OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

Nannyml

NannyML is an open-source Python library for estimating deployed machine-learning model performance, detecting data drift and relating drift alerts to performance changes.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

Nannyml — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use NannyML when you have reference data and later model data to analyze, and need performance estimates or drift analysis before outcome labels are available.

Who it is for

  • Data scientists monitoring deployed tabular classification or regression models.
  • Teams investigating model behavior while ground-truth labels are delayed or unavailable.

Documented uses

  • Estimate post-deployment model performance from model predictions and data.
  • Detect univariate or multivariate data drift in tabular workloads.
  • Rank features with drift alerts and compare selected drift findings with estimated performance.
  • Plot monitoring results for inspection.
Before you choose

Requirements & limits

The project is open source under the Apache-2.0 license. The README documents pip and Conda installation.

  • The documented quick start separates reference data used to fit the estimator from analysis data used for estimation.
  • NannyML identifies confidence-based performance estimation, direct loss estimation and PCA-based reconstruction for multivariate drift detection.
  • The library depends on LightGBM, which may require operating-system-specific binaries.
  • Database connections for reading model inputs or outputs, or exporting monitoring results to a database, require the optional db dependency.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Estimate performance on a later classification dataset

Use a reference dataset to fit a confidence-based estimator, then estimate performance on analysis data and plot the result.

Before you start

  • Python with NannyML and pandas installed.
  • Reference and analysis dataframes with the prediction, predicted-probability and target columns used by the estimator.
  • A chunk size; the example uses 5000.
  • NannyML's LightGBM dependency; installation may require OS-specific binaries.

Steps

  1. Load or prepare the reference and analysis dataframes.

  2. Configure nml.CBPE for binary classification, specify the prediction, predicted-probability and target columns, select a metric such as roc_auc, and set the chunk size.

  3. Fit the estimator on the reference dataframe and call estimate with the analysis dataframe.

  4. Plot the estimated performance results and display the figure.

Expected outcome

A plotted estimate of the selected classification-performance metric for the analysis data.

Guide limitations

  • The quick-start example covers binary classification; it does not provide the full workflow for other problem types.
  • The example specifies a target column, while NannyML's stated post-deployment purpose is to estimate performance when analysis-period targets are unavailable.
  • LightGBM may require OS-specific binaries.
Documentation 1 ↗
Source-based practical guide

Calculate and compare univariate drift with estimated performance

Calculate feature-level drift on analysis data, inspect the highest-alert features and compare a selected feature's drift with estimated model performance.

Before you start

  • Reference and analysis dataframes, a feature-column list and a chosen chunk size.
  • NannyML installed, with the LightGBM dependency described in the README.

Steps

  1. Initialize nml.UnivariateDriftCalculator with the feature columns and chunk size.

  2. Fit the calculator on the reference dataframe and calculate drift on the analysis dataframe.

  3. Rank the drift results with nml.AlertCountRanker and plot selected features.

  4. Filter the drift result for a selected feature in the analysis period, compare it with the estimated-performance result, and plot the comparison.

Expected outcome

Feature-level drift results, ranked features and a plot comparing selected drift with estimated performance.

Guide limitations

  • The README example assumes an estimated-performance result is already available for the comparison.
  • This recipe covers univariate drift; it does not specify a complete workflow for multivariate drift.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗
Source record details
Source availability
Active
Sources checked
13 October 2026
Original website title
NannyML/nannyml · Official homepage title · retained in its source language
How we review sourcesSuggest a correction