Practical overview
When it fits
Use Audit AI when you have model predictions or assessment results and demographic labels, and want to examine group differences as part of model evaluation. It is especially relevant when pass rates, decision thresholds, or comparisons across time and regions matter.
Who it is for
- Data scientists evaluating models used in socially sensitive decisions.
- Practitioners who need to compare assessment or prediction outcomes across demographic groups.
- Teams using pandas and scikit-learn who want fairness checks within a Python analysis workflow.
Documented uses
- Compare pass rates between demographic groups using a bias ratio.
- Apply statistical tests to assess differences in classification or regression outcomes.
- Inspect how test results change at different prediction thresholds.
- Check for differences across time or regions using the described Cochran–Mantel–Haenszel test.
Requirements & limits
The project is an open-source Python library with an MIT license. Its README documents installation with pip and lists Linux, macOS, and Windows as supported platforms.
- The documented checks cover both practical measures such as the four-fifths ratio and statistical tests; these answer different questions.
- The project says significance thresholds depend on the field and use case, so results require context-specific interpretation.
- The documented workflow expects demographic labels and model results, including prediction probabilities in the provided classification example.
- The README identifies pandas and scikit-learn as core dependencies and lists NumPy, SciPy, and pandas as installation requirements.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Run a demographic bias check on model predictions
Pass demographic labels and model prediction results to Audit AI’s documented bias-check function.
Before you start
- Python environment with Audit AI installed.
- pandas, NumPy, SciPy, and scikit-learn dependencies.
- A dataframe containing model features and demographic labels, plus a model that provides prediction probabilities.
Steps
Install the package with the command below.
pip install audit-AISelect the feature columns from the dataframe and obtain prediction probabilities from the model.
Import
bias_test_checkfromauditai.miscand call it with the demographic labels, prediction results, and a category name.
Expected outcome
The documented example returns results for checks including the four-fifths test, Fisher and chi-squared p-values, a z-test, and a Bayes factor.
Guide limitations
- The README’s example uses a gender label and classification probabilities; it does not provide a complete dataset or model-training procedure.
- Significance thresholds depend on the field and use case, and identifying potential bias is not itself a remedy for discrimination.
Plot bias tests across prediction thresholds
Create a visualization of documented bias-test results at different thresholds.
Before you start
- Python environment with Audit AI installed.
- pandas, NumPy, SciPy, scikit-learn, matplotlib, and seaborn.
- A dataframe with feature columns and demographic labels, plus a model that provides prediction probabilities.
Steps
Install the package with the command below.
pip install audit-AISelect model features and obtain prediction probabilities.
Import
plot_threshold_testsfromauditai.vizand pass the demographic labels, results, and category name.
Expected outcome
The function is documented to produce a plot of different tests across thresholds.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 12 October 2026
- Original website title
- pymetrics/audit-ai · Official homepage title · retained in its source language