ReliabilityModel.eval

ReliabilityModel.eval(*, X=None, labels=None, n_bins=5, use_calibrated=False, add_metrics=False)[source]

Reliability diagnostics: calibration curve, empirical conformal coverage, in-domain rate.

Aggregates predict() over a labeled evaluation set into a compact table — per-bin predicted-vs-empirical positive rate (how well calibrated the score is) plus a summary row with the fraction in the applicability domain and the empirical coverage of the conformal sets (which should track 1 - conformal_alpha). Optionally, the calibrated score is binned instead of the raw one, and the Brier score and expected calibration error (ECE) [Guo17] are added as scalar rows, so two calibrations can be compared by number.

Changed in version 1.2.0: The per-bin sample-count column n is named n_samples. Adds optional calibrated scoring and scalar Brier-score / expected-calibration-error rows; passing X without labels now raises instead of evaluating the training labels silently.

Parameters:

  • X (array-like, shape (n_samples, n_features), optional) – Evaluation features. If X and labels are both None, use the training features; passing a held-out labeled set gives an honest estimate because calibration and conformal were fit on the training data.

  • labels (array-like of int, shape (n_samples,), optional) – Evaluation labels, matching X in length; the training labels are used when both X and labels are None. Labels given with X=None are scored against the training features and must match them in length. Passing X without labels raises.

  • n_bins (int, default=5) – Number of equal-width score bins for the calibration curve (and the ECE). Must be an integer of at least 2.

  • use_calibrated (bool, default=False) –

    If True, score the calibrated column (score_calibrated) instead of the raw score: the bins, Brier score, and ECE then describe the calibrated probability. If False, score the raw model probability. If True, an available calibrator is required: fit with calibrate=True and ensure calibration did not fail (fit() warns when it does).

    Added in version 1.2.0.

  • add_metrics (bool, default=False) –

    If True, append two scalar rows for the scored column: the Brier score (bin='brier', mean squared difference between score and label) and the ECE (bin='ece', the n_samples-weighted mean of |mean_score - empirical_pos| over the n_bins bins). Lower is better for both. If False, return only the per-bin and summary rows.

    Added in version 1.2.0.

Returns:

df_eval – Per-bin rows (bin, mean_score, empirical_pos, n_samples) plus a summary row (bin='summary') with the in-domain fraction (mean_score), the empirical conformal coverage (empirical_pos), and the number of evaluated samples (n_samples). With add_metrics=True, a 'brier' and an 'ece' row follow, each holding its value in mean_score (empirical_pos is NaN, n_samples is the number of evaluated samples).

Return type:

pd.DataFrame

Raises:

  • RuntimeError – If called before fit().

  • ValueError – If X is given without labels, X or labels is invalid, their lengths differ, labels holds a value not observed during fit(), n_bins is not an integer of at least 2, use_calibrated or add_metrics is not a bool, or use_calibrated=True while no probability calibrator is available: either the model was fitted with calibrate=False, or its calibration failed (the message names which).

Notes

  • With both parameters left at their defaults, the output is the raw-score table of earlier versions, unchanged.

  • Scoring the calibrated column on the training data flatters the calibrator (it was fit there); pass a held-out X / labels to judge whether calibration helps.

Examples

ReliabilityModel().eval() returns reliability diagnostics: a per-bin calibration curve (predicted vs. empirical positive rate) plus a summary row with the in-domain fraction and the empirical conformal coverage.

import aaanalysis as aa
import numpy as np
from sklearn.datasets import make_classification
aa.options["verbose"] = False
# ``X`` is any feature matrix — e.g. a CPP feature matrix from
# ``SequenceFeature.feature_matrix`` — and ``labels`` are binary. A compact synthetic
# stand-in is used here so the example runs in a second.
X, labels = make_classification(n_samples=140, n_features=10, n_informative=6, random_state=42)
X_train, labels_train, X_new = X[:110], labels[:110], X[110:]

rm = aa.ReliabilityModel(random_state=42).fit(X=X_train, labels=labels_train)

Evaluate on a labeled set (the training data is used if X / labels are omitted); n_bins sets the number of calibration bins.

df_eval = rm.eval(X=X_new, labels=labels[110:], n_bins=5)
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (6, 4)
  bin mean_score empirical_pos n_samples
1 0.00-0.20 0.146750 0.000000 4
2 0.20-0.40 0.286500 0.000000 9
3 0.40-0.60 0.515750 0.666667 6
4 0.60-0.80 0.688300 0.800000 5
5 0.80-1.00 0.869000 1.000000 6
6 summary 0.900000 1.000000 30

Set add_metrics=True to append two scalar rows: the Brier score (bin='brier') and the expected calibration error (bin='ece'), each stored in mean_score. With use_calibrated=True the bins and both metrics describe the calibrated probability (score_calibrated) instead of the raw score, so the two tables answer whether calibration helped. A naive Bayes model on redundant features is strongly over-confident, which makes the effect visible:

from sklearn.naive_bayes import GaussianNB
X, labels = make_classification(n_samples=600, n_features=20, n_informative=3, n_redundant=15,
                                class_sep=0.8, random_state=0)
X_train, labels_train, X_test, labels_test = X[:400], labels[:400], X[400:], labels[400:]
rm = aa.ReliabilityModel(random_state=42).fit(X=X_train, labels=labels_train, model=GaussianNB(),
                                              n_bootstrap=0, calibration_method="isotonic")
df_eval_raw = rm.eval(X=X_test, labels=labels_test, n_bins=5, use_calibrated=False, add_metrics=True)
aa.display_df(df_eval_raw, n_rows=10, show_shape=True)

DataFrame shape: (8, 4)
  bin mean_score empirical_pos n_samples
1 0.00-0.20 0.030605 0.277778 90
2 0.20-0.40 0.311367 0.812500 16
3 0.40-0.60 0.524551 0.800000 10
4 0.60-0.80 0.702804 0.684211 19
5 0.80-1.00 0.964626 0.676923 65
6 summary 0.955000 0.930000 200
7 brier 0.276247 nan 200
8 ece 0.260361 nan 200 
df_eval_cal = rm.eval(X=X_test, labels=labels_test, n_bins=5, use_calibrated=True, add_metrics=True)
aa.display_df(df_eval_cal, n_rows=10, show_shape=True)

DataFrame shape: (8, 4)
  bin mean_score empirical_pos n_samples
1 0.00-0.20 0.029580 0.000000 40
2 0.20-0.40 0.287005 0.294118 17
3 0.40-0.60 0.523290 0.645161 31
4 0.60-0.80 0.649868 0.652632 95
5 0.80-1.00 0.956756 0.941176 17
6 summary 0.955000 0.930000 200
7 brier 0.167682 nan 200
8 ece 0.028048 nan 200