ReliabilityModel.eval

ReliabilityModel.eval(X=None, labels=None, n_bins=5)

Reliability diagnostics: calibration curve, empirical conformal coverage, in-domain rate.

Aggregates predict() over a labeled evaluation set into a compact table — per-bin predicted-vs-empirical positive rate (how well calibrated the score is) plus a summary row with the fraction in the applicability domain and the empirical coverage of the conformal sets (which should track 1 - conformal_alpha).

Parameters:

• X (array-like, optional) – Evaluation features; the training X is used if None (a held-out labeled set gives an honest estimate — calibration and conformal were fit on the training data).

• labels (array-like, optional) – Evaluation labels; the training labels are used if None.

• n_bins (int, default=5) – Number of equal-width score bins for the calibration curve.

Returns:

df_eval – Per-bin rows (bin, mean_score, empirical_pos, n) plus a summary row with the in-domain fraction (mean_score) and the empirical conformal coverage (empirical_pos).

Return type:

pd.DataFrame

Raises:

RuntimeError – If called before fit().

Examples

ReliabilityModel().eval() returns reliability diagnostics: a per-bin calibration curve (predicted vs. empirical positive rate) plus a summary row with the in-domain fraction and the empirical conformal coverage.

import aaanalysis as aa
import numpy as np
from sklearn.datasets import make_classification
aa.options["verbose"] = False
# ``X`` is any feature matrix — e.g. a CPP feature matrix from
# ``SequenceFeature.feature_matrix`` — and ``labels`` are binary. A compact synthetic
# stand-in is used here so the example runs in a second.
X, labels = make_classification(n_samples=140, n_features=10, n_informative=6, random_state=42)
X_train, labels_train, X_new = X[:110], labels[:110], X[110:]

rm = aa.ReliabilityModel(random_state=42).fit(X=X_train, labels=labels_train)

Evaluate on a labeled set (the training data is used if X / labels are omitted); n_bins sets the number of calibration bins.

df_eval = rm.eval(X=X_new, labels=labels[110:], n_bins=5)
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (6, 4)

	bin	mean_score	empirical_pos	n
1	0.00-0.20	0.146750	0.000000	4
2	0.20-0.40	0.286500	0.000000	9
3	0.40-0.60	0.515750	0.666667	6
4	0.60-0.80	0.688300	0.800000	5
5	0.80-1.00	0.869000	1.000000	6
6	summary	0.900000	1.000000	30