ReliabilityModel.predict

ReliabilityModel.predict(X)[source]

Score new samples for reliability (one row per sample).

Applies the references learned by fit() (applicability domain, ensemble / bootstrap, calibrator, conformal calibration) — no model is refitted here, so repeated calls are cheap and deterministic. Each column maps to one axis of the mental model: score_std / ci_* (stability), ood_score / in_domain / ad_* (applicability domain), margin / entropy (calibrated ambiguity), conformal_set (validity), and reliable (the headline flag).

Parameters:

X (array-like, shape (n_samples, n_features)) – Feature matrix to assess (same feature space as the training X).

Returns:

df_rel – One row per sample with columns: score, score_std, ci_low, ci_high, ood_score, in_domain, ad_knn_dist, ad_mahalanobis, ad_leverage, score_calibrated, margin, entropy, conformal_set, reliable.

Return type:

pd.DataFrame

Raises:

Examples

ReliabilityModel().predict() returns the score and the signals that decide whether to believe it, one row per sample. The columns map onto the two trust axes:

  • stability (epistemic): score_std, ci_low / ci_high

  • applicability domain (epistemic / out-of-distribution): ood_score, in_domain, ad_knn_dist / ad_mahalanobis / ad_leverage

  • calibrated sharpness (aleatoric): score_calibrated, margin, entropy

  • validity: conformal_set (pos / neg / both / none)

  • headline: reliable = in-domain and a confident conformal singleton

import aaanalysis as aa
import numpy as np
from sklearn.datasets import make_classification
aa.options["verbose"] = False
# ``X`` is any feature matrix — e.g. a CPP feature matrix from
# ``SequenceFeature.feature_matrix`` — and ``labels`` are binary. A compact synthetic
# stand-in is used here so the example runs in a second.
X, labels = make_classification(n_samples=140, n_features=10, n_informative=6, random_state=42)
X_train, labels_train, X_new = X[:110], labels[:110], X[110:]

rm = aa.ReliabilityModel(random_state=42).fit(X=X_train, labels=labels_train)

Pass a feature matrix in the same space as the training X.

df_rel = rm.predict(X=X_new)
aa.display_df(df_rel, n_rows=10, show_shape=True)

DataFrame shape: (30, 14)
  score score_std ci_low ci_high ood_score in_domain ad_knn_dist ad_mahalanobis ad_leverage score_calibrated margin entropy conformal_set reliable
1 0.162500 0.087913 0.017895 0.307105 0.660842 True 2.087579 2.983001 0.081636 0.000000 1.000000 0.000000 neg True
2 0.604500 0.114868 0.415558 0.793442 0.784138 True 2.477065 3.229642 0.095694 0.569444 0.138889 0.986040 both False
3 0.299500 0.093727 0.145333 0.453667 0.832062 True 2.628456 3.700983 0.125663 0.185606 0.628788 0.692189 neg True
4 0.599500 0.123712 0.396011 0.802989 0.563890 True 1.781309 1.640368 0.024686 0.547222 0.094444 0.993556 both False
5 0.210000 0.096488 0.051291 0.368709 0.851984 True 2.691390 4.028899 0.148918 0.000000 1.000000 0.000000 neg True
6 0.198500 0.108548 0.019954 0.377046 0.816676 True 2.579853 3.254185 0.097154 0.000000 1.000000 0.000000 neg True
7 0.943000 0.035511 0.884590 1.000000 0.650111 True 2.053680 2.494305 0.057079 1.000000 1.000000 0.000000 pos True
8 0.905500 0.075728 0.780938 1.000000 0.636056 True 2.009280 2.599997 0.062018 1.000000 1.000000 0.000000 pos True
9 0.248500 0.085398 0.108033 0.388967 0.703901 True 2.223600 2.983913 0.081686 0.227273 0.545455 0.773227 both False
10 0.586500 0.120427 0.388415 0.784585 0.807499 True 2.550861 3.045127 0.085072 0.513889 0.027778 0.999443 both False

Score is not trust. Take a training positive and push it far from the data: it still looks positive, but it is now out-of-distribution (in_domain=False, ood_score > 1), so reliable=False — a high score on an input the model never saw is extrapolation, not confidence.

X_ood = (X_train[labels_train == 1][0] + 20.0)[None, :]
aa.display_df(rm.predict(X=X_ood), n_rows=10, show_shape=True)

DataFrame shape: (1, 14)
  score score_std ci_low ci_high ood_score in_domain ad_knn_dist ad_mahalanobis ad_leverage score_calibrated margin entropy conformal_set reliable
1 0.482500 0.096482 0.323801 0.641199 13.887164 False 43.869081 21187.420026 448904749.374478 0.513889 0.027778 0.999443 both False