ReliabilityModel.predict

ReliabilityModel.predict(X)[source]

Score new samples for reliability (one row per sample).

Applies the references learned by fit() (applicability domain, ensemble / bootstrap, calibrator, conformal calibration) — no model is refitted here, so repeated calls are cheap and deterministic. Each column maps to one axis of the mental model: score_std / ci_* (stability), ood_score / in_domain / ad_* (applicability domain, banded by ad_status), margin / entropy (score ambiguity), conformal_set (validity), and reliable (the headline flag).

Changed in version 1.2.0: The applicability-domain column ad_knn_dist is named ad_knn, matching its ad_mahalanobis / ad_leverage siblings, and two columns are appended at the end of the table: ad_status (the banded domain verdict) and ad_nearest_train (the closest training row). Apart from the ad_knn_dist rename, previously returned columns keep their position and values.

Parameters:

X (array-like, shape (n_samples, n_features)) – Feature matrix to assess (same feature space as the training X).

Returns:

df_rel – One row per sample with columns: score, score_std, ci_low, ci_high, ood_score, in_domain, ad_knn, ad_mahalanobis, ad_leverage, score_calibrated, margin, entropy, conformal_set, reliable, ad_status, ad_nearest_train.

  • score is the mean positive-class probability across model members; score_std and ci_low / ci_high are its across-member spread and confidence interval.

  • ood_score is the k-nearest-neighbor distance divided by ad_threshold_ when that threshold is positive (otherwise NaN); ad_knn is the raw mean distance. ad_mahalanobis and ad_leverage are auxiliary distance diagnostics (both are NaN when features equal or outnumber samples).

  • score_calibrated is the calibrated positive-class probability (NaN when no calibrator is available); margin and entropy quantify its decisiveness.

  • conformal_set is the split-conformal set ('pos', 'neg', 'both', or 'none'). reliable requires an in-domain singleton, or an in-domain margin of at least 0.5 when no conformal reference is available.

  • ad_status (str, never null): 'inside' (ood_score <= 1), 'borderline' (1 < ood_score <= 1 + ad_borderline), 'outside' (above the band), or 'unknown' (degenerate training reference, ood_score is NaN). in_domain equals ad_status == 'inside' on every row.

  • ad_nearest_train (int): 0-based row index, into the X passed to fit(), of the closest training sample.

Added in version 1.2.0: The ad_status and ad_nearest_train columns (appended at the end).

Return type:

pd.DataFrame

Raises:

  • RuntimeError – If called before fit().

  • ValueError – If X is invalid or has a different number of features than the training data.

Examples

ReliabilityModel().predict() returns the score and the signals that decide whether to believe it, one row per sample. The columns map onto the two trust axes:

  • stability (epistemic): score_std, ci_low / ci_high

  • applicability domain (epistemic / out-of-distribution): ood_score, in_domain, ad_knn / ad_mahalanobis / ad_leverage, and the last two columns ad_status (inside / borderline / outside / unknown) and ad_nearest_train (0-based row index of the closest training sample)

  • calibrated sharpness (aleatoric): score_calibrated, margin, entropy

  • validity: conformal_set (pos / neg / both / none)

  • headline: reliable = in-domain and a confident conformal singleton (or, when no conformal reference is available, a margin of at least 0.5)

import aaanalysis as aa
import numpy as np
from sklearn.datasets import make_classification
aa.options["verbose"] = False
# ``X`` is any feature matrix — e.g. a CPP feature matrix from
# ``SequenceFeature.feature_matrix`` — and ``labels`` are binary. A compact synthetic
# stand-in is used here so the example runs in a second.
X, labels = make_classification(n_samples=140, n_features=10, n_informative=6, random_state=42)
X_train, labels_train, X_new = X[:110], labels[:110], X[110:]

rm = aa.ReliabilityModel(random_state=42).fit(X=X_train, labels=labels_train)

Pass a feature matrix in the same space as the training X.

df_rel = rm.predict(X=X_new)
aa.display_df(df_rel, n_rows=10, show_shape=True)

DataFrame shape: (30, 16)
  score score_std ci_low ci_high ood_score in_domain ad_knn ad_mahalanobis ad_leverage score_calibrated margin entropy conformal_set reliable ad_status ad_nearest_train
1 0.162500 0.087913 0.017895 0.307105 0.660842 True 2.087579 2.983001 0.081636 0.000000 1.000000 0.000000 neg True inside 106
2 0.604500 0.114868 0.415558 0.793442 0.784138 True 2.477065 3.229642 0.095694 0.569444 0.138889 0.986040 both False inside 18
3 0.299500 0.093727 0.145333 0.453667 0.832062 True 2.628456 3.700983 0.125663 0.185606 0.628788 0.692189 neg True inside 16
4 0.599500 0.123712 0.396011 0.802989 0.563890 True 1.781309 1.640368 0.024686 0.547222 0.094444 0.993556 both False inside 72
5 0.210000 0.096488 0.051291 0.368709 0.851984 True 2.691390 4.028899 0.148918 0.000000 1.000000 0.000000 neg True inside 36
6 0.198500 0.108548 0.019954 0.377046 0.816676 True 2.579853 3.254185 0.097154 0.000000 1.000000 0.000000 neg True inside 48
7 0.943000 0.035511 0.884590 1.000000 0.650111 True 2.053680 2.494305 0.057079 1.000000 1.000000 0.000000 pos True inside 80
8 0.905500 0.075728 0.780938 1.000000 0.636056 True 2.009280 2.599997 0.062018 1.000000 1.000000 0.000000 pos True inside 105
9 0.248500 0.085398 0.108033 0.388967 0.703901 True 2.223600 2.983913 0.081686 0.227273 0.545455 0.773227 both False inside 74
10 0.586500 0.120427 0.388415 0.784585 0.807499 True 2.550861 3.045127 0.085072 0.513889 0.027778 0.999443 both False inside 50

Score is not trust. Take a training positive and push it far from the data: it still looks positive, but it is now out-of-distribution (in_domain=False, ood_score > 1), so reliable=False — a high score on an input the model never saw is extrapolation, not confidence.

X_ood = (X_train[labels_train == 1][0] + 20.0)[None, :]
aa.display_df(rm.predict(X=X_ood), n_rows=10, show_shape=True)

DataFrame shape: (1, 16)
  score score_std ci_low ci_high ood_score in_domain ad_knn ad_mahalanobis ad_leverage score_calibrated margin entropy conformal_set reliable ad_status ad_nearest_train
1 0.482500 0.096482 0.323801 0.641199 13.887164 False 43.869081 21187.420026 448904749.374478 0.513889 0.027778 0.999443 both False outside 15

Banded domain verdict. in_domain is a plain yes/no; ad_status says how far outside a sample is. With ad_borderline=0.25 set on fit, a sample is inside if ood_score <= 1, borderline if 1 < ood_score <= 1.25, and outside beyond that (unknown only appears when the training data has no usable spread). in_domain is always ad_status == "inside". ad_nearest_train points to the most similar 0-based training row, a direct handle to inspect why a sample is (or is not) familiar. Here held-out samples are stretched away from the data to populate all three bands.

rm = aa.ReliabilityModel(random_state=42).fit(X=X_train, labels=labels_train, ad_borderline=0.25)
X_shift = np.vstack([X_new[:4], X_new[:4] * 1.4, X_new[:4] * 2.0])
df_rel = rm.predict(X=X_shift)
cols = ["ood_score", "ad_knn", "in_domain", "ad_status", "ad_nearest_train"]
aa.display_df(df_rel[cols], n_rows=12, show_shape=True)

DataFrame shape: (12, 5)
  ood_score ad_knn in_domain ad_status ad_nearest_train
1 0.660842 2.087579 True inside 106
2 0.784138 2.477065 True inside 18
3 0.832062 2.628456 True inside 16
4 0.563890 1.781309 True inside 72
5 1.030476 3.255238 False borderline 106
6 1.053593 3.328264 False borderline 18
7 1.159855 3.663943 False borderline 16
8 0.641815 2.027472 True inside 40
9 1.692150 5.345445 False outside 106
10 1.571221 4.963435 False outside 18
11 1.699758 5.369479 False outside 39
12 0.886400 2.800108 True inside 40

ood_score is ad_knn divided by the fitted threshold rm.ad_threshold_, and the status counts show how the band splits the out-of-domain samples.

print(f"ad_threshold_ = {rm.ad_threshold_:.3f} ({rm.ad_method_}); identity holds: "
      f"{np.allclose(df_rel['ood_score'], df_rel['ad_knn'] / rm.ad_threshold_)}")
df_counts = df_rel["ad_status"].value_counts().rename_axis("ad_status").reset_index()
aa.display_df(df_counts, n_rows=10, show_shape=True)

ad_threshold_ = 3.159 (knn); identity holds: True
DataFrame shape: (3, 2)
  ad_status count
1 inside 6
2 borderline 3
3 outside 3