ReliabilityModel.fit

ReliabilityModel.fit(X, labels, model=None, label_pos=1, k=5, ad_percentile=95.0, ci=90.0, n_bootstrap=20, calibrate=True, calibration_method='isotonic', conformal_alpha=0.1)[source]

Fit the reliability reference from a (fitted or default) model and its training data.

Learns, once, everything predict() needs: the applicability-domain reference, the ensemble / bootstrap source of uncertainty, an optional probability calibrator, and the split-conformal calibration.

Parameters:

  • X (array-like, shape (n_samples, n_features)) – Training feature matrix the model was fitted on (the applicability-domain reference).

  • labels (array-like, shape (n_samples,)) – Binary training labels (exactly two classes).

  • model (estimator, list of estimators, AAPred, or None) – A fitted scikit-learn classifier (predict_proba), a list of fitted estimators (ensemble; uncertainty = their disagreement), a fitted AAPred, or None to fit a default RandomForestClassifier.

  • label_pos (int, default=1) – Positive-class label whose probability is scored.

  • k (int, default=5) – Number of nearest training neighbors for the applicability-domain distance.

  • ad_percentile (float, default=95.0) – Training kNN-distance percentile used as the in_domain boundary.

  • ci (float, default=90.0) – Central width (percent) of the reported score confidence interval.

  • n_bootstrap (int, default=20) – Bootstrap resamples for uncertainty when model is a single estimator (not an ensemble); score is then the bagged mean over the resamples (see Notes). 0 disables the bootstrap and reports the model’s own probability (score_std = 0).

  • calibrate (bool, default=True) – Fit a probability calibrator (needed for meaningful margin / entropy).

  • calibration_method (str, default="isotonic") – "isotonic" or "sigmoid" (Platt), passed to CalibratedClassifierCV.

  • conformal_alpha (float, default=0.1) – Miscoverage level of the split-conformal set (1 - alpha coverage).

Returns:

The fitted instance.

Return type:

ReliabilityModel

Raises:

ValueError – If labels are not binary, label_pos is absent from labels, model is an empty list or lacks predict_proba, a passed AAPred is not fitted, or a numeric parameter is out of range.

Examples

A prediction score and its trustworthiness are different things. A model can be confident that a case is a 0.55 coin-flip (a real, in-distribution ambiguity) and worthless on a 1.0 for an input unlike anything it was trained on. ReliabilityModel returns the score together with the signals that decide whether to believe it — along two independent axes (the aleatoric vs. epistemic distinction):

axis

what it means

measures

aleatoric

irreducible, in-distribution ambiguity (a genuine 0.55)

margin, entropy on a calibrated score

epistemic

the model’s lack of knowledge — out-of-distribution / sparse data

ood_score / in_domain (applicability domain) + score_std (stability)

So confident-about-0.55 (low epistemic, high aleatoric) is trustworthy as “ambiguous”, while worthless-about-1.0 (high epistemic / out-of-distribution) is not trustworthy at any score. The headline flag reliable = in-domain and a confident conformal singleton.

Workflow: fit (learn the references) -> predict (score new samples) -> eval (calibration + coverage diagnostics) -> :class:~aaanalysis.ReliabilityModelPlot. This notebook covers fit; the others cover the rest.

import aaanalysis as aa
import numpy as np
from sklearn.datasets import make_classification
aa.options["verbose"] = False
# ``X`` is any feature matrix — e.g. a CPP feature matrix from
# ``SequenceFeature.feature_matrix`` — and ``labels`` are binary. A compact synthetic
# stand-in is used here so the example runs in a second.
X, labels = make_classification(n_samples=140, n_features=10, n_informative=6, random_state=42)
X_train, labels_train, X_new = X[:110], labels[:110], X[110:]

Simplest form. With model=None a default random forest is fitted and its bootstrap disagreement becomes the uncertainty. verbose and random_state are set on the constructor (random_state makes the bootstrap / calibration / conformal splits reproducible).

rm = aa.ReliabilityModel(random_state=42, verbose=False).fit(X=X_train, labels=labels_train)

Bring your own model. Pass a fitted scikit-learn classifier, a fitted :class:~aaanalysis.AAPred, or a list of fitted estimators as model — a list uses the members’ disagreement as the uncertainty instead of the bootstrap.

from sklearn.ensemble import RandomForestClassifier
models = [RandomForestClassifier(n_estimators=50, random_state=i).fit(X_train, labels_train)
          for i in range(4)]
rm = aa.ReliabilityModel(random_state=42).fit(X=X_train, labels=labels_train, model=models)

Every parameter, grouped. The remaining parameters tune each axis:

  • label_pos — the positive class whose probability is scored

  • applicability domain: k (nearest neighbors), ad_percentile (in-domain boundary)

  • stability: ci (interval width), n_bootstrap (resamples for a single-model uncertainty)

  • calibration: calibrate, calibration_method ("isotonic" or "sigmoid")

  • validity: conformal_alpha (the 1 - alpha conformal coverage level)

rm = aa.ReliabilityModel(random_state=42).fit(
    X=X_train, labels=labels_train, model=None, label_pos=1,
    k=5, ad_percentile=95,                       # applicability domain
    ci=90, n_bootstrap=20,                        # stability
    calibrate=True, calibration_method="isotonic",  # calibration
    conformal_alpha=0.1)                          # conformal validity