ReliabilityModel

class ReliabilityModel(*, verbose=True, random_state=None)[source]

Assess how much to trust each prediction — the reliability of a score, not the score itself.

A high score is not the same as a trustworthy one: a model can be right to call a case a 0.55 toss-up, and badly wrong to call an input it has never seen a 1.0. Reporting only the score hides this. ReliabilityModel returns the score together with the answer to three plain questions — the categories that decide trust:

  1. Has the model seen anything like this before? If the input is unlike the training data, the model is guessing, and the score cannot be trusted no matter how high it is. (the “applicability domain” — ``ood_score`` / ``in_domain``.)

  2. Do repeated models agree? If an ensemble, or the same model refit on resampled data, disagree, the score is shaky. (stability — ``score_std``, ``ci_low`` / ``ci_high``.)

  3. Is the case clear-cut, or a genuine toss-up? Even a familiar, agreed-on case can be a real 50/50; a well-calibrated score near 0.5 means “honestly borderline,” not “broken.” (decisiveness — ``margin`` / ``entropy``; the conformal set can also **abstain*.)*

So the two classic failures separate cleanly:

case

seen before?

clear-cut?

verdict

confident about 0.55

yes

no (real toss-up)

trust it as “borderline”

worthless about 1.0

no (out-of-distribution)

–

do not trust, at any score

The headline flag reliable = familiar and decisive (in_domain and a confident conformal singleton, or margin >= 0.5 when no conformal reference is available). It wraps an already-fitted binary predictor (an AAPred, a TreeModel, or any scikit-learn classifier) plus its training data and adds nothing but reliability — the prediction itself stays with the model.

In uncertainty-quantification terms, questions 1-2 are epistemic uncertainty (the model’s reducible lack of knowledge) and question 3 is aleatoric uncertainty (irreducible ambiguity in the data) [Huellermeier21]. The applicability domain follows the QSAR idea [Sahigara12] (features here are a descriptor space), calibration follows [Guo17] (a raw predict_proba is not a confidence until calibrated), and the conformal set follows [Angelopoulos23].

Warning

Experimental. This class is part of the new v1.1.0 prediction layer and is under active development; its API (signatures, defaults, return objects) may change between minor releases without the usual deprecation cycle. Pin a version if you depend on the current behaviour.

Added in version 1.1.0.

Changed in version 1.2.0: Adds a banded applicability-domain verdict and exposes the fitted k-nearest-neighbor boundary and its decision rule.

Notes

  • Scope. Binary classification only. ad_mahalanobis / ad_leverage are auxiliary diagnostics that need more training samples than features (they are NaN otherwise); the in_domain decision uses the robust k-NN distance and is always valid.

  • ``score`` is the member mean — the ensemble average, or the bootstrap (“bagged”) average for a single model — so it is always the centre of [ci_low, ci_high]. Set n_bootstrap=0 to report a single model’s own probability instead (then score_std is 0).

  • ``score_std`` is a per-sample spread across members. Here it is the standard deviation of one sample’s probability across the ensemble members (or bootstrap resamples). The same-named column in AAPred.eval() and ModelEvaluator.run() means something different: the standard deviation of a metric across cross-validation folds.

  • ``reliable`` is conformal-based (in_domain and a confident conformal singleton) when a conformal reference is available; otherwise it uses in_domain and margin >= 0.5. margin / entropy can therefore disagree with it on a borderline case.

  • Calibration affects score_calibrated / margin / entropy only; when no calibrator is available, the latter two use score. score stays the reported model score. Raw scoring is the default everywhere: eval() bins score and ReliabilityModelPlot.reliability_diagram() draws that curve, whereas eval(use_calibrated=True) returns the calibrated table, from which the same method draws the calibrated curve (both can share one ax). If calibrate=True but no calibrator can be fitted (e.g. a class with a single member, or a model that cannot be cloned), fit() warns, score_calibrated is NaN, and eval(use_calibrated=True) raises naming that reason. For a passed ensemble, the calibrator and conformal reference are built from its first member.

  • Reproducibility. The bootstrap, calibration split, and conformal split are stochastic — set random_state for identical output across fits.

  • Banded domain verdict. ad_status refines the bool in_domain into inside (ood_score <= 1), borderline (1 < ood_score <= 1 + ad_borderline), outside (above the band), and unknown (the training reference has no usable spread, so ood_score is NaN). in_domain always equals ad_status == "inside".

  • All fitted-state attributes carry a trailing underscore and are set by fit().

model_

The assessed model (the fitted single estimator or the list of ensemble members).

Type:

estimator or list of estimators

label_pos_

Positive-class label whose probability is scored.

Type:

int

ad_threshold_

Applicability-domain boundary: the ad_percentile-th percentile of the training samples’ mean distance to their k nearest other training samples (standardized feature space). ood_score is ad_knn / ad_threshold_, so ood_score == 1 is the boundary. A non-positive value marks a degenerate reference (ad_status='unknown').

Added in version 1.2.0.

Type:

float

ad_method_

Decision rule behind ood_score / in_domain / ad_status; always "knn".

Added in version 1.2.0.

Type:

str

See also

Parameters:

Methods

eval(*[, X, labels, n_bins, use_calibrated, ...])

Reliability diagnostics: calibration curve, empirical conformal coverage, in-domain rate.

fit(X, labels, *[, model, label_pos, k, ...])

Fit the reliability reference from a (fitted or default) model and its training data.

predict(X)

Score new samples for reliability (one row per sample).

predict_candidates(df_cand, df_seq, features, *)

Score a set of design candidates for reliability, rebuilding their feature matrix first.

__init__(*, verbose=True, random_state=None)[source]
Parameters:
  • verbose (bool, default=True) – If True, verbose outputs are enabled.

  • random_state (int, optional) – Seed for the bootstrap, calibration split, and conformal split. If None, those stochastic steps use a fresh random state on each fit.

Raises:

ValueError – If verbose is not a bool or random_state is not an integer or None.