ReliabilityModel
- class ReliabilityModel(*, verbose=True, random_state=None)[source]
Assess how much to trust each prediction — the reliability of a score, not the score itself.
A high score is not the same as a trustworthy one: a model can be right to call a case a
0.55toss-up, and badly wrong to call an input it has never seen a1.0. Reporting only the score hides this.ReliabilityModelreturns the score together with the answer to three plain questions — the categories that decide trust:Has the model seen anything like this before? If the input is unlike the training data, the model is guessing, and the score cannot be trusted no matter how high it is. (the “applicability domain” — ``ood_score`` / ``in_domain``.)
Do repeated models agree? If an ensemble, or the same model refit on resampled data, disagree, the score is shaky. (stability — ``score_std``, ``ci_low`` / ``ci_high``.)
Is the case clear-cut, or a genuine toss-up? Even a familiar, agreed-on case can be a real 50/50; a well-calibrated score near
0.5means “honestly borderline,” not “broken.” (decisiveness — ``margin`` / ``entropy``; the conformal set can also **abstain*.)*
So the two classic failures separate cleanly:
case
seen before?
clear-cut?
verdict
confident about
0.55yes
no (real toss-up)
trust it as “borderline”
worthless about
1.0no (out-of-distribution)
–
do not trust, at any score
The headline flag
reliable= familiar and decisive (in_domainand a confident conformal singleton, ormargin >= 0.5when no conformal reference is available). It wraps an already-fitted binary predictor (anAAPred, aTreeModel, or any scikit-learn classifier) plus its training data and adds nothing but reliability — the prediction itself stays with the model.In uncertainty-quantification terms, questions 1-2 are epistemic uncertainty (the model’s reducible lack of knowledge) and question 3 is aleatoric uncertainty (irreducible ambiguity in the data) [Huellermeier21]. The applicability domain follows the QSAR idea [Sahigara12] (features here are a descriptor space), calibration follows [Guo17] (a raw
predict_probais not a confidence until calibrated), and the conformal set follows [Angelopoulos23].Warning
Experimental. This class is part of the new v1.1.0 prediction layer and is under active development; its API (signatures, defaults, return objects) may change between minor releases without the usual deprecation cycle. Pin a version if you depend on the current behaviour.
Added in version 1.1.0.
Changed in version 1.2.0: Adds a banded applicability-domain verdict and exposes the fitted k-nearest-neighbor boundary and its decision rule.
Notes
Scope. Binary classification only.
ad_mahalanobis/ad_leverageare auxiliary diagnostics that need more training samples than features (they areNaNotherwise); thein_domaindecision uses the robust k-NN distance and is always valid.``score`` is the member mean — the ensemble average, or the bootstrap (“bagged”) average for a single model — so it is always the centre of
[ci_low, ci_high]. Setn_bootstrap=0to report a single model’s own probability instead (thenscore_stdis 0).``score_std`` is a per-sample spread across members. Here it is the standard deviation of one sample’s probability across the ensemble members (or bootstrap resamples). The same-named column in
AAPred.eval()andModelEvaluator.run()means something different: the standard deviation of a metric across cross-validation folds.``reliable`` is conformal-based (
in_domainand a confident conformal singleton) when a conformal reference is available; otherwise it usesin_domainandmargin >= 0.5.margin/entropycan therefore disagree with it on a borderline case.Calibration affects
score_calibrated/margin/entropyonly; when no calibrator is available, the latter two usescore.scorestays the reported model score. Raw scoring is the default everywhere:eval()binsscoreandReliabilityModelPlot.reliability_diagram()draws that curve, whereaseval(use_calibrated=True)returns the calibrated table, from which the same method draws the calibrated curve (both can share oneax). Ifcalibrate=Truebut no calibrator can be fitted (e.g. a class with a single member, or a model that cannot be cloned),fit()warns,score_calibratedisNaN, andeval(use_calibrated=True)raises naming that reason. For a passed ensemble, the calibrator and conformal reference are built from its first member.Reproducibility. The bootstrap, calibration split, and conformal split are stochastic — set
random_statefor identical output across fits.Banded domain verdict.
ad_statusrefines the boolin_domainintoinside(ood_score <= 1),borderline(1 < ood_score <= 1 + ad_borderline),outside(above the band), andunknown(the training reference has no usable spread, soood_scoreisNaN).in_domainalways equalsad_status == "inside".All fitted-state attributes carry a trailing underscore and are set by
fit().
- model_
The assessed model (the fitted single estimator or the list of ensemble members).
- Type:
estimator or list of estimators
- ad_threshold_
Applicability-domain boundary: the
ad_percentile-th percentile of the training samples’ mean distance to theirknearest other training samples (standardized feature space).ood_scoreisad_knn / ad_threshold_, soood_score == 1is the boundary. A non-positive value marks a degenerate reference (ad_status='unknown').Added in version 1.2.0.
- Type:
- ad_method_
Decision rule behind
ood_score/in_domain/ad_status; always"knn".Added in version 1.2.0.
- Type:
See also
AAPredfor fitting and deploying the prediction models this class assesses.ReliabilityModelPlotfor visualizing the outputs.
Methods
eval(*[, X, labels, n_bins, use_calibrated, ...])Reliability diagnostics: calibration curve, empirical conformal coverage, in-domain rate.
fit(X, labels, *[, model, label_pos, k, ...])Fit the reliability reference from a (fitted or default) model and its training data.
predict(X)Score new samples for reliability (one row per sample).
predict_candidates(df_cand, df_seq, features, *)Score a set of design candidates for reliability, rebuilding their feature matrix first.
- __init__(*, verbose=True, random_state=None)[source]
- Parameters:
- Raises:
ValueError – If
verboseis not a bool orrandom_stateis not an integer orNone.