ReliabilityModel

class ReliabilityModel(verbose=True, random_state=None)[source]

Bases: Wrapper

Assess how much to trust each prediction — the reliability of a score, not the score itself.

A high score is not the same as a trustworthy one: a model can be right to call a case a 0.55 toss-up, and badly wrong to call an input it has never seen a 1.0. Reporting only the score hides this. ReliabilityModel returns the score together with the answer to three plain questions — the categories that decide trust:

Has the model seen anything like this before? If the input is unlike the training data, the model is guessing, and the score cannot be trusted no matter how high it is. (the “applicability domain” — ``ood_score`` / ``in_domain``.)
Do repeated models agree? If an ensemble, or the same model refit on resampled data, disagree, the score is shaky. (stability — ``score_std``, ``ci_low`` / ``ci_high``.)
Is the case clear-cut, or a genuine toss-up? Even a familiar, agreed-on case can be a real 50/50; a well-calibrated score near 0.5 means “honestly borderline,” not “broken.” (decisiveness — ``margin`` / ``entropy``; the conformal set can also **abstain*.)*

So the two classic failures separate cleanly:

case	seen before?	clear-cut?	verdict
confident about `0.55`	yes	no (real toss-up)	trust it as “borderline”
worthless about `1.0`	no (out-of-distribution)	–	do not trust, at any score

The headline flag reliable = familiar and decisive (in_domain and a confident conformal singleton). It wraps an already-fitted binary predictor (an AAPred, a TreeModel, or any scikit-learn classifier) plus its training data and adds nothing but reliability — the prediction itself stays with the model.

In uncertainty-quantification terms, questions 1-2 are epistemic uncertainty (the model’s reducible lack of knowledge) and question 3 is aleatoric uncertainty (irreducible ambiguity in the data) [Huellermeier21]. The applicability domain follows the QSAR idea [Sahigara12] (features here are a descriptor space), calibration follows [Guo17] (a raw predict_proba is not a confidence until calibrated), and the conformal set follows [Angelopoulos23].

Added in version 1.1.0.

Notes

Scope. Binary classification only. ad_mahalanobis / ad_leverage are auxiliary diagnostics that need more training samples than features (they are NaN otherwise); the in_domain decision uses the robust k-NN distance and is always valid.
``score`` is the member mean — the ensemble average, or the bootstrap (“bagged”) average for a single model — so it is always the centre of [ci_low, ci_high]. Set n_bootstrap=0 to report a single model’s own probability instead (then score_std is 0).
``reliable`` is conformal-based (in_domain and a confident conformal singleton), whereas margin / entropy are a separate calibrated-sharpness readout — they can disagree on a borderline case.
Calibration affects score_calibrated / margin / entropy only; score (and ReliabilityModelPlot.reliability_diagram()) stay on the reported model score. For a passed ensemble, the calibrator and conformal reference are built from its first member.
Reproducibility. The bootstrap, calibration split, and conformal split are stochastic — set random_state for identical output across fits.
All fitted-state attributes carry a trailing underscore and are set by fit().

`eval`([X, labels, n_bins])	Reliability diagnostics: calibration curve, empirical conformal coverage, in-domain rate.
`fit`(X, labels[, model, label_pos, k, ...])	Fit the reliability reference from a (fitted or default) model and its training data.
`predict`(X)	Score new samples for reliability (one row per sample).