ModelEvaluator

class ModelEvaluator(models=None, list_metrics=None, verbose=True, random_state=None)[source]

Bases: Tool

ModelEvaluator: rigorous cross-validated evaluation and paired comparison of models (Tool).

Turns a feature matrix X and labels into an honest performance table rather than a single optimistic hold-out number. run() scores one or more scikit-learn models by repeated stratified cross-validation (multi-seed mean and std over n_cv * n_rounds folds) with percentile bootstrap confidence intervals of the mean; eval() compares two or more models pairwise on the same folds with a signed delta (e.g. ΔMCC) and a two-sided Wilcoxon signed-rank significance test. Both are reproducible under random_state and reuse comp_bootstrap_ci() and scikit-learn metrics — no new dependency.

Where AAPred deploys a fitted model and TreeModel ranks features, ModelEvaluator answers “how well does this model generalize, and is model A really better than model B?” with mean±std, confidence intervals, and a paired significance test.

Warning

Experimental. This class is part of the evolving prediction layer; its API (signatures, defaults, return objects) may change between minor releases without the usual deprecation cycle. Pin a version if you depend on the current behaviour.

Added in version 1.1.0.

Notes

  • All computed-state attributes carry a trailing underscore and are set by run().

  • The per-fold scores held on df_scores_ are what eval() compares, so a comparison never re-runs cross-validation.

See also

  • AAPred for evaluating and deploying prediction models on raw sequences.

  • ModelEvaluatorPlot for visualizing the evaluation and comparison tables.

  • comp_bootstrap_ci() for the percentile bootstrap confidence interval reused here.

Parameters:

Methods

eval([metric, ci, random_state])

Compare the evaluated models pairwise on a single metric over the shared folds.

run(X, labels[, n_cv, n_rounds, metrics, ...])

Evaluate every model by repeated stratified cross-validation with bootstrap CIs.
__init__(models=None, list_metrics=None, verbose=True, random_state=None)[source]
Parameters:

  • models (str, estimator, or list, optional) – Models to evaluate, given as registry name strings (e.g. "svm", "rf"; see aaanalysis.utils.LIST_PRED_MODELS) and/or configured scikit-learn estimator instances, in any mix. Defaults to a single "rf" (RandomForestClassifier). Pass two or more to enable eval() (paired comparison). Each model must implement predict; predict_proba is required only for probability metrics (e.g. roc_auc).

  • list_metrics (list of str, default=["accuracy", "balanced_accuracy", "mcc"]) – Default metrics used by run() when its metrics argument is not given. Each should be one of accuracy, balanced_accuracy, precision, recall, f1, roc_auc, mcc (Matthews correlation coefficient).

  • verbose (bool, default=True) – If True, verbose outputs are enabled.

  • random_state (int, optional) – The seed used by the random number generator. If a positive integer, results of stochastic processes (fold shuffling, bootstrap resampling) are consistent, enabling reproducibility. If None, stochastic processes will be truly random.

Examples

To demonstrate ModelEvaluator, we obtain the DOM_GSEC dataset and its feature matrix:

import aaanalysis as aa
aa.options["verbose"] = False  # Disable verbosity

# DOM_GSEC example dataset + a small feature set (see [Breimann25]_)
df_seq = aa.load_dataset(name="DOM_GSEC")
labels = df_seq["label"].to_list()
df_feat = aa.load_features(name="DOM_GSEC").head(20)

# Build the CPP feature matrix X
sf = aa.SequenceFeature()
df_parts = sf.get_df_parts(df_seq=df_seq)
X = sf.feature_matrix(features=df_feat["feature"], df_parts=df_parts)

Construct a ModelEvaluator with the models to compare, the default list_metrics, verbose, and a random_state for reproducibility:

me = aa.ModelEvaluator(models=["rf", "svm"],
                       list_metrics=["accuracy", "balanced_accuracy", "mcc"],
                       verbose=False,
                       random_state=42)
df_eval = me.run(X, labels)
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (6, 7)

/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
  model metric score score_std ci_low ci_high n_scores
1 rf accuracy 0.809231 0.064659 0.744000 0.843692 5
2 rf balanced_accuracy 0.810256 0.065416 0.744872 0.844231 5
3 rf mcc 0.622289 0.131735 0.491475 0.691125 5
4 svm accuracy 0.848923 0.092943 0.761823 0.920923 5
5 svm balanced_accuracy 0.849359 0.092783 0.761522 0.920513 5
6 svm mcc 0.701250 0.187081 0.523948 0.845931 5