ModelEvaluator.learning_curve

ModelEvaluator.learning_curve(X, labels, *, train_sizes=None, n_cv=5, n_rounds=1, metrics=None, ci=0.95, random_state=None)[source]

Evaluate every model at increasing training sizes to show whether a task is sampling-limited.

Runs the same repeated stratified cross-validation as run(), but within each training fold fits every model on a stratified subset of each size in train_sizes and scores it on the full, unchanged test fold (the test fold is never subsampled and never used for training). Within a fold, the subsets are nested (a larger subset contains the smaller ones). The per-fold scores are aggregated per (model, training size, metric) into a mean, a population std, a percentile bootstrap confidence interval of the mean, and the fold count, by the same aggregation run() uses. A fractional size is resolved within each training fold, so the fraction 1.0 uses every fold’s complete training set and reproduces the scores of run() exactly when both calls use the same resolved random_state (and the same n_cv, n_rounds, and metrics) - also when the training folds differ in size; with random_state=None the two calls shuffle the folds differently, so they then agree only in distribution. An absolute size is used as given in every fold, so it matches run() only where it equals the training fold.

A curve that is still rising at the largest size suggests that more data will help; a curve that has flattened suggests changing the representation or model instead. The decision is left to the user.

Added in version 1.2.0.

Parameters:

  • X (array-like of float, shape (n_samples, n_features)) – Finite feature matrix with at least three samples and two features; rows are the samples evaluated by every training-size curve.

  • labels (array-like of int, shape (n_samples,)) – Binary class labels aligned with X. Values must be exactly 0 and 1; 1 is the positive class for precision, recall, f1, and roc_auc.

  • train_sizes (array-like of float or int, optional) – Training-subset sizes (at least two), either all fractions in (0, 1] or all distinct absolute sample counts (int >= 2). A fraction is resolved within each training fold (rounded down to samples, raised to at least 2, one per class), so 1.0 is every fold’s complete training set; the curve point is labelled by its size in the smallest training fold, and fractions collapsing onto the same label are de-duplicated. An absolute count is used as given in every fold, so counts must be distinct and fit into the smallest training fold. If None, uses [0.2, 0.4, 0.6, 0.8, 1.0]; on sufficiently large data these resolve to five sizes, each with a bootstrap CI. On small data they can collapse to fewer sizes, and at least two distinct sizes are required.

  • n_cv (int, default=5) – Number of stratified cross-validation folds per round. Must be at least 2 and not exceed the smallest class count; increasing it changes the train/test split size and the number of scores aggregated.

  • n_rounds (int, default=1) – Number of cross-validation repeats. Increasing it changes the shuffled splits and yields n_cv * n_rounds fold scores per (model, training size, metric).

  • metrics ({'accuracy', 'balanced_accuracy', 'precision', 'recall', 'f1', 'roc_auc', 'mcc'} or list of str, optional) – Performance metric(s) to compute. If None, uses list_metrics from the constructor; roc_auc requires every model to implement predict_proba.

  • ci (float or None, default=0.95) – Central confidence level in (0, 1) for the percentile bootstrap CI of the mean. If None, skips bootstrap CIs and sets the ci_low / ci_high columns to NaN.

  • random_state (int or None, default=None) – Per-call seed overriding the constructor’s random_state for the folds, the training subsets, and the bootstrap CI. A non-negative integer makes those operations and estimators that support random_state reproducible. If None, the constructor’s random_state is used (and stochastic processes are truly random when that is None as well). aaanalysis.options["random_state"] overrides both unless it is "off".

Returns:

df_curve – Long-format learning-curve table with columns model, train_size (number of training samples, ascending; for a fraction this is the size in the smallest training fold, while every fold uses that same fraction of its own training set), metric, score (mean over folds), score_std (population std over folds), ci_low / ci_high (bootstrap CI of the mean, NaN when ci is None), and n_scores (fold count).

Return type:

pd.DataFrame, shape (n_models * n_train_sizes * n_metrics, 8)

Raises:

  • ValueError – If X or labels are invalid or mismatched, labels are not exactly the two classes 0 and 1, n_cv exceeds the smallest class count, train_sizes mix fractions and counts, resolve to fewer than two distinct sizes, or exceed the smallest training fold, a metric is unknown, a probability metric is requested for a model without predict_proba, or a numeric parameter is out of range.

  • RuntimeError – If an internally constructed training subset overlaps its held-out test fold.

Notes

  • Unlike run(), this method does not change df_scores_ or df_eval_, so a subsequent eval() still compares the models of the last run().

See also

Examples

learning_curve shows whether a prediction task is sampling-limited. It repeats the stratified cross-validation of run on stratified subsets of increasing size of every training fold and scores each model on the full, unchanged test fold. First, the DOM_GSEC dataset and its feature matrix:

import aaanalysis as aa
aa.options["verbose"] = False  # Disable verbosity

# DOM_GSEC example dataset + a small feature set (see [Breimann25]_)
df_seq = aa.load_dataset(name="DOM_GSEC")
labels = df_seq["label"].to_list()
df_feat = aa.load_features(name="DOM_GSEC").head(20)

# Build the CPP feature matrix X
sf = aa.SequenceFeature()
df_parts = sf.get_df_parts(df_seq=df_seq)
X = sf.feature_matrix(features=df_feat["feature"], df_parts=df_parts)

Pass X and labels with the training-subset sizes (train_sizes, fractions of the smallest training fold or absolute sample counts), the number of folds (n_cv), the number of cross-validation repeats (n_rounds), the metrics, the bootstrap confidence level ci, and a per-call random_state. The result has one row per (model, training size, metric):

me = aa.ModelEvaluator(models=["rf", "log_reg"], random_state=42, verbose=False)
df_curve = me.learning_curve(X=X, labels=labels, train_sizes=[0.1, 0.25, 0.5, 0.75, 1.0],
                             n_cv=5, n_rounds=3, metrics=["balanced_accuracy", "mcc"],
                             ci=0.95, random_state=42)
aa.display_df(df_curve, n_rows=10, show_shape=True)

DataFrame shape: (20, 8)
  model train_size metric score score_std ci_low ci_high n_scores
1 rf 10 balanced_accuracy 0.790812 0.092653 0.742089 0.833974 15
2 rf 10 mcc 0.597695 0.181774 0.503134 0.682840 15
3 rf 25 balanced_accuracy 0.789316 0.097634 0.740807 0.837399 15
4 rf 25 mcc 0.590024 0.192078 0.492990 0.685081 15
5 rf 50 balanced_accuracy 0.804701 0.092453 0.758114 0.845315 15
6 rf 50 mcc 0.617305 0.186296 0.523176 0.699471 15
7 rf 75 balanced_accuracy 0.801282 0.100295 0.751923 0.847222 15
8 rf 75 mcc 0.607160 0.199625 0.509310 0.699319 15
9 rf 100 balanced_accuracy 0.803632 0.076990 0.763862 0.838034 15
10 rf 100 mcc 0.612956 0.151909 0.534722 0.680772 15

A score that still rises at the largest train_size suggests that more data will help, while a flat curve suggests changing the representation or model. Absolute sample counts can be given instead of fractions, and ci=None skips the bootstrap:

df_curve = me.learning_curve(X, labels, train_sizes=[10, 20, 40, 80], metrics=["mcc"], ci=None)
aa.display_df(df_curve, n_rows=10, show_shape=True)

DataFrame shape: (8, 8)
  model train_size metric score score_std ci_low ci_high n_scores
1 rf 10 mcc 0.580965 0.246694 nan nan 5
2 rf 20 mcc 0.590384 0.185111 nan nan 5
3 rf 40 mcc 0.587075 0.119189 nan nan 5
4 rf 80 mcc 0.575011 0.122296 nan nan 5
5 log_reg 10 mcc 0.608698 0.188068 nan nan 5
6 log_reg 20 mcc 0.669767 0.167276 nan nan 5
7 log_reg 40 mcc 0.668534 0.164187 nan nan 5
8 log_reg 80 mcc 0.621766 0.139291 nan nan 5