AAPred.predict_oof

AAPred.predict_oof(X, labels, *, label_pos=1, n_cv=5, score_range='proba')[source]

Score the training set with cross-validated out-of-fold per-sample probabilities.

Where predict() deploys models fit on all data to score new proteins, and eval() reports aggregate cross-validated metrics, predict_oof returns the per-sample out-of-fold score for the training data: every sample is scored by models fit on the folds that exclude it (stratified k-fold cross-validation), so the scores are honest and free of the optimistic in-sample bias that scoring the training proteins with predict() would incur. Each configured model is cross-validated independently and the per-model out-of-fold scores are averaged, matching the score / score_std shape of predict() (mean over the ensemble, std across models).

Like eval(), this cross-validates the models given at construction and does not require a prior fit(); it never touches the deployment models in list_models_.

Added in version 1.1.0.

Parameters:

  • X (array-like, shape (n_samples, n_features)) – Feature matrix. Rows typically correspond to samples and columns to features.

  • labels (array-like, shape (n_samples,)) – Class labels for samples in X (typically 1 for the positive class and 0 for the negative class).

  • label_pos (int, default=1) – Label of the positive class whose out-of-fold probability is scored.

  • n_cv (int, default=5) – Number of stratified cross-validation folds (must not exceed the smallest class count).

  • score_range (str, default="proba") – Scale of the returned score / score_std columns: 'proba' for the native probabilities in [0, 1], 'percent' for the same values scaled to [0, 100].

Returns:

df_pred – Per-sample out-of-fold predictions, row-aligned with X, with columns score (mean positive-class probability over the model ensemble) and score_std (std across models; 0 for a single model).

Return type:

pd.DataFrame, shape (n_samples, 2)

See also

  • AAPred.predict() for scoring new proteins with the fitted deployment ensemble.

  • AAPred.eval() for aggregate cross-validated model metrics (df_eval).

Examples

AAPred().predict_oof() returns cross-validated out-of-fold per-sample scores for the training set: every sample is scored by models fit on the folds that exclude it, so the scores are free of the optimistic in-sample bias that scoring the training proteins with :meth:predict would incur. Where :meth:predict deploys models fit on all data to score new proteins and :meth:eval reports aggregate cross-validated metrics, predict_oof is the per-sample counterpart of eval (see [Breimann25]). We first obtain the DOM_GSEC dataset and its feature matrix:

import aaanalysis as aa
aa.options["verbose"] = False  # Disable verbosity

# DOM_GSEC example dataset + its feature set (see [Breimann25]_)
df_seq = aa.load_dataset(name="DOM_GSEC")
labels = df_seq["label"].to_list()
df_feat = aa.load_features(name="DOM_GSEC").head(20)

# Build the CPP feature matrix
sf = aa.SequenceFeature()
df_parts = sf.get_df_parts(df_seq=df_seq)
X = sf.feature_matrix(features=df_feat["feature"], df_parts=df_parts)

predict_oof cross-validates the models given at construction and returns one row per sample (row-aligned with X) with a positive-class score averaged over the ensemble and its score_std across models. Using several models makes the score_std (model agreement) informative:

aap = aa.AAPred(models=["rf", "extra_trees", "log_reg"], random_state=42)
df_pred = aap.predict_oof(X, labels)
aa.display_df(df_pred, n_rows=10, show_shape=True)

DataFrame shape: (126, 2)
  score score_std
1 0.803635 0.049259
2 0.831424 0.068139
3 0.886386 0.038709
4 0.912722 0.031769
5 0.950564 0.048871
6 0.860323 0.021306
7 0.068444 0.022419
8 0.914243 0.048924
9 0.937032 0.033492
10 0.825888 0.069414

label_pos selects which class is scored as positive, and n_cv sets the number of stratified cross-validation folds (it must not exceed the smallest class count):

df_pred = aap.predict_oof(X, labels, label_pos=1, n_cv=3)
aa.display_df(df_pred, n_rows=10, show_shape=True)

DataFrame shape: (126, 2)
  score score_std
1 0.850491 0.036761
2 0.852332 0.083054
3 0.935484 0.035620
4 0.913218 0.049365
5 0.968408 0.044678
6 0.846112 0.031219
7 0.049390 0.015589
8 0.943372 0.038533
9 0.932898 0.062944
10 0.848418 0.018302

score_range sets the scale of the returned scores: 'proba' (default, [0, 1]) or 'percent' ([0, 100]) — convenient when the downstream display or classification already speaks percent:

df_pred = aap.predict_oof(X, labels, score_range="percent")
aa.display_df(df_pred, n_rows=10, show_shape=True)

DataFrame shape: (126, 2)
  score score_std
1 80.363495 4.925876
2 83.142386 6.813886
3 88.638596 3.870922
4 91.272228 3.176885
5 95.056410 4.887054
6 86.032322 2.130603
7 6.844384 2.241948
8 91.424261 4.892395
9 93.703240 3.349161
10 82.588771 6.941356

predict_oof needs no prior :meth:fit and never touches the deployment models in list_models_; a single model simply returns a score_std of 0:

df_pred = aa.AAPred(models="rf", random_state=42).predict_oof(X, labels)
aa.display_df(df_pred, n_rows=10, show_shape=True)

DataFrame shape: (126, 2)
  score score_std
1 0.860000 0.000000
2 0.920000 0.000000
3 0.940000 0.000000
4 0.930000 0.000000
5 0.990000 0.000000
6 0.890000 0.000000
7 0.100000 0.000000
8 0.900000 0.000000
9 0.970000 0.000000
10 0.740000 0.000000