AAPred

class AAPred(models=None, list_model_classes=None, list_model_kwargs=None, list_metrics=None, df_feat=None, df_scales=None, verbose=True, random_state=None)[source]

Bases: Wrapper

AAPred: evaluate and deploy sequence-based prediction models (Wrapper) [Breimann25].

A thin, opinionated wrapper that closes the gap left by feature engineering: given a feature matrix X and labels, it evaluates one or more scikit-learn model classes across metrics by cross-validation and an optional held-out set (eval()), and deploys them by fitting on all data and exposing prediction scores (fit() / predict() / eval()).

Unlike CPPGrid, which optimizes the feature space and scores configurations by feature separation, AAPred takes a fixed feature set and trains models that are kept for deployment. It intentionally does not perform hyperparameter optimization — pass configured estimators and it evaluates and deploys them.

Added in version 1.1.0.

Notes

  • All fitted-state attributes carry a trailing underscore and are set by fit().

See also

  • TreeModel for tree-ensemble Monte-Carlo feature importance and selection.

  • AAPredPlot for visualizing evaluation and prediction results.

  • CPPGrid for optimizing the CPP feature space (upstream of this class).

Parameters:

Methods

eval(X, labels[, X_holdout, labels_holdout, ...])

Evaluate every model across metrics by cross-validation and an optional held-out set.

fit(X, labels[, label_pos, ...])

Fit every model on the full dataset for deployment.

predict(df_seq[, level, threshold, ...])

Predict from raw sequences at a chosen level: whole protein, domain, or residue window.
__init__(models=None, list_model_classes=None, list_model_kwargs=None, list_metrics=None, df_feat=None, df_scales=None, verbose=True, random_state=None)[source]
Parameters:

  • models (str, estimator, or list, optional) – The models to evaluate and deploy, given as registry name strings (e.g. "svm", "rf"; see aaanalysis.utils.LIST_PRED_MODELS) and/or configured scikit-learn estimator instances, in any mix. This is the recommended way to select models; it is mutually exclusive with list_model_classes. Each model must implement predict_proba.

  • list_model_classes (list of Type[ClassifierMixin or BaseEstimator], default=[RandomForestClassifier]) – Model classes to evaluate and deploy (legacy alternative to models). Each must implement predict_proba.

  • list_model_kwargs (list of dict, optional) – Keyword arguments for each model in list_model_classes (same length).

  • list_metrics (list of str, default=["accuracy", "balanced_accuracy", "f1", "roc_auc"]) – Default performance metrics used by eval() when metrics is not given. Each should be one of accuracy, balanced_accuracy, precision, recall, f1, roc_auc.

  • df_feat (pd.DataFrame, shape (n_features, n_feature_info), optional) – CPP feature DataFrame (with a feature column) bound to the model. When given, the feature matrix X is computed internally from a df_seq by the sequence-level predict() method (level='sequence'/'domain'/'window').

  • df_scales (pd.DataFrame, shape (n_letters, n_scales), optional) – Amino acid scales used for internal featurization. Defaults to the bundled AAontology scales when None.

  • verbose (bool, default=True) – If True, verbose outputs are enabled.

  • random_state (int, optional) – The seed used by the random number generator. If a positive integer, results of stochastic processes are consistent, enabling reproducibility. If None, stochastic processes will be truly random.

Examples

The AAPred class evaluates and deploys sequence-based prediction models. We first obtain the DOM_GSEC example dataset and build its CPP feature matrix (see [Breimann25]):

The big picture — three verbs. AAPred trains scikit-learn models on CPP features and then:

  • ``fit`` — train the model(s) on all data, for deployment.

  • ``predict`` — score raw sequences at one of three levels:

    • level='seq' — one score per protein (whole sequence).

    • level='domain' — a boundary-sensitivity scan (score vs. a shifted domain boundary).

    • level='window' — a per-residue profile (sliding window).

  • ``eval`` — compare models across metrics by cross-validation (returns df_eval).

Rule of thumb: fit to deploy, predict to score sequences, eval to compare models.

import aaanalysis as aa
aa.options["verbose"] = False  # Disable verbosity

# DOM_GSEC example dataset + its feature set (see [Breimann25]_)
df_seq = aa.load_dataset(name="DOM_GSEC")
labels = df_seq["label"].to_list()
df_feat = aa.load_features(name="DOM_GSEC").head(20)

# Build the CPP feature matrix
sf = aa.SequenceFeature()
df_parts = sf.get_df_parts(df_seq=df_seq)
X = sf.feature_matrix(features=df_feat["feature"], df_parts=df_parts)

We create an AAPred object (default model: RandomForestClassifier) and evaluate it across metrics by 5-fold cross-validation:

aapred = aa.AAPred(random_state=42)
df_eval = aapred.eval(X, labels)
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (4, 5)
  model metric principle score score_std
1 RandomForestClassifier accuracy cv 0.809231 0.064659
2 RandomForestClassifier balanced_accuracy cv 0.810256 0.065416
3 RandomForestClassifier f1 cv 0.810256 0.059252
4 RandomForestClassifier roc_auc cv 0.887278 0.071059

Further parameters. Select the models to evaluate and deploy by registry name with models (a mix of "svm", "rf", "extra_trees", "log_reg" and/or configured estimators). list_metrics sets the default metrics used by :meth:AAPred.eval, df_scales the amino acid scales used for internal featurization, and df_feat binds a feature set for sequence-level prediction:

df_scales = aa.load_scales()  # bundled AAontology scales (used for internal featurization)
aapred = aa.AAPred(models=["svm", "rf"],
                         list_metrics=["accuracy", "f1", "roc_auc"],
                         df_feat=df_feat, df_scales=df_scales, random_state=42)
df_eval = aapred.eval(X, labels)
aa.display_df(df_eval, n_rows=10, show_shape=True)

/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
/Users/stephanbreimann/Programming/1Packages/aaanalysis/.venv/lib/python3.13/site-packages/sklearn/svm/_base.py:239: FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11. Use CalibratedClassifierCV(SVC(), ensemble=False) instead of SVC(probability=True)
  warnings.warn(
DataFrame shape: (6, 5)
  model metric principle score score_std
1 SVC accuracy cv 0.848923 0.092943
2 SVC f1 cv 0.854063 0.089412
3 SVC roc_auc cv 0.907199 0.073961
4 RandomForestClassifier accuracy cv 0.809231 0.064659
5 RandomForestClassifier f1 cv 0.810256 0.059252
6 RandomForestClassifier roc_auc cv 0.887278 0.071059

Alternatively, the legacy list_model_classes / list_model_kwargs pair selects model classes together with their keyword arguments (mutually exclusive with models):

from sklearn.ensemble import RandomForestClassifier
aapred = aa.AAPred(list_model_classes=[RandomForestClassifier],
                          list_model_kwargs=[{"n_estimators": 50}], random_state=42)
df_eval = aapred.eval(X, labels)
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (4, 5)
  model metric principle score score_std
1 RandomForestClassifier accuracy cv 0.816923 0.055368
2 RandomForestClassifier balanced_accuracy cv 0.817949 0.056082
3 RandomForestClassifier f1 cv 0.818345 0.051525
4 RandomForestClassifier roc_auc cv 0.891765 0.064760