AAPred.eval

AAPred.eval(X, labels, X_holdout=None, labels_holdout=None, metrics=None, n_cv=5, df_seq=None, baseline=None, list_parts=None)[source]

Evaluate every model across metrics by cross-validation and an optional held-out set.

Two evaluation principles are reported: cv (stratified k-fold cross-validation on X) and, when a held-out set is provided, holdout (models fit on X and scored on X_holdout). The result is a long-format table with one row per (model, metric, principle).

Set baseline to compare the bound features against simple, non-positional baseline featurizers (amino-acid / dipeptide / scale composition) built internally from df_seq: each baseline is cross-validated with the same models and folds and its rows are appended, so the whole “CPP vs baseline” comparison comes from one call. This quantifies how much the positional CPP features add over a plain composition encoding.

Added in version 1.1.0.

Parameters:

  • X (array-like, shape (n_samples, n_features)) – Feature matrix used for cross-validation (and, for the holdout principle, training).

  • labels (array-like, shape (n_samples,)) – Class labels for samples in X.

  • X_holdout (array-like, shape (n_holdout, n_features), optional) – Held-out feature matrix. If given, the holdout principle is added.

  • labels_holdout (array-like, shape (n_holdout,), optional) – Class labels for X_holdout. Required if X_holdout is given.

  • metrics (list of str, optional) – Performance metrics to compute. Defaults to list_metrics from the constructor.

  • n_cv (int, default=5) – Number of stratified cross-validation folds (must not exceed the smallest class count).

  • df_seq (pd.DataFrame, shape (n_samples, n_seq_info), optional) – DataFrame containing an entry column with unique protein identifiers and sequence information (the same input accepted by SequenceFeature.get_df_parts()). Must be row-aligned with X / labels (row i is the same sample) and built with the same part geometry as X for a fair comparison — the alignment cannot be verified from the opaque X, only len(df_seq) == len(labels) is checked. Required when baseline is set, ignored otherwise.

  • baseline (bool, str, or list of str, optional) – Baseline featurizer(s) to cross-validate alongside the bound features. True uses the scale-composition baseline ('scale'); a str or list selects among 'scale' (SequenceFeature.scale_composition()), 'aac' (SequenceFeature.aa_composition()), and 'dpc' (SequenceFeature.dipeptide_composition()). None (default) adds no baseline. Baselines average over list_parts with SequenceFeature.get_df_parts()’ default JMD lengths; match the geometry used to build X so the comparison is not confounded.

  • list_parts (str or list of str, optional) – Sequence parts averaged into each baseline (passed to the featurizers). Defaults to the whole tmd_jmd span. Used only when baseline is set.

Returns:

df_eval – Long-format evaluation table with columns model, metric, principle, score, and score_std (score_std is NaN for the holdout principle). When baseline is given, a leading features column is added ('cpp' for the bound-feature rows, the baseline kind for each baseline’s cross-validation rows); with baseline=None the table is unchanged (5 columns).

Return type:

pd.DataFrame, shape (n_rows, 5) — or (n_rows, 6) in baseline mode

Examples

To demonstrate AAPred().eval(), we obtain the DOM_GSEC dataset and its feature matrix:

import aaanalysis as aa
aa.options["verbose"] = False  # Disable verbosity

# DOM_GSEC example dataset + its feature set (see [Breimann25]_)
df_seq = aa.load_dataset(name="DOM_GSEC")
labels = df_seq["label"].to_list()
df_feat = aa.load_features(name="DOM_GSEC").head(20)

# Build the CPP feature matrix
sf = aa.SequenceFeature()
df_parts = sf.get_df_parts(df_seq=df_seq)
X = sf.feature_matrix(features=df_feat["feature"], df_parts=df_parts)

/Users/stephanbreimann/Programming/1Packages/wt-335-baseline-eval/aaanalysis/feature_engineering/_backend/cpp_run.py:164: UserWarning: CPP is using the Python kernel fallback — the compiled Cython extension is not available in this install. Output is bit-exact with the Cython path but ~2x slower. Reinstall via pip install --force-reinstall aaanalysis to fetch a prebuilt wheel.
  warnings.warn(

eval scores every model across metrics by stratified cross-validation and returns a long-format table (one row per model, metric, and evaluation principle):

aapred = aa.AAPred(random_state=42)
df_eval = aapred.eval(X, labels)
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (4, 5)
  model metric principle score score_std
1 RandomForestClassifier accuracy cv 0.809231 0.064659
2 RandomForestClassifier balanced_accuracy cv 0.810256 0.065416
3 RandomForestClassifier f1 cv 0.810256 0.059252
4 RandomForestClassifier roc_auc cv 0.887278 0.071059

Providing a held-out set adds the holdout principle beside cross-validation (score_std is NaN for the single held-out estimate):

from sklearn.model_selection import train_test_split
X_tr, X_ho, y_tr, y_ho = train_test_split(X, labels, test_size=0.3, random_state=0, stratify=labels)
df_eval = aapred.eval(X_tr, y_tr, X_holdout=X_ho, labels_holdout=y_ho)
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (8, 5)
  model metric principle score score_std
1 RandomForestClassifier accuracy cv 0.806536 0.105052
2 RandomForestClassifier accuracy holdout 0.842105 nan
3 RandomForestClassifier balanced_accuracy cv 0.808333 0.105556
4 RandomForestClassifier balanced_accuracy holdout 0.842105 nan
5 RandomForestClassifier f1 cv 0.800611 0.117547
6 RandomForestClassifier f1 holdout 0.833333 nan
7 RandomForestClassifier roc_auc cv 0.916667 0.078567
8 RandomForestClassifier roc_auc holdout 0.896122 nan

The metrics and number of cross-validation folds are controlled by metrics and n_cv:

df_eval = aapred.eval(X, labels, metrics=["balanced_accuracy", "roc_auc"], n_cv=3)
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (2, 5)
  model metric principle score score_std
1 RandomForestClassifier balanced_accuracy cv 0.793651 0.059391
2 RandomForestClassifier roc_auc cv 0.863190 0.061409

Set baseline to compare the CPP features against simple, non-positional composition baselines built internally from df_seq — scale (scale-average), aac (amino-acid composition), and dpc (dipeptide composition), each averaged over list_parts. Every baseline is cross-validated with the same models and folds, and the rows are tagged in a leading features column (cpp for the bound features), so the whole “CPP vs baseline” comparison comes from one call:

df_eval = aapred.eval(X, labels, df_seq=df_seq,
                      baseline=["scale", "aac", "dpc"], list_parts="tmd_jmd")
aa.display_df(df_eval, n_rows=10, show_shape=True)

DataFrame shape: (16, 6)
  features model metric principle score score_std
1 cpp RandomForestClassifier accuracy cv 0.809231 0.064659
2 cpp RandomForestClassifier balanced_accuracy cv 0.810256 0.065416
3 cpp RandomForestClassifier f1 cv 0.810256 0.059252
4 cpp RandomForestClassifier roc_auc cv 0.887278 0.071059
5 scale RandomForestClassifier accuracy cv 0.754462 0.094264
6 scale RandomForestClassifier balanced_accuracy cv 0.757051 0.095218
7 scale RandomForestClassifier f1 cv 0.749096 0.093903
8 scale RandomForestClassifier roc_auc cv 0.845217 0.075511
9 aac RandomForestClassifier accuracy cv 0.706154 0.090286
10 aac RandomForestClassifier balanced_accuracy cv 0.706410 0.091143