Skip to content

Evaluation Report

After you export predictions with test(), BaseModel can turn that predictions file into a standalone evaluation report — a Markdown summary with task-specific metrics and diagnostic plots. It is a companion to interpretation: interpretation explains why the model predicts, the evaluation report quantifies how well it predicts.

Prerequisites

The report is generated from the predictions file written by test() — a tab-separated file with entity_id, prediction, and ground_truth columns. No checkpoint or re-run is needed; the report reads only that file.

Plotting requires the interpretability extra

The report's plots are rendered with matplotlib/seaborn, installed via the interpretability optional dependencies — the same extra used for SHAP reports.

Basic Usage

Point evaluate_predictions() at the predictions file and tell it the task type. It writes the Markdown report to output_path and the plots to a plots/ directory beside it, and also returns the report as a string:

Python
from pathlib import Path
from monad.ui.evaluation import EvaluationConfig, evaluate_predictions
from monad.ui.module import BinaryClassificationTask

evaluate_predictions(
    predictions_path=Path("./results/predictions.tsv"),
    output_path=Path("./results/report.md"),
    task=BinaryClassificationTask(),
    config=EvaluationConfig(model_name="Churn", threshold_strategy="best_f1"),
)

config is optional — omit it and a sensible default is built for the task.

Metrics and Plots by Task Type

The report adapts to the task. evaluate_predictions() dispatches on the task you pass:

Task type Key metrics Plots
Binary / Multiclass Accuracy, Precision, Recall, F1, AUROC, Average Precision (PR-AUC), confusion matrix Score distribution, PR curve, ROC curve, confusion matrix, cumulative gain, lift
Regression RMSE, MAE, R², MAPE, SMAPE, Pearson, Spearman, residual mean/std Predicted-vs-actual, residuals, residual distribution, error distribution/trend, cumulative error, calibration
Multilabel Per-class summary table (Average Precision, AUROC, F1, support) plus full binary-style subreports for the selected top classes The six binary plots per detailed class
Recommendation HR@K, Recall@K, Precision@K, MAP@K, NDCG@K (across k_values), MRR, catalog coverage Metrics-vs-K, metric distributions, recall by ground-truth size, most-recommended items

Single-class ground truth

For binary/multiclass, if the test set contains only one class, AUROC and Average Precision are reported as NaN (with a warning) — there is no negative (or positive) class to rank against.

Configuration

EvaluationConfig covers binary, multiclass, and regression tasks:

Field Type Default Description
model_name str "Model" Display name used in the report.
threshold_strategy "best_f1" | "fixed" "best_f1" How the classification threshold is chosen. best_f1 sweeps the PR curve for the F1-optimal threshold.
fixed_threshold float | None None Threshold (0–1) to use when threshold_strategy="fixed".
plot_output_dir Path | None None Where plots are written. Defaults to a plots/ directory beside the report.
delimiter str "\t" Delimiter of the predictions file.

Threshold strategy is strict

threshold_strategy="fixed" requires fixed_threshold to be set, and "best_f1" requires it to be left unset. There is no implicit 0.5 default — a mismatch raises a validation error.

Recommendation and multilabel tasks use dedicated subclasses. Pass the subclass that matches your task — supplying a plain EvaluationConfig to a recommendation or multilabel task raises a TypeError.

RecommendationEvaluationConfig adds:

Field Type Default Description
k_values list[int] [1, 5, 10, 20, 50] K values for the ranking metrics.
k_focus int 10 K used for per-user distribution plots and the top-recommendations table.
top_recommendations_n int 20 Number of items shown in the most-recommended-items diagnostics.

MultiLabelEvaluationConfig adds:

Field Type Default Description
class_names list[str] | None None One name per label column; must match the column count.
per_class_reports "none" | "top_k" | "all" "top_k" How many per-class subreports to generate.
per_class_top_k int 10 Number of classes to detail when per_class_reports="top_k".
per_class_rank_by "average_precision" | "auroc" | "f1_score" | "support" "average_precision" Metric used to rank classes for top_k selection.

Cap multilabel subreports on large catalogs

A full subreport (one Markdown file plus six plots) per label is expensive when you have hundreds of classes. By default only the top 10 classes get a subreport; raise per_class_top_k, or set per_class_reports="all" when you truly need every class.

Output Layout

For a binary model, evaluate_predictions() produces:

results/
├── report.md
└── plots/
    ├── score_distribution.png
    ├── pr_curve.png
    ├── roc_curve.png
    ├── confusion_matrix.png
    ├── cumulative_gain.png
    └── lift_curve.png

Multilabel runs additionally write one <class_name>_report.md (with its own plots) per detailed class.

Resource Description
Testing Against Ground Truth Run test() and export the predictions file this report consumes
Interpretation Explain why the model predicts, via attribution scores
Reference: Testing Parameters Full TestingParams and OutputType reference
Model Configuration Default metrics and output types per task type