Held-out reports

Every endpoint ships with a held-out score measured on rows whose target values the model never saw when predicting them, stated with its split, seed, baseline, and data snapshot. It is computed at endpoint creation, at every refresh, and at every base upgrade.

How it is computed

The score is measured on rows the model did not see: a stratified test partition is held out (default 80/20), the endpoint predicts those rows, and the predictions are scored against the true values. The report records the method (in_context_holdout), the exact partition sizes, and the seed. A refresh scores a fresh split on the renewed snapshot.

Every score is stated against a baseline, with the metric set by task type:

• Classification: accuracy, ROC-AUC (one-vs-one for multiclass), per-class; baseline is the majority class.
• Regression: RMSE, NRMSE, R-squared, plus quantile calibration (coverage of predicted intervals); baseline is the mean or last value.
• Anomaly: precision, recall, PR-AUC at the chosen threshold where labels exist; score separation and threshold, flagged unsupervised, where they do not; baseline is the random or expected rate.
• Imputation: NRMSE on held-out known cells.

What the score measures

Held-out is a creation-time estimate on unseen rows from the same dataset, tied to a data snapshot. Live production accuracy is a separate measurement, taken from labels that arrive after deployment. The report stamps the snapshot it ran on.

What every report contains

Two layers: a badge everywhere, a report on click. Every number travels with its provenance.

 +-----------------------------+
 | Held-out   96.1%            |      the badge, inline on every endpoint output
 | 4,480 test rows             |
 +-----------------------------+

 + HELD-OUT REPORT -------------------------------------------------+
 | Target: crm_contacts.c3   Data: dsv_91f2 · 2026-08-12 09:31      |
 | Method: in-context holdout, seed 7    Train 17,920 / Test 4,480  |
 |                                                                  |
 | Held-out accuracy   96.1%   (upgrade report r_a7d0, schema-2)    |
 | ROC-AUC             0.974                                        |
 | Baseline (majority) 71.0%   +25.1 pp lift                        |
 |                                                                  |
 | Confidence          probabilities (see model card)               |
 | Confidence spread   most predictions high-confidence             |
 | Per-class:  retained 97%  ·  churned 91%                         |
 | Weakest slice:  tenure<3mo  -> 84%                               |
 +------------------------------------------------------------------+

Confidences are probabilities; calibration results per model are in the model card. For mapping tasks, each aligned pair carries its own confidence in the cross-table map.

Naming and history

Every create, refresh, and upgrade emits exactly one report. Display name: {endpoint}.{base}.{op}.{YYYY-MM-DD}, with the base abbreviated (s1, s2); stable id: r_....

 churn.s1.create.2026-03-10    r_8f3a   pinned (production baseline)
 churn.s1.refresh.2026-08-12   r_b5e1   fresh split; reflects removals
 churn.s2.upgrade.2026-08-14   r_a7d0   new base

Star any report on the platform’s Reports page to pin it: pinned reports float to the top of the endpoint’s history. Compare takes any two reports side by side, which is where the upgrade delta is read. Reports are kept for the life of the org, listed newest first with pinned reports first; deleting an endpoint does not delete its reports. Reference: Reports.