Building reports

Turning a predictions parquet into the aggregate-only JSON bundle a site uploads.

External summary and rank eligibility

The task headline is External AUROC: eligible external-site AUROCs weighted by test hospitalizations, never daily prediction rows. Daily tasks use each site's published pooled landmark AUROC. For the continuous-task discrimination method, see Heagerty PJ, Zheng Y. Biometrics. 2005;61(1):92–105 (doi:10.1111/j.0006-341X.2005.030814.x). The site computes its estimates and bootstrap intervals; the website copies them. MIMIC-CLIF remains a reference and does not contribute. Site estimates and confidence intervals stay unchanged. No overall CI is reported; averaging site interval endpoints does not establish one.

Summary policy flair-manuscript-2026-10-18 fixes eleven external health systems for mortality, extubation failure and ICU readmission, and seven for LTAC. LTAC excludes Health Systems 4, 7 and 9 for local practice and Health System 5 for its 0.6% test-hospitalization event rate; eligibility requires greater than 1%. Summary event rates use test hospitalizations. Previously published estimates outside this analysis stay inspectable with exclusion labels; bundles already withheld by the build are not restored.

A missing or invalid required AUROC or test count leaves the summary unavailable; its denominator never silently shrinks. Report selections do not change it. The API retains pooled for compatibility with auroc_ci: null.

Formal ranks are pending. The exports do not fully establish identical task versions, evaluation protocols and per-site test partitions across submissions. Different task-script hashes occur; a shared package version does not certify equivalence. Changes to these conditions or eligible sites require a new ranking version and reference-model reevaluation. Once verified, exact ties must share a rank. Descriptive ordering uses available aggregate precision before display rounding; lost exporter precision cannot be recovered. The manuscript did not evaluate comparative leaderboard rankings.

How FLAIR reads these results

We treat the leaderboard as an entry point. We prioritize risk ranking across sites, calibration of probabilities, and the consequences of a clinically relevant threshold.

Risk ranking: inspect each site's AUROC and uncertainty. An overall rank can conceal differences; AUROC alone does not establish accurate probabilities or clinical benefit.
Calibration: compare predicted risk with observed outcomes. Episodic reports show equal-count reliability bins, not a smoothed curve or curve-level confidence band. Continuous reports show slopes over time; a slope near 1 does not establish calibration across the full risk range.
Threshold explorer: select a health system and adjust the probability cutoff. Read PPV, NNE, sensitivity and alert rate together. PPV is the share of flagged hospitalizations with the outcome; NNE is the number flagged per true positive (approximately 1/PPV). Alert rate counts flagged hospitalizations, not repeated bedside alerts. The clinical button restores the task's reference threshold; exploration does not change the published policy.
Clinical utility: episodic decision curves compare the model's net benefit with the same site's default strategies. Overlays use each site's own act-on-all comparator and a common zero act-on-none line. Thresholds should reflect the consequences of missed outcomes and unnecessary action. A test-set optimum does not establish a clinical policy, and retrospective net benefit does not demonstrate benefit in practice.

Continuous operating metrics use maximum predicted risk per hospitalization; episodic metrics use one prediction per hospitalization. Alarm metrics flag risk at or above the cutoff. Extubation's clinical action is below the threshold; its decision curve evaluates that action separately. The alarm readout must not be relabelled as that action.

Detailed reports initially focus on the first reporting external-validation site in published order, falling back to the first available site. This is a navigation default, not a recommendation. Choose one health system, all loaded sites, or a custom comparison. Summary retains all published site estimates; report filters never change the ranking.

Each task page states its potential clinical use and the evidence FLAIR prioritizes. Explore local thresholds in Summary opens the same detailed explorer directly. Its four primary metrics use percentages to one decimal and NNE to one decimal, including CI bounds. Positive percentages that would round to zero show <0.1%; actual zeros show 0.0%. All threshold metrics and detailed precision retains every operating metric, the original proportion scale and confusion counts. Switching views and retrying unavailable reports preserve site, section and cutoff selections. Each model's report keeps its own selections.

Reports pair ROC, calibration and decision curves with their key metric tables. Single-site pairs sit alongside each other on wider screens and stack on phones; all-site tables use their own horizontal scroll area. Continuous reports show AUROC and calibration slope over time, followed by the published pooled estimates. The threshold explorer shows sensitivity and PPV curves alongside the four primary readout metrics; the cutoff marker and readout update together. Other original curves and detailed tables remain available in named expanders. These layout choices do not alter published values, confidence intervals or ranking.

Secondary metrics, precision–recall curves, threshold-selection rules and training cohorts remain in named expanders. We retain AUPRC and F1 for supplementary inspection, alongside subgroup results and test-population context; they do not replace calibration or clinical utility.

Available evidence has limits. Outcome-stratified risk distributions and smoothed calibration curves with curve-level uncertainty are not exported. Continuous tasks have no reliability-bin curves or decision curves on the maximum-over-stay surface. Missing or suppressed estimates stay explicit; suppressed counts remain <10.

This is FLAIR's presentation policy, informed by Van Calster et al., Lancet Digital Health 2025. The paper recommends AUROC, calibration plots, clinical utility assessment and outcome-stratified probability distributions. Current exports support only part of that assessment. The paper does not prescribe FLAIR's ranking formula or website layout.

The key choice is --mode - how a task's predictions are scored.

The command

flair build-report preds.parquet \
  --task icu_daily_mortality \
  --out ./report/ \
  --mode continuous \
  --cohort cohort.parquet \
  --viz            # or --no-viz

flag	default	purpose
`--task`	(required)	task name
`--out`	(required)	output directory for the JSON bundle
`--mode`	inferred	`episodic` \| `continuous`
`--cohort`	-	joins demographics + `window_number` when absent from preds
`--viz/--no-viz`	on	render local sanity-check PNGs (needs `flair-benchmark[viz]`; never uploaded)

Reports are always scored on the held-out test split; there is no override, so cross-site numbers are comparable.

The predictions parquet

Required columns: hospitalization_join_id, prediction_id, split, <label_column>.

binary → y_prob required; y_pred optional (derived as y_prob ≥ 0.5 when absent)
multiclass → y_pred (class_probs_<class> optional)
regression → y_pred_value

hospitalization_join_id (the stitched hospitalization) is the report unit; hospitalization_id may be carried for traceability but is not the evaluation key. Carry window_number and demographics through too (or pass --cohort and they're joined in on prediction_id). y_prob must be a valid probability in [0, 1] with no nulls and no NaNs. Only binary tasks are scored today; multiclass and regression files are validated, then reported with null metrics.

What the bundle looks like

One folder per task. Results are grouped by evaluation arm, and the report files inside an arm depend on the task's mode.

<task>/
  table1.json                        cohort table, shared across arms
  codes.parquet                      feature dictionary (aggregated into /vocab)
  report/
    local/                           trained and evaluated here
      overall.json                   EPISODIC tasks only
      landmark.json                  CONTINUOUS tasks only
      hospitalization_level.json     both
      viz/*.png                      local sanity checks, never uploaded
    external_validation/             model trained elsewhere, evaluated here
      ...same three files...
    transfer/                        internal diagnostic
      ...same three files...

The presence of a file is the mode signal. overall.json means episodic; landmark.json means continuous. Never both.

Which arm gets published

The site a model was trained on publishes its local arm. Every other site publishes its external_validation arm. transfer/ is an internal diagnostic and is never published.

There is no fallback. If a site owes an external_validation arm and only local is present, the snapshot build fails rather than publishing a site's own-data score in a column labelled external validation.

Choosing --mode

mode	use for	files produced
episodic	one-per-stay tasks (extubation failure, ICU readmission)	`overall.json` + `hospitalization_level.json`
continuous	one-per-window tasks (mortality task, discharge to LTACs)	`landmark.json` + `hospitalization_level.json`

If you omit --mode it is inferred from META's temporal_pattern (continuous → continuous, anything else → episodic, with a warning for episode tasks to pass --mode explicitly).

Repeated predictions within a hospitalization are correlated, and performance may change over time, so once-per-hospitalization tasks use episodic evaluation and daily tasks use landmark-based evaluation.

The landmark view freezes the clock at each point in the stay, scores one row per hospitalization at that lead time, and repeats across the risk-set lead times → an honest skill-and-calibration curve over time. It also carries a pooled block: AUROC, AUPRC, Brier and calibration slope across all landmarks, each with a stay-level bootstrap CI computed at the site. Landmarks cover the first 30 prediction days (META['landmark_horizon']), and a landmark with fewer than 10 hospitalizations at risk is suppressed whole. Pooled AUROC is weighted by case-control pairs at each landmark, AUPRC by cases, and Brier score by observations; the pooled calibration slope comes from a logistic model relating outcomes to the log-odds of predicted risk across landmarks.
The peak surface collapses each hospitalization to its maximum predicted risk - deployment-like for screening discrimination and operating-point selection. Calibration and net benefit are omitted there because peak-selection breaks calibration (the "peak-calibration trap"), and the file states that reason in net_benefit_omitted_reason so the website can show it. Decision curves are not calculated here because maximum risk is not a probability tied to a single decision time.

Thresholds and decision curves

Threshold summaries report sensitivity, specificity, positive and negative predictive values, F1 score, flag rate, and number needed to evaluate at thresholds from 0.01 to 0.99, flagging scores at or above the threshold. Calibration curves use ten bins containing approximately equal numbers of predictions. Decision curves estimate net benefit relative to default action strategies. Extubation decision curves instead evaluate action below the threshold, with benefit and harm weights reversed accordingly. Subgroup estimates use task-defined thresholds.

What each file reports

file	contents
overall.json	episodic only. `discrimination`: AUROC, AUPRC, the prevalence baseline, and ROC/PR curves sampled on a fixed 101-point grid (0.00-1.00). `calibration`: Brier score, slope, intercept, and ten equal-count reliability bins (a bin below 10 hospitalizations nulls its observed rate).
landmark.json	continuous only. `pooled`: AUROC, AUPRC, Brier and calibration slope across landmarks. `by_landmark`: the same four metrics at each lead time.
hospitalization_level.json	both modes, on the stay-level surface (`stay` for episodic, `peak` for continuous). `by_threshold`: the full 99-point sweep (0.01-0.99) with confusion counts and sensitivity/specificity/PPV/NPV/F1/FPR/NNE/flag rate at every threshold, plus net benefit on the episodic surface only; the clinical policy is one flagged row. `selected_thresholds`: the Youden-J (marked in-sample) and pre-specified clinical cutpoints (a site whose sweep degenerates — no positive outcomes in its hold-out — ships only the clinical rule). `fairness`: per-subgroup (sex, race, ethnicity, age-bin) rates and AUROC, plus demographic-parity and equalized-odds gaps.
table1.json	the cohort table, broken out by total / train / test split, with date ranges, prevalence, and continuous + categorical covariates. The cohort tab shows sex as Female only, ethnicity as Hispanic only, and race with Unknown + Other merged into “Unknown/Other” (`≥N` when one merged part is suppressed).

Metrics carry a 95% percentile-bootstrap CI from 500 replicates with a fixed seed, resampling hospitalizations to preserve dependence among repeated predictions (flag rate has no band; NNE's band is the PPV band inverted). Intervals are not adjusted for multiplicity across tasks, sites, or thresholds and should not be used to infer statistically significant differences between sites. Cell suppression (<10) and subgroup dropping (estimates for landmarks and subgroups with fewer than 10 eligible hospitalizations are omitted) are applied so nothing identifying leaves the site, and confusion-matrix counts are masked in pairs to limit recovery of suppressed counts; --viz writes matching PNGs under report/<arm>/viz/ for local validation only. The website renders every chart from the JSON, so the PNGs are never uploaded or served.

Each file also describes its own limits - surface, score_rule, min_cell_count, selection_note, net_benefit_omitted_reason - and the public report renders those sentences rather than dropping a panel without explanation.

← Feature engineering Reference baseline →