FLAIR

Federated Leaderboard for Assessment of ICU Risk Models

Compare performance across health systems. Patient records stay local.
View leaderboard Read the guide
pip install flair-benchmark coming soon
Build on public MIMIC-CLIF. Then test across hospitals without moving patient data.

One evaluation framework. Room for your model.

  1. Clinical task

    Cohort, outcome and prediction time

    Standardized by FLAIR

  2. Feature engineering

    How clinical data become model inputs

    Selected by the developer

  3. Prediction model

    The model and inference code to evaluate

    Selected by the developer

  4. Evaluation

    Shared metrics and site-level reports

    Standardized by FLAIR

The same clinical task and evaluation at every health system. Patient records stay local; only aggregate reports return to the leaderboard.

The story from model to result

Follow one model from development to a released, site-by-site result.
How a model moves through FLAIR A model owner, either a CLIF site team or an outside researcher, submits a frozen model with inference code or a container. The Benchmark Committee reviews it for reproducibility, clinical face validity, and code safety, the FLAIR Portal distributes it across the institutional privacy boundary, each participating health system runs it locally on its own CLIF tables, a site data steward approves the aggregate reports, and the leaderboard displays site estimates. Patient records stay at the institution. MODEL DEVELOPMENT REVIEW AND COORDINATION PARTICIPATING HEALTH SYSTEM Institutional privacy boundary 1 Submitfrozen model 2 Approvesubmission 3 Approved model 4 Executelocally Aggregatereports 5 Approved aggregatereports only 6 Display siteestimates Model owner CLIF site team or outside researcher Develop and test on MIMIC-CLIF or local CLIF Benchmark Committee Review reproducibility, clinical face validity, and code safety with site data stewards FLAIR Portal Register submissions Distribute approved models Site CLIF tables Longitudinal ICU records held under local control Run submitted model Shared task and evaluation on local CLIF data Site data steward Reviews aggregate reports Approves results for release Leaderboard Site-specific estimates after approved release

Patient records, engineered features, and individual predictions remain at the institution.

Model owners include CLIF site teams as well as outside researchers: a model built at one CLIF site runs at every other site. Its home site reports local results; every other site reports external validation.

  1. Submit frozen modelA CLIF site team or outside researcher develops and tests on MIMIC-CLIF or local CLIF, then submits a frozen model with inference code or a container.
  2. Approve submissionThe Benchmark Committee of clinicians, methodologists, and site data stewards reviews reproducibility, clinical face validity, and code safety.
  3. Distribute approved modelThe FLAIR Portal sends the approved model across the privacy boundary to each health system.
  4. Execute locallyEach site runs the shared task and evaluation on its own CLIF tables.
  5. Return approved aggregatesThe site data steward reviews aggregate reports; only approved ones go back.
  6. Display site estimatesThe leaderboard shows site-specific estimates after approved release.
~960,000
ICU hospitalizations across the consortium
12
sites: 11 health systems + MIMIC-CLIF
2011–2025
years of data at the health systems
4
benchmark tasks

Counted from the published snapshot. Each site contributes its largest task cohort (hospitalizations, train and test splits together), so a stay is not counted twice across tasks. MIMIC-CLIF dates are shifted and are left out of the year range.

Simple ranking first. Deep results when you need them.

FLAIR ranks models by task, then lets you open the site-level details behind the rank. That matters because a pooled AUROC (an n-weighted mean across sites) can hide exactly the cross-site failure mode this benchmark is designed to surface.

1 model · 4 tasks · 12 participating sites
Open leaderboard Request a run

What you can inspect

The public board stays simple, but the details are one click away.

Task rankings

Open a task to compare external AUROC summaries and inspect each site’s results.

View rankings

Site differences

Open a model to compare site-level scores and recompute pooled performance with sites included or removed.

Compare sites

Full reports

Open a site report for calibration, decision curves, subgroup fairness, lead-time behavior, and cohort summaries.

Read report guide

Prediction tasks

Each task specifies eligibility, prediction time, outcome, threshold, action direction, and evaluation protocol.
Mortality task
outcome continuous one-per-window
Discharge to LTACs
outcome continuous one-per-window
Extubation failure
intervention episode one-per-stay
ICU readmission
outcome episode one-per-stay

Citation

Preprint - please check the repository for the canonical reference.
@article{flair2026,
  title   = {FLAIR: A Model-to-Data Benchmark for Multicenter Evaluation
             of ICU Prediction Models},
  author  = {Chaudhari, Vaishvik and Parker, William F. and
             Ingraham, Nicholas E. and Gao, Catherine A. and Liao, Zewei and
             Lyons, Patrick G. and Chhikara, Kaveri and Barker, Anna K. and
             Park-Egan, Brenna and Ortiz, Alexander C. and
             Schmid, Benjamin E. and Weissman, Gary E. and
             Hochberg, Chad H. and Liao, Wan-Ting and
             Amaral, Andre C. K. B. and Beaulieu-Jones, Brett K. and
             Buell, Kevin G. and Rojas, Juan C.},
  year    = {2026},
  note    = {Preprint}
}