Tasks & the task maker

The four v1 benchmarks, the cohort contract, and how to author your own task.

The four v1 tasks

Each task is a versioned Python module that specifies eligibility, prediction time, outcome, threshold, action direction, and evaluation protocol. Each task links to a potential clinical decision: goals-of-care conversations (mortality task), early rehabilitation and care-transition planning (discharge to LTACs), ventilator management (extubation failure), and discharge readiness assessment (ICU readmission).

Daily-task discrimination and calibration use pooled landmarks; alarm summaries use the maximum risk per hospitalization. Extubation decision curves evaluate action below the threshold, whereas high-risk alarms flag scores at or above it. LTACs denotes long-term acute care hospitals. The actions are evaluated retrospectively and are not clinical recommendations.

Label and eligibility rules

• Mortality task. The endpoint includes discharge disposition recorded as expired or hospice. A missing disposition, or one recorded as still admitted, is labelled negative.
• Daily tasks. Predictions occur at 07:00 during the first ICU stay. The evaluation covers hospitalizations still eligible for prediction at each of the first 30 prediction days.
• Extubation failure. The prediction occurs 30 minutes before the first confirmed extubation; reintubation is assessed during the 24 hours after extubation. Extubations followed by death within 24 hours, or without a complete 24-hour observation, are excluded unless reintubation occurs within that interval.

Task taxonomy

flair list-tasks

The recommended report mode follows from the temporal pattern: episode → episodic; continuous → continuous (pooled landmarks: the mortality task and discharge to LTACs). See Building reports.

Loading a task → cohort

flair load-task icu_daily_mortality \
  --clif-config clif_config.json \
  --out cohort.parquet \
  [--train-end YYYY-MM-DD] [--test-start YYYY-MM-DD]

This runs the task's build() and writes cohort.parquet (one row per prediction_id) plus cohort.table1.json (cohort characteristics, auto-generated).

The cohort contract

prediction_id is {hospitalization_join_id}-{hospitalization_id}-{patient_id}-{window_number}, so each prediction traces to its stitched hospitalization, the specific hospitalization, the patient, and the window. No shipped task sets feature_cutoff_dttm away from prediction_dttm; extubation failure instead moves the prediction itself to 30 minutes before extubation. Reports carry a version identifier covering the task code, evaluation policies, and the splitting procedure.

Train and test partitions

Hospitalizations, not patients, are assigned to training and test sets, at the grain of the stitched hospitalization (hospitalization_join_id): one stitched hospitalization is never split across the two, and a patient can contribute more than one hospitalization. Three regimes, in priority order:

Outside MIMIC-CLIF, rows predicted after a fixed data cutoff (31 December 2025) are dropped before any split. Performance is reported on test hospitalizations only.

Authoring a new task (the "task maker")

A task is a single Python file dropped into flair_benchmark/tasks/<name>.py that exposes a META dict and a build() function. The catalog auto-discovers it - no registry edits.

META - required fields

The taxonomy triple (validated; reviewers read this before any code):

For binary tasks, also required: clinical_threshold (a float in (0, 1) and a multiple of 0.01, so it lands on the 99-point threshold sweep) and clinical_direction ("above" or "below"). Also validated: report_mode ("episodic" or "continuous") and, for continuous tasks, landmark_horizon (a positive int; 30 daily landmarks for the mortality task and discharge to LTACs). Plus descriptive fields: name, display_name, task_type, label_column, positive_class, clinical_action, windows, cohort docs, evaluation_metrics.

build() - signature & contract

def build(clif_config: str, train_end: str | None,
          test_start: str | None) -> pl.DataFrame:
    ...
    return finalize_cohort(df, OUTPUT_COLS, META)

It must end with finalize_cohort(df, OUTPUT_COLS, META), which selects your OUTPUT_COLS and stamps on window_number + prediction_id (and feature_cutoff_dttm, defaulted to prediction_dttm when absent). It does not drop rows whose split came back null, so filter them after assign_split(...) as every shipped task does.

Author decision checklist

1. Cohort scope - all-comers ICU? hospital-wide? specific inclusion/exclusion?
2. Label anchor - what clinical event defines the positive label?
3. Prediction timing - one-per-stay (episode) or one-per-window (continuous)?
4. Threshold + direction - at what risk do you act, and above or below?
5. prediction_dttm - when the prediction is made.
6. feature_cutoff_dttm - does the prediction_dttm row itself leak the outcome? If the anchor is the event (an extubation, a transfer, a procedure), set the cutoff strictly earlier; otherwise leave the default.

Acceptance

Researchers and ICU clinicians can propose new tasks for Benchmark Committee review. Each proposal specifies the clinical question, eligible hospitalizations, the information available at prediction, the outcome label, and the evaluation procedure.

A new task enters the federated suite after: reproducibility against MIMIC-CLIF, a clinician-led face-validity review, and site-governance acceptance at a minimum of two consortium sites.

Contribute a task via GitHub

Tasks ship through a pull request to the FLAIR repo. End-to-end, from a fork to a task that shows up on this leaderboard:

1. Fork the repo. The FLAIR library repository, then branch (e.g. task/<name>).
2. Add the task file. Drop flair_benchmark/tasks/<name>.py with a META dict and a build() ending in finalize_cohort(df, OUTPUT_COLS, META) - see Authoring a new task above. The catalog auto-discovers it; no registry edits.
3. Validate locally. Reproduce against MIMIC-CLIF before opening the PR:

   flair list-tasks                 # confirm your task is discovered
   flair load-task <name> \
     --clif-config clif_config.json \
     --out cohort.parquet

   Check the generated cohort.table1.json for sane cohort characteristics.
4. Mirror the task into the website catalog. In the website repo (flair_website, a separate pull request), add one entry to data/tasks.json so the task surfaces on the leaderboard and task pages - the catalog is hand-mirrored from the upstream META dicts. Required fields:

   field	notes
   id	must match the task file name / META.name; the leaderboard joins tasks ↔ scores on this
   display_name	human title
   color	blue / purple / sage / terracotta (UI dot/pill)
   task_class, temporal_pattern, prediction_unit	the taxonomy triple (copy from META)
   task_type	e.g. binary
   clinical_threshold, clinical_direction	float in (0,1) + above/below (binary tasks)
   clinical_action	the act taken at threshold
   label_column	e.g. label_mortality
   summary	one-line prose (prediction window + cohort)
   report_mode	episodic / continuous (see Building reports)

5. Regenerate the snapshot (website maintainers, once a site returns a non-PHI result dump for the task): drop it under runs_data/flair_export/<site>/<site>_baseline_non_phi_for_upload/<name>/ and run python scripts/build_data.py to rebuild data/leaderboard.json + data/reports/. Without a run, the task lists with no scores yet.
6. Open the pull request against the FLAIR library repository. CI plus a clinician-led face-validity review run on the PR.
7. Acceptance follows the criteria above - reproducibility vs MIMIC-CLIF and site-governance acceptance at a minimum of two consortium sites.