ML4H 2026 rebuttal: hazard-scored override, channel probes, event pins, GEMINI stages, comparison tools - #274
Merged
Merged
Conversation
The submission gives no model size or training settings. This script renders them from each flagship run's banked config.json (MIMIC-IV full_run_v10, eICU-CRD eicu_full_v10; GEMINI's config was never exported and prints as such) and appends the GBM comparator's settings read from the code itself: HistGradientBoostingClassifier, the four GBM_GRID configurations, the 400-round budget with the best round picked on a subject-grouped 10% split, and the 609-feature panel with its 110 count features computed from feature_names/feature_groups. Parameter counts are supplied through --params (run name -> integer) and print as tbd until the VMs compute them. Co-Authored-By: Claude Fable 5.1 <[email protected]>
interventions.py gains --hazard-heads: the same streaming pass reads the per-event hazard heads at the alert protocol's landmark rows under every mode and reports, per event and horizon, the AUROC against the landmark outcome and the mean P(event within h), plus paired subject-clustered bootstrap differences (truth minus none, truth minus flip, flip minus none) on both, attached to the left mode's entry. Landmark rows come from alerts.py's own mask and outcome rule (position_visit_starts is factored out of collect_model_scores so both callers share it), the AUROC bootstrap is uncertainty.bootstrap_auroc_delta as in scripts/alerts_cis.py, and the mean-risk bootstrap draws whole subjects in vectorised blocks. Only landmark rows are kept in memory. Off by default: the JSON is byte for byte what it was. Tests cover the unchanged schema, the row set against collect_model_scores, interval coverage and collapse, and whole-subject resampling. Co-Authored-By: Claude Fable 5.1 <[email protected]>
scripts/probe_channel.py dumps, on a frozen mixture-bottleneck checkpoint, the concept probabilities k, the unnamed slot's probability u, the poles w+/w- and the unnamed embedding z_u at every scored held-out position (seeded, shard-stratified subsample), then fits fresh linear (or --mlp) readouts on a split by subject and scores exact next-event top-1 on the held-out subjects: k_only, k_plus_u, z_named, poles_mean_k, poles_both, h_bar, and the model's own head as the reference. Each carries a subject-clustered bootstrap CI and a paired CI on the ratio to the model's accuracy; the CTL leakage probes run on the same split. ConceptBottleneck gains mixture_parts(), which forward now uses, so the poles a probe reads are the poles the forward pass mixes. Hazard heads at landmarks are not dumped; the docs paragraph says why, gives the VM command lines and a runtime guess. Co-Authored-By: Claude Fable 5.1 <[email protected]>
… the ML4H rebuttal (WP5) Three self-contained, idempotent steps that write only aggregate JSON into scripts/gemini/out/evals/ through the existing key whitelist: - alerts-cis <run>: scripts/alerts_cis.py on the run's alerts_rows_allshards.parquet (the current GBM refit; GEMINI_ALERTS_TAG defaults to _allshards), hazard vs gbm, 1000 paired draws. Prints the dump's row and subject counts first; GEMINI_CIS_MAX_SUBJECTS maps to --max-subjects. alerts_cis.py gains n_subjects per cell, the dump counts, and a flat per-cell summary block. - panel-coverage <run>: new scripts/panel_coverage.py reports which of the 48 panel signals resolve on a source (GEMINI: 15 of 48, no non-invasive blood pressure) and, given metadata/codes.parquet, which resolved prefixes were charted. Names only. - cohort-counts <run>: new scripts/cohort_counts.py over the shards a run used: subjects, admissions, hospitals from hadm_id_hospital.parquet, admission years, LOS, sex and age where charted, and per-event prevalence under the alerts leg's own onset definitions. Counts under 10 leave as "<10". Works for any MEDS source dir. Tests pin each step's whitelist to the script's output keys so a new field refuses in CI rather than on the node (the #238 incident). docs/ml4h2026_rebuttal_plan.md WP5 carries the three commands. Co-Authored-By: Claude Fable 5.1 <[email protected]>
load_run pinned a checkpoint's concept list but rebuilt its hazard heads from today's alert registry. On eICU that registry gained sepsis3 with PR #222 (SOFA source config), so eicu_full_v10 (80 = 5 events x 16 bins) failed load_state_dict against a 96-row head, with sepsis3 spliced into the middle of the list so even a width match would have misread rows. - legacy_concept_pins: pinned_event_names next to pinned_concept_names, resolved from the run's own run_pins.json first, then LEGACY_EVENT_PINS by run name (eicu_full_v10, eicu_full_L_v10, eicu_full_ADD_v10, eicu_full_DEC_v10, eicu_full_DEC_v11, eicu_full_DEC_v12, eicu_full_DEC_v12_steer: the pre-#222 eICU runs, all five heads in the order printed at their training commits). check_event_count and checkpoint_num_events mirror the concept checks and name the run, both counts and the fix. The same runs gain the 26-concept pin. - train.build_model takes an optional event_names override; unset, behaviour is unchanged. hazard_event_names_for is the one place the unpinned list comes from. Training writes run_pins.json (concepts and events, in slot/head order) right after building the model. - load_run threads the pinned events into build_model and runs check_event_count before load_state_dict. Co-Authored-By: Claude Fable 5.1 <[email protected]>
…ebuttal/integration
…buttal/integration
…uttal/integration
…uttal/integration
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A --bank-on-cpu run on an A100 streamed all 37 held-out shards and then died in compute_ctl: the random projection was built on the default device and multiplied against banks on the other one, and the probe fits never moved batches to the probe device at all, so any bank/probe device split failed. 45 minutes of streaming were lost with it. leakage.py: _random_projection takes a device and compute_ctl builds it on the bank's device (and follows each bank in flat); the categorical and masked-multilabel fits move every batch to the probe device and score the tuning split batch by batch; scoring and ICL logits go through a batched predictor that returns CPU tensors, so roc_auc_score and numpy never see a CUDA tensor. probe_channel.py: the CTL banks stay on the bank device (the flag's whole point); the streamed bank is saved to <output-json stem>.bank.pt right after the pass (--bank-path overrides, --from-bank PATH reloads it and skips the checkpoint); the readout block is written before the CTL probes run and the ctl block is added at the end; the JSON is append-only (a file with a ctl block is refused without --overwrite, one without it is filled in place); the bank size is logged in GB. Tests: CPU-bank CTL with an explicit probe device and the device-safe projection helper, batched prediction, the --from-bank round trip (same readout accuracies, no model load), the fill-ctl path, and the refusals. Co-Authored-By: Claude Fable 5.1 <[email protected]>
scripts/compare_runs_paired.py inner-joins two alerts_rows.parquet dumps per event on (subject_id, visit_id, time_hours), refuses when more than 0.1% of either dump's rows are unmatched (both counts and the unmatched count in the message), and reports per (event, horizon) both hazard-head AUROCs with the paired subject-clustered bootstrap of b minus a (bootstrap_auroc_delta, reused, not re-implemented), the two GBM refits side by side without a bootstrap, next-event metrics from each inference_results.json, and a markdown table. Tests cover shifted scores, identical scores, a 5% row drop, duplicate keys, label disagreement, a two-subject clustering fixture, and main(). Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #274 +/- ##
==========================================
+ Coverage 89.89% 90.41% +0.52%
==========================================
Files 67 67
Lines 10666 11009 +343
==========================================
+ Hits 9588 9954 +366
+ Misses 1078 1055 -23
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Everything built for the ML4H 2026 author response (reviews Oct 5, response Oct 5-12). Plan and banked results:
docs/ml4h2026_rebuttal_plan.md, section "Results so far".Inference and analysis
odyssey/inference/interventions.py:--hazard-headsscores the label override (none / truth / flip / random) on the per-event hazard heads at the alert protocol's landmark rows, with paired subject-clustered bootstraps of truth minus none, truth minus flip and flip minus none. Off by default; old outputs reproduce byte for byte.scripts/probe_channel.py: fresh readouts on the concept probabilities, the poles and the full bottleneck (plus the CTL leakage probes), subject-split, with a saved feature bank and staged JSON writes.scripts/compare_runs_paired.py: paired hazard AUROC deltas between two runs on identical landmark rows; refuses row-set or label mismatches.scripts/make_hparams_table.py,scripts/cohort_counts.py,scripts/panel_coverage.py.Fixes
run_pins.jsonwritten at training time, legacy table for the pre-Add eICU SOFA support: 26/29 -> 29/29 concept resolution #222 eICU runs). Loadingeicu_full_v10on main failed with a head-width mismatch because Sepsis-3 now resolves on eICU.odyssey/inference/leakage.py: device-safe probes (bank on CPU, model on GPU).scripts/alerts_cis.py: subject counts and a flat summary block.GEMINI
scripts/gemini/run.shsteps:alerts-cis,panel-coverage,cohort-counts, each exporting aggregate JSON through the existing whitelist.Results banked with this code (under
research_journal/figure_data/, gitignored): override on hazards is inert on both databases; the forecast runs through the concept embeddings, not the probabilities (poles with k fixed recover everything, k alone recovers 45% on MIMIC-IV and 72% on eICU-CRD); the bottleneck costs under one AUROC point per cell like-for-like; RandInt at full scale gives no lever; GEMINI resolves 15 of the 48 panel signals.Checks
Pre-commit hooks pass;
uv run pytest tests: 1,503 passed, 18 skipped at the integration point (later commits added tests for the probe fix and the comparison script, all passing).🤖 Generated with Claude Code