X-Ray

Chest X-ray pathology classification trained separately on each demographic subgroup, then evaluated across every other subgroup, to measure how far performance drops when a model is used outside the population it was trained on.

Five subgroups: female, male, young, middle-aged, old. Two model families: a DenseNet121 following the Stanford CheXNeXt setup, and an Xception model in Keras. Four pathologies: Nodule, Infiltration, Effusion, Atelectasis.

The question

Train a model on female patients only, and it does well on female patients. What does it do on male patients, or on old patients? Every subgroup model here is evaluated on all five validation sets, which gives a 5 by 5 matrix where the diagonal is in-domain and everything else is a domain shift.

Data

NIH ChestX-ray14, downloaded from the twelve official archives at https://nihcc.app.box.com/v/ChestXray-NIHCC.

The subgroup splits are pre-built CSVs (train_F.csv, valid_F.csv, train_young.csv and so on) that live on the author's Google Drive and are not in this repository. Each subgroup has 10,000 training images and 4,000 validation images. There is no held-out test set. The age boundaries separating young, middle and old are not recorded in any file here.

Labels are multi-label over four pathologies rather than the full fourteen, in the order Nodule, Infiltration, Effusion, Atelectasis.

Models

DenseNet121 (PyTorch). ImageNet-pretrained torchvision DenseNet121 with the classifier replaced by a 4-unit linear layer. MultiLabelSoftMarginLoss, Adam at 1e-4, batch size 16, 224 by 224 input, 10 epochs, ReduceLROnPlateau with patience 1, early stopping after 3 epochs without improvement. No augmentation is enabled, so the training and validation transforms are identical. The hyperparameters are dumped verbatim in each */ep_10/run_dir/params.txt and are the same for all five runs.

At prediction time every saved epoch checkpoint is loaded and their sigmoid outputs are averaged, so each subgroup result is a small ensemble. The ensemble size varies (6 for female, 8 for male, middle and old, 10 for young) because early stopping fired at different epochs.

Xception (TensorFlow / Keras). ImageNet-pretrained Xception with the top removed, then global average pooling, a 1024-unit ReLU dense layer, and a 4-unit sigmoid output. Trained in two phases: 9 epochs with the base frozen at batch size 64 and learning rate 1e-4, then 1 fine-tuning epoch with everything unfrozen at batch size 32 and 1e-5. Binary cross-entropy, 299 by 299 input, balanced class weights. Only the female and male Xception runs are in the repo.

Results

DenseNet121, macro AUC over the four pathologies

Rows are the subgroup a model was trained on, columns the validation set it was scored against.

Trained on	valid F	valid M	valid young	valid middle	valid old
Female	0.8292	0.7645	0.8006	0.7917	0.7704
Male	0.7557	0.8276	0.8056	0.7939	0.7664
young	0.7806	0.7850	0.9279	0.7564	0.7245
middle	0.7958	0.8039	0.7777	0.8197	0.7540
old	0.7840	0.7966	0.7641	0.7644	0.8737

The diagonal is the highest value in every row without exception. The steepest fall is the young model, 0.9279 on young patients against 0.7245 on old patients, a gap of 0.203. The gender pair is more symmetric: female on female 0.8292 against female on male 0.7645, male on male 0.8276 against male on female 0.7557.

Age transfers worse than gender. Each age model loses most on the age group furthest from it, and the two gender models sit close together on all three age sets.

These numbers come from the .npy filenames under each predict/ directory, which encode the roc_auc_score computed at prediction time.

DenseNet121, per-pathology detail

From the training log retained in the notebook, which belongs to the old model on valid_old. Best validation loss 0.2996 at epoch 5, training stopped after epoch 9.

Epoch	Nodule	Infiltration	Effusion	Atelectasis	Weighted mean ROC AUC
1	0.7495	0.7136	0.8632	0.7567	0.771
3	0.8108	0.7545	0.8958	0.8079	0.814
5	0.8410	0.7885	0.9127	0.8391	0.842
7	0.8460	0.8062	0.9075	0.8431	0.849
9	0.8473	0.8396	0.9224	0.8647	0.870

Effusion is consistently the easiest of the four and Infiltration the hardest. Training loss falls to 0.028 by epoch 9 while validation loss bottoms out at epoch 5 and then rises, which is plain overfitting, though the ensembling over checkpoints softens it.

Xception, per-pathology AUC

Trained on	Evaluated on	Nodule	Infiltration	Effusion	Atelectasis	Mean
F	F	0.6575	0.6560	0.8693	0.7049	0.7219
F	M	0.5927	0.6419	0.8676	0.6979	0.7000
F	young	0.6595	0.6724	0.8449	0.7258	0.7257
F	middle	0.6049	0.6479	0.8436	0.7083	0.7012
F	old	0.6136	0.6458	0.8282	0.6430	0.6827
M	F	0.5834	0.5853	0.8306	0.7156	0.6787
M	M	0.5915	0.6210	0.8685	0.7624	0.7108
M	young	0.6043	0.6136	0.8263	0.7650	0.7023
M	middle	0.5789	0.6000	0.8330	0.7470	0.6897
M	old	0.6078	0.6229	0.8163	0.7104	0.6894

Xception is behind DenseNet121 everywhere, by roughly 0.10 macro AUC in-domain. Its one epoch of fine-tuning also made things worse: training AUC peaked at 0.7335 in epoch 9 with the base frozen and dropped to 0.7016 after unfreezing.

The confusion matrices in Xeception/result_2/ show why the mean AUC understates the problem. At the chosen thresholds the model predicts Nodule positive zero times in all five groups, and Infiltration between 1 and 31 times out of roughly 800 positives. Effusion and Atelectasis carry the whole result.

What's in the repo

DenseNet121.ipynb      Everything for the PyTorch side: data loading, model,
                       training, checkpoint-ensemble prediction, ROC plotting,
                       Grad-CAM

CheXneXt_gender/Female/ep_10/    DenseNet121 female model
Male/ep_10/                      DenseNet121 male model
young/ep_10/                     DenseNet121 young model
middle/ep_10/                    DenseNet121 middle model
old/ep_10/                       DenseNet121 old model
  run_dir/params.txt             Hyperparameters for the run
  predict/<group>/predictions/<timestamp>/
    <auc>-valid_<group>.npy      4000 x 4 sigmoid probabilities
    params.json                  Which checkpoints were ensembled, and their losses

Xeception/
  Xeception.ipynb                Keras Xception training and evaluation
  grad_cam.ipynb                 Keras Grad-CAM over five saved models
  result_1/{F,M}/                Per-epoch metrics, predictions, per-class ROC CSVs, ROC plots
  result_2/                      Confusion matrices per group and pathology

result_ROC/                      31 comparison plots, grouped by what they compare:
                                 one model across demographics, one demographic
                                 across models, both models per pathology, and
                                 all ten models on one axis

The directory name Xeception is a typo for Xception. It is left alone here so the paths match what is on disk.

Running it

Both notebooks were written for Google Colab and cannot run as checked in. They mount Google Drive, read the split CSVs and saved checkpoints from /content/drive/MyDrive/..., and none of those files are in the repository. DenseNet121.ipynb downloads the NIH archives itself, so the images are reproducible, but the subgroup split CSVs would have to be rebuilt from Data_Entry_2017.csv and the exact age boundaries are not recorded anywhere.

No trained weights are committed. What is preserved is the prediction arrays, the hyperparameters, the checkpoint loss history, and the plots.

Caveats

• result_ROC/ holds images only. The AUC values on those charts are rendered into the PNGs and are not available as text.
• The Total_AUC column in the Xception per-class ROC CSVs is a duplicate of Nodule_AUC, caused by an index reset in the script that split the callback output. Ignore it.
• The accuracy figures in the Xception CSVs (around 0.10 to 0.16) are Keras exact-match multi-label accuracy across all four labels at once, not per-class accuracy.
• The Grad-CAM cells at the end of DenseNet121.ipynb list the trained checkpoints but never load them, so they produce saliency maps for an ImageNet DenseNet121, not for the chest X-ray models.
• The confusion_matrices.csv files record the evaluation group but not which trained model produced the predictions.
• young/ep_10/predict/young/ contains the same result twice, in predictions/1732372765129/ and in predictions1732908849427/. The second is a missing slash in the output path.
• Some comments in DenseNet121.ipynb are Chinese text stored with the wrong encoding and render as mojibake.

Credit

The PyTorch training, evaluation and prediction code in DenseNet121.ipynb is adapted from the Stanford ML Group's CheXNeXt reference implementation. The argument parser, the Dataset and evaluate helpers, the weighted loss, the checkpoint-ensembling predictor and the best-model selection are all from there, with the original usage docstring left in place. The runs here deviate from that documented recipe: 224 pixel input instead of 512, batch size 16 instead of 8, and no horizontal flip or class weighting.

The DenseNet class body is the standard torchvision implementation. Images are from the NIH Clinical Center ChestX-ray14 dataset.