Multi-temporal crop and land-cover segmentation on Harmonized Landsat–Sentinel (HLS) imagery, with a phenological account of where and why segmentation models fail.
13-class semantic segmentation over 3,083 multi-temporal HLS chips (6 bands × 3 timesteps, 224×224, EPSG:5070). The goal was not a leaderboard number: it was to build a comparison that could be trusted, then explain the errors it produced in terms of crop physiology rather than loss curves.
| Model | Test mIoU | Test OA |
|---|---|---|
| Random Forest (24 per-pixel features) | 0.351 | 0.547 |
| U-Net, cross-entropy | 0.448 | 0.641 |
| U-Net, Dice + class-weighted CE | 0.475 | 0.660 |
| Prithvi-EO-1.0-100M, fine-tuned | 0.433 | 0.614 |
Published reference points on this dataset (IBM/NASA): U-Net 0.426, Prithvi-EO-1.0-100M 0.427, Prithvi-EO-2.0-300M 0.486. These use a different split and are context rather than a like-for-like claim.
Two things worth stating plainly:
- The tuned U-Net exceeds both published 100M-scale reference numbers. Loss design and class weighting mattered more here than model scale.
- The fine-tuned foundation model underperformed the from-scratch U-Net under an identical protocol. This is reported rather than omitted — see Foundation-model comparison.
Water is nearly solved (0.777) on spectral distinctiveness alone. Corn and soy — the two classes a single-date classifier struggles most with — reach 0.562 and 0.570, and the next section explains why.
Mean NDVI trajectories were computed per class across the three timesteps, then compared against the tuned U-Net's symmetric confusion matrix.
Classes with similar phenological trajectories are confused at predictable rates: r = −0.487, p < 0.00001 (n = 78 class pairs).
Slope distance — how the curve changes rather than where it sits — is also predictive (r = −0.384, p = 0.00051), though the two measures are collinear and a joint regression cannot separate their contributions (standardised β: level −0.015, slope −0.010). The defensible claim is that phenological similarity predicts confusion, not that shape matters independently of level.
| Pair | Curve distance | Confusion rate |
|---|---|---|
| Forest ↔ Wetlands | 0.057 | 0.205 |
| NatVeg ↔ Dev/Barren | 0.029 | 0.119 |
| NatVeg ↔ Alfalfa | 0.172 | 0.213 |
| Corn ↔ Soy | 0.076 | 0.092 |
Corn and soy are the interesting exception. They begin the season nearly identical (NDVI 0.321 vs 0.319) and their curves cross between the mid and late acquisitions — corn peaks and senesces earlier, soybeans keep climbing. No single timestep separates them; the trajectory does. That is the clearest available demonstration of why multi-temporal input carries information a single-date classifier cannot access, and it is measured here rather than asserted.
Winter wheat is the other clean case: the only class whose NDVI genuinely falls across the season (0.443 → 0.393 → 0.306), consistent with mid-summer harvest exposing bare soil.
The two pairs the phenology analysis predicts are outlined in red. Forest↔Wetlands is the strongest off-diagonal block in the matrix; corn↔soy is visibly weaker despite comparable curve distance.
Alfalfa scores poorly (0.346 IoU) despite a distinctive mean curve. The initial hypothesis was high between-field variance — alfalfa is cut several times per season, so fields sit at different points in the cut cycle on any given date, and a mean curve would hide that.
The data rejected it. Alfalfa has among the lowest between-field standard deviation (0.138), and across all classes the correlation between within-class variance and IoU is positive (r = +0.585, n = 13, suggestive rather than established at this sample size). Water has the highest variance (0.216) and the highest IoU (0.777).
The revised account: variance arising from spectral distinctiveness helps, variance arising from ambiguity hurts, and the original framing conflated the two. Alfalfa's low score is better explained by its 0.213 confusion rate against Natural Vegetation, which occupies 21% of pixels — when two classes are confusable, the rarer one absorbs the errors.
Tile-disjoint splits. Chips are grouped by HLS tile before splitting, so no tile contributes to more than one partition (train 2,151 chips / 291 tiles; val 482 / 62; test 450 / 63; tile overlap: empty). Chip-level random splitting would place spatially adjacent, near-duplicate chips on both sides of the boundary and inflate every number reported here.
Identical protocol across models. Same splits, same loss (class-weighted CE + Dice), same schedule and epoch budget for both the U-Net and the fine-tuned Prithvi encoder. The only intentional difference is the learning rate: the pretrained encoder trains at 1e-4 while the randomly initialised decoder and head use 1e-3, to avoid washing out the pretrained representation in early iterations.
Verified weight loading. load_state_dict(..., strict=False) silently skips tensors whose names do not match. An earlier version of the encoder used nn.Sequential for the MLP block, producing mlp.0 / mlp.2 where the checkpoint expects mlp.fc1 / mlp.fc2 — 48 tensors, roughly two-thirds of the transformer, would have remained randomly initialised while the run still trained and produced a plausible number. The loader now asserts an exact match and verifies numerically:
missing: []
unexpected: []
tensors not matching checkpoint: []
loaded 150/150 encoder tensors
Any "fine-tuned foundation model" result without a check of this kind should be treated as unverified.
Training sets were subsampled by tile, not by chip — sampling 10% of chips at random still touches nearly every tile and is a thinner sample of the same geography, not a label-scarce regime. All runs use a reduced 25-epoch budget, which is why the 100% row differs from the headline table.
| Labels | Chips | U-Net | Prithvi |
|---|---|---|---|
| 10% | 188 | 0.291 | 0.242 |
| 25% | 486 | 0.363 | 0.338 |
| 100% | 2,151 | 0.445 | 0.409 |
Retention at 10% of labels: U-Net 65.3%, Prithvi 59.2%.
The expectation going in was that pretraining would pay off most under label scarcity. It did not, at this scale and on this dataset. The from-scratch model retained more of its full-supervision performance at every fraction tested.
Prithvi-EO-1.0-100M was fine-tuned as a controlled comparison arm: a ViT-B encoder with 3D patch embedding (6 bands × 3 timesteps, 16×16 spatial patches, 589 tokens including CLS), pretrained by masked autoencoding on HLS imagery. The MAE decoder is discarded and replaced with a convolutional upsampling head.
It reached 0.433 mIoU, below the tuned U-Net's 0.475 and below IBM's own published 0.427 for the same checkpoint.
Plausible contributors, none individually verified: a single T4 constrains batch size to 8 and rules out the longer schedules foundation-model fine-tuning typically wants; the segmentation head here is a simple convolutional decoder rather than the UperNet-style head used in the reference implementation; and 2,151 training chips is a regime where a well-regularised U-Net is competitive by default. IBM's own results place the 100M model within noise of their U-Net, so the direction of this result is not anomalous — the 300M and 600M variants are where their scaling advantage appears.
The honest summary is that model scale was not the binding constraint on this dataset. Loss design was.
notebooks/
crop-segmentation.ipynb end-to-end: data, splits, models, analysis
figures/
phenology.png per-class NDVI trajectories
confusion_unet.png confusion matrix, tuned U-Net
per_class_iou.png per-class IoU vs class frequency
label_efficiency.png mIoU vs label fraction
results/
cm_unet_v2.npy confusion matrix, tuned U-Net
label_sweep.json label-efficiency sweep
per_class_iou.md per-class IoU table
splits.csv tile-disjoint split assignment
Requires a HuggingFace token with access to the gated ibm-nasa-geospatial repositories.
pip install torch rasterio pandas scikit-learn huggingface_hub matplotlib scipyRun the notebook top to bottom. On a single T4 the full sequence — U-Net, Prithvi fine-tune, and the label-efficiency sweep — takes roughly 4 hours. All randomness is seeded (random_state=42); splits are written to results/splits.csv for exact reproduction.
ibm-nasa-geospatial/multi-temporal-crop-classification — HLS chips with USDA Cropland Data Layer labels, 13 classes. Reflectance is scaled by 1e-4; nodata (−9999) is masked; label 0 is treated as ignore.
- Single split, single seed. No confidence intervals on the headline mIoU figures.
- The variance and frequency correlations are computed over 13 classes and should be read as suggestive.
- Comparisons against published IBM/NASA numbers use a different (stricter) split.
- NDVI trajectories use class means over a 400-chip sample; within-class distributions are summarised only by standard deviation.