imagecache: Re-unpack a layer retired while joining its flight

[igooch]

Fixes #1076

The bug

ensureLayer and retireLayer deliberately share one singleflight key per layer — that sharing is the reuse/retire interlock — so either can join a flight the other is leading. The interlock was one-sided about joins: retireLayer treats a join as a veto (its closure never ran, so it learned nothing), but ensureLayer drew the opposite conclusion from the same situation. Do returned the other call's nil error, and ensureLayer returned the layer path as if verified.

When the flight it joined was a retirement, that path had just been renamed aside and nothing unpacked it back. pull recorded the layer anyway, and its re-verify failed a healthy pull with layer dir vanished during pull (evicted?), surfacing as an RPC retry. The window is a few microseconds wide, hence CI-only.

Correcting the diagnosis in #1076

The issue blames an eviction cutoff postdating ensureLayer's mtime refresh. That can't fire here: with the default minAge = 2m, cutoff = now - 2m while a refreshed dir has mtime ≈ now, so retireLayer always vetoes. That window is real only at minAge ≈ 0.

Evidence

Counting joins that leave no fs dir behind matches the failures exactly (34→34, 149→149, 191→191, 1→1): no failure ever occurred without a join, the rate tracks the flight-window width, and guarding the join drops it to zero while the joins continue.

A/B on this branch, same harness, 200 iterations per head start:

head start	without fix	with fix
100µs	66	0
200µs	182	0
400µs	186	0

The fix

Make the flight say what it did. Every layer-flight closure now returns a flightOp through the singleflight, so a caller that joined knows which operation it joined and treats the two cases differently:

• Joined an ensure: the collapse working as designed — one download shared by every waiter, success and failure alike. Sharing failures is deliberate: under a persistent failure (registry down, corrupt blob), per-waiter retries would multiply full-size download attempts by the number of waiters.
• Joined a retirement: no ensure-work happened, whatever its outcome — renamed away, vetoed, or failed, nothing it reports settles anything for a caller that wants the layer present. Go around and lead a flight of our own, which reuses the layer (refreshing its mtime inside the flight, preserving the interlock) or unpacks it, as the pool dictates.

Re-entry is bounded by maxEnsureLayerFlights — singleflight has no fairness, so "retry until we lead" has no bound of its own — with a ctx check between flights. No livelock: retireLayer vetoes on the symmetric join.

Presence checks now go through layerFSPresent, which reads only ErrNotExist as absence. A transient stat error (EIO and friends) is reported instead of being mistaken for a missing layer and triggering a re-download that would fail at the commit rename anyway.

Unchanged on purpose

pull's re-verify and TestPullReverifyFailsCleanlyOnYankedLayer stay. The re-verify guards a dir removed outside the flight, which the singleflight can't see. Fixing the cause didn't require weakening the backstop.

Testing

Three deterministic tests pin the three join cases. Each holds the layer flight open the way a real leader would and releases it only once the ensureLayer under test is observably parked joining it (waitForFlightJoiner scans the goroutine dump — releasing on a timer would let a loaded machine close the flight early and pass the test without ever exercising the join path):

• TestEnsureLayerJoiningRetireFlightRepacksLayer — joined a retirement that renamed the dir away: unpack it again. Fails on main, passes here.

• TestEnsureLayerJoiningFailedFlightKeepsLiveLayer — joined a retirement whose rename failed: the layer is live and usable, so lead a flight and return it rather than failing the pull with an eviction-internal error.

• TestEnsureLayerJoiningFailedEnsureSharesError — joined a failed ensure: propagate the leader's error rather than piling on download attempts.

• Full package -race: 70 pass, 0 fail. TestConcurrentEnsureImageAndEvict: 500 consecutive -race runs clean.

• make verify clean except metrics.sh (needs Docker), proto-fmt.sh (needs clang-format), shellcheck.sh — the latter two fail identically on main, and this touches no metrics, protos, or shell scripts.

Follow-up (separate issue)

TestConcurrentEnsureImageAndEvict tests less than it appears: over 5,000 iterations it produced 33 candidates, skipped all as fresh, and removed zero records — the EnsureImage hit beats the evictor ~99.3% of the time, so the eviction-wins branch is almost never taken.

[Taahir Ahmed (ahmedtd)]

Can you clean up the PR description? It's very difficult to follow.