What's wrong
LeastRecentlyUsedEvictionPolicy.Evict() deletes objects coldest-first until TotalBytes <= MaxSize * LowWaterMark (LeastRecentlyUsedEvictionPolicy.cs#L27-L65).`` Nothing compares a single object's size against that target. The miss path (ObjectRouteHandler.cs ~L335) and the upload tee (~L229) stage and publish an object of any size.
If one object is larger than the low-water target (90% of MaxSize by default), a sweep:
- deletes every other object, and the store is still over target;
- then deletes the large object as well, even though it is the hottest object because it was just fetched.
The store ends up empty. If that object keeps being requested, for example a large .umap or a build artifact that every CI pod pulls, the cycle repeats every MaintenanceInterval:
- the next miss re-fetches and re-stores it;
- the next sweep wipes everything again.
The hit ratio stays near zero, and the whole volume is rewritten every interval. This happens for an object between LowWaterMark × MaxSize and MaxSize, which fits the budget, as well as for one larger than MaxSize.
Repro
This test uses the Create/StoreAsync helpers in GitLfsCache.Tests/Storage/EvictionTests.cs:
(ObjectStore store, LeastRecentlyUsedEvictionPolicy policy, FakeTimeProvider time) = Create("1000B"); // target 900B
for (int i = 0; i < 5; i++) { await StoreAsync(store, 100, (char)('a' + i)); time.Advance(TimeSpan.FromHours(1)); }
await StoreAsync(store, 950, 'z'); // fits the budget, exceeds the low-water mark
EvictionResult result = policy.Evict();
// evicted=6 bytes=1450 remaining=0 total=0
Suggested fix
- Don't cache an object that can't survive a sweep. The href token already carries
Size. When token.Size is over a cap, skip TryOpenStaging and relay the object uncached, as the staging-failure path already does. The cap could be a new Store:MaxObjectSize setting, or default to a fraction of MaxSize × LowWaterMark. Apply the same rule in UploadAsync, and count these in a metric such as gitlfscache.objects_too_large.
- As a backstop, the sweep should not delete an object larger than the remaining excess while smaller objects can still be freed, and it should never empty the store for one object.
Acceptance criteria
- With
MaxSize=1000B and LowWaterMark=0.9, a 950-byte download is served correctly. Either it isn't stored, or it's stored without the store being emptied.
- An object larger than
MaxSize is relayed uncached, and the metric records it.
- The existing eviction tests still pass.
What's wrong
LeastRecentlyUsedEvictionPolicy.Evict()deletes objects coldest-first untilTotalBytes <= MaxSize * LowWaterMark(LeastRecentlyUsedEvictionPolicy.cs#L27-L65).`` Nothing compares a single object's size against that target. The miss path (ObjectRouteHandler.cs~L335) and the upload tee (~L229) stage and publish an object of any size.If one object is larger than the low-water target (90% of
MaxSizeby default), a sweep:The store ends up empty. If that object keeps being requested, for example a large
.umapor a build artifact that every CI pod pulls, the cycle repeats everyMaintenanceInterval:The hit ratio stays near zero, and the whole volume is rewritten every interval. This happens for an object between
LowWaterMark × MaxSizeandMaxSize, which fits the budget, as well as for one larger thanMaxSize.Repro
This test uses the
Create/StoreAsynchelpers inGitLfsCache.Tests/Storage/EvictionTests.cs:Suggested fix
Size. Whentoken.Sizeis over a cap, skipTryOpenStagingand relay the object uncached, as the staging-failure path already does. The cap could be a newStore:MaxObjectSizesetting, or default to a fraction ofMaxSize × LowWaterMark. Apply the same rule inUploadAsync, and count these in a metric such asgitlfscache.objects_too_large.Acceptance criteria
MaxSize=1000BandLowWaterMark=0.9, a 950-byte download is served correctly. Either it isn't stored, or it's stored without the store being emptied.MaxSizeis relayed uncached, and the metric records it.