Skip to content

One cached object larger than the low-water target makes the next eviction sweep empty the whole store, including that object #98

Description

@matt-edmondson

What's wrong

LeastRecentlyUsedEvictionPolicy.Evict() deletes objects coldest-first until TotalBytes <= MaxSize * LowWaterMark (LeastRecentlyUsedEvictionPolicy.cs#L27-L65).`` Nothing compares a single object's size against that target. The miss path (ObjectRouteHandler.cs ~L335) and the upload tee (~L229) stage and publish an object of any size.

If one object is larger than the low-water target (90% of MaxSize by default), a sweep:

  1. deletes every other object, and the store is still over target;
  2. then deletes the large object as well, even though it is the hottest object because it was just fetched.

The store ends up empty. If that object keeps being requested, for example a large .umap or a build artifact that every CI pod pulls, the cycle repeats every MaintenanceInterval:

  • the next miss re-fetches and re-stores it;
  • the next sweep wipes everything again.

The hit ratio stays near zero, and the whole volume is rewritten every interval. This happens for an object between LowWaterMark × MaxSize and MaxSize, which fits the budget, as well as for one larger than MaxSize.

Repro

This test uses the Create/StoreAsync helpers in GitLfsCache.Tests/Storage/EvictionTests.cs:

(ObjectStore store, LeastRecentlyUsedEvictionPolicy policy, FakeTimeProvider time) = Create("1000B"); // target 900B
for (int i = 0; i < 5; i++) { await StoreAsync(store, 100, (char)('a' + i)); time.Advance(TimeSpan.FromHours(1)); }
await StoreAsync(store, 950, 'z');   // fits the budget, exceeds the low-water mark
EvictionResult result = policy.Evict();
// evicted=6 bytes=1450 remaining=0 total=0

Suggested fix

  • Don't cache an object that can't survive a sweep. The href token already carries Size. When token.Size is over a cap, skip TryOpenStaging and relay the object uncached, as the staging-failure path already does. The cap could be a new Store:MaxObjectSize setting, or default to a fraction of MaxSize × LowWaterMark. Apply the same rule in UploadAsync, and count these in a metric such as gitlfscache.objects_too_large.
  • As a backstop, the sweep should not delete an object larger than the remaining excess while smaller objects can still be freed, and it should never empty the store for one object.

Acceptance criteria

  • With MaxSize=1000B and LowWaterMark=0.9, a 950-byte download is served correctly. Either it isn't stored, or it's stored without the store being emptied.
  • An object larger than MaxSize is relayed uncached, and the metric records it.
  • The existing eviction tests still pass.

Activity

  1. matt-edmondson commented on Oct 7, 2026

    @matt-edmondson
    ContributorAuthor

    Triage

    • Category: Bug
    • Priority: High. One large, frequently requested object (a .umap, a CI artifact) empties the whole store on every MaintenanceInterval. The hit ratio drops to about zero and the volume is rewritten each cycle. Nothing reports it, and it happens with default settings for any object between 90% and 100% of MaxSize, as well as for any object larger than MaxSize.
    • Area / suggested assignment: Storage and eviction (LeastRecentlyUsedEvictionPolicy.Evict), plus the admission paths in ObjectRouteHandler (miss staging around L335, upload tee around L229). It needs one owner across both, because the primary fix is at admission and the backstop is in the sweep.
    • Duplicates: none found.
    • In progress: no open PR covers this. The open PRs Bump Testably.Abstractions.Testing from 7.0.2 to 7.1.1 #83 and Bump Testably.Abstractions.FileSystem.Interface from 10.3.0 to 10.4.0 #53 are dependency bumps.
    • Next step: add an admission cap (Store:MaxObjectSize, defaulting below MaxSize × LowWaterMark) that relays oversized objects uncached and counts them in a metric. Then make the sweep skip an object larger than the remaining excess while smaller objects can still be freed. The repro in the issue is ready to use as the regression test.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions