Skip to content

perf: the discovery sweep (~6.4s stat/fingerprint pass over the full tree) dominates every dated command #1104

Description

@iamtoruk

Successor to #1034/#1057 (closed 2026-08-22). Measured on a real 21k-file / 9-provider corpus: warm, day-scoped codeburn today spends ~6.4s wall / ~5.6s CPU, and the dominant term is the scope-independent discovery pass — walking and stat()/fingerprinting every file in every provider tree regardless of the query's date range. Eliminating 89% of the redundant re-parses (#1057's mechanism, verified working) moved neither wall-clock nor CPU, proving the sweep owns the runtime.

Directions worth exploring (each needs its own never-lose analysis):

  • Scope the WALK, not just the load: prune directory subtrees by mtime (a project dir untouched since before the range can't contain in-range appends — but beware: mtime of a dir doesn't propagate from deep children on all filesystems)
  • fs-events/watch journal between runs (the resident serve child could maintain a dirty-set so warm queries stat only changed files)
  • Batch stat via readdir-with-types + only fingerprint files whose (size,mtime) moved vs the cached envelope — measure how much of the 6.4s is stat syscalls vs fingerprint hashing vs JS overhead first

Baseline numbers and methodology live in the #1057 settlement comment. The prize: warm dated commands in low hundreds of ms on corpora this size — the 'sub-second warm' target the perf sprint originally aimed at.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions