Repository navigation
feat(extract): detect generated files by their content banner - #220
Merged
Merged
Conversation
Go's convention marks a generated file in its content, not its name, so a `payroll.go` carrying `// Code generated by protoc-gen-go. DO NOT EDIT.` beside hand-written code was invisible to a path-only check and could outrank the implementation it shadows. The verdict is now decided at index time and stored on the file row, so a query never reads a header to rank a result. The flag only ever ADDS to the path signal: a file the path table already demotes stays demoted, and a header scan can promote nothing. The window is bounded, and precision comes from the patterns rather than the `generat` stem pre-filter — a comment that merely discusses code generation is not a banner, which the corpus pins with a one-clause `by` line and a banner declared inside a function body as negatives. All three extraction write paths compute it from the source they already read — fresh index, incremental sync, and the outdated rebuild — because filling a default at any one of them would make `sync` disagree with `index --force` and break the byte-equality invariant the golden suite rests on. Each path has its own test. SCHEMA_VERSION 7 -> 8 carries the column. EXTRACTION_VERSION 8 -> 9 is what makes an existing index classify outdated and re-extract; a DDL-only bump would leave every old row stuck at zero forever, since sync skips unchanged files. Verified in both directions across two binaries: the new one upgrades a v8 index with `2 reindexed, 0 skipped`, and the old one refuses a v9 index as `future` rather than misreading it. STORAGE_PROTOCOL stays 2 — bumping it would make an old binary judge every phase future, a wider refusal than this warrants. Sixteen corpora regenerate for the column plus a new go corpus; nodes, edges and refs are byte-identical, because the flag never enters extraction output. mini and godot additionally normalise their index CREATE order, which is expected: they are migrated artifacts where the v7 migration appended an index the newer corpora take from the base schema.
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #220 +/- ##
==========================================
+ Coverage 95.28% 95.31% +0.03%
==========================================
Files 140 141 +1
Lines 77287 78022 +735
==========================================
+ Hits 73641 74367 +726
- Misses 3646 3655 +9
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier 4 item 11 of the post-
v1.5.0upstream sync (#1500, plus the Wrangler double-bybanner from57e0854).The defect
Go marks a generated file in its content, not its name. A
payroll.gocarrying// Code generated by protoc-gen-go. DO NOT EDIT.beside hand-written code was invisible to our path-only check:The control that made the reproduction conclusive: renaming the same file to
payroll.pb.go— content unchanged — already flipped the ranking, proving the demotion pipeline worked and only the signal was missing.Conservative by construction
The flag only ever ADDS to the path signal. A file the path table already demotes stays demoted; a header scan can promote nothing. Precision comes from the patterns, not the
generatstem pre-filter — the Go corpus pins a one-clausebyline and a banner declared inside a function body as negatives, so a comment that merely discusses code generation is not a banner.All three write paths, separately verified
codegraph-extract's engine returns anExtractionResultand never builds aFileRecord— three independent sites do, each already holding&source:index --forcemain.rs:4870sync.rs:760migrate.rs:130All three call the same
detect_generated_file(relative, &source). Filling a default at any one would makesyncdisagree withindex --forceand break the byte-equality invariant the whole golden suite rests on, so each has its own test. Zero hardcoded defaults remain in production code (the one atmain.rs:6962is inside#[cfg(test)]).The version bump is load-bearing, proven across two binaries
State: outdated (built with extraction version 8)→sync→ 2 reindexed, 0 skipped, column added, flags1/0State: future (built with extraction version 9)— fails closed, does not misread0 skippedis the point: a DDL-only bump would have skipped both unchanged files and left them stuck atgenerated = 0forever.SCHEMA_VERSION7→8 carries the column,EXTRACTION_VERSION8→9 forces re-extraction,STORAGE_PROTOCOLstays 2 because bumping it would make an old binary judge every phase future — a wider refusal than this warrants.Golden movement, exactly as enumerated
files.jsonschema.sqlcolby.dbnodes.json/edges.json/refs.jsonZero movement in the extraction artifacts is the proof the flag never enters extraction output — any movement there would have meant a leak. A new
reference/golden/go/corpus pins the verdict itself: 3 ones and 3 zeros, the negatives being files that mention generation without a banner.miniandgodotadditionally normalise their index CREATE order. That is expected and was predicted: they are migrated artifacts where the v7 migration appendedidx_edges_identity, while newer corpora take it from the base schema in alphabetical position —sqlite_masterorder reflects CREATE order andcompare_schemais a strict string comparison.Verification
make cigreen: 133 suites / 3408 passed / 0 failed (baseline 133 / 3371).reference/golden/erlang/edges.jsonstill holds its 2 legitimate recursivecallsself-edges.nodeandexplore, and both upgrade directions.The plan cleared three review rounds. Round 1 found the design not implementable as written — the function it cited never built a
FileRecord, and the missing piece would have silently broken sync/rebuild byte-equality. Round 2 found a criterion that ran the wrong code path (indexinstead ofsync, which is what reachesmigrate_project), a corpus count that was never counted, and three mutually inconsistent statements aboutmini.