fix(ENGKNOW-3670): don't rebuild gord dictionary through the .gord.link on a cache hit - #131
Conversation
There was a problem hiding this comment.
Pull request overview
Fixes a result-cache regression where rerunning pgor ... | write x.gord on a cache hit could crash with Not a directory by ensuring dictionary generation resolves .gord.link entries to the real gord folder and skips unnecessary rebuilds when thedict.gord already exists.
Changes:
- Update
GeneralQueryHandler.generateDictionaryFileto resolve.gord.linkpaths and only rebuildthedict.gordwhen missing, targeting the resolved folder. - Add a ScalaTest regression suite covering both “cache hit with dict present” and “cache hit with dict missing (fallback)” scenarios.
- Add design + implementation-plan documentation for ENGKNOW-3670 take-2.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| gortools/src/main/scala/gorsat/QueryHandlers/GeneralQueryHandler.scala | Resolve cache link paths before dictionary existence checks and rebuilds to avoid writing through .gord.link. |
| gortools/src/test/scala/gorsat/Utilities/UTestGenerateDictionaryFile.scala | New regression tests ensuring no rebuild/crash on cache hits and no writing under the raw link path. |
| docs/superpowers/specs/2026-07-22-engknow-3670-gor-rewriting-gord-fails-take-2-design.md | Design write-up documenting root cause, fix approach, and scope boundaries. |
| docs/superpowers/plans/2026-07-22-engknow-3670-gor-rewriting-gord-fails-take-2.md | Step-by-step implementation plan and test expectations for the fix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
f923139 to
0571c75
Compare
0571c75 to
68446e0
Compare
…nk on a cache hit On re-run with an active result cache, GeneralQueryHandler.generateDictionaryFile received the raw `<fingerprint>.gord.link` cache entry. It checked project-root/ thedict.gord (never present) so it always rebuilt, and rebuilt through the unresolved link, writing thedict.gord (and its atomic-write temp sibling) inside the `.gord.link` FILE -> "Not a directory". Resolve the link to the real gord folder first; skip the rebuild when thedict.gord is already there (the normal cache-hit case), and target the resolved folder in the rare rebuild fallback so no path is ever concatenated onto the link file. The now unused fileRoot parameter (and its executeBatch local) is removed. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
68446e0 to
ed16508
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
gortools/src/test/scala/gorsat/Utilities/UTestPGorGordFolderDelete.scala:173
Files.walk(root)returns ajava.util.stream.Streamthat should be closed. As written, the stream is never closed, which can leak file descriptors and cause flaky tests on some platforms (notably Windows). Wrap it in try/finally and close the stream.
/** Absolute paths of every regular file under `root` whose name contains "-temp-". */
private def tempFilesUnder(root: Path): List[Path] =
Files.walk(root).iterator().asScala.filter(Files.isRegularFile(_))
.filter(_.getFileName.toString.contains("-temp-")).toList
ed16508 to
087175e
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
gortools/src/test/scala/gorsat/Utilities/UTestPGorGordFolderDelete.scala:173
Files.walk(root)returns a Java Stream that must be closed. As written,tempFilesUnderleaks the stream (can cause file-handle leaks / directory locks, especially on Windows), which can make tests flaky over time.
private def tempFilesUnder(root: Path): List[Path] =
Files.walk(root).iterator().asScala.filter(Files.isRegularFile(_))
.filter(_.getFileName.toString.contains("-temp-")).toList
…ite paths White-box (UTestPGorGordFolderDelete.scala): fold in the ENGKNOW-3670 regression alongside the existing take-1 delete-gate suite, reusing its stub-cache session. (1) cache hit with thedict.gord present -> rebuild skipped, no crash, no temp file under the link; (2) fallback with thedict.gord missing -> rebuild targets the resolved folder, never writes through the raw `.gord.link` (asserts the outcome's cause chain names neither ".gord.link" nor "Not a directory", so it fails against the pre-fix code). No end-to-end test can catch this locally: TestUtils.runGorPipe uses LocalFileCacheClient, which re-runs the write when a cached link does not resolve, masking the server-cache crash. End-to-end (UTestGorWriteFolder.java): add testPgorOverwriteGordFolderWith- DifferentDataOnCacheMiss -- overwrite an existing gord folder with disjoint data on a cache miss (different query => different signature => real delete-and-rewrite) and assert only the second write's rows remain (no superset/stale dictionary). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
087175e to
9136780
Compare
Problem
pgor [...] | write x.gordcrashes withNot a directorywhen re-run against an active result cache (stg):Second attempt at ENGKNOW-3656 (#128), which fixed the local-cache case but not the server/result-cache case.
Root cause
On a cache hit,
GeneralQueryHandler.executeBatchhandsgenerateDictionaryFilethe raw<fingerprint>.gord.linkcache-entry path. That method then:fileRoot/thedict.gord(the project root — never present), so the skip-guard never fired and it rebuilt the dictionary on every cache hit; andwriteOutGorDictionaryFolderstring-concatenatedthedict.gord(and its atomic-write-temp-sibling) onto the.gord.linkfile →Not a directory.The result-cache stores a completed gord folder as a
.gord.linkwhose content is the real folder path, so on a genuine cache hit the rebuild is unnecessary.Fix
GeneralQueryHandler.generateDictionaryFilenow:.gord.linkto the real folder (resolveUrl(cacheFile, false).getFullPath()— same mode already used atGeneralQueryHandler.scala:80);thedict.gordalready exists in the resolved folder (the normal re-run case — crash gone, and the wasteful every-hit rebuild is removed for local caches too);thedict.gordmissing) rebuilds targeting the resolved folder, never the raw link.Scope: take-1 delete-gate in
PGor/PartGor/ParallelandwriteOutGorDictionaryFolder's path construction are untouched.Tests
New
UTestGenerateDictionaryFile(white-box, stub.gord.link→ folder):thedict.gordpresent → no rebuild, no crash, no-temp-file;thedict.gord→ fallback never writes through the raw.gord.link(captures the outcome and asserts the cause chain names neither.gord.linknorNot a directory; fails against pre-fix code)../gradlew :gortools:testScala→ 666 succeeded, 0 failed (incl. take-1'sUTestPGorGordFolderDelete). pgor/partgor/write JUnit suites green.🤖 Generated with Claude Code