fix(packages): read ebuild.lock back, so it pins what it records - #146
Kartikey1306 wants to merge 11 commits into
Conversation
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
4114be7 to
90010fd
Compare
srpatcha
left a comment
There was a problem hiding this comment.
Review — ebuild#146 "fix(packages): read ebuild.lock back, so it pins what it records"
head: 90010fda3d4fd990a6d2fd9a6d150a6d1188354a author: Kartikey1306 ci: pass (30 green, 0 red; only Create GitHub Release skipped)
Verdict: First look at f4788f2..90010fd (one commit, 6 files) as the body asks; the other 12
files belong to #132. The fix is right and I verified all four behaviours through the real CLI, not
through the tests: without a lock zlib resolves to 1.3.0, with a lock to 1.2.13, a lock naming a
version no recipe provides is refused, and a lock whose recorded URL no longer matches the recipe
is refused — both with clean exit 1 and an error that says what to do. Three findings, one of
which is the mirror image of the PR's best quality: the two failure paths the author designed
have excellent diagnostics, and the one they did not think about prints a fifty-line PyYAML
traceback.
Findings
| # | Severity | File:line | Finding | Recommended fix |
|---|---|---|---|---|
| 1 | Medium (P2) | ebuild/packages/lockfile.py:37-56; reachable from ebuild/cli/commands.py:89 |
A corrupt ebuild.lock now crashes the build with a traceback, and this PR is what makes that reachable. load() calls yaml.safe_load(f) with no handler. Until this commit nothing called load(), so a malformed lock was inert; now every configure, build and package runs it. Verified end to end with the real CLI on a project whose lock had one unterminated quote: exit 1 with a 50-line yaml.scanner.ScannerError traceback ending in click and PyYAML frames, after the log had already printed Resolving packages.... The call sites catch FileNotFoundError, ConfigError, RecipeError, ResolveError, FetchError and BuildError (commands.py:1364-1371); yaml.YAMLError is in none of them. IsADirectoryError and PermissionError escape the same way — I reproduced both out of load() directly, and neither is a FileNotFoundError, so neither is caught. What makes this worth a finding rather than a nit is the comment at :50-51: "a hand-edited or truncated file must not turn into a KeyError deep inside the resolver." A truncated file is precisely the case that still raises, one frame earlier than the one the comment guards. The sanitising dict comprehension below it handles a well-formed YAML file of the wrong shape — I confirmed packages: [zlib], zlib: 'a-string', zlib: [1,2,3] and an empty file all yield {} cleanly — which is the easier half. This repo already has the pattern one file over: commands.py:407-411 does except yaml.YAMLError as e: log.error(f"… is not valid YAML: {e}"); raise SystemExit(1). |
Wrap the read and raise the project's own error type so the existing handlers catch it: try: raw = yaml.safe_load(f) except yaml.YAMLError as e: raise ResolveError(f"{self.lock_path} is not valid YAML: {e}. Fix it, or delete it to resolve afresh.") — and catch OSError around the open() for the directory and permission cases. ResolveError is already handled at all three call sites and already produces [error] Error: …, so this lands the corrupt-lock case in the same shape as the two refusals this PR wrote by hand. One test per case; the truncated fixture is two lines. |
| 2 | Low (P3) | ebuild/packages/lockfile.py:76; ebuild/packages/resolver.py:186-200 |
build is recorded in the lock and never read — the exact shape of the defect this PR exists to fix. Lockfile.lock() writes four fields per package: version, url, checksum, build. _check_locked_bytes() compares ("url", "checksum"). So a recipe that keeps its name, version, URL and checksum and changes build: cmake to build: make passes the lock unchallenged and produces a different artifact from the same source. The PR's own argument — "the version reproduces the name; the lock is there to reproduce the bytes" — applies: the build system is part of what turns the source bytes into the installed bytes. Small and contained; I am rating it Low because a recipe edit of that kind is rarer than a URL change, not because the reasoning is weaker. |
Either add "build" to the tuple at :196 (comparing against recipe.build_system, which is the attribute name — getattr(recipe, "build") would AttributeError, so this needs a field→attribute map, not a fourth string), or stop recording it in lock(). Recording a field nobody reads is what #145 was about. |
| 3 | Low (P3) | docs/architecture.md:183; docs/ generally |
ebuild.lock has no user-facing documentation, and this PR is what makes it authoritative. grep -rn "ebuild.lock" docs/*.md README.md returns zero matches; the only description anywhere is architecture.md:183, "Lockfile (lockfile.py) — records exact resolved versions for reproducibility", inside an internal component list. That sentence was false before this PR and is true after, so nothing is made wrong (brief item 11 is not tripped) — but the change introduces a workflow a user will need and cannot look up: an existing lock now silently outranks "give me the newest", and the way to move forward is to delete the file or name the version. Today that appears only inside two error strings and the CHANGELOG. §9.2 asks for "reproducible lockfiles/manifests for production builds" and §25.2 for "excellent errors with suggested fixes"; the errors are excellent, and there is nowhere to read about the feature before hitting one. |
A short docs/ section, or a paragraph in the packages guide: what ebuild.lock is, the request → lock → newest precedence, that it is rewritten after every resolution, that it should be committed for reproducible builds, and how to upgrade (delete it, or pin explicitly). Ten lines. |
Recorded as a limit, not a finding: _check_locked_bytes() skips any field the lock recorded as
empty, and lock() records recipe.checksum verbatim — so for a recipe with no checksum, the lock
pins a URL, which is mutable content, not bytes. That is the correct behaviour given what the
recipe provides and the docstring says so ("Only fields the lock recorded are compared"); it does
mean the body's "the lock is there to reproduce the bytes" holds only for recipes that carry a
checksum. Worth a sentence wherever finding 3's documentation lands.
What this PR gets right
The precedence order is the right one and is argued rather than asserted: the request is a
statement of intent, the lock is a record, and the record is rewritten from the result — so
"explicit request outranks the lock" is not an arbitrary tie-break. Both refusals are errors
rather than fallbacks, which is the whole point; a lockfile that silently resolved to something
else would be worse than no lockfile. And the error strings are genuinely good — each one names
the file, the package, both values, and three ways out. _locked_entries() filtering out
explicitly-pinned names before resolution starts, rather than checking precedence at each node, is
also the cleaner of the two available shapes.
Verification performed for this review
Detached worktree at .ai/autoreview/state/scratch/ebuild-146 on refs/pull/146/head, using the
run's shared venv. The ebuild checkout has a dirty working tree — TASKS.md,
ebuild/cli/integration.py, tests/ebuild/test_integration_initramfs_security.py and an untracked
smart-sensor/ — and I left it exactly as found, re-checked after all work. Nothing was
committed or pushed. The CLI probes ran against throwaway projects under /tmp.
| Check | Result |
|---|---|
pytest tests/ -q |
PASS — 690 passed, 1 skipped, 6.21s — matches the body exactly |
pytest tests/unit/test_resolver.py tests/unit/test_lockfile_cli.py -q |
PASS — 18 passed |
ruff 0.16.5 check . |
PASS — All checks passed! |
| End to end, real CLI, two recipes (zlib 1.2.13 and 1.3.0), no lock | Packages to install: zlib v1.3.0 — newest, as before |
| Same project, lock pinning 1.2.13 | Packages to install: zlib v1.2.13 — the headline claim, confirmed against the binary, not the unit tests |
| Lock pinning v9.9.9, which no recipe provides | exit 1, [error] Error: ebuild.lock pins 'zlib' at v9.9.9, and no recipe provides that version. Restore the recipe, or request the version you want explicitly, or delete ebuild.lock to resolve afresh. |
| Lock pinning v1.2.13 with a stale URL | exit 1, [error] Error: ebuild.lock pins 'zlib' v1.2.13 with url '…/OLD.tar.gz', but the recipe for that version now has '…/zlib-1.2.13.tar.gz'. … |
Corrupt lock (unterminated quote), real CLI ebuild configure |
exit 1, 50-line yaml.scanner.ScannerError traceback — finding 1 |
Lockfile.load() shape probes |
packages: [zlib], zlib: 'string', zlib: [1,2,3], empty file → all {} clean ✅ · truncated, tab-indented, unclosed flow mapping → ScannerError/ParserError ❌ · lock path a directory → IsADirectoryError ❌ · mode 000 → PermissionError ❌ |
Negative control — locked = {} forced in resolve(), rebuilt |
3 of 18 FAIL: test_lockfile_pins_a_package_the_request_leaves_open, test_a_locked_version_no_recipe_provides_is_an_error_not_a_fallback, test_the_lock_notices_a_recipe_whose_bytes_changed. The tests genuinely pin the new behaviour. (The other two lock tests assert the absence of an effect and pass either way — correct for controls.) Restored; git status clean. |
| Was anything weakened? | No. No test disabled, no assertion removed, no lint rule loosened, no || true, no permission widened. resolve() gains an optional parameter, so existing callers still work. |
| Public API change (brief item 8) | PackageResolver.resolve(requested, lockfile=None) — additive and backward compatible. The user-visible behaviour change (an existing lock now outranks "newest") is stated in the body and the CHANGELOG, with the migration path (delete the lock, or request a version). Item 8 satisfied; finding 3 is that it is stated nowhere a user would look. |
Architecture conformance
Conforms. §21 places ebuild in Tier 1 — Foundation; every file is inside the owning repo and
§21.1 is not engaged. §5.1 dependency direction holds and the new import is downward within the
package: packages/resolver.py → packages/lockfile.py → packages/recipe.py, with no cycle
(resolver already reached recipe via registry). "eBuild understands the complete graph but is
not a runtime dependency" is untouched — none of this ships to a device. The clause this PR serves
is §9.2, "Reproducible lockfiles/manifests for production builds", and it is the difference
between claiming that and having it: before this commit ebuild.lock was written and never read,
so §9.2 was satisfied by a file, not by a behaviour, and §28's evidence policy would have rated it
Planned described in the present tense. It is now Implemented with functional tests, which
is what §28 requires for that word. §23.2's "eBuild project format — backward-compatible
project/manifest migration policy" is respected: the lock format is unchanged, lockfile_version: 1
still, and old locks are read by the new code.
Proposed changes
Before merge, small:
lockfile.py:42-43 wrap yaml.safe_load in except yaml.YAMLError -> ResolveError,
and the open() in except OSError; the existing call-site
handlers then catch it (finding 1)
tests/ a truncated-lock fixture and an unreadable-lock case
After, not blocking:
lockfile.py:76 check "build" too, or stop recording it (finding 2)
docs/ ten lines on what ebuild.lock is and how to upgrade (finding 3)
No fix PR opened. Finding 1 is Medium, and the brief's autofix rule is High-only. It is also
in lockfile.py's load() as this branch rewrites it — a branch cut from origin/master, which
is what fix-start.sh produces, would be patching a function with a different body. It belongs to
the author, in this PR.
Not checked
- mypy — NOT RUN. Not installed in the scratch env. The body's "mypy clean" is unverified
locally; the greenTest (Python 3.x, os)legs imply theType check (mypy)step passed, since
ci.ymlruns it before the tests with nocontinue-on-error. Inferred. Note that finding
2's suggested fix would need a field→attribute map rather than a fourth string in the tuple, and
mypy would not catch the naive version becausegetattr()is untyped. - yamllint — NOT RUN. Not installed.
- No package was actually fetched or built. My end-to-end runs used
https://example.invalid
URLs and stopped at resolution, which is the code this PR changes. I did not verify that a
locked resolution then fetches, checksums and builds the pinned archive — that path is unchanged
by this diff but is where a lockfile's value is finally realised. - Concurrency — NOT EXAMINED.
_install_packagesbuilds packages across threads
(commands.py:104-118), andlockfile.save()runs after. I did not check what twoebuild
processes in one project directory do toebuild.lock;save()is a plain truncating write with
no locking, and that was true before this PR as well. pyteston Python 3.10 and 3.11 — NOT RUN. My venv is 3.12. CI covers them.- The #132 half of the 18-file diff was not reviewed here — it is reviewed separately in this
same run at headf4788f2c. If #132 changes, the 690-passed figure above no longer describes
what will merge. mergeStateStatusisBLOCKED—REVIEW_REQUIREDplus the #132 stack.
Automated architecture review of 90010fda3d4f — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.
|
All three taken, in 1 — a corrupt lock crashed with a traceback, and this PR made that reachable. Agreed, including the point about my own comment: a truncated file did still raise, one frame earlier than the one the comment guarded. 2 — 3 — no user-facing documentation. Gates on |
srpatcha
left a comment
There was a problem hiding this comment.
Review — ebuild#146 "fix(packages): read ebuild.lock back, so it pins what it records"
head: bf2a776 author: Kartikey1306 ci: pass — 30 green, 0 red; only Create GitHub Release skipped
Verdict: Follow-up on one new commit, bf2a776, which answers all three findings from
ebuild-146-90010fda.md. All three are resolved, and I confirmed each by execution rather
than by reading: the build-field comparison refuses a drifted recipe through the real CLI, the
corrupt-lock path reports [error] … is not valid YAML with no traceback, and the six new tests
all fail against 90010fd's implementation — the author's own negative control, reproduced.
One residual case of finding 1 survives the fix and I reproduced it end to end, plus one
cosmetic nit in a new test. Both are Low.
Follow-up on the previous review
| Prior # | Sev | Status | Evidence at this head |
|---|---|---|---|
| 1 | Medium (P2) | Resolved for the cases it named; one residual — see new finding 4 | lockfile.py:61-73 wraps the open()/yaml.safe_load() in except yaml.YAMLError and except OSError, raising a new LockfileError; commands.py:89-95 converts it to ResolveError at the single load() call site, which the three handlers at commands.py:1078, :1271 and :1374 already catch. Declining to raise ResolveError from lockfile.py to avoid a resolver↔lockfile cycle is the right call. Reproduced through the real CLI: unterminated quote → exit 1, [error] Package error: … ebuild.lock is not valid YAML: …, no traceback. A directory in place of the file → LockfileError: … could not be read. |
| 2 | Low (P3) | Resolved, and generalised further than asked | Lockfile.CHECKED_FIELDS = (("url","url"), ("checksum","checksum"), ("build","build_system")) at lockfile.py:39-46 is the single list both sides use, with the field→attribute map the previous review said would be needed. resolver.py:193 iterates it. Verified through the real CLI, not only the tests: a recipe with build: make against a lock recording build: cmake, same version/URL/checksum → exit 1, [error] Package error: ebuild.lock pins 'zlib' v1.3.0 with build 'cmake', but the recipe for that version now has 'make'. …; with the lock corrected to make the same project resolves ([info] Packages to install: zlib v1.3.0) and proceeds to fetch. test_every_field_the_lock_records_is_one_the_resolver_compares asserts set(lock entry) - {"version"} == set(CHECKED_FIELDS), which is the part that stops this recurring. |
| 3 | Low (P3) | Resolved | docs/dependency-management.md:363-387 — a 25-line ebuild.lock section covering what it records, request → lock → newest precedence, that drift in a locked field is refused and why, that it should be committed, how to upgrade (name the version, or delete the file), and that a corrupt lock errors while a wrong-shape one is sanitised entry by entry. It is the only file under docs/ that mentions ebuild.lock and it is reachable from README.md:91. docs/architecture.md:183 ("records exact resolved versions for reproducibility") is now true rather than aspirational. |
The previous review's "recorded as a limit, not a finding" — that a recipe with no checksum
leaves the lock pinning a mutable URL — is not covered by the new documentation. Not a
finding; noted so it is not mistaken for covered.
Review of the new commit on its own merits
| # | Severity | File:line | Finding | Recommended fix |
|---|---|---|---|---|
| 4 | Low (P3) | ebuild/packages/lockfile.py:61-73 |
The corrupt-lock fix misses the non-UTF-8 case, which reaches the terminal as the same traceback finding 1 was about. open(..., encoding="utf-8") is inside the try, but UnicodeDecodeError is a ValueError — neither yaml.YAMLError nor OSError — so it escapes LockfileError, escapes the ResolveError handlers, and propagates out of the command. Reproduced end to end with a real ebuild build on a project whose ebuild.lock contains a 0xff byte: exit 1 preceded by a 44-line UnicodeDecodeError traceback ending in yaml/reader.py … <frozen codecs>, after the log had printed Resolving packages... — byte for byte the shape finding 1 described. A UTF-16-saved lock (the BOM alone does it) behaves identically. Rated Low rather than Medium because the cases a hand-edit or truncation actually produces — bad YAML syntax, an unreadable path — are now handled, and save() only ever writes ASCII; this needs a foreign encoding, a bad binary merge, or a partial overwrite. |
One word: except (yaml.YAMLError, UnicodeDecodeError) as e: at :63, keeping the same message (PyYAML's own reader raises yaml.reader.ReaderError for a decode failure it detects itself, so the two belong in one clause). A test alongside test_load_refuses_a_lock_that_is_not_yaml — path.write_bytes(b"packages:\n zlib: {version: '\xff'}\n") — is two lines. |
| 5 | Low (P3) | tests/unit/test_resolver.py:277-287 |
registry = make_registry_with_checksums(...) is assigned and then overwritten four lines later by registry = PackageRegistry(); the first call is there only for the zlib.yaml it writes, which the next statement appends build: make to. The name says the opposite of what the call is for, and a reader has to run it to find out. Ruff does not flag it because the name is later reused. |
Drop the binding: make_registry_with_checksums(tmp_path, {...}) on its own line, or name it _. Cosmetic. |
Nothing was weakened. No test disabled, no assertion removed, no lint rule loosened, no
|| true, no permission widened — checked across the whole follow-up diff. LockfileError is a
new public name in ebuild.packages.lockfile; it is purely additive, Lockfile.load()'s
signature is unchanged, and the behaviour change (a previously-uncaught traceback becomes a
clean error) is stated in the CHANGELOG entry bf2a776 rewrites. Brief item 8 satisfied.
Architecture conformance
Conforms, re-checked against the follow-up diff. §21 places ebuild in Tier 1 — Foundation;
every file is inside the owning repo, so §21.1 is not engaged. §5.1 dependency direction holds
and no new edge is introduced: resolver.py already imported Lockfile for Lockfile.FILENAME
and now also reads Lockfile.CHECKED_FIELDS from it; lockfile.py still imports only
packages/recipe.py, so there is no cycle — which is why the author converts LockfileError
to ResolveError in the CLI rather than raising ResolveError from lockfile.py. "eBuild
understands the complete graph but is not a runtime dependency" is untouched; none of this ships
to a device.
The clause this PR serves is §9.2, "Reproducible lockfiles/manifests for production builds",
and bf2a776 tightens it: the recorded build system is now part of what the lock reproduces,
which is the difference between reproducing a package's name and its installed artifact.
§25.2's "excellent errors with suggested fixes" now holds on the failure path that previously
printed a PyYAML traceback — for every input except finding 4's. §23.2's "eBuild project format
— backward-compatible project/manifest migration policy" is respected: lockfile_version: 1 is
unchanged and an old lock without a build field is still read, because _check_locked_bytes()
skips a field the lock recorded as empty.
No new proposal appended. Nothing here reveals a gap in the master design; §9.2 named the
requirement correctly and this is the implementation catching up to it.
Proposed changes
Small, before or after merge:
lockfile.py:63 except (yaml.YAMLError, UnicodeDecodeError) as e: (finding 4)
tests/unit/test_resolver.py a two-line non-UTF-8 fixture beside
test_load_refuses_a_lock_that_is_not_yaml
tests/unit/test_resolver.py:277 drop the discarded `registry =` (finding 5)
Blocking nothing. #132 still has to land first; this is stacked on it.
No fix PR opened. Both findings are Low and the brief's autofix rule is High-only. Both are
also in files as this branch rewrites them, so a branch cut from origin/master — which is what
fix-start.sh produces — would be patching different bodies. They belong to the author, in
this PR.
Verification performed for this review
The ebuild checkout has a dirty working tree (TASKS.md, ebuild/cli/integration.py,
tests/ebuild/test_integration_initramfs_security.py modified, smart-sensor/ untracked) and
was not touched — no worktree, checkout, stash or reset; re-checked after all work. The PR
head was fetched read-only into refs/autoreview/pr146; everything below ran against
git archive bf2a7762 | tar -x into a temporary directory outside the repository, with
click/pyyaml/ninja resolved from pyproject.toml into a throwaway CPython 3.12
environment. CLI probes ran against throwaway projects under /tmp. Nothing was committed or
pushed.
| Check | Result |
|---|---|
pytest 9.1.1 tests/ -q |
PASS — 696 passed, 1 skipped in 5.28s. Exactly the figure the body claims (690 + 6) |
ruff 0.16.5 check . |
PASS — All checks passed! |
mypy . --ignore-missing-imports --no-strict-optional --exclude '^(layers|core|promo)/' — CI's exact invocation from ci.yml:68 |
PASS — "Success: no issues found in 108 source files". The previous review recorded mypy as NOT RUN; it is run here |
Negative control — lockfile.py, resolver.py, commands.py restored to 90010fd, the six new tests re-run |
6 failed, 0 passed. test_the_lock_notices_a_recipe_whose_build_system_changed, test_every_field_the_lock_records_is_one_the_resolver_compares, test_load_refuses_a_lock_that_is_not_yaml, test_load_refuses_a_lock_it_cannot_open, test_install_packages_reports_a_corrupt_lock_as_a_resolve_error, test_build_command_prints_a_corrupt_lock_without_a_traceback. The author's claimed control, reproduced |
Real CLI — lock records build: cmake, recipe says build: make, same version/URL/checksum |
exit 1, [error] Package error: ebuild.lock pins 'zlib' v1.3.0 with build 'cmake', but the recipe for that version now has 'make'. The lock exists to notice this; … — finding 2 confirmed against the binary |
Real CLI, control — lock corrected to build: make |
resolves: [info] Packages to install: zlib v1.3.0, proceeds to fetch (then fails on example.invalid, as designed) |
Real CLI — ebuild.lock with an unterminated quote |
exit 1, [error] … ebuild.lock is not valid YAML: …, no traceback — finding 1's headline case confirmed fixed |
Lockfile.load() — lock path is a directory |
LockfileError: … could not be read: [Errno 21] Is a directory … |
Lockfile.load() — lock with a NUL byte |
LockfileError: … is not valid YAML: unacceptable character … |
Lockfile.load() — lock with a 0xff byte, and a UTF-16-encoded lock |
UNCAUGHT UnicodeDecodeError in both — finding 4 |
Real CLI — ebuild build on a project with a 0xff byte in ebuild.lock |
exit 1 with a 44-line UnicodeDecodeError traceback printed to the terminal — finding 4 confirmed end to end |
Lockfile.load() call sites |
exactly one, commands.py:90, and it is the one wrapped |
ResolveError handlers reached from it |
commands.py:1078, :1271, :1374 — all three catch it |
git diff --stat 90010fd bf2a776 |
7 files, +196/−10; CHANGELOG.md, docs/dependency-management.md, commands.py, lockfile.py, resolver.py, and the two test files |
grep -rln "ebuild.lock" docs/ README.md |
docs/dependency-management.md only; reachable from README.md:91 |
CI on this head (checks.txt) |
30 pass, 0 fail, incl. policy / Policy / Linked Issue, CodeQL, Analyze (Python), 9 Test (3.10/3.11/3.12 × ubuntu/macos/windows) legs, 11 EoSim targets, CI Gate, Simulation Sanity Gate |
| PR metadata | mergeable: MERGEABLE, mergeStateStatus: BLOCKED, reviewDecision: REVIEW_REQUIRED |
Not checked
- yamllint — NOT RUN. Not installed. The body's yamllint claim is unverified locally; this
commit changes no YAML, so nothing in it is exposed by that gap. - The line-ending claim was not verified. The author reports
ebuild/cli/commands.py
2935 → 2940 CR bytes, one per added line. I did not count CR bytes;git archivenormalises
under.gitattributes, so the export is the wrong artifact to measure it from. - No package was actually fetched or built. Every end-to-end probe used
https://example.invalidand stopped at resolution or at the download error. The path from a
locked resolution through fetch → checksum → build is unchanged by this diff and remains
unverified by me. pyteston Python 3.10 and 3.11 — NOT RUN. 3.12 only; CI covers the others and is green.- Concurrency — still NOT EXAMINED, as in the previous review.
lockfile.save()is a plain
truncating write with no locking and no error handling of its own: a read-only project
directory would traceback out ofsave()the wayload()used to. That is pre-existing, not
introduced here, and I did not probe it. - Codecov reports 96.92% patch coverage, 2 lines missing (
commands.py1 partial,
plugins/__init__.py1 — the latter is #132's file). I did not chase which branch in
commands.pyis uncovered; on inspection the only new branch there is theexcept LockfileErrorarm, which two of the new tests exercise. - The #132 half of the 19-file diff is not re-reviewed here — reviewed separately at head
f4788f2c. If #132 changes, the 696-passed figure no longer describes what will merge. mergeStateStatus: BLOCKED—REVIEW_REQUIREDplus the #132 stack. Land #132 first, as
the body says.
Automated architecture review of bf2a77628d47 — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.
|
Both taken, in Finding 4 — non-UTF-8 lock. Finding 5 — the overwritten
|
CI -- ebuild has been red on master since the 09-08 batch merge, and the first failing step (ruff) has hidden the ones behind it. Lint (ruff, all nine Test legs): - test_build_dir_resolution.py imported shutil twice (F811). - test_package_recipe.py lost its trailing newline (W292). - test_ci_gate.py had `import itertools` / `import re` two hundred lines down (E402) -- my own embeddedos-org#103, replayed onto a file that had moved. These three hunks are byte-identical to embeddedos-org#122's, so either PR merging first leaves the other clean. Type check and tests (never reached on master since 09-08): - ebuild/packages/index_sync.py calls PackageRecipe.to_dict(), which embeddedos-org#111 defined and embeddedos-org#112 -- merged five minutes later from a base without it -- deleted in its replay. mypy names it once; pytest fails nine test_index_sync cases with AttributeError. The method is restored verbatim from embeddedos-org#111 (cc90078): it emits the `package:`/`build:` keys parse_recipe() reads back, which an asdict() replacement would not. Vendored core drift: - embeddedos-org#109 (dba3d83) edited core/eos/docs/three-way-alignment.md, a vendored copy pinned to eos 5544c98, so drift went 44 -> 45 and the guard failed as designed. Reverted to the pinned content (blob 7f9c8c1, the same bytes as eos:docs/three-way-alignment.md at the pin). The alignment note belongs in ebuild's own docs or upstream in eos, not in the snapshot. OSSF Scorecard: - ossf/[email protected] pulls gcr.io/openssf/scorecard-action, and gcr.io now refuses the pull ("requires billing to be enabled"). v2.4.3 pulls from ghcr.io; eos already pins it and its Scorecard job is green. Not in this PR: EoSim Sanity's Windows/macOS legs install a wheel that has never been published; embeddedos-org#121 (srpatcha) already replaces that with the clone the other legs use. Verified locally: ruff clean, yamllint clean, mypy clean over 107 files, 680 passed / 1 skipped, scripts/check_vendor_drift.py 44/44 and 46/46.
…/3.11, yamllint on Windows Both surfaced on this branch's first CI run, once ruff let the job get past its first step. - ebuild/plugins/__init__.py: on Python 3.10 and 3.11 the stubs type entry_points() as the deprecated mapping, and its .get() wants an EntryPoints default, so mypy fails with arg-type. The line carried a '# type: ignore[attr-defined]' -- the wrong error code, so it suppressed nothing. Spelled out with a cast, byte-identical to embeddedos-org#122's hunk (54605f0). - .yamllint.yml: the Windows runners check out with core.autocrlf=true, so every YAML file arrives as CRLF and the default new-lines: unix rule rejected every line. The step was added on 09-03 and had never passed on that leg. new-lines: platform accepts the checkout's own convention.
…e checkout's line ending new-lines: platform was the wrong fix. The Windows runners' autocrlf turns LF files into CRLF -- except a file that already carries a stray CR, which git leaves alone, and auto-assign.yml had one on its last line. So under 'platform' Windows expected CRLF and got LF on that file's first line, and the leg was red again for the opposite reason. Pin *.yml and *.yaml to eol=lf so every OS lints the same bytes, keep yamllint's default unix rule, and drop the stray CR.
…h a round-trip test PackageRecipe.to_dict() was written before install_args existed and was never taught about it, so a recipe that went through index_sync came back from the cache with install_args empty while every other field survived. It also returned the recipe's own list objects, so a caller that appended to what it got back edited the recipe behind its back. install_args is now emitted after build_args, matching the order parse_recipe() reads them, and every list field is copied on the way out. index_sync's entry-to-recipe mapping carries install_args too; without that the field could not arrive from an index at all. The new round-trip test builds a recipe with every field set and asserts parse_recipe(safe_load(safe_dump(to_dict()))) equals it; against the previous to_dict() it fails on install_args. A second test checks the lists are copies, and test_index_sync gains a case that an index entry's install_args reaches the cached YAML.
actions/checkout, ossf/scorecard-action and codeql-action/upload-sarif were referenced by moving tags. A tag can be re-pointed; a commit cannot, and Scorecard itself flags unpinned actions. Each is now pinned to the commit its tag resolved to on 2026-09-14, with the tag kept in a trailing comment, the same shape linked-issue.yml already uses.
The changelog records what the master repair changed and why. CONTRIBUTING gains a note for Windows contributors: .gitattributes now pins YAML to LF, but the attribute governs future checkouts and commits, not files already in a working tree, so an existing clone needs one git add --renormalize . (or a fresh clone) before yamllint stops seeing CRLF.
…working tree git add --renormalize . re-applies the clean filter to the index; it never rewrites files, so a clone with CRLF YAML still has CRLF YAML afterwards (reproduced in a scratch clone with core.autocrlf=true: two CRs before, two after, status clean). What re-checks the files out is git rm --cached -r . && git reset --hard HEAD, or a fresh clone. CONTRIBUTING and the changelog now say that. The changelog entry also lists every file it describes and the plugins/__init__.py type-check fix it had left out, and the to_dict() comment no longer claims to follow parse_recipe()'s order.
The review at 43b0337 found the one-line assertion pinning install_args next to build_args and, six lines above it, recipe.py's docstring saying key order does not matter. Both are true: order is not a correctness property, because parse_recipe() reads every key by name, and it is a stability property, because index_sync writes the dict as cached YAML that humans diff. The assertion now says which of the two it is guarding and what to do when a reordering is deliberate, so a future failure reads as intentional rather than as a mystery. No behaviour change. ruff clean; pytest 683 passed, 1 skipped.
The lockfile was written after every resolution and never read: nothing called Lockfile.load(), and _install_packages built the Lockfile only after resolve() had already chosen versions. An unpinned package therefore resolved to the newest recipe on every machine and every run, while an ebuild.lock sat in the project claiming to pin it. The architecture doc's "records exact resolved versions for reproducibility" described a file that had no effect on resolution. PackageResolver.resolve() takes an optional Lockfile now. Precedence is the request, then the lock, then the newest version in the registry: an explicit request is the statement of intent and outranks the record, and the caller rewrites the record from the result as before. A locked version no recipe provides is an error that names ebuild.lock and the three ways out, not a silent fallback to something else -- silently resolving to something else is exactly what the lock exists to prevent. A locked entry that recorded a URL or checksum is compared with the recipe it resolves to, and the same version behind different bytes is refused: the version reproduces the name; the lock is there to reproduce the bytes. _install_packages loads the lock before resolving and passes it in. Lockfile.load() keeps only well-formed entries so a hand-edited file cannot become a KeyError inside the resolver. Tests: the lock pins an open package to 1.2.13 where the newest is 1.3.0; an explicit request outranks it; a locked version no recipe provides is a ResolveError; a changed checksum is refused and the real one resolves; a locked package nothing reaches is ignored; malformed entries are dropped; and _install_packages hands the resolver the lock that was on disk and rewrites it afterwards. Against the previous resolver five of the six resolver tests fail (the sixth exercises Lockfile.load() alone); against the previous _install_packages the CLI test fails.
… document it Three follow-ups to reading ebuild.lock back. A lock that is not YAML crashed every configure, build and package with a yaml.scanner.ScannerError traceback, because load() is now on the path and had no handler; a directory or an unreadable file did the same one frame earlier. Lockfile.load() raises LockfileError for both, and the one call site in the CLI converts it to ResolveError, which every caller already catches and prints as "[error] ...". A lock of the wrong shape is still sanitised entry by entry, as before. lock() recorded four fields and the resolver compared two of them, so a recipe that kept its URL and checksum and changed build: cmake to make passed the lock unchallenged -- a field recorded and never read, which is the defect the lock itself was. Lockfile.CHECKED_FIELDS is now the one list both sides use (field name -> PackageRecipe attribute, since the attribute is build_system), and a test asserts that everything lock() records other than the version key is in it. docs/dependency-management.md gains an ebuild.lock section: what it records, the request -> lock -> newest precedence, that drift in a locked field is refused, that it should be committed, and how to upgrade. Six new tests, all failing against 90010fd: the unterminated-quote lock through Lockfile.load(), through _install_packages(), and end to end through `ebuild build` (exit 1, the message, no traceback); a directory in place of the lock; the build-system drift; and the record/compare list invariant.
…ne that is not YAML The corrupt-lock handler caught yaml.YAMLError and OSError. A lock holding a byte that is not UTF-8 -- a foreign editor, a UTF-16 save, a bad binary merge -- fails in the codec before the parser sees it, and UnicodeDecodeError is a ValueError, so it escaped LockfileError, escaped the ResolveError handlers, and reached the terminal as the same 44-line traceback the handler was added to stop. Caught now, with the same message and remedy. A 0xff byte and a UTF-16 BOM are both tests; against bf2a776 the first raises the UnicodeDecodeError. tests/unit/test_resolver.py: the registry binding that was assigned only for the recipe file its call writes, then overwritten, is a bare call with a comment saying what it is for.
c26d9f9 to
ba4e248
Compare
|
Status at the current head — my last comment ended "Still on It was written against Re-verified at
Same counts as the pre-restack run, which is the expected outcome for a replay. The single skip is One thing worth stating plainly, because it cost me a wrong reading here first: three tests that shell out to a compiler ( The findings from the last round are unchanged and still covered: |
Stacked on #132 (master is red without it). Review
4054494..ba4e248for this change alone: three commits (restacked once, onto #132's rebase4054494aftermastermoved toc37e995;f4788f2..c26d9f9was the same content on the previous base --51d4642/3a941c6/ba4e248are90010fd/bf2a776/c26d9f9replayed, no conflicts, none of #122's files are touched here).90010fdis the fix (6 files);bf2a776answers the first review's three findings;c26d9f9answers the second review's findings 4 and 5 (ebuild/packages/lockfile.py,tests/unit/test_resolver.py).bf2a776covered (ebuild/packages/lockfile.py,ebuild/packages/resolver.py,ebuild/cli/commands.py,docs/dependency-management.md, the two test files,CHANGELOG.md).Problem (ebuild #145)
ebuild.lockwas written after every package resolution and never read. Nothing calledLockfile.load(), and_install_packagesconstructed theLockfileonly afterPackageResolver.resolve()had already chosen versions — the code, in order:So an unpinned package resolved to the newest recipe on every machine and every run, while an
ebuild.locksat in the project claiming to pin it.docs/architecture.md("records exact resolved versions for reproducibility") and the CHANGELOG's "guaranteeing reproducible builds (§9.2)" described a file that had no effect on resolution.Change
PackageResolver.resolve(requested, lockfile=None); the CLI loads the lock before resolving and passes it in, and rewrites it from the result as before.Precedence, in order:
Two refusals, because silently resolving to something else is exactly what the lock exists to prevent:
ResolveError: ebuild.lock pins 'zlib' at v9.9.9, and no recipe provides that version. Restore the recipe, or request the version you want explicitly, or delete ebuild.lock to resolve afresh.Lockfile.load()keeps only well-formed entries, so a hand-edited file cannot become aKeyErrorinside the resolver.Tests (Verified; five of the six resolver tests fail against the previous resolver, and the CLI test fails against the previous
_install_packages)test_lockfile_pins_a_package_the_request_leaves_open— zlib at 1.2.13 and 1.3.0; without a lock → 1.3.0, with the lock → 1.2.13test_an_explicit_request_outranks_the_locktest_a_locked_version_no_recipe_provides_is_an_error_not_a_fallbacktest_the_lock_notices_a_recipe_whose_bytes_changed— and the control with the real checksum resolvestest_a_lock_entry_the_request_does_not_reach_is_ignoredtest_load_drops_malformed_entriestests/unit/test_lockfile_cli.py—_install_packageshands the resolver the lock that was on disk and rewrites it afterwardsFull suite on this head: ruff, yamllint, mypy (
mypy . --ignore-missing-imports --no-strict-optional --exclude '^(layers|core|promo)/') clean;pytest696 passed / 1 skipped.Review follow-up (
bf2a776)yaml.scanner.ScannerErrortraceback, and this PR made that reachableLockfile.load()raisesLockfileErrorfor a lock that is not YAML (ebuild.lock is not valid YAML: ... Fix it, or delete it to resolve afresh.) and for one it cannot open (OSError: a directory, a permission)._install_packagesconverts it toResolveErrorat the one load site, so all three callers print[error] ...as they already do for the two refusals. Tests: throughLockfile.load()(unterminated quote; a directory in place of the file), through_install_packages(), and end to end throughebuild buildwith the real project fixture shape -- exit 1, the message, noTraceback.buildwas recorded and never comparedLockfile.CHECKED_FIELDS = (("url","url"), ("checksum","checksum"), ("build","build_system"))is the one list bothlock()'s readers use;_check_locked_bytes()iterates it (field name → attribute, since the attribute isbuild_system).test_the_lock_notices_a_recipe_whose_build_system_changed(with the control), andtest_every_field_the_lock_records_is_one_the_resolver_compares, which assertsset(lock entry) - {"version"} == set(CHECKED_FIELDS)so a fifth recorded field cannot go uncompared again.docs/dependency-management.mdgains anebuild.locksection: what it records, request → lock → newest precedence, that drift in a locked field is refused, commit it, how to upgrade (name the version, or delete the file), and how a corrupt vs. wrong-shape lock is treated.Negative control for the follow-up: with
90010fd'slockfile.py,resolver.pyandcommands.pyrestored, all six new tests fail (the CLI one prints the ScannerError traceback).Second review follow-up (
c26d9f9)UnicodeDecodeErrortracebackexcept (yaml.YAMLError, UnicodeDecodeError)with the same message and remedy.test_load_refuses_a_lock_that_is_not_utf8: a0xffbyte and a UTF-16 BOM both raiseLockfileError; againstbf2a776the first raises theUnicodeDecodeError.pytest697 passed / 1 skipped onc26d9f9; ruff, yamllint, mypy clean. Onba4e248(the restack): ruff, yamllint, mypy (CI's invocation) clean;pytest694 passed / 1 skipped, with 3 not run here --test_end_to_end_build_from_outside_produces_the_binary,test_editing_a_header_triggers_a_rebuild,test_measures_a_real_binarylink a real C binary and this host's linker is currently broken (ld: tapi error: malformed file … MacOSX27.0.sdk … unknown architecture, identical at the base4054494); CI covers them.Closing issue
Fixes #145