Repository navigation
test(sqlite): disk-full injection fail-closed and recovery matrix - #9126
Conversation
…y store Batch-3 slice 4 of issue loopx-project#4224: storage-exhaustion matrix binding to the documented injected-limit vs real-ENOSPC distinction in docs/reference/sqlite-authority-store.md. - Freeze PRAGMA max_page_count at the occupied page count on every write connection of a real child process (genuine SQLITE_FULL path) and force exhaustion on the checkpoint-bearing commit: typed provider_transaction_failed, receipts never claim success, retries fail durably with zero durable footprint, reads keep proving the intact state, and a never-armed session restores service without repair, including fresh reuse of the failed operation id. - Contrast control: identical padded workload applies without the quota (byte-identical retry replays applied) and the logical 16 MiB cap stays a separately typed store_capacity_exhausted failure. - Real-OS case: fill a bounded macOS hdiutil sparse volume until the kernel returns ENOSPC; the failing commit is typed (or ambiguous for receipt reconciliation) and never fakes success, reads fail closed, and after the volume grows the store resumes on exactly the applied prefix. Oracles are independent of the store's proof code: engine integrity_check, exact raw row snapshots, frozen page count and canonical ledger bytes. Beyond loopx-project#4328's single-commit capacity-full case this pins the typed classification, checkpoint-transaction exhaustion, durable retries, reads under constraint, restoration and the real-ENOSPC path. Signed-off-by: Lihua <[email protected]>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=low
未发现阻塞性问题。评审 head:25af666b671c5a80395a850938d749924bef6055。
动机 / Motivation
维护 SQLite authority store 的开发者在修改事务与恢复代码时,需要发现磁盘写满后的假成功、部分提交或重试丢失。此前仅有小数据库容量注入覆盖;本次在第 65 次 checkpoint 写入失败后核对整条已提交记录,并验证释放容量后继续提交。checkpoint 是用于恢复的状态快照;它与新 head、事件、回执共同提交,任何一项单独前进都会破坏后续恢复。
新增测试已在真实 SQLite 上证明失败不发布新状态,恢复后可继续提交并正确重放。不修改产品运行行为,也不宣称断电、大规模历史或非 macOS 真实磁盘写满已获验证。这里完成的是容量失败回归保护这一独立测试切片,并未把通过几个小测试等同于整个存储项目完成。
改动思路 / Approach
原子提交规则仍由现有 SqliteAuthorityStore 持有;故障注入与卷操作只在测试进程中,没有新增产品策略或状态所有者。本 PR 交付可重复的容量失败与恢复回归保护,不扩展运行时能力或宣称全部耐久性资格。
测试使用两个相互补充的故障来源:子进程在 BEGIN IMMEDIATE 前冻结 SQLite 页数上限,得到稳定、可重复的引擎 SQLITE_FULL;macOS 专用用例则填满一个可丢弃的小卷,观察真实操作系统 ENOSPC,再扩容、重新挂载和恢复。两者都经过产品事务入口,没有用 mock 返回预期的成功或失败。
我比较了已有 sqlite_authority_process.ts 的 max_page_count 注入。它可以继续保留,但不能替代本次跨 checkpoint、完整 ledger、重复 operation id 和扩容后的恢复组合。把 hook 放进产品代码会增加不必要的运行时入口;仅保留真实卷测试又会失去跨平台的确定性覆盖。父进程保持独立的期望 ledger,并直接查询 SQLite,避免子进程自产预期结果。
具体改动 / Changes
完整差异为三个文件,新增 733 行、删除 0 行:731 行测试与子进程辅助器,加两条 tsconfig 测试编译入口;没有产品代码、持久化格式或用户配置变化。
规格依据是 docs/reference/sqlite-authority-store.md,固定版本 cc5392e9538e81487e9e551b915dabc68c085c25。以下为便于引用而给规格段落分配的评审标签,并非文档原有编号:
- SQLITE-ATOMIC:head、receipt、events、outbox 同一事务提交。本次检查失败前后数据库 head、checkpoint 和计数、完整扫描及独立 ledger,一致性覆盖已实现。
- SQLITE-ROLLBACK:COMMIT 前失败回滚;COMMIT 出错必须保留歧义并通过 receipt 对账。本次把确定性配额失败与真实 ENOSPC 分开,后者允许文档规定的歧义,然后按已提交前缀恢复,覆盖已实现。
- SQLITE-BOUNDARY:小型临时数据库测试不构成断电或大规模恢复资格。这些资格仍在本切片范围之外;实际 macOS 卷测试只增加该平台的容量耗尽证据。
关键代码讲解
tests/control_plane_ts/sqlite_authority_disk_full.test.ts:263的注入测试先写入 64 条记录,确认已有一个 checkpoint,然后使第 65 次携带 1 MiB 数据的写入需要增长数据库。相同 id 重试与不同 id 的失败都必须返回failed/provider_transaction_failed,不能携带新 cursor 或 provider revision。接着读取 head、失败操作的缺失回执及 SQL 状态,释放约束后提交并验证重放,没有停在“成功产生错误”这一步。tests/control_plane_ts/sqlite_authority_disk_full_process.ts:47的DatabaseSync.prototype.exec包装仅存在于单独启动的测试子进程。armed 后才在准确的 BEGIN IMMEDIATE 位置设置 max_page_count,其余 SQL 继续执行原方法;因此真正受测的是现有存储入口的事务与回滚逻辑。若将来入口改写导致 hook 失效,主测试会因写入意外成功而失败。tests/control_plane_ts/sqlite_authority_disk_full.test.ts:407的对照测试在没有页数约束时完成同样的 padded checkpoint,并单独检查 16 MiB 逻辑请求上限。它排除了“请求本来就无效,所以看起来像容量失败”的伪阳性。tests/control_plane_ts/sqlite_authority_disk_full.test.ts:451的真实卷测试使用 24 MiB 临时 sparse volume,遇到失败或歧义后检查可读已提交前缀或明确不可用状态,再扩至 64 MiB,对账并继续写入。清理与卷生命周期属于测试,不转移给产品用户。
对主干的风险 / Risks
PASS:精确 head 的三个用例全部通过,0 失败、0 跳过;本机实际执行了 hdiutil ENOSPC 与扩容恢复用例。PASS:TypeScript noEmit 编译。PASS:仓库 premerge 的 4 项直接检查和 7 项 canary,无 hold。远端 CI 按本次评审策略未查询,不能把这些本地结果解读为远端 CI 通过。
PASS:将同一份 head 测试与辅助器复制到临时 harness,只更换指向所选 checkout 的 import,在不可变基线 cc5392e 与当前 head 上分别运行,两边均为 3 通过、0 失败、0 跳过。比较保留 ledger、receipt、cursor、checkpoint 和错误形态,忽略临时路径、随机数据库 incarnation 与耗时。
预期 FAIL:仅禁用临时辅助器中 armed 的配额 hook,再运行注入用例,测试在第 307 行捕获 applied 与预期 failed 不符并以 1 退出;原始 head 则通过。这说明断言确实能发现注入失效,而非只证明脚本能跑完。
主要剩余风险是测试本身的维护成本和平台差异。真实卷分支限 macOS;其他平台仍依赖确定性 SQLite 页数约束,不能称为真实 OS ENOSPC 验证。断电与大规模历史恢复也未测试。该 PR 是对已存在产品入口的 coverage-only 保护,没有新增生产 gate、默认关闭能力、agent 指令或 actor authority;测试内的 schema 和 command union 不成为共享持久化协议。
我的整体评价 / Verdict
APPROVE。对长程运行的影响判为 preserved:现有恢复与重放语义不变,新增回归保护能发现会污染后续工作的假成功与部分提交。对用户体验判为 preserved:普通 CLI、frontend 和 Lark 操作没有新增配置、导航或确认步骤。完整三文件 diff 仅包含测试、子进程辅助器和 tsconfig include,没有渲染界面变化。视觉未验证范围:none for rendered presentation;非 macOS 真实 OS ENOSPC 仍未验证。
733 行对于一个测试切片不算少,但独立 ledger、无配额对照、逻辑上限区别和真实卷恢复分别排除不同的错误解释,且没有增加产品热路径或迁移成本。当前不建议为压缩测试行数而创建共享故障注入框架;已有 store owner 足够,测试内局部协议也无需扩展公共 vocabulary。基线与 head 的真实 SQLite 对照、恢复后的实际进展及故意破坏断言提供了批准依据;批准不代表其他耐久性资格、合并准备状态或未查询的远端 CI 已通过。
English verdict: APPROVE — head 25af666; no blocking findings. Exact-head and paired baseline/head SQLite tests passed, including macOS ENOSPC recovery; disabling the quota hook failed as expected. Power-loss and large-history qualification remain outside scope.
|
Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · xhigh 合并后补审 [P2] 真实 ENOSPC 用例会接受普通 SQLite 错误填盘循环遇到第一个通用事务错误就退出,只核对 provider 状态和已写入前缀。用同一临时 APFS 卷,在第 2 笔事务触发普通的 SQLite 缺表错误,SQLite 返回错误码 1。当时卷仍有约 20 MiB 可用,独立 2 MiB 写入和同步成功,整个 ENOSPC 用例仍通过。建议在触发容量路径处要求独立错误来源证据,并加一个普通 SQLite 错误必须被拒绝的对照。填盘结束条件 [P2] 按回执核对可读的歧义提交,再判断账本前缀用例允许 COMMIT 返回 ambiguous,却在扩容前把可读快照的 cursor 和历史笔数固定为原先已确认前缀。实际提交第 2 笔后,在测试进程模拟 COMMIT 确认丢失,provider 返回 commit_outcome_unknown,读回 cursor 2 且回执存在。用例先在 cursor 1 的断言失败,未到扩容后的对账分支。建议在任何可读快照上用原操作回执解释歧义,明确失败则保持原前缀。读回断言 动机 / Motivation维护 SQLite authority store 的工程师,需要在改变提交和恢复逻辑时发现磁盘耗尽后的假成功、残留状态或重复回放。 旧的容量用例在普通一笔提交上限制 SQLite 页数。新增矩阵在带 checkpoint 的提交上检查整个账本,另外在临时 macOS 卷上演练容量耗尽后的恢复。 确定性的页数上限测试能验证第 65 笔失败后滚回,恢复容量后接续写入,且与 16 MiB 业务请求限制区分。 本次补审限定在新增测试、测试进程和 TypeScript 配置。断电、跨平台真实磁盘、大规模历史和 D2 完整验收仍在本切片之外。 真实卷用例把任何通用事务拒绝都当成磁盘写满,也会在回执尚未对账前拒绝已经持久化的合法歧义提交。 改动思路 / Approach原 SqliteAuthorityStore 继续拥有真实事务和回执。页限、临时卷和账本断言只用于测试;复用现有提供方可以隔离被测错误来源。 确定性子进程页限、相同负载无遮拦对照、真实 macOS 临时卷与恢复、同一测试编译配置。 复用已有子进程隔离,同时让物理卷的错误来源和已确认的提交结果分别有独立证据。 具体改动 / Changes精确提交 25af666。以 2000c39 的 SQLite 存储合约检查。原有合约
关键代码讲解测试子进程在 BEGIN IMMEDIATE 前注入 max_page_count。第 65 笔写入需扩页,账本、历史扫描和回执提供独立 oracle;无页限时相同写入应成功。macOS 用例创建并销毁临时 APFS 镜像,当前把第一条通用失败放进容量分支,检查读出前缀后扩容并写入。tsconfig 只注册这两个测试文件,未改产品状态、CLI 或写入口。 对主干的风险 / Risks精确提交与主干各三个存储测试通过,TypeScript noEmit 通过。两个局部对照分别证实当前测试误放行非容量错误,以及误拒绝已经落盘的歧义。 两个受控替换都使用真实 SQLite 和临时卷。反例在精确提交及主干 85efe20 上行为一致。现有写入与恢复 owner 保留。未验证断电、非 macOS 物理容量故障、完整 D2 和远端 CI。控制组用于测试辨别能力。 我的整体评价 / Verdict无页面或交互呈现变更。实际验证使用临时 APFS 镜像、真实 SQLite 与独立本地进程。 保留确定性页限与无遮拦对照。建议补齐真实卷测试对错误来源和歧义提交的判断,再用两条反例验证。当前发现涉及测试覆盖。 English verdict: REQUEST_CHANGES — The real-volume case accepts a non-capacity SQLite error and rejects a readable, durably applied ambiguous commit. |
Part of #4224 (batch 3 slice 4, the 2026-11-20 kickoff: disk-full injection for the SQLite authority store). Test-only: no
loopx/**change.What this adds
The storage-exhaustion evidence row the kickoff named, binding every assertion to the contract already documented in
docs/reference/sqlite-authority-store.md— writes runBEGIN IMMEDIATE/synchronous=FULLwith "Before-COMMIT failures roll back; a COMMIT error reports an ambiguous outcome for receipt reconciliation", and the real-process regressions'max_page_countexhaustion is the injected-limit path ("small disposable-database tests, not power-loss, operating-system ENOSPC or large-history recovery qualification"):tests/control_plane_ts/sqlite_authority_disk_full.test.ts— three tests, one disposable store/volume each (~7 s total).tests/control_plane_ts/sqlite_authority_disk_full_process.ts— the spawned writer: once armed, every write connection freezesPRAGMA max_page_countat the occupied page count (genuineSQLITE_FULLin the engine; writes fitting already-allocated pages still pass — the same shape a full disk has). No fault hooks enter product code; the parent never patches anything.tsconfig.control-plane.json— registers both files.Coverage (kickoff points 1–4)
checkpoints+commits+headin one transaction) after a 64-commit seeded window. A real-OS case fills a bounded 24 MB macOS hdiutil sparse volume with unique ~1 MiB projections until the kernel returns ENOSPC.provider_transaction_failed(neverapplied, never ambiguous, never a protocol violation, never the logical 16 MiBstore_capacity_exhaustedcap); receipts staymissing; retries (byte-identical and distinct) keep failing while the quota stays armed; the raw engine oracle (PRAGMA integrity_check, exact row snapshot, frozen page count) proves the failed transactions allocated nothing. Reads keep proving the intact state inside the constrained process (head/receipt/audit/scan at cursor 64), and an unconstrained new process sees the byte-identical pre-exhaustion snapshot with no repair step. Under real ENOSPC the failing commit is typed (or ambiguous-for-reconciliation if the error lands on COMMIT) and never fakes success, and reads fail closed asprovider_read_unavailable— the documented real-ENOSPC vs injected-limit difference, now pinned from both sides.verifiedat 66 commits / 2 checkpoints with the full ledger byte-equal. The real-OS case grows the volume (detach → resize → attach) and resumes on exactly the applied prefix, failed id missing, fresh commit appended, integrity ok.appliedat the original cursor), and a >16 MiB commit fails with the separately typedstore_capacity_exhausted— proving the injection assertions bind to engine quota exhaustion, not to payload shape or the logical cap.Beyond #4328
#4328's
capacity-fullcase injects exhaustion on one ordinary commit of a one-commit store and assertsstatus === "failed"plus state preservation. This PR additionally pins: the typed classification, exhaustion inside a checkpoint-publishing transaction, durable zero-footprint retries under continued exhaustion, provider reads under the constraint, raw-engine oracles, post-restoration service without repair (including failed-id reuse), the un-injected contrast, and the real-OS ENOSPC path (off-darwin the volume test skips with the boundary disclosed in the skip message).Validation
Tested revision:
25af666b671c5a80395a850938d749924bef6055on top ofcc5392e9538e81487e9e551b915dabc68c085c25, Node 22.22.3 / SQLite 3.51.3 (qualified SQLite D2 runtime), macOS arm64.node --no-warnings --experimental-sqlite --experimental-strip-types --test tests/control_plane_ts/sqlite_authority_disk_full.test.ts)npm run typecheck:control-planeuv run --extra test loopx canary premerge --from-git-diff --git-diff-base cc5392e9538e81487e9e551b915dabc68c085c25: diff checks, RFC bilingual, 7/7 canaries, 0 holdsNo
loopx/**file changed; no RFC pair touched (bilingual check passed on the empty diff).