Skip to content

fix(extraction): the background writeback never changes reviewed lines - #625

Open
privatejoel wants to merge 1 commit into
mainfrom
fix/extraction-writeback-guard
Open

privatejoel wants to merge 1 commit into
mainfrom
fix/extraction-writeback-guard

Conversation

@privatejoel

Copy link
Copy Markdown
Collaborator

Summary

Plan C-2 (docs/SO_EXECUTION_COHESION.md, R-F1 (b)), with decision C-D1 accepted.

The defect: the background extraction writeback did not read the order status. On a large PO, a person can review, approve and push the page-1 lines before the job ends. The job then replaced every line, with the operator edits and _mapped_item. Nothing recorded it. Also, a cancel that landed during the merge was overwritten.

What the writeback does now (src/api/cron/extraction_jobs.js):

Order status Lines Status Approval Record
DRAFT written, mapped with mapLinesToItemMaster kept cleared only if the content changed (as before) none
PENDING_REVIEW, APPROVED kept DRAFT, with correction_reason (C-D1) cleared, as a return to DRAFT clears it in [id].js finding + row
EXPORTED_TO_TALLY, FAILED_TALLY_IMPORT, RECONCILED, DUPLICATE, REUSED, BLOCKED kept kept kept finding + row
CANCELLED no write to the order kept kept none
  • The plan text names only PENDING_REVIEW and APPROVED. The owner accepted the reading "never changes reviewed lines" for every status past DRAFT. This PR applies that reading. A status past APPROVED keeps its status and approval.
  • If the full extraction reads the same content as the order (same payload hash), the job writes nothing.
  • The new lines are held on the finding lines_changed_after_review in orders.rule_findings. The finding has blocks: true and source: "extraction", so:
    • the approve PATCH, the ERP push and the Tally push refuse the order until an approver resolves it (existing gate, blocking-findings.js);
    • a routine rule_findings overwrite carries it forward (mergeBlockersForward).
  • The exceptions list shows the row lines_changed_after_review with blocks: "approval".
  • An audit row order_lines_held_after_review records each hold.
  • The SO workspace has a control on that row: "Load the new lines into the line grid". It puts the held lines in the grid as unsaved edits. The operator checks them and selects Save line edits. An approver then resolves the finding on the Why tab. Nothing is saved by the control itself.

Races:

  • The order write is compare-and-set on updated_at (the 001_init trigger moves it on every write). If 0 rows change, the job reads the order again and applies the rule once more. If the order changes again, the job fails with last_error "order writeback failed: the order changed twice while the job wrote it". job.result keeps the merged extraction. A write error fails the job as before.
  • Each job stage (profile, chunk, chunk done, extract to merge) is compare-and-set on the status the tick read. Before this, a cancel that landed while a chunk ran was set back to extracting or merging, and the job then wrote the order.
  • The cancel endpoint (orders/extraction_jobs_id.js) refuses a job in merging with 409 JOB_WRITING, because the job is writing the order. Its update is compare-and-set on the status it read: if the worker moved the job on, it returns 409 JOB_MOVED.

In-flight job: GET /api/orders/exceptions reads the order's newest queued or running extraction_jobs row (tenant-scoped). The builder raises extraction_in_flight with blocks: "send". The next step "Send for approval" is then disabled.

docs/AGENT_READY_SO_FLOW.md lists the two new types and the options use_new_lines and cancel_extraction.

Deploy notes

  • No migration. The held lines live in orders.rule_findings (jsonb, exists since 001).
  • No new requireAction. No new permission.
  • No setting to turn on.

Tests

New, behavioural:

  • src/v3-app/api-extraction-writeback-guard.test.js (15 tests): DRAFT writes and maps; PENDING_REVIEW goes to DRAFT with the reason and holds the lines; approve then gets 409 ORDER_HAS_UNRESOLVED_BLOCKER; the same lines write nothing; APPROVED keeps its lines and loses its approval; EXPORTED_TO_TALLY keeps lines, status and approval; a second late job replaces the open held lines; CANCELLED gets no write; an order cancel between the read and the write is not overwritten; a send for review between the read and the write gets a hold; an order that changes on both tries fails the job and keeps job.result; a job cancel during a chunk is not overwritten; the cancel endpoint refuses a merging job and a job the worker moved on.
  • src/v3-app/api-order-exceptions.in-flight.test.js (6 tests): the row for each in-flight status, none for a terminal job, Send for approval stays disabled, and the real endpoint reads only this tenant's job.

Changed:

  • api-order-reconcile.hash.test.js: three C-1 cases used a PENDING_REVIEW order to show that the writeback moves the hash. That writeback no longer happens, so those cases use a DRAFT. The approved-order case now checks the C-2 rule: the lines and the hash stay, and the approval goes.
  • api-extraction-jobs-continuation.test.js: the double now reads the order as a DRAFT and returns the written row.

Mutation check. I undid each fix, ran the two new files, and restored it. A no-op control passed. All 18 mutations failed at least one test:

# Mutation Killed by
M1 write lines at every status in review, same lines, approve refused
M2 skip mapLinesToItemMaster DRAFT maps, held lines mapped
M3 no return to DRAFT in review, APPROVED, send-for-review race
M4 approval not cleared APPROVED
M5 write to a CANCELLED order CANCELLED, order-cancel race
M6 no updated_at compare-and-set the three race tests
M7 treat 0 rows as written send-for-review race, changed twice
M8 mark completed after two 0-row writes changed twice
M9 append a second open finding second late job
M10 finding not a blocker in review, approve refused, EXPORTED
M11 no stage compare-and-set (extract to merge) job cancel during a chunk
M12 allow cancel while merging cancel endpoint, merging
M13 no compare-and-set in the cancel endpoint cancel endpoint, worker moved on
M14 no in-flight row builder, next step, endpoint
M15 in-flight row does not block send builder, next step, endpoint
M16 job read not tenant-scoped endpoint, other tenant
M17 endpoint does not pass the job endpoint
M18 no lines_changed_after_review row type in review, APPROVED, EXPORTED

Gates, on a clean npm ci in this worktree:

  • npm run check: pass.
  • npm run lint: 0 errors (511 warnings, none on a changed line).
  • npm run build: pass. Build outputs are not committed.
  • npm run verify: pass. It rewrote docs/RBAC_AUDIT.md and docs/WRITE_PATH_AUDIT.md with changes this PR did not cause (admin screen, roles). I restored both.
  • npm run eval:golden: pass, aggregate 1.000.
  • NODE_OPTIONS=--no-experimental-webstorage npx vitest run --maxWorkers=2: 628 files, 7388 tests, all pass.

Note: with the shared node_modules symlink (installed 2026-07-07), check:imports and build fail on origin/main too. The symlink has @simplewebauthn/server 11.0.0 (the code needs 14) and no @fontsource packages. A clean install fixes both.

Not covered

  • The workspace control has no UI test. The server side is tested.
  • The other three job stage compare-and-set lines (profile, chunk, chunk done) are the same line as the tested extract-to-merge one. No test reaches them, because each needs the source bytes and a model call.
  • When a job holds lines, the workspace toast still says "Line items landed" (BackgroundExtractionWatch reads the job, not the order). The banner and the exceptions row tell the truth after the reload.
  • The local exceptions build in the workspace has no job, so it shows no extraction_in_flight row. The server list, which the workspace uses when it answers, has it.
  • A job stuck in merging cannot be cancelled from the API. The worker retries it when its lease ends. This is marked shortcut: in the cancel endpoint.
  • After the operator saves the held lines, an approver still has to resolve the finding. Saving the lines does not resolve it.
  • BLOCKED keeps its status. The plan does not name it.
  • The held lines can make rule_findings large on a long PO.

Open questions for the owner:

  1. If the engineer saves the held lines unchanged, should that resolve the finding without an approver? Today an approver resolves it.
  2. Should a BLOCKED order go back to DRAFT, as PENDING_REVIEW does? Today it keeps BLOCKED.

🤖 Generated with Claude Code

Plan C-2 (R-F1 b), with decision C-D1.

The background job wrote its lines without a read of the order status.
On a large PO, a person can review, approve and push the page-1 lines
before the job ends. The job then replaced every line, with the
operator edits and item mappings. Nothing recorded it.

Now the job writes lines only to a DRAFT, and maps them to the item
master. Past DRAFT, it keeps the reviewed lines. It holds the new lines
on a blocking finding, lines_changed_after_review. An order in review
or approved goes back to DRAFT with the reason, and loses its approval.
A later status keeps its status and approval. A cancelled order gets
no write.

The order write is compare-and-set on updated_at. A save or a cancel
that lands during the job is not overwritten. The job reads the order
again once. If the order changes again, the job fails and keeps its
result. Each job stage is compare-and-set on its status, so an operator
cancel stays. The cancel endpoint refuses a job that is merging.

A queued or running job is the row extraction_in_flight. It blocks
send.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@vercel

vercel Bot commented Oct 19, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
anvil Ready Ready Preview Oct 19, 2026 4:02pm UTC

This branch was successfully deployed

1 active deployment
Preview — 5a15667e Deployed Oct 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant