Skip to content

perf: annotate carbonah E002/E003 false positives (efficiency round 2b) - #185

Closed
tina4stack wants to merge 3 commits into
v3from
refactor/efficiency-round2b
Closed

tina4stack wants to merge 3 commits into
v3from
refactor/efficiency-round2b

Conversation

@tina4stack

Copy link
Copy Markdown
Owner

Efficiency round 2b — carbonah E002 (N+1) + E003 (unbounded)

carbonah 0.3.3 flagged 9 E002 (N+1) and 1 E003 (unbounded) in tina4_python.
I traced each against the source. Every one is a heuristic false positive or a
behaviour-locked intentional pattern — zero genuine, safely-fixable findings.
Each is annotated # carbonah:ignore <CODE> with a one-line reason, so the reported
count drops with zero runtime change: the production diff is 100% comments.

carbonah lint: before → after

Rule Before After
E002 (N+1) 9 0
E003 (unbounded) 1 0
E004 (polling loop) 12 11
C001 / D003 / E001 / E005 / H001 2 / 2 / 3 / 29 / 4 unchanged

E004 12→11 is a same-loop side effect: annotating the E002 in session's
_ensure_table retry loop also clears carbonah's coupled "polling loop" E004 for
that same bounded 5-attempt backoff — itself a false positive. No behaviour changed.

Findings — all left (documented), none needed a code fix

File:line Rule Verdict Reason
database/adapter.py:770 E002 false positive the anti-N+1: build_batch_inserts collapses N single-row INSERTs into ~1 multi-row statement — one round-trip per chunk, not per row
database/adapter.py:783 E002 intentional row-at-a-time fallback for statements that cannot collapse safely (RETURNING / upsert / Firebird / ragged)
dev_admin/init.py:1029 E002 false positive admin SQL console runs the developer's own multi-statement batch; each is distinct arbitrary SQL
migration/runner.py:275 E002 intentional one-time v2→v3 backfill; per-row UPDATE+commit isolates failures (a bad row would abort the whole PostgreSQL txn) and keeps log-and-continue
migration/runner.py:829 E002 intentional sequential apply DDL from one migration file — ordered, distinct
migration/runner.py:898 E002 intentional sequential rollback DDL from one .down.sql file
migration/runner.py:906 E002 intentional one DELETE per rolled-back migration, each inside its own per-migration transaction
orm/model.py:1363 E002 intentional one spatial-index DDL per PointField at CREATE TABLE
session/init.py:229 E002 false positive race-safe CREATE TABLE; the while is a bounded retry, executes once on success
dev_admin/init.py:1108 E003 false positive single-row lookup by primary key via fetch_one (WHERE id = ? → at most one row)

The migration backfill (275) is the one genuinely inefficient loop, but its per-row
commit is deliberate — a single batch would abort the whole PostgreSQL transaction on
one bad row and break the log-and-continue contract. Fixing it would regress behaviour
on a one-time cold path for no hot-path win, so it is left with a reason.

Characterization tests (real SQLite, no mocks — mutation-proven)

  • execute_many collapsible: build_batch_inserts returns 1 statement for 200 rows (query-count) and all 200 land with the right affected_rows/data (result).
  • RETURNING batch: not collapsed ([]), rows still written via the fallback loop.
  • Migration apply + rollback: correct end state.

carbonah measure (batch workload, 3 runs each)

Branch median SCI 0.00022239 vs baseline 0.00021779 — within noise; grade APlus both.

Verification

  • Local SQLite: characterization (4) + 117 batch / migration / ORM tests green.
  • Lab (real Postgres + MySQL + MSSQL): 81 passed, 4 skipped ([needs:firebird], no live Firebird provisioned).
  • tina4 metrics: unaffected — no file became a new offender; comments add no complexity.

Do not merge — for review.

🤖 Generated with Claude Code

andrevanzuydam and others added 3 commits September 30, 2026 14:37
Pin the behaviour carbonah's N+1 heuristic misreads, with a real result AND a
real query count on SQLite (no mocks): execute_many collapses 200 rows into one
statement (anti-N+1), the RETURNING fallback still writes every row, and the
migration apply+rollback loop reaches the right end state. Mutation-proven.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Co-Authored-By: Tina4 <[email protected]>
Signed-off-by: Andre van Zuydam <[email protected]>
carbonah 0.3.3 flagged 9 E002 (N+1) + 1 E003 (unbounded). Investigation found
0 genuine, safely-fixable findings: all are heuristic false positives or
behaviour-locked intentional patterns — the execute_many batch primitive (the
anti-N+1 itself), its row-at-a-time fallback, sequential migration DDL, a
per-migration rollback DELETE inside its own txn, a spatial-index DDL per
PointField, a race-safe CREATE TABLE retry loop, the admin SQL console running
the developer's own multi-statement batch, and a single-row PK lookup via
fetch_one. Each is annotated '# carbonah:ignore <CODE>' with a one-line reason,
so the reported count drops with zero runtime change (production diff is 100%
comments). The migration v2->v3 backfill keeps its per-row commit deliberately:
a single batch would abort the whole PostgreSQL transaction on one bad row and
break the log-and-continue contract.

carbonah lint: E002 9->0, E003 1->0. E004 12->11 as a same-loop side effect —
the session retry-loop annotation also clears carbonah's coupled 'polling loop'
finding for that same bounded 5-attempt backoff (also a false positive).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Co-Authored-By: Tina4 <[email protected]>
Signed-off-by: Andre van Zuydam <[email protected]>
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Co-Authored-By: Tina4 <[email protected]>
Signed-off-by: Andre van Zuydam <[email protected]>
@tina4stack

Copy link
Copy Markdown
Owner Author

Superseded by the Round 3 .carbonah.toml approach. This PR was inline carbonah:ignore annotations only (0 genuine perf fixes — the E002/E003 findings are the execute_many batch primitive, migration DDL, race-safe CREATE, admin console, and a PK lookup). Per the decision to centralise carbonah suppressions in .carbonah.toml, the reasoned ignores are folded into the repo's .carbonah.toml in Round 3. Closing.

@tina4stack tina4stack closed this Sep 30, 2026
@tina4stack
tina4stack deleted the refactor/efficiency-round2b branch September 30, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants