diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 55ccfd1..8748fcc 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,53 +1,274 @@ -# Knowledge consolidation — 15 open PRs (#17–#40) → one reconciled state +# Knowledge flush — 3 insight(s) -The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the -harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75 -unique pages, with the same insight landing at up to 3 different paths across -up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the -#6–#13 consolidation — this branch carries the reconciled end-state and the 15 -PRs are closed in its favor. +Queue drained: `~/.dev-loop/queue/cd5970a5-…jsonl` (2 rows), `~/.dev-loop/queue/d5a117bb-…jsonl` (1 row). +All three came out of one investigation (Hibernate's application-level not-null check +and what it does to audit columns), so they route to two pages, not three. ## Verified best-practice -Every adopted page's sources were carried from its originating PR's flush, where -they were live-verified at flush time; no new URLs were introduced during -consolidation (checked mechanically: every `http(s)` URL in every merged page -appears in a source PR's diff; every added body line in amended pages traces to -a source PR hunk — orphan-line verification). Confidence fields were kept as the -originating flushes set them, except client-side-rate-limiting where the union -of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports -`verified` for the load-bearing claims. One subagent's fabricated content (12 -files matching neither main nor any PR, with invented source URLs) was detected -by the same verification and replaced with true PR content. +### I1 — attribute `PropertyValueException: not-null property references a null or transient value` by path shape, not by the word "transient"; and exclude `@PreUpdate` from cause and fix on the UPDATE path + +**Claim.** Both throw sites share one hardcoded message literal, so the wording +carries no discriminating information; a dotted path means a composite +(`@Embeddable`) sub-attribute, a plain name means a top-level attribute. On UPDATE the +check runs before `EntityUpdateAction` exists, so no `@PreUpdate`/`@PostUpdate` +listener has run — a listener is neither a candidate cause nor a viable fix. + +**Sources checked and how.** + +- `Nullability.java` (5.6) fetched raw and read on disk: line 108-112 throws with + `persister.getPropertyNames()[i]`; line 119-123 throws with + `buildPropertyPath( persister.getPropertyNames()[i], breakProperties )`. Same literal + both times. `checkSubElementsNullability` (line 143-167) enters + `checkComponentNullability` only for `isComponentType()` (`CompositeType`) or a + collection whose **element type** is composite — so dots have exactly two producers. +- Uniqueness of the literal: `gh api search/code 'q="not-null property references a + null or transient value" repo:hibernate/hibernate-orm'` → 5 hits, of which + **1 is production code** (`engine/internal/Nullability.java`); the other 4 are tests. + So no other class emits this message. +- Ordering: `DefaultFlushEntityEventListener.scheduleUpdate` — source comment "check + nullability but do not doAfterTransactionCompletion command execute" then + `new Nullability( session ).checkNullability( values, persister, true )` (5.6 line + 320) immediately followed by `new EntityUpdateAction(...)` (line 325). Re-checked on + branch **6.6** (253 → 258) and **7.0** (253 → 258) — same order in all three. + `EntityUpdateAction.execute()` (line 170) calls `preUpdate()` (line 176), i.e. after + the check. +- INSERT side: `AbstractEntityInsertAction.nullifyTransientReferencesIfNotAlready()` + (6.6, lines 122-126) runs `nullifyTransientReferences( getState() )` and then + `new Nullability( getSession() ).checkNullability( getState(), getPersister(), false )` + — which is where "or transient" in the message comes from. +- Distinct exception for the genuine unsaved-instance case: + `gh api search/code 'q="object references an unsaved transient instance"'` → 1 + production hit (`metamodel/mapping/EntityIdentifierMapping.java`), a different + message. +- Counter-source read for the folklore this corrects: + https://www.baeldung.com/hibernate-not-null-error attributes the same message to two + causes ("null value for a column marked nullable = false" / "an association + referencing an unsaved instance") and never mentions the path shape. + +**Confidence: verified** (primary source read on three branches + reproducible +`gh api` code searches). + +### I2 — Hibernate's core not-null check is silently off when Bean Validation is active and `hibernate.check_nullability` is unset; measure it instead of inferring it + +**Sources checked.** + +- https://docs.hibernate.org/orm/6.6/javadocs/org/hibernate/cfg/ValidationSettings.html + — `CHECK_NULLABILITY`: "Enable nullability checking, raises an exception if an + attribute marked as not null is null at runtime"; "Defaults to disabled if Bean + Validation is present in the classpath and annotations are used, or enabled + otherwise." +- `TypeSafeActivator.applyCallbackListeners`, read on disk for **5.6** (lines 114-115) + and **6.6** (lines 116-117): guarded by `modes.contains( CALLBACK ) || + modes.contains( AUTO )`, then, under the comment "de-activate not-null tracking at + the core level when Bean Validation is present unless the user explicitly asks for + it", `if ( cfgService.getSettings().get( CHECK_NULLABILITY ) == null ) + … setCheckNullability( false )`. So the toggle is driven by a classpath/dependency + change, exactly as the candidate claimed. +- `Nullability` line 69-73 corroborates in-source: "Typically when Bean Validation is + on, we don't want to validate null values at the Hibernate Core level. Hence the + checkNullability setting." +- Measurement API existence checked before writing it into a directive: + `SessionFactoryImplementor.getSessionFactoryOptions()` (javadoc: "Get the options + used to build this factory") and `SessionFactoryOptions.isCheckNullability()` + (`boolean isCheckNullability()`). +- Skip conditions written into directive 4 read from `Nullability` lines 92-104 + (`getPropertyInsertability`/`getPropertyUpdateability`, `UNFETCHED_PROPERTY`, + `GenerationTiming.NEVER`); that `@UpdateTimestamp` is in-memory generated was + confirmed via `UpdateTimestamp.java` (`@ValueGenerationType(generatedBy = + UpdateTimestampGeneration.class)`) → `UpdateTimestampGeneration.getGenerationTiming() + == GenerationTiming.ALWAYS` with a non-null `getValueGenerator()`. +- Enforcement-layer claim cross-read at + https://thorben-janssen.com/hibernate-tips-whats-the-difference-between-column-nullable-false-and-notnull/ + — `@Column(nullable = false)` "adds a not null constraint to the database column, if + Hibernate creates the database table definition"; `@NotNull` is what Bean Validation + checks on pre-update/pre-persist. This is also why that article describes Hibernate + "just executing the SQL UPDATE" — it describes the Bean-Validation-present default, + i.e. the check switched off. + +**Confidence: verified.** + +### I3 — an audit column is evidence only about the writer that sets it, so all-NULL `update_dt` cannot separate "never updated" from "every update failed" + +**Sources checked.** + +- Spring Data JPA `AuditingEntityListener` read on disk: `touchForCreate` is + `@PrePersist` (line 84-85), `touchForUpdate` is `@PreUpdate` (line 104-105) — so + `@LastModifiedDate` is written by a JPA lifecycle callback and inherits the ordering + established in I1. +- Bulk-DML claim taken from the primary Hibernate guide + (https://docs.hibernate.org/orm/6.6/querylanguage/html_single/Hibernate_Query_Language.html): + "The effect of an `update` or `delete` statement is not reflected in the persistence + context, nor in the state of entity objects held in memory at the time the statement + is executed"; `@Version` is untouched unless the statement is `versioned`. I looked + for a primary statement that callbacks specifically are not invoked and did **not** + find one in the 6.6 user guide, the 6.6 query-language guide, or the Jakarta + Persistence 3.1 spec page (§4.10's body was not in the served content). The page + therefore states the doc-backed fact (no persistence-context effect → no flush action + → no callback) and does not assert a spec quotation it cannot cite. +- Trigger axis: https://www.postgresql.org/docs/current/sql-createtrigger.html — "A + trigger that is marked `FOR EACH ROW` is called once for every row that the operation + modifies." The doc states per-row firing; client-independence is the structural + consequence of the trigger living on the relation, and the page words it that way + rather than quoting the doc for it. +- Field evidence carried over from the session: PRD `manage.building_tenant_floor_info` + — 574 rows with a NULL business code, all with `update_dt` NULL, whose UPDATEs were + dying in `Nullability` before `EntityUpdateAction` was created. + +**Confidence: verified** for the mechanism (callback ordering, `@PreUpdate` writer, +bulk-DML persistence-context semantics); the production observation that motivated it +is labelled as a field observation in the page's Sources. ## Existing-layer check -- Merged-main near-dup scan before consolidation: pairwise Jaccard over - title + "When this applies" across all 141 merged pages → **0 flagged pairs**; - previously merged content carries no duplication. -- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one - canonical page each (rate limiting 8→1, call-site enumeration 7→folded into - the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1, - env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1, - harness-mediated results 2→1, leaked artifacts 2→1, orchestration category - naming unified). Three near-pairs kept distinct after trigger comparison, - with mutual `related:` links (differential setup vs interpretation; expansion - semantics vs off-switch design; import-time tactics vs level choice). -- 24 existing pages received union-merged amendments; additions already present - in main (from #16/#20) were skipped, and all non-canonical `related:` ids - were remapped to canonical page ids (post-merge broken-link scan: 0). +Routed via `INDEX.md` → `backend` (application-code concern, JVM subtree) and +`databases` (reading live rows to derive a claim), then read each domain index and +every page whose "load when" overlapped. + +Pages read: backend-java-jpa-entity-mapping, backend-java-jpa-persistence-context, backend-java-kotlin-frameworks-and-jpa, databases-schema-design-nullability-and-defaults, databases-data-survey-surveying-live-data-for-a-rule, databases-schema-design-soft-delete + +Also read (non-page routing/schema files): `INDEX.md`, `wiki/backend/index.md`, +`wiki/backend/java/index.md`, `wiki/databases/index.md`, `wiki/debugging/index.md`, +`AGENTS.md`, `templates/page.md`. + +**Overlaps and what I did.** + +| Existing page | Overlap | Decision | +|---------------|---------|----------| +| `backend-java-jpa-persistence-context` | Owns flush timing and dirty checking — the *when does flush happen* question. Says nothing about the validation that runs inside `scheduleUpdate` or about callback ordering | No duplication; new page links it and stays off flush-timing | +| `backend-java-kotlin-frameworks-and-jpa` | Rule 3 covers "Kotlin non-null property vs nullable column"; rule 5 covers `@field:NotNull` targets | Closest existing coverage, but it is a Kotlin *class-shape* page, not a runtime-enforcement page. Kept separate; the new page defers the `@field:` target to it by id rather than restating it | +| `databases-schema-design-nullability-and-defaults` | Column-side NOT NULL/defaults design | Complementary axis (declaring vs interpreting/enforcing); linked from both new pages | +| `databases-data-survey-surveying-live-data-for-a-rule` | Same category and the same failure family ("a survey result read as more than it is"), but its subject is deriving mapping/enum/normalization rules and the empty-result substitution | Not a merge: adding an audit-column directive would contradict its own "When this applies". New sibling page in the same category, one-way link to it | +| `debugging/signals/reading-error-messages` (index line only) | Generic error-reading methodology | Left alone — the insight is stack-specific mechanics, which AGENTS.md routes to the owning domain | + +**Conflicts flagged.** None. No existing directive says the opposite of anything +ingested here. + +**Related links added.** New JPA page → `backend-java-jpa-persistence-context`, +`backend-java-kotlin-frameworks-and-jpa`, +`databases-schema-design-nullability-and-defaults`, +`databases-data-survey-audit-columns-as-update-evidence`. New data-survey page → +`databases-data-survey-surveying-live-data-for-a-rule`, +`databases-schema-design-nullability-and-defaults`, +`backend-java-jpa-not-null-check-and-lifecycle-callbacks`. Inline id references also +made to `databases-schema-design-soft-delete`. Back-links into +`surveying-live-data-for-a-rule.md` were deliberately **not** added: PR #78 edits that +file's `related:` line, and a one-way link satisfies invariant 4 without contending +for the same line. + +## Open-PR check + +Listed with `gh pr list --repo choiyounggi/dev-loop --state open --search "head:knowledge/"` +→ 23 open heads: #86, #80, #79, #78, #76, #74, #73, #72, #69, #68, #66, #64, #62, #61, +#58, #57, #56, #55, #52, #51, #50, #49, #47. + +Per-head `wiki/**` file lists pulled with `gh pr view --json files` (the heads are +cross-repo forks — `git fetch origin ` fails with "couldn't find remote ref", so +`gh pr diff` was used for content). Heads touching the two categories I ingest into: + +| Head | Files in my categories | Content overlap? | +|------|------------------------|------------------| +| #73 | `wiki/backend/java/jpa/raw-jdbc-inside-a-jpa-transaction.md`, `wiki/backend/java/index.md` | **No.** Subject is a raw JDBC connection used inside a JPA transaction. Same category and same index file, different case | +| #78 | `wiki/databases/data-survey/catalog-statistics-as-current-state.md`, `surveying-live-data-for-a-rule.md`, `wiki/databases/index.md` | **No.** Diffed in full: its subject is catalog estimates (`relpages`/`reltuples`/`pg_stats`) being stale relative to the heap — a staleness-of-statistics case, not a writer-coverage case. Family resemblance only | +| #74 | `wiki/databases/index.md` (+ query-optimization pages) | No — index row only | + +Verdicts: **I1 → new**, **I2 → new** (both into one page, same mechanism/one case per +AGENTS.md rule 1), **I3 → new**. Nothing folded, nothing dropped. + +**Contention to expect at merge (not duplication):** three additive index rows — +`wiki/backend/java/index.md` (also touched by #73) and `wiki/databases/index.md` (also +touched by #74 and #78). Each is a one-row table append; resolve by keeping both rows. +`wiki/backend/index.md` was intentionally not modified (it routes to the subtree index +only), which keeps this PR off the file that #76/#72/#68/#58/#55/#51 all edit. + +I3 does **not** link `databases-data-survey-catalog-statistics-as-current-state` +(#78's new page) even though the pairing would be apt: that id does not exist on `main` +yet, and invariant 4 would break if #78 is rejected. Worth adding as a `related:` on +whichever of the two merges second. ## Routing decision -- New categories: `infrastructure/agent-orchestration` (5 pages; unified the - competing `orchestration`/`agent-orchestration` names), `databases/data-survey` - (1), `qa/deliverables` (1). All other pages route into existing categories. -- Canonical-path decisions: rate limiting → `backend/common/reliability/` - (sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr - diagnostics → `platforms/processes/` (concern spans beyond shells); leaked - artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site - enumeration → the existing `backend/common/change-impact/` page. -- All 38 new pages listed in their domain indexes (nearest-index rule; backend - routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain - summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter, - ids, related-links, index coverage, size, qualifiers, staleness → 0 findings. +| Insight | Target | New category? | +|---------|--------|---------------| +| I1 + I2 | `backend/java/jpa/not-null-check-and-lifecycle-callbacks.md` (id `backend-java-jpa-not-null-check-and-lifecycle-callbacks`), **new page** in the existing `jpa` category | No — `backend/java/jpa` already exists and holds entity-mapping/persistence-context | +| I3 | `databases/data-survey/audit-columns-as-update-evidence.md` (id `databases-data-survey-audit-columns-as-update-evidence`), **new page** in the existing `data-survey` category | No — `data-survey` exists (created for `surveying-live-data-for-a-rule`) and its remit is exactly "reading live data as evidence" | + +**Why I1 and I2 share one page.** They are the same mechanism seen from two sides: the +exception thrown by `Nullability` (I1) and whether `Nullability` runs at all (I2). Both +are answered from the same source lines, and a reader arriving with either question +needs the other half — splitting them would produce two pages whose "load when" lines +each pull in the other. + +**Why I3 is not on the JPA page.** Its situation is "I am looking at production rows +and about to state a behavioural conclusion" — a survey, reached from the `databases` +domain, and true for any callback-written audit column (not only Hibernate's). The JPA +page owns the ORM mechanism; the data-survey page owns the evidential rule and links to +it. + +**Domain hints honoured.** Queue hints were `backend` (I1, I2) and `databases` (I3); +both matched the routing that `INDEX.md` produced independently. + +Sizes: 91 and 73 body lines (limit 120). Both pages carry `confidence: verified` and +`last_verified: 2026-08-13`. + +**Citation ledger** (12 distinct URLs across the two pages, so the reviewer can spot a +gap rather than trust a blanket claim): + +| URL | How it was opened this session | +|-----|-------------------------------| +| hibernate-orm `engine/internal/Nullability.java` (5.6) | raw fetch → read on disk (226 lines) | +| hibernate-orm `event/internal/DefaultFlushEntityEventListener.java` (6.6) | raw fetch of 5.6 read on disk; 6.6 and 7.0 fetched and grepped for the two call sites | +| hibernate-orm `action/internal/AbstractEntityInsertAction.java` (6.6) | raw fetch → grepped call site | +| hibernate-orm `boot/beanvalidation/TypeSafeActivator.java` (6.6) | raw fetch → read lines 95-125; 5.6 equivalent also fetched | +| `ValidationSettings` javadoc (6.6) | WebFetch (quote extracted) | +| `SessionFactoryOptions` javadoc (6.6) | WebFetch (signature confirmed) | +| Hibernate Query Language guide (6.6) | WebFetch (mutation-statement quote extracted) | +| spring-data-jpa `AuditingEntityListener.java` | raw fetch → grepped annotations | +| `thorben-janssen.com` @Column vs @NotNull | WebFetch (quotes extracted) | +| `baeldung.com/hibernate-not-null-error` | WebFetch (used as the counter-source) | +| PostgreSQL `CREATE TRIGGER` | WebFetch (FOR EACH ROW quote extracted) | +| — | Each of the blob/javadoc/guide URLs *as written in the frontmatter* additionally + re-checked with `curl -o /dev/null -w '%{http_code}' -L` → 200 (9 of them; the two + WebFetch-only pages above were opened by WebFetch, not curl) | + +Two things this flush did **not** establish, stated so a later revise can close them: +a primary-source quotation that bulk JPQL skips lifecycle callbacks specifically (the +Jakarta Persistence §4.10 body was not in the served spec page), and the Hibernate 7.x +equivalent of `TypeSafeActivator` (checked on 5.6 and 6.6 only; the flush-order check +did cover 7.0). + +## Decision Log + +- **Intent.** Drain 3 queued `★ Insight` candidates from one investigation into the + wiki as a single reviewed PR: 2 new pages, no merges into existing pages, no + candidate dropped. +- **Chose** one page for I1+I2 (`backend/java/jpa/not-null-check-and-lifecycle-callbacks`) + because both answer from the same `Nullability` source lines and either question + needs the other's answer. **Rejected** splitting them into an "error attribution" + page and a "configuration" page — their "load when" lines would each pull in the + other, which AGENTS.md rule 1 exists to prevent. +- **Chose** `databases/data-survey` for I3. **Rejected** merging it into + `surveying-live-data-for-a-rule` (its "When this applies" is scoped to deriving + mapping/normalization/enum rules — an audit-column directive would contradict its own + trigger) and **rejected** putting it on the JPA page (the evidential rule holds for + any callback-written audit column, not only Hibernate's). +- **Rejected** adding a back-link inside `surveying-live-data-for-a-rule.md`: open PR + #78 edits that file's `related:` line, and invariant 4 is satisfied by the one-way + link from the new page. +- **Rejected** linking `databases-data-survey-catalog-statistics-as-current-state` + (apt, but it exists only on #78's head — a broken id if #78 is rejected). +- **Left out on purpose:** no edit to `wiki/backend/index.md` (it routes to the subtree + index only), which keeps this PR off the file six open PRs already touch. +- **Downgraded one claim rather than sourcing it loosely:** the candidate asserted bulk + JPQL "skips callbacks". The page states the doc-backed persistence-context fact and + the mechanism it implies; the callbacks-specifically quotation is listed above as not + established from a primary source. +- **Cross-Check:** no independent adversarial reviewer was run — subagent/CLI + delegation was outside this invocation's scope. What *was* done, first-hand this + session: Hibernate sources read on disk for 5.6/6.6/7.0 (flush-order verified on all + three), `gh api search/code` used to prove the message literal has exactly one + production occurrence, every frontmatter URL opened (ledger above, 9 additionally + re-checked for HTTP 200), all 8 `related:`/inline ids resolved against `wiki/` by + script, both pages measured under the 120-line limit, and the flush gate's own + `Pages read:` resolution reproduced locally. Treat the two "not established" items + above as the known gaps. diff --git a/log.md b/log.md index c930fc2..7907922 100644 --- a/log.md +++ b/log.md @@ -43,3 +43,4 @@ Append-only. Format: `## [YYYY-MM-DD] ` and you must decide which attribute and which code path +produced it; you are about to fix it inside a `@PreUpdate` listener; or you are asking +why an entity's `nullable = false` was never enforced before (or stopped being). + +Column-side nullability design → [databases-schema-design-nullability-and-defaults]. +Reading the failed rows' audit columns → [databases-data-survey-audit-columns-as-update-evidence]. + +## Do this + +1. Identify the attribute from the **path shape**, not from the words "or transient". + `Nullability` throws this message from two sites that share one hardcoded literal — + it is the only production occurrence of that string in hibernate-orm — so the + wording says nothing about which site fired: + +| Path in the message | Means | Inspect | +|---------------------|-------|---------| +| No dot (`title`) | A top-level attribute of the entity held `null` when the check ran | That attribute's column mapping, and every path that assembles the entity | +| Contains a dot (`address.city`) | A sub-attribute of a composite value (`@Embeddable`) held `null`; only `checkSubElementsNullability` → `buildPropertyPath` produces dots, and it recurses into `CompositeType` | The embeddable's own `nullable = false` attributes — the owning entity attribute (`address`) was non-null | +| Contains a dot and the parent attribute is a collection | The same, reached through a collection whose **element type** is composite (`@ElementCollection` of `@Embeddable`); the first loaded non-null element decides | The element class's not-null attributes | + +2. On the UPDATE path, drop `@PreUpdate`/`@PostUpdate` from both the suspect list and + the fix. `DefaultFlushEntityEventListener.scheduleUpdate` runs + `new Nullability( session ).checkNullability( values, persister, … )` and only then + adds `EntityUpdateAction` to the action queue; the callbacks fire inside that + action's `execute()`. The order is the same in 5.6, 6.6 and 7.0. So when this + exception is thrown, no `@PreUpdate` listener has run on that entity: a listener + cannot have nulled the value, and a listener cannot supply it. +3. Fix the value where the entity state is assembled — the service, mapper, or + deserializer that produced the instance — or change the declared nullability if + "absent" is a real state of the domain. +4. Before ruling an attribute out, check whether the loop even examined it. + `Nullability` skips an attribute when it is not insertable (INSERT) or not + updatable (UPDATE), when its value is `UNFETCHED_PROPERTY` (lazy, not loaded), and + when Hibernate generates the value in memory (`GenerationTiming != NEVER` — e.g. + `@CreationTimestamp`, `@UpdateTimestamp`). +5. Measure whether the check is active instead of inferring it from the mapping. + `hibernate.check_nullability` "Defaults to disabled if Bean Validation is present in + the classpath and annotations are used, or enabled otherwise": + `TypeSafeActivator.applyCallbackListeners` calls `setCheckNullability( false )` + whenever the validation mode is `CALLBACK`/`AUTO` **and** the setting has no value. + Adding or removing a dependency such as `spring-boot-starter-validation` therefore + flips it. Read it, or assert it: + +| To establish | Do | +|--------------|----| +| The effective setting at startup | `emf.unwrap( SessionFactoryImplementor.class ).getSessionFactoryOptions().isCheckNullability()` | +| That the behaviour holds for this build | A test that flushes an entity whose `nullable = false` attribute is null and expects `PropertyValueException` | + +6. When the app-level check must hold regardless of which dependencies are on the + classpath, set `hibernate.check_nullability=true` explicitly. `@Column(nullable = + false)` alone does not give you a Bean Validation constraint — it "adds a not null + constraint to the database column, if Hibernate creates the database table + definition" — so with the core check off, the enforcement left is the DB constraint. + Add `@NotNull` (Kotlin: `@field:NotNull` → + [backend-java-kotlin-frameworks-and-jpa]) when you want the validator to reject it. + +## Edge cases + +| Case | Then | +|------|------| +| The same message on `persist()`/INSERT | The check runs in `AbstractEntityInsertAction.nullifyTransientReferencesIfNotAlready()`, immediately after `nullifyTransientReferences( getState() )` — a to-one attribute holding an unsaved instance is nulled first, then reported as null. That is where "or transient" comes from; the reported path is still the plain attribute name | +| The path names an attribute whose DB column is nullable | The entity declares not-null while the column allows NULL; existing rows can already violate it, and they surface only once this check is on (directive 5) | +| No exception at all despite a null on a `nullable = false` attribute | The check is off — the statement reaches the database, where a real NOT NULL constraint raises a `ConstraintViolationException` and a nullable column accepts the row silently | +| The failing attribute is `@UpdateTimestamp`/`@CreationTimestamp` | It is skipped by the check (directive 4); the reported path belongs to another attribute | +| The write path is a bulk JPQL/HQL `update` or native SQL | No entity flush happens, so this check never runs — the database constraint is the only gate → [databases-data-survey-audit-columns-as-update-evidence] | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Add a `@PreUpdate` listener that fills the missing value | Set it where the entity state is assembled (directive 3) | The check runs before `EntityUpdateAction` exists, so no `@PreUpdate` has run — the listener is never reached on the failing flush | +| Read "or transient" as evidence that an unsaved association caused it | Read the path shape (directive 1); on INSERT, treat a nulled transient reference as one of the causes | One literal is shared by both throw sites; the distinct unsaved-instance error carries its own message, "object references an unsaved transient instance" | +| Conclude the check is on because the mapping says `nullable = false` | Read `isCheckNullability()` or assert the exception in a test (directive 5) | Bean Validation on the classpath disables the core check unless the setting is explicit | +| Set `hibernate.check_nullability=false` to get past the exception | Supply the value, or relax the declared nullability | Disabling moves the failure to the DB constraint, or writes the incomplete row when the column is nullable | + +## Sources + +- https://github.com/hibernate/hibernate-orm/blob/5.6/hibernate-core/src/main/java/org/hibernate/engine/internal/Nullability.java — two throw sites share the literal `"not-null property references a null or transient value"`; the dotted path comes only from `buildPropertyPath(...)` via `checkSubElementsNullability`, which recurses into `CompositeType` and into collections whose element type is composite; the loop skips non-checkable, `UNFETCHED_PROPERTY`, and `GenerationTiming != NEVER` attributes; comment: "Typically when Bean Validation is on, we don't want to validate null values at the Hibernate Core level. Hence the checkNullability setting." +- https://github.com/hibernate/hibernate-orm/blob/6.6/hibernate-core/src/main/java/org/hibernate/event/internal/DefaultFlushEntityEventListener.java — `scheduleUpdate`: "check nullability but do not doAfterTransactionCompletion command execute" → `new Nullability( session ).checkNullability(...)` precedes `new EntityUpdateAction(...)` (same order on 5.6 and 7.0) +- https://github.com/hibernate/hibernate-orm/blob/6.6/hibernate-core/src/main/java/org/hibernate/action/internal/AbstractEntityInsertAction.java — `nullifyTransientReferencesIfNotAlready()` nullifies transient references and then runs the CREATE-type nullability check +- https://github.com/hibernate/hibernate-orm/blob/6.6/hibernate-core/src/main/java/org/hibernate/boot/beanvalidation/TypeSafeActivator.java — "de-activate not-null tracking at the core level when Bean Validation is present unless the user explicitly asks for it": guarded by validation mode `CALLBACK`/`AUTO`, then `if ( cfgService.getSettings().get( CHECK_NULLABILITY ) == null ) … setCheckNullability( false )` +- https://docs.hibernate.org/orm/6.6/javadocs/org/hibernate/cfg/ValidationSettings.html — `CHECK_NULLABILITY`: "Enable nullability checking, raises an exception if an attribute marked as not null is null at runtime"; "Defaults to disabled if Bean Validation is present in the classpath and annotations are used, or enabled otherwise" +- https://docs.hibernate.org/orm/6.6/javadocs/org/hibernate/boot/spi/SessionFactoryOptions.html — `boolean isCheckNullability()` exposes the effective setting +- https://thorben-janssen.com/hibernate-tips-whats-the-difference-between-column-nullable-false-and-notnull/ — `@Column(nullable = false)` "adds a not null constraint to the database column, if Hibernate creates the database table definition" and otherwise leaves validation to the database; `@NotNull` is what Bean Validation checks on pre-persist/pre-update +- https://www.baeldung.com/hibernate-not-null-error — the widely repeated two-cause framing this page corrects: the same message attributed to "a null value for a column marked with nullable = false" and to "an association referencing an unsaved instance", with no mention of the path shape diff --git a/wiki/databases/data-survey/audit-columns-as-update-evidence.md b/wiki/databases/data-survey/audit-columns-as-update-evidence.md new file mode 100644 index 0000000..6687da3 --- /dev/null +++ b/wiki/databases/data-survey/audit-columns-as-update-evidence.md @@ -0,0 +1,87 @@ +--- +id: databases-data-survey-audit-columns-as-update-evidence +domain: databases +category: data-survey +applies_to: [postgresql, mysql, general] +confidence: verified +sources: + - https://github.com/spring-projects/spring-data-jpa/blob/main/spring-data-jpa/src/main/java/org/springframework/data/jpa/domain/support/AuditingEntityListener.java + - https://docs.hibernate.org/orm/6.6/querylanguage/html_single/Hibernate_Query_Language.html + - https://github.com/hibernate/hibernate-orm/blob/6.6/hibernate-core/src/main/java/org/hibernate/event/internal/DefaultFlushEntityEventListener.java + - https://www.postgresql.org/docs/current/sql-createtrigger.html +last_verified: 2026-08-13 +related: [databases-data-survey-surveying-live-data-for-a-rule, databases-schema-design-nullability-and-defaults, backend-java-jpa-not-null-check-and-lifecycle-callbacks] +--- + +# Audit Columns as Evidence About a Row's Update History + +## When this applies + +You are surveying live rows and about to turn an audit column into a behavioural +claim: `update_dt`/`updated_at`/`modified_by` is NULL (or unchanged) on every row of +interest and you read that as "these rows were never modified", "this feature is +unused", or "the bad values came in at insert and nothing touched them since". Also +when an incident investigation needs to know whether writes to those rows were +*attempted*. + +Deriving a mapping or enum rule from a survey → [databases-data-survey-surveying-live-data-for-a-rule]. + +## Do this + +1. **Identify what writes the column before reading it as history**, and bound the + claim to that writer: + +| Written by | Records | A NULL therefore means | +|------------|---------|------------------------| +| ORM lifecycle callback (`@PreUpdate`; Spring Data's `AuditingEntityListener.touchForUpdate` is `@PreUpdate`) | Only updates that reached the entity's flush action | No *successful, entity-level* update ran | +| Application code assigning the field | Only the code paths that assign it | No update through those paths | +| DB trigger (`BEFORE UPDATE … FOR EACH ROW`) or a generated/`ON UPDATE` column | Every statement the database executed on the row, whatever the client | The row's columns were not updated | + +2. **State the bounded claim in the deliverable**: "no update ran through the path that + stamps this column, as of " — not "never updated". Name the writer you found + in directive 1 alongside the count. +3. **Enumerate the write paths that leave the column untouched** before concluding + anything, and check each against the code: + +| Path | Why the column stays as it was | +|------|-------------------------------| +| The update failed pre-flush | Validation that runs before the update action is scheduled — e.g. Hibernate's not-null check → [backend-java-jpa-not-null-check-and-lifecycle-callbacks] — throws before any `@PreUpdate` listener runs | +| Bulk JPQL/HQL `update`, or native SQL | "The effect of an `update` or `delete` statement is not reflected in the persistence context": no entity flush, so no callback and no `@Version` bump unless the statement is `versioned` | +| Another service, migration, or manual SQL writes the table | It never loaded the entity, so the ORM's auditing was never in the path | +| The transaction rolled back after the callback set the field | The in-memory stamp is discarded with the transaction | + +4. **Judge update history on an independent axis** — a history/audit table, the + application logs for the writing endpoint, CDC/WAL, or a DB-side trigger installed + going forward. Record which axis the conclusion rests on. +5. **Require one positive control before acting on the absence.** Find at least one row + whose audit column *is* set by the same writer (or a test that exercises it). Absent + that, the NULLs are equally explained by "the writer never worked here", and any + decision built on them (backfill, drop the column, close the ticket as "unused") is + resting on an unmeasured mechanism. + +## Edge cases + +| Case | Then | +|------|------| +| Only old rows are NULL | The auditing was added later — find the migration or the earliest non-null value and read NULL as "before instrumentation", not as behaviour | +| `updated_at` equals `created_at` on every row | The writer stamps both at insert; equality is evidence of no update only if you have confirmed the update path stamps it (directive 5) | +| Every non-null value shares one timestamp | A backfill or bulk migration wrote them; those rows carry no per-row update history | +| The column is `NOT NULL DEFAULT now()` | It cannot distinguish "inserted" from "updated" at all — pair it with a separate insert timestamp or a history table | +| The claim needed is "did anyone *read*/attempt this" | Audit columns cannot answer it in any configuration; go to application logs or the DB's statement/audit logging | +| Rows are soft-deleted | The delete may run as an update through a different path than the business update → [databases-schema-design-soft-delete] | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Conclude "never modified" from all-NULL audit columns | State the bounded claim (directive 2), then confirm on an independent axis (directive 4) | The column records successes of one path; failed, bulk, and out-of-band writes leave it untouched | +| Treat the audit column as the failure timeline in an incident | Use application logs or a history table for attempts, and keep the audit column for confirmed successes | The investigation's subject is the failing write, which is exactly the event the column cannot record | +| Add `@PreUpdate` auditing to close the gap you just found | Add a DB trigger (or generated column) when the requirement is "every statement, whatever the client" | Callback auditing is bypassed by bulk DML, native SQL, and other services by construction | + +## Sources + +- https://github.com/spring-projects/spring-data-jpa/blob/main/spring-data-jpa/src/main/java/org/springframework/data/jpa/domain/support/AuditingEntityListener.java — `AuditingEntityListener.touchForCreate` is annotated `@PrePersist` and `touchForUpdate` `@PreUpdate`, so `@LastModifiedDate` is written by a JPA lifecycle callback +- https://docs.hibernate.org/orm/6.6/querylanguage/html_single/Hibernate_Query_Language.html — mutation statements: "The effect of an `update` or `delete` statement is not reflected in the persistence context, nor in the state of entity objects held in memory at the time the statement is executed"; "It's the responsibility of the client program to maintain synchronization of state held in memory with the database"; `update` leaves `@Version` attributes alone unless the `versioned` keyword is used +- https://github.com/hibernate/hibernate-orm/blob/6.6/hibernate-core/src/main/java/org/hibernate/event/internal/DefaultFlushEntityEventListener.java — `scheduleUpdate` runs the nullability check before `EntityUpdateAction` is queued, so a pre-flush validation failure precedes every `@PreUpdate` listener +- https://www.postgresql.org/docs/current/sql-createtrigger.html — "A trigger that is marked `FOR EACH ROW` is called once for every row that the operation modifies"; the trigger is defined on the relation, not in a client, which is what makes it the axis independent of the application's write path +- Field observation 2026-08-13 (PostgreSQL, `manage.building_tenant_floor_info`): all 574 rows with a NULL business code also had `update_dt` NULL, which read as "never edited"; those rows' UPDATEs were in fact failing in Hibernate's `Nullability` check before `EntityUpdateAction` was created, so the `@PreUpdate` stamp never ran. The same repository also updates a table by bulk JPQL without setting `updateDt`, a second path with the same signature diff --git a/wiki/databases/index.md b/wiki/databases/index.md index 66083a7..d781793 100644 --- a/wiki/databases/index.md +++ b/wiki/databases/index.md @@ -50,6 +50,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| +| [audit-columns-as-update-evidence](data-survey/audit-columns-as-update-evidence.md) | About to read `update_dt`/`updated_at`/`modified_by` (NULL, or equal to the insert timestamp) as evidence that rows were never modified or a feature is unused; an incident needs to know whether writes to those rows were *attempted*; deciding whether ORM callback auditing or a DB trigger is the right writer for the claim you need to make | | [surveying-live-data-for-a-rule](data-survey/surveying-live-data-for-a-rule.md) | A task says to sample real data to decide a mapping table, normalization/canonicalization rule, enum value set, or parsing rule; a `GROUP BY`/`DISTINCT` survey came back with zero rows; deciding what evidence replaces the data when the table is empty; recording in the deliverable which evidence a rule was actually derived from | ## sqlite