roachtest: ldr/transactional/kv0/shutdown_node failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.ldr/transactional/kv0/shutdown_node failed with artifacts on master @ 2bf53f5e2960d67b69a8d3576c03e3e9fc3dae2a:

Failed with:

(sql_runner.go:324).Scan: error scanning '&{0x295571c458c0 <nil>}': pq: at or near "on_constraint_conflict": syntax error
(1) pq: at or near "on_constraint_conflict": syntax error
Error types: (1) *pq.Error
test artifacts and logs in: /artifacts/ldr/transactional/kv0/shutdown_node/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21513554-1789192452-224-n7cpu8-0001		10.91.102.167
teamcity-21513554-1789192452-224-n7cpu8-0002		10.91.102.165
teamcity-21513554-1789192452-224-n7cpu8-0003		10.91.103.15
teamcity-21513554-1789192452-224-n7cpu8-0004		10.91.102.220
teamcity-21513554-1789192452-224-n7cpu8-0005		10.91.103.24
teamcity-21513554-1789192452-224-n7cpu8-0006		10.91.103.38
teamcity-21513554-1789192452-224-n7cpu8-0007		10.91.97.14

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=8
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=false
• metamorphicLeases=default
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

/cc @cockroachlabs/cdc

Improve this report!

Jira issue: CRDB-68335

[blathers-crl]

cc @cockroachdb/cdc

[sjain022]

/investigate

[github-actions]

Investigation: ldr/transactional/kv0/shutdown_node

Investigated failure: issue body / first failure
Failure SHA: 2bf53f5e2960d67b69a8d3576c03e3e9fc3dae2a (master, v26.4.0-alpha.2)
Confidence: high

What This Test Does

ldr/transactional/kv0/shutdown_node sets up bidirectional Logical Data
Replication (LDR) in transactional mode between two clusters, runs the kv0
workload, then shuts down a node on one side to verify the LDR job survives a
node failure. This variant is metamorphically run in mixed-version mode:
one of the two clusters may be started on a supported predecessor release
(minimum V26_3), the other on current (v26.4).

Where the Failure Occurs

The test fails during setup, ~1 minute in, before any node is shut down —
so the "shutdown_node" logic is not implicated. It fails while issuing the
CREATE LOGICAL REPLICATION STREAM statement on the destination cluster:

(sql_runner.go:324).Scan: error scanning '&{...}': pq: at or near "on_constraint_conflict": syntax error

Stack trace (failure_1.log):

setupLDR.func2  pkg/cmd/roachtest/tests/logical_data_replication.go:1574  (r.Scan after QueryRow)
setupLDR        pkg/cmd/roachtest/tests/logical_data_replication.go:1585  (startLDR on the LEFT cluster)
TestLDROnNodeShutdown  ...:758

From test.log:

logical_data_replication.go:1408: using predecessor version v26.3.1 for left cluster
logical_data_replication.go:1428: left cluster version: v26.3.1, right cluster version: current

The stream created on the right (current/v26.4) cluster succeeds
(line 1582);
the stream created on the left (v26.3.1) cluster
(line 1585)
fails to parse, because v26.3.1 does not have the ON_CONSTRAINT_CONFLICT
grammar keyword.

Analysis

Likely cause (high confidence): the roachtest unconditionally adds the
ON_CONSTRAINT_CONFLICT WITH-option in transactional mode, without gating on
cluster version, so it is sent to a v26.3 predecessor that cannot parse it.

Evidence — multiple independent pieces converge:

1. The ON_CONSTRAINT_CONFLICT grammar keyword and the FK/UC conflict-policy
   feature were added to master on 2026-08-31 in commit
   32cdcfc5a7f ("logical,sql: gate FK and UC replication on a
   conflict-resolution policy"). This is not present on release-26.3
   (verified: git grep ON_CONSTRAINT_CONFLICT origin/release-26.3 is empty;
   the commit is not an ancestor of release-26.3). So v26.3.x binaries reject
   the token as a syntax error.

2. constraintConflictOptions
   (lines 1234-1245)
   returns ON_CONSTRAINT_CONFLICT = 'pause' (or 'dlq') for every
   transactional-mode run, with no version parameter and no version check.
   It is appended to the statement at
   line 1550.

3. The sibling gate for the DLQ already gets this right.
   maybeEnableDLQ
   (lines 1453-1473)
   explicitly disables the DLQ unless both clusters are current:
   if leftVersion != "current" || rightVersion != "current" { return false }
   with the comment "We can only enable the DLQ if both versions are on 26.4."
   The ON_CONSTRAINT_CONFLICT = 'pause' path was not given the same guard.

4. The artifact confirms the exact scenario: left = v26.3.1, right = current;
   the failing Scan is at line 1574 reached via startLDR on the left
   (v26.3.1) cluster at line 1585.

Why it's not just this test: any transactional-mode LDR variant that (a)
allows a V26_3 predecessor and (b) can place that predecessor on a cluster
that runs CREATE LOGICAL REPLICATION STREAM will hit this. Issue #175305
(ldr/transactional/conflict failed, same day, same on_constraint_conflict syntax error) is the same root cause.

Suggested fix: version-gate the conflict-policy option the same way the DLQ
is gated. Thread the left/right versions into constraintConflictOptions (or
compute the option in the same place dlqEnabled is computed) and omit
ON_CONSTRAINT_CONFLICT entirely when either cluster is a predecessor
(!= "current"). The kv0 table has no FK or secondary unique constraint, so
transactional LDR does not require a conflict policy for it — omitting the
option on mixed-version runs is safe. (On a same-version v26.4 run the option is
accepted; it is only strictly required for tables carrying FK/UC constraints.)
The TODO(darryl): we can remove this check in 27.1 note on the DLQ gate
applies equally to the new gate.

Related Issues and PRs

• roachtest: ldr/transactional/conflict failed #175305 roachtest: ldr/transactional/conflict failed — OPEN,
  release-blocker, filed the same day (2026-09-12) with the identical
  on_constraint_conflict syntax error. Same root cause (sibling transactional
  LDR test, mixed-version predecessor). Should be fixed together.
• Commit 32cdcfc5a7f "logical,sql: gate FK and UC replication on a
  conflict-resolution policy" (2026-08-31) — introduced the
  ON_CONSTRAINT_CONFLICT grammar (master/v26.4 only).
• Commit 41875ddec39 "roachtest: metamorphically enable the DLQ in
  transactional LDR tests" (2026-09-02) — added maybeEnableDLQ's version gate;
  the pause policy was left ungated.
• roachtest: ldr/transactional/kv0/network_partition failed #174880 ldr/transactional/kv0/network_partition failed (CLOSED) — same test
  family but a different failure mode (network partition), not obviously
  related.

Timeline

• 2026-08-31 32cdcfc5a7f adds ON_CONSTRAINT_CONFLICT grammar + FK/UC
  policy requirement to master (v26.4). Not backported to release-26.3.
• 2026-09-02 41875ddec39 adds DLQ metamorphic enablement with a
  both-clusters-current version gate; the pause conflict option is not gated.
• 2026-09-12 14:52 test run on master @ 2bf53f5e picks v26.3.1 for the
  left cluster; the CREATE LOGICAL REPLICATION STREAM ... ON_CONSTRAINT_CONFLICT = 'pause' on the left cluster fails to parse. Issue filed.

Detailed evidence and investigation notes

Log Excerpts

test.log:

9:  logical_data_replication.go:1408: using predecessor version v26.3.1 for left cluster
22: logical_data_replication.go:1428: left cluster version: v26.3.1, right cluster version: current
111: test failure #1: (sql_runner.go:324).Scan: error scanning '&{...}':
     pq: at or near "on_constraint_conflict": syntax error
139: --- FAIL: ldr/transactional/kv0/shutdown_node (161.25s)

failure_1.log (trimmed):

error scanning '&{...}': pq: at or near "on_constraint_conflict": syntax error
  | sqlutils.(*Row).Scan            sql_runner.go:324
  | tests.setupLDR.func2            logical_data_replication.go:1574
  | tests.setupLDR                  logical_data_replication.go:1585
  | tests.TestLDROnNodeShutdown     logical_data_replication.go:758

Code References

constraintConflictOptions — no version awareness:

func constraintConflictOptions(mode mode, dlqEnabled bool) []string {
	if mode != ModeTransactional {
		return nil
	}
	if dlqEnabled {
		return []string{"ON_CONSTRAINT_CONFLICT = 'dlq'", fmt.Sprintf("DLQ = %s.%s", ...)}
	}
	return []string{"ON_CONSTRAINT_CONFLICT = 'pause'"}
}

maybeEnableDLQ — the version gate that the pause path is missing:

// We can only enable the DLQ if both versions are on 26.4.
// TODO(darryl): we can remove this check in 27.1.
if leftVersion != "current" || rightVersion != "current" {
	return false
}

Grammar-presence verification

$ git grep -c ON_CONSTRAINT_CONFLICT origin/release-26.3 -- pkg/sql/parser/sql.y   # (empty)
$ git merge-base --is-ancestor 32cdcfc5a7f origin/release-26.3 ; echo $?           # non-zero: NOT an ancestor

Things Ruled Out

• Node-shutdown / failover bug: ruled out. The failure is at setup (line
  1574/1585) at ~14:52:05, before StopE is ever called. The test never reaches
  the shutdown phase.
• Server-side LDR/transactional-mode defect: ruled out. The error is a
  client-side SQL parse error (*pq.Error, "syntax error") returned
  immediately, not a runtime replication error.
• Runtime-assertions build artifact: the issue carries
  B-runtime-assertions-enabled, but the failure is a plain parser rejection
  unrelated to assertions.

Other Failure Occurrences

Only one occurrence in this thread (the issue body). Cross-thread: #175305
shows the identical on_constraint_conflict syntax error on the same day — same
failure mode, same fix.

Tooling Feedback

• git log -S <string> over the blobless clone timed out (2m) on
  pkg/sql/parser/sql.y; git merge-base --is-ancestor + git grep <ref> were
  the fast substitutes. No blocker, just a note that pickaxe searches are slow
  in this environment.
• All required artifacts (TeamCity artifacts.zip, test.log, failure_1.log)
  and git history were accessible. No denied tools affected this investigation.

---

Workflow run: https://github.com/cockroachdb/cockroach/actions/runs/34702438432

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

---

Generated by claude-opus-4-8 at high effort. Use /investigate fable-5 max to go deeper.

Workflow run · Was this investigation helpful? Leave a 👍 or 👎 on this comment.