Skip to content

fix(core,eth): fix nonce too low in debug_traceTransaction - #2587

Closed
gzliudan wants to merge 2 commits into
XinFinOrg:dev-upgradefrom
gzliudan:fix-trace-replay-a
Closed

gzliudan wants to merge 2 commits into
XinFinOrg:dev-upgradefrom
gzliudan:fix-trace-replay-a

Conversation

@gzliudan

@gzliudan gzliudan commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Proposed changes

Debug tracing rebuilds the pre-state of a transaction by re-executing the transactions that come before it. stateAtTransaction, the replay behind debug_traceTransaction and debug_traceCall, did that with core.ApplyMessage, which knows nothing about the routing block processing applies. While the XDCX receiver fork is active (mainnet TIPXDCXBlock 38,383,838TIPXDCXReceiverDisableBlock 80,370,900), transactions to the system addresses 0x91/0x92/0x93/0x94 are handled by ApplyEmptyTransaction: no EVM execution, no nonce check, no nonce increment and no state change. A plain ApplyMessage replay executes them as ordinary EVM transactions and bumps the sender nonce, so every following transaction of the same sender is replayed against a nonce the chain never had. That replay also credited the block fee to the zero address instead of the coinbase owner and skipped the historical balance bypass.

The same trailing ApplyMessage and the same unconditional skip sit in the block-level APIs, debug_intermediateRoots and traceBlock; those two are fixed separately, because they are different defects with their own probes.

Symptoms

  • nonce too lowdebug_traceTransaction on any transaction that follows a system-address transaction of the same sender, while the fork is active. stateAtTransaction never skipped, so it bumped a nonce block processing leaves alone. Apothem block 48,667,667 (0x2e69c13, transactions 5–7), mainnet block 39,083,312 (0x2545d30, transaction 5).

Fix

Two commits.

  1. refactor(core): ApplyTransactionWithEVM decided the routing inline, so every replay caller had to re-derive it and each copy drifted. The decision moves into routeTransaction, the per-transaction finalisation into finaliseTxState and the sign-transaction nonce handling into applySignTransactionNonce, so a replay can follow exactly the same path. The log a non-EVM handler records moves into addNonEVMTxLog for the same reason: it is not a receipt artefact, StateDB.AddLog advances the block-wide log count every later log takes its index from, and a copy that dropped it would make the following transactions report indexes the chain never had. The commit also adds core.ApplyTransactionForReplay, the entry point a replay should use instead of core.ApplyMessage: it shares the routing, records the same log through the same helper, and skips the receipt and its bloom, which a replay does not need. It refuses an EVM that carries a tracer (it never fires OnTxStart/OnTxEnd) or a state that is not a *state.StateDB, with sentinel errors the caller can tell apart from a transaction failure. No behaviour change.
  2. fix(eth): stateAtTransaction goes through core.ApplyTransactionForReplay instead of core.ApplyMessage. A replay that was handed an EVM it cannot use is reported as it is instead of being blamed on the transaction. The tracers test backend mirrors the same change so it keeps modelling the production behaviour.

Upstream

No upstream fix to port: geth has no non-EVM transaction concept, so neither the routing nor the trailing ApplyMessage have a geth counterpart.

Tests

  • core/state_processor_test.goTestApplyTransactionForReplayKeepsTheNonEVMTxLog: drives the replay and block processing of a transaction to a system address (the trading state address with the receiver fork active, and the block signers address) and compares the logs they record, so the replay cannot lose the log that advances the block-wide log count the following transactions index their logs with; without it the replay records no log where block processing records one.
  • eth/state_accessor_test.goTestStateAtTransactionReplayKeepsNonceLessSenderNonce: replays the block behind the issue (first transaction to the trading state address, receiver fork active from genesis) and asserts the sender nonce after the replay is still 0; the old replay left it at 1, which is what made the following transaction fail with nonce too low.
  • eth/tracers/api_test.goTestTraceTransactionSkipNonceTransactions: drives debug_traceTransaction for the skip-nonce transaction itself and for the follower, over both receiver fork settings.

End-to-end verification

The branch binary was run against archive nodes of both live networks, on the same data directory and the same node as the baseline binary (dev-upgrade @ cdce8fc5c), and the tracing RPCs were compared on the same blocks.

Mainnet block 39,083,312 (0x2545d30, 14 transactions, inside the window; tx 0 goes to 0x92, tx 5 to 0x90 from the same sender with the same nonce) and Apothem block 48,667,667 (0x2e69c13, 8 transactions; tx 0 to 0x92, tx 5 to 0x89 from the same sender with the same nonce):

probe baseline this branch
debug_traceTransaction (the follower) tracing failed: nonce too low ok

Manual test plan: start a node on that data directory with --rpcapi debug and call debug_traceTransaction on 0x2545d30 and 0x2e69c13; the baseline binary fails both and this branch passes them.

Regression: on both networks this branch imported testnet and mainnet segments normally, with no bad block, no panic and no error attributable to the change.

Types of changes

  • fix: A bug fix
  • refactor: A code change that neither fixes a bug nor adds a feature
  • test: Adding missing tests or correcting existing tests
  • build / ci / chore / docs / feat / perf / revert / style

Impacted Components

  • Geth
  • Not sure (the changes are the debug JSON-RPC surface plus a behaviour-preserving refactor of the block processing routing in core/state_processor.go)
  • Consensus
  • Account
  • Network
  • Smart Contract
  • External components

Checklist

  • This PR has sufficient test coverage (unit/integration test)
  • Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet (see "End-to-end verification")
  • Tested the backwards compatibility — the change only affects replays; block processing keeps the same conditions, finalisation and nonce handling, and no API, RPC method or state format changes
  • Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs — not done for this branch. It was run against live testnet and mainnet archive nodes instead (segments imported, no bad block, no panic, no new error), and the tracing RPCs were compared block by block against the baseline binary.
  • Tested with XDC nodes running this version co-exist with those running the previous version — the branch ran on the live networks next to the rest of the network without being dropped
  • Relevant documentation has been updated as part of this PR

Relation to other work

This is the first of three PRs that split a single debug tracing series, so every defect is reviewable on its own. It carries the shared routing and the replay entry point the other two build on.

The second PR (#2588) fixes debug_intermediateRoots — it replayed through ApplyMessage and dropped every transaction sent to the system addresses, so it returned one root fewer than the block has transactions and its roots did not match block processing. The third fixes the traces and intermediate roots of the block that activates TIPSigning, which did not remove the legacy block signers account block processing removes before the first transaction.

#2581 fixes the unconditional skip in traceBlock and in the state feeder of the JS tracer path — the nonce too high and the null hole in debug_traceBlock*. It carries its own copy of the skipNonceForkCases / newSkipNonceBackend fixtures this PR introduces, because its block-level tests build on them; the second of the two PRs to merge drops that copy, which is the only textual overlap between them.

#2578 fixes the give-up paths of the same stateAtTransaction function whose replay this PR rewrites. Both branches touch eth/state_accessor.go; merging #2578 first keeps this one a trivial rebase.

ApplyTransactionWithEVM decided inline which handler a transaction goes
through, so any caller that has to replay a transaction had to re-derive that
decision on its own — and every copy drifts from block processing the moment
the routing changes.

Pull the decision into routeTransaction, the per-transaction finalisation into
finaliseTxState, and the sender nonce handling of ApplySignTransaction into
applySignTransactionNonce, so a replay can follow exactly the same path. The
log both non-EVM handlers record moves into addNonEVMTxLog for the same reason:
it is not a receipt artefact, StateDB.AddLog advances the block wide log count
that every later log takes its index from, so a copy that skipped it would make
the following transactions report indexes the chain never had.

Also add core.ApplyTransactionForReplay, the entry point a replay should use
instead of core.ApplyMessage: it shares the routing above, records the same log
through the same helper, and skips the receipt and its bloom, which a replay
does not need. It refuses an EVM that carries a tracer, as it never fires the
OnTxStart/OnTxEnd hooks, and it takes the state to replay on from evm.StateDB,
so the finalisation, the nonce handling and the TRC21 fee handling cannot land
on a state other than the one the execution wrote to. Both refusals are sentinel
errors, so a caller can tell them apart from a transaction failure. The replay
paths converted in the commits that follow go through it.

No behaviour change: the conditions, the finalisation, the nonce handling and
the log are the ones ApplyTransactionWithEVM already used. finaliseTxState takes
the state the EVM executes against rather than the plain *state.StateDB, because
a hooked state reports the balance burnt by self-destructed accounts to the
tracer on Finalise (core/state/statedb_hooked.go); IntermediateRoot forwards
unchanged either way. TestApplyTransactionForReplayKeepsTheNonEVMTxLog drives
the replay and block processing of a transaction to a system address and
compares the logs they record.
stateAtTransaction rebuilds the pre-state of a transaction by replaying the
transactions before it, and replayed them with core.ApplyMessage. That
diverges from block processing for transactions sent to the XDCX system
addresses 0x91/0x92/0x93/0x94: while the XDCX receiver fork is active those
are handled by ApplyEmptyTransaction, which does not execute the EVM and
leaves the sender nonce untouched, whereas the replay executed them as
ordinary EVM transactions and bumped the nonce. The block fee was credited to
the zero address instead of the coinbase owner, and the historical balance
bypass was not applied either.

Every following transaction of the same sender reuses that nonce, so
debug_traceTransaction failed for it and for everything after it that needed
the replay to get past it: on Apothem block 0x2e69c13 indices 5, 6 and 7 all
failed with nonce too low.

Replay through core.ApplyTransactionForReplay instead, the entry point block
processing and the other replay paths share: it routes those addresses exactly
like block processing, and skips the receipt, its logs and the bloom, which a
replay does not need. A replay that was handed an EVM it cannot use is
reported as it is instead of being blamed on the transaction. Mirror the same
change in the tracers test backend so it keeps modelling the production
behaviour.

Add TestStateAtTransactionReplayKeepsNonceLessSenderNonce, which fails on the
old replay with sender nonce after replay = 1 want 0, and
TestTraceTransactionSkipNonceTransactions for the debug_traceTransaction path,
with the skipNonceForkCases and newSkipNonceBackend fixtures it drives both
receiver fork settings with.

Refs: gzliudan/XDPoSChain#256
@coderabbitai

coderabbitai Bot commented Sep 19, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 6291f546-fb44-4a91-b07a-6c8f40874871

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant