Skip to content

fix(eth/tracers): fix the missing intermediate roots of a block - #2590

Open
gzliudan wants to merge 2 commits into
XinFinOrg:dev-upgradefrom
gzliudan:fix-trace-replay-intermediate-roots
Open

gzliudan wants to merge 2 commits into
XinFinOrg:dev-upgradefrom
gzliudan:fix-trace-replay-intermediate-roots

Conversation

@gzliudan

@gzliudan gzliudan commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Proposed changes

debug_intermediateRoots rebuilds the pre-state of a block by re-executing the transactions before each one. It did that with core.ApplyMessage, which knows nothing about the routing block processing applies, and it also dropped every transaction sent to the XDCX system addresses 0x91/0x92/0x93/0x94 with tx.IsSkipNonceTransaction(), a condition that does not consult the fork at all.

Inside the XDCX receiver fork window (mainnet TIPXDCXBlock 38,383,838TIPXDCXReceiverDisableBlock 80,370,900) those transactions are handled by ApplyEmptyTransaction: no EVM execution, no nonce check, no nonce increment and no state change, so the skip keeps the state right but leaves one root missing. Outside the window block processing does execute them and does bump the nonce the following transaction relies on, so dropping them loses exactly that increment and the replay hits the error on the next transaction. The replay also credited the block fee to the zero address instead of the coinbase owner and skipped the historical balance bypass, so even the roots it did return were not the ones block processing produces.

The same unconditional skip in traceBlock and in the state feeder of the JS tracer path — the one that makes debug_traceBlock* fail with nonce too high outside the window and leave a null hole inside it — is a separate change, submitted as #2581.

Symptoms

  • One root fewer than the block has transactions from debug_intermediateRoots whenever the receiver fork is active (mainnet, one root fewer per skip-nonce transaction).
  • Truncated root list outside the window: IntermediateRoots hits the replay error caused by the missing nonce increment and silently returns the roots collected so far (mainnet block 37,849,457, 0x2418971, 204 transactions, returns 94 roots).
  • The last debug_intermediateRoots root diverges from the block state root whenever the block charges a non-zero fee, because the replay credited it to the zero address instead of the coinbase owner (mainnet block 107,084,658, 0x661FB72).

Fix

IntermediateRoots replays through core.ApplyTransactionForReplay, the entry point that shares the routing with ApplyTransactionWithEVM but skips the receipt, its logs and the bloom, and it stops skipping the transactions sent to the system addresses. Every transaction then gets an intermediate root, a nonce-less one leaves the state untouched so its root is the previous one, the coinbase owner is credited the way block processing credits it, and the historical balance bypass applies. An EVM this replay cannot use is reported as it is instead of returning a short root list.

Upstream

No upstream fix to port: geth has no non-EVM transaction concept, so neither the trailing ApplyMessage nor the unconditional skip have a geth counterpart.

Tests

  • eth/tracers/api_test.goTestIntermediateRootsMatchesBlockProcessing: compares the roots against a replay through core.ApplyTransactionWithEVM, pins the last root to the block root after block finalisation and fails when the coinbase owner fee is dropped; the base has no intermediate-roots test, so this one is also the first to pin the replay instead of only counting its roots. The case drives a fee paying transaction, a transaction to an XDCX system address and a follower of the same sender reusing its nonce.

End-to-end verification

The branch binary was run against archive nodes of both live networks, on the same data directory and the same node as the baseline binary (dev-upgrade @ cdce8fc5c), and the tracing RPCs were compared on the same blocks.

Mainnet block 37,849,457 (0x2418971, 204 transactions, outside the receiver fork window, contains sign transactions):

probe baseline this branch
debug_intermediateRoots 94 roots for a 204-transaction block (truncated at the failure) 204 roots, last root == block stateRoot

Mainnet block 39,083,312 (0x2545d30, 14 transactions, inside the window; tx 0 goes to 0x92, tx 5 to 0x90 from the same sender with the same nonce) and Apothem block 48,667,667 (0x2e69c13, 8 transactions; tx 0 to 0x92, tx 5 to 0x89 from the same sender with the same nonce):

probe baseline this branch
debug_intermediateRoots 13 / 7 roots 14 / 8 roots, last root == block stateRoot

Mainnet block 107,084,658 (0x661FB72, 6 transactions, after the receiver-disable fork, no system-address transaction, baseFeePerGas 12.5 Gwei): only debug_intermediateRoots differs — the baseline last root is not the block stateRoot, this branch's is.

Manual test plan: start a node on that data directory with --rpcapi debug and call debug_intermediateRoots on 0x2418971, 0x2545d30 and 0x2e69c13; the baseline binary truncates or diverges on every one of them and this branch passes them.

Regression: on both networks this branch imported testnet and mainnet segments normally, with no bad block, no panic and no error attributable to the change.

Types of changes

  • fix: A bug fix
  • refactor: A code change that neither fixes a bug nor adds a feature
  • test: Adding missing tests or correcting existing tests
  • build / ci / chore / docs / feat / perf / revert / style

Impacted Components

  • Geth
  • Not sure (the change is the replay inside the debug_intermediateRoots JSON-RPC method)
  • Consensus
  • Account
  • Network
  • Smart Contract
  • External components

Checklist

  • This PR has sufficient test coverage (unit/integration test)
  • Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet (see "End-to-end verification")
  • Tested the backwards compatibility — the change only affects a replay; block processing keeps the same conditions, finalisation and nonce handling, and no API, RPC method or state format changes
  • Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs — not done for this branch. It was run against live testnet and mainnet archive nodes instead (segments imported, no bad block, no panic, no new error), and the tracing RPCs were compared block by block against the baseline binary.
  • Tested with XDC nodes running this version co-exist with those running the previous version — the branch ran on the live networks next to the rest of the network without being dropped
  • Relevant documentation has been updated as part of this PR

Relation to other work

This is the second of three PRs that split a single debug tracing series. It depends on the replay entry point #2589 adds (core.ApplyTransactionForReplay, together with the shared routing it follows); with that PR merged, this branch rebases trivially.

The third (#2591) fixes the traces and intermediate roots of the block that activates TIPSigning, which did not remove the legacy block signers account block processing removes before the first transaction.

To keep reviewability, this PR keeps its test in a file of its own (eth/tracers/api_intermediate_roots_test.go) instead of appending to the shared eth/tracers/api_test.go, so it does not overlap the other two PRs textually.

#2581 fixes the unconditional skip in traceBlock and in the state feeder of the JS tracer path — the nonce too high and the null hole in debug_traceBlock*. It carries its own copy of the skipNonceForkCases / newSkipNonceBackend fixtures, which are introduced by the first PR of this series, because its block-level tests build on them.

ApplyTransactionWithEVM decided inline which handler a transaction goes
through, so any caller that has to replay a transaction had to re-derive that
decision on its own — and every copy drifts from block processing the moment
the routing changes.

Pull the decision into routeTransaction, the per-transaction finalisation into
finaliseTxState, and the sender nonce handling of ApplySignTransaction into
applySignTransactionNonce, so a replay can follow exactly the same path. The
log both non-EVM handlers record moves into addNonEVMTxLog for the same reason:
it is not a receipt artefact, StateDB.AddLog advances the block wide log count
that every later log takes its index from, so a copy that skipped it would make
the following transactions report indexes the chain never had.

Also add core.ApplyTransactionForReplay, the entry point a replay should use
instead of core.ApplyMessage: it shares the routing above, records the same log
through the same helper, and skips the receipt and its bloom, which a replay
does not need. It refuses an EVM that carries a tracer, as it never fires the
OnTxStart/OnTxEnd hooks, and it takes the state to replay on from evm.StateDB,
so the finalisation, the nonce handling and the TRC21 fee handling cannot land
on a state other than the one the execution wrote to. Both refusals are sentinel
errors, so a caller can tell them apart from a transaction failure. The replay
paths converted in the commits that follow go through it.

No behaviour change: the conditions, the finalisation, the nonce handling and
the log are the ones ApplyTransactionWithEVM already used. finaliseTxState takes
the state the EVM executes against rather than the plain *state.StateDB, because
a hooked state reports the balance burnt by self-destructed accounts to the
tracer on Finalise (core/state/statedb_hooked.go); IntermediateRoot forwards
unchanged either way. TestApplyTransactionForReplayKeepsTheNonEVMTxLog drives
the replay and block processing of a transaction to a system address and
compares the logs they record.
@coderabbitai

coderabbitai Bot commented Sep 19, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 229be4a2-5e24-4d59-810a-e68d776d34cd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Automated coverage is missing for the documented post-receiver-fork nonce regression.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Fixes debug_intermediateRoots by replaying transactions through block-processing-compatible routing.

Changes:

  • Preserves roots for native XDCX/sign transactions.
  • Replays fee ownership, nonce handling, and historical balance behavior correctly.
  • Adds regression tests comparing replayed roots and native transaction effects.
File Description
eth/​tracers/​api.go Uses the shared replay transaction path.
eth/​tracers/​api_intermediate_roots_test.go Tests roots against block processing.
core/​state_processor.go Adds shared routing and replay helpers.
core/​state_processor_test.go Tests native replay logs and nonce behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread eth/tracers/api_intermediate_roots_test.go Outdated
debug_intermediateRoots rebuilt the pre-state of a block with
core.ApplyMessage, which knows nothing about the routing block processing
applies, and skipped every transaction sent to the XDCX system addresses
outright. While the receiver fork is active those transactions are handled by
ApplyEmptyTransaction, which neither checks nor increments the sender nonce and
leaves the state untouched, whereas the replay executed them as ordinary EVM
transactions and bumped the nonce; the block fee was credited to the zero
address instead of the coinbase owner, and the historical balance bypass was
not applied either.

It therefore returned one root fewer than the block has transactions, and the
roots it did return were not the ones block processing produces.

Replay through core.ApplyTransactionForReplay, the entry point that shares the
routing with ApplyTransactionWithEVM but skips the receipt, its logs and the
bloom, and stop skipping those transactions: every transaction then gets an
intermediate root, and a nonce-less one leaves the state untouched, so its root
is the previous one.

Add TestIntermediateRootsMatchesBlockProcessing, which compares the roots
against a replay through core.ApplyTransactionWithEVM, pins the last one to the
block root after the block finalisation and fails when the coinbase owner fee
is dropped; the base has no intermediate-roots test, so this one is also the
first to pin the replay instead of only counting its roots.

No corresponding fix exists upstream: geth has no non-EVM transaction concept.
@gzliudan
gzliudan force-pushed the fix-trace-replay-intermediate-roots branch from 7f8d548 to bdda0d9 Compare September 19, 2026 10:32
@gzliudan
gzliudan requested a balanced review from Copilot September 19, 2026 10:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The fix is well tested, but it refactors consensus-sensitive transaction processing and requires final human validation.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants