Skip to content

fix(blockchain): refuse a rollback target below the archived block bodies - #1405

Merged
vipwzw merged 4 commits into
33cn:masterfrom
bysomeone:fix-rollback-archive-floor
Oct 1, 2026
Merged

vipwzw merged 4 commits into
33cn:masterfrom
bysomeone:fix-rollback-archive-floor

Conversation

@bysomeone

@bysomeone bysomeone commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1404.

-rollback has no guard against the chunk-archive floor, and a target below it leaves a datadir that cannot be started. On a node whose bodies had been archived up to 578999, -rollback 546819 removed 321190 blocks, then panicked on 578999 with ErrHashNotExist, and the datadir had to be replaced even though everything above the floor was still intact.

The boundary is the startup window, not the deletion loop

The archiver deletes block bodies, and rollback loads every block above its target, so the target has to clear the lowest stored body for the deletion loop to work at all. That is not sufficient: the tip then stands at the target, and the next start reads the bodies above it. InitCache loads [tip-DefCacheSize, tip] (128 by default) and [tip-HighAllowPackHeight-LowAllowPackHeight+1, tip] (600+200-1 = 799 by default, types/const.go) and panics on the first block it cannot load, so the tip has to clear the archived range by the larger of the two windows.

Measured against a real InitCache, with the bodies at or below 999 deleted:

tip=999  -> panic, ErrHashNotExist at height 871
tip=1000 -> panic, ErrHashNotExist at height 872
tip=1128 -> panic (tx-height window)
tip=1799 -> starts

So the lowest target that restarts is the floor plus 799 with the default configuration, which is 800 blocks shallower than the floor alone. Both windows are read from the configuration (DefCacheSize, types.HighAllowPackHeight/LowAllowPackHeight, which chain.go overrides from the module config), and the window only applies when TxHeight is enabled — InitCache returns early without reading anything otherwise.

The floor comes from the chunk records, at the archiver's frontier

Not from ChunkblockNum times the cursor: chunkShardHandle writes each record with the chunk size in effect at the time, and DeleteBlockBody deletes exactly the range that record names. A node whose ChunkblockNum was lowered between two starts would compute a floor that is too low and reopen the gap. The configured size remains the fallback for a datadir that has a cursor but no record.

Nor from the cursor alone. The cursor says where the previous run stopped, and the archiver keeps going in the same process: SetQueueClient starts chunkDeleteRoutine before Rollbackblock runs (util/cli/chain33.go), it fires every minute, and the loop stops DelRollbackChunkNum (10, chunkshard.go) chunks short of the newest archived chunk. A node that never drained its backlog therefore has a cursor well behind the last chunk it is about to delete, and a target the guard had cleared could still be deleted into by the next tick — the same death, one tick later. The floor is taken from the chunk at max(maxDeletedChunk, maxSerialChunkNum - DelRollbackChunkNum - 1), the last chunk the archiver will ever delete, read from that chunk's own record.

A record that cannot be read is not "nothing was archived". BlockStore.GetKey reports a failed read as types.ErrNotFound, so a read failure and a missing record were the same case, and both fell back to the configured chunk size — the bound that is too low exactly when it matters, since ChunkblockNum may have been reduced after the bodies were dropped. Only db.ErrNotFoundInDb falls back now; anything else refuses the rollback and logs the chunk and the error.

The cursor itself still cannot make the floor too low: it advances even when a chunk has no record or a height inside it is skipped, and both cases only raise the derived floor.

Also in here

  • Rollbackblock exits 0 when the target is at or above the tip. That is a no-op, not a refusal; a service manager with Restart=always would otherwise report a failure for it. Refusals stay non-zero, so a refusal still cannot be mistaken for a completed rollback.
  • The cursor is read with its error. GetMaxDeletedChunkNum reported -1 both for "nothing archived yet" and for a failed read, and -1 means "no floor applies" — a safety check must not fail open on a read failure. GetMaxDeletedChunkNumWithErr is added for that caller; the existing method keeps its behaviour for the archiver, which uses the value as "where to resume".

Not done here: making InitCache tolerate a missing body instead of panicking. It is the more robust fix, since any cause that brings the tip near the archive would stop locking the datadir, but it changes startup behaviour for every cause rather than this one and deserves its own change and discussion.

Tests

  • TestRollbackTargetMustLeaveTheStartupWindowIntact builds the archived datadir for real (chunk record written, bodies deleted, cursor set), asserts the guard refuses the height one below the accepted one, and then starts the chain at both heights: the accepted target does not panic in InitCache, the refused one does.
  • TestRollbackFloorFollowsTheArchiverNotTheCursor builds a datadir with two chunks dropped, the cursor still on the first and ten chunks of backlog, and asserts the floor is the second chunk's record (1500), not the cursor's (1000). The pre-change function returned 1000 and cleared a target below the real floor. Fails before, passes after.
  • TestRollbackFloorRefusesAnUnreadableRecord: an undecodable record has to reach the caller instead of the fallback floor. Fails before, passes after.
  • TestRollbackFloorFallsBackToTheConfiguredChunkSize and TestRollbackFloorWithoutTxHeight cover the two branches that had none: the fallback, and TxHeight=false — bityuan's configuration, where InitCache reads nothing and no window applies. These pass before as well; they pin behaviour rather than discriminate.
  • TestNeedRollbackArchiveFloor follows the new boundary instead of the old one.
  • Rollbackblock's exit codes and the -rollback flag help follow the same rule.

go test ./blockchain/ -count=1 passes; gofmt and go vet are clean.

🤖 Generated with Claude Code

…dies

The chunk archiver deletes block bodies at and below
(maxDeletedChunkNum+1)*ChunkblockNum-1. Rollback loads every block above its
target, so a target below that floor fails part way down the deletion loop --
and that failure cannot be recovered from: the tip pointer has already been
moved to a height whose body is gone, so the database can no longer be started.
On a node that had archived up to 578999, rolling back to 546819 removed 321190
blocks and then panicked on 578999 with ErrHashNotExist; that datadir had to be
replaced, even though everything above the floor was still intact.

Check the floor in NeedRollback, where the other "this rollback cannot work"
conditions already live, and refuse with an error naming the target, the floor
and how it was derived. GetMaxDeletedChunkNum reports -1 when nothing has been
archived, in which case no body has been deleted and no floor applies.

Rollbackblock now exits non-zero when it did not roll back, so a caller cannot
mistake a refusal for a completed rollback -- it used to exit 0 either way.

The -rollback flag help now says the target cannot go below the archived bodies
and that the operation cannot be undone.

Fixes 33cn#1404.

Co-authored-by: Claude <[email protected]>
@33cn

33cn commented Sep 28, 2026

Copy link
Copy Markdown
Owner

复查:下界少算了一个"启动窗口",放行的目标仍会让节点起不来

方向是对的,NeedRollback 里加下界、非 0 退出、帮助文本这几处都没问题。但 blockchain/rollback.go:47 的阈值 rollHeight < bodyFloor-1 偏松了 800 个区块:它只保证了回滚删除循环(Rollback() 遍历 (rollHeight, tip])读得到 body,却漏掉了回滚结束后链的 tip 立在 rollHeight 上、下次启动时还要读 tip 下方一大段 body。

代码路径

  • 启动:SetQueueClient → InitBlockChain → InitCache(blockchain/chain.go:296-303、chain.go:515)
  • InitCache 两个循环都是 GetBlock 出错即 panic(chain.go:527-534、chain.go:538-546):
    • loop1(TxHeight 打开时执行):[tip-DefCacheSize, tip],默认 128
    • loop2:[tip-(HighAllowPackHeight+LowAllowPackHeight)+1, tip],默认 600+200 → 799
  • GetBlock → LoadBlock → multiGetBody:高度靠近 tip 时命中"该高度范围应该保存在本地"分支直接 ErrHashNotExist(blockstore.go:1655)。归档只删 body,height→hash 索引还在,所以 GetBlock 必然走过第一步再死在这里。

TxHeight 在默认配置里就是 true(types/defaultcfg.go:11、cmd/chain33/chain33.toml:6;bityuan.toml 为 false),所以这不是边角配置。

实测(PR head 72b0ff5,调用真实 InitCache)

temp leveldb 写入 0..1999 个区块,删除 ≤999 的 body(即 bodyFloor=1000),回滚后按各目标调 InitCache:

tip=999  (本 PR 允许的最深目标) -> InitCache PANIC: getBlock err=ErrHashNotExist, height=871
tip=1000 (bodyFloor 本身)       -> InitCache PANIC: getBlock err=ErrHashNotExist, height=872
tip=1128 (bodyFloor+128)        -> InitCache PANIC: ErrHashNotExist          // txHeight 窗口
tip=1799 (bodyFloor+799)        -> 正常返回,节点可启动

即可重启的最小目标是 bodyFloor + max(DefCacheSize, HighAllowPackHeight+LowAllowPackHeight-1)(默认 bodyFloor+799),而不是 bodyFloor-1。

这也顺带修正了 #1404 的叙事:那次部分回滚后 tip 停在 579000,它自身 body 还在,起不来是因为 578872..578999 的 body 早被归档删掉,重启时倒在 InitCache 里——不是回滚循环,也不是"tip 指向了 body 已丢失的高度"。

建议(二选一,或都做)

  1. 收紧阈值(最小改动,注意这两个常量可被 module config 覆盖,见 blockchain/chain.go:664-665,不要写死 799):
bodyFloor := (maxDeletedChunk + 1) * chain.cfg.ChunkblockNum
lowest := bodyFloor
if chain.client.GetConfig().IsEnable("TxHeight") {
    w := types.HighAllowPackHeight + types.LowAllowPackHeight - 1 // 启动时 txHeight 缓存窗口
    if chain.cfg.DefCacheSize > w {
        w = chain.cfg.DefCacheSize
    }
    lowest = bodyFloor + w
}
if rollHeight < lowest { ... refuse ... }
  1. 让 InitCache 容忍缺失 body(跳过 + warn,而不是 panic)。这更稳:任何原因把 tip 带到归档下界附近(不止 -rollback)都不会再锁死数据库,回滚守卫也不必把一个 799 块的运营窗口写进安全判断。

顺带:blockchain/rollback_test.go:80-100 里"599999 接受、599998 拒绝"把当前这个边界写死了,改阈值时要一起改。

其它几点(次要)

  • 退出码范围超出预期:Rollbackblock 现在对 NeedRollback 的所有 false 分支都 Exit(1),其中 curHeight <= rollHeight(目标 ≥ tip)和 ForkKvmvccmavl 是改动前就有的分支,前者属于良性 no-op。用 Restart=always 的 systemd / 监控会把它当失败。建议区分退出码(0 = 无事可做,非 0 = 拒绝),或在帮助里写明。
  • 守卫是 fail-open:GetMaxDeletedChunkNum 对任何错误都返回 -1(blockstore.go:1802-1813),包括真实读盘错误,而 -1 被当作"未归档、无下界"。安全判断上这是反的方向,建议把 dbm.ErrNotFoundInDb 与其它错误区分开。
  • 下界由当前配置推导:ChunkInfo.Start/End 已存在库里(chunkshard.go:181-188),maxDeletedChunkNum 那个 chunk 的 End+1 才是权威值。两次启动之间改了 chunkblockNum(调小)会让公式算出偏低的 floor,重新打开缺口。
  • MaxDeletedChunkNum 是"最后尝试的 chunk":CheckDeleteBlockBody 里即使 DeleteBlockBody 返回 nil 或块内部分高度被 continue 跳过,游标照样前进(chunkshard.go:99-118)。方向保守(floor 偏高 → 多拒绝),但建议注释里写明,免得以后被按"最后成功删除"理解。
  • 测试缺口:新单测只覆盖判词,且断言的是错误边界;缺的正是"归档过 body 的数据目录 → 回滚 → 重新起链"这条路径。上面那段临时测试的思路可以直接改成集成用例。

认可的部分

守卫放在 NeedRollback 与既有"这次回滚做不成"的判断并列,位置合理;-1(未归档)的处理正确,归档关闭的节点与现有测试行为不变;拒绝日志带全了 target / floor / maxDeletedChunkNum / chunkblockNum 这些推导输入,排障友好;PR 描述坦率标注了 success path 未在真实 datadir 上验证。

建议本条修掉、并把边界单测和一条"归档后回滚并重启"的用例补上再合。

The floor was one chunk too low: it only kept the deletion loop able to read
every body above the target, and missed that the tip then *stands* at the
target, where the next start reads the bodies above it. InitCache loads
[tip-DefCacheSize, tip] and [tip-HighAllowPackHeight-LowAllowPackHeight+1, tip]
and panics on the first block it cannot load, so a target at the floor deletes
happily and then leaves a datadir that cannot start -- 800 blocks of margin
short with the default 128 / 600+200 windows (measured: with the bodies at or
below 999 gone, tip 999 and tip 1000 panic in InitCache, tip 1799 starts).

- Take the floor from the chunk records, not from ChunkblockNum times the
  cursor: chunkShardHandle writes each record with the chunk size in effect at
  the time and DeleteBlockBody deletes exactly the range it names, so lowering
  ChunkblockNum between two starts would otherwise compute a floor that is too
  low. The configured size remains the fallback when no record is stored.
- Add the startup window when TxHeight is enabled, reading both windows from
  the configuration rather than hardcoding the defaults.
- Read the cursor with its error. It reported -1 both for "nothing archived
  yet" and for a failed read, and -1 means "no floor applies" -- a safety check
  must not fail open on a read failure.
- Exit 0 when the target is at or above the tip: that is a no-op, and a service
  manager would otherwise report a failure for it. Refusals stay non-zero.

Tests: TestRollbackTargetMustLeaveTheStartupWindowIntact archives a chunk for
real (record plus deleted bodies), asserts the guard refuses the height one
below the accepted one, and starts the chain at both -- the accepted target
does not panic and the refused one does. Rollbackblock's exit codes and the
flag help follow the same rule.

Co-authored-by: Claude <[email protected]>
@bysomeone

Copy link
Copy Markdown
Collaborator Author

感谢复查,这条抓得很准 —— 我的下界确实少了整个启动窗口。已按建议 1 改完(6b5b9e1),并把你列的次要项一起处理了。

主问题:下界少了 800 块

先在代码上复核了你的路径:InitCache(blockchain/chain.go:515-546)在 TxHeight 打开时读 [tip-DefCacheSize, tip] 与 [tip-HighAllowPackHeight-LowAllowPackHeight+1, tip],任一 GetBlock 失败即 panic;TxHeight 关闭时函数提前 return,两个窗口都不读。默认 128 与 799 取大 → 这就是我漏掉的余量。

改法与你给的伪代码一致,但两个窗口都从配置读、不写死:DefCacheSize 用 chain.cfg.DefCacheSize,窗口用 types.HighAllowPackHeight + types.LowAllowPackHeight - 1(chain.go:664-665 会从 module config 覆盖这两个包级变量),并且只在 cfg.IsEnable("TxHeight") 时叠加。

边界单测按新阈值改了,并补上你要的那条集成用例 TestRollbackTargetMustLeaveTheStartupWindowIntact:真写一条 chunk 记录、真删掉它标定的 body(DeleteBlockBody 只返回 kv,要像生产那样自己写回,这点我第一次写测试时也踩了)、设好游标,然后断言 —— 守卫拒绝"可接受下界 - 1",并且在两个高度上分别真的启动一次链:可接受的那个 InitCache 不 panic,被拒绝的那个 panic。也就是把你实测的那个现象固化成了回归用例。

下界改成取 chunk 记录(你的第 3 条次要项)

你说得对,(cursor+1)*ChunkblockNum 是拿当前配置推历史范围。现在优先读 calcChunkNumToHash(cursor) 那条 ChunkInfo 的 End+1(chunkShardHandle 写入时用的就是当时的 chunk 大小,DeleteBlockBody 删的也正是这条记录标定的范围),读不到记录才回退到配置公式。游标是"最后尝试的 chunk"这一点也写进注释了(按你提醒的语义,方向只会让 floor 偏高,不会偏低)。

次要项

  • 退出码:Rollbackblock 现在先判"目标 ≥ tip"→ 记日志 + Exit(0)(无事可做,避免 Restart=always 的服务被判失败);其余 NeedRollback 为 false 的分支才 Exit(1)。
  • fail-open:新增 GetMaxDeletedChunkNumWithErr,把 dbm.ErrNotFoundInDb("还没归档过")与其它读错误分开;守卫遇到读失败直接拒绝。原方法保留原行为给归档协程用(它把值当"从哪继续")。
  • 帮助文本:-rollback 的说明改成"拒绝节点起不来的目标(归档 body + 启动缓存窗口),拒绝时非 0 退出"。

没有做的:建议 2(让 InitCache 容忍缺 body)

我认为方向对且更稳,但它改的是所有原因导致的启动路径(不只是 -rollback),影响面比这个 PR 大 —— 值得单独一个改动 + 单独讨论,所以先留在外面。已在 PR 描述里写明这是有意不做的。

关于 #1404 的叙事

你补的启动窗口这一层是对的,我采纳了;不过我们那次实际是倒在回滚循环里(rollback.go 的 LoadBlock(578999) panic,ErrHashNotExist),不是重启时才暴露 —— 两条路径都存在,我按更宽的那条(启动窗口)收紧了阈值,PR 描述也按这个改写了。

@bysomeone
bysomeone requested a review from 33cn September 28, 2026 07:46
archivedBodyFloor read the deletion cursor as if it were the end of the archive, but
the cursor only says where the previous run stopped. CheckDeleteBlockBody keeps going
in this process -- SetQueueClient starts the routine before the rollback runs, and it
fires every minute -- and it stops DelRollbackChunkNum chunks short of the newest
archived chunk, so a node with ten chunks of backlog will delete bodies the guard had
already cleared. That is the same failure the guard exists to prevent: the deletion
loop panics on the first missing body, or the next start panics in InitCache.

The floor now comes from the chunk at max(maxDeletedChunk, maxSerialChunkNum -
DelRollbackChunkNum - 1) -- the last chunk the archiver will ever delete -- read from
that chunk's own record, and the record is still what decides the height.

The record read also failed open. Any error, a real read failure included, fell back
to (chunk+1) * ChunkblockNum, and that formula uses the current configuration, which
is too low whenever ChunkblockNum was reduced after the bodies were dropped --
GetKey reports a failed read as types.ErrNotFound, so the two cases were one. Only
db.ErrNotFoundInDb, a datadir from before the records were written, falls back now;
anything else refuses the rollback and logs the chunk and the error.

Tests, against the pre-change function:
- TestRollbackFloorFollowsTheArchiverNotTheCursor: cursor on chunk 0, chunk 1 dropped
  with a different chunk size on record, archiver twelve chunks ahead. The floor has
  to be 1500 (chunk 1's record), not 1000; the old function returned 1000 and cleared
  a target below the real floor. Fails before, passes after.
- TestRollbackFloorRefusesAnUnreadableRecord: an undecodable record must reach the
  caller instead of the fallback floor. Fails before, passes after.
- TestRollbackFloorFallsBackToTheConfiguredChunkSize, TestRollbackFloorWithoutTxHeight:
  cover the two branches that had none, the fallback and the config bityuan runs
  (TxHeight=false, where InitCache reads nothing and no window applies). These pass
  before as well -- they pin behaviour, they do not discriminate.

go test ./blockchain/ -count=1 passes, go vet and gofmt are clean.

Co-authored-by: Claude <[email protected]>
@bysomeone

Copy link
Copy Markdown
Collaborator Author

自查后又补了一个提交(8556841),修的是上界之外的另一个洞:下界取错了参照物。

下界原来读的是"删除游标",而归档协程还在跑

游标(GetMaxDeletedChunkNum)只记录上一次运行删到哪。chunkDeleteRoutine 是 SetQueueClient(chain.go:360)起的,比 util/cli/chain33.go:197 的 Rollbackblock 早,60 秒一 tick,循环条件是 toDelete + DelRollbackChunkNum(10) < maxSerialChunkNum(chunkshard.go:102)。所以没排空积压的节点,游标会明显落后于它接下来还要删的最后一个 chunk——守卫放行的目标,下一 tick 就可能被删进去,和 #1404 是同一种死法,只是晚一个 tick。

现在下界取 max(maxDeletedChunk, maxSerialChunkNum - DelRollbackChunkNum - 1) 那条 chunk 记录(归档器最终会删到的最后一个 chunk),高度仍然由记录本身决定。

记录读失败原来是 fail-open

BlockStore.GetKey 把真实的读错误报成 types.ErrNotFound,缺记录则返回 ErrNotFoundInDb,而原写法 if err == nil { ... } 把两者都归到"回退到 (chunk+1) * ChunkblockNum"——那个公式用的是当前配置,ChunkblockNum 被调小过就偏低,正是最需要它准的时候不准。现在只有 ErrNotFoundInDb(记录写入之前的旧 datadir)回退,其它错误一律拒绝回滚并把 chunk 与错误打进日志。

用例(都对着改动前的函数跑过)

  • TestRollbackFloorFollowsTheArchiverNotTheCursor:造出"两个 chunk 已删、游标还停在第一个、归档器已在第 12 个 chunk"的现场,断言下界取第 2 个 chunk 的记录(1500)而不是游标的(1000)——改动前返回 1000,会放行一个下界以下的回滚;改动前失败、改动后通过。
  • TestRollbackFloorRefusesAnUnreadableRecord:记录解不开时必须把错误交出来,而不是用回退值。改动前失败、改动后通过。
  • TestRollbackFloorFallsBackToTheConfiguredChunkSize、TestRollbackFloorWithoutTxHeight:补上原先零覆盖的两个分支(回退分支,以及 TxHeight=false —— 比特元的生产配置,InitCache 不读任何 body、不叠加窗口)。这两条改动前也通过,属于钉行为而非判据。

go test ./blockchain/ -count=1 通过(164s),gofmt/go vet 干净。

@vipwzw
vipwzw requested review from vipwzw and removed request for 33cn October 1, 2026 03:52
@vipwzw
vipwzw merged commit 4d4e8c1 into 33cn:master Oct 1, 2026
13 checks passed
@33cn

33cn commented Oct 1, 2026

Copy link
Copy Markdown
Owner

🎉 This PR is included in version 1.72.4 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@33cn 33cn added the released label Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rollback: no guard against the chunk-archive floor -- a deep rollback panics and leaves the datadir unstartable

3 participants