[MehranMazhar]

Why

Stage was rotated onto new validator keys on 2026-10-29 (clutch-deploy #161). node1 cannot sync across it. Every old block fails with Block author verification failed: expected author <new key>, but found <old key>, about 2,800 times in 30 minutes.

The cause is in verify_block_author. It compared every imported block, sync included, with today's authority list. Blocks signed before a rotation were signed by the old keys, so every one fails. The same happens to any new node. The rotation guide and the rehearsal say history still validates. That is wrong when the authorities change.

This does not heal node1 by itself. node1 was already stuck at about block 144,507 (the real chain passed block 144,508 on 2026-10-23 at 03:30 UTC). When the deploy of 2026-10-29 restarted it at 14:44:56, it made blocks of its own on that old head (144,508 and up, one per minute) before it had synced, because a node does not check if it is behind before it authors. Those blocks are not on the real chain, and a node cannot switch chains. So node1's data must be wiped once and resynced from the other two nodes. With this change and the table below that resync works, and the old keys cannot be used to make a fork while a node is below block 163,015.

What changes

• aura.rs: PastAuthoritySet { until_block, authorities } and Aura.past_sets. A block's set is the first past set whose until_block reaches its height, else the current set. Each set keeps its own slot length (60 / len). With no past sets nothing changes.
• Config: optional [[previous_authorities]] tables, each with until_block (the last block that set authored) and authorities. They go at the end of the TOML file.
• Blockchain::with_previous_authorities, wired in main.rs. It checks the lists at start like authorities (not empty, no duplicates, at most 60), and that they come oldest first.
• Docs: CLAUDE.md, in the Config section.
• CI (own commit, f4a2e13): test.yml runs on ubuntu-24.04 instead of ubuntu-latest. ubuntu-latest became 26.04 in October 2026, and its newer compiler cannot build the pinned librocksdb-sys (RocksDB 8.10 headers lack #include <cstdint>). Earlier green runs restored a cached build, which hid this until the new image changed the cache key. Any other PR to this repo fails the same way until the pin is on main.

What does not change

• Who may author at the head. The head is checked against authorities alone, so a rotated-out key stays dead past until_block.
• Nodes at the head. Only the check of an OLD block reads the past sets, so adding them needs no reset and no coordinated restart. A node without the tables (mainnet today) behaves as before.

Tests

CI is green on f4a2e13 (run 40901110494): 118 unit tests and every integration suite pass. The new tests, by name:

• aura::tests::an_old_block_is_checked_against_the_set_that_authored_it
• aura::tests::without_the_old_set_a_rotated_out_authors_block_is_refused
• aura::tests::the_last_block_of_an_old_set_is_inclusive_and_the_next_one_is_not
• aura::tests::each_of_several_past_sets_covers_its_own_range
• aura::tests::a_past_set_of_another_size_keeps_its_own_slot_length
• aura::tests::a_block_of_an_old_set_is_still_bound_to_real_time
• blockchain::tests::past_authority_sets_must_be_valid_and_oldest_first
• configuration::tests::previous_authorities_default_to_none and previous_authorities_read_as_tables_oldest_first
• a_node_that_missed_the_rotation_syncs_the_blocks_of_the_old_authority (tests/authority_rotation.rs): a node on the new set refuses block 1 from the old key, imports it once the old set is declared, and still refuses the old key at block 2.

Not tried on a real network yet. That is the stage rollout below.

Rollout for stage (after the merge)

A merge builds an image, and stage moves its node pin by itself. Then a clutch-deploy change adds this to the end of config/node-stage/node{1,2,3}.toml:

[[previous_authorities]]
until_block = 163015
authorities = [
    "0x9b6e8afff8329743cac73dbef83ca3cbf9a74c20",
    "0x6fc11ba44483201f6e9c5eba6435805bb94ad080",
    "0xc4f3f661a43e099aedb8e396d9de1a831a1b4adc",
]

Block 163,015 is the last block the old keys authored (explorer: 14:44:40 UTC). Block 163,016 is the first block of the new keys. The order of the two changes does not matter: an image without this change ignores the tables. node1 then needs its data wiped and a resync (see above); about 163,000 blocks at 100 blocks every 10 seconds take several hours.

Follow-up in clutch-deploy: correct docs/AUTHORITY-ROTATION.md and the rehearsal, which claim that history still validates after a rotation.

🤖 Generated with Claude Code