Skip to content

[WIRE-415] OPP: bound the outbound attestation queue - #696

Merged
heifner merged 8 commits into
masterfrom
fix/wire-415-bounded-outbound-queue
Oct 26, 2026
Merged

heifner merged 8 commits into
masterfrom
fix/wire-415-bounded-outbound-queue

Conversation

@heifner

@heifner heifner commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes WIRE-415. buildenv walked every READY attestation of every outpost before applying its byte budget, and every desyndicate adds a READY row paid by sysio, with no limit on how many. A large enough backlog could keep the rosters an outpost needs out of the envelope, or exhaust sysio.epoch::advance, which runs buildenv inline.

  • A byqueue index orders attestations by outpost, status and lane, then id. It replaces bystatus, and the unread bytype and byepoch go with it. OPERATORS and BATCH_OPERATOR_GROUPS form a schedule lane that ships first; every other type keeps its queueout order.
  • buildenv walks only its outpost's READY rows and stops at MAX_ENVELOPE_BYTES or at MAX_ENVELOPE_ATTESTATIONS (64), so its work no longer grows with the backlog. It erases what it ships instead of marking it PROCESSED first; the envelope itself stays in outenvelopes until the outpost acknowledges it ([WNS-18] opp: erase an outbound envelope only once the outpost has consumed it - WIRE-348 #592). Its inbound cleanup reads only that outpost's PROCESSED rows.
  • queueout caps a general-lane row at MAX_GENERAL_ATTESTATION_DATA_BYTES (7,656), what fits beside a full schedule lane (SCHEDULE_LANE_BYTES, 24 KiB; the largest window's rosters encode to 6,806 bytes at [WNS-56] OPP: bound the operator roster to the schedule window and cap queueout #692's 8 × 7 ceiling, and 15,446 even at Solana's 8 × 16). Every row the walk reaches therefore ships, and none can stall the lane behind it.
  • queueout refuses a general-lane row once the outpost already holds MAX_QUEUED_ATTESTATIONS (512), counted in a new outqueue table. A refused desyndicate reverts whole, burn and fee included, and the holder retries after a later epoch ships some. Schedule rows are never refused, since advance queues them.
  • Nothing is dropped: a READY DESYNDICATE_LIQ is a payout whose LIQ is already burned. The bounded walk drains a backlog one budget at a time.

Notes

Tests

  • buildenv_ships_the_schedule_lane_ahead_of_a_backlog: OPERATORS and BATCH_OPERATOR_GROUPS queued behind a backlog ship first, and the backlog keeps its order.
  • buildenv_stops_at_the_attestation_cap: 67 rows ship as 64, then 3, in order.
  • buildenv_ships_only_its_own_outposts_rows: one outpost's buildenv leaves another's rows and count alone.
  • buildenv_erases_only_its_outposts_processed_rows: two outposts accept an envelope with one attestation per lane; one outpost's buildenv erases its own inbound rows and leaves the other's, which that outpost's own build then erases.
  • the_largest_windows_schedule_fits_its_lane_budget: an 8 × 7 window with the longest names and both links encodes within SCHEDULE_LANE_BYTES.
  • queueout_caps_general_rows_to_fit_beside_the_schedule and buildenv_ships_a_general_row_at_its_cap_beside_a_full_schedule: a general row one byte over its cap is refused while a schedule row of that size is not, and a general row at its cap ships beside a full schedule.
  • queueout_refuses_general_rows_past_the_queue_cap: the 513th general row is refused, schedule rows and another outpost are not, and shipping 63 rows frees exactly 63 places.
  • desyndicate_reverts_whole_when_the_outbound_queue_is_full: the holder keeps the shadow; nothing burns and no returns row is written.

contracts_unit_test passes 961/961.

The OPERATORS roster listed every sysio.opreg row, and regoperator is self-authorized, so anyone with accounts could grow it until it no longer fit an envelope. buildenv then aborted inside sysio.epoch::advance, and every later advance failed the same way (CertiK WNS-56, WIRE-406).

Outposts read the roster only to map the window's members to the keys they deliver with, so finishadv now lists just the operators the window seats, each with its current status. setconfig caps the window at Solana's limits, 8 groups of 16, which bounds the roster at 128 entries.

queueout rejects data larger than an otherwise empty envelope can carry, so the producer's transaction fails instead of a later buildenv. buildenv's single-attestation checks are now unreachable invariants.
The docs still described OPERATORS as the full roster and cited the old 100/255/1000 setconfig ceilings. They now describe the roster as the operators the window seats, the ceilings of 8 groups of 16, and queueout's size cap.

The pay cadence notes assumed a 21-operator group, which setconfig now rejects. With the default 7-operator groups the 10-epoch history bound limits the cadence; a 16-operator group is held to 6 by the 100-credit cap.

Comments and docs only; the rebuilt contracts are byte-identical.
buildenv walked every READY attestation of every outpost, and every desyndicate adds one paid by sysio with no limit on the count. A large enough backlog could keep the rosters out of the envelope or exhaust sysio.epoch::advance, which runs buildenv inline (WIRE-415).

A byqueue index orders the attestations table by outpost, status and lane, replacing the status, type and epoch indexes. OPERATORS and BATCH_OPERATOR_GROUPS form a schedule lane that ships first; every other type keeps its queueout order.

buildenv walks only its outpost's READY rows and stops at MAX_ENVELOPE_BYTES or MAX_ENVELOPE_ATTESTATIONS (64), erasing what it ships. Its inbound cleanup reads only that outpost's rows.

queueout caps a general-lane row at what fits beside a full schedule lane (7,656 bytes) and refuses one once the outpost already holds 512, counted in a new outqueue table. A refused desyndicate reverts whole, burn included. Schedule rows are never refused, since advance queues them.

The index change needs fresh state; docs/contract-upgrade-order.md says so.
huangminghuang
huangminghuang previously approved these changes Oct 22, 2026

@huangminghuang huangminghuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the #696 delta (2dcac2f4..cf356755). The Ethereum 128-operator gas-limit issue is tracked on base PR #692 and remains blocking there.

`sysio.epoch::setconfig` now takes at most 7 operators per group, Ethereum's `OPERATORS_PER_GROUP`, instead of Solana's 16. Ethereum applies the OPERATORS roster in one handler call, and at 8 x 16 that call runs out of gas (`OPP_HandlerGasExhausted`, HM's review). At 8 x 7 the roster is at most 56 entries, and on wire-ethereum `next` a roster with 49 operators it has never seen dispatches in one transaction: 9.3M gas, 12.4M with the outbound queue full, of EIP-7825's 16,777,216 (WIRE-342).

At most 7 operators over 10 epochs is 70 payout credits, so the history bound is the cadence limit and the 100-credit cap stays only as a backstop. The emissions tests now cover that bound and the group-size ceiling, and the legacy-cadence recovery runs at 7 per group.
#692 caps groups at 7, so the lane test now mirrors 7 and builds the largest window at 8 x 7: 6,806 bytes, down from 15,446 at 8 x 16. `SCHEDULE_LANE_BYTES` stays 24 KiB, which still holds a window at Solana's 16 per group.
@heifner
heifner force-pushed the fix/wns-56-bounded-roster-and-queueout branch from 139ec03 to 0e82d29 Compare October 23, 2026 20:31
Base automatically changed from fix/wns-56-bounded-roster-and-queueout to master October 24, 2026 18:35
@heifner
heifner dismissed huangminghuang’s stale review October 24, 2026 18:35

The base branch was changed.

Brings in #692 (merged as a62f42c, superseding this branch's copies of its commits), #592, #659, #658, #651 and the rest of master.

- `buildenv` keeps #592's retention: shipped rows are erased once packed, the chain links come from the newest `byoutepoch` row, and the acknowledged-prefix sweep runs before the inbound PROCESSED sweep.
- `shipped_attestations` reads the newest `outenvelopes` row, since unacknowledged emits now stay, and the new readers decode with `fixture_abi_yield_function()` after #684's sweep.
- Two locals in `buildenv` are renamed so they no longer shadow (`by_queue`, `inbound`).
- `sysio.msgch.wasm` rebuilt.
HM asked for an on-chain regression of the PROCESSED range. Both outposts accept an envelope with one attestation per lane; ETH's `buildenv` erases ETH's two inbound rows and leaves SOL's, and SOL's own build through `advance` then erases SOL's.

Change-Id: Id5722a6d44d6d446496119403d7484c7748983df

@huangminghuang huangminghuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed current head 01b5880 after the master merge and follow-up. The new two-outpost/two-lane regression covers the previously missing inbound PROCESSED cleanup and isolation path. The exact-head contracts build and 27 focused JIT cases pass, generated ABI/WASM match, and I found no blocking issues.

@heifner
heifner merged commit c3bbb61 into master Oct 26, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants