turbine: accelerate 32+32 FEC generation and recovery

[7layermagik]

Summary

Reduce CPU overhead in the leader's entry-batching and shred-generation path, and add specialized recovery for exactly one missing data shred in a standard 32+32 FEC set.

Against the current alpenglow-dev head, the serial producer benchmark processes 50,000 maximum-size transactions in 282 ms instead of 734 ms: 2.60× faster on one pinned Zen 5 CPU. The dev baseline already has incremental batch-size accounting; this comparison measures the remaining benefit against that implementation.

Changes

• Track canonical encoded transaction sizes independently of transport wire-size hints, and reuse the tracked size at flush.
• Send generated packets and their already-computed FEC roots directly through BroadcastSession, eliminating owning shred parsing, payload copies, and root reconstruction.
• Pool the fixed 32+32 Reed-Solomon encoder and write generated Merkle proofs in place.
• Use a 61,632-byte producer batch target and set DATA_COMPLETE_SHRED only at the end of a serialized component spanning multiple FEC sets.
• Recover exactly one missing data shred using precomputed GF(256) coefficient rows. Production dispatch requires 32 data + 32 coding shards, exactly one missing data shard, and an available coding shard. Other patterns retain ReconstructSome; reduced multi-missing and all-coding implementations remain experiments.
• Add a deterministic repair simulator using production validation, spooling, repair selection, recovery, and completion. Build initial packet lists once and process natural late arrivals when repair is disabled.

The public parsed-shred API remains available. Repair wire format, request signing, peer selection, and scheduler policy are unchanged.

Benchmark against current alpenglow-dev

Baseline: alpenglow-dev at 7e4e8af1, verified as the branch head on September 6, 2026. It uses incremental size accounting and a 30,816-byte batch target.

Candidate: the PR producer at a598cd05, using its 61,632-byte default. Commit a3b16eba adds the identical standalone benchmark and results without further production-code changes.

Machine and method: AMD Ryzen 7 9700X (Zen 5), Linux amd64, Go 1.26.4, CGO_ENABLED=0, GOAMD64=v1, GOMAXPROCS=1, pinned to logical CPU 6. Results are medians of five alternating samples per arm, each timing one complete workload. The host was shared, with SMT and boost enabled.

50,000-transaction workload	Current alpenglow-dev	This PR	Speedup	Time reduction
1,232-byte transactions, three slots	734.052 ms	282.296 ms	2.60×	61.54%
215-byte transfers, one slot	150.339 ms	71.442 ms	2.10×	52.48%

Control: same batch target on both versions

With both versions using 30,816-byte batches, the gains remain:

50,000-transaction workload	Current alpenglow-dev	PR with 30,816-byte target	Speedup
1,232-byte transactions, three slots	734.052 ms	288.378 ms	2.55×
215-byte transfers, one slot	150.339 ms	72.424 ms	2.08×

Most of the measured improvement survives at the dev head's one-FEC target. Increasing the PR's target to two FECs adds about 2.1% and 1.4% time reduction in these samples. This does not establish the best target for live latency or pipelined throughput.

Workload and scope

Maximum-size transactions are signed legacy transfers with a memo, exactly 1,232 bytes each. The 512-fixture pool is reused, with structural sanitization, signature verification, and canonical byte round trips performed before timing.

The maximum-size workload uses 16,667 + 16,667 + 16,666 transactions across three slots on both versions to fit dev's slot limits. The harness asserts the 20 MiB-minus-reserved-ending-tick entry-byte bound, the 32,768-data-shred cap, and transaction/batch/packet counts.

For this workload, allocated bytes decrease from 2.129 GB to 708.003 MB, and allocations from 3,844,267 to 974,514. The same-target control produces identical batch and packet counts on both heads. With its larger default, the PR emits slightly fewer packets through denser packing.

Included in timing: entry construction and flushes, serialization, erasure coding, Merkle proofs, shred signing, headers/footers/ending ticks, chained roots, and block-ID bookkeeping. The sink only counts emitted packets.

Excluded: fixture setup, transaction execution, admission signature verification, scheduling/reservation policy, asynchronous shred-worker queueing/overlap, peer routing, UDP, and receiver work. These results measure serial producer CPU-stage work, not whole-validator throughput, executed-block latency, or the wait for a batch to fill.

The comparison uses the two branch heads directly. It does not measure this patch rebased onto dev. Dev's newer slot accounting and asynchronous shred worker must be preserved when integrating these optimizations; the PR's target branch remains 7layer/narya-integration.

Standalone benchmark source, exact commits, raw samples, allocation counts, and reproduction steps.

Correctness and validation

Coverage includes:

• All 32 missing-data positions × 32 coding-row choices compared with the general decoder, independent matrix-inverse checks, and invalid-buffer failure atomicity.
• Stable packet fixtures, multi-FEC component round trips, canonical legacy/v0 batch sizes, and resets after automatic and explicit flushes.
• Generated FEC roots checked against packet proofs; direct broadcast packets, indices, chained root, and block ID checked against the parsed-shred path.
• Deterministic simulator traces and natural late arrivals with repair disabled.

Recovery derivation and differential-test methodology.

The production changes passed on a clean snapshot, excluding unrelated local experiments:

• go test -race -count=1 ./pkg/repair ./pkg/turbine/... ./pkg/costmodel ./pkg/blockprod ./cmd/repair-sim
• go vet ./pkg/repair ./pkg/blockprod ./pkg/costmodel ./pkg/turbine/... ./cmd/repair-sim
• Targeted Linux/Zen 5 packet, root, block-ID, and entry-size checks.
• Before/after simulator logical-trace comparisons.
• git diff --check.

The branch-comparison fixtures passed on both Zen 5 binaries. All 30 measured samples passed the count/budget assertions. The added fixture test also passed locally with the race detector, and blockprod vet passed.

A prior whole-repository go test ./... run was not globally green because of unavailable conformance/genesis fixtures and existing lightbringer/sealevel failures.