Repository navigation
Keepword 2.1.0: fix the TLSNotary hang, time out stalled sessions - #16
Merged
Merged
Conversation
About one MPC-TLS session in six stalled while the two witnesses set up MPC-TLS, which hung the tlsn CI job in 4 of 84 attempts. The stall is inside TLSNotary: the pinned revision predates upstream's multiplexer fixes and new executor. Build against TLSNotary's main (2d7a627), where none of 800 sessions hung. Its multiplexer speaks a newer protocol, so a notarization needs both witnesses on 2.1.0. Nothing in keepword-tlsn timed out either. Give a session ten minutes and a request 30 seconds on both sides, abort a session's tasks when its future is dropped so it closes its connection, close gracefully after a refusal so the prover still hears why, and keep the verifier accepting when it runs out of file descriptors instead of exiting. Cap the CI job at 30 minutes.
Under tokio's paused clock the runtime jumps to the next timer whenever it waits, even on I/O, so the time a connection took to close read as about a day whatever it was: the timeout tests passed even if the connection closed at once. Check instead that it is still open just before the timeout runs out, then that it closes, and fail rather than hang if a capture never gives up. The verifier slept a second after any accept error, so a connection reset before it was accepted paused it too. Skip those without the pause, as axum::serve does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Aelthorim · project thread
The TLSNotary test has hung in 4 of 84 CI attempts since 2026-09-29, until each was cancelled: run 53, the v1.2.0 tag push (run 56), and both attempts of #15 (run 82). This fixes the hang, makes notarization time out instead of waiting forever, and keeps the verifier serving when it runs out of file descriptors. It goes into 2.1.0, which has no release yet.
The hang
Before: about one MPC-TLS session in six stalled while the two witnesses set up MPC-TLS (10 of 60 runs of
notarized_capture_verifieshere). Every thread went idle and a sub-task on the prover never sent its first message, so the test waited forever. The stall is inside TLSNotary. The revision we pinned (b9b2e9e) uses tlsn-utils' stream multiplexer from April, from before upstream fixed several ways it could hang: lost driver wakeups that could deadlock it under parallel pressure (tlsn-utils PR 105), and data a peer sent on a stream before the local side opened it being dropped, so the late opener waited forever (PR 111). It also used mpz's old executor, which started a thread per CPU core for every session.After: keepword-tlsn builds against TLSNotary's main (2d7a627, 2026-09-30), which takes those fixes, one more (PR 122) and a new executor (tlsn PR 1188). None of 800 runs hung: 500 in a row at 1.6 s each, then 300 more three at a time. Nothing in our code had to change for it, and sessions now share one thread pool.
It also hung between two witness processes over TCP, not just in the test: with two copies of a scratch test, one serving and one capturing, the old revision hung in 4 of 14 sessions and the new one in none of 11.
Needs both witnesses on 2.1.0. The new multiplexer opens a stream with a SYN flag on its first frame, and ignores frames for streams that weren't opened that way. The old one never sends SYN, and ends the connection with a protocol error when it gets one. Both revisions report version 0.1.0-alpha.16-pre, so TLSNotary's own version check doesn't catch it. Tried both ways with the scratch test: a 2.1.0 prover fails within milliseconds against a 2.0.0 verifier ("commitment protocol failed to receive response caused by: unexpected end of file"), and a 2.0.0 prover hangs on a 2.1.0 verifier (2.0.0 has no timeouts), which fails at once ("failed to receive request"). The changelog says so in the 2.1.0 intro and the entry. Receipts and bundles are unaffected: their format and checks live in
keepword-core.No timeouts
Before: nothing in keepword-tlsn timed out. A verifier that went quiet left
keepword-tlsn capturewaiting forever, and a prover that went quiet, or a connection that never sent its hello, held a verifier's connection and session for as long as it liked. The session's tasks were detached, so after an error or a refusal the connection also stayed open for as long as the peer kept it.After: either side gives up on a session after ten minutes, and on a request (a hello, a receipt lookup) after 30 seconds. A session's tasks are aborted when its future is dropped, so a session that times out or fails closes its connection. When the verifier refuses a prover (limits, proxy mode, a hidden server identity), it closes the session gracefully first, so the reason still arrives: without that, 24 of 30 refused provers saw a dropped connection instead.
A flood of connections stopped the verifier
Before:
VerifierService::servereturned on any accept error, so once the process ran out of file descriptors (about a thousand open connections with the usual limit of 1 024),keepword-tlsn serveexited. Anyone who could reach the port could do that, before any hello.After: it logs the error, waits a second and accepts again. With 64 descriptors and 100 idle connections, the scratch verifier above exited before this change (the next capture was refused); after it, it logged "Too many open files" three times, and the next capture succeeded. A connection reset before it was accepted is skipped without the wait, as
axum::servedoes.CI also gives the tlsn job 30 minutes, so a hang can't hold a runner for six hours.
Testing
crates/keepword-tlsn:cargo fmt --check,cargo clippy --release --all-targetswith-D warnings, andcargo test --release.notarized_capture_verifies, same machine (4 cores), 40 s per run: 10 of 60 runs hung on b9b2e9e; on 2d7a627, 0 of 500 in a row and 0 of 300 run three at a time.