Skip to content

Keepword 2.1.0: fix the TLSNotary hang, time out stalled sessions - #16

Merged
Aelthorim merged 3 commits into
mainfrom
claude/project-thread-n5ghwx
Oct 10, 2026
Merged

Aelthorim merged 3 commits into
mainfrom
claude/project-thread-n5ghwx

Conversation

@Aelthorim

@Aelthorim Aelthorim commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Requested by Aelthorim · project thread

The TLSNotary test has hung in 4 of 84 CI attempts since 2026-09-29, until each was cancelled: run 53, the v1.2.0 tag push (run 56), and both attempts of #15 (run 82). This fixes the hang, makes notarization time out instead of waiting forever, and keeps the verifier serving when it runs out of file descriptors. It goes into 2.1.0, which has no release yet.

The hang

Before: about one MPC-TLS session in six stalled while the two witnesses set up MPC-TLS (10 of 60 runs of notarized_capture_verifies here). Every thread went idle and a sub-task on the prover never sent its first message, so the test waited forever. The stall is inside TLSNotary. The revision we pinned (b9b2e9e) uses tlsn-utils' stream multiplexer from April, from before upstream fixed several ways it could hang: lost driver wakeups that could deadlock it under parallel pressure (tlsn-utils PR 105), and data a peer sent on a stream before the local side opened it being dropped, so the late opener waited forever (PR 111). It also used mpz's old executor, which started a thread per CPU core for every session.

After: keepword-tlsn builds against TLSNotary's main (2d7a627, 2026-09-30), which takes those fixes, one more (PR 122) and a new executor (tlsn PR 1188). None of 800 runs hung: 500 in a row at 1.6 s each, then 300 more three at a time. Nothing in our code had to change for it, and sessions now share one thread pool.

It also hung between two witness processes over TCP, not just in the test: with two copies of a scratch test, one serving and one capturing, the old revision hung in 4 of 14 sessions and the new one in none of 11.

Needs both witnesses on 2.1.0. The new multiplexer opens a stream with a SYN flag on its first frame, and ignores frames for streams that weren't opened that way. The old one never sends SYN, and ends the connection with a protocol error when it gets one. Both revisions report version 0.1.0-alpha.16-pre, so TLSNotary's own version check doesn't catch it. Tried both ways with the scratch test: a 2.1.0 prover fails within milliseconds against a 2.0.0 verifier ("commitment protocol failed to receive response caused by: unexpected end of file"), and a 2.0.0 prover hangs on a 2.1.0 verifier (2.0.0 has no timeouts), which fails at once ("failed to receive request"). The changelog says so in the 2.1.0 intro and the entry. Receipts and bundles are unaffected: their format and checks live in keepword-core.

No timeouts

Before: nothing in keepword-tlsn timed out. A verifier that went quiet left keepword-tlsn capture waiting forever, and a prover that went quiet, or a connection that never sent its hello, held a verifier's connection and session for as long as it liked. The session's tasks were detached, so after an error or a refusal the connection also stayed open for as long as the peer kept it.

After: either side gives up on a session after ten minutes, and on a request (a hello, a receipt lookup) after 30 seconds. A session's tasks are aborted when its future is dropped, so a session that times out or fails closes its connection. When the verifier refuses a prover (limits, proxy mode, a hidden server identity), it closes the session gracefully first, so the reason still arrives: without that, 24 of 30 refused provers saw a dropped connection instead.

A flood of connections stopped the verifier

Before: VerifierService::serve returned on any accept error, so once the process ran out of file descriptors (about a thousand open connections with the usual limit of 1 024), keepword-tlsn serve exited. Anyone who could reach the port could do that, before any hello.

After: it logs the error, waits a second and accepts again. With 64 descriptors and 100 idle connections, the scratch verifier above exited before this change (the next capture was refused); after it, it logged "Too many open files" three times, and the next capture succeeded. A connection reset before it was accepted is skipped without the wait, as axum::serve does.

CI also gives the tlsn job 30 minutes, so a hang can't hold a runner for six hours.

Testing

  • In crates/keepword-tlsn: cargo fmt --check, cargo clippy --release --all-targets with -D warnings, and cargo test --release.
  • Hang rate of notarized_capture_verifies, same machine (4 cores), 40 s per run: 10 of 60 runs hung on b9b2e9e; on 2d7a627, 0 of 500 in a row and 0 of 300 run three at a time.
  • New tests: a refused prover hears why; a verifier hangs up on a connection that never sends its hello, and on a prover that goes quiet in the session; a capture gives up on a quiet verifier and hangs up on it. The last two run on tokio's paused clock, so they take milliseconds, and check that the connection is still open just before each timeout runs out and closes after it. Each fails without its fix: taking out abort-on-drop, the request timeout or the session timeout, or shortening either timeout, fails one of the last two; taking out the graceful close failed the first in 24 of 30 runs. All four together passed 100 runs in a row.
  • The last two passed 5,500 runs under load, with every core busy and two notarizations running alongside.

About one MPC-TLS session in six stalled while the two witnesses set up
MPC-TLS, which hung the tlsn CI job in 4 of 84 attempts. The stall is
inside TLSNotary: the pinned revision predates upstream's multiplexer
fixes and new executor. Build against TLSNotary's main (2d7a627), where
none of 800 sessions hung. Its multiplexer speaks a newer protocol, so a
notarization needs both witnesses on 2.1.0.

Nothing in keepword-tlsn timed out either. Give a session ten minutes
and a request 30 seconds on both sides, abort a session's tasks when its
future is dropped so it closes its connection, close gracefully after a
refusal so the prover still hears why, and keep the verifier accepting
when it runs out of file descriptors instead of exiting. Cap the CI job
at 30 minutes.
Under tokio's paused clock the runtime jumps to the next timer whenever
it waits, even on I/O, so the time a connection took to close read as
about a day whatever it was: the timeout tests passed even if the
connection closed at once. Check instead that it is still open just
before the timeout runs out, then that it closes, and fail rather than
hang if a capture never gives up.

The verifier slept a second after any accept error, so a connection
reset before it was accepted paused it too. Skip those without the
pause, as axum::serve does.
@Aelthorim Aelthorim closed this Oct 10, 2026
@Aelthorim
Aelthorim deleted the claude/project-thread-n5ghwx branch October 10, 2026 13:05
@Aelthorim
Aelthorim restored the claude/project-thread-n5ghwx branch October 10, 2026 13:05
@Aelthorim Aelthorim reopened this Oct 10, 2026
@Aelthorim Aelthorim self-assigned this Oct 10, 2026
@Aelthorim
Aelthorim merged commit f41cf5c into main Oct 10, 2026
11 checks passed
@Aelthorim
Aelthorim deleted the claude/project-thread-n5ghwx branch October 11, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant