amethyst/quic/interop/plans/2026-05-06-interop-runner.md

quic-interop-runner endpoint — Phase 0 scaffolding

Date: 2026-05-06

Why

We want the quic-interop-runner matrix as a bug-finding harness, not a vanity scoreboard. Each peer impl (quiche, aioquic, picoquic, ngtcp2, msquic, mvfst, lsquic, neqo, kwik) enforces different parts of RFC 9000/9001/9002/9114 strictly, so failures triangulate to specific bugs in :quic. The runner's ns-3 sim also exposes loss / reorder / migration scenarios that are awkward to reproduce in unit tests.

What's in Phase 0

• :quic-interop Gradle module (JVM-only application, registered at quic/interop/ via settings.gradle).
• InteropClient.kt reads the runner's env-var contract (ROLE, TESTCASE, REQUESTS, …) and dispatches by testcase. Phase 0 implements only handshake; everything else returns 127 (runner-skip).
• Dockerfile based on martenseemann/quic-network-simulator-endpoint + OpenJDK 21 runtime, copies the installDist output.
• run_endpoint.sh sources the base image's /setup.sh then execs our JVM binary.
• Makefile wrappers: make build, make clean. (A make smoke target previously stood up picoquic + our endpoint on a private Docker bridge to bisect runner failures from impl failures; dropped once the runner reliably exercised both paths.)

Local iteration loop

The fast path: quic/interop/run-matrix.sh clones the runner alongside this repo, sets up a venv, merges our implementations_quic.json snippet, builds the endpoint image, and invokes run.py. All steps are idempotent so repeated invocations just iterate.

# Single test against the most permissive peer:
quic/interop/run-matrix.sh -s aioquic -t handshake

# A focused triangulation:
quic/interop/run-matrix.sh -s aioquic   -t handshake,chacha20
quic/interop/run-matrix.sh -s quic-go   -t handshake,chacha20
quic/interop/run-matrix.sh -s picoquic  -t handshake,chacha20

# Tight inner loop — skip the image rebuild between test selections:
SKIP_BUILD=1 quic/interop/run-matrix.sh -s aioquic -t transfer

Manual flow (if run-matrix.sh doesn't fit):

make -C quic/interop build
# Then in a sibling clone of quic-interop-runner, merge our snippet:
jq -s '.[0] * .[1]' implementations_quic.json \
   ../amethyst/quic/interop/quic-interop-runner-snippet.json \
   > implementations_quic.json.new && mv implementations_quic.json.new implementations_quic.json
python run.py -d -i amethyst -s aioquic -t handshake,chacha20 --log-dir ./logs

Inspect ./logs/<run>/client_qlog/*.qlog in qvis when something breaks.

macOS gotcha — upstream certs.sh patch (auto-applied)

run-matrix.sh step 1a rewrites LC_CTYPE=C tr → LC_ALL=C tr in the upstream certs.sh. macOS BSD tr chokes on /dev/urandom with tr: Illegal byte sequence when only LC_CTYPE is set; LC_ALL wins in the locale-precedence chain and silences it. Without this, the amplificationlimit testcase aborts at cert generation, and every test sequenced after it in TESTCASES_QUIC (handshakeloss, transferloss, handshakecorruption, transfercorruption, ipv6, v2, rebind-port, rebind-addr, connectionmigration, plus the goodput/crosstraffic measurements) silently never runs. The matrix prints the post-mortem summary showing only the 12 testcases that ran before the abort, which looks deceptively complete.

The patch is idempotent and is re-applied on every clone of the runner (the fix is a working-tree edit to ../quic-interop-runner/, not a fork). If you ever delete ../quic-interop-runner/ and re-clone manually, run quic/interop/run-matrix.sh once first or apply the same sed by hand. Upstream PR not yet filed.

Phase ladder (excerpt — full plan in conversation)

Phase	Goal	Tests	Exit criterion
0	Minimum harness	handshake	one test reproducible end-to-end ✅
1	Triangulate handshake bugs	+ versionnegotiation, chacha20	green vs aioquic + quiche + picoquic
2	Streams + loss + multiplexing	+ transfer, multiplexing, *loss, http3	transfer / multiplexing / http3 ✅ landed; loss tests pending
3	Edge cases	retry, resumption, zerortt, keyupdate, rebinding-*, blackhole, amplificationlimit	every test green or unsupported-127 with a written reason
4	CI gate	nightly Phases 1–2; PR-blocking subset on every push	qlogs uploaded as artifacts on red

Phase 2 — landed 2026-05-06

• Minimal Http3GetClient (in :quic-interop, NOT :quic — interop-test surface, not a production HTTP/3 client). Opens the three required client uni streams (control + QPACK encoder + QPACK decoder), sends empty SETTINGS, then per request opens a bidi stream, encodes a HEADERS frame with the four pseudo-headers using the existing literal-only QpackEncoder, FINs, and reassembles HEADERS+DATA frames from the response. Out-of-scope: GOAWAY, PUSH_PROMISE, dynamic QPACK table, trailers, priority.
• transfer + http3 testcases: GET each URL in REQUESTS sequentially, write each body to $DOWNLOADS/<basename>. Status != 200 fails.
• multiplexing testcase: same as transfer but issues each GET in a parallel coroutineScope { async { … } } so the request streams genuinely overlap on the wire (what tshark verifies).
• Aliased sim-driven testcases — these reuse the same client code paths; the runner injects the network condition via the ns-3 sim. Failures here are exactly the bug-finding signal we want, since they exercise loss recovery / RTT estimator / congestion behaviour against real peers:
  • transferloss → transfer (random packet loss)
  • transfercorruption → transfer (random bit-flip; AEAD AUTH FAIL → drop + retransmit)
  • longrtt → transfer (emulated high-latency link)
  • goodput → transfer (throughput floor)
  • crosstraffic → transfer (competing UDP flows on the same link)
  • handshakeloss → handshake (loss during handshake — tests CRYPTO retransmit)

Phase 3 — landed 2026-05-07 (post-quic-go interop)

• ALPN per testcase (:quic-interop's Alpn enum + per-test switch in InteropClient.main). quic-go enforces strictly with TLS no_application_protocol (CRYPTO_ERROR 0x178); aioquic / picoquic accept either. Convention: h3 for http3/multiplexing, hq-interop everywhere else.
• HqInteropGetClient (HTTP/0.9 over QUIC) — open bidi, send GET /path\r\n, FIN, read body until server FINs. No framing, no QPACK. ~30 lines.
• Multi-stream FIN delivery fix in QuicConnection.closeAllSignals(): pre-fix iterated only the connection-wide signal channels, leaving every per-stream incomingChannel open after teardown — coroutines suspended on stream.incoming.collect { ... } hung forever. Fix iterates streamsList on close. Three regression tests in MultiStreamFinDeliveryTest.
• retry + ipv6 testcases enabled in dispatch. retry rides on agent 3's RFC 9000 §17.2.5 + RFC 9001 §5.8 implementation. ipv6 should "just work" via JDK's DatagramChannel v6 support; runner-validated when run.

Validated against (as of 2026-05-07 evening)

Latest run results before pushing the multi-ALPN fix acfe815e1:

Peer	handshake	chacha20	transfer	http3	multiplexing
aioquic	✓	✓	✓	✓	✕ (channel-saturation, agent C investigating)
picoquic	✕ (alpn=hq-interop unsupported, fixed in acfe815e1)	✕	✕	✓	✕
quic-go	(untested post-fix)

After acfe815e1's multi-ALPN offer, predictions:

• picoquic returns to 4/4 (server picks h3, we run Http3GetClient).
• quic-go: handshake / chacha20 / transfer / http3 should green (server picks hq-interop for non-h3 tests).
• aioquic: still 4/5; multiplexing held back by channel-saturation bug.

Phase 4 — landed 2026-05-07 (overnight agents)

All three agents merged onto the branch with three rounds of fixup (each agent's worktree was based on main, not the branch HEAD, so they clobbered each other's changes during merge):

• Agent A — VN-handling defense in :quic: configurable initialVersion on QuicConnection, applyVersionNegotiation state machine, vnConsumed latch, downgrade defenses. NOTE: the runner does not have a versionnegotiation testcase (it has v2, which tests QUIC v2 — we're v1-only). Code stays as defensive support for any server that throws a VN at us, but unused by the matrix.
• Agent B — qlog observer: QlogObserver interface in :quic with NoOp default + 12 event hooks (packet-sent/received/dropped, key-updated, conn-started/closed, loss-detected, PTO-fired, transport-params, ALPN, version). QlogWriter (Jackson-backed JSON-NDJSON) in :quic-interop. InteropClient reads $QLOGDIR the runner already sets and writes client.sqlog per testcase. Drag straight into qvis.quictools.info to see the trace.
• Agent C — peer-uni-stream drainer: drainPeerInitiatedUniStreamsIntoBlackHole helper on QuicConnection, wired into Http3GetClient.init(scope). Fixes the multiplexing channel-saturation symptom (server's uni streams accumulated bytes in 64-chunk per-stream channels until parser tore down with INTERNAL_ERROR ~4.5s into a multi-stream run).

Phase 5 — landed 2026-05-07 (post-overnight bug-hunt)

The full overnight session pulled hard on the bugs the matrix exposed. qlog turned out to be the unblocking tool — every fix in this phase came from staring at a client.sqlog and matching it against the RFC pages it referenced.

Commit	Diagnosis pattern
bd9d717df	IOException: Stream closed from QlogWriter mid-test → observer threw into the send loop, killed connections that had already completed transfer.
99a1a91de	qlog stops at t=400ms, no inbound packets → per-event writer.flush() stalled the connection lock on macOS Docker filesystem virtualization.
c0d7b6031	aioquic CONNECTION_CLOSE: Packet contains no CRYPTO frame after PTO probe → our PTO emitted bare PING, not CRYPTO retransmit. Restored agent 2's wiring lost in the qlog merge.
17b80270d	runner's retry verdict Client reset the packet number. Check failed for PN 0 → applyRetry reset Initial PN to 0; RFC 9001 §5.7 says PN namespace continues across Retry. New LevelState.resetForRetry helper.
9a74d1d5d	multiplexing qlog showed STREAM frames still arriving at t=31s when local timeout fired → bumped TRANSFER_TIMEOUT_SEC 30→60. Doesn't fix throughput, just lets the test budget match the workload.
32ccbd2b2	coroutineScope { urls.map { async }.map { it.await() } } blocks on slowest hung stream → per-stream withTimeoutOrNull so a single bad stream surfaces as status=0 instead of hanging the matrix.

After all of these, expectation is aioquic / picoquic ≥6/7 (M might still be flaky). retry green via qlog-driven debugging is the marquee result.

Phase 6 — landed 2026-05-10 (matrix expansion + flakiness sweep)

Three-round flakiness sweep across four peers (aioquic, picoquic, quic-go, quinn). Round 1 = full 24-testcase matrix; rounds 2 and 3 = 11-test audio-critical subset (handshake, transfer, multiplexing, transferloss, transfercorruption, handshakeloss, handshakecorruption, longrtt, blackhole, keyupdate, retry).

quinn added to the default set this phase — was previously untested; cohort coverage demanded it because most Rust-based Nostr / MoQ relays use it on the server side. Validated identical to the existing three:

Peer	Pass	Unsupported	Fail	Goodput (kbps)	Crosstraffic (kbps)
aioquic	19/22	E, CM, V2	—	9071 ± 53	5084 ± 257
picoquic	20/22	V2	CM (expected — no client-initiated migration)	9234 ± 12	7405 ± 140
quic-go	19/22	E, CM, V2	—	9449 ± 8	6276 ± 218
quinn	20/22	CM, V2	—	9322 ± 6	4117 ± 2283 (env-inflated; see below)

Across 528 test executions (4 peers × 1 full round + 4 peers × 2 audio-critical rounds × 11 tests, plus 10-iter measurements):

• Zero result flakiness. No testcase that passed on one round ever failed on another, and vice versa.
• Two stall classes, both environmental (Docker Desktop on macOS, not in :quic):
  • Type A (compose-init lockup) — docker compose up hangs for ~15-16 min between Python's command-log and compose's first Docker API call. Hit 4 testcases: picoquic r2 transfer, quic-go r2 handshakecorruption, quinn r1 rebind-port/rebind-addr (both), quinn r3 handshakeloss. Test always succeeds once compose starts. qlog confirms the QUIC connection itself completes in 10-30s. The quinn-r1 crosstraffic std-dev (±2283 kbps) is entirely driven by 2 of 5 iterations being trapped in this stall — the 3 unaffected iters were 111/138/97 s = consistent with peers.
  • Type B (pyshark XML stall) — runner's _get_packets() iteration through tshark's XML via pyshark takes 16+ min on rebind/crosstraffic pcaps. Direct tshark CLI on the same pcap: 0.6 s. pyshark wrapper: 66 s. Multiple passes per testcase amplify it.
• Mitigations documented in plans/2026-05-06-interop-runner.md Docker Desktop section (separate investigation note).

Still open

• v2 — server demands QUIC v2 (RFC 9369). We're v1-only. Implementing v2 is its own project (different Initial-secret derivation, different transport-parameter encoding, different long-header type bits — not just a version-number swap).
• Multiplexing throughput on Mac+Rosetta — measured at ~23 streams/sec (aioquic processed 1359 GETs in 58s). The runner's multiplexing test generates 1999 files; even with a 60s budget we only get 70% of them open before the test ends, with most coroutines failing to surface a status=200 in time for the file-write loop. Throughput is hard-bottlenecked by :quic's single conn.lock serializing the send loop, the read loop, and openBidiStream across all 1999 awaiting coroutines. Under this lock contention pattern, parallelism doesn't help — coroutines queue on the lock anyway. Possible fixes (non-trivial):
  • Per-level / per-stream lock split (big refactor; touches almost everything in :quic/connection).
  • Batch concurrency via a Semaphore capping in-flight client.get() calls to e.g. 64 at a time. Doesn't lift the throughput ceiling but reduces lock thrash and probably lets MORE files complete in 60s.
  • QPACK dynamic-table support so aioquic doesn't need to fall back to literal encoding (would help on the response-decode side). None are urgent; this is a stress-test scenario, not a normal-traffic one. Filed for follow-up.
• Server role — entire stack is client-only. Implementing the server side would unlock handshakeloss against aioquic (currently ? because aioquic-server doesn't support it), and would make us a self-validating peer.

Concurrency

run-matrix.sh is NOT safe to run in parallel. The runner's docker-compose.yml hardcodes container_name: sim/server/client — Docker enforces those globally. Use a sequential loop:

for peer in aioquic picoquic quic-go quinn; do
    quic/interop/run-matrix.sh -s $peer -t handshake,chacha20,...
done

Default peer set

The four-peer baseline that every change touching :quic should be checked against is aioquic, picoquic, quic-go, and quinn. The rationale is product-shaped, not protocol-shaped: this is the cohort that the Nostr / MoQ relay ecosystem our users actually run their servers on is built on. quinn in particular underwrites most of the Rust relay stacks, so an interop regression there is a user-visible regression — it must stay flawless. quinn was added to the default set 2026-05-10 after running cleanly across the same audio-critical 3-round flakiness sweep used for the other three peers.

Explicitly unsupported testcases (return 127, runner skips)

Testcase	Reason
versionnegotiation	QuicConnectionWriter hard-codes QuicVersion.V1; needs a configurable initial version + VN-response retry path. In progress (background agent).
resumption	session ticket parsing + persistence not yet implemented
zerortt	depends on resumption + early-data path
keyupdate	KEY_PHASE bit handling not yet implemented in 1-RTT
rebinding-port, rebinding-addr	client-side connection migration (re-bind UDP socket, NEW_CONNECTION_ID rotation) not implemented
amplificationlimit	server-side test, N/A for client role
blackhole	inverse test (verifies we fail on dead network in bounded time); needs special handling
ipv6	UdpSocket IPv6 path not exercised; risky to claim without testing

Phase 1a — landed 2026-05-06

• SSLKEYLOGFILE writer in InteropClient (NSS Key Log Format) so Wireshark decrypts the sim's pcap captures. Backed by:
  • TlsClient.clientRandom (new public read-only property, captured in start() before sending ClientHello).
  • QuicConnection.extraSecretsListener (new optional constructor param, chained after the connection's own key-installation listener; no-op default, so production callers are unaffected).
• chacha20 testcase: forces ChaCha20-Poly1305 only via new QuicConnection.cipherSuites knob (threaded through TlsClient → buildQuicClientHello → TlsClientHello).

Phase 1b — open

• QLOGDIR: :quic has no qlog observer infrastructure yet. Wiring needs hooks at packet send/recv, frame dispatch, recovery, and TLS state transitions, plus the qlog JSON-NDJSON schema. Sized as its own design doc before implementation.
• versionnegotiation: QuicConnectionWriter hard-codes QuicVersion.V1 in two call sites; threading a configurable initial-version through and wiring the response-handling path is non-trivial. Defer.
• Server role: we are client-first. Reassess after Phase 3.
• WebTransport is not part of the standard interop matrix; it needs a separate harness against moq-rs / chrome-headless.