amethyst/quic/interop/plans/2026-05-06-interop-runner.md

quic-interop-runner endpoint — Phase 0 scaffolding

Date: 2026-05-06

Why

We want the quic-interop-runner matrix as a bug-finding harness, not a vanity scoreboard. Each peer impl (quiche, aioquic, picoquic, ngtcp2, msquic, mvfst, lsquic, neqo, kwik) enforces different parts of RFC 9000/9001/9002/9114 strictly, so failures triangulate to specific bugs in :quic. The runner's ns-3 sim also exposes loss / reorder / migration scenarios that are awkward to reproduce in unit tests.

What's in Phase 0

• :quic-interop Gradle module (JVM-only application, registered at quic/interop/ via settings.gradle).
• InteropClient.kt reads the runner's env-var contract (ROLE, TESTCASE, REQUESTS, …) and dispatches by testcase. Phase 0 implements only handshake; everything else returns 127 (runner-skip).
• Dockerfile based on martenseemann/quic-network-simulator-endpoint + OpenJDK 21 runtime, copies the installDist output.
• run_endpoint.sh sources the base image's /setup.sh then execs our JVM binary.
• Makefile wrappers: make build, make clean. (A make smoke target previously stood up picoquic + our endpoint on a private Docker bridge to bisect runner failures from impl failures; dropped once the runner reliably exercised both paths.)

Local iteration loop

The fast path: quic/interop/run-matrix.sh clones the runner alongside this repo, sets up a venv, merges our implementations_quic.json snippet, builds the endpoint image, and invokes run.py. All steps are idempotent so repeated invocations just iterate.

# Single test against the most permissive peer:
quic/interop/run-matrix.sh -s aioquic -t handshake

# A focused triangulation:
quic/interop/run-matrix.sh -s aioquic   -t handshake,chacha20
quic/interop/run-matrix.sh -s quic-go   -t handshake,chacha20
quic/interop/run-matrix.sh -s picoquic  -t handshake,chacha20

# Tight inner loop — skip the image rebuild between test selections:
SKIP_BUILD=1 quic/interop/run-matrix.sh -s aioquic -t transfer

Manual flow (if run-matrix.sh doesn't fit):

make -C quic/interop build
# Then in a sibling clone of quic-interop-runner, merge our snippet:
jq -s '.[0] * .[1]' implementations_quic.json \
   ../amethyst/quic/interop/quic-interop-runner-snippet.json \
   > implementations_quic.json.new && mv implementations_quic.json.new implementations_quic.json
python run.py -d -i amethyst -s aioquic -t handshake,chacha20 --log-dir ./logs

Inspect ./logs/<run>/client_qlog/*.qlog in qvis when something breaks.

Phase ladder (excerpt — full plan in conversation)

Phase	Goal	Tests	Exit criterion
0	Minimum harness	handshake	one test reproducible end-to-end ✅
1	Triangulate handshake bugs	+ versionnegotiation, chacha20	green vs aioquic + quiche + picoquic
2	Streams + loss + multiplexing	+ transfer, multiplexing, *loss, http3	transfer / multiplexing / http3 ✅ landed; loss tests pending
3	Edge cases	retry, resumption, zerortt, keyupdate, rebinding-*, blackhole, amplificationlimit	every test green or unsupported-127 with a written reason
4	CI gate	nightly Phases 1–2; PR-blocking subset on every push	qlogs uploaded as artifacts on red

Phase 2 — landed 2026-05-06

• Minimal Http3GetClient (in :quic-interop, NOT :quic — interop-test surface, not a production HTTP/3 client). Opens the three required client uni streams (control + QPACK encoder + QPACK decoder), sends empty SETTINGS, then per request opens a bidi stream, encodes a HEADERS frame with the four pseudo-headers using the existing literal-only QpackEncoder, FINs, and reassembles HEADERS+DATA frames from the response. Out-of-scope: GOAWAY, PUSH_PROMISE, dynamic QPACK table, trailers, priority.
• transfer + http3 testcases: GET each URL in REQUESTS sequentially, write each body to $DOWNLOADS/<basename>. Status != 200 fails.
• multiplexing testcase: same as transfer but issues each GET in a parallel coroutineScope { async { … } } so the request streams genuinely overlap on the wire (what tshark verifies).
• Aliased sim-driven testcases — these reuse the same client code paths; the runner injects the network condition via the ns-3 sim. Failures here are exactly the bug-finding signal we want, since they exercise loss recovery / RTT estimator / congestion behaviour against real peers:
  • transferloss → transfer (random packet loss)
  • transfercorruption → transfer (random bit-flip; AEAD AUTH FAIL → drop + retransmit)
  • longrtt → transfer (emulated high-latency link)
  • goodput → transfer (throughput floor)
  • crosstraffic → transfer (competing UDP flows on the same link)
  • handshakeloss → handshake (loss during handshake — tests CRYPTO retransmit)

Phase 3 — landed 2026-05-07 (post-quic-go interop)

• ALPN per testcase (:quic-interop's Alpn enum + per-test switch in InteropClient.main). quic-go enforces strictly with TLS no_application_protocol (CRYPTO_ERROR 0x178); aioquic / picoquic accept either. Convention: h3 for http3/multiplexing, hq-interop everywhere else.
• HqInteropGetClient (HTTP/0.9 over QUIC) — open bidi, send GET /path\r\n, FIN, read body until server FINs. No framing, no QPACK. ~30 lines.
• Multi-stream FIN delivery fix in QuicConnection.closeAllSignals(): pre-fix iterated only the connection-wide signal channels, leaving every per-stream incomingChannel open after teardown — coroutines suspended on stream.incoming.collect { ... } hung forever. Fix iterates streamsList on close. Three regression tests in MultiStreamFinDeliveryTest.
• retry + ipv6 testcases enabled in dispatch. retry rides on agent 3's RFC 9000 §17.2.5 + RFC 9001 §5.8 implementation. ipv6 should "just work" via JDK's DatagramChannel v6 support; runner-validated when run.

Validated against (as of 2026-05-07 evening)

Latest run results before pushing the multi-ALPN fix acfe815e1:

Peer	handshake	chacha20	transfer	http3	multiplexing
aioquic	✓	✓	✓	✓	✕ (channel-saturation, agent C investigating)
picoquic	✕ (alpn=hq-interop unsupported, fixed in acfe815e1)	✕	✕	✓	✕
quic-go	(untested post-fix)

After acfe815e1's multi-ALPN offer, predictions:

• picoquic returns to 4/4 (server picks h3, we run Http3GetClient).
• quic-go: handshake / chacha20 / transfer / http3 should green (server picks hq-interop for non-h3 tests).
• aioquic: still 4/5; multiplexing held back by channel-saturation bug.

Phase 4 — landed 2026-05-07 (overnight agents)

All three agents merged onto the branch with three rounds of fixup (each agent's worktree was based on main, not the branch HEAD, so they clobbered each other's changes during merge):

• Agent A — VN-handling defense in :quic: configurable initialVersion on QuicConnection, applyVersionNegotiation state machine, vnConsumed latch, downgrade defenses. NOTE: the runner does not have a versionnegotiation testcase (it has v2, which tests QUIC v2 — we're v1-only). Code stays as defensive support for any server that throws a VN at us, but unused by the matrix.
• Agent B — qlog observer: QlogObserver interface in :quic with NoOp default + 12 event hooks (packet-sent/received/dropped, key-updated, conn-started/closed, loss-detected, PTO-fired, transport-params, ALPN, version). QlogWriter (Jackson-backed JSON-NDJSON) in :quic-interop. InteropClient reads $QLOGDIR the runner already sets and writes client.sqlog per testcase. Drag straight into qvis.quictools.info to see the trace.
• Agent C — peer-uni-stream drainer: drainPeerInitiatedUniStreamsIntoBlackHole helper on QuicConnection, wired into Http3GetClient.init(scope). Fixes the multiplexing channel-saturation symptom (server's uni streams accumulated bytes in 64-chunk per-stream channels until parser tore down with INTERNAL_ERROR ~4.5s into a multi-stream run).

Phase 5 — landed 2026-05-07 (post-overnight bug-hunt)

The full overnight session pulled hard on the bugs the matrix exposed. qlog turned out to be the unblocking tool — every fix in this phase came from staring at a client.sqlog and matching it against the RFC pages it referenced.

Commit	Diagnosis pattern
bd9d717df	IOException: Stream closed from QlogWriter mid-test → observer threw into the send loop, killed connections that had already completed transfer.
99a1a91de	qlog stops at t=400ms, no inbound packets → per-event writer.flush() stalled the connection lock on macOS Docker filesystem virtualization.
c0d7b6031	aioquic CONNECTION_CLOSE: Packet contains no CRYPTO frame after PTO probe → our PTO emitted bare PING, not CRYPTO retransmit. Restored agent 2's wiring lost in the qlog merge.
17b80270d	runner's retry verdict Client reset the packet number. Check failed for PN 0 → applyRetry reset Initial PN to 0; RFC 9001 §5.7 says PN namespace continues across Retry. New LevelState.resetForRetry helper.
9a74d1d5d	multiplexing qlog showed STREAM frames still arriving at t=31s when local timeout fired → bumped TRANSFER_TIMEOUT_SEC 30→60. Doesn't fix throughput, just lets the test budget match the workload.
32ccbd2b2	coroutineScope { urls.map { async }.map { it.await() } } blocks on slowest hung stream → per-stream withTimeoutOrNull so a single bad stream surfaces as status=0 instead of hanging the matrix.

After all of these, expectation is aioquic / picoquic ≥6/7 (M might still be flaky). retry green via qlog-driven debugging is the marquee result.

Still open

• v2 — server demands QUIC v2 (RFC 9369). We're v1-only. Implementing v2 is its own project (different Initial-secret derivation, different transport-parameter encoding, different long-header type bits — not just a version-number swap).
• Multiplexing throughput on Mac+Rosetta — the timeout bump unblocks tests that were finishing in time. The deeper fix is per-stream backpressure (parser suspends when channel near capacity instead of tearing down or dropping). See :quic's Audit-4 #3 decision: prefer fail-fast over silent-drop on per-stream channel saturation. Needs a re-think for multi-stream-heavy workloads.
• Server role — entire stack is client-only. Implementing the server side would unlock handshakeloss against aioquic (currently ? because aioquic-server doesn't support it), and would make us a self-validating peer.

Concurrency

run-matrix.sh is NOT safe to run in parallel. The runner's docker-compose.yml hardcodes container_name: sim/server/client — Docker enforces those globally. Use a sequential loop:

for peer in aioquic picoquic quic-go; do
    quic/interop/run-matrix.sh -s $peer -t handshake,chacha20,...
done

Explicitly unsupported testcases (return 127, runner skips)

Testcase	Reason
versionnegotiation	QuicConnectionWriter hard-codes QuicVersion.V1; needs a configurable initial version + VN-response retry path. In progress (background agent).
resumption	session ticket parsing + persistence not yet implemented
zerortt	depends on resumption + early-data path
keyupdate	KEY_PHASE bit handling not yet implemented in 1-RTT
rebinding-port, rebinding-addr	client-side connection migration (re-bind UDP socket, NEW_CONNECTION_ID rotation) not implemented
amplificationlimit	server-side test, N/A for client role
blackhole	inverse test (verifies we fail on dead network in bounded time); needs special handling
ipv6	UdpSocket IPv6 path not exercised; risky to claim without testing

Phase 1a — landed 2026-05-06

• SSLKEYLOGFILE writer in InteropClient (NSS Key Log Format) so Wireshark decrypts the sim's pcap captures. Backed by:
  • TlsClient.clientRandom (new public read-only property, captured in start() before sending ClientHello).
  • QuicConnection.extraSecretsListener (new optional constructor param, chained after the connection's own key-installation listener; no-op default, so production callers are unaffected).
• chacha20 testcase: forces ChaCha20-Poly1305 only via new QuicConnection.cipherSuites knob (threaded through TlsClient → buildQuicClientHello → TlsClientHello).

Phase 1b — open

• QLOGDIR: :quic has no qlog observer infrastructure yet. Wiring needs hooks at packet send/recv, frame dispatch, recovery, and TLS state transitions, plus the qlog JSON-NDJSON schema. Sized as its own design doc before implementation.
• versionnegotiation: QuicConnectionWriter hard-codes QuicVersion.V1 in two call sites; threading a configurable initial-version through and wiring the response-handling path is non-trivial. Defer.
• Server role: we are client-first. Reassess after Phase 3.
• WebTransport is not part of the standard interop matrix; it needs a separate harness against moq-rs / chrome-headless.