amethyst/quic/interop/plans/2026-05-06-interop-runner.md

quic-interop-runner endpoint — Phase 0 scaffolding

Date: 2026-05-06

Why

We want the quic-interop-runner matrix as a bug-finding harness, not a vanity scoreboard. Each peer impl (quiche, aioquic, picoquic, ngtcp2, msquic, mvfst, lsquic, neqo, kwik) enforces different parts of RFC 9000/9001/9002/9114 strictly, so failures triangulate to specific bugs in :quic. The runner's ns-3 sim also exposes loss / reorder / migration scenarios that are awkward to reproduce in unit tests.

What's in Phase 0

• :quic-interop Gradle module (JVM-only application, registered at quic/interop/ via settings.gradle).
• InteropClient.kt reads the runner's env-var contract (ROLE, TESTCASE, REQUESTS, …) and dispatches by testcase. Phase 0 implements only handshake; everything else returns 127 (runner-skip).
• Dockerfile based on martenseemann/quic-network-simulator-endpoint + OpenJDK 21 runtime, copies the installDist output.
• run_endpoint.sh sources the base image's /setup.sh then execs our JVM binary.
• Makefile wrappers: make build, make clean. (A make smoke target previously stood up picoquic + our endpoint on a private Docker bridge to bisect runner failures from impl failures; dropped once the runner reliably exercised both paths.)

Local iteration loop

The fast path: quic/interop/run-matrix.sh clones the runner alongside this repo, sets up a venv, merges our implementations_quic.json snippet, builds the endpoint image, and invokes run.py. All steps are idempotent so repeated invocations just iterate.

# Single test against the most permissive peer:
quic/interop/run-matrix.sh -s aioquic -t handshake

# A focused triangulation:
quic/interop/run-matrix.sh -s aioquic   -t handshake,chacha20
quic/interop/run-matrix.sh -s quic-go   -t handshake,chacha20
quic/interop/run-matrix.sh -s picoquic  -t handshake,chacha20

# Tight inner loop — skip the image rebuild between test selections:
SKIP_BUILD=1 quic/interop/run-matrix.sh -s aioquic -t transfer

Manual flow (if run-matrix.sh doesn't fit):

make -C quic/interop build
# Then in a sibling clone of quic-interop-runner, merge our snippet:
jq -s '.[0] * .[1]' implementations_quic.json \
   ../amethyst/quic/interop/quic-interop-runner-snippet.json \
   > implementations_quic.json.new && mv implementations_quic.json.new implementations_quic.json
python run.py -d -i amethyst -s aioquic -t handshake,chacha20 --log-dir ./logs

Inspect ./logs/<run>/client_qlog/*.qlog in qvis when something breaks.

Phase ladder (excerpt — full plan in conversation)

Phase	Goal	Tests	Exit criterion
0	Minimum harness	handshake	one test reproducible end-to-end ✅
1	Triangulate handshake bugs	+ versionnegotiation, chacha20	green vs aioquic + quiche + picoquic
2	Streams + loss + multiplexing	+ transfer, multiplexing, *loss, http3	transfer / multiplexing / http3 ✅ landed; loss tests pending
3	Edge cases	retry, resumption, zerortt, keyupdate, rebinding-*, blackhole, amplificationlimit	every test green or unsupported-127 with a written reason
4	CI gate	nightly Phases 1–2; PR-blocking subset on every push	qlogs uploaded as artifacts on red

Phase 2 — landed 2026-05-06

• Minimal Http3GetClient (in :quic-interop, NOT :quic — interop-test surface, not a production HTTP/3 client). Opens the three required client uni streams (control + QPACK encoder + QPACK decoder), sends empty SETTINGS, then per request opens a bidi stream, encodes a HEADERS frame with the four pseudo-headers using the existing literal-only QpackEncoder, FINs, and reassembles HEADERS+DATA frames from the response. Out-of-scope: GOAWAY, PUSH_PROMISE, dynamic QPACK table, trailers, priority.
• transfer + http3 testcases: GET each URL in REQUESTS sequentially, write each body to $DOWNLOADS/<basename>. Status != 200 fails.
• multiplexing testcase: same as transfer but issues each GET in a parallel coroutineScope { async { … } } so the request streams genuinely overlap on the wire (what tshark verifies).
• Aliased sim-driven testcases — these reuse the same client code paths; the runner injects the network condition via the ns-3 sim. Failures here are exactly the bug-finding signal we want, since they exercise loss recovery / RTT estimator / congestion behaviour against real peers:
  • transferloss → transfer (random packet loss)
  • transfercorruption → transfer (random bit-flip; AEAD AUTH FAIL → drop + retransmit)
  • longrtt → transfer (emulated high-latency link)
  • goodput → transfer (throughput floor)
  • crosstraffic → transfer (competing UDP flows on the same link)
  • handshakeloss → handshake (loss during handshake — tests CRYPTO retransmit)

Explicitly unsupported testcases (return 127, runner skips)

Testcase	Reason
versionnegotiation	QuicConnectionWriter hard-codes QuicVersion.V1; needs a configurable initial version + VN-response retry path
resumption	session ticket parsing + persistence not yet implemented
zerortt	depends on resumption + early-data path
keyupdate	KEY_PHASE bit handling not yet implemented in 1-RTT
retry	RetryPacket parses but isn't fully wired into the connection feed-loop yet — claiming support without verifying would mask bugs
rebinding-port, rebinding-addr	client-side connection migration (re-bind UDP socket, NEW_CONNECTION_ID rotation) not implemented
amplificationlimit	server-side test, N/A for client role
blackhole	inverse test (verifies we fail on dead network in bounded time); needs special handling
ipv6	UdpSocket IPv6 path not exercised; risky to claim without testing

Phase 1a — landed 2026-05-06

• SSLKEYLOGFILE writer in InteropClient (NSS Key Log Format) so Wireshark decrypts the sim's pcap captures. Backed by:
  • TlsClient.clientRandom (new public read-only property, captured in start() before sending ClientHello).
  • QuicConnection.extraSecretsListener (new optional constructor param, chained after the connection's own key-installation listener; no-op default, so production callers are unaffected).
• chacha20 testcase: forces ChaCha20-Poly1305 only via new QuicConnection.cipherSuites knob (threaded through TlsClient → buildQuicClientHello → TlsClientHello).

Phase 1b — open

• QLOGDIR: :quic has no qlog observer infrastructure yet. Wiring needs hooks at packet send/recv, frame dispatch, recovery, and TLS state transitions, plus the qlog JSON-NDJSON schema. Sized as its own design doc before implementation.
• versionnegotiation: QuicConnectionWriter hard-codes QuicVersion.V1 in two call sites; threading a configurable initial-version through and wiring the response-handling path is non-trivial. Defer.
• Server role: we are client-first. Reassess after Phase 3.
• WebTransport is not part of the standard interop matrix; it needs a separate harness against moq-rs / chrome-headless.