amethyst/quic/interop/plans/2026-05-06-interop-runner.md

# quic-interop-runner endpoint — Phase 0 scaffolding

Date: 2026-05-06

## Why

We want the [`quic-interop-runner`](https://github.com/quic-interop/quic-interop-runner)
matrix as a **bug-finding harness**, not a vanity scoreboard. Each peer impl
(quiche, aioquic, picoquic, ngtcp2, msquic, mvfst, lsquic, neqo, kwik) enforces
different parts of RFC 9000/9001/9002/9114 strictly, so failures triangulate
to specific bugs in `:quic`. The runner's ns-3 sim also exposes loss /
reorder / migration scenarios that are awkward to reproduce in unit tests.

## What's in Phase 0

- `:quic-interop` Gradle module (JVM-only application, registered at
  `quic/interop/` via `settings.gradle`).
- `InteropClient.kt` reads the runner's env-var contract (`ROLE`, `TESTCASE`,
  `REQUESTS`, …) and dispatches by testcase. Phase 0 implements only
  `handshake`; everything else returns `127` (runner-skip).
- `Dockerfile` based on `martenseemann/quic-network-simulator-endpoint` +
  OpenJDK 21 runtime, copies the `installDist` output.
- `run_endpoint.sh` sources the base image's `/setup.sh` then execs our JVM
  binary.
- `Makefile` wrappers: `make build`, `make clean`. (A `make smoke`
  target previously stood up picoquic + our endpoint on a private Docker
  bridge to bisect runner failures from impl failures; dropped once
  the runner reliably exercised both paths.)

## Local iteration loop

The fast path: `quic/interop/run-matrix.sh` clones the runner alongside
this repo, sets up a venv, merges our `implementations_quic.json` snippet,
builds the endpoint image, and invokes `run.py`. All steps are
idempotent so repeated invocations just iterate.

```
# Single test against the most permissive peer:
quic/interop/run-matrix.sh -s aioquic -t handshake

# A focused triangulation:
quic/interop/run-matrix.sh -s aioquic   -t handshake,chacha20
quic/interop/run-matrix.sh -s quic-go   -t handshake,chacha20
quic/interop/run-matrix.sh -s picoquic  -t handshake,chacha20

# Tight inner loop — skip the image rebuild between test selections:
SKIP_BUILD=1 quic/interop/run-matrix.sh -s aioquic -t transfer
```

Manual flow (if `run-matrix.sh` doesn't fit):

```
make -C quic/interop build
# Then in a sibling clone of quic-interop-runner, merge our snippet:
jq -s '.[0] * .[1]' implementations_quic.json \
   ../amethyst/quic/interop/quic-interop-runner-snippet.json \
   > implementations_quic.json.new && mv implementations_quic.json.new implementations_quic.json
python run.py -d -i amethyst -s aioquic -t handshake,chacha20 --log-dir ./logs
```

Inspect `./logs/<run>/client_qlog/*.qlog` in qvis when something breaks.

### macOS gotcha — upstream `certs.sh` patch (auto-applied)

`run-matrix.sh` step 1a rewrites `LC_CTYPE=C tr` → `LC_ALL=C tr` in
the upstream `certs.sh`. macOS BSD `tr` chokes on `/dev/urandom` with
`tr: Illegal byte sequence` when only `LC_CTYPE` is set; `LC_ALL` wins
in the locale-precedence chain and silences it. Without this, the
`amplificationlimit` testcase aborts at cert generation, and every
test sequenced after it in `TESTCASES_QUIC`
(`handshakeloss`, `transferloss`, `handshakecorruption`,
`transfercorruption`, `ipv6`, `v2`, `rebind-port`, `rebind-addr`,
`connectionmigration`, plus the goodput/crosstraffic measurements)
silently never runs. The matrix prints the post-mortem summary
showing only the 12 testcases that ran before the abort, which looks
deceptively complete.

The patch is idempotent and is re-applied on every clone of the
runner (the fix is a working-tree edit to `../quic-interop-runner/`,
not a fork). If you ever delete `../quic-interop-runner/` and
re-clone manually, run `quic/interop/run-matrix.sh` once first or
apply the same `sed` by hand. Upstream PR not yet filed.

## Phase ladder (excerpt — full plan in conversation)

| Phase | Goal | Tests | Exit criterion |
|---|---|---|---|
| 0 | Minimum harness | `handshake` | one test reproducible end-to-end ✅ |
| 1 | Triangulate handshake bugs | + `versionnegotiation`, `chacha20` | green vs aioquic + quiche + picoquic |
| 2 | Streams + loss + multiplexing | + `transfer`, `multiplexing`, `*loss`, `http3` | `transfer` / `multiplexing` / `http3` ✅ landed; loss tests pending |
| 3 | Edge cases | `retry`, `resumption`, `zerortt`, `keyupdate`, `rebinding-*`, `blackhole`, `amplificationlimit` | every test green or unsupported-127 with a written reason |
| 4 | CI gate | nightly Phases 1–2; PR-blocking subset on every push | qlogs uploaded as artifacts on red |

## Phase 2 — landed 2026-05-06

- Minimal `Http3GetClient` (in `:quic-interop`, NOT `:quic` — interop-test
  surface, not a production HTTP/3 client). Opens the three required
  client uni streams (control + QPACK encoder + QPACK decoder), sends
  empty SETTINGS, then per request opens a bidi stream, encodes a HEADERS
  frame with the four pseudo-headers using the existing literal-only
  `QpackEncoder`, FINs, and reassembles HEADERS+DATA frames from the
  response. Out-of-scope: GOAWAY, PUSH_PROMISE, dynamic QPACK table,
  trailers, priority.
- `transfer` + `http3` testcases: GET each URL in `REQUESTS` sequentially,
  write each body to `$DOWNLOADS/<basename>`. Status != 200 fails.
- `multiplexing` testcase: same as `transfer` but issues each GET in a
  parallel `coroutineScope { async { … } }` so the request streams
  genuinely overlap on the wire (what tshark verifies).
- Aliased sim-driven testcases — these reuse the same client code paths;
  the runner injects the network condition via the ns-3 sim. Failures
  here are exactly the bug-finding signal we want, since they exercise
  loss recovery / RTT estimator / congestion behaviour against real peers:
  - `transferloss` → transfer (random packet loss)
  - `transfercorruption` → transfer (random bit-flip; AEAD AUTH FAIL → drop + retransmit)
  - `longrtt` → transfer (emulated high-latency link)
  - `goodput` → transfer (throughput floor)
  - `crosstraffic` → transfer (competing UDP flows on the same link)
  - `handshakeloss` → handshake (loss during handshake — tests CRYPTO retransmit)

## Phase 3 — landed 2026-05-07 (post-quic-go interop)

- ALPN per testcase (`:quic-interop`'s `Alpn` enum + per-test switch in
  `InteropClient.main`). quic-go enforces strictly with TLS
  `no_application_protocol` (CRYPTO_ERROR 0x178); aioquic / picoquic accept
  either. Convention: `h3` for `http3`/`multiplexing`, `hq-interop`
  everywhere else.
- `HqInteropGetClient` (HTTP/0.9 over QUIC) — open bidi, send `GET /path\r\n`,
  FIN, read body until server FINs. No framing, no QPACK. ~30 lines.
- Multi-stream FIN delivery fix in `QuicConnection.closeAllSignals()`: pre-fix
  iterated only the connection-wide signal channels, leaving every per-stream
  `incomingChannel` open after teardown — coroutines suspended on
  `stream.incoming.collect { ... }` hung forever. Fix iterates `streamsList`
  on close. Three regression tests in `MultiStreamFinDeliveryTest`.
- `retry` + `ipv6` testcases enabled in dispatch. `retry` rides on agent 3's
  RFC 9000 §17.2.5 + RFC 9001 §5.8 implementation. `ipv6` should "just work"
  via JDK's `DatagramChannel` v6 support; runner-validated when run.

## Validated against (as of 2026-05-07 evening)

Latest run results before pushing the multi-ALPN fix `acfe815e1`:

| Peer | handshake | chacha20 | transfer | http3 | multiplexing |
|---|---|---|---|---|---|
| aioquic   | ✓ | ✓ | ✓ | ✓ | ✕ (channel-saturation, agent C investigating) |
| picoquic  | ✕ (alpn=hq-interop unsupported, fixed in `acfe815e1`) | ✕ | ✕ | ✓ | ✕ |
| quic-go   | (untested post-fix) | | | | |

After `acfe815e1`'s multi-ALPN offer, predictions:
- picoquic returns to 4/4 (server picks `h3`, we run Http3GetClient).
- quic-go: handshake / chacha20 / transfer / http3 should green (server picks `hq-interop` for non-h3 tests).
- aioquic: still 4/5; multiplexing held back by channel-saturation bug.

## Phase 4 — landed 2026-05-07 (overnight agents)

All three agents merged onto the branch with three rounds of fixup
(each agent's worktree was based on `main`, not the branch HEAD, so
they clobbered each other's changes during merge):

- **Agent A — VN-handling defense in `:quic`**: configurable
  `initialVersion` on `QuicConnection`, `applyVersionNegotiation`
  state machine, `vnConsumed` latch, downgrade defenses. *NOTE*: the
  runner does not have a `versionnegotiation` testcase (it has `v2`,
  which tests QUIC v2 — we're v1-only). Code stays as defensive
  support for any server that throws a VN at us, but unused by the
  matrix.
- **Agent B — qlog observer**: `QlogObserver` interface in `:quic`
  with NoOp default + 12 event hooks (packet-sent/received/dropped,
  key-updated, conn-started/closed, loss-detected, PTO-fired,
  transport-params, ALPN, version). `QlogWriter` (Jackson-backed
  JSON-NDJSON) in `:quic-interop`. `InteropClient` reads `$QLOGDIR`
  the runner already sets and writes `client.sqlog` per testcase.
  Drag straight into qvis.quictools.info to see the trace.
- **Agent C — peer-uni-stream drainer**: `drainPeerInitiatedUniStreamsIntoBlackHole`
  helper on `QuicConnection`, wired into `Http3GetClient.init(scope)`.
  Fixes the multiplexing channel-saturation symptom (server's uni
  streams accumulated bytes in 64-chunk per-stream channels until
  parser tore down with INTERNAL_ERROR ~4.5s into a multi-stream run).

## Phase 5 — landed 2026-05-07 (post-overnight bug-hunt)

The full overnight session pulled hard on the bugs the matrix exposed.
qlog turned out to be the unblocking tool — every fix in this phase
came from staring at a `client.sqlog` and matching it against the RFC
pages it referenced.

| Commit | Diagnosis pattern |
|---|---|
| `bd9d717df` | `IOException: Stream closed` from QlogWriter mid-test → observer threw into the send loop, killed connections that had already completed transfer. |
| `99a1a91de` | qlog stops at t=400ms, no inbound packets → per-event `writer.flush()` stalled the connection lock on macOS Docker filesystem virtualization. |
| `c0d7b6031` | aioquic `CONNECTION_CLOSE: Packet contains no CRYPTO frame` after PTO probe → our PTO emitted bare PING, not CRYPTO retransmit. Restored agent 2's wiring lost in the qlog merge. |
| `17b80270d` | runner's retry verdict `Client reset the packet number. Check failed for PN 0` → `applyRetry` reset Initial PN to 0; RFC 9001 §5.7 says PN namespace continues across Retry. New `LevelState.resetForRetry` helper. |
| `9a74d1d5d` | multiplexing qlog showed STREAM frames still arriving at t=31s when local timeout fired → bumped `TRANSFER_TIMEOUT_SEC` 30→60. Doesn't fix throughput, just lets the test budget match the workload. |
| `32ccbd2b2` | `coroutineScope { urls.map { async }.map { it.await() } }` blocks on slowest hung stream → per-stream `withTimeoutOrNull` so a single bad stream surfaces as status=0 instead of hanging the matrix. |

After all of these, expectation is aioquic / picoquic ≥6/7 (M
might still be flaky). retry green via qlog-driven debugging is
the marquee result.

## Phase 6 — landed 2026-05-10 (matrix expansion + flakiness sweep)

Three-round flakiness sweep across four peers (aioquic, picoquic,
quic-go, quinn). Round 1 = full 24-testcase matrix; rounds 2 and 3 =
11-test audio-critical subset (handshake, transfer, multiplexing,
transferloss, transfercorruption, handshakeloss, handshakecorruption,
longrtt, blackhole, keyupdate, retry).

**quinn added** to the default set this phase — was previously
untested; cohort coverage demanded it because most Rust-based Nostr /
MoQ relays use it on the server side. Validated identical to the
existing three:

| Peer | Pass | Unsupported | Fail | Goodput (kbps) | Crosstraffic (kbps) |
|---|---|---|---|---|---|
| aioquic  | 19/22 | E, CM, V2 | — | 9071 ± 53 | 5084 ± 257 |
| picoquic | 20/22 | V2        | CM (expected — no client-initiated migration) | 9234 ± 12 | 7405 ± 140 |
| quic-go  | 19/22 | E, CM, V2 | — | 9449 ±  8 | 6276 ± 218 |
| quinn    | 20/22 | CM, V2    | — | 9322 ±  6 | 4117 ± 2283 (env-inflated; see below) |

Across **528 test executions** (4 peers × 1 full round + 4 peers × 2
audio-critical rounds × 11 tests, plus 10-iter measurements):
- **Zero result flakiness.** No testcase that passed on one round ever
  failed on another, and vice versa.
- **Two stall classes**, both environmental (Docker Desktop on macOS,
  not in `:quic`):
  - **Type A (compose-init lockup)** — `docker compose up` hangs for
    ~15-16 min between Python's command-log and compose's first Docker
    API call. Hit 4 testcases: picoquic r2 transfer, quic-go r2
    handshakecorruption, quinn r1 rebind-port/rebind-addr (both),
    quinn r3 handshakeloss. Test always succeeds once compose starts.
    qlog confirms the QUIC connection itself completes in 10-30s. The
    quinn-r1 crosstraffic std-dev (±2283 kbps) is entirely driven by
    2 of 5 iterations being trapped in this stall — the 3 unaffected
    iters were 111/138/97 s = consistent with peers.
  - **Type B (pyshark XML stall)** — runner's `_get_packets()`
    iteration through tshark's XML via pyshark takes 16+ min on
    rebind/crosstraffic pcaps. Direct tshark CLI on the same pcap:
    0.6 s. pyshark wrapper: 66 s. Multiple passes per testcase
    amplify it.
- Mitigations documented in `plans/2026-05-06-interop-runner.md`
  Docker Desktop section (separate investigation note).

## Still open

- **`v2`** — server demands QUIC v2 (RFC 9369). We're v1-only.
  Implementing v2 is its own project (different Initial-secret
  derivation, different transport-parameter encoding, different
  long-header type bits — not just a version-number swap).
- **Multiplexing throughput on Mac+Rosetta** — measured at ~23 streams/sec
  (aioquic processed 1359 GETs in 58s). The runner's multiplexing test
  generates 1999 files; even with a 60s budget we only get 70% of them
  open before the test ends, with most coroutines failing to surface a
  status=200 in time for the file-write loop. Throughput is hard-bottlenecked
  by `:quic`'s single `conn.lock` serializing the send loop, the read
  loop, and `openBidiStream` across all 1999 awaiting coroutines. Under
  this lock contention pattern, parallelism doesn't help — coroutines
  queue on the lock anyway.
  Possible fixes (non-trivial):
    - Per-level / per-stream lock split (big refactor; touches almost
      everything in `:quic/connection`).
    - Batch concurrency via a `Semaphore` capping in-flight `client.get()`
      calls to e.g. 64 at a time. Doesn't lift the throughput ceiling but
      reduces lock thrash and probably lets MORE files complete in 60s.
    - QPACK dynamic-table support so aioquic doesn't need to fall back
      to literal encoding (would help on the response-decode side).
  None are urgent; this is a stress-test scenario, not a normal-traffic
  one. Filed for follow-up.
- **Server role** — entire stack is client-only. Implementing the
  server side would unlock `handshakeloss` against aioquic
  (currently `?` because aioquic-server doesn't support it), and
  would make us a self-validating peer.

## Concurrency

`run-matrix.sh` is NOT safe to run in parallel. The runner's
docker-compose.yml hardcodes `container_name: sim/server/client` —
Docker enforces those globally. Use a sequential loop:

```
for peer in aioquic picoquic quic-go quinn; do
    quic/interop/run-matrix.sh -s $peer -t handshake,chacha20,...
done
```

## Default peer set

The four-peer baseline that every change touching `:quic` should be
checked against is `aioquic`, `picoquic`, `quic-go`, and `quinn`. The
rationale is product-shaped, not protocol-shaped: this is the cohort
that the Nostr / MoQ relay ecosystem our users actually run their
servers on is built on. quinn in particular underwrites most of the
Rust relay stacks, so an interop regression there is a user-visible
regression — it must stay flawless. quinn was added to the default
set 2026-05-10 after running cleanly across the same audio-critical
3-round flakiness sweep used for the other three peers.

## Explicitly unsupported testcases (return 127, runner skips)

| Testcase | Reason |
|---|---|
| `versionnegotiation` | `QuicConnectionWriter` hard-codes `QuicVersion.V1`; needs a configurable initial version + VN-response retry path. **In progress (background agent)**. |
| `resumption` | session ticket parsing + persistence not yet implemented |
| `zerortt` | depends on `resumption` + early-data path |
| `keyupdate` | KEY_PHASE bit handling not yet implemented in 1-RTT |
| `rebinding-port`, `rebinding-addr` | client-side connection migration (re-bind UDP socket, NEW_CONNECTION_ID rotation) not implemented |
| `amplificationlimit` | server-side test, N/A for client role |
| `blackhole` | inverse test (verifies we *fail* on dead network in bounded time); needs special handling |
| `ipv6` | `UdpSocket` IPv6 path not exercised; risky to claim without testing |

## Phase 1a — landed 2026-05-06

- `SSLKEYLOGFILE` writer in `InteropClient` (NSS Key Log Format) so Wireshark
  decrypts the sim's pcap captures. Backed by:
  - `TlsClient.clientRandom` (new public read-only property, captured in
    `start()` before sending ClientHello).
  - `QuicConnection.extraSecretsListener` (new optional constructor param,
    chained after the connection's own key-installation listener; no-op
    default, so production callers are unaffected).
- `chacha20` testcase: forces ChaCha20-Poly1305 only via new
  `QuicConnection.cipherSuites` knob (threaded through `TlsClient` →
  `buildQuicClientHello` → `TlsClientHello`).

## Phase 1b — open

- `QLOGDIR`: `:quic` has no qlog observer infrastructure yet. Wiring needs
  hooks at packet send/recv, frame dispatch, recovery, and TLS state
  transitions, plus the qlog JSON-NDJSON schema. Sized as its own design
  doc before implementation.
- `versionnegotiation`: `QuicConnectionWriter` hard-codes `QuicVersion.V1`
  in two call sites; threading a configurable initial-version through and
  wiring the response-handling path is non-trivial. Defer.
- Server role: we are client-first. Reassess after Phase 3.
- WebTransport is **not** part of the standard interop matrix; it needs a
  separate harness against `moq-rs` / chrome-headless.