Removes the `hang-interop` + `browser-interop` jobs from
`build.yml` (added in commit `21947bc5` after a 10/10 stability
sweep × 22 tests = 220/220 hard-pass). Cold-cache cost is
~10 min hang + ~13 min browser; warm-cache critical-path add
is ~6 min via the parallel browser job. Most PRs don't touch
audio / MoQ / QUIC, so paying that cost on every PR is
net-negative for the change-pattern the repo sees.
Adds `nestsClient/tests/README.md` covering:
- when to run (changes to nip53 / moq-lite session / audio
pipeline / MoqLiteNests* / ReconnectingNests* / :quic /
the sidecars themselves);
- quick-start gradle commands for hang-only, browser-only,
combined;
- prerequisites + first-run cache costs;
- all the configuration knobs (incl. the
`-DnestsHangInteropTraceRelay=true` trace-capture switch
added during the routing investigation);
- known limitations (hot-swap browser soft-pass,
framesPerGroup pin-vs-prod gap, I7 cycle 2 truncation);
- a 4-step debug recipe for triaging a flaking scenario.
Plan updates:
- `2026-05-07-t16-closure-roadmap.md` — Priority 3 marked
⏸ DEFERRED instead of ✅ CLOSED, pointing at the new README.
- `2026-05-07-cross-stack-interop-ci-gating.md` — status
changed to ⏸ DEFERRED; YAML shape preserved verbatim in the
plan for the next revisit.
- `2026-05-06-cross-stack-interop-test-results.md`,
`2026-05-06-cross-stack-interop-test-gap-matrix.md` — CI
integration § rewritten to "manual-run only" + README link.
The trace-capture instrumentation in `NativeMoqRelayHarness`
stays in place; it's useful for future flake triage even
without CI.
Self-audit of commits `d7f87971` (trace capture) and `f8dc9c59`
(hot-swap soft-pass revert) caught four issues; this commit
addresses them.
`NativeMoqRelayHarness.kt`:
- `ProcessOutputDrainer.start`: tolerate `bufferedWriter()`
failures so a misconfigured trace-log dir (parent gone, disk
full, etc.) doesn't kill the drain thread and deadlock the
relay subprocess on a full stdout pipe (~64 KB Linux pipe
buffer). Fall back to ring-only capture and System.err-warn,
matching the pattern the relay-startup error handler already
uses.
- Hoist `Regex("[^A-Za-z0-9._-]")` to `tagSanitiser` so we don't
recompile per relay boot.
- Rename `LOG_TIMESTAMP_FMT` → `logTimestampFmt` (it's a runtime
`val`, not a `const val`; existing convention is camelCase for
runtime, SCREAMING_SNAKE for compile-time constants like
`PORT_READY_TIMEOUT_MS`).
`BrowserInteropTest.kt`:
- `chromium_listener_speaker_hot_swap_does_not_crash`: prune the
`pcm.size <= warmupSamples` early-return + the trailing comment
about the skipped FFT. After the soft-pass revert the
post-warmup branch had no assertions, so the early-return was
dead code. Reduce to a single decoderErrors assertion + kdoc
spelling out the soft-pass and pointing at the hang-tier T12
counterpart.
- Update the kdoc to reflect what the test ACTUALLY asserts (it
used to claim FFT-peak coverage that no longer applies).
Other audit findings deferred (not real bugs, low priority):
- `ConcurrentLinkedQueue.size()` O(n) per line in the drainer.
- Per-line `writer.flush()` syscall — kept intentionally for
hung-test post-mortem; documented inline.
- BrowserInteropTest at 1140+ lines could split helpers — pre-
existing situation, not a regression.
Follow-up to commit `029329af`: a verification sweep (5×) showed
the `pcm.size > warmupSamples` hard floor on
`chromium_listener_speaker_hot_swap_does_not_crash` flakes 3/5 —
empirical capture varies 2880–7680 samples (= 60–160 ms TOTAL),
straddling the 4800-sample warmup threshold. The browser-side
@moq/lite 0.2.x re-attach behaviour across `Active::Ended →
Active` is fundamentally unreliable; ANY hard floor that
excludes the regression mode ("swap killed the WT session
entirely") fail-flakes because the steady state itself sits
right at that threshold.
Per `2026-05-07-tighten-cross-stack-assertions.md`'s Risk § option
(a) — "If any scenario fail-flakes after tightening, [...] revert
the tightening on that scenario" — this commit reverts the
hot-swap floor to a documented soft-pass that prints to
System.err when triggered. T12 (group sequence carry across
hot-swaps) is still asserted by the hang-tier counterpart
(`speaker_hot_swap_does_not_crash` in `HangInteropTest`), which
hard-asserts the full post-swap window decodes the 440 Hz peak.
The deferred "browser hot-swap re-attach" follow-up in
`2026-05-06-cross-stack-interop-test-results.md` captures the
underlying @moq/lite client work that would let this scenario
become a hard-floor test in the future.
T16 closure-roadmap Priority 3
(`2026-05-07-cross-stack-interop-ci-gating.md`).
Reverses commits `6829ab72` ("drop hang-interop job") and `b94737de`
("drop hang-interop + browser-interop jobs") with the path tweak
that the browser harness moved from `nestsClient-browser-interop/`
to `nestsClient/tests/browser-interop/` (commit `bd7b166f`).
Both jobs gated on `lint`, run `ubuntu-latest`, 30 min timeout each.
Linux-only — the cargo install of `moq-relay` 0.10.x has nontrivial
native deps (aws-lc-sys, ring) that take ~6 min cold + ~30 s warm;
caching is keyed on `Cargo.lock + REV`. Browser job adds bun 1.3.11
+ Playwright Chromium caches.
Stability bar:
- 5/5 sweep on HangInteropTest + BrowserInteropTest with hardened
assertions = 110/110 pass (commit `f6894792` summary).
- A second 5x sweep is in flight (target: 10/10 per plan).
5/5 sweep × 22 tests = 110/110 hard-pass post-recalibration
(commits `04be38ad` + `029329af`). Marks
`2026-05-07-tighten-cross-stack-assertions.md` and the roadmap's
Priority 2 closed; documents the per-scenario floors that landed,
including the one weaker-than-specced floor on the browser
hot-swap (deferred follow-up). Priority 3 (CI gating) is now
unblocked.
The first 5× sweep after Priority 2's tightening (commit `04be38ad`)
exposed two scenarios where my chosen thresholds didn't match the
post-merge browser path:
1. **`chromium_listener_mid_broadcast_mute_shortens_pcm`** —
the plan recommended tightening the no-mute upper bound from
5.5 s to 5.0 s. Empirical post-`:quic`-merge steady state is
~5.1–5.2 s (Chromium AudioDecoder ramp-up + harness window
padding); 5.0 s tripped 5/5 sweeps. Reverted to 5.5 s — still
excludes the 6 s "push embedded silence instead of FIN"
regression mode.
2. **`chromium_listener_speaker_hot_swap_does_not_crash`** —
the plan recommended a 0.5 s floor. Empirical post-merge sample
count is ~100–160 ms (warmup window only). Looks like
Chromium's `@moq/lite` 0.2.x client tears down its catalog/audio
subscriptions when it sees `Announce::Ended → Active` in rapid
succession instead of re-attaching. Tracked as a deferred
follow-up in `2026-05-06-cross-stack-interop-test-results.md`.
Replaced the 0.5 s floor with `pcm.size > warmupSamples`
(catches "swap killed the WT session entirely"). The hang-tier
counterpart (`speaker_hot_swap_does_not_crash`) hard-asserts the
full post-swap window decodes the 440 Hz peak, so T12
protection is intact via the hang tier; the browser tier here
only asserts the WT session survived the swap. FFT assertion
removed because the captured window is too short post-merge for
the FFT to resolve a 440 Hz peak with halfWindowHz=5.
Per the plan's Risk § option (b): widen the threshold ONLY if the
new value still excludes the regression mode the test was designed
to catch.
T16 closure-roadmap Priority 2 (`2026-05-07-tighten-cross-stack-assertions.md`).
Replaces every `if (pcm.size <= warmupSamples) return@runBlocking`
short-circuit in `BrowserInteropTest` with a sample-count
`assertTrue` floor:
- I2 late-join: `≥ 1.5 s after warmup` (was vacuous-pass on cold-launch flake)
- I3 mute-window: `≥ 2.5 s` lower bound + tightened `< 5.0 s` upper (was `< 5.5 s`)
- I4 stereo: `≥ 1 s × 2 channels` (= sample-rate × 2 floats)
- I5 hot-swap: `≥ 0.5 s after warmup`
- I9 packet-loss: `≥ 0.5 s after warmup`
- I14 decoder-no-errors: `decoderOutputs ≥ 4` (3 warmup + ≥ 1 audio)
- Browser-publish baseline + reconnect: `runBrowserPublishKotlinListen`
helper hard-asserts on listener side instead of System.err-printing
and returning early.
These were soft-passes because the pre-merge `:quic` post-handshake
bidi-drop bug produced 0-frame outcomes ~50 % of the time, and we
didn't want to fail-flake the suite while the actor was still
unidentified. Trace capture in commit `b2a42d9a` pinned that
actor on `:quic`; merging `origin/main` (commit `8f8251a5`) closed
it; commit `eea746a6` documented the closure.
Sweep verification of the hardened assertions is pending — the
hardening commit lands first so the verification sweep runs
against committed state.
Gap matrix updated to reflect the post-merge stability + hard
floors landing.
Pre-fix QlogWriter only flushed in close(); the 60s runner timeout
SIGKILLs the JVM before runTransferTest reaches its qlogWriter?.close().
On every failed quic-go transferloss, the trace ended at exactly 32768
bytes — 4 × 8KB BufferedWriter blocks — masking ~50 seconds of
late-connection behavior. Made every interop debugging session start
with "is this a connection wedge or a qlog wedge?".
Per-event flush was the original shape and was removed in 99a1a91de
because it caused multi-ms stalls on macOS Docker virtualized
filesystems (broke handshakes mid-flight). 250 ms is the compromise:
cheap enough to not stall the send path, fine-grained enough to
capture per-PTO behavior under heavy loss.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two interop-runner gaps closed in one InteropClient pass plus a
QuicConnection snapshot helper:
1. multiconnect testcase. The runner's handshakeloss /
handshakecorruption tests reuse TESTCASE_CLIENT=multiconnect — 50
sequential connections, each fetching a 1KB file under 30% packet
drop or bit-flip, with the runner verifying _count_handshakes()==50
in the pcap. Pre-fix our InteropClient dispatch returned 127 (skip)
for "multiconnect", so both tests showed as ?(L1, C1). Added
runMulticonnectTest: loops fresh socket + conn + driver + GET +
close per URL. Per-iteration qlog files at $QLOGDIR/client-N.sqlog
so a stuck iteration leaves a focused trace.
2. multiplex pacing against quic-go. Pre-fix the parallel path
chunked the URL list into fixed groups of MULTIPLEX_PARALLELISM=64.
Worked against aioquic + picoquic (initial_max_streams_bidi=128)
but blew up against quic-go (advertises 100, ramps slowly via
MAX_STREAMS_BIDI bumps): second chunk pushed cumulative used past
limit, threw QuicStreamLimitException. Now each iteration takes
min(MULTIPLEX_PARALLELISM, peerMaxStreamsBidi - used). When budget
hits 0, brief 50ms idle waits for the peer's bump.
New QuicConnection.localBidiStreamsUsedSnapshot() exposes the
consumed-side counter; combined with the existing
peerMaxStreamsBidiSnapshot() the InteropClient computes the live
available budget without holding streamsLock.
Result against quic-go: H, M, LR, L2, C2, C1 pass; was 0/7 at
session start (handshake itself failed pre-ALPN-fix), 4/6 after
key-update fix, now 6/7. Only L1 (handshakeloss) remains as
multiconnect-under-30%-drop flake (same flake picoquic shows).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
quic-go initiates a 1-RTT key update partway through every transferloss
or transfercorruption test (KEY_PHASE bit flips 0→1 around server pn=100
by default). Pre-fix our parser used the OLD application keys for every
post-update packet, AEAD-failed all of them, never sent another ACK,
the server fell into PTO mode, and throughput collapsed (~24kbps over
60s vs the 10Mbps the path supports).
The fix is end-to-end:
- ShortHeaderPacket.peekKeyPhase: HP-unmasks just the first byte to
surface the key-phase bit BEFORE running AEAD. The parser uses this
to pick the right keys instead of paying for a doomed AEAD.
- QuicConnection: tracks the live application secrets (server- and
client-side) and current send/receive key phase, plus a
previousReceiveProtection slot for RFC §6.1 reorder-window decryption.
deriveNextPhaseReceiveKeys derives the next phase via
HKDF-Expand-Label("quic ku", "", Hash.length) without committing;
commitKeyUpdate installs them only after AEAD has succeeded, then
rolls the send side forward in lockstep so our next outbound
carries the matching KEY_PHASE bit (peer needs that to confirm the
rotation completed). HP key is NOT rotated, per spec.
- QuicConnectionParser.feedShortHeaderPacket: three-way dispatch on
the peeked bit — matches current → live keys; matches retained
previous → previous keys (reordered packet); else → derive
next-phase, attempt AEAD, commit on success.
- QuicConnectionWriter: ShortHeaderPlaintextPacket(... keyPhase =
conn.currentSendKeyPhase) at both 1-RTT build sites (steady-state
and CONNECTION_CLOSE).
We don't drive key updates ourselves — only echo the peer's. Avoids
the bookkeeping for RFC 9001 §6.6 packet-count limits and the safety
benefits of voluntary rotation aren't load-bearing at our connection
scale.
Tests: peekKeyPhase round-trip + long-header rejection;
2-byte-pn round-trip when largestReceived is far behind (the original
suspected-but-not-actual cause before the key-phase reveal).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
RFC 9002 §6.2.4 says a PTO probe SHOULD retransmit unacked data, not
emit a bare PING. Two gaps in our handler surfaced via interop:
1. Handshake CRYPTO past the 1-RTT-keys-up boundary. The pre-fix
handler gated the requeue on `application.sendProtection == null`
so once 1-RTT keys were derived, our Finished (still inflight at
Handshake level until the peer ACKs it) was never retransmitted.
Lost Finished → server never confirms handshake → never sends
HANDSHAKE_DONE → connection wedges with ACK-only handshake packets
bouncing forever. Surfaced by handshakeloss against aioquic at 30%
drop rate (multiconnect iter 12 stuck at t=52s, zero handshake_done).
2. STREAM data when the peer never ACKs anything. Our loss detection
gates on `pn < largestAckedPn`, which never advances when every one
of our 1-RTT packets is dropped or corrupted en route. Surfaced by
handshakecorruption: we send H3 init streams + GET in 1-RTT pn=0,
gets corrupted, server never decrypts, never ACKs. Pre-fix the
STREAM bytes were never retransmitted; the GET stalled.
Fix: handlePtoFired now requeues inflight CRYPTO at every active
pre-application level (Initial AND Handshake) regardless of 1-RTT
state, and walks streamsList to re-queue inflight STREAM bytes when
1-RTT keys are up. requeueAllInflight is a no-op when nothing is
inflight, so calling on already-ACKed / already-discarded levels is
harmless.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
QuicConnection.alpnList → TlsClient.offeredAlpns was captured but never
threaded into buildQuicClientHello. The builder accepted only an
"additionalAlpn" parameter and hardcoded "h3" first, so our wire
ClientHello always carried just [h3] regardless of caller intent.
quic-go enforces strictly with TLS alert 120 (no_application_protocol,
CRYPTO_ERROR 376) when none of the offered ALPNs match its server
config — handshake failed at the very first server response against
quic-go's hq-interop testcases.
Replaced the awkward additionalAlpn shape with `alpns: List<ByteArray>`
(default [h3] for backward-compat) and threaded TlsClient.offeredAlpns
through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
longrtt failed against aioquic with "Expected at least 2 ClientHellos.
Got: 1" because our client started sending Initials before the sim's
ns3 + tcpdump capture finished initializing. Only the PTO retransmit
hit the wire — the original ClientHello was sent during the sim's
~1s readiness window and never captured. aioquic, picoquic, and
quic-go all gate their client launch on /wait-for-it.sh sim:57832.
We didn't.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
club.minnced:opus-java doesn't ship natives for darwin-aarch64, so the
sine-440 round-trip test failed on Apple Silicon dev machines and blocked
the pre-push hook. Catch the IllegalStateException from encoder/decoder
construction and use JUnit Assume to skip; Linux x86_64 CI keeps coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 1 of `2026-05-07-moq-relay-routing-investigation.md` ran a
5× sweep on `HangInteropTest` with per-test moq-relay trace
capture. Failure rate: 3/5, all in
`late_join_listener_still_decodes_tail`. Cross-referencing the
relay trace (`encoding self=Subscribe …catalog.json` on conn{id=0}
at T+2.07 s) with the speaker's `Log.d("NestTx")` lines (only
`ANNOUNCE inbound prefix=''` at T=0; NO `SUBSCRIBE inbound id=0`
in the failing window) shows the relay correctly forwards the
upstream SUBSCRIBE — the bidi just never reaches
`MoqLiteSession.handleInboundBidi`. So the prior plan's
"moq-relay 0.10.x per-broadcast subscribe-routing race"
hypothesis is wrong.
Re-scoped Priority 1 of the closure roadmap to point at
`:quic`'s peer-bidi surfacing path instead. The actual bug is
between QUIC's bidi-accept and `incomingBidiStreams()` flow.
Spotless-formatted Kotlin imports.
Trace artefacts preserved under
`nestsClient/plans/artefacts/2026-05-07-routing-race-disproven/`
so the next agent (QUIC owner) doesn't have to re-run the sweep.
Walks through `moq-relay/src/connection.rs`,
`moq-lite/src/lite/publisher.rs`, `moq-relay/src/cluster.rs` and
`moq-lite/src/model/{origin,broadcast}.rs` to refine the routing-race
hypotheses (`H1` / `H1b` / `H1c`). Documents that even main HEAD
`bdda6bd1` keeps `recv_subscribe`'s synchronous `consume_broadcast`
lookup; commit `bea9b3a` only added `wait_for_broadcast` as an opt-in
alternative and an explicit TODO acknowledging the race in
`moq-relay/src/web.rs:325`. So Step 4 (version bump past 0.10.25)
won't yield a fix — the most viable upstream change would convert
`recv_subscribe` to `origin.wait_for_broadcast(path).await` with a
bounded deadline. Concrete trace from Step 1 still needed to pick
between the surviving hypotheses.
`NativeMoqRelayHarness` gains a `testTag` parameter on `shared` /
`resetShared` and writes the relay subprocess's combined stdout/stderr
to `nestsClient/build/relay-logs/<tag>-<seq>-<ts>.log` whenever
`-DnestsHangInteropTraceRelay=true` is set. The subprocess runs with
`RUST_LOG=info,moq_relay=trace,moq_lite=trace,moq_native=debug` so the
per-broadcast subscribe-routing path investigated in
`nestsClient/plans/2026-05-07-moq-relay-routing-investigation.md`
(Step 1) becomes visible.
`HangInteropTest` and `BrowserInteropTest` now expose a JUnit 4
`TestName` rule and forward the method name into `resetShared`, so
each scenario's per-method log is easy to locate by name when
cross-referencing with the speaker-side `Log.d("NestTx")` lines
already captured in JUnit XML's `<system-err>`.
When the runner's compliance-check phase produces traces but the
actual testcase phase doesn't (matrix runs against multiple
testcases sometimes lose later containers' stderr), the previous
inspect script greppped the runner's stdout and silently produced
nothing.
Now: check per-testcase client/output.txt and output.txt first; fall
back to the tee'd .stdout.log narrowed to this testcase's window.
On total miss, print actionable hints (rebuild with DEBUG, run in
isolation).
The 'run-<timestamp>.stdout.log' siblings also match the 'run-*'
glob — zsh in particular returns them mixed with the actual run
dirs. The for-loop now filters to directories only.
(The summarize-matrix script was already OK — it does ls -1d
followed by a head -1, and run dirs come first in mtime order.)
Previously the script only showed server stderr — but the bug
investigation needs the CLIENT's runtime traces, which are in the
runner's tee'd stdout (${RUN_DIR}.stdout.log).
awk-narrows the lines to the segment between this testcase's
'Running test case: X' marker and the next one (each testcase runs
a fresh container, so the trace lines between markers are exactly
this testcase's run). Then dumps:
- first 50 [boot] / [interop] / [batch] / [writer.app] lines
- stream_frames=N histogram for the testcase
Useful when debugging a specific testcase failure that requires
seeing the writer's per-drain decisions.
The longrtt testcase (1 file, serial path) was failing because
client.get(authority, path) → prepareRequest() opens a stream and
queues the GET, but never calls driver.wakeup(). The data sits in
the queue until the PTO timer fires (~1 s later) — a fatal delay
on a 1.5 s RTT link with an 8 s docker-compose timeout.
The parallel path explicitly wakes after prepareRequests returns,
but the serial path was missing the equivalent nudge.
Smoking gun from the inspect-testcase output:
- 1 KB file, handshake at t=4502, response packets at t=4684/4685
- NO outgoing packet between t=4499 (ack) and t=4687 (ack) that
contained the GET request — it never went out via prepareRequest
Fix: prepareRequest in both Http3GetClient and HqInteropGetClient
now calls driver.wakeup() after enqueuing the request. The
@Suppress("UNUSED_PARAMETER") on HqInteropGetClient.driver was
also stale — it's used now.
The longrtt testcase failed at 8s with only 1.2 KB received (out of
3 MB requested). Trace from inspect-testcase:
handshake completed at t=4502 (3 RTTs at 750ms one-way as expected)
first stream byte arrived at t=4694, ~200ms after
test killed at t=8s with code=0x0 (graceful close from us)
After handshake, ~3.5s of transfer time was available before the
docker-compose timeout. With our default
initialMaxStreamDataBidiLocal = 1 MB, the peer sends 1 MB then stalls
until our parser fires a MAX_STREAM_DATA bump. At 1.5s RTT each stall
is one round-trip lost (~1.5s). For a 3 MB file that's 2 stalls = 3s
of pure flow-control idle on top of CC slow-start.
Setting initialMaxData and initialMaxStreamData{BidiLocal,
BidiRemote,Uni} to 32 MB removes flow control as a bottleneck for
the largest interop transfers (longrtt 5 MB, transfercorruption few
MB) and lets the peer's congestion control alone drive the rate.
Doesn't help handshake duration (which is RTT-bound and correct).
Doesn't help slow-start ramp (CC, server-side). Should still close
the gap on longrtt.
Usage:
./quic/interop/inspect-testcase.sh longrtt
Auto-finds the most recent run dir with the named testcase, then
emits a focused one-screen report:
- runner status line (from the tee'd .stdout.log)
- file sizes generated for that testcase
- qlog event-type histogram
- transport_parameters (peer's flow-control budget)
- connection_closed events (spec violations / explicit failures)
- last 10 sent/received packets (steady-state shape)
- first/last packet timestamps + total received count
(transfer rate hint)
- server stderr tail
8 plan files updated to reflect what's actually shipped vs. what
each doc said before:
- 2026-05-06-cross-stack-interop-test.md: 'Spec — ready to
implement' → 'Implemented and merged' with pointers to results,
gap matrix, and closure roadmap.
- 2026-05-06-cross-stack-interop-test-results.md: scenario
inventory updated to show all branches merged into
claude/cross-stack-interop-test-XAbYB; suite-flake caveats
flagged with cross-refs to the routing investigation; file
inventory now lists BrowserInteropTest + PlaywrightDriver +
the moved nestsClient/tests/browser-interop/ tree (no more 'in
sister branches not yet merged' section).
- 2026-05-06-cross-stack-interop-test-gap-matrix.md: T8/T10/T11/
T12/T13 all show hang ✅ + browser ✅; I13/I14 'NOT YET LANDED'
callouts removed; the 'in sister branch' qualifier on T13 / I7
/ I6 dropped; coverage-holes section replaced with caveat
pointers to the routing investigation.
- 2026-05-06-i4-stereo-cross-stack-scenario.md: 'Spec — ready for
implementation' → 'Landed' with PR #2755 + the three test
scenarios that ship the assertion.
- 2026-05-06-phase4-browser-harness.md: 'Spec — ready for
implementation' → 'Landed', with notes on how the implementation
diverged (used Container.Legacy.Consumer not @moq/watch; cert
pinning via serverCertificateHashes; I2/I3/I4/I13/I14 originally
deferred but later landed).
- 2026-05-06-phase4-browser-harness-results.md: status updated to
reflect 4.D CI removed per maintainer ask; layout note for
the nestsClient/tests/browser-interop/ relocation; full
scenario list reconciled with results doc.
- 2026-05-07-late-join-catalog-flake-investigation.md: status
updated to mention 00f6cba31 + 207057374 were reverted, and
link to 2026-05-07-moq-relay-routing-investigation.md as the
action plan. Adds note that the flake also affects four
browser-tier scenarios after Browser I7 landed.
- 2026-05-01-quic-stream-cliff-investigation.md: adds an HCgOY-
override banner — recommendation 2 (framesPerGroup = 5) was
superseded 4 days later by HCgOY field tests landing the prod
default at 50. Cross-refs the reconciliation + production-
rerun plans. Open follow-up #3 ('reset to 1') updated with
the same context.
No code or test changes.
Mirrors the existing nestsClient/tests/hang-interop/ layout — all
T16 cross-stack interop test infrastructure (Rust sidecars + bun
browser harness) now lives under nestsClient/tests/.
Updates:
- nestsClient/build.gradle.kts: browserInteropDir path.
- PlaywrightDriver.kt + BrowserInteropTest.kt: kdoc comments.
- All plan docs in nestsClient/plans/ that referenced the old
path.
Compiles clean post-move. No CI changes (those jobs are
intentionally not wired per 2026-05-07-cross-stack-interop-ci-gating.md;
when re-added they'll reference the new path).
Two bash-3.2 issues:
- Multiline process substitution '< <(...)' with embedded
comments triggered 'bad substitution: no closing ')''.
Reworked as imperative pushes to an array.
- 'set -u' rejects '${SEARCH_FILES[@]}' on an empty array.
Disabled '-u' since the artifact-fallback path doesn't need
any search files.
Plans the path from 'infra shipped' to 'full coverage with
correct behaviours'. Five new plan docs in nestsClient/plans/:
1. 2026-05-07-t16-closure-roadmap.md — index + priority order.
Three sequential steps + one independent track.
2. 2026-05-07-moq-relay-routing-investigation.md (Priority 1)
— root-cause the moq-relay 0.10.x per-broadcast subscribe-
routing race. Step-by-step: capture relay-side trace, write
minimum reproducer, file upstream OR bump moq-relay version.
Smoking gun + hypotheses already in place from the
late-join-flake investigation; this plan turns that into
actionable next steps.
3. 2026-05-07-tighten-cross-stack-assertions.md (Priority 2)
— once the routing race is closed, replace the five soft-
pass scenarios in BrowserInteropTest with hard floors.
Lists each scenario, its current soft-pass behavior, and
the proposed hard threshold.
4. 2026-05-07-cross-stack-interop-ci-gating.md (Priority 3)
— re-add the hang-interop + browser-interop GitHub Actions
jobs that were dropped in 6829ab727 / b94737de7. Includes
the exact YAML to restore and a 10/10 sweep stability bar
before merge.
5. 2026-05-07-framespergroup-production-rerun.md (independent
track) — re-run the HCgOY two-phone field tests against
current nostrnests production at multiple framesPerGroup
values to settle whether the test pin (5) and production
default (50) can converge.
Estimated total: 2.5-3.5 days of focused work to fully close
T16. After these four: I7 post-reconnect cliff and I12 GOAWAY
remain open as genuinely upstream-territory items, tracked in
their existing investigation docs.
Two paired changes:
(1) run-matrix.sh now tees the runner's full stdout (pre-grep-
filter) to <RUN_LOG_DIR>.stdout.log — sibling rather than
inside the log dir because run.py refuses to start if its own
--log-dir already exists.
(2) summarize-matrix.sh checks that file FIRST when searching for
'Test: X took Y, status:' lines — it has the authoritative
runner output that wasn't being saved before.
Old runs (without the tee) fall back to qlog inspection.
Some runner versions don't write 'Test: X took Y, status:' lines
into the per-testcase output.txt — they only print to the runner's
own stdout (which our run-matrix.sh consumes via grep filter and
loses). Without status lines we can still infer outcomes from
artifacts:
- qlog has a connection_closed event with a reason → FAILED with
that reason (e.g. peer CLOSE 'no CRYPTO frame')
- qlog has packets received but no close → ran something, status
genuinely uncertain
- no qlog → connection probably never made it past TLS
Tagged [inf] so users can distinguish runner-reported status from
inferred status.
The 'Test: X took Y, status: TestResult.Z' line is written by
run.py to its own stdout, not the per-testcase output.txt the
script was grepping. Scan common locations (per-testcase output.txt
fallback, run dir log files, runner-logs root) and accept the
runner's optional leading timestamp format.
Full matrix takes long enough to be vulnerable to terminal hangs,
docker OOM, or just user CTRL-C. Per-testcase output.txt files
hold status lines we can scrape without re-running anything.
Usage:
./quic/interop/summarize-matrix.sh
Output:
TESTCASE RESULT TIME
-------------------- --------------- ----------
handshake ✓ SUCCEEDED 2.5s
transfer ✓ SUCCEEDED 3.1s
multiplexing ✓ SUCCEEDED 5.2s
retry ✕ FAILED 8.3s
ecn ? UNSUPPORTED 2.6s
Helps iterate when the matrix is too long to run end-to-end on
macOS Docker.
The smoking gun from the 2026-05-07 multiplex run boot log:
[boot] DEBUG=1; ...; TESTCASE=transfer; ROLE=client
[boot] transfer mode: parallel=false urls=1999
quic-interop-runner sets TESTCASE_CLIENT=transfer for ALL the
transfer-family testcases (transfer, multiplexing, transferloss,
transfercorruption). Discrimination between transfer (1 file) and
multiplexing (~2000 files) happens by URL count, NOT by TESTCASE
name — so our check `parallel = (testcase == "multiplexing")` was
always false, even for the multiplexing test, and we always took
the serial fallback path: client.get(authority, path) per URL,
opening one stream + awaiting before the next. That's why the wire
showed exactly one stream per RTT for the entire 60s.
Fix: parallel = (token count of REQUESTS env var) > 1. Effectively:
- 1 URL → transfer testcase, serial path
- >1 URL → multiplexing testcase, batched-parallel path
Verified the writer's batched coalescing already works in unit
tests (MultiplexingCoalescingTest, MultiplexingAioquicTpsTest both
green at ~9 streams/packet). With this dispatch fix, the live
runner should finally reach the batched code path.
Bumped WRITER_DEBUG_BUILD_ID so the next [boot] line confirms
the fix is deployed.
Hypothesis: the [interop] / [batch] logs are missing because we're
hitting the SERIAL branch (parallel=false), not the parallel one —
which would explain perfectly the writer trace pattern:
- streamsView grows by 1 every 2-3 drains
- active stays at 6 or 7 (3 H3 init + at most 4 chunk streams)
- one stream per RTT cadence on the wire
That's exactly what client.get(authority, path) looks like in
sequence. parallel=true would call prepareRequests for chunks of 64.
Two new unconditional log lines (low-volume, control-flow only,
NOT in hot paths):
1. [boot] now includes TESTCASE and ROLE — to verify the runner
is sending TESTCASE=multiplexing as expected
2. [boot] transfer mode: parallel=BOOL urls=N — confirms which
branch we took
If parallel=false despite TESTCASE=multiplexing, the bug is in our
testcase-to-parallel mapping (line 192 of InteropClient.kt).
If parallel=true but [interop]/[batch] still missing, the bug is
elsewhere.
Four sketches, queued by impact, each grounded in current code paths
and observed benchmark numbers:
- event-ingestion-batching: SQLite group commit + EVENT pipelining +
off-thread Schnorr verify. Targets 5–10× EPS on a fast SSD.
- live-broadcast-fanout-index: indexed filter matching to replace the
O(N_subs × N_filters) per-event walk in LiveEventStore. Targets
flat fanout p99 up to high subscriber counts.
- connection-scaling: shrink the per-session outQueue footprint
(currently the dominant per-conn cost), tune Ktor CIO group sizes,
reduce JSON parse allocations. Targets 10 000+ concurrent conns.
- negentropy-large-corpus: id-and-time-only snapshot path so NEG-OPEN
on a 5M-event store doesn't materialise full Event objects, plus
bounded-window defaults and concurrent-session caps.
Each plan names the verification benchmark to add. Plans are queued,
not committed work — README orders them by expected impact.
The [batch]/[interop] traces aren't appearing in the user's runs
even after rebuild. To distinguish 'env var not set' from 'binary
doesn't have the code', emit a [boot] line at startup that:
1. Reports the QUIC_INTEROP_DEBUG env var value
2. Reports whether writerDebugEnabled was flipped on
3. Includes a build-id constant from WriterDebug.kt
If the user's run shows '[boot] DEBUG=1; ... build_id=2026-05-07-
batch-log-v1', we know the latest debug code IS deployed. If the
boot line is missing OR shows an older build_id, the docker image
served stale bytecode (gradle/docker layer caching) and a clean
rebuild is needed.
Inspect script now greps [boot] and surfaces it BEFORE the rest of
the writer-side debug section.
Closes the last gap in the T16 browser-tier coverage. Adds:
publish.ts (browser-side publisher harness):
- Refactored to a per-cycle openSession() that can be re-opened
after a session drop. Audio source pump (Oscillator →
MediaStreamTrackProcessor → AudioEncoder) survives across
cycles; only the moq-lite Connection + Producer get rebuilt.
- New ?reconnectAfterMs=N URL param: cycles the moq session at
N ms — drops Connection, builds a fresh one, re-publishes the
same broadcast suffix. Relay sees Announce::Ended → Active.
- dst.channelCount/Mode/Interpretation pinned to fix
'EncodingError: Input audio buffer is incompatible with codec
parameters' (MediaStreamDestinationNode defaulted to stereo
while AudioEncoder was configured mono).
- serverCertificateHashes pinning via ?certSha256= URL param.
Same channel as listen.ts.
- Exposes window.__framesIn (encoded frames pumped) and
window.__publishCycle (reconnect cycle count) for the test
side to assert on the publisher's behavior even when the
listener-side relay-routing flake produces 0 captured samples.
PlaywrightDriver.openPublishPage:
- serverCertHashB64 + reconnectAfterMs params threaded into the
page URL.
harness.spec.ts: framesIn + cycles meta fields surfaced.
NativeMoqRelayHarness.resetShared() added (mirrors the fix from
the HangInteropTest path on claude/cross-stack-interop-test-XAbYB)
so per-method @BeforeTest can boot a fresh relay subprocess and
keep the per-subscriber forward queues / announce tables clean
between scenarios.
BrowserInteropTest:
- @BeforeTest gate() now calls resetShared().
- chromium_publisher_baseline_kotlin_listener_decodes — companion
smoke test that exercises Chromium-publish → Kotlin-listen
WITHOUT reconnect. Hard-asserts publisher framesIn ≥ 100; soft-
asserts FFT peak (vacuous-pass on listener-side relay flake).
- chromium_publisher_reconnect_kotlin_listener_recovers — the
actual I7 reverse scenario. Hard-asserts publisher cycles ≥ 1
(cycle code path fired) AND framesIn ≥ 100; soft-asserts
≥ 2.5 s of decoded mono PCM with 440 Hz peak.
- Both share runBrowserPublishKotlinListen helper that drives
the Kotlin side via connectReconnectingNestsListener (the
wrapper's opener-throws retry path masks Chromium's cold-launch
lag, during which the listener subscribes before the publisher
is alive).
- Playwright timeout bumped to speakerSeconds + 180 s — second
consecutive run pays a bigger Chromium boot cost.
Coverage status: each new test passes individually. Suite-mode
runs hit the same upstream relay-routing flake documented in
2026-05-07-late-join-catalog-flake-investigation.md (relay accepts
the wire SUBSCRIBE but doesn't forward it upstream); we soft-pass
listener assertions to surface that as a known-flake without
masking real publisher-side regressions.
set -euo pipefail kills the script on no-match grep. The run-09:45:21
output.txt has [writer.app] lines but no [batch] / [interop] (those
were added in a later commit). The empty subset grep aborted the
script silently after printing the section header.
Previous version only grepped '\[writer' so the [batch] entry/exit
logs and [interop] chunk-size logs were silently filtered out. Now
shows them BEFORE the [writer.app] dump so they appear at the top.
Claude