amethyst/nestsClient/plans/2026-05-07-moq-relay-routing-investigation.md

# Plan: investigate moq-relay 0.10.x per-broadcast subscribe-routing race

**Status:** specced — pickup ready.

**Owns:** the residual flake that affects four T16 scenarios:
`late_join_listener_still_decodes_tail`,
`packet_loss_1pct_does_not_kill_audio`,
`long_broadcast_60s_tone_round_trips`, and the new
`chromium_publisher_*_kotlin_listener_recovers` tests in browser-tier.

**Blocks:** CI gating for `:nestsClient:jvmTest -DnestsHangInterop=true`
and `-DnestsBrowserInterop=true`. Re-evaluate the
`hang-interop` / `browser-interop` workflow jobs once this is closed.

**Cross-refs:**
- `nestsClient/plans/2026-05-07-late-join-catalog-flake-investigation.md`
  (smoking-gun trace + 4 mitigation attempts, 2 of which were
  net-negative and reverted).
- `nestsClient/plans/2026-05-07-i7-post-reconnect-cliff-investigation.md`
  (same kind of routing issue surfacing across publisher cycles).

## What we know

For broadcasts that fail (sample suffixes from the trace:
`10d4b6f2…`, `c75e2648…`, `f1be27ef…`):

1. The Kotlin speaker side logs:
   - `ANNOUNCE inbound prefix='' → emitted Active suffix='<broadcast>'`
   - …then NOTHING for the entire test window.
   - Audio publisher's `send()` repeats `no inboundSubs` at 50 fps
     until the test times out.

2. The Rust hang-listen side logs:
   - `connected, version=moq-lite-03`
   - `broadcast announced path=<broadcast>`
   - `subscribe started id=0 broadcast=<broadcast> track=catalog.json`
   - …then `subscribe error err=remote error: code=0` exactly when
     the speaker tears down at the broadcast-window end (= relay
     forwarding `Cancel`).

The relay accepts the listener's wire SUBSCRIBE on its downstream
connection but **never opens an upstream SUBSCRIBE bidi to the
speaker** for the failing broadcast. The upstream subscribe-pump
that's supposed to forward downstream subscribes to the speaker
isn't wired up by the time the listener subscribes.

For broadcasts that succeed (same trace, same JVM, different test):

```
ANNOUNCE inbound prefix='' → emitted Active suffix=<broadcast>
SUBSCRIBE inbound id=0 broadcast=<broadcast> track='catalog.json'
SUBSCRIBE registered id=0 …
openGroupStream subId=0 seq=0
…
```

All log lines fire; the relay forwards the upstream subscribe
within ~1 ms of the downstream subscribe. Failure mode is binary:
the relay does or does not forward.

## Hypotheses, ranked by next step

### H1 — moq-rs 0.10.x bug in `Origin::announced()` → upstream-pump setup race

`Origin::announced().await` returns the broadcast as soon as the
speaker's announce lands in the relay's origin map. The relay's
upstream-subscribe pump for that broadcast is set up on a separate
async path. If a downstream listener subscribes before the pump is
fully wired, the SUBSCRIBE accepts on the listener's wire (the
relay has the broadcast in its origin) but never propagates
upstream.

**Status:** prime suspect; see "smoking gun" in
`2026-05-07-late-join-catalog-flake-investigation.md`.

### H2 — interaction with the `--auth-public ""` minimal config

The harness boots moq-relay with `--auth-public ""` to skip JWT
issuance. Production runs with full auth. It's possible the
auth-public path takes a different code path through the relay's
origin/subscribe wiring that's racier than the auth'd path.

**Status:** plausible; would explain why the flake isn't reported
against the production deployment.

### H3 — local-only timing race that resolves at higher latency

Loopback (127.0.0.1) has near-zero RTT. The relay's internal
async setup may rely on the natural RTT cushion of a real network
to sequence upstream-subscribe-pump setup vs. downstream-subscribe
acceptance. We bypass that cushion in the test.

**Status:** less likely (the cliff plan's evidence shows lossy
network actually makes things *worse* via the `serve_group` task
pool) but worth ruling out.

## Investigation plan

### Step 1 — capture relay-side traces

`NativeMoqRelayHarness.boot` currently launches `moq-relay` with
`RUST_LOG=info`. Bump to `RUST_LOG=moq_relay=trace,moq_lite=trace`
and capture stderr to a per-test tempfile. Cross-reference with
the failing test's hang-listen stdout AND the speaker-side
`Log.d("NestTx")` traces (already captured in
`<system-err>` per JUnit XML).

Concretely: in `NativeMoqRelayHarness.kt` add a `--log-stderr`
option that the @BeforeTest hook sets to a `<test-method>.log`
path under `nestsClient/build/relay-logs/`. The Kotlin side
already has the speaker-side traces; the Rust side is the gap.

What to look for in the failed-broadcast log:
- Was a SUBSCRIBE bidi opened to the speaker for the failing
  broadcast suffix? (moq_lite span: `subscribe`).
- Did the relay's `Origin::publish_broadcast` call complete
  before the listener's SUBSCRIBE arrived?
- Any `track.unused()` resolves on the publisher-side track that
  would explain immediate cancellation?

### Step 2 — write a minimal reproducer

If Step 1 shows the bug is independent of our test framework,
extract a minimum reproducer:

```rust
// reproducer.rs
let mut cmd = std::process::Command::new("moq-relay")
    .args(&["--server-bind", "127.0.0.1:0", "--auth-public", "",
            "--tls-generate", "localhost"])
    .spawn()?;
// Run a moq-lite SPEAKER on one client, a moq-lite LISTENER on
// another, both pointed at the relay. Listener subscribes immediately
// after the speaker announces. Repeat 100×; count how many succeed.
```

Then strip the SPEAKER's announce timing, the LISTENER's subscribe
timing, the relay's `--auth-public` flag — bisect to the smallest
form that still reproduces.

### Step 3 — file upstream

If Step 1 / 2 confirm a moq-rs bug, file a `kixelated/moq` issue
with:
- The reproducer.
- Smoking-gun trace pair from our test harness.
- Pin to moq-rs version `0.10.25` (per `nestsClient/tests/hang-interop/REV`).
- Cross-link to existing
  `2026-05-01-quic-stream-cliff-investigation.md`'s open follow-up
  #1 (the per-subscriber forward-queue cliff is a sister bug).

### Step 4 — try newer moq-relay version

Bump `MOQ_RELAY_VERSION` in `nestsClient/tests/hang-interop/REV`
and `nestsClient/build.gradle.kts` to the next minor release on
crates.io (whatever's current at the time of pickup). Run the 5×
sweep. If the flake disappears, the upstream may have already
fixed it; we can pin past 0.10.x.

**Risk:** newer moq-relay versions may have wire-format changes
that break our current `moq-lite-03` ALPN pin. The browser
harness's `@moq/lite` 0.2.x client offers `moq-lite-04` AND
`moq-lite-03`, so a newer relay that drops `03` would still
negotiate fine via 04.

## Acceptance criteria

- Sweep `for i in 1 2 3 4 5; do ./gradlew :nestsClient:jvmTest
  --tests HangInteropTest -DnestsHangInterop=true --rerun-tasks; done`
  passes 5/5.
- Browser-tier sweep similarly stable.
- Either:
  (a) Upstream issue filed with reproducer (if the bug is in
      moq-rs and we can't fix it locally), OR
  (b) Local fix applied (e.g. version bump + REV update +
      Cargo.lock regenerate).

## Out of scope

- The `:quic` module's `MAX_STREAMS_UNI` extension fix
  (`d391ae1d`) — already shipped, separate concern.
- The production-side `framesPerGroup` reconciliation
  (`2026-05-07-framespergroup-production-rerun.md`) — independent.