amethyst/nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md

# QUIC stream cliff against nostrnests.com — investigation plan

**⚠️ Recommendation 2 (cadence tuning) was overridden 4 days
later.** Subsequent two-phone production tests on
`claude/fix-nests-audio-receiver-HCgOY` (commit `a36ccb569`,
2026-05-05) showed `framesPerGroup = 5` itself cliffs after ~13 s
on the same production deployment, just slower than `framesPerGroup
= 1`'s 3 s. Production now defaults to `framesPerGroup = 50`
(1 stream/sec). The interop tests still pin `5` because the local
`moq-relay 0.10.25 --auth-public ""` minimal setup hits a *different*
cliff (per-stream byte volume) at 50.

Both values are correct in their own environments. See
`nestsClient/plans/2026-05-07-framespergroup-reconciliation.md`
for the full reconciliation. A planned re-run of the HCgOY field
tests against the current production deployment is queued in
`nestsClient/plans/2026-05-07-framespergroup-production-rerun.md`
to settle whether the two rigs can converge.

**Status: PRODUCTION-FIXED via two-layer fix.** The investigation closed
with one true bug fix in `:quic` and one tuning change in
`NestMoqLiteBroadcaster`. A third layer — exposing moq-lite's
`max_latency` knob to listener callers — is now wired through the
public API so future tuning can happen at the call site without
touching the listener internals.

  1. **Bug fix.** `:quic` now emits `MAX_STREAMS_UNI` extensions to
     widen the listener's peer-initiated stream-id cap (commit
     `d391ae1d`). Closes the original 99-frame cliff for short
     broadcasts. Pre-fix, `:quic` only ever *parsed* inbound
     `MaxStreamsFrame` — it never sent the outbound counterpart, so the
     peer's stream-id cap stayed at our [QuicConnectionConfig.initialMaxStreamsUni]
     default (`100`) for the lifetime of the connection. The relay
     forwards each broadcast group as a fresh peer-initiated uni
     stream, so any broadcast longer than 100 frames silently truncated
     at the listener.

  2. **Cadence tuning.** `NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP = 5`
     packs five Opus frames per moq-lite group, dropping client
     uni-stream creation from 50/sec to 10/sec — comfortably under the
     production nostrnests relay's sustained per-subscriber forward
     ceiling of ~40 streams/sec, which the moq-lite spec explicitly
     leaves to relay implementations to size. Closes the long-broadcast
     residual where the relay's per-subscriber queue would overflow
     after 6–14 seconds of continuous push and silently terminate
     forwarding to that subscriber.

  3. **API knob.** [NestsListener.subscribeSpeaker] now takes a
     `maxLatencyMs: Long = 0L` parameter, plumbed into the moq-lite
     SUBSCRIBE frame's `max_latency` field. `0` (default) preserves
     prior behavior — the relay falls back to its own `MAX_GROUP_AGE
     = 30 s`. Low-latency live audio callers can set e.g. `500L` so
     the listener prefers fresh frames over stale buffered ones during
     transient relay backpressure. Per the moq-lite spec, the relay
     aggregates `max_latency` across subscribers with `0` winning, so
     a single legacy listener wedges everyone — but for a fully-tuned
     deployment this is the on-wire knob the spec gives us.

Listener-side flow-control snapshots in the round-2 sweep confirmed
the residual was strictly relay-side: every uni stream the relay
opened to the listener delivered cleanly to the application
(`peerInitiatedUni == received + 1`, where `+1` is the WT control
stream); `pendingBytes == 0`; cap headroom unused. The relay simply
stopped opening new streams to the listener once its per-subscriber
buffer overflowed. Reducing the stream-creation rate to 10/sec keeps
the relay's queue from ever filling.

Late-join latency cost from `framesPerGroup = 5`: ≤ 100 ms initial
audio gap (one group boundary) instead of ≤ 20 ms with `framesPerGroup
= 1`. Imperceptible for live audio.

## Decisions vs. bugs — the spec audit

The earlier draft of this plan called the relay's behavior a stack of
"upstream bugs". A spec audit (moq-transport-17, moq-lite Lite-03,
RFC 9000) showed that's not accurate. The honest framing:

- **Our `:quic` not emitting `MAX_STREAMS_UNI` was a real bug** —
  RFC 9000 §4.6 / §19.11 require the receiver to extend the peer's
  stream-id cap by sending `MAX_STREAMS` once existing streams have
  been consumed. Not optional. Fixed in `d391ae1d`.

- **The relay's per-subscriber forward queue policy is a deliberate
  architectural decision in a spec gap, not a bug.** The moq-lite
  Lite-03 spec's "Congestion" section explicitly says: *"MoQ puts
  each subscriber in control"* via the per-subscribe `max_latency`
  field. The relay is **expected** to keep streams flowing while
  groups are within tolerance, and to evict groups (not subscribers)
  when they age past tolerance. moq-rs's choices —
  - unbounded per-track group queue with age-only eviction
    (`MAX_GROUP_AGE = 30 s` default),
  - `serve_group()` blocking on `open_uni().await` with no per-task
    timeout,
  - `FuturesUnordered` task pool for forward tasks (unbounded),
  - no proactive lagging-consumer detection / RESET / STOP_SENDING
  — are all spec-permitted (`MAY`-level) implementation choices. They
  collectively *can* starve a subscriber whose downstream is
  temporarily slow, but the spec puts the burden of fixing that on
  the subscriber via `max_latency`, not on the relay. Calling these
  "bugs" in the prior draft was incorrect.

- **Our prior shipped behavior of `max_latency = 0` (= unlimited) on
  every SUBSCRIBE was the missing piece.** With `0`, the relay was
  doing exactly what the spec asks: hold groups for up to its
  `MAX_GROUP_AGE` default, then evict. We never told it we cared about
  freshness over completeness. The new `subscribeSpeaker(pubkey,
  maxLatencyMs = …)` parameter lets a caller fix that — though the
  defaults stay at `0` for back-compat with the JS reference watcher.

- **Worth filing upstream:** an issue at `kixelated/moq` *suggesting*
  (not asserting as a bug) that the relay expose its per-subscriber
  queue depth as a config knob, and that `serve_group()` `open_uni()`
  take a deadline derived from the active subscriber's smallest
  `max_latency`. This is a feature request, not a defect report.

## Source audit (moq-rs 0.10.25)

Read against the version production nostrnests is running
(`moq-relay 0.10.25-5063dbfe`):

1. **Per-track group queue is unbounded.**
   `moq/rs/moq-lite/src/model/track.rs:69-90` —
   ```rust
   groups: VecDeque<Option<(GroupProducer, tokio::time::Instant)>>,
   ```
   Inserts are unconditional (subscriber.rs:363-393); only eviction
   is age-based at `MAX_GROUP_AGE = 30 s` (track.rs:29). When a
   subscriber falls behind, groups pile up indefinitely until they age
   out 30 s later — which is what the spec wants if `max_latency = 0`.

2. **`serve_group` blocks on `open_uni().await` with no timeout.**
   `moq/rs/moq-lite/src/lite/publisher.rs:317-379` —
   ```rust
   let stream = session.open_uni().await.map_err(Error::from_transport)?;
   ```
   No `select!` against a deadline. If the subscriber's Quinn CWND
   has collapsed or its advertised `MAX_STREAMS_UNI` is exhausted,
   this `await` blocks the task indefinitely. Spec-permitted but
   user-hostile under sustained load.

3. **Unbounded task pool feeding the awaits.**
   The publisher pushes blocked `serve_group` tasks into a
   `FuturesUnordered` (publisher.rs:325, 346). The receive loop (line
   334) keeps spawning more serve tasks as upstream groups arrive.
   No backpressure path back to the publisher to slow upstream
   ingestion. Combined with (1)+(2), once any in-flight write blocks,
   queued tasks pile up and the per-track queue ages everything out.

4. **`max_concurrent_uni_streams = 10000`** in
   `moq/rs/moq-native/src/quinn.rs:30-33` (override of Quinn's
   default 1024). High enough that we never hit the connection's
   stream-id cap on our listener side at typical audio rates —
   confirmed by our `peerMaxStreamsUniNow=10000` snapshots — so the
   stall is *not* about Quinn's stream-concurrency limit.

5. **No proactive "lagging consumer" detection.** The relay never
   gives up on a slow subscriber via RESET / STOP_SENDING. From the
   subscriber's POV the subscription stays "active",
   `peerInitiatedUni` is frozen, and the connection stays alive —
   exactly the symptom we observed.

These are architectural choices, not spec violations. The
`framesPerGroup = 5` mitigation works because it cuts the upstream
publisher rate to ~10 streams/sec — slow enough that even a
temporarily-stalled subscriber's queue drains before any of (1)/(2)/(3)
accumulate beyond recovery.

## What `flowControlSnapshot` proved (the smoking gun)

After wiring [QuicConnection.flowControlSnapshot] into [SendTraceScenario],
running `sweep_frames_200` against production produced:

```
fc-pre:        peerInitMaxData=4611686018427387903   peerInitMaxStreamDataUni=1250000
               peerInitMaxStreamsUni=10000           sendCredit=4611686018427387903
               consumed=786                          peerMaxStreamsUniNow=10000
               nextLocalUniIdx=1                     pendingBytes=0   pendingStreams=0/4
fc-post-pump:  sendCredit=4611686018427387903        consumed=18729
               peerMaxStreamsUniNow=10000            nextLocalUniIdx=201
               pendingBytes=0                        pendingStreams=0/205
fc-post-grace: (identical to fc-post-pump)
sub[0]:        received=99/200                       firstSentButLost=99
```

This rules out every speaker-side hypothesis at once:

- **`peerInitMaxData = 2⁶²−1`** ⇒ ~5 EB connection-level send credit;
  we used **18 KB**. Not a connection-level flow-control wedge.
- **`peerInitMaxStreamsUni = 10000`, `nextLocalUniIdx = 201`** ⇒
  speaker opened all 200 broadcast streams + 1 control stream, well
  under cap. Not a peer-granted stream-id wedge.
- **`pendingBytes = 0`, `pendingStreams = 0/205`** ⇒ no bytes are
  stuck in any stream's send buffer; everything we wrote made it past
  the writer.
- **`consumed` is identical between post-pump and post-grace** ⇒ we
  weren't even *trying* to send more after the pump finished.

So the speaker → relay path was healthy. The 99-frame ceiling on the
listener side had to be on the **relay → listener** half of the
connection.

Listener-side, our [QuicConnection] advertises `initial_max_streams_uni
= 100` at handshake. The relay can open up to 100 uni streams to us
*for the lifetime of the connection* unless we send `MAX_STREAMS_UNI(N)`
to extend it. Grepping `:quic` for `MaxStreamsFrame` showed it was only
ever *parsed inbound* — no outbound emission. So the peer's stream-id
allowance was capped at 100 forever. Production validation:
`sweep_frames_200` went from `received=99/200` (pre-fix) to
`received=200/200` (post-fix).

## Sweep evidence (production, before mitigation)

`NostrnestsProdAudioTransmissionTest` (commit `2da18859`-era, before
any QUIC changes), one row per scenario:

| Scenario | streams opened | `received` | cliff |
|---|---|---|---|
| `sweep-frames50` | 50 | 50/50 | none ✅ |
| `sweep-baseline` 100 frames | 100 | 99/100 | last 1 |
| any cadence (5/10/40/80/200 ms) at 100 frames | 100 | 99/100 | last 1 |
| any payload (80 B / 1 KB / 4 KB) at 100 frames | 100 | 99/100 | last 1 |
| `sweep-frames200` | 200 | **99**/200 | hard at 99 |
| `sweep-frames400` | 400 | **99**/400 | hard at 99 |
| `sweep-30s` (1500 frames) | 1500 | **99**/1500 | hard at 99 |
| `sweep-120s` (6000 frames) | 6000 | 52/6000 | hard at 52 |
| `sweep-burst` (cadence 0) | 100 | 65/100 | hard at 65 |
| `sweep-payload-16kb` 100 frames × 16 KB | 100 | 49/100 | hard at 49 |
| **`sweep-frames-per-group-5`** | **20** groups | **100/100** | none ✅ |
| **`sweep-frames-per-group-20`** | **5** groups | **100/100** | none ✅ |
| **`sweep-frames-per-group-all`** | **1** group | **100/100** | none ✅ |

The signal is unambiguous: **fewer streams → no loss, regardless of
frame count, payload, or duration**. The cliff scaled with stream-
opening rate, not byte volume or wall-clock time.

## Sweep evidence (after MAX_STREAMS_UNI fix)

Full prod sweep (all 27 scenarios, commit `c0269859`):

**Now perfect (100% received):** `baseline 100×20ms`, `frames200`,
`frames50`, `1kb`, `4kb`, `cad10`, `cad40`, `cad80`, `cad200`,
`2subs` (both subs), `slowconsumer`, `fpg5`, `fpg20`, `fpg-all`.
14 of 27 scenarios.

**Improved but still partial (residual relay-side loss):**

| Scenario | Pre-fix | Post-fix | Pattern |
|---|---|---|---|
| `frames400` | 99/400 | 368/400 | last 32 frames lost (tail) |
| `30s` (1500 frames) | 99/1500 | 653/1500 | cliffed at frame 653 mid-pump |
| `120s` (6000 frames) | 52/6000 | 61/6000 | cliffed at frame 61, t≈2 s |
| `burst` cadence=0 | 65/100 | 96/100 | last 4 lost (tail) |
| `pause` 50+5 s+50 | 99/100 | 32/100 | regressed; pause kills it |
| `cad5` 100×5 ms | 99/100 | 54/100 | regressed (faster burst) |
| `3subs` sub[2] | varied | 77/100 | one sub lags |

The cliff is no longer at "exactly 99 streams" — it's now timing/load-
sensitive. The speaker side remains pristine (`fc-post-pump`:
`consumed > 0`, `pendingBytes = 0`, `nextLocalUniIdx` matches frame
count) — the loss is on the relay→listener path. Closing the residual
required the `framesPerGroup = 5` cadence change.

## Round 2 — chasing the residual (preserved for context)

Three test-side changes ruled out collector-side biases before
concluding the residual was relay-side:

### 2a. Test-side stdout serialisation

`SendTraceScenario.Scenario.verbosePerFrame` defaulted to `true`,
which logged a per-frame `tx i=…` and `rx[idx] gid=…` line via
`InteropDebug.checkpoint`. Under JUnit's stdout capture, that's a
synchronous write per event. At 50 frames/sec sustained the capture
thread can serialise the receive coroutine and starve the QUIC read
loop, biasing the `received` count downward. **Default flipped to
`false`**; specific tests opt in for debugging.

### 2b. Test-side `CopyOnWriteArrayList` add cost

The collector accumulated arrivals in a `CopyOnWriteArrayList`, which
is O(N) per add. For 1500-frame runs that's ~1.1 M element copies and
~35 MB of memory ops cumulative. Replaced with
`Collections.synchronizedList(ArrayList(capacityHint))` for O(1)
amortized adds.

### 2c. Listener-side kernel UDP receive buffer

`UdpSocket.connect` did not configure `SO_RCVBUF`, so the kernel's
default applied (~200 KB on Linux/macOS). At MTU-sized datagrams
that's room for ~130 packets. Bumped to 4 MiB at socket-bind time
(Linux doubles + clamps via `rmem_max`).

Added lifetime UDP datagram counters (`receivedDatagramCount`,
`receivedByteCount`, `receiveBufferSizeBytes`) to the `UdpSocket`
expect surface; the driver wires them into the existing
[QuicConnection.flowControlSnapshot] via a `udpStatsSupplier` hook.
The `fc-pre / fc-post-pump / fc-post-grace` lines now include
`udpDatagrams=…  udpBytes=…  udpRcvBuf=…` so future sweeps can
correlate "frames missing on subscription" against "datagrams the
kernel actually delivered".

None of these fully closed the residual — the cliff after round-2 was
still relay-side, finally addressed by `framesPerGroup = 5`.

## Symptom (preserved for context)

When the speaker pumps Opus frames at the production 20 ms cadence
and packs **one frame per moq-lite group** (= one client-initiated
QUIC unidirectional stream per Opus frame), the listener stops
receiving frames somewhere around the 99th frame. From there to
end-of-broadcast, **`MoqLitePublisherHandle.send` keeps returning
`true` for every subsequent frame**, so the application has no signal
that anything is wrong; the audio simply cuts out a few seconds in.

The bug is invisible to:

- `NestMoqLiteBroadcaster`'s outer `runCatching {…}.onFailure {…}`
  (no exception is thrown on the silent-loss path).
- `MoqLitePublisherHandle.send`'s return value (always `true` past
  the cliff).
- `QuicConnection`'s peer-cap check.

**Reproduces against:** `https://moq.nostrnests.com:4443` (production
relay).

## Reproducing / verifying

```bash
./gradlew :nestsClient:jvmTest \
  --tests com.vitorpamplona.nestsclient.interop.NostrnestsProdAudioTransmissionTest \
  -DnestsProd=true -DnestsInteropDebug=true --rerun-tasks
grep -E "sub\[|missing=" \
  nestsClient/build/test-results/jvmTest/TEST-com.vitorpamplona.nestsclient.interop.NostrnestsProdAudioTransmissionTest.xml \
  | sort
```

Acceptance criterion: every sweep row showing `received=N/N
missing=[]`, except the explicit late-join scenarios
(`sweep_late_subscribe_after_25` / `_after_50`) which legitimately
report `received < N` due to from-latest semantics.

## Open follow-ups

1. **File a feature request at `kixelated/moq`** describing the
   per-subscriber forward-queue cliff and proposing (a) per-deployment
   tuning of the unbounded `FuturesUnordered` task pool, and (b) a
   deadline on `serve_group()`'s `open_uni().await` derived from the
   active subscriber's smallest `max_latency`. This is a feature
   request, not a bug report.

2. **Surface `maxLatencyMs` in the audio-room VM/UI** if a future
   product requirement wants per-listener freshness preference. The
   listener wire path now supports it; nothing in the current
   `NestViewModel` consumes it.

3. **Reset `DEFAULT_FRAMES_PER_GROUP` back to 1** if a future relay
   version ships either a config knob or a fix for the per-subscriber
   forward starvation. The 100 ms late-join initial gap is the only
   remaining audio-quality tradeoff from the current mitigation.

   **Update 2026-05-05:** the value did NOT stay at `5` — HCgOY field
   tests overrode it to `50`. See the banner at the top of this
   doc. The "reset to 1" follow-up still applies in spirit (use the
   smallest value that doesn't cliff) but the right value is now
   data-dependent on whichever production deployment is being
   targeted. Tracked in
   `nestsClient/plans/2026-05-07-framespergroup-production-rerun.md`.