amethyst/nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md

# QUIC stream cliff against nostrnests.com — investigation plan

**Status: PRODUCTION-FIXED via two-layer fix.**

  1. `:quic` now emits `MAX_STREAMS_UNI` extensions to widen the
     listener's peer-initiated stream-id cap (commit `d391ae1d`).
     Closes the original 99-frame cliff for short broadcasts.

  2. `NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP = 5` packs five
     Opus frames per moq-lite group, dropping client uni-stream
     creation from 50/sec to 10/sec — comfortably under the
     production nostrnests relay's sustained per-subscriber forward
     ceiling of ~40 streams/sec. Closes the long-broadcast residual
     where the relay's per-subscriber queue would overflow after 6–14
     seconds of continuous push and silently terminate forwarding to
     that subscriber.

Listener-side flow-control snapshots in the round-2 sweep confirmed
the residual was strictly relay-side: every uni stream the relay
opened to the listener delivered cleanly to the application
(`peerInitiatedUni == received + 1`, where `+1` is the WT control
stream); `pendingBytes == 0`; cap headroom unused. The relay simply
stopped opening new streams to the listener once its per-subscriber
buffer overflowed. Reducing the stream-creation rate to 10/sec keeps
the relay's queue from ever filling.

Late-join latency cost: ≤ 100 ms initial audio gap (one group
boundary) instead of ≤ 20 ms with `framesPerGroup = 1`. Imperceptible
for live audio.

The remaining open work is **upstream**: file an issue at
`kixelated/moq` describing the per-subscriber forward queue
limitation so future relay versions can either lift the ceiling or
expose it as a config knob.

**Original root cause: FIXED.** Our `:quic` client never emitted
`MAX_STREAMS_*` frames to extend the peer-initiated stream-id cap.
[QuicConnectionConfig.initialMaxStreamsUni] (default `100`) was the
*lifetime* maximum the peer could ever open. The relay forwards each
broadcast group as a fresh peer-initiated uni stream, so any
broadcast longer than 100 frames silently truncated at the listener.
Fixed by tracking [QuicConnection.peerInitiatedUniCount] and emitting
`MAX_STREAMS_UNI(newCap)` from [appendFlowControlUpdates] once the
peer's usage crosses the half-window threshold — same pattern as the
existing `MAX_DATA` / `MAX_STREAM_DATA` extension.

Production validation: `sweep_frames_200` against `nostrnests.com`
went from `received=99/200 missing=[99-199]` (pre-fix) to
`received=200/200 missing=[]` (post-fix). The
[NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP] mitigation has
been reverted from `5` back to `1` so the broadcaster matches the
JS reference's wire shape (one Opus frame per moq-lite group → ≤ 20 ms
late-join initial gap).

## Residual loss after the MAX_STREAMS fix

A full prod sweep (all 27 scenarios, commit `c0269859`) showed:

**Now perfect (100% received):** `baseline 100×20ms`, `frames200`,
`frames50`, `1kb`, `4kb`, `cad10`, `cad40`, `cad80`, `cad200`,
`2subs` (both subs), `slowconsumer`, `fpg5`, `fpg20`, `fpg-all`.
14 of 27 scenarios.

**Improved but still partial:**

| Scenario | Pre-fix | Post-fix | Pattern |
|---|---|---|---|
| `frames400` | 99/400 | 368/400 | last 32 frames lost (tail) |
| `30s` (1500 frames) | 99/1500 | 653/1500 | cliffed at frame 653 mid-pump |
| `120s` (6000 frames) | 52/6000 | 61/6000 | cliffed at frame 61, t≈2 s |
| `burst` cadence=0 | 65/100 | 96/100 | last 4 lost (tail) |
| `16kb` 100×16 KB | 49/100 | 37/100 | regressed slightly |
| `pause` 50+5 s+50 | 99/100 | 32/100 | regressed; pause kills it |
| `cad5` 100×5 ms | 99/100 | 54/100 | regressed (faster burst) |
| `3subs` sub[2] | varied | 77/100 | one sub lags |

The cliff is no longer at "exactly 99 streams" — it's now timing/load-
sensitive and varies between runs. The speaker side remains pristine
(`fc-post-pump`: `consumed > 0`, `pendingBytes = 0`, `nextLocalUniIdx`
matches frame count) — so the loss is on the relay→listener path.

`fc-post-grace` is identical to `fc-post-pump` for these scenarios
(no late delivery during grace), confirming the listener never
recovers once it stalls.

## Round 2 — chasing the residual

After the original `MAX_STREAMS_UNI` fix, residual loss appeared in
sustained / bursty scenarios. Three round-2 changes against
likely contributors:

### 2a. Test-side stdout serialisation (hypothesis 2)

`SendTraceScenario.Scenario.verbosePerFrame` defaulted to `true`,
which logged a per-frame `tx i=…` and `rx[idx] gid=…` line via
`InteropDebug.checkpoint`. Under JUnit's stdout capture, that's a
synchronous write per event. At 50 frames/sec sustained the
capture thread can serialise the receive coroutine and starve the
QUIC read loop, biasing the `received` count downward. **Default
flipped to `false`**; specific tests opt in for debugging.

### 2b. Test-side `CopyOnWriteArrayList` add cost

The collector accumulated arrivals in a `CopyOnWriteArrayList`,
which is O(N) per add — copying the entire underlying array on
every `+=`. For 1500-frame runs that's ~1.1 M element copies and
~35 MB of memory ops cumulative across the run. Replaced with
`Collections.synchronizedList(ArrayList(capacityHint))` for O(1)
amortized adds. Snapshot at end via
`synchronized(sink) { ArrayList(sink) }`.

### 2c. Listener-side kernel UDP receive buffer (hypothesis 1)

`UdpSocket.connect` did not configure `SO_RCVBUF`, so the kernel's
default applied (~200 KB on Linux/macOS, similarly small on
Android). At MTU-sized datagrams that's room for ~130 packets. A
multi-second moq-lite broadcast that the relay fans out across
multiple subscribers can transiently exceed this — anything
queued past `rmem` is silently dropped by the kernel and never
reaches our QUIC stack, manifesting as "subscription stops
mid-broadcast even though `publisher.send` keeps returning true".
Bumped to 4 MiB at socket-bind time (Linux doubles + clamps via
`rmem_max`, so the effective cap is whatever the kernel allows up
to `min(8 MiB, rmem_max)`).

Added lifetime UDP datagram counters (`receivedDatagramCount`,
`receivedByteCount`, `receiveBufferSizeBytes`) to the `UdpSocket`
expect surface; the driver wires them into the existing
`QuicConnection.flowControlSnapshot` via a `udpStatsSupplier` hook.
The `fc-pre / fc-post-pump / fc-post-grace` lines now include
`udpDatagrams=…  udpBytes=…  udpRcvBuf=…` so a future sweep can
correlate "frames missing on subscription" against "datagrams the
kernel actually delivered".

### Hypotheses still open after round 2

3. **moq-rs relay per-subscriber queue policy.** Even if our
   listener kernel + collector are infinitely fast, the relay
   itself may have an in-flight uni-stream cap per subscriber that
   drops new groups once the listener falls behind. The
   asymmetric loss in `sweep_3subs sub[2]` (77/100 while sub[0]
   and sub[1] got 100/100) is consistent with a per-subscriber
   queue rather than a connection- or relay-wide limit. Worth
   filing an issue at `kixelated/moq` once we have one more sweep
   confirming the round-2 changes don't already close the gap.

4. **Receive-side `MAX_STREAM_DATA` thresholds.** Probably not
   the cause for 80-byte audio frames (1 MB per-stream window vs
   ~80 bytes used per stream) but may bite the 16 KB-payload
   scenario. Worth a follow-up only if `sweep_payload_16kb`
   doesn't recover.

## Re-running after round 2

The acceptance criterion is the same: every sweep row showing
`received=N/N missing=[]`, except the explicit late-join scenarios
(`sweep_late_subscribe_after_25` / `_after_50`) which legitimately
report `received < N` due to from-latest semantics.

**Reproduces against:** `https://moq.nostrnests.com:4443` (production
relay). Not consistently reproducible against the local
`kixelated/moq` reference relay (the local harness shows the same
shape but with much higher run-to-run variance).

## Symptom

When the speaker pumps Opus frames at the production 20 ms cadence
and packs **one frame per moq-lite group** (= one client-initiated
QUIC unidirectional stream per Opus frame), the listener stops
receiving frames somewhere around the 99th frame. From there to
end-of-broadcast, **`MoqLitePublisherHandle.send` keeps returning
`true` for every subsequent frame**, so the application has no signal
that anything is wrong; the audio simply cuts out a few seconds in.

The bug is invisible to:

- `NestMoqLiteBroadcaster`'s outer `runCatching {…}.onFailure {…}`
  (no exception is thrown on the silent-loss path).
- `MoqLitePublisherHandle.send`'s return value (always `true` past
  the cliff).
- `QuicConnection`'s peer-cap check (`nextLocalUniIndex >= peerMaxStreamsUni`
  doesn't fire because the relay raises `MAX_STREAMS_UNI` ahead of
  our consumption — pre-fix tests confirmed `firstFalseSend = -1`
  even at 1500 frames).

## Sweep evidence (production, before mitigation)

`NostrnestsProdAudioTransmissionTest` (commit
`2da18859`-era, before any QUIC changes), one row per scenario:

| Scenario | streams opened | `received` | cliff |
|---|---|---|---|
| `sweep-frames50` | 50 | 50/50 | none ✅ |
| `sweep-baseline` 100 frames | 100 | 99/100 | last 1 |
| any cadence (5/10/40/80/200 ms) at 100 frames | 100 | 99/100 | last 1 |
| any payload (80 B / 1 KB / 4 KB) at 100 frames | 100 | 99/100 | last 1 |
| `sweep-frames200` | 200 | **99**/200 | hard at 99 |
| `sweep-frames400` | 400 | **99**/400 | hard at 99 |
| `sweep-30s` (1500 frames) | 1500 | **99**/1500 | hard at 99 |
| `sweep-120s` (6000 frames) | 6000 | 52/6000 | hard at 52 |
| `sweep-burst` (cadence 0) | 100 | 65/100 | hard at 65 |
| `sweep-payload-16kb` 100 frames × 16 KB | 100 | 49/100 | hard at 49 |
| **`sweep-frames-per-group-5`** | **20** groups | **100/100** | none ✅ |
| **`sweep-frames-per-group-20`** | **5** groups | **100/100** | none ✅ |
| **`sweep-frames-per-group-all`** | **1** group | **100/100** | none ✅ |

The signal is unambiguous: **fewer streams → no loss, regardless of
frame count, payload, or duration**. The cliff scales with stream-
opening rate, not with byte volume or wall-clock time.

The number "99" is suspicious — it's exactly QUIC's typical
`initial_max_streams_uni = 100` minus one. But:

- Pre-mitigation `firstFalseSend = -1` for tests pumping 1500+
  streams says the cap WAS being raised dynamically. So the cliff
  is NOT `openUniStream` rejecting past the local cap.
- A 16 KB payload variant cliffed at 49 streams (~800 KB total),
  while 80-byte variants cliffed at 99 (~8 KB total). These are
  inconsistent if the cause were a single fixed cap.
- The cliff is *consistent* across many scenarios at ~99 — too
  consistent for plain network packet loss.

## What we tried and what we learned

### Attempt 1 — make `openUniStream` suspend instead of throw

Branch / commit: `f0705e3a` on `claude/audio-transmission-tests-zKzlB`,
**reverted in `96a585a6`**.

Rationale: the obvious failure mode is `peerMaxStreamsUni` being hit
without us ever waiting for the relay to extend it; we'd just throw
locally. So we'd:

1. Make `QuicConnection.openBidiStream` / `openUniStream` suspend
   on a `CompletableDeferred<Unit>` notifier when the cap is hit,
   re-acquire the lock and re-check on every wake.
2. Fire the notifier from `QuicConnectionParser` whenever an inbound
   `MAX_STREAMS` frame raises a cap.
3. Emit `STREAMS_BLOCKED` (RFC 9000 §19.14) when an opener registers
   itself blocked, so the peer knows we want more credit.
4. Wake blocked openers with `QuicConnectionClosedException` on
   close so they don't hang.

Unit tests for the suspend / wake / STREAMS_BLOCKED / close paths
pass (`PeerStreamLimitTest`).

**Result against production:** the cliff didn't move. Same scenarios
still hit ~99 frames received, and the run-to-run variance got
*worse* (numbers swung between 19/100 and 99/100 on consecutive
identical baseline runs against the same relay). The fix also
introduced no observable change in the per-frame `tx i=… ok dt=…us`
traces (`firstFalseSend = -1` still on every test), which means the
suspend path isn't being triggered in production — the peer's cap
isn't what we're hitting.

The fix is RFC-correct on its own (we should suspend rather than
throw, and we should send STREAMS_BLOCKED), but it does not address
the production cliff. Reverted to keep the tree clean while we
chase the real cause.

### Attempt 2 — pack multiple frames per moq-lite group (shipping)

Commit on `claude/audio-transmission-tests-zKzlB` (this commit).

Rationale: `sweep_frames_per_group_*` was 100/100 across every
multi-frame-per-group variant. The cliff scales with the number of
new uni streams opened, not with anything else. Packing N Opus
frames per moq-lite group reduces the new-stream rate by N×.

Default chosen: `framesPerGroup = 5` ≈ 100 ms of audio per group.
Stream-creation rate drops from 50/sec to 10/sec, well below the
cliff.

Trade-off: a brand-new subscriber that attaches mid-broadcast picks
up at the next group boundary per moq-lite "from-latest" semantics.
With 5 frames/group the late-join initial gap is up to 100 ms
(perceptually inaudible). With 1 frame/group it was up to 20 ms.

This is a **mitigation, not a fix.** Other workloads with a high
per-stream cost (file transfer, metadata fan-out, anything that
naturally wants a fresh stream per message) would still hit the
cliff if they exceed roughly 50 streams/second.

## What `flowControlSnapshot` proved (the smoking gun)

After wiring [QuicConnection.flowControlSnapshot] into [SendTraceScenario],
running `sweep_frames_200` against production produced:

```
fc-pre:        peerInitMaxData=4611686018427387903   peerInitMaxStreamDataUni=1250000
               peerInitMaxStreamsUni=10000           sendCredit=4611686018427387903
               consumed=786                          peerMaxStreamsUniNow=10000
               nextLocalUniIdx=1                     pendingBytes=0   pendingStreams=0/4
fc-post-pump:  sendCredit=4611686018427387903        consumed=18729
               peerMaxStreamsUniNow=10000            nextLocalUniIdx=201
               pendingBytes=0                        pendingStreams=0/205
fc-post-grace: (identical to fc-post-pump)
sub[0]:        received=99/200                       firstSentButLost=99
```

This rules out every speaker-side hypothesis at once:

- **`peerInitMaxData = 2⁶²−1`** ⇒ ~5 EB connection-level send credit;
  we used **18 KB**. Not a connection-level flow-control wedge.
- **`peerInitMaxStreamsUni = 10000`, `nextLocalUniIdx = 201`** ⇒
  speaker opened all 200 broadcast streams + 1 control stream, well
  under cap. Not a peer-granted stream-id wedge.
- **`pendingBytes = 0`, `pendingStreams = 0/205`** ⇒ no bytes are
  stuck in any stream's send buffer; everything we wrote made it
  past the writer.
- **`consumed` is identical between post-pump and post-grace** ⇒ we
  weren't even *trying* to send more after the pump finished.

So the speaker → relay path was healthy. The 18 KB of stream data
made it onto the wire. The 99-frame ceiling on the listener side
had to be on the **relay → listener** half of the connection.

Listener-side, our [QuicConnection] advertises `initial_max_streams_uni
= 100` (from [QuicConnectionConfig], default 100) at handshake. The
relay can open up to 100 uni streams to us *for the lifetime of the
connection* unless we send `MAX_STREAMS_UNI(N)` frames to extend it.
Grepping `:quic` for `MaxStreamsFrame` showed it was only ever
*parsed inbound* — there was no outbound emission anywhere. So the
peer's stream-id allowance was capped at 100 forever.

For MoQ over WebTransport this is fatal: the listener's relay
forwards every broadcast group as a new peer-initiated uni stream.
With one group per Opus frame at 20 ms cadence, the 100-stream cap
is exhausted at frame 99 → relay-side blocked → no more groups → no
more audio.

The fix mirrors the existing [appendFlowControlUpdates] pattern for
`MAX_DATA` / `MAX_STREAM_DATA`: track lifetime peer-initiated stream
count, emit `MAX_STREAMS_UNI(currentCount + initialCap)` when count
crosses the half-window threshold.

## Hypotheses still on the table

In rough order of likelihood, none confirmed:

1. **Connection-level send credit** (`initial_max_data` / `MAX_DATA`).
   If the relay's connection-level grant is ~8 KB and only extends
   on `DATA_BLOCKED`, we'd cliff at ~99 frames × ~85 bytes = ~8.4 KB.
   `:quic` parses inbound `MAX_DATA` (`QuicConnectionParser.kt:266`)
   but never emits `DATA_BLOCKED` when our writer hits
   `connBudget == 0` in `appendStreamFrames`. That asymmetry would
   stall the producer with no signal. The 16 KB payload data point
   (cliff at 49 × 16 KB ≈ 800 KB) doesn't match a single fixed
   8 KB cap, but it could match a different per-byte budget the
   relay calibrates to its initial offer. Worth instrumenting.

2. **Per-stream send credit** (`initial_max_stream_data_uni` /
   `MAX_STREAM_DATA`). Each new uni stream gets its own credit
   window from the peer's `initial_max_stream_data_uni`. If that's
   small enough to require a `MAX_STREAM_DATA` on every stream and
   the peer is conservative about issuing it, the writer would
   buffer frames forever waiting for credit while `publisher.send`
   reports success. `:quic` honours the per-stream credit
   (`QuicConnectionWriter.kt:280`) but never emits
   `STREAM_DATA_BLOCKED`.

3. **Reference-relay default `MAX_STREAMS_UNI` extension policy.**
   `kixelated/moq-rs` is built on Quinn; Quinn's stream-cap
   extension is automatic when streams complete. If the production
   nostrnests deployment uses a custom Quinn config that only
   extends in response to `STREAMS_BLOCKED` (which we don't send
   on the un-fixed code path), we'd starve. Attempt 1 added
   STREAMS_BLOCKED emission and didn't shift the cliff, weak
   evidence against this hypothesis.

4. **Relay-side per-subscription buffer.** If the relay has a
   per-subscriber forward buffer that's smaller than the total
   in-flight uni-stream count, and it drops streams that don't
   fit, the cliff would scale with stream count and be invisible
   to the speaker. Hardest to investigate without relay logs.

## Next steps to actually fix

The investigation order I'd take, given the data:

1. **Instrument outbound `appendStreamFrames`.** Log every iteration
   where `connBudget <= 0` (connection-level credit exhausted) or
   where a stream's `streamRemaining <= 0` (per-stream credit
   exhausted). Re-run the prod sweep with this on. If the cliff
   correlates with one of those, the hypothesis is confirmed and the
   fix is to emit `DATA_BLOCKED` / `STREAM_DATA_BLOCKED` and on the
   write path, suspend the writer until a fresh `MAX_DATA` /
   `MAX_STREAM_DATA` arrives.

2. **Capture the production peer transport parameters.** Dump
   `tp.initialMaxData`, `tp.initialMaxStreamDataUni`, and
   `tp.initialMaxStreamsUni` once at handshake time so we know
   exactly what the relay grants. Compare against the cliff
   thresholds.

3. **Test with `:quic` flow-control diagnostics in place.** Re-run
   `sweep_frames200` and `sweep_30s` — the cliff is rock-solid at
   exactly 99 in production, so a single pass is enough to confirm
   the hypothesis.

4. **If it's not connection / stream data flow control,** instrument
   the relay (file an issue on `nostrnests/nests` or `kixelated/moq`)
   asking whether their relay imposes a per-subscriber stream
   buffer that drops on overflow.

## Re-running the sweep after a candidate fix

The test infrastructure is already in place:

```bash
./gradlew :nestsClient:jvmTest \
  --tests com.vitorpamplona.nestsclient.interop.NostrnestsProdAudioTransmissionTest \
  -DnestsProd=true -DnestsInteropDebug=true --rerun-tasks
grep -E "sub\[|missing=" \
  nestsClient/build/test-results/jvmTest/TEST-com.vitorpamplona.nestsclient.interop.NostrnestsProdAudioTransmissionTest.xml \
  | sort
```

Acceptance criterion: `sweep-30s` reports `received≈1500/1500`
without any `framesPerGroup` mitigation. That removes the only
remaining audio-quality tradeoff (the 100 ms late-join initial gap)
and lets us reset `DEFAULT_FRAMES_PER_GROUP` back to 1 to match the
JS reference broadcaster's wire shape.

## Files / lines to touch when picking this up

- `quic/src/commonMain/kotlin/com/vitorpamplona/quic/connection/QuicConnectionWriter.kt`
  — `appendStreamFrames` is where the writer skips streams with no
  credit; instrument here.
- `quic/src/commonMain/kotlin/com/vitorpamplona/quic/connection/QuicConnection.kt:160-165`
  — `sendConnectionFlowCredit` / `sendConnectionFlowConsumed`. The
  values to log + diff.
- `quic/src/commonMain/kotlin/com/vitorpamplona/quic/connection/TransportParameters.kt`
  — `decode` is where peer TPs land at handshake.
- `quic/src/commonMain/kotlin/com/vitorpamplona/quic/connection/QuicConnectionParser.kt`
  — currently parses `MAX_DATA`, `MAX_STREAM_DATA`, `MAX_STREAMS`,
  `STREAMS_BLOCKED` (last one ignored). The reverse —
  `DATA_BLOCKED` / `STREAM_DATA_BLOCKED` emission — would live in
  `QuicConnectionWriter`.
- `nestsClient/src/jvmTest/kotlin/com/vitorpamplona/nestsclient/interop/SendTraceScenario.kt`
  — extend per-frame trace with `connBudget` / `streamRemaining`
  if the writer instrumentation surfaces them via diagnostics on
  `QuicConnection`.