diff --git a/nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md b/nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md index 630273fa9..7bd7d40b8 100644 --- a/nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md +++ b/nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md @@ -1,11 +1,37 @@ # QUIC stream cliff against nostrnests.com — investigation plan -**Status: PARTIALLY FIXED.** Original 99-frame production cliff is -gone (verified `sweep_frames_200` at `received=200/200 missing=[]`, -plus baseline / cad / payload sweeps now 100/100). Residual loss -appears in extreme scenarios (very long broadcasts, high payload -sizes, mid-stream pauses, very fast bursts) — see "Residual loss -after the MAX_STREAMS fix" section below. +**Status: PRODUCTION-FIXED via two-layer fix.** + + 1. `:quic` now emits `MAX_STREAMS_UNI` extensions to widen the + listener's peer-initiated stream-id cap (commit `d391ae1d`). + Closes the original 99-frame cliff for short broadcasts. + + 2. `NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP = 5` packs five + Opus frames per moq-lite group, dropping client uni-stream + creation from 50/sec to 10/sec — comfortably under the + production nostrnests relay's sustained per-subscriber forward + ceiling of ~40 streams/sec. Closes the long-broadcast residual + where the relay's per-subscriber queue would overflow after 6–14 + seconds of continuous push and silently terminate forwarding to + that subscriber. + +Listener-side flow-control snapshots in the round-2 sweep confirmed +the residual was strictly relay-side: every uni stream the relay +opened to the listener delivered cleanly to the application +(`peerInitiatedUni == received + 1`, where `+1` is the WT control +stream); `pendingBytes == 0`; cap headroom unused. The relay simply +stopped opening new streams to the listener once its per-subscriber +buffer overflowed. Reducing the stream-creation rate to 10/sec keeps +the relay's queue from ever filling. + +Late-join latency cost: ≤ 100 ms initial audio gap (one group +boundary) instead of ≤ 20 ms with `framesPerGroup = 1`. Imperceptible +for live audio. + +The remaining open work is **upstream**: file an issue at +`kixelated/moq` describing the per-subscriber forward queue +limitation so future relay versions can either lift the ceiling or +expose it as a config knob. **Original root cause: FIXED.** Our `:quic` client never emitted `MAX_STREAMS_*` frames to extend the peer-initiated stream-id cap. diff --git a/nestsClient/src/commonMain/kotlin/com/vitorpamplona/nestsclient/audio/NestMoqLiteBroadcaster.kt b/nestsClient/src/commonMain/kotlin/com/vitorpamplona/nestsclient/audio/NestMoqLiteBroadcaster.kt index 2bbd865c2..409949979 100644 --- a/nestsClient/src/commonMain/kotlin/com/vitorpamplona/nestsclient/audio/NestMoqLiteBroadcaster.kt +++ b/nestsClient/src/commonMain/kotlin/com/vitorpamplona/nestsclient/audio/NestMoqLiteBroadcaster.kt @@ -49,27 +49,42 @@ class NestMoqLiteBroadcaster( * varint-prefixed frames; [MoqLitePublisherHandle.endGroup] FINs * the stream and the next [send] starts a fresh group. * - * **Default = 1 (one Opus frame per group).** Matches the JS - * reference broadcaster's wire shape and gives any late-joining - * subscriber a sub-20 ms initial audio gap (moq-lite "from-latest" - * semantics — new subscribers pick up at the next group boundary). + * **Default = 5 (≈ 100 ms of audio per group → 10 streams/sec).** * - * Earlier versions defaulted to 5 as a mitigation for a production - * cliff at frame ~99: the relay would only forward the first ~100 - * uni streams to a listener for the lifetime of the connection. - * The actual root cause was on the listener's *receive* side — - * our `:quic` never emitted `MAX_STREAMS_UNI` frames to extend - * the peer-initiated stream-id cap, so the relay's initial 100 - * uni-stream allowance was the lifetime maximum. Once - * [com.vitorpamplona.quic.connection.QuicConnectionWriter] - * started emitting periodic `MAX_STREAMS_*` extensions, packing - * stopped being necessary. See - * `nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md`. + * The story: * - * Set to a higher value to amortise per-stream overhead (CPU, - * relay bookkeeping) at the cost of late-join latency: with - * `framesPerGroup = 5` the late-join initial gap is ≤ 100 ms - * instead of ≤ 20 ms. + * Round 1 found that the production relay's lifetime peer-uni + * stream cap of 100 was bitten by our `:quic` never emitting + * `MAX_STREAMS_UNI` extensions; that fix landed and short + * broadcasts (≤ 100 frames) now deliver 100/100. + * + * Round 2 surfaced a *separate* residual cliff on **sustained** + * broadcasts: the production relay's per-subscriber forward + * pipeline runs at ≈ 40 streams/sec sustained. With one Opus + * frame per group = 50 streams/sec, the relay falls behind by + * ~9 streams/sec; once its per-subscriber buffer fills (after + * 6–14 seconds of sustained push) it stops forwarding to that + * subscriber entirely. Production sweeps reproduced this on + * `sweep_30s` (610/1500), `sweep_120s` (227/6000), + * `sweep_frames400` (36/400 in some runs, 400/400 in others — + * highly variance-prone). Listener-side flow-control snapshots + * confirm every stream the relay forwarded reached the + * application; the relay simply stops opening new uni streams + * once its queue overflows. + * + * Packing 5 frames per group cuts stream-creation rate to 10/sec, + * comfortably below the relay's sustained-forward ceiling. Sweep + * results with `framesPerGroup = 5` show 100/100 across every + * scenario including 30 s and 120 s broadcasts. Late-join gap + * grows from ≤ 20 ms (one frame per group) to ≤ 100 ms (five + * frames per group) — imperceptible for live audio rooms. + * + * See `nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md`. + * + * Set to 1 to match the JS reference broadcaster's wire shape + * exactly — fine for short / bursty broadcasts and useful when + * pointing at a relay deployment without the per-subscriber + * forward limit (e.g. self-hosted moq-rs with tuned config). */ private val framesPerGroup: Int = DEFAULT_FRAMES_PER_GROUP, ) { @@ -198,10 +213,14 @@ class NestMoqLiteBroadcaster( companion object { /** - * Default moq-lite group size = 1 Opus frame per group, matching - * the JS reference broadcaster's wire shape. See [framesPerGroup] - * kdoc for the full rationale + history. + * Default moq-lite group size = 5 Opus frames ≈ 100 ms of audio. + * Picked to keep the QUIC uni-stream creation rate + * (10 streams/sec at 20 ms cadence) under the production + * nostrnests relay's sustained per-subscriber forward + * ceiling (~40 streams/sec) while still giving late-joining + * subscribers a sub-100 ms initial audio gap. See + * [framesPerGroup] kdoc for the full rationale + history. */ - const val DEFAULT_FRAMES_PER_GROUP: Int = 1 + const val DEFAULT_FRAMES_PER_GROUP: Int = 5 } }