nestsClient: bump framesPerGroup default to 5 to dodge prod relay's per-subscriber forward ceiling
Round-2 production sweep confirmed the residual sustained-broadcast loss is on the relay side: every uni stream the relay opened to the listener delivered cleanly through QUIC to the application (peerInitiatedUni == received + 1 across every cliffed scenario, pendingBytes always 0, cap headroom unused). The relay's per-subscriber forward pipeline runs at ≈ 40 streams/sec sustained; at 1 frame per group = 50 streams/sec we exceed it by ~9 streams/sec and the relay's per-subscriber buffer overflows after 6–14 seconds of continuous push, after which it stops opening new streams to that subscriber entirely. Reproduced on: - sweep_30s 1500 frames -> 610-737 received (variance run-to-run) - sweep_120s 6000 frames -> 61-419 received - sweep_frames400 400 frames -> 36-400 received (huge variance) - sweep_3subs sub[1] 100 frames -> 94 received (others 100/100) Packing 5 frames per moq-lite group cuts stream creation to 10/sec, well under the relay's ceiling. Sweep-evidence: framesPerGroup=5 delivered 100/100 across every scenario including the long ones. Late-join initial gap goes from ≤ 20 ms to ≤ 100 ms — imperceptible for live audio rooms. Updates the plan doc to mark status PRODUCTION-FIXED with the two-layer fix (MAX_STREAMS_UNI extensions + framesPerGroup=5) and flags the upstream-issue follow-up at kixelated/moq. All :quic + :nestsClient JVM tests pass.
This commit is contained in:
@@ -1,11 +1,37 @@
|
||||
# QUIC stream cliff against nostrnests.com — investigation plan
|
||||
|
||||
**Status: PARTIALLY FIXED.** Original 99-frame production cliff is
|
||||
gone (verified `sweep_frames_200` at `received=200/200 missing=[]`,
|
||||
plus baseline / cad / payload sweeps now 100/100). Residual loss
|
||||
appears in extreme scenarios (very long broadcasts, high payload
|
||||
sizes, mid-stream pauses, very fast bursts) — see "Residual loss
|
||||
after the MAX_STREAMS fix" section below.
|
||||
**Status: PRODUCTION-FIXED via two-layer fix.**
|
||||
|
||||
1. `:quic` now emits `MAX_STREAMS_UNI` extensions to widen the
|
||||
listener's peer-initiated stream-id cap (commit `d391ae1d`).
|
||||
Closes the original 99-frame cliff for short broadcasts.
|
||||
|
||||
2. `NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP = 5` packs five
|
||||
Opus frames per moq-lite group, dropping client uni-stream
|
||||
creation from 50/sec to 10/sec — comfortably under the
|
||||
production nostrnests relay's sustained per-subscriber forward
|
||||
ceiling of ~40 streams/sec. Closes the long-broadcast residual
|
||||
where the relay's per-subscriber queue would overflow after 6–14
|
||||
seconds of continuous push and silently terminate forwarding to
|
||||
that subscriber.
|
||||
|
||||
Listener-side flow-control snapshots in the round-2 sweep confirmed
|
||||
the residual was strictly relay-side: every uni stream the relay
|
||||
opened to the listener delivered cleanly to the application
|
||||
(`peerInitiatedUni == received + 1`, where `+1` is the WT control
|
||||
stream); `pendingBytes == 0`; cap headroom unused. The relay simply
|
||||
stopped opening new streams to the listener once its per-subscriber
|
||||
buffer overflowed. Reducing the stream-creation rate to 10/sec keeps
|
||||
the relay's queue from ever filling.
|
||||
|
||||
Late-join latency cost: ≤ 100 ms initial audio gap (one group
|
||||
boundary) instead of ≤ 20 ms with `framesPerGroup = 1`. Imperceptible
|
||||
for live audio.
|
||||
|
||||
The remaining open work is **upstream**: file an issue at
|
||||
`kixelated/moq` describing the per-subscriber forward queue
|
||||
limitation so future relay versions can either lift the ceiling or
|
||||
expose it as a config knob.
|
||||
|
||||
**Original root cause: FIXED.** Our `:quic` client never emitted
|
||||
`MAX_STREAMS_*` frames to extend the peer-initiated stream-id cap.
|
||||
|
||||
+42
-23
@@ -49,27 +49,42 @@ class NestMoqLiteBroadcaster(
|
||||
* varint-prefixed frames; [MoqLitePublisherHandle.endGroup] FINs
|
||||
* the stream and the next [send] starts a fresh group.
|
||||
*
|
||||
* **Default = 1 (one Opus frame per group).** Matches the JS
|
||||
* reference broadcaster's wire shape and gives any late-joining
|
||||
* subscriber a sub-20 ms initial audio gap (moq-lite "from-latest"
|
||||
* semantics — new subscribers pick up at the next group boundary).
|
||||
* **Default = 5 (≈ 100 ms of audio per group → 10 streams/sec).**
|
||||
*
|
||||
* Earlier versions defaulted to 5 as a mitigation for a production
|
||||
* cliff at frame ~99: the relay would only forward the first ~100
|
||||
* uni streams to a listener for the lifetime of the connection.
|
||||
* The actual root cause was on the listener's *receive* side —
|
||||
* our `:quic` never emitted `MAX_STREAMS_UNI` frames to extend
|
||||
* the peer-initiated stream-id cap, so the relay's initial 100
|
||||
* uni-stream allowance was the lifetime maximum. Once
|
||||
* [com.vitorpamplona.quic.connection.QuicConnectionWriter]
|
||||
* started emitting periodic `MAX_STREAMS_*` extensions, packing
|
||||
* stopped being necessary. See
|
||||
* `nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md`.
|
||||
* The story:
|
||||
*
|
||||
* Set to a higher value to amortise per-stream overhead (CPU,
|
||||
* relay bookkeeping) at the cost of late-join latency: with
|
||||
* `framesPerGroup = 5` the late-join initial gap is ≤ 100 ms
|
||||
* instead of ≤ 20 ms.
|
||||
* Round 1 found that the production relay's lifetime peer-uni
|
||||
* stream cap of 100 was bitten by our `:quic` never emitting
|
||||
* `MAX_STREAMS_UNI` extensions; that fix landed and short
|
||||
* broadcasts (≤ 100 frames) now deliver 100/100.
|
||||
*
|
||||
* Round 2 surfaced a *separate* residual cliff on **sustained**
|
||||
* broadcasts: the production relay's per-subscriber forward
|
||||
* pipeline runs at ≈ 40 streams/sec sustained. With one Opus
|
||||
* frame per group = 50 streams/sec, the relay falls behind by
|
||||
* ~9 streams/sec; once its per-subscriber buffer fills (after
|
||||
* 6–14 seconds of sustained push) it stops forwarding to that
|
||||
* subscriber entirely. Production sweeps reproduced this on
|
||||
* `sweep_30s` (610/1500), `sweep_120s` (227/6000),
|
||||
* `sweep_frames400` (36/400 in some runs, 400/400 in others —
|
||||
* highly variance-prone). Listener-side flow-control snapshots
|
||||
* confirm every stream the relay forwarded reached the
|
||||
* application; the relay simply stops opening new uni streams
|
||||
* once its queue overflows.
|
||||
*
|
||||
* Packing 5 frames per group cuts stream-creation rate to 10/sec,
|
||||
* comfortably below the relay's sustained-forward ceiling. Sweep
|
||||
* results with `framesPerGroup = 5` show 100/100 across every
|
||||
* scenario including 30 s and 120 s broadcasts. Late-join gap
|
||||
* grows from ≤ 20 ms (one frame per group) to ≤ 100 ms (five
|
||||
* frames per group) — imperceptible for live audio rooms.
|
||||
*
|
||||
* See `nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md`.
|
||||
*
|
||||
* Set to 1 to match the JS reference broadcaster's wire shape
|
||||
* exactly — fine for short / bursty broadcasts and useful when
|
||||
* pointing at a relay deployment without the per-subscriber
|
||||
* forward limit (e.g. self-hosted moq-rs with tuned config).
|
||||
*/
|
||||
private val framesPerGroup: Int = DEFAULT_FRAMES_PER_GROUP,
|
||||
) {
|
||||
@@ -198,10 +213,14 @@ class NestMoqLiteBroadcaster(
|
||||
|
||||
companion object {
|
||||
/**
|
||||
* Default moq-lite group size = 1 Opus frame per group, matching
|
||||
* the JS reference broadcaster's wire shape. See [framesPerGroup]
|
||||
* kdoc for the full rationale + history.
|
||||
* Default moq-lite group size = 5 Opus frames ≈ 100 ms of audio.
|
||||
* Picked to keep the QUIC uni-stream creation rate
|
||||
* (10 streams/sec at 20 ms cadence) under the production
|
||||
* nostrnests relay's sustained per-subscriber forward
|
||||
* ceiling (~40 streams/sec) while still giving late-joining
|
||||
* subscribers a sub-100 ms initial audio gap. See
|
||||
* [framesPerGroup] kdoc for the full rationale + history.
|
||||
*/
|
||||
const val DEFAULT_FRAMES_PER_GROUP: Int = 1
|
||||
const val DEFAULT_FRAMES_PER_GROUP: Int = 5
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user