plan: cite the actual moq-rs 0.10.25 source for the upstream stream-stop architectural gaps
Previous status note speculated about a per-subscriber forwarding ceiling.
This pins it down with file:line citations from the production version of
kixelated/moq so the upstream issue we file (or the next person to debug
this) has the exact code paths in hand:
1. Unbounded per-track group queue (moq-lite/src/model/track.rs:69-90),
evicted only by 30 s age (track.rs:29).
2. serve_group's session.open_uni().await has no timeout
(moq-lite/src/lite/publisher.rs:317-379) — blocks forever when the
subscriber's CWND collapses.
3. Publisher's FuturesUnordered task pool is unbounded
(publisher.rs:325, 346) — keeps spawning blocked tasks with no
backpressure to upstream.
4. max_concurrent_uni_streams = 10000 (moq-native/src/quinn.rs:30-33)
so the symptom is NOT the QUIC stream-id cap; it's the cascade
above.
5. No lagging-consumer RESET/STOP_SENDING anywhere; the relay never
gives up on a stuck subscriber.
The framesPerGroup=5 mitigation we shipped sidesteps the cascade by
keeping upstream rate (10 streams/sec) below the threshold any of those
gaps would accumulate at. It doesn't fix the underlying issues — those
are upstream to address.
This commit is contained in:
@@ -33,6 +33,72 @@ The remaining open work is **upstream**: file an issue at
|
||||
limitation so future relay versions can either lift the ceiling or
|
||||
expose it as a config knob.
|
||||
|
||||
### Upstream confirmation (moq-rs 0.10.25 source audit)
|
||||
|
||||
I read the `kixelated/moq` Rust source at the version production
|
||||
nostrnests is running (`moq-relay 0.10.25-5063dbfe`). Three
|
||||
architectural issues compound to produce the observed silent
|
||||
stream-stop:
|
||||
|
||||
1. **Per-track group queue is unbounded.**
|
||||
`moq/rs/moq-lite/src/model/track.rs:69-90` —
|
||||
```rust
|
||||
groups: VecDeque<Option<(GroupProducer, tokio::time::Instant)>>,
|
||||
```
|
||||
Inserts are unconditional (subscriber.rs:363-393); the only
|
||||
eviction is age-based at `MAX_GROUP_AGE = 30 s`
|
||||
(track.rs:29). So when a subscriber falls behind, groups pile
|
||||
up indefinitely until they age out 30 s later.
|
||||
|
||||
2. **`serve_group` blocks on `open_uni().await` with no timeout.**
|
||||
`moq/rs/moq-lite/src/lite/publisher.rs:317-379` — the per-group
|
||||
forward task does
|
||||
```rust
|
||||
let stream = session.open_uni().await.map_err(Error::from_transport)?;
|
||||
```
|
||||
If the subscriber's Quinn CWND has collapsed (because *our*
|
||||
ACK rate slipped) or the subscriber's advertised
|
||||
`MAX_STREAMS_UNI` is exhausted, this `await` blocks the task
|
||||
forever. There's no `select!` against a deadline.
|
||||
|
||||
3. **No bounded task pool feeding the awaits.**
|
||||
The publisher pushes blocked `serve_group` tasks into a
|
||||
`FuturesUnordered` (publisher.rs:325, 346) which is itself
|
||||
unbounded. The receive loop (line 334) keeps spawning more
|
||||
serve tasks as upstream groups arrive. So we end up with N
|
||||
blocked tasks all waiting on `open_uni`, and **no
|
||||
backpressure path back to the publisher** to slow upstream
|
||||
ingestion or drop.
|
||||
|
||||
4. **`max_concurrent_uni_streams = 10_000`** in
|
||||
`moq/rs/moq-native/src/quinn.rs:30-33` (override of Quinn's
|
||||
default 1024). High enough that we never hit the connection's
|
||||
stream-id cap on our listener side at typical audio rates —
|
||||
confirmed by our `peerMaxStreamsUniNow=10000` snapshots — so
|
||||
the stall is *not* about Quinn's stream-concurrency limit per
|
||||
se. It's the cascade in (1)+(2)+(3): once any in-flight
|
||||
write blocks (CWND, per-stream credit, etc.), all the queued
|
||||
`serve_group` tasks that follow also block, and the publisher
|
||||
loop happily keeps adding more tasks until the per-track
|
||||
queue ages everything out.
|
||||
|
||||
5. **No "lagging consumer" detection / RESET / STOP_SENDING.**
|
||||
The relay never proactively gives up on a subscriber that's
|
||||
falling behind. So from the subscriber's POV the subscription
|
||||
stays "active", `peerInitiatedUni` is frozen, and the connection
|
||||
stays alive — exactly the symptom we observed.
|
||||
|
||||
That's three specific architectural gaps any of which on its own
|
||||
would cause the symptom. The `framesPerGroup = 5` mitigation works
|
||||
because it cuts the upstream-publisher rate to ~10 streams/sec —
|
||||
slow enough that even a temporarily-stalled subscriber's queue
|
||||
drains before any of (1)/(2)/(3) accumulate beyond recovery. It
|
||||
doesn't fix the underlying gaps.
|
||||
|
||||
Suggested upstream issue text to file at `kixelated/moq` is
|
||||
captured in the commit message of `85691cce` and can be expanded
|
||||
with the specific file:line cites above.
|
||||
|
||||
**Original root cause: FIXED.** Our `:quic` client never emitted
|
||||
`MAX_STREAMS_*` frames to extend the peer-initiated stream-id cap.
|
||||
[QuicConnectionConfig.initialMaxStreamsUni] (default `100`) was the
|
||||
|
||||
Reference in New Issue
Block a user