589ced580c690d9dd45c482b19af034008e90b73
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Claude
|
589ced580c |
docs(nests): refresh all T16 + cliff plans to current state
8 plan files updated to reflect what's actually shipped vs. what each doc said before: - 2026-05-06-cross-stack-interop-test.md: 'Spec — ready to implement' → 'Implemented and merged' with pointers to results, gap matrix, and closure roadmap. - 2026-05-06-cross-stack-interop-test-results.md: scenario inventory updated to show all branches merged into claude/cross-stack-interop-test-XAbYB; suite-flake caveats flagged with cross-refs to the routing investigation; file inventory now lists BrowserInteropTest + PlaywrightDriver + the moved nestsClient/tests/browser-interop/ tree (no more 'in sister branches not yet merged' section). - 2026-05-06-cross-stack-interop-test-gap-matrix.md: T8/T10/T11/ T12/T13 all show hang ✅ + browser ✅; I13/I14 'NOT YET LANDED' callouts removed; the 'in sister branch' qualifier on T13 / I7 / I6 dropped; coverage-holes section replaced with caveat pointers to the routing investigation. - 2026-05-06-i4-stereo-cross-stack-scenario.md: 'Spec — ready for implementation' → 'Landed' with PR #2755 + the three test scenarios that ship the assertion. - 2026-05-06-phase4-browser-harness.md: 'Spec — ready for implementation' → 'Landed', with notes on how the implementation diverged (used Container.Legacy.Consumer not @moq/watch; cert pinning via serverCertificateHashes; I2/I3/I4/I13/I14 originally deferred but later landed). - 2026-05-06-phase4-browser-harness-results.md: status updated to reflect 4.D CI removed per maintainer ask; layout note for the nestsClient/tests/browser-interop/ relocation; full scenario list reconciled with results doc. - 2026-05-07-late-join-catalog-flake-investigation.md: status updated to mention |
||
|
Claude
|
897287be5f |
nestsClient: expose moq-lite max_latency on subscribeSpeaker + correct plan framing
The moq-lite Lite-03 spec puts subscriber-side group-staleness control in
max_latency on the SUBSCRIBE frame: 0 = unlimited (relay falls back to its
MAX_GROUP_AGE = 30 s default); a positive value tells the relay to evict
groups older than that in favour of newer ones during transient backpressure.
Plumb it through:
NestsListener.subscribeSpeaker(pubkey, maxLatencyMs = 0L)
└─ MoqLiteNestsListener → MoqLiteSession.subscribe(.., maxLatencyMillis)
└─ DefaultNestsListener (IETF) ignores — no equivalent on that wire
└─ ReconnectingHandle re-issues across reconnects
Default stays at 0L for back-compat with the JS reference watcher; callers
opt in for low-latency live audio.
Also rewrites nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md
with corrected framing: of the four "upstream issues" the prior draft listed,
only our own missing MAX_STREAMS_UNI emission was a real bug (RFC 9000 §4.6
violation, fixed in
|
||
|
Claude
|
9841e3ca2c |
plan: cite the actual moq-rs 0.10.25 source for the upstream stream-stop architectural gaps
Previous status note speculated about a per-subscriber forwarding ceiling.
This pins it down with file:line citations from the production version of
kixelated/moq so the upstream issue we file (or the next person to debug
this) has the exact code paths in hand:
1. Unbounded per-track group queue (moq-lite/src/model/track.rs:69-90),
evicted only by 30 s age (track.rs:29).
2. serve_group's session.open_uni().await has no timeout
(moq-lite/src/lite/publisher.rs:317-379) — blocks forever when the
subscriber's CWND collapses.
3. Publisher's FuturesUnordered task pool is unbounded
(publisher.rs:325, 346) — keeps spawning blocked tasks with no
backpressure to upstream.
4. max_concurrent_uni_streams = 10000 (moq-native/src/quinn.rs:30-33)
so the symptom is NOT the QUIC stream-id cap; it's the cascade
above.
5. No lagging-consumer RESET/STOP_SENDING anywhere; the relay never
gives up on a stuck subscriber.
The framesPerGroup=5 mitigation we shipped sidesteps the cascade by
keeping upstream rate (10 streams/sec) below the threshold any of those
gaps would accumulate at. It doesn't fix the underlying issues — those
are upstream to address.
|
||
|
Claude
|
85691cce26 |
nestsClient: bump framesPerGroup default to 5 to dodge prod relay's per-subscriber forward ceiling
Round-2 production sweep confirmed the residual sustained-broadcast loss is on the relay side: every uni stream the relay opened to the listener delivered cleanly through QUIC to the application (peerInitiatedUni == received + 1 across every cliffed scenario, pendingBytes always 0, cap headroom unused). The relay's per-subscriber forward pipeline runs at ≈ 40 streams/sec sustained; at 1 frame per group = 50 streams/sec we exceed it by ~9 streams/sec and the relay's per-subscriber buffer overflows after 6–14 seconds of continuous push, after which it stops opening new streams to that subscriber entirely. Reproduced on: - sweep_30s 1500 frames -> 610-737 received (variance run-to-run) - sweep_120s 6000 frames -> 61-419 received - sweep_frames400 400 frames -> 36-400 received (huge variance) - sweep_3subs sub[1] 100 frames -> 94 received (others 100/100) Packing 5 frames per moq-lite group cuts stream creation to 10/sec, well under the relay's ceiling. Sweep-evidence: framesPerGroup=5 delivered 100/100 across every scenario including the long ones. Late-join initial gap goes from ≤ 20 ms to ≤ 100 ms — imperceptible for live audio rooms. Updates the plan doc to mark status PRODUCTION-FIXED with the two-layer fix (MAX_STREAMS_UNI extensions + framesPerGroup=5) and flags the upstream-issue follow-up at kixelated/moq. All :quic + :nestsClient JVM tests pass. |
||
|
Claude
|
46af65591a |
chase residual stream loss: bump SO_RCVBUF to 4 MiB + diagnostic UDP counters + cheaper test collector
Three targeted changes against the residual sustained-load stream loss
that survived the MAX_STREAMS_UNI fix (long broadcasts and bursty
scenarios still cliffed mid-pump). Each addresses one of the round-1
hypotheses; the next prod sweep should tell us how much each
contributes.
quic/UdpSocket:
- Bump SO_RCVBUF to 4 MiB at bind time. The kernel default (~200 KB
on Linux/macOS, similar on Android) holds barely 130 MTU-sized
datagrams; a multi-second moq-lite broadcast that the relay fans
out to several subscribers transiently overflows this. Anything
queued past `rmem` is silently dropped by the kernel and never
reaches our QUIC stack.
- Add lifetime diagnostic counters (receivedDatagramCount,
receivedByteCount, receiveBufferSizeBytes) on the expect surface
so commonMain code can read them via the new udpStatsSupplier
hook on QuicConnection without each platform having to surface its
own native bookkeeping.
- QuicConnectionDriver wires the supplier on start; the connection's
flowControlSnapshot bundles the stats into a new
QuicFlowControlSnapshot.udp field.
QuicConnection.flowControlSnapshot:
- Now also surfaces advertisedMaxStreamsUni / advertisedMaxStreamsBidi
and peerInitiatedUniCount / peerInitiatedBidiCount so a sweep can
see the listener-side stream-cap evolution alongside speaker-side
counters that were already there.
SendTraceScenario:
- verbosePerFrame default flipped from true to false. The per-frame
`tx i=…` and `rx[idx] gid=…` logs go through InteropDebug ->
JUnit's stdout capture; at 50 frames/sec the capture thread
serialises the receive coroutine and starves the QUIC read loop,
biasing the recorded received count downward on long runs. Tests
that need the per-frame timeline opt in explicitly.
- Replace the receive-side CopyOnWriteArrayList sink (O(N) per add,
~1.1M element copies cumulative for a 1500-frame run) with
Collections.synchronizedList(ArrayList(capacityHint)) — O(1)
amortised. Snapshot under the same lock at end of run so the
JDK's iterator-locking contract is satisfied.
- fc-pre / fc-post-pump / fc-post-grace lines now include
`udpDatagrams=… udpBytes=… udpRcvBuf=…` plus
`advertisedMaxStreamsUni=… peerInitiatedUni=…` so the next sweep's
diagnostic dump shows directly whether the kernel actually
delivered datagrams the relay sent.
Plan doc: nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md
documents round-2 changes and lists the remaining open hypotheses
(relay-side per-subscriber queue policy, receive-side
MAX_STREAM_DATA threshold).
All :quic + :nestsClient JVM tests pass.
|
||
|
Claude
|
f1fc239ba4 |
plan: status update — original cliff fixed, residual loss documented
Production sweep on commit
|
||
|
Claude
|
c0269859b3 |
nestsClient: revert framesPerGroup default to 1 now that the QUIC fix lands frames
Production sweep_frames_200 confirms received=200/200 missing=[] after the :quic MAX_STREAMS_UNI extension landed in |
||
|
Claude
|
d391ae1db9 |
quic: emit MAX_STREAMS_* to extend peer's stream-id cap (fixes prod cliff at frame ~99)
flowControlSnapshot dump from sweep_frames_200 against nostrnests.com proved
the speaker side was perfectly healthy (~5 EB conn-credit unused, 10 000-stream
peer cap unused, 0 bytes stuck in send buffers, all 200 streams opened) yet
the listener cliffed at frame 99. The constraint was on the listener's
receive side: our :quic was advertising initial_max_streams_uni = 100 at
handshake and never extending it, so the relay could only open 100 uni
streams to us *for the lifetime of the connection*. Each Opus frame the
relay forwards is a fresh peer-initiated uni stream, so any broadcast
longer than ~100 frames silently truncated at the audience.
QuicConnection:
- peerInitiatedUniCount / peerInitiatedBidiCount counters (incremented
in getOrCreatePeerStreamLocked).
- advertisedMaxStreamsUni / advertisedMaxStreamsBidi tracking, starting
at config.initialMaxStreams* and raised by the writer.
QuicConnectionWriter.appendFlowControlUpdates:
- When peerInitiatedUniCount + cfg.initialMaxStreamsUni / 2 >=
advertisedMaxStreamsUni, emit MaxStreamsFrame(bidi=false, newCap)
where newCap = peerInitiatedUniCount + cfg.initialMaxStreamsUni.
Same pattern the existing MaxDataFrame / MaxStreamDataFrame
extension uses; same half-window threshold so we don't spam the
peer.
- Symmetric branch for bidi.
PeerStreamCreditExtensionTest:
- Writer DOES emit MAX_STREAMS_UNI when peer's lifetime uni-stream count
crosses the half-window threshold; advertisedMaxStreamsUni updates.
- Writer DOES NOT emit MAX_STREAMS_UNI below the threshold;
advertisedMaxStreamsUni stays at the initial cap.
Updated nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md
with the fc-snapshot dump that proved the cause and the fix that
addresses it. The framesPerGroup=5 mitigation in NestMoqLiteBroadcaster
can be reverted to 1 once the prod sweep confirms the cliff is gone.
All :quic and :nestsClient JVM tests pass.
|
||
|
Claude
|
10ad69f1af |
nestsClient: pack 5 Opus frames per moq-lite group to dodge the prod stream cliff
Production sweep against nostrnests.com cliffed at exactly 99 frames received
across every multi-frame scenario (frames200/400, 30s, 120s, etc.) when the
broadcaster opened one QUIC uni stream per Opus frame. The investigation in
nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md walks through
why the obvious "raise the stream cap" hypothesis didn't pan out (publisher.send
was never returning false past the cap, the local QuicConnection was already
seeing the relay extend MAX_STREAMS_UNI dynamically).
The same sweep showed sweep_frames_per_group_5 / _20 / _all delivering 100/100
frames cleanly. The cliff scales with stream-creation rate, not byte volume or
total stream count — so packing N frames per group (one uni stream per N frames
instead of per frame) bypasses whatever the underlying limit is.
NestMoqLiteBroadcaster:
- Add framesPerGroup constructor param, default 5 (≈ 100 ms of audio per
moq-lite group). Stream-creation rate drops from 50/sec to 10/sec, well
below the production cliff threshold.
- Counter resets after each endGroup; trailing partial group flushes its
FIN on capture EOF so the listener doesn't sit on a half-open stream.
MoqLiteNestsSpeaker + connectNestsSpeaker:
- Plumb framesPerGroup through to the broadcaster. Default flows from
NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP so production callers
get the mitigation without code changes; tests can override to 1 to
keep their groupId-per-frame assertions.
Tests:
- All harness interop tests (RoundTrip, LateJoin, MultiPeer, Mute,
Unsubscribe, SpeakerClose, both Reconnecting variants) plus the prod
NostrnestsProdAudioTransmissionTest now pass framesPerGroup = 1
explicitly to preserve their per-frame group/object id assertions.
- All :quic and :nestsClient JVM tests pass.
Trade-off: a brand-new subscriber that joins mid-broadcast picks up at the
next group boundary. With 5 frames/group the late-join initial gap is up
to 100 ms (perceptually inaudible for live audio). The previous wire shape
gave at most 20 ms.
Investigation plan for the underlying QUIC cliff lives in
nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md. The
attempted-then-reverted suspend/STREAMS_BLOCKED commit is summarised
under "Attempt 1" in that doc.
|