chase residual stream loss: bump SO_RCVBUF to 4 MiB + diagnostic UDP counters + cheaper test collector

Three targeted changes against the residual sustained-load stream loss
that survived the MAX_STREAMS_UNI fix (long broadcasts and bursty
scenarios still cliffed mid-pump). Each addresses one of the round-1
hypotheses; the next prod sweep should tell us how much each
contributes.

quic/UdpSocket:
  - Bump SO_RCVBUF to 4 MiB at bind time. The kernel default (~200 KB
    on Linux/macOS, similar on Android) holds barely 130 MTU-sized
    datagrams; a multi-second moq-lite broadcast that the relay fans
    out to several subscribers transiently overflows this. Anything
    queued past `rmem` is silently dropped by the kernel and never
    reaches our QUIC stack.
  - Add lifetime diagnostic counters (receivedDatagramCount,
    receivedByteCount, receiveBufferSizeBytes) on the expect surface
    so commonMain code can read them via the new udpStatsSupplier
    hook on QuicConnection without each platform having to surface its
    own native bookkeeping.
  - QuicConnectionDriver wires the supplier on start; the connection's
    flowControlSnapshot bundles the stats into a new
    QuicFlowControlSnapshot.udp field.

QuicConnection.flowControlSnapshot:
  - Now also surfaces advertisedMaxStreamsUni / advertisedMaxStreamsBidi
    and peerInitiatedUniCount / peerInitiatedBidiCount so a sweep can
    see the listener-side stream-cap evolution alongside speaker-side
    counters that were already there.

SendTraceScenario:
  - verbosePerFrame default flipped from true to false. The per-frame
    `tx i=…` and `rx[idx] gid=…` logs go through InteropDebug ->
    JUnit's stdout capture; at 50 frames/sec the capture thread
    serialises the receive coroutine and starves the QUIC read loop,
    biasing the recorded received count downward on long runs. Tests
    that need the per-frame timeline opt in explicitly.
  - Replace the receive-side CopyOnWriteArrayList sink (O(N) per add,
    ~1.1M element copies cumulative for a 1500-frame run) with
    Collections.synchronizedList(ArrayList(capacityHint)) — O(1)
    amortised. Snapshot under the same lock at end of run so the
    JDK's iterator-locking contract is satisfied.
  - fc-pre / fc-post-pump / fc-post-grace lines now include
    `udpDatagrams=… udpBytes=… udpRcvBuf=…` plus
    `advertisedMaxStreamsUni=… peerInitiatedUni=…` so the next sweep's
    diagnostic dump shows directly whether the kernel actually
    delivered datagrams the relay sent.

Plan doc: nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md
documents round-2 changes and lists the remaining open hypotheses
(relay-side per-subscriber queue policy, receive-side
MAX_STREAM_DATA threshold).

All :quic + :nestsClient JVM tests pass.
This commit is contained in:
Claude
2026-05-01 15:07:09 +00:00
parent 253680fd82
commit 46af65591a
6 changed files with 246 additions and 46 deletions
@@ -57,52 +57,79 @@ matches frame count) — so the loss is on the relay→listener path.
(no late delivery during grace), confirming the listener never
recovers once it stalls.
## Next investigation: relay-side per-subscription buffer
## Round 2 — chasing the residual
The post-fix profile suggests a *relay-side* per-subscriber forward
buffer or rate-limiter that drops new uni streams once the listener
falls behind. Hypotheses to test:
After the original `MAX_STREAMS_UNI` fix, residual loss appeared in
sustained / bursty scenarios. Three round-2 changes against
likely contributors:
1. **Listener QUIC RX coalescing.** Our `:quic` reads packets from
the UDP socket one at a time. Under high stream-creation rate
the kernel's UDP receive buffer might drop datagrams before we
pick them up. `recv` overhead per packet (we use a fresh
`ByteBuffer` per call in `UdpSocket.kt:60-70`) compounds. Worth
profiling with `dropwatch` or just `cat /proc/net/snmp | grep -i
udp` (RcvbufErrors counter) during a stuck run.
### 2a. Test-side stdout serialisation (hypothesis 2)
2. **Receive-side `MAX_STREAM_DATA` extension.** Symmetric to the
`MAX_STREAMS` fix: if our writer's threshold for re-crediting
per-stream data is too conservative under high stream count,
the relay's per-stream send credit to us could starve. Current
threshold is "consumed > limit - window/2" with `window =
1 MB`. With 80-byte audio frames we'd never hit it on a single
stream, but for the 16 KB payload test, 8 KB consumed per
stream is well under the 500 KB threshold — so this isn't the
primary cause.
`SendTraceScenario.Scenario.verbosePerFrame` defaulted to `true`,
which logged a per-frame `tx i=…` and `rx[idx] gid=…` line via
`InteropDebug.checkpoint`. Under JUnit's stdout capture, that's a
synchronous write per event. At 50 frames/sec sustained the
capture thread can serialise the receive coroutine and starve the
QUIC read loop, biasing the `received` count downward. **Default
flipped to `false`**; specific tests opt in for debugging.
3. **Application-thread stalling on the listener side.** Our test
`SendTraceScenario` collector logs every `rx[…]` line via
`InteropDebug.checkpoint`, which writes to stdout under a JUnit
capture. Under 50+ stream/sec rates the stdout flush may
serialize the receive coroutine. Worth disabling
`verbosePerFrame` and re-running long scenarios — if numbers
improve dramatically, the receive side is application-rate
limited, not transport-bound.
### 2b. Test-side `CopyOnWriteArrayList` add cost
4. **moq-rs relay's `--cluster-replication` / per-subscriber queue
policy.** Some upstream knobs control how many in-flight uni
streams the relay queues per subscriber before dropping new
ones. Worth checking `moq-relay --help | grep -i buffer` or
filing an issue at `kixelated/moq` for the relay's behaviour
spec under saturation.
The collector accumulated arrivals in a `CopyOnWriteArrayList`,
which is O(N) per add — copying the entire underlying array on
every `+=`. For 1500-frame runs that's ~1.1 M element copies and
~35 MB of memory ops cumulative across the run. Replaced with
`Collections.synchronizedList(ArrayList(capacityHint))` for O(1)
amortized adds. Snapshot at end via
`synchronized(sink) { ArrayList(sink) }`.
The MAIN production bug (audio cuts out mid-broadcast at frame ~99
in two-user rooms) is FIXED. The residual is a separate
investigation that mostly affects extreme scenarios — long
broadcasts, large payloads, fan-out asymmetry, fast bursts — and
should be tracked as a separate plan once one of the hypotheses
above is confirmed.
### 2c. Listener-side kernel UDP receive buffer (hypothesis 1)
`UdpSocket.connect` did not configure `SO_RCVBUF`, so the kernel's
default applied (~200 KB on Linux/macOS, similarly small on
Android). At MTU-sized datagrams that's room for ~130 packets. A
multi-second moq-lite broadcast that the relay fans out across
multiple subscribers can transiently exceed this — anything
queued past `rmem` is silently dropped by the kernel and never
reaches our QUIC stack, manifesting as "subscription stops
mid-broadcast even though `publisher.send` keeps returning true".
Bumped to 4 MiB at socket-bind time (Linux doubles + clamps via
`rmem_max`, so the effective cap is whatever the kernel allows up
to `min(8 MiB, rmem_max)`).
Added lifetime UDP datagram counters (`receivedDatagramCount`,
`receivedByteCount`, `receiveBufferSizeBytes`) to the `UdpSocket`
expect surface; the driver wires them into the existing
`QuicConnection.flowControlSnapshot` via a `udpStatsSupplier` hook.
The `fc-pre / fc-post-pump / fc-post-grace` lines now include
`udpDatagrams=… udpBytes=… udpRcvBuf=…` so a future sweep can
correlate "frames missing on subscription" against "datagrams the
kernel actually delivered".
### Hypotheses still open after round 2
3. **moq-rs relay per-subscriber queue policy.** Even if our
listener kernel + collector are infinitely fast, the relay
itself may have an in-flight uni-stream cap per subscriber that
drops new groups once the listener falls behind. The
asymmetric loss in `sweep_3subs sub[2]` (77/100 while sub[0]
and sub[1] got 100/100) is consistent with a per-subscriber
queue rather than a connection- or relay-wide limit. Worth
filing an issue at `kixelated/moq` once we have one more sweep
confirming the round-2 changes don't already close the gap.
4. **Receive-side `MAX_STREAM_DATA` thresholds.** Probably not
the cause for 80-byte audio frames (1 MB per-stream window vs
~80 bytes used per stream) but may bite the 16 KB-payload
scenario. Worth a follow-up only if `sweep_payload_16kb`
doesn't recover.
## Re-running after round 2
The acceptance criterion is the same: every sweep row showing
`received=N/N missing=[]`, except the explicit late-join scenarios
(`sweep_late_subscribe_after_25` / `_after_50`) which legitimately
report `received < N` due to from-latest semantics.
**Reproduces against:** `https://moq.nostrnests.com:4443` (production
relay). Not consistently reproducible against the local