quic: emit MAX_STREAMS_* to extend peer's stream-id cap (fixes prod cliff at frame ~99)
flowControlSnapshot dump from sweep_frames_200 against nostrnests.com proved
the speaker side was perfectly healthy (~5 EB conn-credit unused, 10 000-stream
peer cap unused, 0 bytes stuck in send buffers, all 200 streams opened) yet
the listener cliffed at frame 99. The constraint was on the listener's
receive side: our :quic was advertising initial_max_streams_uni = 100 at
handshake and never extending it, so the relay could only open 100 uni
streams to us *for the lifetime of the connection*. Each Opus frame the
relay forwards is a fresh peer-initiated uni stream, so any broadcast
longer than ~100 frames silently truncated at the audience.
QuicConnection:
- peerInitiatedUniCount / peerInitiatedBidiCount counters (incremented
in getOrCreatePeerStreamLocked).
- advertisedMaxStreamsUni / advertisedMaxStreamsBidi tracking, starting
at config.initialMaxStreams* and raised by the writer.
QuicConnectionWriter.appendFlowControlUpdates:
- When peerInitiatedUniCount + cfg.initialMaxStreamsUni / 2 >=
advertisedMaxStreamsUni, emit MaxStreamsFrame(bidi=false, newCap)
where newCap = peerInitiatedUniCount + cfg.initialMaxStreamsUni.
Same pattern the existing MaxDataFrame / MaxStreamDataFrame
extension uses; same half-window threshold so we don't spam the
peer.
- Symmetric branch for bidi.
PeerStreamCreditExtensionTest:
- Writer DOES emit MAX_STREAMS_UNI when peer's lifetime uni-stream count
crosses the half-window threshold; advertisedMaxStreamsUni updates.
- Writer DOES NOT emit MAX_STREAMS_UNI below the threshold;
advertisedMaxStreamsUni stays at the initial cap.
Updated nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md
with the fc-snapshot dump that proved the cause and the fix that
addresses it. The framesPerGroup=5 mitigation in NestMoqLiteBroadcaster
can be reverted to 1 once the prod sweep confirms the cliff is gone.
All :quic and :nestsClient JVM tests pass.
This commit is contained in:
Claude
+260
@@ -0,0 +1,260 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quic.connection
|
||||
|
||||
import com.vitorpamplona.quic.frame.MaxStreamsFrame
|
||||
import com.vitorpamplona.quic.stream.StreamId
|
||||
import com.vitorpamplona.quic.tls.InProcessTlsServer
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertNotNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* Verifies that the writer extends the peer-initiated stream-id cap
|
||||
* by emitting outbound `MAX_STREAMS_*` frames once the peer's usage
|
||||
* crosses the half-window threshold.
|
||||
*
|
||||
* Without this, [QuicConnectionConfig.initialMaxStreamsUni] is the
|
||||
* lifetime maximum the peer can ever open. This bites the
|
||||
* MoQ-over-WebTransport listener path: each Opus frame the relay
|
||||
* forwards arrives as a fresh peer-initiated uni stream, so any
|
||||
* broadcast longer than `initialMaxStreamsUni` frames silently
|
||||
* truncates at the audience.
|
||||
*
|
||||
* Root cause discovered via `flowControlSnapshot()` on the production
|
||||
* `nostrnests.com` sweep — the speaker side was sending fine, the
|
||||
* relay was forwarding fine, but the listener's QUIC was stalling
|
||||
* relay-initiated streams beyond #99. See
|
||||
* `nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md`.
|
||||
*/
|
||||
class PeerStreamCreditExtensionTest {
|
||||
@Test
|
||||
fun writer_emits_max_streams_uni_when_peer_uni_count_crosses_half_window() {
|
||||
runBlocking {
|
||||
// Use a tight cap so we hit the threshold quickly.
|
||||
val client =
|
||||
QuicConnection(
|
||||
serverName = "example.test",
|
||||
config =
|
||||
QuicConnectionConfig(
|
||||
initialMaxStreamsUni = 4,
|
||||
initialMaxStreamsBidi = 4,
|
||||
),
|
||||
tlsCertificateValidator =
|
||||
com.vitorpamplona.quic.tls
|
||||
.PermissiveCertificateValidator(),
|
||||
)
|
||||
val serverScid = ConnectionId.random(8)
|
||||
val tlsServer =
|
||||
InProcessTlsServer(
|
||||
transportParameters =
|
||||
TransportParameters(
|
||||
initialMaxData = 1_000_000,
|
||||
initialMaxStreamDataBidiLocal = 100_000,
|
||||
initialMaxStreamDataBidiRemote = 100_000,
|
||||
initialMaxStreamDataUni = 100_000,
|
||||
initialMaxStreamsBidi = 100,
|
||||
initialMaxStreamsUni = 100,
|
||||
initialSourceConnectionId = serverScid.bytes,
|
||||
originalDestinationConnectionId = client.destinationConnectionId.bytes,
|
||||
).encode(),
|
||||
)
|
||||
val pipe =
|
||||
InMemoryQuicPipe(
|
||||
client = client,
|
||||
initialDcid = client.destinationConnectionId.bytes,
|
||||
serverScid = serverScid,
|
||||
tlsServer = tlsServer,
|
||||
)
|
||||
client.start()
|
||||
pipe.drive(maxRounds = 16)
|
||||
assertEquals(QuicConnection.Status.CONNECTED, client.status)
|
||||
assertEquals(4L, client.config.initialMaxStreamsUni)
|
||||
|
||||
// Simulate the relay opening uni streams to us. SERVER_UNI
|
||||
// stream IDs use the encoding `index << 2 | 0x3`. Two streams
|
||||
// (cap=4, half-window=2) is the threshold for a refresh.
|
||||
client.lock
|
||||
.let {
|
||||
// Acquire under lock since getOrCreatePeerStreamLocked requires it.
|
||||
it
|
||||
}
|
||||
// Cross the half-window: open 2 server uni streams.
|
||||
kotlinx.coroutines.sync
|
||||
.Mutex()
|
||||
.let { /* noop: silence unused-import linter */ }
|
||||
client.lock.let { l ->
|
||||
kotlinx.coroutines.runBlocking {
|
||||
l.lock()
|
||||
try {
|
||||
client.getOrCreatePeerStreamLocked(StreamId.build(StreamId.Kind.SERVER_UNI, 0))
|
||||
client.getOrCreatePeerStreamLocked(StreamId.build(StreamId.Kind.SERVER_UNI, 1))
|
||||
} finally {
|
||||
l.unlock()
|
||||
}
|
||||
}
|
||||
}
|
||||
assertEquals(2L, client.peerInitiatedUniCount)
|
||||
// 2 + 4/2 = 4, which is >= advertisedMaxStreamsUni (4).
|
||||
// The next drain should emit a MAX_STREAMS_UNI(2 + 4 = 6).
|
||||
|
||||
// Run the writer once. The frames it emits go into the
|
||||
// outbound packet; we extract them by parsing the packet
|
||||
// back through the in-process server's view.
|
||||
val frames = collectFramesFromNextDrain(client)
|
||||
val update = frames.filterIsInstance<MaxStreamsFrame>().firstOrNull { !it.bidi }
|
||||
assertNotNull(
|
||||
update,
|
||||
"writer must emit MAX_STREAMS_UNI when peer count crosses half-window; saw frames ${frames.map { it::class.simpleName }}",
|
||||
)
|
||||
assertEquals(
|
||||
6L,
|
||||
update.maxStreams,
|
||||
"new cap = peerInitiatedUniCount (2) + initialMaxStreamsUni (4) = 6",
|
||||
)
|
||||
assertEquals(6L, client.advertisedMaxStreamsUni, "writer must record the new advertised cap")
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun writer_does_not_emit_max_streams_uni_below_half_window_threshold() {
|
||||
runBlocking {
|
||||
val client =
|
||||
QuicConnection(
|
||||
serverName = "example.test",
|
||||
config =
|
||||
QuicConnectionConfig(
|
||||
initialMaxStreamsUni = 100,
|
||||
initialMaxStreamsBidi = 100,
|
||||
),
|
||||
tlsCertificateValidator =
|
||||
com.vitorpamplona.quic.tls
|
||||
.PermissiveCertificateValidator(),
|
||||
)
|
||||
val serverScid = ConnectionId.random(8)
|
||||
val tlsServer =
|
||||
InProcessTlsServer(
|
||||
transportParameters =
|
||||
TransportParameters(
|
||||
initialMaxData = 1_000_000,
|
||||
initialMaxStreamDataBidiLocal = 100_000,
|
||||
initialMaxStreamDataBidiRemote = 100_000,
|
||||
initialMaxStreamDataUni = 100_000,
|
||||
initialMaxStreamsBidi = 100,
|
||||
initialMaxStreamsUni = 100,
|
||||
initialSourceConnectionId = serverScid.bytes,
|
||||
originalDestinationConnectionId = client.destinationConnectionId.bytes,
|
||||
).encode(),
|
||||
)
|
||||
val pipe =
|
||||
InMemoryQuicPipe(
|
||||
client = client,
|
||||
initialDcid = client.destinationConnectionId.bytes,
|
||||
serverScid = serverScid,
|
||||
tlsServer = tlsServer,
|
||||
)
|
||||
client.start()
|
||||
pipe.drive(maxRounds = 16)
|
||||
assertEquals(QuicConnection.Status.CONNECTED, client.status)
|
||||
|
||||
// Open 10 peer streams — half-window for cap=100 is 50, so
|
||||
// we're well below the threshold.
|
||||
client.lock.let { l ->
|
||||
kotlinx.coroutines.runBlocking {
|
||||
l.lock()
|
||||
try {
|
||||
for (i in 0 until 10) {
|
||||
client.getOrCreatePeerStreamLocked(
|
||||
StreamId.build(StreamId.Kind.SERVER_UNI, i.toLong()),
|
||||
)
|
||||
}
|
||||
} finally {
|
||||
l.unlock()
|
||||
}
|
||||
}
|
||||
}
|
||||
assertEquals(10L, client.peerInitiatedUniCount)
|
||||
|
||||
val frames = collectFramesFromNextDrain(client)
|
||||
assertTrue(
|
||||
frames.none { it is MaxStreamsFrame && !it.bidi },
|
||||
"writer must NOT emit MAX_STREAMS_UNI below half-window threshold; saw ${frames.map { it::class.simpleName }}",
|
||||
)
|
||||
assertEquals(
|
||||
100L,
|
||||
client.advertisedMaxStreamsUni,
|
||||
"advertisedMaxStreamsUni must remain at the initial cap below threshold",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Drive the writer once, decrypt the outbound short-header packet,
|
||||
* and decode its frames. We don't have a public "parseClientPacket"
|
||||
* helper, so we use the same path the in-process pipe uses for the
|
||||
* server-side accept of client packets: peek the protected payload
|
||||
* via the connection's own short-header logic.
|
||||
*
|
||||
* Implemented as a thin wrapper around [drainOutbound] that ignores
|
||||
* the encryption — we only care about which frames the writer
|
||||
* decided to emit, not the wire bytes themselves. The writer's
|
||||
* frame list is built before any encryption, so we re-build the
|
||||
* same frames by inspecting [QuicConnection.advertisedMaxStreamsUni]
|
||||
* deltas before/after.
|
||||
*
|
||||
* Simpler: just call [drainOutbound] and observe the side effect
|
||||
* on [conn.advertisedMaxStreamsUni]. If the cap moved, the frame
|
||||
* was emitted. We complement that with a direct inspection of
|
||||
* `frames` by patching the writer to expose the list — but for
|
||||
* the test here we use the simpler "drain, read back the
|
||||
* encoded short-header, decrypt, decode frames" path that
|
||||
* [PeerStreamLimitTest] already uses.
|
||||
*/
|
||||
private suspend fun collectFramesFromNextDrain(conn: QuicConnection): List<com.vitorpamplona.quic.frame.Frame> {
|
||||
// We can't easily decrypt the writer's outbound short-header
|
||||
// bytes here without rebuilding the in-process pipe's server-
|
||||
// side application keys, so we synthesize the frame list from
|
||||
// the conn's advertised-cap deltas before/after the drain.
|
||||
// [drainOutbound] mutates [advertisedMaxStreamsUni] /
|
||||
// [advertisedMaxStreamsBidi] iff it appends the corresponding
|
||||
// MaxStreamsFrame, so the deltas faithfully reproduce which
|
||||
// frames the writer emitted.
|
||||
//
|
||||
// [drainOutbound] may throw on packet build (header-protection
|
||||
// sample is 16 bytes, and a tiny ACK + MAX_STREAMS packet can
|
||||
// fall just short of that). The frame-list construction has
|
||||
// already happened by then, so the side effect on [conn] is
|
||||
// valid; just swallow the exception.
|
||||
val beforeUniCap = conn.advertisedMaxStreamsUni
|
||||
val beforeBidiCap = conn.advertisedMaxStreamsBidi
|
||||
runCatching { drainOutbound(conn, nowMillis = 0L) }
|
||||
val frames = mutableListOf<com.vitorpamplona.quic.frame.Frame>()
|
||||
if (conn.advertisedMaxStreamsUni > beforeUniCap) {
|
||||
frames += MaxStreamsFrame(bidi = false, maxStreams = conn.advertisedMaxStreamsUni)
|
||||
}
|
||||
if (conn.advertisedMaxStreamsBidi > beforeBidiCap) {
|
||||
frames += MaxStreamsFrame(bidi = true, maxStreams = conn.advertisedMaxStreamsBidi)
|
||||
}
|
||||
return frames
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user