diag(quic-interop): dump first 10 packets fully + add live-driver-flow synth test
Two adds for the multiplex investigation.
(1) inspect-multiplexing.sh: previous histograms aggregate over the
whole run. Add full-frame-array dump of the first 10 packet_sent
AND first 10 packet_received events so we can see whether the
FIRST chunk burst (~13 streams in one packet) or dribbled (1 per).
(2) MultiplexingAioquicTpsTest: synchronous drain test using EXACTLY
the TPs aioquic gave us in the failing run (initial_max_data=1MB,
initial_max_stream_data_bidi_remote=1MB, initial_max_streams_bidi=
128) and ~80-byte HEADERS-frame-sized payloads. PASSES with 7
packets / 9.1 streams per packet — proving the writer's coalescing
is fine under aioquic's flow-control budget. So the bug is NOT in
the writer; it's in the live driver flow that
MultiplexingCoalescingTest doesn't exercise (concurrent send loop
+ parser + real socket).
The first-10 dump from inspect should localize this further:
- if first packet has 13 stream frames → writer burst, bug is
server-side timing or driver-loop scheduling
- if first packet has 1 stream frame → writer producing 1 per call
in production for some condition my synchronous test doesn't
cover
https://claude.ai/code/session_01HcvfQq1ttPV9PkRoJb4nyT
This commit is contained in:
@@ -76,7 +76,19 @@ if [[ -n "$QLOG" ]]; then
|
|||||||
| sort -n | uniq -c
|
| sort -n | uniq -c
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "=============== peer transport_parameters (initial_max_data is the key) ==============="
|
echo "=============== first 10 packet_sent events (full) ==============="
|
||||||
|
# The very first stream-bearing packets reveal whether we burst
|
||||||
|
# the chunk's 64 streams OR dribbled them out one per RTT. Look
|
||||||
|
# at the `frames` array — if it has 13 stream entries, writer
|
||||||
|
# is bursting; if 1, writer is regressed.
|
||||||
|
grep '"name":"transport:packet_sent"' "$QLOG" | head -n 10
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=============== first 10 packet_received events (full) ==============="
|
||||||
|
# If we sent N streams in burst, server should respond with N
|
||||||
|
# responses interleaved as it processes them. Spread tells us
|
||||||
|
# server-side serial cost.
|
||||||
|
grep '"name":"transport:packet_received"' "$QLOG" | head -n 10
|
||||||
# If initial_max_data is small (e.g. < 1000 bytes), our writer
|
# If initial_max_data is small (e.g. < 1000 bytes), our writer
|
||||||
# gets throttled to one stream per packet because connBudget
|
# gets throttled to one stream per packet because connBudget
|
||||||
# exhausts after the first stream. Each subsequent stream has to
|
# exhausts after the first stream. Each subsequent stream has to
|
||||||
|
|||||||
+151
@@ -0,0 +1,151 @@
|
|||||||
|
/*
|
||||||
|
* Copyright (c) 2025 Vitor Pamplona
|
||||||
|
*
|
||||||
|
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||||
|
* this software and associated documentation files (the "Software"), to deal in
|
||||||
|
* the Software without restriction, including without limitation the rights to use,
|
||||||
|
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||||
|
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||||
|
* subject to the following conditions:
|
||||||
|
*
|
||||||
|
* The above copyright notice and this permission notice shall be included in all
|
||||||
|
* copies or substantial portions of the Software.
|
||||||
|
*
|
||||||
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||||
|
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||||
|
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||||
|
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||||
|
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||||
|
*/
|
||||||
|
package com.vitorpamplona.quic.connection
|
||||||
|
|
||||||
|
import com.vitorpamplona.quic.frame.StreamFrame
|
||||||
|
import com.vitorpamplona.quic.tls.InProcessTlsServer
|
||||||
|
import kotlinx.coroutines.runBlocking
|
||||||
|
import kotlin.test.Test
|
||||||
|
import kotlin.test.assertEquals
|
||||||
|
import kotlin.test.assertTrue
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reproduce the aioquic-2026-05-07 multiplex-on-the-wire failure in
|
||||||
|
* a unit test. Server qlog showed:
|
||||||
|
* - 1406 packets with exactly 1 STREAM frame, 1 with 3, never more
|
||||||
|
* - first stream packets at 1665 / 1709ms (one RTT apart, NOT burst)
|
||||||
|
* - per-RTT cadence throughout
|
||||||
|
*
|
||||||
|
* Hypothesis: writer drains 1 stream per packet despite many being
|
||||||
|
* queued — under conditions specific to aioquic's TPs:
|
||||||
|
* - initial_max_data = 1MB (matches qlog)
|
||||||
|
* - initial_max_stream_data_bidi_remote = 1MB (matches qlog)
|
||||||
|
* - initial_max_streams_bidi = 128 (matches qlog)
|
||||||
|
*
|
||||||
|
* MultiplexingCoalescingTest passes with `initial_max_data = 100MB`
|
||||||
|
* — which is 100x more credit than aioquic gives. This test mirrors
|
||||||
|
* the exact aioquic TPs to see if the smaller credit triggers the
|
||||||
|
* 1-stream-per-packet shape.
|
||||||
|
*
|
||||||
|
* If this test asserts ≤6 packets and PASSES, the bug is NOT in the
|
||||||
|
* synchronous drain path — it's in the live driver flow (which
|
||||||
|
* MultiplexingCoalescingTest doesn't exercise).
|
||||||
|
*
|
||||||
|
* If it FAILS with one stream per packet, we have a deterministic
|
||||||
|
* reproduction and can fix the writer.
|
||||||
|
*/
|
||||||
|
class MultiplexingAioquicTpsTest {
|
||||||
|
@Test
|
||||||
|
fun multiplex_64_streams_with_aioquic_TPs_should_still_coalesce() =
|
||||||
|
runBlocking {
|
||||||
|
val client = handshakedClientMatchingAioquic()
|
||||||
|
|
||||||
|
// ~80-byte HTTP/3 HEADERS frame payload — what
|
||||||
|
// Http3GetClient actually emits per stream after QPACK
|
||||||
|
// encoding. Smaller payloads (50 bytes) under-stress the
|
||||||
|
// coalescing path.
|
||||||
|
val n = 64
|
||||||
|
val payloadPerStream = 80
|
||||||
|
val streams =
|
||||||
|
client.openBidiStreamsBatch((0 until n).toList()) { stream, i ->
|
||||||
|
stream.send.enqueue(ByteArray(payloadPerStream) { (i and 0xff).toByte() })
|
||||||
|
stream.send.finish()
|
||||||
|
stream
|
||||||
|
}
|
||||||
|
assertEquals(n, streams.size)
|
||||||
|
|
||||||
|
val packets = mutableListOf<ByteArray>()
|
||||||
|
while (true) {
|
||||||
|
val pkt = drainOutbound(client, nowMillis = 1L) ?: break
|
||||||
|
packets += pkt
|
||||||
|
}
|
||||||
|
|
||||||
|
val totalBytes = packets.sumOf { it.size }
|
||||||
|
val streamsPerPacket = n.toDouble() / packets.size.coerceAtLeast(1)
|
||||||
|
println(
|
||||||
|
"[MultiplexingAioquicTpsTest] 64 streams of $payloadPerStream bytes " +
|
||||||
|
"drained into ${packets.size} packets " +
|
||||||
|
"(${(streamsPerPacket * 10).toInt() / 10.0} streams/pkt, " +
|
||||||
|
"totalBytes=$totalBytes)",
|
||||||
|
)
|
||||||
|
|
||||||
|
// Same threshold as MultiplexingCoalescingTest. 64 streams
|
||||||
|
// × ~110 bytes (80 payload + ~30 overhead) ≈ 7 KB; at
|
||||||
|
// ~1100-byte payload budget per packet that's 6-7 packets.
|
||||||
|
assertTrue(
|
||||||
|
packets.size <= 8,
|
||||||
|
"expected ≤8 packets, got ${packets.size} — " +
|
||||||
|
"writer regressed to one-stream-per-packet under aioquic TPs",
|
||||||
|
)
|
||||||
|
assertTrue(
|
||||||
|
streamsPerPacket >= 8.0,
|
||||||
|
"expected ≥8 streams/packet, got $streamsPerPacket",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun handshakedClientMatchingAioquic(): QuicConnection =
|
||||||
|
runBlocking {
|
||||||
|
val client =
|
||||||
|
QuicConnection(
|
||||||
|
serverName = "example.test",
|
||||||
|
config =
|
||||||
|
QuicConnectionConfig(
|
||||||
|
// Mirror our actual local TPs from the qlog.
|
||||||
|
initialMaxData = 16L * 1024 * 1024,
|
||||||
|
initialMaxStreamsBidi = 100,
|
||||||
|
initialMaxStreamsUni = 10000,
|
||||||
|
initialMaxStreamDataBidiLocal = 1L * 1024 * 1024,
|
||||||
|
initialMaxStreamDataBidiRemote = 1L * 1024 * 1024,
|
||||||
|
initialMaxStreamDataUni = 1L * 1024 * 1024,
|
||||||
|
),
|
||||||
|
tlsCertificateValidator =
|
||||||
|
com.vitorpamplona.quic.tls
|
||||||
|
.PermissiveCertificateValidator(),
|
||||||
|
)
|
||||||
|
val serverScid = ConnectionId.random(8)
|
||||||
|
val tlsServer =
|
||||||
|
InProcessTlsServer(
|
||||||
|
transportParameters =
|
||||||
|
TransportParameters(
|
||||||
|
// Mirror aioquic-qns 2026-05-07 from the qlog.
|
||||||
|
initialMaxData = 1L * 1024 * 1024,
|
||||||
|
initialMaxStreamDataBidiLocal = 1L * 1024 * 1024,
|
||||||
|
initialMaxStreamDataBidiRemote = 1L * 1024 * 1024,
|
||||||
|
initialMaxStreamDataUni = 1L * 1024 * 1024,
|
||||||
|
initialMaxStreamsBidi = 128,
|
||||||
|
initialMaxStreamsUni = 128,
|
||||||
|
initialSourceConnectionId = serverScid.bytes,
|
||||||
|
originalDestinationConnectionId = client.destinationConnectionId.bytes,
|
||||||
|
).encode(),
|
||||||
|
)
|
||||||
|
val pipe =
|
||||||
|
InMemoryQuicPipe(
|
||||||
|
client = client,
|
||||||
|
initialDcid = client.destinationConnectionId.bytes,
|
||||||
|
serverScid = serverScid,
|
||||||
|
tlsServer = tlsServer,
|
||||||
|
)
|
||||||
|
client.start()
|
||||||
|
pipe.drive(maxRounds = 16)
|
||||||
|
assertEquals(QuicConnection.Status.CONNECTED, client.status)
|
||||||
|
client
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user