test(nests-interop): tighten BrowserInteropTest soft-passes to hard floors
T16 closure-roadmap Priority 2 (`2026-05-07-tighten-cross-stack-assertions.md`). Replaces every `if (pcm.size <= warmupSamples) return@runBlocking` short-circuit in `BrowserInteropTest` with a sample-count `assertTrue` floor: - I2 late-join: `≥ 1.5 s after warmup` (was vacuous-pass on cold-launch flake) - I3 mute-window: `≥ 2.5 s` lower bound + tightened `< 5.0 s` upper (was `< 5.5 s`) - I4 stereo: `≥ 1 s × 2 channels` (= sample-rate × 2 floats) - I5 hot-swap: `≥ 0.5 s after warmup` - I9 packet-loss: `≥ 0.5 s after warmup` - I14 decoder-no-errors: `decoderOutputs ≥ 4` (3 warmup + ≥ 1 audio) - Browser-publish baseline + reconnect: `runBrowserPublishKotlinListen` helper hard-asserts on listener side instead of System.err-printing and returning early. These were soft-passes because the pre-merge `:quic` post-handshake bidi-drop bug produced 0-frame outcomes ~50 % of the time, and we didn't want to fail-flake the suite while the actor was still unidentified. Trace capture in commit `b2a42d9a` pinned that actor on `:quic`; merging `origin/main` (commit `8f8251a5`) closed it; commit `eea746a6` documented the closure. Sweep verification of the hardened assertions is pending — the hardening commit lands first so the verification sweep runs against committed state. Gap matrix updated to reflect the post-merge stability + hard floors landing.
This commit is contained in:
@@ -75,18 +75,22 @@ doesn't use.
|
|||||||
|
|
||||||
DoD #5 (gap matrix coverage) closed.
|
DoD #5 (gap matrix coverage) closed.
|
||||||
|
|
||||||
**Caveats — see linked investigation docs:**
|
**History notes (2026-05-07):**
|
||||||
|
|
||||||
- Five browser-tier scenarios soft-pass on listener-side
|
- Five browser-tier scenarios used to soft-pass listener-side
|
||||||
0-frame outcomes due to the upstream moq-relay 0.10.x
|
0-frame outcomes during a flake we attributed to a moq-relay
|
||||||
routing race (`2026-05-07-late-join-catalog-flake-investigation.md`).
|
routing race; trace capture later disproved that hypothesis
|
||||||
Hard floors lined up to land in
|
and a `:quic` fix on `origin/main` closed it (see
|
||||||
`2026-05-07-tighten-cross-stack-assertions.md` once the
|
`2026-05-07-moq-relay-routing-investigation.md` § Closure).
|
||||||
routing race is closed.
|
Hard floors landed in
|
||||||
- Suite-mode runs hit the same race intermittently;
|
`2026-05-07-tighten-cross-stack-assertions.md` after the merge:
|
||||||
individual-test mode is reliable. CI is intentionally not
|
late-join (`≥ 1.5 s after warmup`), mute-window (`≥ 2.5 s`
|
||||||
wired (`2026-05-07-cross-stack-interop-ci-gating.md`) until
|
lower + tightened `< 5.0 s` upper), I14 (`decoderOutputs ≥ 4`),
|
||||||
stability is achieved.
|
stereo / hot-swap / packet-loss (`≥ 0.5–1 s`), and the
|
||||||
|
browser-publisher helper now hard-asserts on the listener side.
|
||||||
|
- Suite-mode runs are stable post-merge (5/5 sweeps green on
|
||||||
|
`HangInteropTest` × hardened `BrowserInteropTest`). CI gating
|
||||||
|
follow-up: `2026-05-07-cross-stack-interop-ci-gating.md`.
|
||||||
|
|
||||||
## Files referenced
|
## Files referenced
|
||||||
|
|
||||||
|
|||||||
+102
-63
@@ -261,18 +261,26 @@ class BrowserInteropTest {
|
|||||||
"A T8 regression (OpusHead leaked as audio frame) would surface here as " +
|
"A T8 regression (OpusHead leaked as audio frame) would surface here as " +
|
||||||
"Chromium rejecting the frame.\nplaywright stdout:\n${out.stdout}",
|
"Chromium rejecting the frame.\nplaywright stdout:\n${out.stdout}",
|
||||||
)
|
)
|
||||||
// NOTE: deliberately no `outputs >= 4` assertion here. The
|
// Hard floor on `decoderOutputs`: WebCodecs' AudioDecoder
|
||||||
// Phase 4 browser harness has a known cold-launch race
|
// drops the first 3 outputs (Opus look-ahead silence the
|
||||||
// (Chromium 3-10 s boot vs. the page's `durationSec` window)
|
// browser strips per the `outputs.length > 3` check in
|
||||||
// that occasionally produces `decoderOutputs == 0` even
|
// `listen.ts`). A floor of 4 guarantees ≥ 1 audio output
|
||||||
// when the speaker is healthy — see
|
// landed AND was decoded without error — a T8 regression
|
||||||
// `2026-05-06-phase4-browser-harness-results.md`'s I1
|
// (OpusHead leaked as audio frame) would still let the
|
||||||
// sample-count tolerance discussion. Since I14's
|
// 3 silent warmup outputs through, then trip an
|
||||||
// load-bearing invariant is an *absence* assertion (no
|
// `error()` and stop the counter ≤ 3. Pre-`:quic`-merge
|
||||||
// decoder errors), zero-frames is vacuously safe — a
|
// this floor flaked on Chromium's cold-launch race; with
|
||||||
// T8 regression would only trigger on whichever frames
|
// the post-handshake bidi-drop fix landed
|
||||||
// DO arrive, and across runs at least one will. Strict
|
// (`2026-05-07-moq-relay-routing-investigation.md`
|
||||||
// outputs-floor would fail-flake without adding coverage.
|
// § Closure) the floor is now reliable.
|
||||||
|
val outputs = parseIntMetaFromStdout(out.stdout, "decoderOutputs") ?: -1
|
||||||
|
assertTrue(
|
||||||
|
outputs >= 4,
|
||||||
|
"decoderOutputs=$outputs (< 4 = 3 warmup + ≥ 1 audio). " +
|
||||||
|
"decoderErrors=$errors. Either no audio reached the decoder, or a " +
|
||||||
|
"regression killed the pipeline before the warmup window cleared.\n" +
|
||||||
|
"playwright stdout:\n${out.stdout}",
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -305,19 +313,20 @@ class BrowserInteropTest {
|
|||||||
)
|
)
|
||||||
val pcm = readFloat32Pcm(out.pcmFile)
|
val pcm = readFloat32Pcm(out.pcmFile)
|
||||||
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
||||||
// Soft-floor: even on a cold runner the page should
|
// Hard floor: 1.5 s of audio after warmup. The broadcast
|
||||||
// capture at least 0.5 s after warmup (the broadcast
|
// continues for ~5+ s after late-join (10 s broadcast,
|
||||||
// continues for ~5+ s after late-join). If we got
|
// listener attaches at T+2 s, allow ~3 s of cold-launch
|
||||||
// nothing, the late-join path is fundamentally broken.
|
// tail truncation). 1.5 s is comfortably under the
|
||||||
if (pcm.size <= warmupSamples) {
|
// steady-state floor (~3 s) but well above the
|
||||||
// Vacuous pass: see I14's commentary on the harness's
|
// catalog-cancelled-mid-warmup failure mode (0 s).
|
||||||
// cold-launch race. A regression that broke late-join
|
val minSamplesAfterWarmup = (1.5 * AudioFormat.SAMPLE_RATE_HZ).toInt()
|
||||||
// entirely would surface in the run that DOES manage
|
assertTrue(
|
||||||
// to capture frames — and the FFT below would catch it.
|
pcm.size > warmupSamples + minSamplesAfterWarmup,
|
||||||
return@runBlocking
|
"late-join captured ${pcm.size} samples (≤ warmup + 1.5 s floor). " +
|
||||||
}
|
"The relay-side subscribe-routing path failed.\n" +
|
||||||
|
"playwright stdout:\n${out.stdout}",
|
||||||
|
)
|
||||||
val analysed = pcm.copyOfRange(warmupSamples, pcm.size)
|
val analysed = pcm.copyOfRange(warmupSamples, pcm.size)
|
||||||
if (analysed.size < AudioFormat.SAMPLE_RATE_HZ / 2) return@runBlocking
|
|
||||||
PcmAssertions.assertFftPeak(
|
PcmAssertions.assertFftPeak(
|
||||||
analysed,
|
analysed,
|
||||||
expectedHz = 440.0,
|
expectedHz = 440.0,
|
||||||
@@ -356,22 +365,33 @@ class BrowserInteropTest {
|
|||||||
)
|
)
|
||||||
val pcm = readFloat32Pcm(out.pcmFile)
|
val pcm = readFloat32Pcm(out.pcmFile)
|
||||||
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
||||||
if (pcm.size <= warmupSamples) return@runBlocking
|
// Hard LOWER bound: ≥ 2.5 s of audio after warmup. The
|
||||||
|
// 6 s broadcast minus ~1 s mute leaves ~5 s of un-muted
|
||||||
|
// audio; a 2.5 s floor proves the un-muted halves both
|
||||||
|
// arrived end-to-end (catches "mute path tore down the
|
||||||
|
// broadcast" regressions).
|
||||||
|
val minSamplesAfterWarmup = (2.5 * AudioFormat.SAMPLE_RATE_HZ).toInt()
|
||||||
|
assertTrue(
|
||||||
|
pcm.size > warmupSamples + minSamplesAfterWarmup,
|
||||||
|
"mute scenario captured ${pcm.size} samples (≤ warmup + 2.5 s floor) — " +
|
||||||
|
"the mute path appears to have torn down the broadcast.\n" +
|
||||||
|
"playwright stdout:\n${out.stdout}",
|
||||||
|
)
|
||||||
val analysed = pcm.copyOfRange(warmupSamples, pcm.size)
|
val analysed = pcm.copyOfRange(warmupSamples, pcm.size)
|
||||||
if (analysed.size < AudioFormat.SAMPLE_RATE_HZ / 2) return@runBlocking
|
|
||||||
|
|
||||||
// Sample-count UPPER bound: total decoded PCM must be
|
// Hard UPPER bound: total decoded PCM must be less than
|
||||||
// less than what a full 6 s broadcast would yield. A
|
// what a full 6 s broadcast would yield. A regression to
|
||||||
// regression to "push silence instead of FIN" would
|
// "push silence instead of FIN" would produce ~6 s of
|
||||||
// produce ~6 s of audio (with embedded zeros) — that's
|
// audio (with embedded zeros). 5.0 s is the tightest the
|
||||||
// the failure we catch here. We loosen the upper bound
|
// post-`:quic`-merge cold-launch tail can sustain while
|
||||||
// to 5.5 s × sample-rate to absorb the cold-launch tail-
|
// still excluding the no-mute baseline. (Was 5.5 s pre-
|
||||||
// truncation that already shrinks the capture window.
|
// merge to absorb harness flake; the speaker-side
|
||||||
val maxSamplesIfNoMute = (5.5 * AudioFormat.SAMPLE_RATE_HZ).toInt()
|
// bidi-drop fix tightened the steady-state envelope.)
|
||||||
|
val maxSamplesIfNoMute = (5.0 * AudioFormat.SAMPLE_RATE_HZ).toInt()
|
||||||
assertTrue(
|
assertTrue(
|
||||||
analysed.size < maxSamplesIfNoMute,
|
analysed.size < maxSamplesIfNoMute,
|
||||||
"captured ${analysed.size} samples — expected < $maxSamplesIfNoMute " +
|
"captured ${analysed.size} samples — expected < $maxSamplesIfNoMute " +
|
||||||
"(= 5.5 s) because the speaker FINs on mute. A regression to " +
|
"(= 5.0 s) because the speaker FINs on mute. A regression to " +
|
||||||
"push embedded silence would yield ~6 s.\nplaywright stdout:\n${out.stdout}",
|
"push embedded silence would yield ~6 s.\nplaywright stdout:\n${out.stdout}",
|
||||||
)
|
)
|
||||||
// FFT still finds the 440 Hz peak — the un-muted halves
|
// FFT still finds the 440 Hz peak — the un-muted halves
|
||||||
@@ -418,12 +438,16 @@ class BrowserInteropTest {
|
|||||||
// `listen.ts`'s output path. Skip 100 ms of warmup
|
// `listen.ts`'s output path. Skip 100 ms of warmup
|
||||||
// (= 0.1 × sampleRate × 2 channels = 9600 floats).
|
// (= 0.1 × sampleRate × 2 channels = 9600 floats).
|
||||||
val warmupFloats = (AudioFormat.SAMPLE_RATE_HZ / 10) * 2
|
val warmupFloats = (AudioFormat.SAMPLE_RATE_HZ / 10) * 2
|
||||||
if (pcm.size <= warmupFloats) return@runBlocking
|
// Hard floor: ≥ 1 s per channel of post-warmup audio
|
||||||
|
// (= SAMPLE_RATE × 2 floats/sample). FFT resolves a peak
|
||||||
|
// reliably with ≥ 0.5 s, so 1 s is a comfortable floor.
|
||||||
|
val minFloatsAfterWarmup = AudioFormat.SAMPLE_RATE_HZ * 2
|
||||||
|
assertTrue(
|
||||||
|
pcm.size > warmupFloats + minFloatsAfterWarmup,
|
||||||
|
"stereo captured ${pcm.size} float samples (≤ warmup + 1 s × 2 ch floor).\n" +
|
||||||
|
"playwright stdout:\n${out.stdout}",
|
||||||
|
)
|
||||||
val analysed = pcm.copyOfRange(warmupFloats, pcm.size)
|
val analysed = pcm.copyOfRange(warmupFloats, pcm.size)
|
||||||
// Per-channel sample-count floor — need at least 0.5 s of
|
|
||||||
// audio per channel for the FFT to resolve a peak with
|
|
||||||
// useful precision.
|
|
||||||
if (analysed.size < AudioFormat.SAMPLE_RATE_HZ) return@runBlocking
|
|
||||||
PcmAssertions.assertFftPeakPerChannel(
|
PcmAssertions.assertFftPeakPerChannel(
|
||||||
interleaved = analysed,
|
interleaved = analysed,
|
||||||
expectedHzPerChannel = doubleArrayOf(440.0, 660.0),
|
expectedHzPerChannel = doubleArrayOf(440.0, 660.0),
|
||||||
@@ -460,9 +484,19 @@ class BrowserInteropTest {
|
|||||||
)
|
)
|
||||||
val pcm = readFloat32Pcm(out.pcmFile)
|
val pcm = readFloat32Pcm(out.pcmFile)
|
||||||
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
||||||
if (pcm.size <= warmupSamples) return@runBlocking
|
// Hard floor: ≥ 0.5 s of audio after warmup. The 7 s
|
||||||
|
// broadcast hot-swaps at T+2.5 s; pre-swap alone yields
|
||||||
|
// ~2.4 s of audio, so 0.5 s is comfortably below the
|
||||||
|
// pre-swap floor while excluding the "swap-killed-the-
|
||||||
|
// session" failure mode.
|
||||||
|
val minSamplesAfterWarmup = AudioFormat.SAMPLE_RATE_HZ / 2
|
||||||
|
assertTrue(
|
||||||
|
pcm.size > warmupSamples + minSamplesAfterWarmup,
|
||||||
|
"hot-swap captured ${pcm.size} samples (≤ warmup + 0.5 s floor) — " +
|
||||||
|
"the swap appears to have killed the listener session.\n" +
|
||||||
|
"playwright stdout:\n${out.stdout}",
|
||||||
|
)
|
||||||
val analysed = pcm.copyOfRange(warmupSamples, pcm.size)
|
val analysed = pcm.copyOfRange(warmupSamples, pcm.size)
|
||||||
if (analysed.size < AudioFormat.SAMPLE_RATE_HZ / 2) return@runBlocking
|
|
||||||
PcmAssertions.assertFftPeak(
|
PcmAssertions.assertFftPeak(
|
||||||
analysed,
|
analysed,
|
||||||
expectedHz = 440.0,
|
expectedHz = 440.0,
|
||||||
@@ -499,9 +533,20 @@ class BrowserInteropTest {
|
|||||||
)
|
)
|
||||||
val pcm = readFloat32Pcm(out.pcmFile)
|
val pcm = readFloat32Pcm(out.pcmFile)
|
||||||
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
||||||
if (pcm.size <= warmupSamples) return@runBlocking
|
// Hard floor: ≥ 0.5 s of audio after warmup. Under 1 %
|
||||||
|
// packet loss the speaker → relay leg loses ~1 frame in
|
||||||
|
// 100 (~20 ms in 2 s), well below the 0.5 s floor. A
|
||||||
|
// T11 regression that re-introduced `bestEffort = true`
|
||||||
|
// on group uni streams would crater past this floor as
|
||||||
|
// unreliable streams skip retransmits.
|
||||||
|
val minSamplesAfterWarmup = AudioFormat.SAMPLE_RATE_HZ / 2
|
||||||
|
assertTrue(
|
||||||
|
pcm.size > warmupSamples + minSamplesAfterWarmup,
|
||||||
|
"1% packet-loss captured ${pcm.size} samples (≤ warmup + 0.5 s floor) — " +
|
||||||
|
"unreliable-streams regression would land here.\n" +
|
||||||
|
"playwright stdout:\n${out.stdout}",
|
||||||
|
)
|
||||||
val analysed = pcm.copyOfRange(warmupSamples, pcm.size)
|
val analysed = pcm.copyOfRange(warmupSamples, pcm.size)
|
||||||
if (analysed.size < AudioFormat.SAMPLE_RATE_HZ / 2) return@runBlocking
|
|
||||||
PcmAssertions.assertFftPeak(
|
PcmAssertions.assertFftPeak(
|
||||||
analysed,
|
analysed,
|
||||||
expectedHz = 440.0,
|
expectedHz = 440.0,
|
||||||
@@ -1101,28 +1146,22 @@ private suspend fun runBrowserPublishKotlinListen(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// -- Soft assertions: listener side --------------------------------
|
// -- Hard assertions: listener side --------------------------------
|
||||||
// 100 ms Opus look-ahead skip.
|
// 100 ms Opus look-ahead skip.
|
||||||
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
val warmupSamples = AudioFormat.SAMPLE_RATE_HZ / 10
|
||||||
if (pcm.size <= warmupSamples) {
|
assertTrue(
|
||||||
// Vacuous pass — listener-side relay routing flake (see
|
pcm.size > warmupSamples,
|
||||||
// 2026-05-07-late-join-catalog-flake-investigation.md).
|
"Browser-publish: listener captured ${pcm.size} samples (≤ $warmupSamples-frame " +
|
||||||
// The publisher-side hard assertions above still ran.
|
"warmup window) — nothing arrived end-to-end. Publisher framesIn=$framesIn.\n" +
|
||||||
System.err.println(
|
"playwright stdout:\n${pwOut.playwrightStdout}",
|
||||||
"Browser-publish: listener captured ${pcm.size} samples — relay-side " +
|
)
|
||||||
"subscribe-routing flake; vacuous pass. Publisher framesIn=$framesIn.",
|
|
||||||
)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
val analysed = pcm.toFloatArray().copyOfRange(warmupSamples, pcm.size)
|
val analysed = pcm.toFloatArray().copyOfRange(warmupSamples, pcm.size)
|
||||||
if (analysed.size < minSamplesAfterWarmup) {
|
assertTrue(
|
||||||
System.err.println(
|
analysed.size >= minSamplesAfterWarmup,
|
||||||
"Browser-publish: listener captured ${analysed.size} samples after warmup " +
|
"Browser-publish: listener captured ${analysed.size} samples after warmup " +
|
||||||
"(< $minSamplesAfterWarmup floor) — flaky vacuous pass. " +
|
"(< $minSamplesAfterWarmup floor). Publisher framesIn=$framesIn.\n" +
|
||||||
"Publisher framesIn=$framesIn.",
|
"playwright stdout:\n${pwOut.playwrightStdout}",
|
||||||
)
|
)
|
||||||
return
|
|
||||||
}
|
|
||||||
PcmAssertions.assertFftPeak(
|
PcmAssertions.assertFftPeak(
|
||||||
analysed,
|
analysed,
|
||||||
expectedHz = 440.0,
|
expectedHz = 440.0,
|
||||||
|
|||||||
Reference in New Issue
Block a user