nick/amethyst
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(nests): T16 closure roadmap + 4 follow-up plans

Browse Source 
Plans the path from 'infra shipped' to 'full coverage with
correct behaviours'. Five new plan docs in nestsClient/plans/:

1. 2026-05-07-t16-closure-roadmap.md — index + priority order.
   Three sequential steps + one independent track.

2. 2026-05-07-moq-relay-routing-investigation.md (Priority 1)
   — root-cause the moq-relay 0.10.x per-broadcast subscribe-
   routing race. Step-by-step: capture relay-side trace, write
   minimum reproducer, file upstream OR bump moq-relay version.
   Smoking gun + hypotheses already in place from the
   late-join-flake investigation; this plan turns that into
   actionable next steps.

3. 2026-05-07-tighten-cross-stack-assertions.md (Priority 2)
   — once the routing race is closed, replace the five soft-
   pass scenarios in BrowserInteropTest with hard floors.
   Lists each scenario, its current soft-pass behavior, and
   the proposed hard threshold.

4. 2026-05-07-cross-stack-interop-ci-gating.md (Priority 3)
   — re-add the hang-interop + browser-interop GitHub Actions
   jobs that were dropped in 6829ab727 / b94737de7. Includes
   the exact YAML to restore and a 10/10 sweep stability bar
   before merge.

5. 2026-05-07-framespergroup-production-rerun.md (independent
   track) — re-run the HCgOY two-phone field tests against
   current nostrnests production at multiple framesPerGroup
   values to settle whether the test pin (5) and production
   default (50) can converge.

Estimated total: 2.5-3.5 days of focused work to fully close
T16. After these four: I7 post-reconnect cliff and I12 GOAWAY
remain open as genuinely upstream-territory items, tracked in
their existing investigation docs.
This commit is contained in:
Claude
2026-05-07 14:36:21 +00:00
parent 806cbe7a3b
commit 32065901c8
5 changed files with 723 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nestsClient/plans/2026-05-07-cross-stack-interop-ci-gating.md
+171
View File
@@ -0,0 +1,171 @@
# Plan: wire CI gating for the cross-stack interop suite
**Status:** specced — pickup ready.
**Depends on:**
- `2026-05-07-moq-relay-routing-investigation.md` closed
- `2026-05-07-tighten-cross-stack-assertions.md` closed
- 5/5 sweep stability verified
This is the FINAL step of the T16 closure. With stable hard-pass
suites, CI gating becomes safe and meaningful.
## What's needed
### A) `.github/workflows/build.yml` — the hang-interop job
The job was originally part of this branch but removed per
maintainer ask in commit `6829ab727` ("ci(nests): drop hang-interop
job from build.yml") because the suite was flaky. Resurrect the
exact same shape:
```yaml
hang-interop:
needs: lint
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v6
- uses: actions/setup-java@v5
with: { distribution: 'zulu', java-version: 21 }
- uses: gradle/actions/setup-gradle@v4
with:
cache-read-only: ${{ github.ref != 'refs/heads/main' }}
- uses: dtolnay/rust-toolchain@stable
- uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
nestsClient/tests/hang-interop/target
~/.cache/amethyst-nests-interop/hang-interop-cargo
key: ${{ runner.os }}-cargo-${{ hashFiles('nestsClient/tests/hang-interop/Cargo.lock', 'nestsClient/tests/hang-interop/REV') }}
restore-keys: |
${{ runner.os }}-cargo-
- name: Run cross-stack interop suite
run: ./gradlew :nestsClient:jvmTest -DnestsHangInterop=true
- uses: actions/upload-artifact@v7
if: failure()
with:
name: Hang Interop Test Reports
path: nestsClient/build/reports/tests/jvmTest/
```
The `git show 6829ab727 -- .github/workflows/build.yml` reverse
gives the exact diff to re-add. Linux-only is correct: the cargo
install of moq-relay 0.10.x has nontrivial native deps
(aws-lc-sys, ring) that take 5+ min cold; cached runs ~30 s.
macOS / Windows would double matrix cost without catching new
defects.
### B) `.github/workflows/build.yml` — the browser-interop job
Same shape as A, plus bun + Playwright caching:
```yaml
browser-interop:
needs: lint
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
# ...same checkout + JDK + Gradle + Rust + cargo cache as hang-interop...
- uses: oven-sh/setup-bun@v2
with: { bun-version: 1.3.11 }
- uses: actions/cache@v4
with:
path: |
nestsClient-browser-interop/node_modules
nestsClient-browser-interop/dist
key: ${{ runner.os }}-bun-${{ hashFiles('nestsClient-browser-interop/package.json', 'nestsClient-browser-interop/bun.lock') }}
- uses: actions/cache@v4
with:
path: ~/.cache/ms-playwright
key: ${{ runner.os }}-playwright-${{ hashFiles('nestsClient-browser-interop/package.json') }}
- name: Run browser cross-stack interop suite
run: |
./gradlew :nestsClient:jvmTest \
--tests "com.vitorpamplona.nestsclient.interop.native.BrowserInteropTest" \
-DnestsHangInterop=true \
-DnestsBrowserInterop=true
- uses: actions/upload-artifact@v7
if: failure()
with:
name: Browser Interop Test Reports
path: |
nestsClient/build/reports/tests/jvmTest/
nestsClient-browser-interop/test-results/
nestsClient-browser-interop/playwright-report/
```
Same `git show b94737de7 -- .github/workflows/build.yml` reverse
gives the exact diff (`feat/nests-browser-interop`'s removal
commit).
### C) Cross-link with `:cli` interop tests
The existing `nests-interop` opt-in pattern already lives in
`cli/tests/nests/nests-interop.sh`. Confirm both new jobs run
in parallel with that without resource contention. They use
different ports (NativeMoqRelayHarness reserves `ServerSocket(0)`)
so they're independent at the network level.
## Stability bar
Before flipping the CI switch, run:
```
for i in 1 2 3 4 5 6 7 8 9 10; do
echo "=== run $i ==="
./gradlew :nestsClient:jvmTest \
--tests HangInteropTest \
--tests BrowserInteropTest \
-DnestsHangInterop=true \
-DnestsBrowserInterop=true \
--rerun-tasks 2>&1 | grep -E "FAILED]|BUILD"
done
```
10/10 BUILD SUCCESSFUL. If even one fails, do NOT wire CI; loop
back to the routing investigation.
## CI runtime budget
- Hang-interop job: ~3-4 min on warm cache (one suite run, 60 s
long-broadcast scenario dominates), ~8 min cold (cargo install
moq-relay).
- Browser-interop job: ~5-7 min warm (Chromium boot × N
scenarios), ~10 min cold (Playwright install).
- Both run in parallel after `lint`.
Total CI overhead: ~5-10 min on the critical path beyond the
existing build matrix. Acceptable.
## Documentation updates
After CI is green:
1. `nestsClient/plans/2026-05-06-cross-stack-interop-test-results.md`
— replace the "CI integration: Not wired" section with "wired,
tracking flake-rate at 0/N runs".
2. `nestsClient/plans/2026-05-06-cross-stack-interop-test-gap-matrix.md`
— replace `#6 CI integration: ⏸ deferred` with `✅ live`.
3. Pick a maintainer to monitor the first 2 weeks of CI runs and
bisect any new flake immediately (don't let it accumulate as
"known flake" again).
## Acceptance criteria
- Both jobs added to `build.yml` and merge to main.
- 10/10 sweep before merge.
- First 2 weeks post-merge: ≥ 95% green rate. If lower, the
routing investigation isn't really done — pull the jobs again
until it is.
## Optional follow-ups
- **Add I-12 GOAWAY scenario IF an IETF moq-transport target lands.**
Currently N/A in moq-lite-03 per
`cross-stack-interop-test-results.md`'s I12 section. If an IETF
target ever ships, this is the cross-stack regression test.
- **Surface `framesPerGroup` as a per-deployment config** if the
framesPerGroup-rerun outcome shows the two rigs can't converge
(see `2026-05-07-framespergroup-production-rerun.md`).
nestsClient/plans/2026-05-07-framespergroup-production-rerun.md
+132
View File
@@ -0,0 +1,132 @@
# Plan: re-run HCgOY field tests against current production
**Status:** specced — pickup ready (needs prod-rig access).
**Cross-ref:** `nestsClient/plans/2026-05-07-framespergroup-reconciliation.md`
documents the conflict between the cliff plan's value (5) and
HCgOY's value (50). This plan settles which is current truth.
## What we're trying to settle
Production currently runs `NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP = 50`
based on HCgOY two-phone field tests at commit `6e4df4a`
(2026-05-05) which observed:
| `framesPerGroup` | streams/sec @ 50 fps | observed cliff window |
|---|---|---|
| 1 | 50 | ~3 s |
| 5 | 10 | ~13 s |
| 10 | 5 | ~16 s |
| 50 | 1 | not reached |
| 100 | 0.5 | never observed |
Interop tests pin `5` because the local `--auth-public ""` minimal
relay setup hits a *different* cliff (per-stream byte volume) at
`framesPerGroup = 50`.
The production deployment may have changed since 2026-05-05:
- nostrnests may have updated their `moq-relay` version
- their resource limits may have shifted
- the upstream `kixelated/moq` may have addressed one or both
cliffs
We don't know without re-running. **Cliff value at `framesPerGroup
= 5` may now be unbounded — if so, both rigs can converge on 5
and the test pin matches the prod default.**
## Test setup
### Rig A — interop env (already exists)
`./gradlew :nestsClient:jvmTest --tests HangInteropTest -DnestsHangInterop=true`
runs against local `moq-relay 0.10.25 --auth-public "" --tls-generate localhost`.
Long-broadcast scenario `long_broadcast_60s_tone_round_trips` is
the existing 60-second sustained-stream test pinned at
`framesPerGroup = 5`.
To probe other values, parameterize the helper:
```kotlin
// HangInteropTest.kt — runSpeakerToHangListen helper
private suspend fun runSpeakerToHangListen(
speakerSeconds: Int,
framesPerGroup: Int = 5, // ← new parameter
// ...existing params
): HangListenOutput { ... }
```
Then add scenarios `long_broadcast_60s_framesPerGroup_50` etc.
that pin different values. Expected outcomes today (per the
2026-05-01 cliff plan):
- `framesPerGroup = 5` — passes (current pin)
- `framesPerGroup = 10` — passes
- `framesPerGroup = 50` — fails (per-stream byte volume cliff
at the local minimal relay)
### Rig B — production deployment (needs maintainer access)
This is the gap. Rerunning the HCgOY two-phone field test pattern
needs:
- Two physical Android devices
- A nostrnests room (production endpoint
`wss://nostrnests.com/v0/ws` per `NestsConnect.kt`)
- The diagnostic-build of Amethyst that emits the cliff-detector
trace logs (see commit `6e4df4a`'s logcat run from 18:37:43..18:38:08)
The maintainer should run the same test pattern at:
- `framesPerGroup = 5` (current test value)
- `framesPerGroup = 10`
- `framesPerGroup = 25` (untested, midpoint)
- `framesPerGroup = 50` (current prod value)
- `framesPerGroup = 100` (full group; the cliff plan's
`fpg-all` reference)
For each, broadcast for 120 s and observe:
- Total streams forwarded by the relay
- Time-to-cliff if any (when the listener-side flow-control
snapshot stops incrementing `peerInitiatedUni`)
- Audio dropouts (perceptual + sample-count)
## Decision matrix after data lands
| Rig A passes at | Rig B passes at | Decision |
|---|---|---|
| 5, 10 | 5, 10, 25, 50, 100 | Keep prod 50; test pins 5 (current state) |
| 5, 10, 50 | 5, 10, 25, 50, 100 | Both rigs converge → unify on 50, test pin matches prod |
| 5, 10 | 50, 100 only (5 still cliffs) | Current state is correct; document permanently as "two cliffs in one binary" |
| 5 only | 50, 100 only (5 still cliffs) | The two cliffs are real; consider per-environment config |
| 5, 10, 50 | 50, 100 only (5 still cliffs) | Test rig fixed; production cliff still hits at 5. Test pin doesn't catch prod regression — bigger problem. |
The "decision" column drives the production-side change (or
non-change) to `DEFAULT_FRAMES_PER_GROUP`.
## What lands as code
After Rig B data is in:
1. Update kdoc on `NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP`
citing the new run's logcat dates / commit.
2. If the values converge, change the default to match. Update
the test-side `framesPerGroup = 5` pin to match the new
default — keep both rigs aligned.
3. If values still diverge, document it explicitly as a known
environment-dependent value. Consider exposing
`framesPerGroup` as a per-deployment config (currently only
exposed as a constructor parameter — wire to a config knob if
product wants per-deployment tuning).
4. Update `nestsClient/plans/2026-05-07-framespergroup-reconciliation.md`'s
"Recommendation" section with the data-driven outcome.
## Acceptance criteria
- A logcat dump from Rig B with `framesPerGroup = 5` for ≥ 60 s
showing whether the cliff still hits at ~13 s.
- Decision logged in the framesPerGroup reconciliation doc.
- If a value change lands, the test-side pin and production
default agree.
## Out of scope
- The local interop env's per-stream byte cliff at
`framesPerGroup = 50`. That's a separate thread; addressing it
would require either a different relay configuration or
patching moq-relay itself.
nestsClient/plans/2026-05-07-moq-relay-routing-investigation.md
+183
View File
@@ -0,0 +1,183 @@
# Plan: investigate moq-relay 0.10.x per-broadcast subscribe-routing race
**Status:** specced — pickup ready.
**Owns:** the residual flake that affects four T16 scenarios:
`late_join_listener_still_decodes_tail`,
`packet_loss_1pct_does_not_kill_audio`,
`long_broadcast_60s_tone_round_trips`, and the new
`chromium_publisher_*_kotlin_listener_recovers` tests in browser-tier.
**Blocks:** CI gating for `:nestsClient:jvmTest -DnestsHangInterop=true`
and `-DnestsBrowserInterop=true`. Re-evaluate the
`hang-interop` / `browser-interop` workflow jobs once this is closed.
**Cross-refs:**
- `nestsClient/plans/2026-05-07-late-join-catalog-flake-investigation.md`
(smoking-gun trace + 4 mitigation attempts, 2 of which were
net-negative and reverted).
- `nestsClient/plans/2026-05-07-i7-post-reconnect-cliff-investigation.md`
(same kind of routing issue surfacing across publisher cycles).
## What we know
For broadcasts that fail (sample suffixes from the trace:
`10d4b6f2…`, `c75e2648…`, `f1be27ef…`):
1. The Kotlin speaker side logs:
- `ANNOUNCE inbound prefix='' → emitted Active suffix='<broadcast>'`
- …then NOTHING for the entire test window.
- Audio publisher's `send()` repeats `no inboundSubs` at 50 fps
until the test times out.
2. The Rust hang-listen side logs:
- `connected, version=moq-lite-03`
- `broadcast announced path=<broadcast>`
- `subscribe started id=0 broadcast=<broadcast> track=catalog.json`
- …then `subscribe error err=remote error: code=0` exactly when
the speaker tears down at the broadcast-window end (= relay
forwarding `Cancel`).
The relay accepts the listener's wire SUBSCRIBE on its downstream
connection but **never opens an upstream SUBSCRIBE bidi to the
speaker** for the failing broadcast. The upstream subscribe-pump
that's supposed to forward downstream subscribes to the speaker
isn't wired up by the time the listener subscribes.
For broadcasts that succeed (same trace, same JVM, different test):
```
ANNOUNCE inbound prefix='' → emitted Active suffix=<broadcast>
SUBSCRIBE inbound id=0 broadcast=<broadcast> track='catalog.json'
SUBSCRIBE registered id=0 …
openGroupStream subId=0 seq=0
…
```
All log lines fire; the relay forwards the upstream subscribe
within ~1 ms of the downstream subscribe. Failure mode is binary:
the relay does or does not forward.
## Hypotheses, ranked by next step
### H1 — moq-rs 0.10.x bug in `Origin::announced()` → upstream-pump setup race
`Origin::announced().await` returns the broadcast as soon as the
speaker's announce lands in the relay's origin map. The relay's
upstream-subscribe pump for that broadcast is set up on a separate
async path. If a downstream listener subscribes before the pump is
fully wired, the SUBSCRIBE accepts on the listener's wire (the
relay has the broadcast in its origin) but never propagates
upstream.
**Status:** prime suspect; see "smoking gun" in
`2026-05-07-late-join-catalog-flake-investigation.md`.
### H2 — interaction with the `--auth-public ""` minimal config
The harness boots moq-relay with `--auth-public ""` to skip JWT
issuance. Production runs with full auth. It's possible the
auth-public path takes a different code path through the relay's
origin/subscribe wiring that's racier than the auth'd path.
**Status:** plausible; would explain why the flake isn't reported
against the production deployment.
### H3 — local-only timing race that resolves at higher latency
Loopback (127.0.0.1) has near-zero RTT. The relay's internal
async setup may rely on the natural RTT cushion of a real network
to sequence upstream-subscribe-pump setup vs. downstream-subscribe
acceptance. We bypass that cushion in the test.
**Status:** less likely (the cliff plan's evidence shows lossy
network actually makes things *worse* via the `serve_group` task
pool) but worth ruling out.
## Investigation plan
### Step 1 — capture relay-side traces
`NativeMoqRelayHarness.boot` currently launches `moq-relay` with
`RUST_LOG=info`. Bump to `RUST_LOG=moq_relay=trace,moq_lite=trace`
and capture stderr to a per-test tempfile. Cross-reference with
the failing test's hang-listen stdout AND the speaker-side
`Log.d("NestTx")` traces (already captured in
`<system-err>` per JUnit XML).
Concretely: in `NativeMoqRelayHarness.kt` add a `--log-stderr`
option that the @BeforeTest hook sets to a `<test-method>.log`
path under `nestsClient/build/relay-logs/`. The Kotlin side
already has the speaker-side traces; the Rust side is the gap.
What to look for in the failed-broadcast log:
- Was a SUBSCRIBE bidi opened to the speaker for the failing
broadcast suffix? (moq_lite span: `subscribe`).
- Did the relay's `Origin::publish_broadcast` call complete
before the listener's SUBSCRIBE arrived?
- Any `track.unused()` resolves on the publisher-side track that
would explain immediate cancellation?
### Step 2 — write a minimal reproducer
If Step 1 shows the bug is independent of our test framework,
extract a minimum reproducer:
```rust
// reproducer.rs
let mut cmd = std::process::Command::new("moq-relay")
.args(&["--server-bind", "127.0.0.1:0", "--auth-public", "",
"--tls-generate", "localhost"])
.spawn()?;
// Run a moq-lite SPEAKER on one client, a moq-lite LISTENER on
// another, both pointed at the relay. Listener subscribes immediately
// after the speaker announces. Repeat 100×; count how many succeed.
```
Then strip the SPEAKER's announce timing, the LISTENER's subscribe
timing, the relay's `--auth-public` flag — bisect to the smallest
form that still reproduces.
### Step 3 — file upstream
If Step 1 / 2 confirm a moq-rs bug, file a `kixelated/moq` issue
with:
- The reproducer.
- Smoking-gun trace pair from our test harness.
- Pin to moq-rs version `0.10.25` (per `nestsClient/tests/hang-interop/REV`).
- Cross-link to existing
`2026-05-01-quic-stream-cliff-investigation.md`'s open follow-up
#1 (the per-subscriber forward-queue cliff is a sister bug).
### Step 4 — try newer moq-relay version
Bump `MOQ_RELAY_VERSION` in `nestsClient/tests/hang-interop/REV`
and `nestsClient/build.gradle.kts` to the next minor release on
crates.io (whatever's current at the time of pickup). Run the 5×
sweep. If the flake disappears, the upstream may have already
fixed it; we can pin past 0.10.x.
**Risk:** newer moq-relay versions may have wire-format changes
that break our current `moq-lite-03` ALPN pin. The browser
harness's `@moq/lite` 0.2.x client offers `moq-lite-04` AND
`moq-lite-03`, so a newer relay that drops `03` would still
negotiate fine via 04.
## Acceptance criteria
- Sweep `for i in 1 2 3 4 5; do ./gradlew :nestsClient:jvmTest
--tests HangInteropTest -DnestsHangInterop=true --rerun-tasks; done`
passes 5/5.
- Browser-tier sweep similarly stable.
- Either:
(a) Upstream issue filed with reproducer (if the bug is in
moq-rs and we can't fix it locally), OR
(b) Local fix applied (e.g. version bump + REV update +
Cargo.lock regenerate).
## Out of scope
- The `:quic` module's `MAX_STREAMS_UNI` extension fix
(`d391ae1d`) — already shipped, separate concern.
- The production-side `framesPerGroup` reconciliation
(`2026-05-07-framespergroup-production-rerun.md`) — independent.
nestsClient/plans/2026-05-07-t16-closure-roadmap.md
+123
View File
@@ -0,0 +1,123 @@
# T16 closure roadmap — full coverage with correct behaviours
**Goal state.** Every spec'd cross-stack scenario green in suite-mode
sweeps, asserting its full design intent (no soft-passes, no vacuous
threshold loosening), with CI gating live and stable.
**Where we are.** The merged `claude/cross-stack-interop-test-XAbYB`
branch ships 22 of 23 spec'd scenarios; each passes individually.
Suite-mode runs hit a residual moq-relay 0.10.x routing race on a
specific subset (~40-60% flake rate). Five scenarios soft-pass the
listener side as a known-flake mitigation. CI is intentionally
unwired pending stability.
This roadmap takes the suite from "passes individually" to "passes
in suite + CI" through three sequential plans. None should be
parallelized — each unblocks the next.
## Priority 1 — `2026-05-07-moq-relay-routing-investigation.md`
**Why first.** The race is the root cause of every soft-pass and
the reason CI isn't wired. Without resolving it, downstream plans
mask flake rather than catch regressions.
**What lands.**
- Either an upstream moq-relay version bump that closes the bug,
OR a documented relay configuration tweak that does.
- Or, if neither: a filed `kixelated/moq` issue with reproducer +
trace pair, plus a documented decision to keep CI unwired until
upstream resolves.
**Acceptance bar.** 5/5 sweep BUILD SUCCESSFUL on the existing
HangInteropTest + BrowserInteropTest with their CURRENT soft-pass
assertions intact. (The next step tightens those.)
## Priority 2 — `2026-05-07-tighten-cross-stack-assertions.md`
**Why second.** Once the suite is stable, every soft-pass that
returned vacuous-pass on listener-side 0-frame outcomes is now
HIDING regressions instead of side-stepping flakes. Replace each
with a hard floor.
**What lands.**
- Five BrowserInteropTest scenarios get hard sample-count + FFT
floors (or tightened existing ones).
- Gap matrix updated to reflect hard-pass coverage.
**Acceptance bar.** 5/5 sweep AGAIN, this time with hard
assertions. If anything fail-flakes, the routing investigation
isn't really done — loop back.
## Priority 3 — `2026-05-07-cross-stack-interop-ci-gating.md`
**Why third.** Stability + hard-asserts in place → CI is now a
net positive (catches regressions, doesn't burn maintainer time
on false reds).
**What lands.**
- Re-add `hang-interop` job (was at commit `6829ab727`'s parent;
`git show 6829ab727 -- .github/workflows/build.yml` reverse
gives the exact diff).
- Re-add `browser-interop` job (same pattern, plus bun +
Playwright caches).
- Documentation update across the results plan + gap matrix.
**Acceptance bar.** 10/10 sweep before merge; ≥ 95% CI green
rate over the first 2 weeks. If lower, the upstream race isn't
fully closed — pull the jobs.
## Independent track — `2026-05-07-framespergroup-production-rerun.md`
This one **doesn't block the closure roadmap**. It can run any
time after Priority 1 is done; it settles whether the test pin
(5) and production default (50) can converge, or whether they
must remain different. Either outcome is shippable.
**What lands.**
- Logcat data from a fresh two-phone field test against current
nostrnests production at multiple `framesPerGroup` values.
- A data-driven decision on whether to change the production
default, the test pin, or neither.
**Why it's parallelizable.** Doesn't gate the test suite or CI;
it gates a one-line code change to `NestMoqLiteBroadcaster`'s
default constant.
## After all four close — what remains
Two open items, both genuinely upstream:
1. **I7 post-reconnect listener cliff** —
`2026-05-07-i7-post-reconnect-cliff-investigation.md`. The I7
reverse scenario passes its 2.5 s threshold but a regression
test of "all post-reconnect data arrives" would require the
moq-relay 0.10.x per-broadcast forward queue fix. Same upstream
class as the routing race.
2. **I12 GOAWAY** — only re-emerges if an IETF moq-transport
target lands (currently moq-lite-03 only). Tracked in
`2026-05-06-cross-stack-interop-test-results.md`.
Beyond those: T16 reaches "full coverage with correct behaviours"
when this roadmap closes.
## Estimated wallclock
- Priority 1: 1–2 days (depends on whether upstream version bump
fixes it, or we have to file + wait for upstream).
- Priority 2: 0.5 day (mechanical replacement of soft-passes
with floors, plus rerun verification).
- Priority 3: 0.5 day (re-add CI jobs, run the 10× sweep, merge).
- Independent track (framesPerGroup): 0.5 day (needs prod-rig
access).
Total: 2.5–3.5 days of focused work to take T16 from "infra
shipped" to "fully closed".
## Plan files
- `2026-05-07-moq-relay-routing-investigation.md`
- `2026-05-07-tighten-cross-stack-assertions.md`
- `2026-05-07-cross-stack-interop-ci-gating.md`
- `2026-05-07-framespergroup-production-rerun.md`
- (this file) `2026-05-07-t16-closure-roadmap.md`
nestsClient/plans/2026-05-07-tighten-cross-stack-assertions.md
+114
View File
@@ -0,0 +1,114 @@
# Plan: tighten cross-stack interop assertions to hard-pass
**Status:** specced — pickup ready.
**Depends on:** `2026-05-07-moq-relay-routing-investigation.md`
must be closed first (the soft-passes exist *because* of that flake;
removing them while the flake is unresolved produces fail-flakes,
not regression catches).
## Why soft passes exist today
Five scenarios currently soft-pass (vacuous-pass on listener-side
0-frame outcomes) to keep the test suite from fail-flaking on the
upstream relay-routing race documented in
`2026-05-07-late-join-catalog-flake-investigation.md`:
| Scenario | File | Soft-pass behavior |
|---|---|---|
| `chromium_listener_late_join_still_decodes_tail` | `BrowserInteropTest.kt` | `if (pcm.size <= warmupSamples) return` |
| `chromium_listener_mid_broadcast_mute_shortens_pcm` | `BrowserInteropTest.kt` | same |
| `chromium_decoder_no_errors_through_warmup_window` (I14) | `BrowserInteropTest.kt` | no `decoderOutputs >= 4` floor |
| `chromium_publisher_baseline_kotlin_listener_decodes` | `BrowserInteropTest.kt` | hard-asserts publisher framesIn; soft-asserts listener |
| `chromium_publisher_reconnect_kotlin_listener_recovers` (Browser I7) | `BrowserInteropTest.kt` | same as baseline |
All five have hard assertions on the `framesIn` / publisher-side
behavior; the listener-side is what's soft. None of these are
reaching their full design intent.
## Soft-pass justification audit (per scenario)
Re-read each scenario's kdoc. The soft-pass is honest right now
(captured 0 frames means harness flake, not regression). Once the
relay-routing race is fixed, the listener side becomes deterministic
and the soft-pass is no longer load-bearing — at that point, the
soft-pass HIDES regressions (a real T8/T11/T13 break could land in
a 0-frame outcome and pass vacuously).
## Tighten plan
### Step 1 — confirm sweep stability
After the routing investigation lands, run:
```
for i in 1 2 3 4 5; do
echo "=== run $i ==="
./gradlew :nestsClient:jvmTest \
--tests HangInteropTest \
--tests BrowserInteropTest \
-DnestsHangInterop=true \
-DnestsBrowserInterop=true \
--rerun-tasks 2>&1 | grep -E "FAILED]|BUILD"
done
```
5/5 BUILD SUCCESSFUL with 0 `FAILED` lines = stability achieved.
### Step 2 — replace each soft-pass with a hard floor
For each scenario in the table above, remove the
`if (pcm.size <= warmupSamples) return` short-circuit and replace
with a meaningful sample-count floor. Tighten thresholds based on
observed steady-state numbers (see each scenario's kdoc for what
"steady-state" looks like — most run ≥ 1 s of audio in green-state).
Concretely:
- **Late-join**: `assertTrue(pcm.size >= ...)` floor.
Steady-state captures ~3 s on a 5 s broadcast with 2 s late-join,
minus warmup. Threshold: `≥ 1.5 s` — comfortably under the
steady-state but well over zero.
- **Mute-window**: tighten the upper bound (current 5.5 s) to
~5.0 s. Add a lower bound asserting `≥ 2.5 s` — proves audio
arrived AND the muted segment shortened the total.
- **I14**: re-add `decoderOutputs >= 4` (3 warmup + ≥ 1 audio).
Current absence-only assertion is partial coverage.
- **Browser publisher baseline + reconnect**: remove the
vacuous-pass branches, add a `≥ 0.5 s of audio after warmup`
floor for baseline and `≥ 2.5 s` for reconnect (matches the
hang-tier I7 threshold).
### Step 3 — reverify
Re-run the 5× sweep. All scenarios must hard-pass 5/5. If a
scenario flakes after tightening, the relay-routing investigation
isn't fully done and we revert the tightening on that scenario
until it is.
### Step 4 — update the gap matrix
`nestsClient/plans/2026-05-06-cross-stack-interop-test-gap-matrix.md`
currently lists I14 with "browser ⏳" pending; flip to "✅" once
its hard floor is in. Same for any I-scenarios that now have
hard floors on both tiers.
## Acceptance criteria
- All BrowserInteropTest scenarios run with hard sample-count
AND FFT-peak assertions (no `return@runBlocking` short-circuits
on pcm.size).
- All HangInteropTest scenarios already hard-pass — no change
needed there.
- Gap matrix updated to reflect hard-pass coverage on each T#.
- Results plan updated to remove the "soft-pass on flake"
language.
## Risk: post-tightening flake
If any scenario fail-flakes after tightening, the routing
investigation isn't really done. Don't paper over with a wider
threshold; that's the same trap as the soft-passes. Either:
(a) revert the tightening on that scenario and keep
investigating, OR
(b) widen the threshold ONLY if the new value still excludes
the regression mode the test was designed to catch.