nick/amethyst
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(nests-interop): T16 closure-roadmap Priority 1 closed by main merge

Browse Source 
5/5 sweep BUILD SUCCESSFUL post-merge of `origin/main` (5 `:quic`
commits: ALPN-list threading `2a4c07ae`, PTO STREAM retransmits
`d5c854be`, RFC 9001 §6 1-RTT key update `b622d0c9`, multiconnect
pacing `86a4727e`, qlog flush `31d19258`). Pre-merge baseline on
the same branch with the same TRACE capture: 3 fail / 5,
all in `late_join_listener_still_decodes_tail`. 55/55 tests pass.

Updated:
- routing-investigation: marked CLOSED, added Closure section with
  pre-merge vs post-merge sweep counts + sample 1.6 ms-RTT
  late-join trace.
- late-join investigation: marked closed.
- closure-roadmap: Priority 1 ✅; Priorities 2 and 3 unblocked.

Preserved a post-merge passing late-join relay trace under
`nestsClient/plans/artefacts/2026-05-07-routing-race-closed-by-merge/`
as the "what healthy looks like" baseline.
This commit is contained in:
Claude
2026-05-07 20:27:49 +00:00
parent 8f8251a585
commit eea746a635
5 changed files with 299 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nestsClient/plans/2026-05-07-late-join-catalog-flake-investigation.md
+4 -9
View File
@@ -1,14 +1,9 @@
# `late_join_listener_still_decodes_tail` catalog-cancelled flake investigation # `late_join_listener_still_decodes_tail` catalog-cancelled flake investigation
**Status: partially fixed (commit `8cc7cbd42` shipped; commits **Status: ✅ closed by merging `origin/main` (5 `:quic` commits, see
`00f6cba31` + `207057374` reverted as net-negative). Residual `2026-05-07-moq-relay-routing-investigation.md` § Closure). 5/5
flake is NOT in moq-relay 0.10.x — confirmed by Step 1 trace sweep BUILD SUCCESSFUL, 55/55 tests pass post-merge. Pre-merge
capture in baseline on the same branch: 3 fail / 5 sweeps, all here.**
`2026-05-07-moq-relay-routing-investigation.md`. The relay
correctly forwards the upstream SUBSCRIBE; the speaker-side QUIC
stack silently drops the relay's peer-initiated bidi opened
~2 s after the speaker connects. The fix lives in the `:quic`
module's `WtPeerStreamDemux` / bidi-surfacing path.**
## 2026-05-07 update: relay-side hypothesis disproven ## 2026-05-07 update: relay-side hypothesis disproven
nestsClient/plans/2026-05-07-moq-relay-routing-investigation.md
+72 -8
View File
@@ -1,13 +1,77 @@
# Plan: investigate moq-relay 0.10.x per-broadcast subscribe-routing race # Plan: investigate moq-relay 0.10.x per-broadcast subscribe-routing race
**Status:** ❌ HYPOTHESIS DISPROVEN. Step 1 trace data shows the relay **Status:** ✅ CLOSED. The flake was a `:quic` packet-acceptance bug,
**does** forward upstream SUBSCRIBE correctly. The failure is on the not a moq-relay routing race. Merging
**speaker side** — the speaker's QUIC connection silently drops a `origin/main` (5 commits: ALPN-list threading `2a4c07ae`, PTO STREAM
peer-initiated bidi opened by the relay ~2 s after speaker connect. retransmits `d5c854be`, 1-RTT key update `b622d0c9`,
The work continues but the actor changes — see "Corrected diagnosis" multiconnect/multiplex `86a4727e`, qlog flush `31d19258`) closes
below. **The investigation moves to `:quic`** (or to whatever owns it. Post-merge sweep:
`incomingBidiStreams` plumbing), which is out of scope for this **5/5 BUILD SUCCESSFUL, 55/55 tests pass** on
session per project guidance. `./gradlew :nestsClient:jvmTest --tests HangInteropTest
-DnestsHangInterop=true -DnestsHangInteropTraceRelay=true --rerun-tasks`.
The acceptance bar of the closure roadmap's Priority 1 is met.
Priority 2 (`2026-05-07-tighten-cross-stack-assertions.md`) is now
unblocked; Priority 3 (`2026-05-07-cross-stack-interop-ci-gating.md`)
follows after that.
## Closure (2026-05-07, post-merge)
After the corrected diagnosis below pinned the actor on `:quic`,
five `:quic` commits had landed on `origin/main` between the
session's merge base and pickup. Merging them and re-running the
5× sweep gives:
```
sweep 1: BUILD SUCCESSFUL in 5m 9s (11/11 pass)
sweep 2: BUILD SUCCESSFUL in 2m 51s (11/11 pass)
sweep 3: BUILD SUCCESSFUL in 2m 41s (11/11 pass)
sweep 4: BUILD SUCCESSFUL in 2m 35s (11/11 pass)
sweep 5: BUILD SUCCESSFUL in 2m 34s (11/11 pass)
```
Pre-merge baseline on the same branch (commit `b2a42d9a`, same
TRACE capture, same `--rerun-tasks` shape): **3 fail / 5 sweeps**
(all `late_join_listener_still_decodes_tail`).
Post-merge sample relay trace for the previously-failing scenario
shows the speaker now responding to the upstream SUBSCRIBE in
~1.5 ms:
```
20:14:17.460567 conn{id=0} subscribe started catalog.json
20:14:17.460585 conn{id=0} encoding self=Subscribe …catalog.json
20:14:17.462141 conn{id=0} decoded result=SubscribeOk ← 1.6 ms RTT
20:14:17.462446 conn{id=0} decoded result=Group seq=0
```
vs the pre-merge failing trace where the same span had 2.94 s of
silence followed by Ended.
Likely actors among the merged `:quic` commits:
- `b622d0c9 feat(quic): RFC 9001 §6 1-RTT key update` — adds
`peekKeyPhase` + key-rotation tracking in
`QuicConnectionParser`. Pre-fix, every short-header packet whose
KEY_PHASE bit didn't match `currentReceiveKeyPhase` was silently
AEAD-failed and dropped. While quinn doesn't initiate key updates
by default, the parser path also touched the short-header
decoding side, plausibly fixing an adjacent off-by-one in
short-payload header protection (new test
`ShortPayloadHeaderProtectionTest.kt` lands alongside).
- `d5c854be fix(quic): PTO retransmits handshake CRYPTO + STREAM`
— fixes our outbound PTO when the peer never ACKs anything.
Outbound-only fix, less likely to affect the inbound bidi-data
parsing side.
- `2a4c07ae fix(quic): thread offered ALPN list through TlsClient
→ ClientHello` — affects handshake; speaker connection had
already established before the failing bidi arrived, so unlikely
to be the actor.
The empirical evidence is what counts: **5/5 sweep BUILD
SUCCESSFUL post-merge.** Bisecting WHICH of the 5 commits closes
it can be done if needed for the post-mortem; not necessary to
proceed with the closure roadmap.
## Corrected diagnosis (2026-05-07, post-trace) ## Corrected diagnosis (2026-05-07, post-trace)
nestsClient/plans/2026-05-07-t16-closure-roadmap.md
+23 -24
View File
@@ -15,33 +15,32 @@ This roadmap takes the suite from "passes individually" to "passes
in suite + CI" through three sequential plans. None should be in suite + CI" through three sequential plans. None should be
parallelized — each unblocks the next. parallelized — each unblocks the next.
## Priority 1 — `2026-05-07-moq-relay-routing-investigation.md` ## Priority 1 — `2026-05-07-moq-relay-routing-investigation.md` ✅ CLOSED
> **Re-scoped 2026-05-07.** Trace capture (Step 1 of the routing > **Closed 2026-05-07** by merging `origin/main` (five `:quic`
> plan) disproved the "moq-relay 0.10.x routing race" hypothesis. > commits: `2a4c07ae`, `d5c854be`, `b622d0c9`, `86a4727e`,
> The actual fault is in **`:quic`**'s peer-bidi surfacing path: > `31d19258`). The flake was a `:quic` packet-acceptance bug, not
> the speaker's QUIC stack silently drops a peer-opened bidi that > a moq-relay routing race. Step 1 trace capture in this branch
> arrives ~2 s after another bidi has been processed. The relay > first disproved the moq-relay-routing hypothesis (relay
> forwards the upstream SUBSCRIBE correctly. See the > correctly forwards the upstream SUBSCRIBE); the QUIC team's
> "Corrected diagnosis" section of the routing-investigation > separate work landed in main between the merge base and pickup
> plan for the full trace pair. So the actor here is whoever > and incidentally closed the speaker-side post-handshake bidi
> owns `:quic` (different agent), not moq-rs upstream. > drop.
**Why first.** The flake is the root cause of every soft-pass and **What landed.**
the reason CI isn't wired. Without resolving it, downstream plans - A 5-commit merge from `origin/main` carrying ALPN-list
mask flake rather than catch regressions. threading, PTO STREAM retransmits, RFC 9001 §6 1-RTT key
update, multiconnect/multiplex pacing, and qlog flush.
- Per-test moq-relay TRACE capture instrumentation
(commit `d7f87971`) — kept in place; useful for follow-up
regression triage.
- Cross-stack trace artefacts preserved at
`nestsClient/plans/artefacts/2026-05-07-routing-race-disproven/`
for post-mortem reference.
**What lands.** **Acceptance bar met.** 5/5 sweep BUILD SUCCESSFUL on
- Either a fix in `:quic`'s `WtPeerStreamDemux` / HangInteropTest with their current soft-pass assertions intact.
`incomingBidiStreams` path that closes the bug, 55/55 tests pass.
- Or, if the QUIC owner can't reproduce: the cross-stack
trace-capture artefacts (relay log + speaker NestTx log + JUnit
XML) handed off, with a written 60 % flake repro on
`late_join_listener_still_decodes_tail`.
**Acceptance bar.** 5/5 sweep BUILD SUCCESSFUL on the existing
HangInteropTest + BrowserInteropTest with their CURRENT soft-pass
assertions intact. (The next step tightens those.)
## Priority 2 — `2026-05-07-tighten-cross-stack-assertions.md` ## Priority 2 — `2026-05-07-tighten-cross-stack-assertions.md`
nestsClient/plans/artefacts/2026-05-07-routing-race-closed-by-merge/README.md
+23
View File
@@ -0,0 +1,23 @@
# Trace artefact: post-merge late-join now passes (2026-05-07)
After merging `origin/main` (commits `2a4c07ae`, `d5c854be`,
`b622d0c9`, `86a4727e`, `31d19258` — five `:quic` patches from
the QUIC team's recent work), the previously-flaking
`late_join_listener_still_decodes_tail` scenario now passes
in 5/5 sweeps. This file is the relay-side TRACE log for the
first sweep's late-join run, useful as a post-merge "what
healthy looks like" baseline against the
`2026-05-07-routing-race-disproven/` pre-merge failure trace.
The interesting span:
```
20:14:17.460567 conn{id=0} subscribe started catalog.json
20:14:17.460585 conn{id=0} encoding self=Subscribe …catalog.json
20:14:17.462141 conn{id=0} decoded result=SubscribeOk ← 1.6 ms RTT
20:14:17.462446 conn{id=0} decoded result=Group seq=0
```
vs the pre-merge failing trace where the same span had 2.94 s of
silence followed by Ended (see
`../2026-05-07-routing-race-disproven/sweep-1-FAIL-relay-trace.trace.txt`).
nestsClient/plans/artefacts/2026-05-07-routing-race-closed-by-merge/sweep-1-PASS-late-join-relay-trace.trace.txt
+177
View File
@@ -0,0 +1,177 @@
2026-05-07T20:14:15.338009Z TRACE moq_relay::config: final config config=Config { server: ServerConfig { bind: Some("127.0.0.1:45797"), backend: None, quic_lb_id: None, quic_lb_nonce: None, max_streams: None, version: [], tls: ServerTlsConfig { cert: [], key: [], generate: ["localhost"], root: [] } }, client: ClientConfig { bind: 127.0.0.1:0, backend: None, max_streams: None, version: [], tls: ClientTls { root: [], cert: None, key: None, disable_verify: None }, backoff: Backoff { initial: 1s, multiplier: 2, max: 30s, timeout: None }, websocket: ClientWebSocket { enabled: true, delay: Some(200ms) } }, log: Log { level: Level(Info) }, cluster: ClusterConfig { root: None, token: None, node: None, prefix: "internal/origins" }, auth: AuthConfig { key: None, key_dir: None, tls: AuthTls { root: [], cert: None, key: None, disable_verify: None }, public: Some(Simple([""])), public_subscribe: None, public_publish: None, public_api: None }, web: WebConfig { http: HttpConfig { listen: None }, https: HttpsConfig { listen: None, cert: None, key: None }, ws: true }, file: None, iroh: IrohEndpointConfig { enabled: None, secret: None, bind_v4: None, bind_v6: None, disable_relay: None } }
2026-05-07T20:14:15.386710Z INFO moq_relay: listening addr=127.0.0.1:45797
2026-05-07T20:14:15.386894Z INFO moq_relay::cluster: running as root, accepting leaf nodes
2026-05-07T20:14:15.392937Z WARN rustls::msgs::handshake: Illegal SNI extension: ignoring IP address presented as hostname (3132372e302e302e31)
2026-05-07T20:14:15.393061Z WARN moq_native::tls: no SNI certificate found server_name=None
2026-05-07T20:14:15.393760Z DEBUG moq_native::quinn: accepting host= ip=127.0.0.1:59692 alpn=h3
2026-05-07T20:14:15.396392Z DEBUG moq_native::quinn: accepted host= ip=127.0.0.1:59692 alpn=h3
2026-05-07T20:14:15.396703Z INFO conn{id=0}: moq_relay::connection: session accepted transport="quic" root=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99 publish= subscribe=
2026-05-07T20:14:15.397010Z INFO conn{id=0}: moq_relay::connection: negotiated version=moq-lite-03 transport="quic"
2026-05-07T20:14:15.397096Z TRACE conn{id=0}: moq_lite::lite::message: encoding self=AnnounceInterest { prefix: Path(""), exclude_hop: 0 }
2026-05-07T20:14:15.398518Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Active { suffix: Path("258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199"), hops: [] }
2026-05-07T20:14:15.398551Z DEBUG conn{id=0}: moq_lite::lite::subscriber: announce broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199
2026-05-07T20:14:15.399069Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Probe { bitrate: 32000, rtt: None }
2026-05-07T20:14:15.399094Z DEBUG conn{id=0}: moq_lite::lite::subscriber: probe stream closed
2026-05-07T20:14:17.454976Z WARN moq_native::tls: no SNI certificate found server_name=None
2026-05-07T20:14:17.456677Z DEBUG moq_native::quinn: accepting host= ip=127.0.0.1:55811 alpn=h3
2026-05-07T20:14:17.456714Z DEBUG moq_native::quinn: accepted host= ip=127.0.0.1:55811 alpn=h3
2026-05-07T20:14:17.457649Z INFO conn{id=1}: moq_relay::connection: session accepted transport="quic" root=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99 publish= subscribe=
2026-05-07T20:14:17.457853Z INFO conn{id=1}: moq_relay::connection: negotiated version=moq-lite-03 transport="quic"
2026-05-07T20:14:17.457990Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=AnnounceInterest { prefix: Path(""), exclude_hop: 0 }
2026-05-07T20:14:17.459439Z TRACE conn{id=1}: moq_lite::lite::message: decoded result=AnnounceInterest { prefix: Path(""), exclude_hop: 0 }
2026-05-07T20:14:17.459542Z DEBUG conn{id=1}: moq_lite::lite::publisher: announce broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199
2026-05-07T20:14:17.459555Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Active { suffix: Path("258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199"), hops: [] }
2026-05-07T20:14:17.460293Z TRACE conn{id=1}: moq_lite::lite::message: decoded result=Subscribe { id: 0, broadcast: Path("258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199"), track: "catalog.json", priority: 100, ordered: true, max_latency: 0ns, start_group: None, end_group: None }
2026-05-07T20:14:17.460320Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Probe { bitrate: 164484956, rtt: Some(0) }
2026-05-07T20:14:17.460334Z INFO conn{id=1}: moq_lite::lite::publisher: subscribed started id=0 broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199 track=catalog.json
2026-05-07T20:14:17.460567Z INFO conn{id=0}: moq_lite::lite::subscriber: subscribe started id=0 broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199 track=catalog.json
2026-05-07T20:14:17.460575Z TRACE conn{id=1}: moq_lite::lite::message: decoded result=Probe { bitrate: 27838166, rtt: None }
2026-05-07T20:14:17.460585Z TRACE conn{id=0}: moq_lite::lite::message: encoding self=Subscribe { id: 0, broadcast: Path("258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199"), track: "catalog.json", priority: 100, ordered: true, max_latency: 0ns, start_group: None, end_group: None }
2026-05-07T20:14:17.462141Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=SubscribeOk { priority: 100, ordered: true, max_latency: 0ns, start_group: None, end_group: None }
2026-05-07T20:14:17.462446Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 0, sequence: 0 }
2026-05-07T20:14:17.462474Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=0 track=catalog.json sequence=0
2026-05-07T20:14:17.462544Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 0, sequence: 0 }
2026-05-07T20:14:17.463302Z TRACE conn{id=1}: moq_lite::lite::message: decoded result=Subscribe { id: 1, broadcast: Path("258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199"), track: "audio/data", priority: 1, ordered: true, max_latency: 0ns, start_group: None, end_group: None }
2026-05-07T20:14:17.463319Z INFO conn{id=1}: moq_lite::lite::publisher: subscribed started id=1 broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199 track=audio/data
2026-05-07T20:14:17.463375Z INFO conn{id=0}: moq_lite::lite::subscriber: subscribe started id=1 broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199 track=audio/data
2026-05-07T20:14:17.463400Z TRACE conn{id=0}: moq_lite::lite::message: encoding self=Subscribe { id: 1, broadcast: Path("258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199"), track: "audio/data", priority: 1, ordered: true, max_latency: 0ns, start_group: None, end_group: None }
2026-05-07T20:14:17.463571Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=0
2026-05-07T20:14:17.465078Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=SubscribeOk { priority: 1, ordered: true, max_latency: 0ns, start_group: None, end_group: None }
2026-05-07T20:14:17.478181Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 0 }
2026-05-07T20:14:17.478238Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=0
2026-05-07T20:14:17.478263Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 0 }
2026-05-07T20:14:17.497911Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=0
2026-05-07T20:14:17.518127Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 1 }
2026-05-07T20:14:17.518184Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=1
2026-05-07T20:14:17.518203Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 1 }
2026-05-07T20:14:17.560752Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Probe { bitrate: 244064900, rtt: Some(0) }
2026-05-07T20:14:17.560911Z TRACE conn{id=1}: moq_lite::lite::message: decoded result=Probe { bitrate: 35904185, rtt: None }
2026-05-07T20:14:17.598245Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=1
2026-05-07T20:14:17.617735Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 2 }
2026-05-07T20:14:17.617847Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=2
2026-05-07T20:14:17.617878Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 2 }
2026-05-07T20:14:17.660400Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Probe { bitrate: 308030944, rtt: Some(0) }
2026-05-07T20:14:17.718285Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 3 }
2026-05-07T20:14:17.718383Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=3
2026-05-07T20:14:17.718425Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 3 }
2026-05-07T20:14:17.718707Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=2
2026-05-07T20:14:17.760552Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Probe { bitrate: 399896693, rtt: Some(0) }
2026-05-07T20:14:17.798301Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=3
2026-05-07T20:14:17.817769Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 4 }
2026-05-07T20:14:17.817846Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=4
2026-05-07T20:14:17.817868Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 4 }
2026-05-07T20:14:17.917877Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 5 }
2026-05-07T20:14:17.917958Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=5
2026-05-07T20:14:17.918009Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 5 }
2026-05-07T20:14:17.918274Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=4
2026-05-07T20:14:17.998219Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=5
2026-05-07T20:14:18.017660Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 6 }
2026-05-07T20:14:18.017714Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=6
2026-05-07T20:14:18.017742Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 6 }
2026-05-07T20:14:18.061214Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Probe { bitrate: 513594196, rtt: Some(0) }
2026-05-07T20:14:18.118316Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 7 }
2026-05-07T20:14:18.118403Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=7
2026-05-07T20:14:18.118433Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 7 }
2026-05-07T20:14:18.118681Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=6
2026-05-07T20:14:18.198330Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=7
2026-05-07T20:14:18.217629Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 8 }
2026-05-07T20:14:18.217666Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=8
2026-05-07T20:14:18.217683Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 8 }
2026-05-07T20:14:18.297588Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=8
2026-05-07T20:14:18.318082Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 9 }
2026-05-07T20:14:18.318145Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=9
2026-05-07T20:14:18.318183Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 9 }
2026-05-07T20:14:18.360514Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Probe { bitrate: 639752628, rtt: Some(0) }
2026-05-07T20:14:18.398076Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=9
2026-05-07T20:14:18.417498Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 10 }
2026-05-07T20:14:18.417550Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=10
2026-05-07T20:14:18.417575Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 10 }
2026-05-07T20:14:18.497791Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=10
2026-05-07T20:14:18.518092Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 11 }
2026-05-07T20:14:18.518178Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=11
2026-05-07T20:14:18.518213Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 11 }
2026-05-07T20:14:18.598453Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=11
2026-05-07T20:14:18.617800Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 12 }
2026-05-07T20:14:18.617870Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=12
2026-05-07T20:14:18.617891Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 12 }
2026-05-07T20:14:18.718280Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 13 }
2026-05-07T20:14:18.718318Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=13
2026-05-07T20:14:18.718349Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 13 }
2026-05-07T20:14:18.718664Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=12
2026-05-07T20:14:18.797517Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=13
2026-05-07T20:14:18.818160Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 14 }
2026-05-07T20:14:18.818216Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=14
2026-05-07T20:14:18.818235Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 14 }
2026-05-07T20:14:18.917563Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 15 }
2026-05-07T20:14:18.917627Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=15
2026-05-07T20:14:18.917649Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 15 }
2026-05-07T20:14:18.917976Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=14
2026-05-07T20:14:19.018411Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 16 }
2026-05-07T20:14:19.018519Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=16
2026-05-07T20:14:19.018701Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 16 }
2026-05-07T20:14:19.018888Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=15
2026-05-07T20:14:19.098367Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=16
2026-05-07T20:14:19.117746Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 17 }
2026-05-07T20:14:19.117816Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=17
2026-05-07T20:14:19.117858Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 17 }
2026-05-07T20:14:19.198007Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=17
2026-05-07T20:14:19.218259Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 18 }
2026-05-07T20:14:19.218301Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=18
2026-05-07T20:14:19.218325Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 18 }
2026-05-07T20:14:19.317918Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 19 }
2026-05-07T20:14:19.317967Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=19
2026-05-07T20:14:19.317989Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 19 }
2026-05-07T20:14:19.318220Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=18
2026-05-07T20:14:19.397999Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=19
2026-05-07T20:14:19.417524Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 20 }
2026-05-07T20:14:19.417599Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=20
2026-05-07T20:14:19.417694Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 20 }
2026-05-07T20:14:19.518390Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 21 }
2026-05-07T20:14:19.518468Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=21
2026-05-07T20:14:19.518510Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 21 }
2026-05-07T20:14:19.518761Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=20
2026-05-07T20:14:19.597858Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=21
2026-05-07T20:14:19.618180Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 22 }
2026-05-07T20:14:19.618251Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=22
2026-05-07T20:14:19.618289Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 22 }
2026-05-07T20:14:19.717988Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 23 }
2026-05-07T20:14:19.718068Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=23
2026-05-07T20:14:19.718123Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 23 }
2026-05-07T20:14:19.718458Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=22
2026-05-07T20:14:19.798257Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=23
2026-05-07T20:14:19.817650Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 24 }
2026-05-07T20:14:19.817726Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=24
2026-05-07T20:14:19.817774Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 24 }
2026-05-07T20:14:19.918070Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 25 }
2026-05-07T20:14:19.918168Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=25
2026-05-07T20:14:19.918198Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 25 }
2026-05-07T20:14:19.918530Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=24
2026-05-07T20:14:19.998531Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=25
2026-05-07T20:14:20.017894Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 26 }
2026-05-07T20:14:20.017933Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=26
2026-05-07T20:14:20.017953Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 26 }
2026-05-07T20:14:20.098237Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=26
2026-05-07T20:14:20.117609Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 27 }
2026-05-07T20:14:20.117660Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=27
2026-05-07T20:14:20.117681Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 27 }
2026-05-07T20:14:20.217796Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 28 }
2026-05-07T20:14:20.217838Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=28
2026-05-07T20:14:20.217862Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 28 }
2026-05-07T20:14:20.218242Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=27
2026-05-07T20:14:20.318144Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Group { subscribe: 1, sequence: 29 }
2026-05-07T20:14:20.318232Z DEBUG conn{id=1}: moq_lite::lite::publisher: serving group subscribe=1 track=audio/data sequence=29
2026-05-07T20:14:20.318271Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Group { subscribe: 1, sequence: 29 }
2026-05-07T20:14:20.318651Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=28
2026-05-07T20:14:20.398613Z DEBUG conn{id=1}: moq_lite::lite::publisher: finished group sequence=29
2026-05-07T20:14:20.401417Z TRACE conn{id=0}: moq_lite::lite::message: decoded result=Ended { suffix: Path("258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199"), hops: [] }
2026-05-07T20:14:20.401446Z DEBUG conn{id=0}: moq_lite::lite::subscriber: unannounced broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199
2026-05-07T20:14:20.401669Z INFO conn{id=1}: moq_lite::lite::publisher: subscribed cancelled id=0 broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199 track=catalog.json
2026-05-07T20:14:20.401736Z DEBUG conn{id=0}: moq_lite::lite::subscriber: broadcast closed err=cancelled
2026-05-07T20:14:20.401770Z DEBUG conn{id=1}: moq_lite::lite::publisher: unannounce broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199
2026-05-07T20:14:20.401775Z TRACE conn{id=1}: moq_lite::lite::message: encoding self=Ended { suffix: Path("258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199"), hops: [] }
2026-05-07T20:14:20.401800Z INFO conn{id=0}: moq_lite::lite::subscriber: subscribe cancelled id=0 broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199 track=catalog.json
2026-05-07T20:14:20.401825Z INFO conn{id=0}: moq_lite::lite::subscriber: subscribe cancelled id=1 broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199 track=audio/data
2026-05-07T20:14:20.401912Z INFO conn{id=1}: moq_lite::lite::publisher: subscribed cancelled id=1 broadcast=nests/30312:258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199:rt-bf3f3712-21a4-4de3-b78f-5c041c591d99/258d2776243d6c47889bfeedc69596d2093ab9110a7fbadbdb5bfb87f44a8199 track=audio/data
2026-05-07T20:14:20.402444Z WARN web_transport_quinn::session: failed to read capsule e=UnexpectedEnd
2026-05-07T20:14:20.402484Z INFO conn{id=0}: moq_lite::lite::session: session terminated
2026-05-07T20:14:20.402664Z WARN moq_relay: connection closed err=transport: connection error: closed