The previous design wrapped a mutable HashSet in an AtomicReference,
which makes the *reference* thread-safe but not the set's internals.
The historical-replay closure ran `seenIds.load()?.add(event.id)`
from the subscriber's coroutine while `deliver` ran
`seen.contains(event.id)` from the publisher's coroutine —
concurrent add/contains can corrupt buckets or throw CME.
The pre-index implementation didn't have this race because both
operations ran on the same Flow collector coroutine. Index-driven
dispatch put them on different coroutines.
Switch to AtomicReference<Set<String>?> over an immutable Set with a
CAS-loop on add. Reads in `deliver` always see a fully-constructed
snapshot. Single-writer in steady state so the CAS typically
succeeds first try; the loop only matters across the
`seenIds.store(null)` post-EOSE handoff.
Also:
- hoist `NostrSignerSync(keys[target])` out of the per-iteration
loop in LoadBenchmark.fanoutScaling.
- add FilterIndex tests for `tagsAll` selection, multi-char tag
fall-through, and re-register key-union semantics.
Verifies the FilterIndex path in LiveEventStore: each event matches
exactly ONE of N subscribers, so without an index the relay walks
all N subs per event (O(N)); with the author-keyed index the lookup
is O(1) and per-event latency stays flat as N grows.
Different from fanoutLatency (broadcast: 1 event reaches every
sub). Here per-event work is lookup-bound rather than
delivery-bound. Configurable via -DfanoutScalingSubs (comma list)
and -DfanoutScalingEvents.
Measured (laptop, JDK 21, full sweep):
100 subs, 2k events: p50=1.35ms p99=5.71ms
1000 subs, 2k events: p50=1.05ms p99=3.05ms
5000 subs, 1k events: p50=1.01ms p99=2.96ms
p50 ~1ms across N — the predicted O(1) scaling. Default events count
adapts downward at high N to stay below WebSocketSessionPump's
8192-frame outbound cap (test-client read side is the bottleneck
above ~5k subs, not the relay).
Plan doc updated with the measured numbers in the "How to verify"
section.
Replaces the ConcurrentHashMap<Observable, Observable> registry with
a FilterIndex<Observable>. observeNotes / observeEvents /
observeNewEvents(Filter) call register(filter, observer); the
predicate-only observeNewEvents(predicate) overload uses
registerUnindexed because the index can't introspect an opaque
predicate.
refreshNewNoteObservers now iterates index.candidatesFor(event)
instead of every observer. refreshDeletedNoteObservers still uses
forEach — the index doesn't help on deletes (every observer might
hold the deleted note in its result set, no event-shape to consult).
Same FilterIndex<S> the relay's LiveEventStore uses; one structure,
two call sites, identical fanout shape.
Replace the SharedFlow-based broadcast in LiveEventStore with an
inverted index over filter-bearing subscribers. Each REQ registers
its filters into a per-store FilterIndex<LiveSubscription>; insert()
calls index.candidatesFor(event) and only delivers to candidates
whose Filter.match still passes. Cuts the per-event walk from
O(N_subs * N_filters) to a few hash lookups plus match() over a
small candidate set.
FilterIndex itself lives next to Filter.kt (commonMain, KMP-friendly,
AtomicReference + COW) so other call sites with the same shape
(LocalCache.observables, ObservableEventStore.changes) can reuse it.
Each filter contributes entries on its single most-selective dimension
(ids > authors > tags > tagsAll > kinds > unindexed) to keep buckets
narrow and avoid Set-dedupe work in candidatesFor.
The historical-replay race the previous SharedFlow + onSubscription
handoff closed is preserved by registering BEFORE replay starts and
deduping seen ids until EOSE.
The runner's keyupdate testcase has TESTCASE_CLIENT=keyupdate (server
runs plain transfer). The runner verifies the pcap shows BOTH sides
emit packets in phase 1 — pre-fix our receive-only key-update path
satisfied a server-initiated rotation but not this test, because
aioquic's transfer-server doesn't rotate spontaneously. Result: 0
phase-1 packets either direction, "Expected to see packets sent with
key phase 1 from both client and server".
QuicConnection.initiateKeyUpdate() (now public) is the send-side
analogue of commitKeyUpdate: derives next-phase secrets for both
directions via HKDF-Expand-Label "quic ku", installs as live
(reusing old HP keys per RFC §6.1), flips currentSendKeyPhase +
currentReceiveKeyPhase together. The receive side has to roll too
because the peer responds in the new phase — leaving currentReceive
at 0 would force feedShortHeaderPacket to take the
deriveNextPhase-then-commit path on the response and orphan the
keys we just installed in previousReceiveProtection.
InteropClient adds an `initiateKeyUpdate` flag to runTransferTest;
the keyupdate dispatch sets it true. After awaitHandshake (TLS done,
1-RTT keys derived) the flag-flow polls briefly for status=CONNECTED
(HANDSHAKE_DONE arrived → handshake confirmed per RFC 9001 §6.5
prerequisite) before calling initiateKeyUpdate, then sends the GET.
The GET goes out in phase 1, the server mirrors phase 1 in its
response, runner is satisfied.
Also added ecn, amplificationlimit, blackhole to the runTransferTest
dispatch (all reuse the plain-transfer flow; the runner verifies
behaviour via pcap independent of any client-side dance). aioquic
phase 3 result: ✓(retry, keyupdate, blackhole),
?(resumption, zerortt, ecn — feature gaps requiring session tickets,
0-RTT, and IP-layer ECT codepoints respectively),
amplificationlimit blocked by a runner-side cert-gen bug on macOS
(tr LC_CTYPE=C doesn't suppress UTF-8 errors, the chainlen=9 cert
inflation step fails).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Marks Sketches A and B done, with a note that A took the simpler
Channel.UNLIMITED + AtomicInteger cap path the original Risks section
called out, sidestepping the channel-swap that the plan first
sketched.
Records that the streaming-filter slice of Sketch C landed in Quartz,
and that the larger envelope-streaming work the plan called out is
unnecessary because MessageDeserializer / CommandDeserializer /
EventDeserializer were already streaming — only the filter sub-object
went through readTree.
Adds the verification benchmarks (connectionsHeldOpen10k,
connectionsHeldOpenWithFanout) to the verification section, with a
correction that what's measured is JVM heap not OS RSS.
Carries forward the not-done items (fan-out de-duplication,
filter-matching index, Netty engine) into the open-work section,
pointing at live-broadcast-fanout-index.md for the highest-leverage
remaining work.
After the handshake timeout bump and the faster PTO landed, the last
remaining flake was picoquic's handshakecorruption iter ~35: the
handshake recovers from 2-3 PTO rounds and smoothed_rtt is left at
~1s (RFC 9002 §5.2 takes the sample from the largest-acked packet's
SEND time, and that's the PTO retransmit, not the original).
post-handshake PTO is then 3s+, doubling. Three doublings under 30%
bit-flip eat 24s before the GET retransmit lands — 30s is a cliff.
60s gives the slow-recovery iterations real headroom. Total budget:
50 iters × ~3s typical = 150s, plus a few 60s outliers, comfortably
within the runner's 300s testcase budget.
Verified clean: aioquic, picoquic, quic-go each pass all 7 tests
(handshake, multiplexing, longrtt, transferloss, transfercorruption,
handshakeloss, handshakecorruption). 21/21.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-merge audit findings on the connection-scaling perf changes:
- connectionsHeldOpenWithFanout populated firstSeenNs but never read
it — only lastReceiveNs feeds the p50/p99 latency metric. Drop the
unused map and rename the latency map to lastReceiveNs to match
what it actually holds.
- connectionsHeldOpen10k printed/asserted on a variable named rssMb
that was actually JVM heap (`Runtime.totalMemory - freeMemory`).
Rename to heapMb, force a GC + 200 ms settle before reading so the
number reflects retained bytes rather than connect-ramp churn, and
update the assertion message to say "JVM heap" not "heap usage".
The 10s HANDSHAKE_TIMEOUT and 60s TRANSFER_TIMEOUT were tuned for
single-connection tests against well-behaved peers. multiconnect under
30% packet drop / bit-flip routinely needs three to four PTO rounds
just for the handshake — the 10s default hit "handshake_failed" mid-
recovery on the unlucky iter. Bump per-iter handshake to 30s and
transfer to 30s; 50 iters × ~5s typical = ~250s within the runner's
300s testcase budget, with headroom for the slow-recovery iters.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
multiconnect handshakeloss / handshakecorruption tail-fail under the
runner's 30% packet drop / bit-flip scenarios because each PTO retransmit
chance gives ~49% one-side success (0.7² for both directions clear). With
INITIAL_RTT=333ms the first PTO fires at 999 ms and doubling tops out
at ~5 attempts in 30s — across 50 sequential connections, ~5% probability
some iteration runs out of retransmits before the per-iter budget.
Two coupled changes:
1. INITIAL_RTT_MS 333→100. RFC 9002 §6.2.2 spec-allowed (the standard
default but explicitly configurable). Matches Chrome and
Firefox/neqo. Pre-sample PTO is now 300 ms instead of 999 ms;
doubling fits ~8 retransmit attempts in 30s instead of 5,
pushing per-iter loss-recovery success past 99% under 30% drop.
Spurious retransmits on slow paths are harmless (peer dedupes
by packet number) and smoothed_rtt converges in one round-trip.
2. QuicConnectionDriver always uses lossDetection.ptoBaseMs() for the
PTO timer, including before the first RTT sample. Pre-fix the
driver hardcoded 1000ms as a "handshake-timeout safety floor"
that ignored INITIAL_RTT_MS entirely — the PTO was always 1s
pre-handshake regardless of the constant. Now both pre- and
post-sample regimes go through the same calculation.
max_ack_delay is gated to APPLICATION space (RFC 9002 §6.2.1) so
pre-handshake PTOs aren't padded with the peer's quoted delay.
Two pre-existing tests (PtoTest, QuicLossDetectionTest) hard-coded
expected PTO durations derived from the old 333 ms constant; updated
them to express the relationships in terms of INITIAL_RTT_MS so future
tweaks don't desync.
Result: 21/21 against aioquic, picoquic, quic-go (handshake,
multiplexing, longrtt, transferloss, transfercorruption,
handshakeloss, handshakecorruption all pass on each peer).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The relay-inbound path (REQ / COUNT / NEG-OPEN) was the only place
ManualFilterDeserializer still went through `jp.codec.readTree(jp)`,
materializing a full ObjectNode per filter before walking it. The
Command/Message/Event deserializers next door already stream straight
off the JsonParser; this brings filter parsing onto the same shape.
- Add ManualFilterDeserializer.fromJson(JsonParser): mirrors
EventDeserializer's hand-rolled token loop. Dispatches on field
name, reads the seven fixed keys + dynamic #x / &x tag arrays via
nextToken / nextTextValue / longValue / intValue, and drops invalid
entries silently (same tolerance as the tree-based mapNotNull path).
- Wire the four internal call sites — three in CommandDeserializer
(REQ, COUNT, NEG-OPEN) and the standalone FilterDeserializer — to
the streaming overload.
- Keep the existing fromJson(ObjectNode) overload as-is for any
external/cross-format adapter that already has a tree in hand.
For a single REQ with N filters this drops N ObjectNode trees per
inbound frame, which at 10k connections × ~5 filters × 1 msg/s is
~50k tree allocations/sec we don't have to do. No behavioral change:
all quartz JVM tests pass (including the cross-mapper round-trip
suite that compares Jackson and KotlinSerialization output) and
geode's relay tests pass.
Implements geode/plans/2026-05-07-connection-scaling.md to push the
single-relay connection ceiling past the ~2k floor measured by
LoadBenchmark.connectionsHeldOpen.
- WebSocketSessionPump: switch outQueue to Channel.UNLIMITED bounded
by an AtomicInteger backlog cap. Idle connections no longer reserve
an 8 192-slot fixed buffer; lazy-allocated head segments cost only
a few hundred bytes per idle session. Slow-client policy preserved:
once outstanding > MAX_OUTGOING_BUFFER, the queue is closed and
the connection drops, so NIP-01 ordering is never silently
violated.
- LocalRelayServer / RelayConfig.NetworkSection: expose CIO
connectionGroupSize / workerGroupSize / callGroupSize so big-VM
operators can tune Ktor's event-loop pools without forking. Switch
to the serverConfig {} + configure {} embeddedServer overload so
CIO tunables can be set.
- LoadBenchmark: add connectionsHeldOpen10k (asserts 10 000 idle
WebSockets settle under a 1 GiB heap ceiling) and
connectionsHeldOpenWithFanout (5 000 subscribers, 10 EPS fan-out
for 10 s, reports p50/p99 last-fanout latency).
- config.example.toml: document the three CIO tunables and the
ulimit -n requirement for operators targeting >1k connections.
Pre-fix QlogWriter only flushed in close(); the 60s runner timeout
SIGKILLs the JVM before runTransferTest reaches its qlogWriter?.close().
On every failed quic-go transferloss, the trace ended at exactly 32768
bytes — 4 × 8KB BufferedWriter blocks — masking ~50 seconds of
late-connection behavior. Made every interop debugging session start
with "is this a connection wedge or a qlog wedge?".
Per-event flush was the original shape and was removed in 99a1a91de
because it caused multi-ms stalls on macOS Docker virtualized
filesystems (broke handshakes mid-flight). 250 ms is the compromise:
cheap enough to not stall the send path, fine-grained enough to
capture per-PTO behavior under heavy loss.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two interop-runner gaps closed in one InteropClient pass plus a
QuicConnection snapshot helper:
1. multiconnect testcase. The runner's handshakeloss /
handshakecorruption tests reuse TESTCASE_CLIENT=multiconnect — 50
sequential connections, each fetching a 1KB file under 30% packet
drop or bit-flip, with the runner verifying _count_handshakes()==50
in the pcap. Pre-fix our InteropClient dispatch returned 127 (skip)
for "multiconnect", so both tests showed as ?(L1, C1). Added
runMulticonnectTest: loops fresh socket + conn + driver + GET +
close per URL. Per-iteration qlog files at $QLOGDIR/client-N.sqlog
so a stuck iteration leaves a focused trace.
2. multiplex pacing against quic-go. Pre-fix the parallel path
chunked the URL list into fixed groups of MULTIPLEX_PARALLELISM=64.
Worked against aioquic + picoquic (initial_max_streams_bidi=128)
but blew up against quic-go (advertises 100, ramps slowly via
MAX_STREAMS_BIDI bumps): second chunk pushed cumulative used past
limit, threw QuicStreamLimitException. Now each iteration takes
min(MULTIPLEX_PARALLELISM, peerMaxStreamsBidi - used). When budget
hits 0, brief 50ms idle waits for the peer's bump.
New QuicConnection.localBidiStreamsUsedSnapshot() exposes the
consumed-side counter; combined with the existing
peerMaxStreamsBidiSnapshot() the InteropClient computes the live
available budget without holding streamsLock.
Result against quic-go: H, M, LR, L2, C2, C1 pass; was 0/7 at
session start (handshake itself failed pre-ALPN-fix), 4/6 after
key-update fix, now 6/7. Only L1 (handshakeloss) remains as
multiconnect-under-30%-drop flake (same flake picoquic shows).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
quic-go initiates a 1-RTT key update partway through every transferloss
or transfercorruption test (KEY_PHASE bit flips 0→1 around server pn=100
by default). Pre-fix our parser used the OLD application keys for every
post-update packet, AEAD-failed all of them, never sent another ACK,
the server fell into PTO mode, and throughput collapsed (~24kbps over
60s vs the 10Mbps the path supports).
The fix is end-to-end:
- ShortHeaderPacket.peekKeyPhase: HP-unmasks just the first byte to
surface the key-phase bit BEFORE running AEAD. The parser uses this
to pick the right keys instead of paying for a doomed AEAD.
- QuicConnection: tracks the live application secrets (server- and
client-side) and current send/receive key phase, plus a
previousReceiveProtection slot for RFC §6.1 reorder-window decryption.
deriveNextPhaseReceiveKeys derives the next phase via
HKDF-Expand-Label("quic ku", "", Hash.length) without committing;
commitKeyUpdate installs them only after AEAD has succeeded, then
rolls the send side forward in lockstep so our next outbound
carries the matching KEY_PHASE bit (peer needs that to confirm the
rotation completed). HP key is NOT rotated, per spec.
- QuicConnectionParser.feedShortHeaderPacket: three-way dispatch on
the peeked bit — matches current → live keys; matches retained
previous → previous keys (reordered packet); else → derive
next-phase, attempt AEAD, commit on success.
- QuicConnectionWriter: ShortHeaderPlaintextPacket(... keyPhase =
conn.currentSendKeyPhase) at both 1-RTT build sites (steady-state
and CONNECTION_CLOSE).
We don't drive key updates ourselves — only echo the peer's. Avoids
the bookkeeping for RFC 9001 §6.6 packet-count limits and the safety
benefits of voluntary rotation aren't load-bearing at our connection
scale.
Tests: peekKeyPhase round-trip + long-header rejection;
2-byte-pn round-trip when largestReceived is far behind (the original
suspected-but-not-actual cause before the key-phase reveal).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
RFC 9002 §6.2.4 says a PTO probe SHOULD retransmit unacked data, not
emit a bare PING. Two gaps in our handler surfaced via interop:
1. Handshake CRYPTO past the 1-RTT-keys-up boundary. The pre-fix
handler gated the requeue on `application.sendProtection == null`
so once 1-RTT keys were derived, our Finished (still inflight at
Handshake level until the peer ACKs it) was never retransmitted.
Lost Finished → server never confirms handshake → never sends
HANDSHAKE_DONE → connection wedges with ACK-only handshake packets
bouncing forever. Surfaced by handshakeloss against aioquic at 30%
drop rate (multiconnect iter 12 stuck at t=52s, zero handshake_done).
2. STREAM data when the peer never ACKs anything. Our loss detection
gates on `pn < largestAckedPn`, which never advances when every one
of our 1-RTT packets is dropped or corrupted en route. Surfaced by
handshakecorruption: we send H3 init streams + GET in 1-RTT pn=0,
gets corrupted, server never decrypts, never ACKs. Pre-fix the
STREAM bytes were never retransmitted; the GET stalled.
Fix: handlePtoFired now requeues inflight CRYPTO at every active
pre-application level (Initial AND Handshake) regardless of 1-RTT
state, and walks streamsList to re-queue inflight STREAM bytes when
1-RTT keys are up. requeueAllInflight is a no-op when nothing is
inflight, so calling on already-ACKed / already-discarded levels is
harmless.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- @Volatile ScheduledPostNotifier.channel: two workers firing in the
same window can race on ensureChannel from different IO threads.
createNotificationChannel is idempotent, but the field write needs a
visibility fence so both threads see the cached reference.
- @Volatile PoolEventOutbox.eventOutbox map ref + PoolEventOutboxState
.relaysRemaining set ref. The new pendingPublishRelaysFor polling
path reads these from the WorkManager IO thread; mutations still
happen on NostrClient's IO scope. Pre-existing visibility gap that
this poll surface exposed; @Volatile is the minimal fix.
- ScheduledPostsScreen.SectionLabel: read the locale from
LocalConfiguration.current.locales[0] (the Compose-resolved locale)
rather than Locale.getDefault() (the system locale, which can drift
from app config and breaks Turkish I/i casing).
- Quartz: expose pendingPublishRelaysFor(eventId) on INostrClient
- Worker: after publish(), poll pendingPublishRelaysFor every 500ms up
to OK_TIMEOUT_SEC=30s
- notify user when a scheduled post fires or fails
- Switch list-screen formatters from java.text.DateFormat to
java.time.format.DateTimeFormatter. DateFormat / SimpleDateFormat
are documented not-thread-safe; the file-scope singletons are at
risk under any future multi-threaded recomposition path.
DateTimeFormatter is immutable and thread-safe.
- LogoutButton: when decodePublicKeyAsHexOrNull fails, bail before
the Toast fires so we don't claim "Logged out" while logOff's
coroutine has aborted its cleanup. Theoretical case (npub from
prefs is well-formed in practice) but the previous flow was
misleading on the failure path.
- ScheduledPostStore.purgeStale: document the
terminatedAtSec ?: lastAttemptAtSec ?: createdAtSec fallback so
the legacy-row migration semantics are clear from the source.
- DrawerContent: drop fully-qualified inline references, add proper
imports for Amethyst, ScheduledPostStatus, and Material3 Badge.
- ScheduledPostsScreen: drop the redundant `posts` collection — derive
the empty-state branch from `groups` directly. Use `group.day` as the
sticky-header key instead of an interpolated string. Cache the SHORT
time and FULL date `DateFormat` instances at file scope (matches the
pattern in TimeAgoFormatter). Pass `today` into `DayHeader` so we
don't call `LocalDate.now()` per recomposition.
- ScheduledPostsViewModel: drop the intermediate public `posts`
StateFlow now that no consumer needs it; fold the filter+sort into
the `groupedPosts` chain.
- ScheduledPostStoreTest: collapse the two `newStore()` overloads into
one with a defaulted `now: () -> Long` parameter.
- Delete scheduled posts on logout
- presets, list grouping, drawer badge, always-on prompt, logout toast
- bech hardening, retention, live countdown
- Replace `bechToBytes()` chains at three account sites with the null-safe
`decodePrivateKeyAsHexOrNull` / `decodePublicKeyAsHexOrNull`. A malformed
npub no longer crashes the LogoutButton composable tree or leaves the
logoff path half-cleaned (deleted account row but cache + scheduled
posts still in memory).
- catch CancellationException explicitly
- document the deliberate tradeoff of holding the data mutex across the disk write in persist()
- migrate hardcoded strings to R.string.*
- Store.mutate: use value equality (==) instead of reference (===)
- Store.persist: drop the redundant parent.exists() check
- ViewModel: drop the Context parameter from publishNow(id)
- Screen: memoize extractPreview via remember(post.id) so JSON scanning doesn't run on every recomposition of a row.
- Screen: replace hardcoded Color(0xFF...) literals
- Screen.formatPublishMoment: delegate the past-tense branch to the existing timeAgoNoDot() helper instead of hand-rolled TimeUnit math.
feat(scheduled-posts): add screen + drawer entry to view, push, or delete
feat(scheduled-posts): use scheduled time as created_at + add diagnostic logs
feat(scheduled-posts): add picker UI and toolbar toggle to post composer
feat(scheduled-posts): wire schedule branch into ShortNotePostViewModel
feat(scheduled-posts): add storage + worker for delayed post publishing
QuicConnection.alpnList → TlsClient.offeredAlpns was captured but never
threaded into buildQuicClientHello. The builder accepted only an
"additionalAlpn" parameter and hardcoded "h3" first, so our wire
ClientHello always carried just [h3] regardless of caller intent.
quic-go enforces strictly with TLS alert 120 (no_application_protocol,
CRYPTO_ERROR 376) when none of the offered ALPNs match its server
config — handshake failed at the very first server response against
quic-go's hq-interop testcases.
Replaced the awkward additionalAlpn shape with `alpns: List<ByteArray>`
(default [h3] for backward-compat) and threaded TlsClient.offeredAlpns
through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
longrtt failed against aioquic with "Expected at least 2 ClientHellos.
Got: 1" because our client started sending Initials before the sim's
ns3 + tcpdump capture finished initializing. Only the PTO retransmit
hit the wire — the original ClientHello was sent during the sim's
~1s readiness window and never captured. aioquic, picoquic, and
quic-go all gate their client launch on /wait-for-it.sh sim:57832.
We didn't.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
club.minnced:opus-java doesn't ship natives for darwin-aarch64, so the
sine-440 round-trip test failed on Apple Silicon dev machines and blocked
the pre-push hook. Catch the IllegalStateException from encoder/decoder
construction and use JUnit Assume to skip; Linux x86_64 CI keeps coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the runner's compliance-check phase produces traces but the
actual testcase phase doesn't (matrix runs against multiple
testcases sometimes lose later containers' stderr), the previous
inspect script greppped the runner's stdout and silently produced
nothing.
Now: check per-testcase client/output.txt and output.txt first; fall
back to the tee'd .stdout.log narrowed to this testcase's window.
On total miss, print actionable hints (rebuild with DEBUG, run in
isolation).
The 'run-<timestamp>.stdout.log' siblings also match the 'run-*'
glob — zsh in particular returns them mixed with the actual run
dirs. The for-loop now filters to directories only.
(The summarize-matrix script was already OK — it does ls -1d
followed by a head -1, and run dirs come first in mtime order.)
Previously the script only showed server stderr — but the bug
investigation needs the CLIENT's runtime traces, which are in the
runner's tee'd stdout (${RUN_DIR}.stdout.log).
awk-narrows the lines to the segment between this testcase's
'Running test case: X' marker and the next one (each testcase runs
a fresh container, so the trace lines between markers are exactly
this testcase's run). Then dumps:
- first 50 [boot] / [interop] / [batch] / [writer.app] lines
- stream_frames=N histogram for the testcase
Useful when debugging a specific testcase failure that requires
seeing the writer's per-drain decisions.
The longrtt testcase (1 file, serial path) was failing because
client.get(authority, path) → prepareRequest() opens a stream and
queues the GET, but never calls driver.wakeup(). The data sits in
the queue until the PTO timer fires (~1 s later) — a fatal delay
on a 1.5 s RTT link with an 8 s docker-compose timeout.
The parallel path explicitly wakes after prepareRequests returns,
but the serial path was missing the equivalent nudge.
Smoking gun from the inspect-testcase output:
- 1 KB file, handshake at t=4502, response packets at t=4684/4685
- NO outgoing packet between t=4499 (ack) and t=4687 (ack) that
contained the GET request — it never went out via prepareRequest
Fix: prepareRequest in both Http3GetClient and HqInteropGetClient
now calls driver.wakeup() after enqueuing the request. The
@Suppress("UNUSED_PARAMETER") on HqInteropGetClient.driver was
also stale — it's used now.
The longrtt testcase failed at 8s with only 1.2 KB received (out of
3 MB requested). Trace from inspect-testcase:
handshake completed at t=4502 (3 RTTs at 750ms one-way as expected)
first stream byte arrived at t=4694, ~200ms after
test killed at t=8s with code=0x0 (graceful close from us)
After handshake, ~3.5s of transfer time was available before the
docker-compose timeout. With our default
initialMaxStreamDataBidiLocal = 1 MB, the peer sends 1 MB then stalls
until our parser fires a MAX_STREAM_DATA bump. At 1.5s RTT each stall
is one round-trip lost (~1.5s). For a 3 MB file that's 2 stalls = 3s
of pure flow-control idle on top of CC slow-start.
Setting initialMaxData and initialMaxStreamData{BidiLocal,
BidiRemote,Uni} to 32 MB removes flow control as a bottleneck for
the largest interop transfers (longrtt 5 MB, transfercorruption few
MB) and lets the peer's congestion control alone drive the rate.
Doesn't help handshake duration (which is RTT-bound and correct).
Doesn't help slow-start ramp (CC, server-side). Should still close
the gap on longrtt.
Usage:
./quic/interop/inspect-testcase.sh longrtt
Auto-finds the most recent run dir with the named testcase, then
emits a focused one-screen report:
- runner status line (from the tee'd .stdout.log)
- file sizes generated for that testcase
- qlog event-type histogram
- transport_parameters (peer's flow-control budget)
- connection_closed events (spec violations / explicit failures)
- last 10 sent/received packets (steady-state shape)
- first/last packet timestamps + total received count
(transfer rate hint)
- server stderr tail
8 plan files updated to reflect what's actually shipped vs. what
each doc said before:
- 2026-05-06-cross-stack-interop-test.md: 'Spec — ready to
implement' → 'Implemented and merged' with pointers to results,
gap matrix, and closure roadmap.
- 2026-05-06-cross-stack-interop-test-results.md: scenario
inventory updated to show all branches merged into
claude/cross-stack-interop-test-XAbYB; suite-flake caveats
flagged with cross-refs to the routing investigation; file
inventory now lists BrowserInteropTest + PlaywrightDriver +
the moved nestsClient/tests/browser-interop/ tree (no more 'in
sister branches not yet merged' section).
- 2026-05-06-cross-stack-interop-test-gap-matrix.md: T8/T10/T11/
T12/T13 all show hang ✅ + browser ✅; I13/I14 'NOT YET LANDED'
callouts removed; the 'in sister branch' qualifier on T13 / I7
/ I6 dropped; coverage-holes section replaced with caveat
pointers to the routing investigation.
- 2026-05-06-i4-stereo-cross-stack-scenario.md: 'Spec — ready for
implementation' → 'Landed' with PR #2755 + the three test
scenarios that ship the assertion.
- 2026-05-06-phase4-browser-harness.md: 'Spec — ready for
implementation' → 'Landed', with notes on how the implementation
diverged (used Container.Legacy.Consumer not @moq/watch; cert
pinning via serverCertificateHashes; I2/I3/I4/I13/I14 originally
deferred but later landed).
- 2026-05-06-phase4-browser-harness-results.md: status updated to
reflect 4.D CI removed per maintainer ask; layout note for
the nestsClient/tests/browser-interop/ relocation; full
scenario list reconciled with results doc.
- 2026-05-07-late-join-catalog-flake-investigation.md: status
updated to mention 00f6cba31 + 207057374 were reverted, and
link to 2026-05-07-moq-relay-routing-investigation.md as the
action plan. Adds note that the flake also affects four
browser-tier scenarios after Browser I7 landed.
- 2026-05-01-quic-stream-cliff-investigation.md: adds an HCgOY-
override banner — recommendation 2 (framesPerGroup = 5) was
superseded 4 days later by HCgOY field tests landing the prod
default at 50. Cross-refs the reconciliation + production-
rerun plans. Open follow-up #3 ('reset to 1') updated with
the same context.
No code or test changes.
Mirrors the existing nestsClient/tests/hang-interop/ layout — all
T16 cross-stack interop test infrastructure (Rust sidecars + bun
browser harness) now lives under nestsClient/tests/.
Updates:
- nestsClient/build.gradle.kts: browserInteropDir path.
- PlaywrightDriver.kt + BrowserInteropTest.kt: kdoc comments.
- All plan docs in nestsClient/plans/ that referenced the old
path.
Compiles clean post-move. No CI changes (those jobs are
intentionally not wired per 2026-05-07-cross-stack-interop-ci-gating.md;
when re-added they'll reference the new path).
Two bash-3.2 issues:
- Multiline process substitution '< <(...)' with embedded
comments triggered 'bad substitution: no closing ')''.
Reworked as imperative pushes to an array.
- 'set -u' rejects '${SEARCH_FILES[@]}' on an empty array.
Disabled '-u' since the artifact-fallback path doesn't need
any search files.
Plans the path from 'infra shipped' to 'full coverage with
correct behaviours'. Five new plan docs in nestsClient/plans/:
1. 2026-05-07-t16-closure-roadmap.md — index + priority order.
Three sequential steps + one independent track.
2. 2026-05-07-moq-relay-routing-investigation.md (Priority 1)
— root-cause the moq-relay 0.10.x per-broadcast subscribe-
routing race. Step-by-step: capture relay-side trace, write
minimum reproducer, file upstream OR bump moq-relay version.
Smoking gun + hypotheses already in place from the
late-join-flake investigation; this plan turns that into
actionable next steps.
3. 2026-05-07-tighten-cross-stack-assertions.md (Priority 2)
— once the routing race is closed, replace the five soft-
pass scenarios in BrowserInteropTest with hard floors.
Lists each scenario, its current soft-pass behavior, and
the proposed hard threshold.
4. 2026-05-07-cross-stack-interop-ci-gating.md (Priority 3)
— re-add the hang-interop + browser-interop GitHub Actions
jobs that were dropped in 6829ab727 / b94737de7. Includes
the exact YAML to restore and a 10/10 sweep stability bar
before merge.
5. 2026-05-07-framespergroup-production-rerun.md (independent
track) — re-run the HCgOY two-phone field tests against
current nostrnests production at multiple framesPerGroup
values to settle whether the test pin (5) and production
default (50) can converge.
Estimated total: 2.5-3.5 days of focused work to fully close
T16. After these four: I7 post-reconnect cliff and I12 GOAWAY
remain open as genuinely upstream-territory items, tracked in
their existing investigation docs.
Two paired changes:
(1) run-matrix.sh now tees the runner's full stdout (pre-grep-
filter) to <RUN_LOG_DIR>.stdout.log — sibling rather than
inside the log dir because run.py refuses to start if its own
--log-dir already exists.
(2) summarize-matrix.sh checks that file FIRST when searching for
'Test: X took Y, status:' lines — it has the authoritative
runner output that wasn't being saved before.
Old runs (without the tee) fall back to qlog inspection.
Claude