Ktor's WebSocket layer is unbounded by default, so [limits].max_ws_frame_bytes
was a strfry-parity tuning knob nobody actually used — drop the whole
LimitsSection from StaticConfig and the matching constructor param from
KtorRelay. The negentropy-large-corpus plan note is updated accordingly.
maxAdminBodyBytes is a real defense-in-depth cap (NIP-98 forces us to
read the body for the payload sha256 before we can authenticate, so an
unbounded read is a pre-auth DoS vector). Keep the cap, but stop
exposing it as an operator knob — it was never plumbed through to
StaticConfig and 1 MiB is ~1000× any plausible NIP-86 RPC payload.
Hardcoded at the Nip86HttpRoute construction site with the rationale
inline. Nip86HttpRoute.maxBodyBytes stays so tests can drive the 413
boundary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename RelayConfig → StaticConfig and the persistence/RelayStateStore →
config/RuntimeConfig so the two configuration tiers — boot-time TOML
versus operator-mutable runtime state — are obvious from the class
names. The `persistence/` package is gone; everything related to config
lives in `config/`.
For overlapping fields (NIP-11 info doc, pubkey / kind allow-deny
lists), StaticConfig now seeds RuntimeConfig on first boot via the new
RuntimeConfig(seed = …) constructor and an AuthorizationSeed type. The
runtime file wins from then on: later edits to [authorization] in the
TOML are ignored once an admin RPC has written the snapshot. As a
consequence, KindAllowDenyPolicy and PubkeyAllowDenyPolicy are no
longer stacked in geode's policy chain — BanListPolicy already reads
the same lists from the BanStore, and double-stacking would silently
diverge after the first admin mutation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames in the geode module to make naming honest:
- Relay -> RelayEngine (the transport-agnostic core)
- LocalRelayServer -> KtorRelay (Ktor/CIO transport; defaults to 127.0.0.1
but supports public deployment, so "Local" was misleading)
- RelayHub -> InProcessRelays (URL-keyed registry that doubles as a
WebsocketBuilder for in-JVM tests; "Hub"/"Pool" implied substitutable
resources, but each URL maps to a distinct relay)
Also moves the test-only helpers `preload` and `publish` out of
RelayEngine and into a testFixtures extension file. As part of that,
`publish` now waits for the relay's OK reply before returning so that
publish-then-subscribe is deterministic — previously the fire-and-forget
submit let a subscription register after publish returned but before the
ingest queue fanned out, causing ephemeral kinds (not persisted) to leak
to late subscribers. Fixes the flaky
`Nip01ComplianceTest.ephemeralEventIsNotStoredAndDoesNotShowOnFollowupReq`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Conflicts:
- quartz/.../LiveEventStore.kt: main added IngestQueue (group-commit)
and a fire-and-forget submit() path; this branch replaced the
SharedFlow live fanout with FilterIndex-driven dispatch. Resolved
by keeping main's submit/insert pipeline shape (IngestQueue ingest
ctor param, submit() callback, insert() wrapping submit via
CompletableDeferred) but routing the on-Accepted fanout through
FilterIndex.candidatesFor instead of newEventStream.tryEmit.
Added private fanout(event) helper. Kept main's
snapshotIdsForNegentropy method.
- geode/.../LoadBenchmark.kt: both sides added a new @Test method.
Kept fanoutScaling (this branch) and publishPipelinedSingleClient
(main).
Align the NIP-77 large-corpus plan with strfry's actual defaults:
500_000-byte frame cap (not 64 KB), no default since-window, 200
shared sub cap, max_sync_events overflow protection, exact NEG-ERR
wording, and 32-byte id storage. Add a strfry round-trip benchmark
and call out the BTreeLMDB pre-built tree as a follow-up.
Tier 3 used to say operators "must omit VerifyPolicy from their
policy chain" when parallelVerify is on — that turned out to be
the AUTH-verify regression caught in the audit. Updated the plan
to describe the real wiring: VerifyPolicy was split into a
parameterised base with two singletons, and composePolicy swaps in
VerifyAuthOnlyPolicy so AUTH commands keep signature verification
even when the IngestQueue takes EVENT verify.
Verifies the FilterIndex path in LiveEventStore: each event matches
exactly ONE of N subscribers, so without an index the relay walks
all N subs per event (O(N)); with the author-keyed index the lookup
is O(1) and per-event latency stays flat as N grows.
Different from fanoutLatency (broadcast: 1 event reaches every
sub). Here per-event work is lookup-bound rather than
delivery-bound. Configurable via -DfanoutScalingSubs (comma list)
and -DfanoutScalingEvents.
Measured (laptop, JDK 21, full sweep):
100 subs, 2k events: p50=1.35ms p99=5.71ms
1000 subs, 2k events: p50=1.05ms p99=3.05ms
5000 subs, 1k events: p50=1.01ms p99=2.96ms
p50 ~1ms across N — the predicted O(1) scaling. Default events count
adapts downward at high N to stay below WebSocketSessionPump's
8192-frame outbound cap (test-client read side is the bottleneck
above ~5k subs, not the relay).
Plan doc updated with the measured numbers in the "How to verify"
section.
Closes the last item on the event-ingestion-batching plan: signature
verification no longer serialises on each connection's WebSocket
pump. Instead, IngestQueue takes a `verify: ((Event) -> Boolean)?`
hook and fan-outs the per-batch verify across Dispatchers.Default
(`coroutineScope { events.map { async { verify(it) } }.awaitAll() }`)
before opening the SQLite transaction. Failed verifies pre-mark
Rejected and skip the insert.
Wiring:
- NostrServer takes `parallelVerify: Boolean = false` (opt-in to
preserve existing behaviour for direct library users).
- geode.Relay forwards a matching flag.
- Main.kt enables it whenever signature checking is on (config
`[options].parallel_verify = true`, default true), and when so,
composePolicy is told to skip VerifyPolicy from the chain to
avoid double-verifying every event.
- New CLI escape hatch `--no-parallel-verify` for the legacy path.
Bench: adds publishGroupCommitSingleClient (sequential publish-and-
confirm; 500 EPS regression floor for the synchronous path) — the
companion to the existing pipelined bench that exercises the
group-commit + parallel-verify wins.
Plan doc updated to describe what shipped (batchInsert + SAVEPOINTs
in Tier 1, IngestQueue mechanics in Tier 2, the verify hook in
Tier 3) and to drop the obsolete `synchronous=NORMAL` confirmation
note — the project ships `synchronous=OFF` and intentionally keeps
that.
Replace the SharedFlow-based broadcast in LiveEventStore with an
inverted index over filter-bearing subscribers. Each REQ registers
its filters into a per-store FilterIndex<LiveSubscription>; insert()
calls index.candidatesFor(event) and only delivers to candidates
whose Filter.match still passes. Cuts the per-event walk from
O(N_subs * N_filters) to a few hash lookups plus match() over a
small candidate set.
FilterIndex itself lives next to Filter.kt (commonMain, KMP-friendly,
AtomicReference + COW) so other call sites with the same shape
(LocalCache.observables, ObservableEventStore.changes) can reuse it.
Each filter contributes entries on its single most-selective dimension
(ids > authors > tags > tagsAll > kinds > unindexed) to keep buckets
narrow and avoid Set-dedupe work in candidatesFor.
The historical-replay race the previous SharedFlow + onSubscription
handoff closed is preserved by registering BEFORE replay starts and
deduping seen ids until EOSE.
OK frames carry the event id, so clients pair replies by id rather
than by arrival order. NIP-01 also treats OK true as "accepted," not
"fsynced." That removes two constraints the plan was carrying:
- no per-connection FIFO requirement on OKs
- no need to delay OKs until after batch fsync
Tier 1 can fan OKs out as soon as the per-row INSERT returns inside
the open transaction, hiding the group-commit fsync entirely from
publisher latency. Tier 2 drops the order-preserving commit log and
just sends OKs straight to outQueue. The pipelined benchmark now
checks one-OK-per-event-id rather than ordering.
Marks Sketches A and B done, with a note that A took the simpler
Channel.UNLIMITED + AtomicInteger cap path the original Risks section
called out, sidestepping the channel-swap that the plan first
sketched.
Records that the streaming-filter slice of Sketch C landed in Quartz,
and that the larger envelope-streaming work the plan called out is
unnecessary because MessageDeserializer / CommandDeserializer /
EventDeserializer were already streaming — only the filter sub-object
went through readTree.
Adds the verification benchmarks (connectionsHeldOpen10k,
connectionsHeldOpenWithFanout) to the verification section, with a
correction that what's measured is JVM heap not OS RSS.
Carries forward the not-done items (fan-out de-duplication,
filter-matching index, Netty engine) into the open-work section,
pointing at live-broadcast-fanout-index.md for the highest-leverage
remaining work.
Four sketches, queued by impact, each grounded in current code paths
and observed benchmark numbers:
- event-ingestion-batching: SQLite group commit + EVENT pipelining +
off-thread Schnorr verify. Targets 5–10× EPS on a fast SSD.
- live-broadcast-fanout-index: indexed filter matching to replace the
O(N_subs × N_filters) per-event walk in LiveEventStore. Targets
flat fanout p99 up to high subscriber counts.
- connection-scaling: shrink the per-session outQueue footprint
(currently the dominant per-conn cost), tune Ktor CIO group sizes,
reduce JSON parse allocations. Targets 10 000+ concurrent conns.
- negentropy-large-corpus: id-and-time-only snapshot path so NEG-OPEN
on a 5M-event store doesn't materialise full Event objects, plus
bounded-window defaults and concurrent-session caps.
Each plan names the verification benchmark to add. Plans are queued,
not committed work — README orders them by expected impact.
Vitor Pamplona