Replace the SharedFlow-based broadcast in LiveEventStore with an
inverted index over filter-bearing subscribers. Each REQ registers
its filters into a per-store FilterIndex<LiveSubscription>; insert()
calls index.candidatesFor(event) and only delivers to candidates
whose Filter.match still passes. Cuts the per-event walk from
O(N_subs * N_filters) to a few hash lookups plus match() over a
small candidate set.
FilterIndex itself lives next to Filter.kt (commonMain, KMP-friendly,
AtomicReference + COW) so other call sites with the same shape
(LocalCache.observables, ObservableEventStore.changes) can reuse it.
Each filter contributes entries on its single most-selective dimension
(ids > authors > tags > tagsAll > kinds > unindexed) to keep buckets
narrow and avoid Set-dedupe work in candidatesFor.
The historical-replay race the previous SharedFlow + onSubscription
handoff closed is preserved by registering BEFORE replay starts and
deduping seen ids until EOSE.
Marks Sketches A and B done, with a note that A took the simpler
Channel.UNLIMITED + AtomicInteger cap path the original Risks section
called out, sidestepping the channel-swap that the plan first
sketched.
Records that the streaming-filter slice of Sketch C landed in Quartz,
and that the larger envelope-streaming work the plan called out is
unnecessary because MessageDeserializer / CommandDeserializer /
EventDeserializer were already streaming — only the filter sub-object
went through readTree.
Adds the verification benchmarks (connectionsHeldOpen10k,
connectionsHeldOpenWithFanout) to the verification section, with a
correction that what's measured is JVM heap not OS RSS.
Carries forward the not-done items (fan-out de-duplication,
filter-matching index, Netty engine) into the open-work section,
pointing at live-broadcast-fanout-index.md for the highest-leverage
remaining work.
Four sketches, queued by impact, each grounded in current code paths
and observed benchmark numbers:
- event-ingestion-batching: SQLite group commit + EVENT pipelining +
off-thread Schnorr verify. Targets 5–10× EPS on a fast SSD.
- live-broadcast-fanout-index: indexed filter matching to replace the
O(N_subs × N_filters) per-event walk in LiveEventStore. Targets
flat fanout p99 up to high subscriber counts.
- connection-scaling: shrink the per-session outQueue footprint
(currently the dominant per-conn cost), tune Ktor CIO group sizes,
reduce JSON parse allocations. Targets 10 000+ concurrent conns.
- negentropy-large-corpus: id-and-time-only snapshot path so NEG-OPEN
on a 5M-event store doesn't materialise full Event objects, plus
bounded-window defaults and concurrent-session caps.
Each plan names the verification benchmark to add. Plans are queued,
not committed work — README orders them by expected impact.
Claude