Files

T

Claude 2c0ad4fbf5 docs(geode): performance plans for future work

Four sketches, queued by impact, each grounded in current code paths
and observed benchmark numbers:

- event-ingestion-batching: SQLite group commit + EVENT pipelining +
  off-thread Schnorr verify. Targets 5–10× EPS on a fast SSD.
- live-broadcast-fanout-index: indexed filter matching to replace the
  O(N_subs × N_filters) per-event walk in LiveEventStore. Targets
  flat fanout p99 up to high subscriber counts.
- connection-scaling: shrink the per-session outQueue footprint
  (currently the dominant per-conn cost), tune Ktor CIO group sizes,
  reduce JSON parse allocations. Targets 10 000+ concurrent conns.
- negentropy-large-corpus: id-and-time-only snapshot path so NEG-OPEN
  on a 5M-event store doesn't materialise full Event objects, plus
  bounded-window defaults and concurrent-session caps.

Each plan names the verification benchmark to add. Plans are queued,
not committed work — README orders them by expected impact.

2026-05-07 14:05:11 +00:00

4.1 KiB

Raw Blame History

NIP-77 negentropy at scale: snapshot memory + chunked replay

Problem

RelaySession delegates NEG-OPEN to NegSessionRegistry.open (quartz/nip01Core/relay/server/NegSessionRegistry.kt), which calls store.snapshotQuery(filters) and feeds the entire result list into NegentropyServerSession. For a relay holding 5 M events that match a broad NEG-OPEN filter ({kinds: [1, 7]}), this is 5 M Event objects materialised in memory before the first NEG-MSG goes out.

The negentropy library itself is fine — it pivots into a sealed StorageVector (id + createdAt only, ~40 bytes/entry). But the store.query<Event>(f) step that produces the input materialises full Event objects with content, tags, sig — call it ~1 KB/event. 5 M × 1 KB = 5 GB transient pressure per concurrent NEG-OPEN.

Two operator-visible symptoms:

NEG-OPEN with a broad filter spikes JVM heap; under load, GC pause stalls every other handler on the same process.
NEG-OPEN latency before the first NEG-MSG response is O(N) — for large stores the client waits seconds for what should be a millisecond round-trip.

Sketch

A — id-and-time-only snapshot path

Negentropy only needs (createdAt, id) pairs. Add a streaming IEventStore.queryIdAndTime(filter) that returns Sequence<Pair<Long, ByteArray>> (or a Flow of small chunks) — no content/tags/sig, no Event allocation. SQLite path is a SELECT on event_headers (the created_at, id columns are already indexed for query plans).

suspend fun snapshotIdsForNegentropy(filter: Filter): IdTimeStream

NegentropyServerSession is rewritten to take that stream and feed it directly into the StorageVector. Memory drops from O(N × 1 KB) to O(N × 40 B) — a 25× reduction; for 5 M events, ~200 MB instead of 5 GB.

B — bounded-window subscriptions

Most NEG-OPENs from real Nostr clients want the last 30 days, not "everything." If the client doesn't supply since, the server can default to a configurable horizon (e.g. 90 days) and surface this in the NIP-11 limitation.negentropy_max_lookback_seconds field. Operators can lift the cap; clients reading the doc know the bound.

This is a NIP-spec-adjacent question more than a code change — needs a comment on whether the spec allows it. nostr-rs-relay does this already.

C — frame-size cap on NEG-MSG

NegentropyServerSession is constructed with frameSizeLimit = 0 (no limit). At very large reconciliations the message can grow large. Set a default frameSizeLimit = 64 * 1024 (matching the typical WS frame budget) so NEG-MSGs don't blow past [limits].max_ws_frame_bytes.

The library already supports this — pure config change in NegSessionRegistry.open.

D — concurrent NEG-OPEN cap

A NEG-OPEN holds session state until NEG-CLOSE (or connection close). Today nothing caps the number of concurrent open negentropy sessions per connection. A misbehaving (or hostile) client could open thousands and pin RAM. Add MAX_NEG_SESSIONS_PER_CONNECTION = 16, send NEG-ERR on overflow.

How to verify

Add to geode.perf.LoadBenchmark:

negentropyOpenLatencyLargeCorpus — preload 1 M events (use fixtures), measure NEG-OPEN → first NEG-MSG latency. Target <100 ms.
negentropyMemoryPressure — open 10 concurrent NEG-OPENs on the same large corpus; measure RSS delta, target <500 MB.

Risks

Sequence/Flow over SQLite cursor: holding a cursor open across the full sync is fragile if the client stalls. Materialise to a smaller in-memory list (just (id, createdAt)) once, reuse for the lifetime of the session. Memory bound is the same.
Defaulting since is a behaviour change: existing clients that expect "everything" silently get a bounded window. Either (a) make it opt-in via RelayConfig.NegentropySection.default_lookback_seconds = null, (b) advertise the cap in NIP-11 so well-behaved clients read it.
Frame-size cap can break older clients: the NIP-77 reference implementation (kmp-negentropy) handles this gracefully — multi-frame reconciliation is in spec — but field-test against a known-working client (e.g. nstart, primal-cache) before flipping the default.

4.1 KiB Raw Blame History Unescape Escape