docs(geode): performance plans for future work
Four sketches, queued by impact, each grounded in current code paths and observed benchmark numbers: - event-ingestion-batching: SQLite group commit + EVENT pipelining + off-thread Schnorr verify. Targets 5–10× EPS on a fast SSD. - live-broadcast-fanout-index: indexed filter matching to replace the O(N_subs × N_filters) per-event walk in LiveEventStore. Targets flat fanout p99 up to high subscriber counts. - connection-scaling: shrink the per-session outQueue footprint (currently the dominant per-conn cost), tune Ktor CIO group sizes, reduce JSON parse allocations. Targets 10 000+ concurrent conns. - negentropy-large-corpus: id-and-time-only snapshot path so NEG-OPEN on a 5M-event store doesn't materialise full Event objects, plus bounded-window defaults and concurrent-session caps. Each plan names the verification benchmark to add. Plans are queued, not committed work — README orders them by expected impact.
This commit is contained in:
Claude
@@ -0,0 +1,100 @@
|
||||
# NIP-77 negentropy at scale: snapshot memory + chunked replay
|
||||
|
||||
## Problem
|
||||
|
||||
`RelaySession` delegates NEG-OPEN to `NegSessionRegistry.open`
|
||||
(`quartz/nip01Core/relay/server/NegSessionRegistry.kt`), which calls
|
||||
`store.snapshotQuery(filters)` and feeds the **entire** result list
|
||||
into `NegentropyServerSession`. For a relay holding 5 M events that
|
||||
match a broad NEG-OPEN filter (`{kinds: [1, 7]}`), this is 5 M
|
||||
`Event` objects materialised in memory before the first NEG-MSG goes
|
||||
out.
|
||||
|
||||
The negentropy library itself is fine — it pivots into a sealed
|
||||
`StorageVector` (id + createdAt only, ~40 bytes/entry). But the
|
||||
`store.query<Event>(f)` step that produces the input materialises full
|
||||
`Event` objects with content, tags, sig — call it ~1 KB/event. 5 M ×
|
||||
1 KB = 5 GB transient pressure per concurrent NEG-OPEN.
|
||||
|
||||
Two operator-visible symptoms:
|
||||
|
||||
1. NEG-OPEN with a broad filter spikes JVM heap; under load, GC pause
|
||||
stalls every other handler on the same process.
|
||||
2. NEG-OPEN latency before the first NEG-MSG response is O(N) — for
|
||||
large stores the client waits seconds for what should be a
|
||||
millisecond round-trip.
|
||||
|
||||
## Sketch
|
||||
|
||||
### A — id-and-time-only snapshot path
|
||||
|
||||
Negentropy only needs `(createdAt, id)` pairs. Add a streaming
|
||||
`IEventStore.queryIdAndTime(filter)` that returns
|
||||
`Sequence<Pair<Long, ByteArray>>` (or a `Flow` of small chunks) —
|
||||
no content/tags/sig, no Event allocation. SQLite path is a SELECT
|
||||
on `event_headers` (the `created_at`, `id` columns are already
|
||||
indexed for query plans).
|
||||
|
||||
```kotlin
|
||||
suspend fun snapshotIdsForNegentropy(filter: Filter): IdTimeStream
|
||||
```
|
||||
|
||||
`NegentropyServerSession` is rewritten to take that stream and feed
|
||||
it directly into the `StorageVector`. Memory drops from O(N × 1 KB)
|
||||
to O(N × 40 B) — a 25× reduction; for 5 M events, ~200 MB instead
|
||||
of 5 GB.
|
||||
|
||||
### B — bounded-window subscriptions
|
||||
|
||||
Most NEG-OPENs from real Nostr clients want the last 30 days, not
|
||||
"everything." If the client doesn't supply `since`, the server can
|
||||
default to a configurable horizon (e.g. 90 days) and surface this in
|
||||
the NIP-11 `limitation.negentropy_max_lookback_seconds` field.
|
||||
Operators can lift the cap; clients reading the doc know the bound.
|
||||
|
||||
This is a NIP-spec-adjacent question more than a code change — needs
|
||||
a comment on whether the spec allows it. nostr-rs-relay does this
|
||||
already.
|
||||
|
||||
### C — frame-size cap on NEG-MSG
|
||||
|
||||
`NegentropyServerSession` is constructed with `frameSizeLimit = 0`
|
||||
(no limit). At very large reconciliations the message can grow large.
|
||||
Set a default `frameSizeLimit = 64 * 1024` (matching the typical WS
|
||||
frame budget) so NEG-MSGs don't blow past `[limits].max_ws_frame_bytes`.
|
||||
|
||||
The library already supports this — pure config change in
|
||||
`NegSessionRegistry.open`.
|
||||
|
||||
### D — concurrent NEG-OPEN cap
|
||||
|
||||
A NEG-OPEN holds session state until NEG-CLOSE (or connection close).
|
||||
Today nothing caps the number of concurrent open negentropy sessions
|
||||
per connection. A misbehaving (or hostile) client could open thousands
|
||||
and pin RAM. Add `MAX_NEG_SESSIONS_PER_CONNECTION = 16`, send NEG-ERR
|
||||
on overflow.
|
||||
|
||||
## How to verify
|
||||
|
||||
Add to `geode.perf.LoadBenchmark`:
|
||||
|
||||
- `negentropyOpenLatencyLargeCorpus` — preload 1 M events (use
|
||||
fixtures), measure NEG-OPEN → first NEG-MSG latency. Target <100 ms.
|
||||
- `negentropyMemoryPressure` — open 10 concurrent NEG-OPENs on the
|
||||
same large corpus; measure RSS delta, target <500 MB.
|
||||
|
||||
## Risks
|
||||
|
||||
- **`Sequence`/`Flow` over SQLite cursor**: holding a cursor open
|
||||
across the full sync is fragile if the client stalls. Materialise
|
||||
to a smaller in-memory list (just (id, createdAt)) once, reuse for
|
||||
the lifetime of the session. Memory bound is the same.
|
||||
- **Defaulting `since` is a behaviour change**: existing clients that
|
||||
expect "everything" silently get a bounded window. Either (a) make
|
||||
it opt-in via `RelayConfig.NegentropySection.default_lookback_seconds
|
||||
= null`, (b) advertise the cap in NIP-11 so well-behaved clients
|
||||
read it.
|
||||
- **Frame-size cap can break older clients**: the NIP-77 reference
|
||||
implementation (kmp-negentropy) handles this gracefully — multi-frame
|
||||
reconciliation is in spec — but field-test against a known-working
|
||||
client (e.g. nstart, primal-cache) before flipping the default.
|
||||
Reference in New Issue
Block a user