Claude
2c0ad4fbf5
Four sketches, queued by impact, each grounded in current code paths and observed benchmark numbers: - event-ingestion-batching: SQLite group commit + EVENT pipelining + off-thread Schnorr verify. Targets 5–10× EPS on a fast SSD. - live-broadcast-fanout-index: indexed filter matching to replace the O(N_subs × N_filters) per-event walk in LiveEventStore. Targets flat fanout p99 up to high subscriber counts. - connection-scaling: shrink the per-session outQueue footprint (currently the dominant per-conn cost), tune Ktor CIO group sizes, reduce JSON parse allocations. Targets 10 000+ concurrent conns. - negentropy-large-corpus: id-and-time-only snapshot path so NEG-OPEN on a 5M-event store doesn't materialise full Event objects, plus bounded-window defaults and concurrent-session caps. Each plan names the verification benchmark to add. Plans are queued, not committed work — README orders them by expected impact.
101 lines
4.1 KiB
Markdown
101 lines
4.1 KiB
Markdown
# NIP-77 negentropy at scale: snapshot memory + chunked replay
|
||
|
||
## Problem
|
||
|
||
`RelaySession` delegates NEG-OPEN to `NegSessionRegistry.open`
|
||
(`quartz/nip01Core/relay/server/NegSessionRegistry.kt`), which calls
|
||
`store.snapshotQuery(filters)` and feeds the **entire** result list
|
||
into `NegentropyServerSession`. For a relay holding 5 M events that
|
||
match a broad NEG-OPEN filter (`{kinds: [1, 7]}`), this is 5 M
|
||
`Event` objects materialised in memory before the first NEG-MSG goes
|
||
out.
|
||
|
||
The negentropy library itself is fine — it pivots into a sealed
|
||
`StorageVector` (id + createdAt only, ~40 bytes/entry). But the
|
||
`store.query<Event>(f)` step that produces the input materialises full
|
||
`Event` objects with content, tags, sig — call it ~1 KB/event. 5 M ×
|
||
1 KB = 5 GB transient pressure per concurrent NEG-OPEN.
|
||
|
||
Two operator-visible symptoms:
|
||
|
||
1. NEG-OPEN with a broad filter spikes JVM heap; under load, GC pause
|
||
stalls every other handler on the same process.
|
||
2. NEG-OPEN latency before the first NEG-MSG response is O(N) — for
|
||
large stores the client waits seconds for what should be a
|
||
millisecond round-trip.
|
||
|
||
## Sketch
|
||
|
||
### A — id-and-time-only snapshot path
|
||
|
||
Negentropy only needs `(createdAt, id)` pairs. Add a streaming
|
||
`IEventStore.queryIdAndTime(filter)` that returns
|
||
`Sequence<Pair<Long, ByteArray>>` (or a `Flow` of small chunks) —
|
||
no content/tags/sig, no Event allocation. SQLite path is a SELECT
|
||
on `event_headers` (the `created_at`, `id` columns are already
|
||
indexed for query plans).
|
||
|
||
```kotlin
|
||
suspend fun snapshotIdsForNegentropy(filter: Filter): IdTimeStream
|
||
```
|
||
|
||
`NegentropyServerSession` is rewritten to take that stream and feed
|
||
it directly into the `StorageVector`. Memory drops from O(N × 1 KB)
|
||
to O(N × 40 B) — a 25× reduction; for 5 M events, ~200 MB instead
|
||
of 5 GB.
|
||
|
||
### B — bounded-window subscriptions
|
||
|
||
Most NEG-OPENs from real Nostr clients want the last 30 days, not
|
||
"everything." If the client doesn't supply `since`, the server can
|
||
default to a configurable horizon (e.g. 90 days) and surface this in
|
||
the NIP-11 `limitation.negentropy_max_lookback_seconds` field.
|
||
Operators can lift the cap; clients reading the doc know the bound.
|
||
|
||
This is a NIP-spec-adjacent question more than a code change — needs
|
||
a comment on whether the spec allows it. nostr-rs-relay does this
|
||
already.
|
||
|
||
### C — frame-size cap on NEG-MSG
|
||
|
||
`NegentropyServerSession` is constructed with `frameSizeLimit = 0`
|
||
(no limit). At very large reconciliations the message can grow large.
|
||
Set a default `frameSizeLimit = 64 * 1024` (matching the typical WS
|
||
frame budget) so NEG-MSGs don't blow past `[limits].max_ws_frame_bytes`.
|
||
|
||
The library already supports this — pure config change in
|
||
`NegSessionRegistry.open`.
|
||
|
||
### D — concurrent NEG-OPEN cap
|
||
|
||
A NEG-OPEN holds session state until NEG-CLOSE (or connection close).
|
||
Today nothing caps the number of concurrent open negentropy sessions
|
||
per connection. A misbehaving (or hostile) client could open thousands
|
||
and pin RAM. Add `MAX_NEG_SESSIONS_PER_CONNECTION = 16`, send NEG-ERR
|
||
on overflow.
|
||
|
||
## How to verify
|
||
|
||
Add to `geode.perf.LoadBenchmark`:
|
||
|
||
- `negentropyOpenLatencyLargeCorpus` — preload 1 M events (use
|
||
fixtures), measure NEG-OPEN → first NEG-MSG latency. Target <100 ms.
|
||
- `negentropyMemoryPressure` — open 10 concurrent NEG-OPENs on the
|
||
same large corpus; measure RSS delta, target <500 MB.
|
||
|
||
## Risks
|
||
|
||
- **`Sequence`/`Flow` over SQLite cursor**: holding a cursor open
|
||
across the full sync is fragile if the client stalls. Materialise
|
||
to a smaller in-memory list (just (id, createdAt)) once, reuse for
|
||
the lifetime of the session. Memory bound is the same.
|
||
- **Defaulting `since` is a behaviour change**: existing clients that
|
||
expect "everything" silently get a bounded window. Either (a) make
|
||
it opt-in via `RelayConfig.NegentropySection.default_lookback_seconds
|
||
= null`, (b) advertise the cap in NIP-11 so well-behaved clients
|
||
read it.
|
||
- **Frame-size cap can break older clients**: the NIP-77 reference
|
||
implementation (kmp-negentropy) handles this gracefully — multi-frame
|
||
reconciliation is in spec — but field-test against a known-working
|
||
client (e.g. nstart, primal-cache) before flipping the default.
|