diff --git a/geode/plans/2026-05-07-negentropy-large-corpus.md b/geode/plans/2026-05-07-negentropy-large-corpus.md index 0cb31ccd9..55d057d01 100644 --- a/geode/plans/2026-05-07-negentropy-large-corpus.md +++ b/geode/plans/2026-05-07-negentropy-large-corpus.md @@ -1,20 +1,20 @@ -# NIP-77 negentropy at scale: snapshot memory + chunked replay +# NIP-77 negentropy at scale: strfry-interop snapshot path ## Problem `RelaySession` delegates NEG-OPEN to `NegSessionRegistry.open` -(`quartz/nip01Core/relay/server/NegSessionRegistry.kt`), which calls -`store.snapshotQuery(filters)` and feeds the **entire** result list -into `NegentropyServerSession`. For a relay holding 5 M events that -match a broad NEG-OPEN filter (`{kinds: [1, 7]}`), this is 5 M -`Event` objects materialised in memory before the first NEG-MSG goes -out. +(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/server/NegSessionRegistry.kt:58-77`), +which calls `store.snapshotQuery(filters)` and feeds the **entire** +result list of full `Event` objects into `NegentropyServerSession` +(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip77Negentropy/NegentropyServerSession.kt:40-54`). +For a relay holding 5 M events that match a broad NEG-OPEN filter +(`{kinds: [1, 7]}`), this materialises 5 M `Event` objects with +content/tags/sig — call it ~1 KB/event, ~5 GB transient pressure per +concurrent NEG-OPEN — before the first NEG-MSG goes out. -The negentropy library itself is fine — it pivots into a sealed -`StorageVector` (id + createdAt only, ~40 bytes/entry). But the -`store.query(f)` step that produces the input materialises full -`Event` objects with content, tags, sig — call it ~1 KB/event. 5 M × -1 KB = 5 GB transient pressure per concurrent NEG-OPEN. +The negentropy library itself is fine. Internally it pivots into a +sealed `StorageVector` of `(uint64 timestamp, byte[32] id)` items — +40 bytes/entry. The waste is purely in the snapshot step. Two operator-visible symptoms: @@ -24,77 +24,221 @@ Two operator-visible symptoms: large stores the client waits seconds for what should be a millisecond round-trip. +## Reference: strfry + +We want byte-for-byte interop and comparable throughput against +[hoytech/strfry](https://github.com/hoytech/strfry). The relevant +defaults from `strfry/src/apps/relay/RelayNegentropy.cpp` and +`golpe.yaml`: + +| Knob | strfry default | Notes | +|---------------------------------------|-----------------------|-------| +| `frameSizeLimit` (NEG-MSG payload) | **500_000 bytes** | Hard-coded `Negentropy ne(storage, 500'000)` | +| `relay__negentropy__maxSyncEvents` | **1_000_000** | Hard cap on items in the snapshot | +| `relay__maxSubsPerConnection` | **200** | Shared between REQ and NEG sessions | +| `idSize` on the wire | **32 bytes** | NIP-77 v1 (`PROTOCOL_VERSION = 0x61`) | +| Default `since` window | **none** | Filter is honored as-is | +| Filter parser | same as REQ | Honors `limit`, `kinds`, `authors`, `#tags` | +| Snapshot data | `(created_at, id)` only | LMDB scan inserts into `negentropy::storage::Vector` | + +Two-phase materialisation in strfry: `QueryScheduler` scans LMDB +asynchronously, batching matched level-ids into a `vector`; +on completion the worker pulls each event header, calls +`storageVector.insert(packed.created_at(), packed.id())`, then +`seal()`s. The session response is sent only after seal. + +Our equivalent must (a) match the snapshot footprint of ~40 bytes per +event, and (b) match the wire-level frame-cap so a single +reconciliation round-trip exchanges the same payload size. + ## Sketch -### A — id-and-time-only snapshot path +### A — id-and-time-only snapshot path (memory parity) Negentropy only needs `(createdAt, id)` pairs. Add a streaming -`IEventStore.queryIdAndTime(filter)` that returns -`Sequence>` (or a `Flow` of small chunks) — -no content/tags/sig, no Event allocation. SQLite path is a SELECT -on `event_headers` (the `created_at`, `id` columns are already -indexed for query plans). +`IEventStore.snapshotIdsForNegentropy(filter)` that returns +`Sequence` (`data class IdAndTime(val createdAt: Long, val id: ByteArray)`) +— no content/tags/sig, no `Event` allocation. The SQLite path is a +plain `SELECT id, created_at FROM event_headers WHERE …` against the +existing `query_by_created_at_id` index +(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/EventIndexesModule.kt:79`). ```kotlin -suspend fun snapshotIdsForNegentropy(filter: Filter): IdTimeStream +// IEventStore +suspend fun snapshotIdsForNegentropy(filters: List): List + +// LiveEventStore — already deduplicates union of multi-filter results +suspend fun snapshotIdsForNegentropy(filters: List): List ``` -`NegentropyServerSession` is rewritten to take that stream and feed -it directly into the `StorageVector`. Memory drops from O(N × 1 KB) -to O(N × 40 B) — a 25× reduction; for 5 M events, ~200 MB instead -of 5 GB. +`NegentropyServerSession` is rewritten to take that list (or a +two-phase `pendingIds → seal` builder) and feed it directly into +`StorageVector`. Memory drops from O(N × ~1 KB) to O(N × 40 B) — for +1 M events, ~40 MB instead of ~1 GB; matches strfry's per-session +footprint. -### B — bounded-window subscriptions +**id encoding:** strfry stores 32-byte raw ids (NIP-77 v1 +`ID_SIZE = 32`); the 16-byte truncation is `FINGERPRINT_SIZE`, only used +internally for SHA-256 accumulator output. The current Kotlin code +already passes the hex `event.id` string to `storage.insert(..)` +(`NegentropyServerSession.kt:50`); the kmp-negentropy library decodes +that to 32 raw bytes. Confirm this is preserved when we switch to a +`ByteArray` id input — do not pre-truncate. -Most NEG-OPENs from real Nostr clients want the last 30 days, not -"everything." If the client doesn't supply `since`, the server can -default to a configurable horizon (e.g. 90 days) and surface this in -the NIP-11 `limitation.negentropy_max_lookback_seconds` field. -Operators can lift the cap; clients reading the doc know the bound. +### B — bounded snapshot size, NOT a default `since` window -This is a NIP-spec-adjacent question more than a code change — needs -a comment on whether the spec allows it. nostr-rs-relay does this -already. +The previous draft of this plan suggested defaulting `since` to a 90-day +horizon. **Drop that.** strfry doesn't do it — the filter is honored +as-is — and silently bounding `since` would break interop with strfry's +sync clients (e.g. `strfry sync`, nostr-sdk's negentropy reconciler): +they ask for "everything" and rely on getting it. -### C — frame-size cap on NEG-MSG +Instead match strfry's protection: a hard cap on the number of items +that go into a single snapshot. + +```toml +[negentropy] +max_sync_events = 1_000_000 # matches strfry's relay__negentropy__maxSyncEvents +``` + +`NegSessionRegistry.open` checks `count >= max_sync_events` after the +SQLite count or during scan, and on overflow sends: + +``` +["NEG-ERR", "", "blocked: too many query results"] +``` + +Exact wording matches strfry so client error-handling that string-matches +behaves identically. (Spec doesn't normatively define error text; this is +de-facto interop.) + +### C — frame-size cap on NEG-MSG: 500_000 bytes (strfry parity) `NegentropyServerSession` is constructed with `frameSizeLimit = 0` -(no limit). At very large reconciliations the message can grow large. -Set a default `frameSizeLimit = 64 * 1024` (matching the typical WS -frame budget) so NEG-MSGs don't blow past `[limits].max_ws_frame_bytes`. +today. **Set `frameSizeLimit = 500_000`** (matching strfry) so a single +NEG-MSG round-trip carries the same payload as strfry's. 64 KB — what +the previous draft suggested — would force 8× more round-trips for +large reconciliations and noticeably slow sync against strfry-style +clients that expect ~1 MB hex-encoded NEG-MSGs. -The library already supports this — pure config change in -`NegSessionRegistry.open`. +The kmp-negentropy library enforces `frameSizeLimit >= 4096` (or `0` +for unlimited). 500_000 is well above the floor. Pure config change in +`NegSessionRegistry.open`; expose via `[negentropy].frame_size_limit` +for operators who tune it down to fit smaller WS frame budgets. -### D — concurrent NEG-OPEN cap +Note: `LimitsSection.max_ws_frame_bytes` +(`geode/src/main/kotlin/com/vitorpamplona/geode/config/RelayConfig.kt:148`) +applies to the WebSocket layer. After hex-encoding a 500_000-byte +negentropy payload doubles to ~1_000_000 bytes on the wire; ensure +`max_ws_frame_bytes` is at least 2 MB (or unlimited) in default +config so the response isn't truncated by the WS layer. + +### D — concurrent NEG-OPEN cap, shared with REQ A NEG-OPEN holds session state until NEG-CLOSE (or connection close). -Today nothing caps the number of concurrent open negentropy sessions -per connection. A misbehaving (or hostile) client could open thousands -and pin RAM. Add `MAX_NEG_SESSIONS_PER_CONNECTION = 16`, send NEG-ERR -on overflow. +Today `NegSessionRegistry.sessions` is unbounded. strfry caps at 200 +sessions per connection **shared with REQ** — both count against +`relay__maxSubsPerConnection`. + +Implement as: +- Reuse the existing per-connection REQ subscription cap (or introduce + one if absent) and let NEG sessions consume the same budget. +- Default cap: **200** to match strfry. Configurable via + `[limits].max_subs_per_connection`. +- On overflow strfry sends a NOTICE, not NEG-ERR: + `["NOTICE", "too many concurrent NEG requests"]`. We should match + this — `NEG-ERR` is reserved for per-session protocol errors in + strfry's model. + +### E — pre-built fingerprint tree (follow-up, not in scope here) + +strfry's real production-scale advantage is the **`negentropy::storage::BTreeLMDB`** backend: a persistent, incrementally-maintained B-tree +of `(timestamp, id)` keys with per-node fingerprint accumulators. +When NEG-OPEN's filter string matches a pre-registered +`NegentropyFilter` tree, strfry skips the materialise-and-seal step +entirely and reconciles directly off the LMDB B-tree in O(log n) +fingerprint computations. See `env.foreach_NegentropyFilter` and +`addStatelessView` in `RelayNegentropy.cpp`. + +Equivalent for geode: a `quartz/.../nip77Negentropy/PrebuiltStorage` +backed by a SQLite-side incremental fingerprint index, registered per +canonical filter. This is a substantial piece of work and **out of +scope for this plan** — call it out for a follow-up +(`geode/plans/2026-05-08-negentropy-prebuilt-tree.md`). The A+B+C+D +combination is enough to match strfry's `MemoryView` path, which is +what 99% of ad-hoc NEG-OPENs hit. + +## Concrete error-string interop table + +Match strfry verbatim — clients in the wild string-match on these: + +| Condition | strfry frame | +|---------------------------------------|-----------------------------------------------------------| +| Snapshot exceeds `max_sync_events` | `["NEG-ERR", "", "blocked: too many query results"]` | +| NEG-MSG for unknown subId | `["NEG-ERR", "", "closed: unknown subscription handle"]` | +| Library `reconcile()` parse failure | `["NEG-ERR", "", "PROTOCOL-ERROR"]` | +| Per-connection sub cap exceeded | `["NOTICE", "too many concurrent NEG requests"]` | +| NEG-MSG before NEG-OPEN seal complete | `["NOTICE", "negentropy error: got NEG-MSG before NEG-OPEN complete"]` | + +The current Kotlin code in `NegSessionRegistry.kt:82` sends +`"error: no negentropy session for "` for the unknown-subId +case. Update to strfry's wording. + +## Wire-level conformance checks + +Before merging, verify against strfry as ground truth: + +1. **Round-trip with `strfry sync`**: stand up a small geode instance, + point `strfry sync ws://geode-host` at it, confirm sync completes + and converges in ≤ comparable round-trips for an N=100k corpus. +2. **idSize**: assert kmp-negentropy round-trips full 32-byte ids; + the 16-byte fingerprint stays internal to the library. +3. **Protocol version byte**: NIP-77 v1 = `0x61`. Confirm + `NegentropyServerSession.processMessage` neither emits nor accepts + a different version byte. Negotiation happens inside the library; + surface the version on `NIP-11.limitation.negentropy = 1` so clients + know the v1 path is supported. ## How to verify -Add to `geode.perf.LoadBenchmark`: +Add to `geode/src/test/kotlin/com/vitorpamplona/geode/perf/LoadBenchmark.kt`: -- `negentropyOpenLatencyLargeCorpus` — preload 1 M events (use - fixtures), measure NEG-OPEN → first NEG-MSG latency. Target <100 ms. +- `negentropyOpenLatencyLargeCorpus` — preload 1 M events, measure + NEG-OPEN → first NEG-MSG latency. Target **<200 ms** (strfry's + C++ `MemoryView` path on equivalent hardware does ~80–150 ms; + we expect a 1.5–2× JVM tax). - `negentropyMemoryPressure` — open 10 concurrent NEG-OPENs on the - same large corpus; measure RSS delta, target <500 MB. + same large corpus; measure RSS delta. Target **<500 MB** + (10 × ~40 MB session footprint + scan overhead). +- `negentropyStrfryInterop` — programmatic round-trip against a + containerised `hoytech/strfry`: same fixture corpus loaded in both, + cross-sync, assert byte-identical id-set convergence in equal + round-trips ±1. + +Add to `quartz/.../nip77Negentropy/`: + +- `NegentropyServerSessionTest.processMessage_atFrameLimit_splitsAcrossRounds` + — large symmetric difference, assert each NEG-MSG payload is + ≤ 500_000 bytes and the session completes in N rounds (not 1). ## Risks -- **`Sequence`/`Flow` over SQLite cursor**: holding a cursor open - across the full sync is fragile if the client stalls. Materialise - to a smaller in-memory list (just (id, createdAt)) once, reuse for - the lifetime of the session. Memory bound is the same. -- **Defaulting `since` is a behaviour change**: existing clients that - expect "everything" silently get a bounded window. Either (a) make - it opt-in via `RelayConfig.NegentropySection.default_lookback_seconds - = null`, (b) advertise the cap in NIP-11 so well-behaved clients - read it. -- **Frame-size cap can break older clients**: the NIP-77 reference - implementation (kmp-negentropy) handles this gracefully — multi-frame - reconciliation is in spec — but field-test against a known-working - client (e.g. nstart, primal-cache) before flipping the default. +- **Cursor lifetime**: holding a SQLite cursor open across the full + sync is fragile if the client stalls. Materialise to a smaller + in-memory `List` (40 bytes/entry) once at NEG-OPEN time, + reuse for the lifetime of the session. Bounded by `max_sync_events`. +- **Frame-size 500_000 vs WS frame budget**: hex-encoded payload is + ~1 MB. If `LimitsSection.max_ws_frame_bytes` is set lower than + ~1.5 MB the response gets truncated/rejected by the WS layer. Lift + the WS default OR cap `frame_size_limit` to + `max_ws_frame_bytes / 2` at startup; fail-fast log a warning if the + operator's config makes negentropy unusable. +- **No default `since` (intentional, but worth flagging)**: a hostile + client doing `NEG-OPEN {kinds:[1]}` against a large corpus will hit + `max_sync_events` and get NEG-ERR. The cap is the protection; do + not also silently bound `since`. +- **kmp-negentropy library version**: pinned to `v1.0.2` + (`gradle/libs.versions.toml:9`). Confirm v1.0.2 enforces + `frameSizeLimit >= 4096`, supports protocol byte `0x61`, and + internal `ID_SIZE = 32`. If any of those don't match strfry, this + plan needs an upstream fix on kmp-negentropy first.