docs(geode): refine negentropy plan for strfry interop
Align the NIP-77 large-corpus plan with strfry's actual defaults: 500_000-byte frame cap (not 64 KB), no default since-window, 200 shared sub cap, max_sync_events overflow protection, exact NEG-ERR wording, and 32-byte id storage. Add a strfry round-trip benchmark and call out the BTreeLMDB pre-built tree as a follow-up.
This commit is contained in:
Claude
@@ -1,20 +1,20 @@
|
||||
# NIP-77 negentropy at scale: snapshot memory + chunked replay
|
||||
# NIP-77 negentropy at scale: strfry-interop snapshot path
|
||||
|
||||
## Problem
|
||||
|
||||
`RelaySession` delegates NEG-OPEN to `NegSessionRegistry.open`
|
||||
(`quartz/nip01Core/relay/server/NegSessionRegistry.kt`), which calls
|
||||
`store.snapshotQuery(filters)` and feeds the **entire** result list
|
||||
into `NegentropyServerSession`. For a relay holding 5 M events that
|
||||
match a broad NEG-OPEN filter (`{kinds: [1, 7]}`), this is 5 M
|
||||
`Event` objects materialised in memory before the first NEG-MSG goes
|
||||
out.
|
||||
(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/server/NegSessionRegistry.kt:58-77`),
|
||||
which calls `store.snapshotQuery(filters)` and feeds the **entire**
|
||||
result list of full `Event` objects into `NegentropyServerSession`
|
||||
(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip77Negentropy/NegentropyServerSession.kt:40-54`).
|
||||
For a relay holding 5 M events that match a broad NEG-OPEN filter
|
||||
(`{kinds: [1, 7]}`), this materialises 5 M `Event` objects with
|
||||
content/tags/sig — call it ~1 KB/event, ~5 GB transient pressure per
|
||||
concurrent NEG-OPEN — before the first NEG-MSG goes out.
|
||||
|
||||
The negentropy library itself is fine — it pivots into a sealed
|
||||
`StorageVector` (id + createdAt only, ~40 bytes/entry). But the
|
||||
`store.query<Event>(f)` step that produces the input materialises full
|
||||
`Event` objects with content, tags, sig — call it ~1 KB/event. 5 M ×
|
||||
1 KB = 5 GB transient pressure per concurrent NEG-OPEN.
|
||||
The negentropy library itself is fine. Internally it pivots into a
|
||||
sealed `StorageVector` of `(uint64 timestamp, byte[32] id)` items —
|
||||
40 bytes/entry. The waste is purely in the snapshot step.
|
||||
|
||||
Two operator-visible symptoms:
|
||||
|
||||
@@ -24,77 +24,221 @@ Two operator-visible symptoms:
|
||||
large stores the client waits seconds for what should be a
|
||||
millisecond round-trip.
|
||||
|
||||
## Reference: strfry
|
||||
|
||||
We want byte-for-byte interop and comparable throughput against
|
||||
[hoytech/strfry](https://github.com/hoytech/strfry). The relevant
|
||||
defaults from `strfry/src/apps/relay/RelayNegentropy.cpp` and
|
||||
`golpe.yaml`:
|
||||
|
||||
| Knob | strfry default | Notes |
|
||||
|---------------------------------------|-----------------------|-------|
|
||||
| `frameSizeLimit` (NEG-MSG payload) | **500_000 bytes** | Hard-coded `Negentropy ne(storage, 500'000)` |
|
||||
| `relay__negentropy__maxSyncEvents` | **1_000_000** | Hard cap on items in the snapshot |
|
||||
| `relay__maxSubsPerConnection` | **200** | Shared between REQ and NEG sessions |
|
||||
| `idSize` on the wire | **32 bytes** | NIP-77 v1 (`PROTOCOL_VERSION = 0x61`) |
|
||||
| Default `since` window | **none** | Filter is honored as-is |
|
||||
| Filter parser | same as REQ | Honors `limit`, `kinds`, `authors`, `#tags` |
|
||||
| Snapshot data | `(created_at, id)` only | LMDB scan inserts into `negentropy::storage::Vector` |
|
||||
|
||||
Two-phase materialisation in strfry: `QueryScheduler` scans LMDB
|
||||
asynchronously, batching matched level-ids into a `vector<uint64_t>`;
|
||||
on completion the worker pulls each event header, calls
|
||||
`storageVector.insert(packed.created_at(), packed.id())`, then
|
||||
`seal()`s. The session response is sent only after seal.
|
||||
|
||||
Our equivalent must (a) match the snapshot footprint of ~40 bytes per
|
||||
event, and (b) match the wire-level frame-cap so a single
|
||||
reconciliation round-trip exchanges the same payload size.
|
||||
|
||||
## Sketch
|
||||
|
||||
### A — id-and-time-only snapshot path
|
||||
### A — id-and-time-only snapshot path (memory parity)
|
||||
|
||||
Negentropy only needs `(createdAt, id)` pairs. Add a streaming
|
||||
`IEventStore.queryIdAndTime(filter)` that returns
|
||||
`Sequence<Pair<Long, ByteArray>>` (or a `Flow` of small chunks) —
|
||||
no content/tags/sig, no Event allocation. SQLite path is a SELECT
|
||||
on `event_headers` (the `created_at`, `id` columns are already
|
||||
indexed for query plans).
|
||||
`IEventStore.snapshotIdsForNegentropy(filter)` that returns
|
||||
`Sequence<IdAndTime>` (`data class IdAndTime(val createdAt: Long, val id: ByteArray)`)
|
||||
— no content/tags/sig, no `Event` allocation. The SQLite path is a
|
||||
plain `SELECT id, created_at FROM event_headers WHERE …` against the
|
||||
existing `query_by_created_at_id` index
|
||||
(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/EventIndexesModule.kt:79`).
|
||||
|
||||
```kotlin
|
||||
suspend fun snapshotIdsForNegentropy(filter: Filter): IdTimeStream
|
||||
// IEventStore
|
||||
suspend fun snapshotIdsForNegentropy(filters: List<Filter>): List<IdAndTime>
|
||||
|
||||
// LiveEventStore — already deduplicates union of multi-filter results
|
||||
suspend fun snapshotIdsForNegentropy(filters: List<Filter>): List<IdAndTime>
|
||||
```
|
||||
|
||||
`NegentropyServerSession` is rewritten to take that stream and feed
|
||||
it directly into the `StorageVector`. Memory drops from O(N × 1 KB)
|
||||
to O(N × 40 B) — a 25× reduction; for 5 M events, ~200 MB instead
|
||||
of 5 GB.
|
||||
`NegentropyServerSession` is rewritten to take that list (or a
|
||||
two-phase `pendingIds → seal` builder) and feed it directly into
|
||||
`StorageVector`. Memory drops from O(N × ~1 KB) to O(N × 40 B) — for
|
||||
1 M events, ~40 MB instead of ~1 GB; matches strfry's per-session
|
||||
footprint.
|
||||
|
||||
### B — bounded-window subscriptions
|
||||
**id encoding:** strfry stores 32-byte raw ids (NIP-77 v1
|
||||
`ID_SIZE = 32`); the 16-byte truncation is `FINGERPRINT_SIZE`, only used
|
||||
internally for SHA-256 accumulator output. The current Kotlin code
|
||||
already passes the hex `event.id` string to `storage.insert(..)`
|
||||
(`NegentropyServerSession.kt:50`); the kmp-negentropy library decodes
|
||||
that to 32 raw bytes. Confirm this is preserved when we switch to a
|
||||
`ByteArray` id input — do not pre-truncate.
|
||||
|
||||
Most NEG-OPENs from real Nostr clients want the last 30 days, not
|
||||
"everything." If the client doesn't supply `since`, the server can
|
||||
default to a configurable horizon (e.g. 90 days) and surface this in
|
||||
the NIP-11 `limitation.negentropy_max_lookback_seconds` field.
|
||||
Operators can lift the cap; clients reading the doc know the bound.
|
||||
### B — bounded snapshot size, NOT a default `since` window
|
||||
|
||||
This is a NIP-spec-adjacent question more than a code change — needs
|
||||
a comment on whether the spec allows it. nostr-rs-relay does this
|
||||
already.
|
||||
The previous draft of this plan suggested defaulting `since` to a 90-day
|
||||
horizon. **Drop that.** strfry doesn't do it — the filter is honored
|
||||
as-is — and silently bounding `since` would break interop with strfry's
|
||||
sync clients (e.g. `strfry sync`, nostr-sdk's negentropy reconciler):
|
||||
they ask for "everything" and rely on getting it.
|
||||
|
||||
### C — frame-size cap on NEG-MSG
|
||||
Instead match strfry's protection: a hard cap on the number of items
|
||||
that go into a single snapshot.
|
||||
|
||||
```toml
|
||||
[negentropy]
|
||||
max_sync_events = 1_000_000 # matches strfry's relay__negentropy__maxSyncEvents
|
||||
```
|
||||
|
||||
`NegSessionRegistry.open` checks `count >= max_sync_events` after the
|
||||
SQLite count or during scan, and on overflow sends:
|
||||
|
||||
```
|
||||
["NEG-ERR", "<subId>", "blocked: too many query results"]
|
||||
```
|
||||
|
||||
Exact wording matches strfry so client error-handling that string-matches
|
||||
behaves identically. (Spec doesn't normatively define error text; this is
|
||||
de-facto interop.)
|
||||
|
||||
### C — frame-size cap on NEG-MSG: 500_000 bytes (strfry parity)
|
||||
|
||||
`NegentropyServerSession` is constructed with `frameSizeLimit = 0`
|
||||
(no limit). At very large reconciliations the message can grow large.
|
||||
Set a default `frameSizeLimit = 64 * 1024` (matching the typical WS
|
||||
frame budget) so NEG-MSGs don't blow past `[limits].max_ws_frame_bytes`.
|
||||
today. **Set `frameSizeLimit = 500_000`** (matching strfry) so a single
|
||||
NEG-MSG round-trip carries the same payload as strfry's. 64 KB — what
|
||||
the previous draft suggested — would force 8× more round-trips for
|
||||
large reconciliations and noticeably slow sync against strfry-style
|
||||
clients that expect ~1 MB hex-encoded NEG-MSGs.
|
||||
|
||||
The library already supports this — pure config change in
|
||||
`NegSessionRegistry.open`.
|
||||
The kmp-negentropy library enforces `frameSizeLimit >= 4096` (or `0`
|
||||
for unlimited). 500_000 is well above the floor. Pure config change in
|
||||
`NegSessionRegistry.open`; expose via `[negentropy].frame_size_limit`
|
||||
for operators who tune it down to fit smaller WS frame budgets.
|
||||
|
||||
### D — concurrent NEG-OPEN cap
|
||||
Note: `LimitsSection.max_ws_frame_bytes`
|
||||
(`geode/src/main/kotlin/com/vitorpamplona/geode/config/RelayConfig.kt:148`)
|
||||
applies to the WebSocket layer. After hex-encoding a 500_000-byte
|
||||
negentropy payload doubles to ~1_000_000 bytes on the wire; ensure
|
||||
`max_ws_frame_bytes` is at least 2 MB (or unlimited) in default
|
||||
config so the response isn't truncated by the WS layer.
|
||||
|
||||
### D — concurrent NEG-OPEN cap, shared with REQ
|
||||
|
||||
A NEG-OPEN holds session state until NEG-CLOSE (or connection close).
|
||||
Today nothing caps the number of concurrent open negentropy sessions
|
||||
per connection. A misbehaving (or hostile) client could open thousands
|
||||
and pin RAM. Add `MAX_NEG_SESSIONS_PER_CONNECTION = 16`, send NEG-ERR
|
||||
on overflow.
|
||||
Today `NegSessionRegistry.sessions` is unbounded. strfry caps at 200
|
||||
sessions per connection **shared with REQ** — both count against
|
||||
`relay__maxSubsPerConnection`.
|
||||
|
||||
Implement as:
|
||||
- Reuse the existing per-connection REQ subscription cap (or introduce
|
||||
one if absent) and let NEG sessions consume the same budget.
|
||||
- Default cap: **200** to match strfry. Configurable via
|
||||
`[limits].max_subs_per_connection`.
|
||||
- On overflow strfry sends a NOTICE, not NEG-ERR:
|
||||
`["NOTICE", "too many concurrent NEG requests"]`. We should match
|
||||
this — `NEG-ERR` is reserved for per-session protocol errors in
|
||||
strfry's model.
|
||||
|
||||
### E — pre-built fingerprint tree (follow-up, not in scope here)
|
||||
|
||||
strfry's real production-scale advantage is the **`negentropy::storage::BTreeLMDB`** backend: a persistent, incrementally-maintained B-tree
|
||||
of `(timestamp, id)` keys with per-node fingerprint accumulators.
|
||||
When NEG-OPEN's filter string matches a pre-registered
|
||||
`NegentropyFilter` tree, strfry skips the materialise-and-seal step
|
||||
entirely and reconciles directly off the LMDB B-tree in O(log n)
|
||||
fingerprint computations. See `env.foreach_NegentropyFilter` and
|
||||
`addStatelessView` in `RelayNegentropy.cpp`.
|
||||
|
||||
Equivalent for geode: a `quartz/.../nip77Negentropy/PrebuiltStorage`
|
||||
backed by a SQLite-side incremental fingerprint index, registered per
|
||||
canonical filter. This is a substantial piece of work and **out of
|
||||
scope for this plan** — call it out for a follow-up
|
||||
(`geode/plans/2026-05-08-negentropy-prebuilt-tree.md`). The A+B+C+D
|
||||
combination is enough to match strfry's `MemoryView` path, which is
|
||||
what 99% of ad-hoc NEG-OPENs hit.
|
||||
|
||||
## Concrete error-string interop table
|
||||
|
||||
Match strfry verbatim — clients in the wild string-match on these:
|
||||
|
||||
| Condition | strfry frame |
|
||||
|---------------------------------------|-----------------------------------------------------------|
|
||||
| Snapshot exceeds `max_sync_events` | `["NEG-ERR", "<subId>", "blocked: too many query results"]` |
|
||||
| NEG-MSG for unknown subId | `["NEG-ERR", "<subId>", "closed: unknown subscription handle"]` |
|
||||
| Library `reconcile()` parse failure | `["NEG-ERR", "<subId>", "PROTOCOL-ERROR"]` |
|
||||
| Per-connection sub cap exceeded | `["NOTICE", "too many concurrent NEG requests"]` |
|
||||
| NEG-MSG before NEG-OPEN seal complete | `["NOTICE", "negentropy error: got NEG-MSG before NEG-OPEN complete"]` |
|
||||
|
||||
The current Kotlin code in `NegSessionRegistry.kt:82` sends
|
||||
`"error: no negentropy session for <subId>"` for the unknown-subId
|
||||
case. Update to strfry's wording.
|
||||
|
||||
## Wire-level conformance checks
|
||||
|
||||
Before merging, verify against strfry as ground truth:
|
||||
|
||||
1. **Round-trip with `strfry sync`**: stand up a small geode instance,
|
||||
point `strfry sync ws://geode-host` at it, confirm sync completes
|
||||
and converges in ≤ comparable round-trips for an N=100k corpus.
|
||||
2. **idSize**: assert kmp-negentropy round-trips full 32-byte ids;
|
||||
the 16-byte fingerprint stays internal to the library.
|
||||
3. **Protocol version byte**: NIP-77 v1 = `0x61`. Confirm
|
||||
`NegentropyServerSession.processMessage` neither emits nor accepts
|
||||
a different version byte. Negotiation happens inside the library;
|
||||
surface the version on `NIP-11.limitation.negentropy = 1` so clients
|
||||
know the v1 path is supported.
|
||||
|
||||
## How to verify
|
||||
|
||||
Add to `geode.perf.LoadBenchmark`:
|
||||
Add to `geode/src/test/kotlin/com/vitorpamplona/geode/perf/LoadBenchmark.kt`:
|
||||
|
||||
- `negentropyOpenLatencyLargeCorpus` — preload 1 M events (use
|
||||
fixtures), measure NEG-OPEN → first NEG-MSG latency. Target <100 ms.
|
||||
- `negentropyOpenLatencyLargeCorpus` — preload 1 M events, measure
|
||||
NEG-OPEN → first NEG-MSG latency. Target **<200 ms** (strfry's
|
||||
C++ `MemoryView` path on equivalent hardware does ~80–150 ms;
|
||||
we expect a 1.5–2× JVM tax).
|
||||
- `negentropyMemoryPressure` — open 10 concurrent NEG-OPENs on the
|
||||
same large corpus; measure RSS delta, target <500 MB.
|
||||
same large corpus; measure RSS delta. Target **<500 MB**
|
||||
(10 × ~40 MB session footprint + scan overhead).
|
||||
- `negentropyStrfryInterop` — programmatic round-trip against a
|
||||
containerised `hoytech/strfry`: same fixture corpus loaded in both,
|
||||
cross-sync, assert byte-identical id-set convergence in equal
|
||||
round-trips ±1.
|
||||
|
||||
Add to `quartz/.../nip77Negentropy/`:
|
||||
|
||||
- `NegentropyServerSessionTest.processMessage_atFrameLimit_splitsAcrossRounds`
|
||||
— large symmetric difference, assert each NEG-MSG payload is
|
||||
≤ 500_000 bytes and the session completes in N rounds (not 1).
|
||||
|
||||
## Risks
|
||||
|
||||
- **`Sequence`/`Flow` over SQLite cursor**: holding a cursor open
|
||||
across the full sync is fragile if the client stalls. Materialise
|
||||
to a smaller in-memory list (just (id, createdAt)) once, reuse for
|
||||
the lifetime of the session. Memory bound is the same.
|
||||
- **Defaulting `since` is a behaviour change**: existing clients that
|
||||
expect "everything" silently get a bounded window. Either (a) make
|
||||
it opt-in via `RelayConfig.NegentropySection.default_lookback_seconds
|
||||
= null`, (b) advertise the cap in NIP-11 so well-behaved clients
|
||||
read it.
|
||||
- **Frame-size cap can break older clients**: the NIP-77 reference
|
||||
implementation (kmp-negentropy) handles this gracefully — multi-frame
|
||||
reconciliation is in spec — but field-test against a known-working
|
||||
client (e.g. nstart, primal-cache) before flipping the default.
|
||||
- **Cursor lifetime**: holding a SQLite cursor open across the full
|
||||
sync is fragile if the client stalls. Materialise to a smaller
|
||||
in-memory `List<IdAndTime>` (40 bytes/entry) once at NEG-OPEN time,
|
||||
reuse for the lifetime of the session. Bounded by `max_sync_events`.
|
||||
- **Frame-size 500_000 vs WS frame budget**: hex-encoded payload is
|
||||
~1 MB. If `LimitsSection.max_ws_frame_bytes` is set lower than
|
||||
~1.5 MB the response gets truncated/rejected by the WS layer. Lift
|
||||
the WS default OR cap `frame_size_limit` to
|
||||
`max_ws_frame_bytes / 2` at startup; fail-fast log a warning if the
|
||||
operator's config makes negentropy unusable.
|
||||
- **No default `since` (intentional, but worth flagging)**: a hostile
|
||||
client doing `NEG-OPEN {kinds:[1]}` against a large corpus will hit
|
||||
`max_sync_events` and get NEG-ERR. The cap is the protection; do
|
||||
not also silently bound `since`.
|
||||
- **kmp-negentropy library version**: pinned to `v1.0.2`
|
||||
(`gradle/libs.versions.toml:9`). Confirm v1.0.2 enforces
|
||||
`frameSizeLimit >= 4096`, supports protocol byte `0x61`, and
|
||||
internal `ID_SIZE = 32`. If any of those don't match strfry, this
|
||||
plan needs an upstream fix on kmp-negentropy first.
|
||||
|
||||
Reference in New Issue
Block a user