diff --git a/cli/plans/2026-04-24-file-event-store-nips.md b/cli/plans/2026-04-24-file-event-store-nips.md new file mode 100644 index 000000000..ade1672a8 --- /dev/null +++ b/cli/plans/2026-04-24-file-event-store-nips.md @@ -0,0 +1,500 @@ +# File-backed event store for `amy` — Part 3: NIPs, Enforcement, Scrub, Testing + +**Status:** plan · **Date:** 2026-04-24 · **Part 3 of 3** +(see `2026-04-24-file-event-store-overview.md`, +`2026-04-24-file-event-store-pipelines.md`) + +Per-NIP behavior, slot enforcement details, tag indexing, scrub, and +test plan. + +--- + +## NIP-01 — Replaceable events (kinds 0, 3, 10000–19999) + +### Enforcement + +- Slot path: `replaceable//.json` +- Slot is a **hardlink** into `events///.json` — same + inode, two names. +- Directory-entry uniqueness is the `UNIQUE(kind, pubkey)` constraint + (parity with `ReplaceableModule.kt:25-60`). + +### Insert flow (inside transaction T2/T5 from Part 2) + +``` +guard = ReplaceableGuard + if slot exists: + Told = read slot JSON → createdAt + if event.createdAt <= Told: REJECT # blocks older from reinsert +install (at T5): + slotTmp = .tmp. + ln events/.json slotTmp + Files.move(slotTmp, slot, ATOMIC_MOVE, REPLACE_EXISTING) + # Old slot's inode loses one link. If it had a canonical hardlink too, + # drop it — mirror SQLite trigger that deletes the old row. + if oldId present: + runDeleteLinksFor(oldId) # same code path as delete() +``` + +### Reading the current version + +``` +cat replaceable/3/.json # latest kind-3 for pubkey +``` + +Direct path open. No scan, no MAX — the slot is the winner. + +### Edge cases + +- **Race between two newer events.** Both pass guard, both try to + install; last `rename` wins atomically. Losing event's canonical + exists orphaned until scrub or until someone queries by its id. + Semantically valid: both were newer than the previous slot. +- **Slot deleted by user.** Next insert of (kind, pubkey) has no + guard and wins. This matches "files come and go" contract. +- **Canonical deleted but slot survives.** Slot still resolves — + hardlink keeps inode alive. Queries work. Scrub re-creates the + canonical from slot. + +--- + +## NIP-01 — Addressable events (kinds 30000–39999) + +### Enforcement + +- Slot path: `addressable///.json` +- SHA-256 of d-tag value avoids filesystem-unsafe characters and + collisions. Parity with `AddressableModule.kt:25-63`. +- Empty d-tag is valid Nostr — hashed as `sha256("")`. + +### Flow + +Same as replaceable, keyed by (kind, pubkey, d_tag). Guard reads +slot's createdAt; install via `rename REPLACE_EXISTING`; cascade- +unlink old canonical and its index hardlinks. + +### Query shortcut + +If a filter supplies `kinds`, `authors`, and a d-tag, the planner +bypasses the driver indexes and opens the slot directly. O(1). + +--- + +## NIP-09 — Event deletion (kind 5) + +### Deletion event insert + +When a kind-5 event is inserted (inside T6 of the pipeline): + +``` +for target in event.deleteEventIds(): + # NIP-09: same author as the deletion + targetPath = events///.json + if targetPath exists AND read(targetPath).pubkey == event.pubkey: + runDeleteLinksFor(target) # cascade unlink + + # Install tombstone (hardlink to the kind-5 event): + tomb = tombstones/id/ + ln event_canonical .tmp + Files.move(.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING_IF_NEWER) + +for addr in event.deleteAddresses(): + hash = sha256(addr.dTag) + slot = addressable///.json + if slot exists AND read(slot).pubkey == event.pubkey: + Told = read slot.createdAt + if Told <= event.createdAt: + oldId = read slot.id + unlink slot + runDeleteLinksFor(oldId) + + tomb = tombstones/addr/// + ln event_canonical .tmp + Files.move(.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING_IF_NEWER) + +for replaceable_kind in event.deleteReplaceableKinds(): + # kinds 0,3,10000-19999 — no d-tag + slot = replaceable//.json + similar cascade + tombstone +``` + +`REPLACE_EXISTING_IF_NEWER` is a helper we implement: if an older +kind-5 already owns the tombstone, and the new one has a later +createdAt, swap it in; otherwise keep the existing. Ensures +tombstone holds the strongest (latest) deletion cutoff, matching +SQLite's "OR" of all deletion events +(`DeletionRequestModule.kt:102-184`). + +### Blocking re-insertion + +Part of T2 guards: + +``` +TombstoneIdCheck: + if tombstones/id/ exists → REJECT + +TombstoneAddrCheck (addressable only): + if tombstones/addr/// exists: + Tdel = read tombstone.createdAt + if event.createdAt <= Tdel → REJECT +``` + +Parity with `DeletionRequestModule.kt:61-75` (BEFORE INSERT trigger +rejecting events whose etag_hash/atag_hash match a kind-5 entry). + +### Gift-wrap deletion by p-tag + +`EventIndexesModule.kt:161-166` uses `pubkey_owner_hash` that, for +`GiftWrapEvent`, is the recipient's p-tag (not the outer pubkey). +We mirror this: `idx/owner//-` points at every +event where `ownerHash = TagNameValueHasher.hash(recipient)`. + +A kind-5 from the recipient deleting by `e`-tag still resolves via +`runDeleteLinksFor(id)`, which walks all hardlinks (including +`idx/owner/`) and unlinks them. + +### Security: author check + +NIP-09 requires deletions be authored by the same pubkey as the +deleted event. SQLite enforces this via `pubkey_owner_hash` in the +DELETE WHERE clause (`DeletionRequestModule.kt:107-115`). We mirror +by reading the target's JSON and comparing `pubkey` — done inside +`runDeleteLinksFor`. + +--- + +## NIP-40 — Expiration + +### Read at insert + +`event.expiration()` returns `Long?` (tag value as unix seconds). + +``` +if exp != null && exp > 0: + if exp <= now: throw ExpiredEventException # T1 guard, mirrors trigger + ln canonical idx/expires_at/- +``` + +Parity with `ExpirationModule.kt:27-95`. + +### Sweep + +`deleteExpiredEvents()` listed in Part 2. Uses the sorted +`idx/expires_at/` — early termination when `` ≥ now. + +Recommended caller: cron or shell loop. + +```sh +while :; do amy store sweep-expired; sleep 900; done +``` + +Matches the `ExpirationWorker` WorkManager pattern from the SQLite +README. + +### Query-time filtering + +Belt-and-suspenders: queries also filter out events where +`exp != null && exp <= now`. Covers sweep-missed windows and +clock skew between writer and reader. + +--- + +## NIP-62 — Right to vanish (kind 62) + +### Insert + +``` +if !event.shouldVanishFrom(relayUrl): return # same scoping as SQLite + +ownerHash = TagNameValueHasher.hash(event.pubkey) + +# Install vanish tombstone (one per pubkey; latest wins) +tomb = tombstones/vanish/ +existing = read tomb if exists +if existing == null || existing.createdAt < event.createdAt: + ln canonical .tmp + Files.move(.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING) + +# Cascade: delete every event from this owner with createdAt < tomb.createdAt +for entry in idx/owner//: + (ts, id) = parse entry filename + if ts < event.createdAt: + runDeleteLinksFor(id) +``` + +Parity with `RightToVanishModule.kt:46-71`. + +### Blocking future inserts + +Part of T2 guards (VanishCheck): + +``` +tomb = tombstones/vanish/ +if tomb exists: + Tvanish = read tomb.createdAt + if event.createdAt <= Tvanish → REJECT +``` + +For GiftWraps, owner is the recipient — a vanish request from the +recipient cascades their received gift-wraps too, matching SQLite's +behaviour via `pubkey_owner_hash`. + +--- + +## NIP-45 — Count + +`count(filter)` / `count(filters)` runs the exact query plan from +Part 2 and returns the number of surviving entries. Parity with +SQLite `SELECT count(*)`. + +Optimisation: if the filter is driver-only (no post-driver code +filter), we just count directory entries — `Files.list().count()` — +without opening any JSON. Linear in the driver's listing size. + +--- + +## NIP-50 — Full-text search + +### Tokenisation + +Reuse Quartz's `SearchableEvent.indexableContent()` (per +`FullTextSearchModule.kt:66-78`). Tokeniser: + +- Lowercase. +- Split on Unicode word boundaries. +- Filter tokens shorter than 3 chars (FTS5 default). +- Keep ASCII hex ids and bech32 identifiers whole (no splitting on + underscores/digits). + +Match the Android FTS5 tokeniser behaviour; a shared helper lives +in `commons/` and is called by both backends to guarantee identical +search results. + +### Index + +``` +idx/fts// # empty hardlink into canonical +``` + +One empty hardlink per (token, event). For a 1000-word note with +500 unique tokens, 500 hardlinks — each costs one directory entry +(~~50 bytes on ext4), no inode. At scale this is the biggest +directory count; `idx/fts/` alone may hold millions of entries. +Still fine — each token dir stays small. + +### Search query + +``` +tokens = tokenize(filter.search) +streams = tokens.map { idx/fts// } +candidates = k-way intersection of streams by +for candidate in candidates: + if passes all other filter predicates: + emit +``` + +Same semantics as SQLite `MATCH` in FTS5 AND-mode. Matches +`QueryBuilder.kt:590-657`. + +### Maintenance + +- On insert: write token hardlinks (T4). +- On delete (any path — direct, replaceable replace, vanish, + expiration): `runDeleteLinksFor(id)` enumerates every possible + token and unlinks. We don't store the token list, so we re-derive + it by reading the canonical content *before* unlinking the + canonical. If the canonical is already gone, we fall back to + `find -samefile` to enumerate remaining hardlinks. + +### Without an index + +`amy store search --raw "keywords"` can fall back to +`ripgrep events/` for forensic use. Always works, even if indexes +are corrupt. + +--- + +## Tag indexing + +### Which tags get indexed + +Matches `DefaultIndexingStrategy` +(`IndexingStrategy.kt:99-102`): tag arrays with at least 2 elements +where `tag[0].length == 1` (single-letter tag names). Non-single- +letter tags are stored in the JSON and visible to queries but not +reverse-indexed. + +### Hash scheme + +Reuse `TagNameValueHasher` +(`quartz/.../sqlite/TagNameValueHasher.kt:30-67`). Reads `.seed` +into a `Long` salt identically to SQLite's `seeds` table. The hash +becomes a 16-char hex string that names the directory: + +``` +idx/tag///- +``` + +Parity with `event_tags.tag_hash`. d-tag, e-tag, a-tag, p-tag, +owner: see below. + +### Special tags + +| Tag | Where indexed | Notes | +|---|---|---| +| `d` | Only as part of `addressable///.json` slot | Not in `idx/tag/d/` | +| `e` | `idx/tag/e//` | Normal | +| `a` | `idx/tag/a//` + addressable slot look-aside | Normal | +| `p` | `idx/tag/p//` | Recipient for GiftWrap also goes to `idx/owner/` | +| owner (synthetic) | `idx/owner//` | Event pubkey for normal events, recipient for GiftWrap — matches `pubkey_owner_hash` (`EventIndexesModule.kt:161-166`) | + +--- + +## Seed file + +`.seed` is 8 random bytes generated on store creation. Read into a +`Long` salt for `TagNameValueHasher`, matching SQLite's `seeds` table +(`SeedModule.kt:26-84`). + +- Written atomically (tmp-rename) on first open if absent. +- Never updated. Corrupt or mismatched seeds invalidate `idx/` — + scrub rebuilds from canonical. +- Permissions `400`. + +--- + +## Scrub / rebuild + +`FsEventStore.scrub()`: + +``` +1. Lock .lock exclusively. +2. Delete .staging/ leftovers. +3. Enumerate all events///.json. +4. For each, re-derive: + idx/kind/, idx/author/, idx/owner/, idx/tag/*/, + idx/fts/*/, idx/expires_at/ + using the same code as T4. createLink is idempotent — NOACT on + collision (or unlink + recreate). +5. Walk idx/ trees; drop entries whose target inode count is zero + (dangling — canonical was deleted). +6. Rebuild replaceable/ and addressable/ slots from canonicals by + grouping (kind,pubkey[,d]) and picking max createdAt. +7. Keep tombstones as-is. (Removing a tombstone file is a user- + intended action; scrub does not resurrect them.) +8. Release lock. +``` + +Cost: O(N) disk reads once per scrub. Acceptable as a manual +`amy store scrub` command. Not run automatically. + +### `compact()` + +Lighter: walk `idx/` only, drop dangling entries. No canonical +reads. O(size of idx trees). + +--- + +## Testing strategy + +### Location + +`commons/src/jvmTest/kotlin/com/vitorpamplona/commons/store/fs/` + +Uses JUnit4 (matches rest of the project). Temp-dir fixture creates +a fresh store per test. + +### Test categories + +1. **Parity tests.** Drive both `SQLiteEventStore` and `FsEventStore` + with identical event streams and filters; assert query results + match. Existing SQLite tests under + `quartz/src/jvmTest/.../sqlite/` become the golden set. This is + the highest-value tier. + +2. **Per-module tests.** One file per enforcement area: + - `ReplaceableSlotTest` — newer wins, older rejected, race + deposes both cleanly. + - `AddressableSlotTest` — same, plus d-tag hash collisions + cannot occur (SHA-256). + - `DeletionTombstoneTest` — by id, by address, by replaceable + kind; re-insert blocked; gift-wrap by owner. + - `VanishTest` — cascade + block, GiftWrap recipient. + - `ExpirationTest` — sweep, early termination, belt-and- + suspenders query filter. + - `SearchTest` — tokeniser parity with SQLite FTS, AND of tokens, + maintenance across replace/delete/expire/vanish. + - `TagAndTest` — NIP-91 intersection, three-tag filters. + - `EphemeralTest` — never written. + - `TransactionTest` — all-or-nothing batch insert, rollback on + exception. + +3. **Crash-safety tests.** Inject failures after each pipeline step + (T1–T8) via a testing hook; reopen; assert scrub converges. + +4. **Concurrency tests.** Two threads / two processes hitting the + same store: assert no corruption, writes serialise, readers see + consistent snapshots. + +5. **External edit tests.** Mid-run, delete a canonical file / a + slot / a tombstone; assert store behaves as spec'd (dangling + index skipped; slot rebuilds; tombstone stops enforcing). + +6. **Property tests.** For a random sequence of (insert, delete, + query, vanish, expire, replace) operations, assert the file + store and SQLite store produce identical `query()` results. + Seed = test parameter. + +### Running + +``` +./gradlew :commons:jvmTest --tests "*.store.fs.*" +``` + +Target: all tests pass before the CLI starts depending on the new +store. + +--- + +## Rollout plan + +1. **Step 1 — skeleton.** `FsEventStore` class, directory layout, + `insert` + `query(ids only)` + `delete(id)`. Round-trip test. +2. **Step 2 — indexes.** kind/author/owner/tag hardlinks; query by + kind, author, tag; `count`. +3. **Step 3 — slots.** Replaceable + addressable enforcement + + tests. +4. **Step 4 — tombstones.** NIP-09 deletion by id and address; + block re-insert; gift-wrap via owner. +5. **Step 5 — expirations.** `idx/expires_at/`, `deleteExpiredEvents`. +6. **Step 6 — vanish.** NIP-62 insert + cascade + block. +7. **Step 7 — FTS.** Tokeniser in `commons/`, `idx/fts/`, search + queries. +8. **Step 8 — transactions, concurrency, scrub.** +9. **Step 9 — wire into `amy`.** `Context` gains a `store: + IEventStore` property; default = `FsEventStore` rooted at + `$AMY_HOME/events/`. +10. **Step 10 — parity test matrix** driven by the existing SQLite + golden tests. + +Each step is a separate commit. Each is behind `AMY_EXPERIMENTAL_ +STORE=1` env guard until step 10 passes. + +--- + +## Out of scope for this plan + +- Android integration — stays on SQLite. +- Desktop opt-in — separate plan once CLI is stable. +- Cross-host sync — users can rsync the directory; that's it. +- Encryption at rest — future plan; would wrap each event file. + +--- + +## Summary of plan split + +| File | Covers | +|---|---| +| `2026-04-24-file-event-store-overview.md` | Goals, layout, feature matrix, API | +| `2026-04-24-file-event-store-pipelines.md` | Insert, query, delete, transaction, concurrency | +| `2026-04-24-file-event-store-nips.md` (this file) | Per-NIP enforcement, slots, tombstones, FTS, scrub, testing | diff --git a/cli/plans/2026-04-24-file-event-store-overview.md b/cli/plans/2026-04-24-file-event-store-overview.md new file mode 100644 index 000000000..6d05d8dff --- /dev/null +++ b/cli/plans/2026-04-24-file-event-store-overview.md @@ -0,0 +1,260 @@ +# File-backed event store for `amy` — Part 1: Overview & Layout + +**Status:** plan · **Date:** 2026-04-24 · **Part 1 of 3** +(see `2026-04-24-file-event-store-pipelines.md`, +`2026-04-24-file-event-store-nips.md`) + +A filesystem-native `IEventStore` for the `cli/` module that matches +every feature of the SQLite store +(`quartz/.../nip01Core/store/sqlite/`) while tolerating the fact that +files may be created or deleted by the user between runs. + +--- + +## Goals + +1. **Full feature parity** with `SQLiteEventStore` — same `IEventStore` + contract, same NIP semantics (01 replaceable/addressable, 09 + deletion, 40 expiration, 45 count, 50 search, 62 vanish, 91 multi- + tag AND). +2. **Human-inspectable.** Every event is a JSON file on disk. `ls`, + `cat`, `jq`, `grep`, `rsync`, `git`, backup tools — all work. +3. **Tolerates user edits.** If the user deletes an event file, the + store converges: dangling index entries become no-ops, tombstones + stop enforcing, replaceable slots get rebuilt from `events/`. +4. **JVM-only, no JNI.** No SQLite, no native deps beyond what the + CLI already carries. Single-user, single-host. +5. **Small-to-medium scale.** Tens of thousands of events. Not a + full Amethyst cache replacement. + +## Non-goals + +- Beating SQLite on throughput for 100k+ event workloads. +- Network/replication (rsync is good enough). +- Multi-tenant; concurrent writers across machines. +- Incremental migration tooling from SQLite (not needed — `amy` has no + existing persisted store). + +--- + +## Guiding principles + +1. **Filesystem primitives enforce invariants.** Directory-entry + uniqueness = `UNIQUE` constraints. `rename(2)` = atomic commit. + Hardlink refcount = cascade delete. `chmod 444` = immutable + tables. `flock` = transaction serialization. +2. **Hardlinks for all indexes.** An index entry is a second *name* + for the canonical event file, never a copy. Deleting any name + drops the refcount; when it hits zero the kernel reclaims. +3. **Derived state is rebuildable.** `idx/`, `replaceable/`, + `addressable/`, `tombstones/` can be regenerated by walking + `events/`. `amy store scrub` does this. +4. **`events/` is the source of truth.** If a file is there, it is + part of the store. If it is gone, it is gone — tombstones aside. +5. **No business logic.** This is a `commons/` module that the CLI + imports. No Nostr-protocol decisions live in `cli/`. + +--- + +## Module placement + +Source: `commons/src/jvmMain/kotlin/com/vitorpamplona/commons/store/fs/` + +- JVM-only (uses `java.nio.file`, `FileChannel.lock`, `Files.createLink`). +- Consumed by `cli/`. Android keeps `SQLiteEventStore`. Desktop can + opt in later if useful. +- Tests under `commons/src/jvmTest/.../store/fs/`. + +Not in `quartz/` because `quartz/` is protocol-only (no storage). +Not in `cli/` because it contains reusable logic. + +--- + +## Directory layout + +Root: configurable, defaults to `$AMY_HOME/events/` (where `$AMY_HOME` +is `~/.amy` by default). + +``` +/ +├── .lock # flock target for write serialization +├── .seed # random Long for hash salting; write-once +├── .version # schema version for future migrations +│ +├── events///.json # canonical. = first 4 hex of id +│ +├── idx/ +│ ├── kind//- # hardlink → events/.../.json +│ ├── author//- # hardlink +│ ├── owner//- # hardlink; gift-wrap recipient +│ ├── tag///- # hardlink; single-letter tags +│ └── fts// # hardlink; inverted index +│ +├── replaceable//.json # hardlink to current winner +├── addressable///.json # hardlink to current winner +│ +└── tombstones/ + ├── id/ # hardlink to the kind-5 event + ├── addr/// # hardlink to kind-5 + └── vanish/ # hardlink to kind-62 +``` + +### Filename conventions + +- **Event files:** `.json`, where `` is the 64-hex event id. + Content is the raw NIP-01 JSON (same bytes the signer produced). +- **Timestamp prefix:** zero-padded 10-digit unix seconds + (`0001713960000`) so lexicographic sort = chronological sort. +- **Hashes:** SHA-256 of the canonical bytes, hex-lowercased. Used + for d-tags and arbitrary tag values (sanitizes user content for + filesystem safety). +- **Owner hash:** `TagNameValueHasher.hash(ownerPubkey)` (MurmurHash3 + salted by `.seed`). Matches SQLite's `pubkey_owner_hash` semantics + — for GiftWraps it is the recipient's p-tag, not the event pubkey. + (`EventIndexesModule.kt:161-166`) + +### Sharding + +- `events///` = 65 536 leaf directories max. Keeps any leaf + under ~a few hundred files for 10 M events. Piggybacks on ext4 + htree / APFS B-tree / NTFS index. +- `idx/*/…/` no sharding; query paths pick one index tree and walk + its subdirectory directly (`idx/kind/1/`, `idx/author//`). +- Long pubkeys / ids are already 64 hex (32 bytes). No length issues. + +### File modes + +- Event files: `444` (read-only). Matches SQLite's "immutable tables" + invariant: triggers block `UPDATE` + (`EventIndexesModule.kt:105-111`). We achieve the same at the OS + layer. +- Directories: `755`. User can still `rm` (write on parent dir only). +- Tombstones: `444` (they *are* event files via hardlink). + +--- + +## Feature parity matrix + +| SQLite feature | File-store mechanism | Enforced by | +|---|---|---| +| `event_headers` | `events///.json` | Write-then-rename atomicity | +| `event_tags` | `idx/tag///-` hardlinks | Set on insert; orphans tolerated | +| `event_fts` | `idx/fts//` hardlinks | Tokenize `SearchableEvent.indexableContent()` | +| `event_expirations` | `idx/expires_at/-` hardlink | Swept by `deleteExpiredEvents()` | +| `event_vanish` | `tombstones/vanish/` hardlink | Checked on insert | +| `seeds` | `.seed` file; 8 random bytes | Written once on create | +| `UNIQUE(kind,pubkey)` for replaceable | `replaceable//.json` slot | Directory-entry uniqueness + atomic rename | +| `UNIQUE(kind,pubkey,d)` for addressable | `addressable///.json` slot | Same | +| Reject ephemeral | Code guard before any file write | `if (event.kind.isEphemeral()) return` | +| Reject expired on insert | Code guard before any file write | `if (event.isExpired()) throw` | +| NIP-09 delete by id | Unlink `events/.../.json` + all hardlinks | Tombstone blocks re-insert | +| NIP-09 delete by address | Unlink addressable slot + canonical | Tombstone blocks re-insert | +| NIP-09 gift-wrap by p-tag | `idx/owner//` walk | Owner hash matches recipient for GiftWrap | +| NIP-62 vanish cascade | Walk `idx/owner//` and unlink | AFTER-insert cascade | +| NIP-62 block future | Check `tombstones/vanish/` | Before every insert | +| NIP-40 deleteExpired sweep | Walk `idx/expires_at/` until now, unlink | Called by CLI cron | +| NIP-45 count | Same planner as query, count results | Streaming | +| NIP-50 search | Intersect `idx/fts//` sets | Tokenizer matches Quartz | +| NIP-91 tag AND | Sorted-list intersection of `idx/tag///` | Streaming `comm -12` equivalent | +| `transaction {}` | `flock(.lock) + stage + commit + unlock` | All writes to staging dir, single rename | +| `vacuum()` | `amy store scrub` — walks `events/`, rebuilds `idx/` etc. | Offline maintenance | +| `analyse()` | No-op; FS already maintains its own indexes | — | + +--- + +## Public API + +Implements `com.vitorpamplona.quartz.nip01Core.store.IEventStore` +(`quartz/.../store/IEventStore.kt:26-60`) in full. + +```kotlin +class FsEventStore( + root: Path, + relayUrl: String, // matches SQLiteEventStore ctor; used for NIP-62 scoping + indexingStrategy: IndexingStrategy = DefaultIndexingStrategy, + clock: Clock = Clock.systemUTC(), // injectable for tests +) : IEventStore { + + // IEventStore + override fun insert(event: Event) + override fun transaction(body: IEventStore.ITransaction.() -> Unit) + override fun query(filter: Filter): List + override fun query(filters: List): List + override fun query(filter: Filter, onEach: (T) -> Unit) + override fun query(filters: List, onEach: (T) -> Unit) + override fun count(filter: Filter): Int + override fun count(filters: List): Int + override fun delete(filter: Filter) + override fun delete(filters: List) + override fun deleteExpiredEvents() + override fun close() + + // FsEventStore extras (parity with SQLiteEventStore) + fun delete(id: HexKey): Int + fun rawQuery(filter: Filter): List // parses JSON to RawEvent + fun rawQuery(filters: List): List + fun planQuery(filter: Filter): String // returns human-readable plan + fun planQuery(filters: List): String + + // Maintenance + suspend fun scrub() // rebuild all derived state + suspend fun compact() // equivalent to vacuum — drops orphans +} +``` + +Signer/event conveniences mirror SQLite's — no behavioural drift. + +--- + +## File-backed counterparts to SQLite pragmas + +| SQLite pragma | File-store analog | +|---|---| +| `journal_mode=WAL` | Staging dir `.staging//` + atomic rename on commit | +| `synchronous=OFF` | Default = no `fsync` per write; `AMY_FSYNC=1` env opts in | +| `cache_size=-32000` | None — rely on OS page cache | +| `foreign_keys=ON` | Implicit: hardlinks are the FK; refcount is the cascade | +| `BEGIN IMMEDIATE` | `FileChannel.lock(.lock)` exclusive advisory lock | + +--- + +## CLI commands (later; out of scope for this plan) + +Anticipated but not implemented here: + +``` +amy store import # from relay / file / stdin +amy store export [filter] # to stdout NDJSON +amy store query # debug +amy store scrub # rebuild derived state +amy store compact # drop orphans +amy store stat # counts, disk usage +``` + +Added in a follow-up roadmap row once the store itself is in place. + +--- + +## Open questions (flagged for review before coding) + +1. **Tag index granularity.** SQLite hashes `(name, value)` into one + `Long`. We could keep the same hash scheme and use the hex of the + Murmur hash as the directory name, giving byte-for-byte parity + with SQLite's index semantics. Proposed: yes — reuse + `TagNameValueHasher`. +2. **Filename collisions on d-tag.** SHA-256 is safe; Murmur is not. + Use SHA-256 for d-tag slots specifically. +3. **Relay URL in constructor.** `SQLiteEventStore` takes a + `relayUrlIdentifier` for NIP-62 scoping (`shouldVanishFrom`). We + mirror that; same semantics. +4. **Empty tombstone semantics.** If a tombstone hardlink is deleted + by the user, enforcement stops for that target. Scrub does **not** + re-create tombstones from lingering kind-5 events in `events/` — + removing the tombstone is treated as an explicit "un-forget" + action. Confirm with user. + +--- + +Next: pipelines +(`2026-04-24-file-event-store-pipelines.md`) — insert, query, delete, +transaction, concurrency. diff --git a/cli/plans/2026-04-24-file-event-store-pipelines.md b/cli/plans/2026-04-24-file-event-store-pipelines.md new file mode 100644 index 000000000..13190cf7f --- /dev/null +++ b/cli/plans/2026-04-24-file-event-store-pipelines.md @@ -0,0 +1,326 @@ +# File-backed event store for `amy` — Part 2: Pipelines + +**Status:** plan · **Date:** 2026-04-24 · **Part 2 of 3** +(see `2026-04-24-file-event-store-overview.md`, +`2026-04-24-file-event-store-nips.md`) + +Covers insert, query, delete, transaction, and concurrency. NIP- +specific enforcement (09/40/50/62) and replaceable/addressable slot +mechanics are in Part 3. + +--- + +## Insert pipeline + +Mirrors `SQLiteEventStore.insertEvent()` +(`quartz/.../sqlite/SQLiteEventStore.kt:189-196`) and +`innerInsertEvent()` (lines 178-187). Module ordering is identical +— only the backing representation changes. + +### Pre-transaction guards (no files touched) + +``` +1. if (event.kind.isEphemeral()) return // EphemeralModule +2. if (event.isExpired()) throw ExpiredEventException // ExpirationModule +``` + +Both match SQLite behaviour exactly — rejected before any I/O. + +### Transaction body + +Acquire `FileChannel.lock(.lock)` for the whole body. Release only +on COMMIT or ROLLBACK. + +``` +T1. Stage event file + stagingDir = .staging// + write stagingDir/event.json (event bytes, fsync if AMY_FSYNC=1) + +T2. Pre-insert veto checks (read-only) + • TombstoneIdCheck: tombstones/id/ exists? → REJECT + • TombstoneAddrCheck: (addressable only) tombstones/addr/// + exists AND Tdel >= event.createdAt? → REJECT + • VanishCheck: tombstones/vanish/ exists AND + Tvanish >= event.createdAt? → REJECT + • ReplaceableGuard: (kinds 0,3,10000-19999) replaceable//.json + exists AND its createdAt >= event.createdAt? → REJECT + • AddressableGuard: (kinds 30000-39999) addressable///.json + exists AND its createdAt >= event.createdAt? → REJECT + Any REJECT → discard staging dir, release lock, return. + +T3. Canonical write + canonical = events///.json + Files.move(stagingDir/event.json, canonical, ATOMIC_MOVE) + Files.setLastModifiedTime(canonical, event.createdAt seconds) + Files.setPosixFilePermissions(canonical, r--r--r--) + +T4. Derived-index hardlinks (all ln(canonical → X)) + • idx/kind//- + • idx/author//- + • idx/owner//- # recipient for GiftWrap + • for each indexed tag (name, value): + idx/tag///- + • if event.expiration() > 0: + idx/expires_at/- + • if event implements SearchableEvent: + for each token in tokenize(event.indexableContent()): + idx/fts// + +T5. Replaceable / Addressable slot (if applicable) + slotTmp = .tmp. + ln canonical slotTmp + Files.move(slotTmp, slot, ATOMIC_MOVE, REPLACE_EXISTING) + # The rename atomically deposes the old winner and installs new. + if oldSlot had different event id: + unlink events/.json # drop canonical + walk idx/... and unlink entries for old id # reuses same logic as delete() + +T6. Deletion-event side effects (if event.kind == 5) + Apply NIP-09 actions — see Part 3. Executes *inside* the lock. + +T7. Vanish-event side effects (if event.kind == 62 && shouldVanishFrom(relay)) + Apply NIP-62 cascade — see Part 3. + +T8. Commit + Release .lock. Remove emptied staging dir. +``` + +### Crash safety at every step + +| Crash between | Post-crash state | Scrub action | +|---|---|---| +| T1 and T3 | Only staging dir exists | Delete `.staging/` on next open | +| T3 and T4 | Canonical written, no indexes | Rebuild indexes by walking `events/` | +| T4 (partial) | Some indexes written | Rebuild indexes (idempotent) | +| T5 (partial) | Slot tmp hardlink exists | Remove `.tmp.*` on next open | +| T5 (after rename) | Old winner still has canonical | Slot takes precedence; old canonical GC'd by scrub | +| T6/T7 (partial) | Some targets unlinked, some not | Re-apply kind-5/62 during scrub | + +The kernel guarantees `rename(2)` atomicity; `link(2)` either +succeeds fully or fails. All failure modes are forward-recoverable +by scrub — never corrupting. + +### Ordering vs SQLite + +SQLite runs triggers *before* our code inserts. We run checks +before writing the canonical (T2), then indexes (T4), then slot +enforcement (T5), then kind-5 / kind-62 side effects (T6/T7). Same +overall guarantees, different sequencing because we have no trigger +mechanism. + +--- + +## Transaction API + +```kotlin +override fun transaction(body: ITransaction.() -> Unit) { + withLock { + val txn = FsTransaction(this) + try { + txn.body() // accumulates staged events + txn.commit() // all T3..T7 for every staged event, then one unlock + } catch (e: Throwable) { + txn.rollback() // delete staging dirs, do nothing else + throw e + } + } +} +``` + +Semantics: + +- Single `flock` for the whole transaction. +- Staging dirs collected; canonical moves happen at commit time. +- If any event in the batch is rejected by T2, the whole txn aborts. +- Matches SQLite's all-or-nothing behaviour + (`SQLiteConnectionExt.kt:71-81`). + +No readers are blocked: they don't hold the lock and see either +pre-commit or post-commit state (atomic rename). + +--- + +## Query pipeline + +Matches the planner structure of `QueryBuilder` +(`quartz/.../sqlite/QueryBuilder.kt:110-172`) but emits directory +walks instead of SQL. + +### Filter normalisation + +Same as SQLite: `filter.toFilterWithDTags()` splits the d-tag out of +the generic tag map, flags `isSimpleQuery` / `isSimpleSearch`. + +### Plan selection + +For each filter, choose one driver index based on cardinality and +availability: + +| Filter contents | Driver index | Rationale | +|---|---|---| +| Only `ids` | direct `events///.json` opens | O(1) per id | +| `kinds` present, no tags | `idx/kind//` | Smallest per-kind fanout | +| `authors` present, no kinds, no tags | `idx/author//` | Author-scoped | +| `kinds` + `authors` + no tags | Pick the smaller of `idx/kind/` vs `idx/author/` by listing size | Parity with SQLite's `query_by_kind_pubkey_created` choice | +| Any `#tag` filter | `idx/tag///` — intersect multiple if AND | Matches SQLite's per-tag INNER JOINs | +| Only `search` | `idx/fts//` — intersect tokens | Parity with FTS MATCH | +| Only `since`/`until` / `limit` | `idx/kind/*/` or full `events/` walk ordered by mtime | See below | +| `d_tag` present (addressable) | `addressable///.json` direct open | Slot lookup — O(1) | + +Chosen driver produces a **sorted stream of `(createdAt, id)` pairs** +in DESC order (lexicographic sort over zero-padded filenames reversed). +Filenames are `-`, so `Files.list(dir).sorted(reverseOrder())` +is the plan. + +### Secondary filters (post-driver) + +Anything not covered by the driver is applied in code, event by +event, after opening the JSON: + +- `since <= createdAt <= until` (already a prefix filter on the driver) +- `authors` / `kinds` / `ids` if not the driver +- d-tag match (addressable) +- additional tag filters +- search tokens (if driver wasn't FTS) +- tombstone exclusions (id + address) +- vanish exclusions (by owner_hash) +- expiration exclusions (`createdAt < now < expiration`) + +### Tag AND (NIP-91) + +For `filter.tagsAll = {#e = [a, b], #p = [c]}`: + +``` +streamA = idx/tag/e// (sorted by - DESC) +streamB = idx/tag/e// +streamC = idx/tag/p// +result = k-way intersection by suffix, preserving DESC order +``` + +Implemented as a sorted-list intersection (Java `Iterator` +merge). Equivalent to SQLite's INNER JOIN per tag +(`QueryBuilder.kt:434-449`). Bounded memory. + +### Multi-filter UNION + +`query(filters: List)` runs each filter's plan, then merges +the resulting streams in DESC order with dedup by id. Per-filter +`limit` applied before merge, as SQLite does +(`QueryBuilder.kt:577-583`). + +### Streaming mode + +`query(filter, onEach: (T) -> Unit)` calls `onEach` as each event is +read and parsed. No intermediate list. Matches SQLite's cursor +callback. + +### `count()` + +Same plan, but open nothing — just count filenames that survive the +post-driver code filters. For filters that require JSON inspection +(tag exact match on non-indexed tags, search tokens), we still open +files; cost scales with matched set size, not store size. + +### `planQuery()` output + +Plain text, like `EXPLAIN QUERY PLAN`: + +``` +FILTER 0 + DRIVER: idx/kind/1/ (12,400 entries) + POST: authors=[], since >= 1713960000, limit 100 +FILTER 1 + DRIVER: idx/tag/e/ab12…/ ∩ idx/tag/p/cd34…/ (est 37 hits) + POST: kinds=[30023], limit 10 +MERGE: 2 filters, DESC by created_at, overall limit 110 +``` + +--- + +## Delete pipeline + +### `delete(filter)` / `delete(filters)` + +Runs the query pipeline to enumerate matching events, then for each: + +``` +1. Read canonical, parse tags (needed to know all hardlink paths). +2. Unlink every derived hardlink: + idx/kind//- + idx/author//- + idx/owner//- + idx/tag///- (for each indexed tag) + idx/expires_at/- (if exp present) + idx/fts// (for each FTS token) +3. If event is the winner of a replaceable/addressable slot: + unlink the slot path. +4. Unlink canonical events///.json. +5. kernel GC's the inode (no other refs). +``` + +All inside `flock`. Matches SQLite's +`DELETE FROM event_headers WHERE row_id IN (...)` with FK cascade. + +### `delete(id: HexKey): Int` + +Direct shortcut. Opens `events///.json`, runs the same +steps 1-5. Returns `1` on success, `0` if the file didn't exist — +matching the SQLite return value from `changes()` +(`SQLiteEventStore.kt:261-264`). + +### `deleteExpiredEvents()` + +``` +now = clock.now().epochSecond +for entry in idx/expires_at/ where < now: + delete(entry.id) +``` + +Optional optimisation: because filenames are `-`, listing is +already sorted, so we can stop at the first entry with ` >= now`. + +--- + +## Concurrency model + +### Single writer, many readers + +- Writers hold an exclusive `flock(.lock)` via `FileChannel.tryLock()`. +- Readers take a shared lock only while listing top-level directory + contents that may be mid-rename; most reads hold no lock (kernel- + atomic `open` / `readdir` is enough). +- Two concurrent `amy` invocations serialise cleanly on the flock; + the second one blocks on `FileChannel.lock()` (blocking) or fails + fast with `tryLock()` (configurable). + +### Reader–writer interaction + +Because every writer mutation is a rename-over-tmp, readers see +either the pre-mutation file or the post-mutation file, never a +torn view. Dangling reads (writer unlinks a file between +`readdir` and `open`) manifest as `NoSuchFileException` and are +treated as "event not in store anymore" — we skip it and keep going. + +### Watch / subscribe (future) + +`java.nio.file.WatchService` on `events/` gives per-platform +inotify/FSEvents/ReadDirectoryChangesW. Lets a future +`amy store tail` stream newly arriving events to stdout without +polling. Not part of this plan but the layout supports it +natively. + +--- + +## Close / reopen + +- `close()` releases any held `FileChannel` and flushes nothing + (writes are already durable as of each rename, modulo the + `AMY_FSYNC` option). +- Reopen is free: the store is stateless. No header page, no WAL + checkpoint. First action after open: scan `.staging/` and delete + leftovers (crash from previous run). + +--- + +Next: NIP specifics and enforcement details +(`2026-04-24-file-event-store-nips.md`).