From d4d2fa46767d3744aed520726e602535f039cb62 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Apr 2026 02:30:09 +0000 Subject: [PATCH] docs(quartz): README for the file-backed event store Sibling to the SQLite store's README. Covers: - what the store is and why (filesystem primitives = invariants: directory-entry uniqueness IS the UNIQUE constraint, rename(2) IS the atomic commit, hardlink refcount IS the FK cascade, flock(2) IS the writer serialisation), - feature-parity table vs SQLite (NIP-01 replaceable / addressable, NIP-09 deletion + tombstones, NIP-40 expiration, NIP-45 count, NIP-50 search, NIP-62 vanish, NIP-91 multi-tag AND), - on-disk layout with every directory tree, - filename conventions (sharding, padded-ts entry names, sha256 vs Murmur, mtime), - internal architecture (FsLayout / FsLockManager / FsIndexer / FsSlots / FsTombstones / FsQueryPlanner / FsSearchTokenizer), - usage examples for insert / query / count / delete / transaction / expiration sweep / scrub / compact / close, - failure-mode table (external edits, crashes, multi-process, full disk) and how the store converges, - pointer to the 113-test suite under jvmTest and the design plan documents under cli/plans/. --- .../quartz/nip01Core/store/fs/README.md | 309 ++++++++++++++++++ 1 file changed, 309 insertions(+) create mode 100644 quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/README.md diff --git a/quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/README.md b/quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/README.md new file mode 100644 index 000000000..9ecede0a8 --- /dev/null +++ b/quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/README.md @@ -0,0 +1,309 @@ +# File-Backed Event Store (`fs`) + +A pure-filesystem `IEventStore` implementation. Each Nostr event is +written as a single JSON file under a sharded directory tree; +hardlinks form every secondary index, slot, and tombstone. No +database engine, no JNI, no schema migration — just `events/`, +`idx/`, `replaceable/`, `addressable/`, `tombstones/`. + +JVM-only (uses `java.nio.file.Files.createLink` and +`FileChannel.lock`). For Android the SQLite store at +[`../sqlite`](../sqlite/README.md) is still the right choice; for the +JVM CLI (`amy`) and for human-inspectable scratch stores, this +implementation is the default. + +## Why a filesystem store? + +| Goal | Outcome | +|---|---| +| Human-inspectable | Every event is a JSON file. `ls`, `cat`, `jq`, `grep`, `find`, `rsync`, `git` all work directly. | +| Tolerant of external edits | Delete an event file by hand → store converges. Dangling `idx/` entries are skipped at read time and cleaned up by `compact()`. | +| No JNI / no JDBC | Runs on any JVM. The code is `expect`/`actual`-friendly so it could be ported to native. | +| Same `IEventStore` contract | Drop-in replacement for `EventStore` (SQLite). Same insert / query / delete / count semantics, same NIP support. | +| Filesystem primitives = invariants | Directory-entry uniqueness IS the `UNIQUE(kind,pubkey[,d])` constraint. `rename(2)` IS the atomic commit. Hardlink refcount IS the FK cascade. `flock(2)` IS the writer serialisation. | + +The trade-off vs. SQLite: filtered queries are O(*directory listing +size*) instead of O(*log n*) and there's no cost-based query planner. +For tens-of-thousands of events on a single host this is fine; for a +full Amethyst-scale cache (hundreds of thousands of events with hot +queries) SQLite still wins. + +## Feature parity with the SQLite store + +Everything documented in [`../sqlite/README.md`](../sqlite/README.md): + +| SQLite feature | This store | +|---|---| +| Insert + retrieve by Nostr filter | ✓ | +| Replaceable events (kinds 0, 3, 10000-19999) — newer wins, older blocked | ✓ via `replaceable//.json` slot | +| Addressable events (kinds 30000-39999) — `(kind,pubkey,d)` uniqueness | ✓ via `addressable///.json` slot | +| Ephemeral events never stored | ✓ rejected pre-write | +| NIP-09 deletions — by id, by address, gift-wrap by `p`-tag, blocks re-insert | ✓ via `tombstones/id/` and `tombstones/addr/` (each tombstone is a hardlink to the kind-5 event) | +| NIP-40 expirations — reject expired-on-insert, periodic sweep | ✓ via `idx/expires_at/` index + `deleteExpiredEvents()` | +| NIP-45 counts | ✓ same planner, just count results | +| NIP-50 full-text search — content tokenisation, AND of tokens | ✓ via `idx/fts//` | +| NIP-91 multi-tag AND | ✓ k-way intersection of `idx/tag///` listings | +| NIP-62 right-to-vanish — cascade + block-future-insert | ✓ via `tombstones/vanish/` (latest cutoff wins) | +| Immutable canonical (no UPDATE) | ✓ event files are write-once; only indexes/tombstones mutate | +| Transactions | ✓ `transaction { … }` holds the flock for the whole batch | + +A parity test +([`FsParityTest`](../../../../../../jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/FsParityTest.kt)) +drives both this store and the SQLite reference with identical event +streams and asserts query results agree. + +## On-disk layout + +``` +/ +├── .lock # flock target (cross-process serialisation) +├── .seed # 8 random bytes; salts every Murmur hash +├── .staging/ # tmp files awaiting atomic rename +│ +├── events///.json # canonical event; mtime = event.createdAt +│ +├── idx/ +│ ├── kind//- # hardlink → events/.../.json +│ ├── author//- # hardlink +│ ├── owner//- # hardlink; recipient for GiftWrap +│ ├── tag///- # hardlink; single-letter tag names only +│ ├── expires_at/- # hardlink; for NIP-40 sweep +│ └── fts//- # hardlink; for NIP-50 search +│ +├── replaceable//.json # hardlink to current winner (kinds 0,3,10000-19999) +├── addressable///.json # hardlink to current winner (kinds 30000-39999) +│ +└── tombstones/ + ├── id/.json # hardlink to the kind-5 event that deleted + ├── addr///.json # hardlink to the kind-5 event for an address + └── vanish/.json # hardlink to the kind-62 event for an owner +``` + +### Filename conventions + +- **Event files**: `.json`, raw NIP-01 event JSON. +- **Sharding**: first 4 hex chars of the event id give the path + `//`, capping any one leaf directory at a few hundred + entries even at million-event scale and piggybacking on ext4 htree + / APFS B-tree / NTFS index for O(log n) lookup. +- **Index entry filenames**: `-` where `` + is the event's `created_at` left-padded to 10 digits. Lexicographic + sort of a directory listing == chronological sort. +- **Hashes**: + - `sha256Hex(d-tag)` — collision-resistant, used wherever the path + IS the uniqueness constraint (addressable slots, address tombstones). + - `hashHex(murmurLong)` — 16 hex chars, used for tag-value indexes + where occasional collisions are caught by the post-filter. +- **mtime**: every canonical write does + `Files.setLastModifiedTime(canonical, FileTime.from(event.createdAt, SECONDS))` + so `ls -t`, `find -newermt`, `rsync -a` all behave naturally. + +## Architecture + +``` +FsEventStore ← orchestrator; implements IEventStore +├── FsLayout ← path helpers + .seed file +├── FsLockManager ← flock(.lock) with per-thread reentry +├── FsIndexer ← link / unlink the idx/ + fts hardlinks for an event +├── FsSlots ← install / evict replaceable + addressable slots +├── FsTombstones ← NIP-09 id / addr + NIP-62 vanish tombstones +├── FsQueryPlanner ← pick driver index, stream candidates DESC by createdAt +└── FsSearchTokenizer ← lowercase + Unicode-letter-or-digit split (≈ FTS5 unicode61) +``` + +Every mutating call (`insert`, `delete`, `transaction`, +`deleteExpiredEvents`, `scrub`, `compact`) acquires the flock once; +nested calls on the same thread re-enter for free. Reads take no +lock — atomic-rename writes mean readers see either the pre- or +post-mutation state, and `NoSuchFileException` on a just-unlinked +candidate is silently skipped. + +## Usage + +### Initialisation + +```kotlin +import com.vitorpamplona.quartz.nip01Core.store.fs.FsEventStore +import java.nio.file.Path + +val store = FsEventStore( + root = Path.of(System.getProperty("user.home"), ".amy", "events-store"), + // optional NIP-62 relay scoping; null = only ALL_RELAYS vanish requests cascade + relay = null, + // optional clock for tests + clock = { com.vitorpamplona.quartz.utils.TimeUtils.now() }, +) +``` + +The directory is created on first use. The `.seed` file (8 random +bytes) is generated atomically on first open and read on every +subsequent open — it salts every tag / owner hash, so it must persist +or every previously-written index entry becomes unreachable. + +### Insert + +```kotlin +store.insert(event) +``` + +Pre-checks (in order, parity with the SQLite triggers): + +1. **Ephemeral kinds** (20000-29999) — silently dropped. +2. **NIP-40 expiration** — rejected if `event.expiration() <= now`. +3. **NIP-09 / NIP-62 tombstones** — id-tombstone always blocks; + address-tombstone blocks if `event.createdAt <= cutoff`; vanish + tombstone blocks if `event.createdAt <= vanish.createdAt`. +4. **Replaceable / addressable slot** — blocks if existing slot's + `createdAt >= event.createdAt`. + +Then writes canonical via `events/.json.tmp` → atomic rename, +links every `idx/` hardlink, installs the slot atomically (evicting +the old winner's canonical + indexes), and applies any NIP-09 / NIP-62 +side effects if the event is a kind-5 / kind-62. + +### Query + +```kotlin +val notes = store.query( + Filter(authors = listOf(pubKey), kinds = listOf(1), limit = 50), +) + +// Streaming variant (no intermediate List) +store.query(filter) { event -> + println(event.content) +} + +// Multi-filter union (deduped by id) +store.query(listOf(filterA, filterB)) +``` + +Driver selection: + +| Filter contents | Driver | +|---|---| +| `ids` | direct canonical opens (mtime gives ordering) | +| `search` | `idx/fts//` intersection | +| `tags`/`tagsAll` | `idx/tag///` (first key wins; rest are post-filter) | +| `kinds` | union of `idx/kind//` | +| `authors` | union of `idx/author//` | +| nothing | full scan via every `idx/kind//` | + +Whatever the driver picks, the orchestrator opens each candidate's +JSON and runs `Filter.match(event)` — so any driver is +correctness-safe; choice only affects how much we scan. + +### Count + +```kotlin +val total = store.count(filter) +``` + +Same plan as `query`, just doesn't materialise events. + +### Delete + +```kotlin +store.delete(eventId) // by id; returns 1 if removed, 0 otherwise +store.delete(filter) // delete all matching +store.delete(listOf(filterA, ...)) // delete union +``` + +Unlinks every index hardlink, clears the slot if this event was the +current winner, then unlinks the canonical. + +### Transactions + +```kotlin +store.transaction { + insert(event1) + insert(event2) + insert(event3) +} +``` + +Holds the flock for the whole body. Inside the body each `insert()` +re-enters the lock for free. Exceptions propagate; events written +before the exception are kept (atomic-per-event, serialised across +writers — not all-or-nothing, since canonical files are individually +atomic). + +### NIP-40 expiration sweep + +```kotlin +store.deleteExpiredEvents() +``` + +Walks `idx/expires_at/`, deletes everything with `exp < now`. Run on +a timer (cron, scheduled coroutine) — every 15 minutes is a sane +default; matches the SQLite store's recommendation. + +### Maintenance + +```kotlin +store.scrub() // rebuild every idx/ entry from the canonical events +store.compact() // drop dangling idx/ entries (where canonical is gone) +``` + +`scrub()` recovers from external edits or partial-write crashes; +slots and tombstones are left alone (a tombstone removed by hand is +treated as deliberate "un-forget"). `compact()` is cheap — never +opens an event JSON, only stats canonicals. + +### Close + +```kotlin +store.close() +``` + +Releases the flock channel. Reopen is free — the store is stateless +on disk; first action after open is sweeping leftover `.staging/` files. + +## Failure modes + +The store is engineered to converge under the kinds of damage a +filesystem-backed cache will actually see: + +| What happened | Effect | +|---|---| +| User deleted an event JSON | Dangling `idx/` entries are skipped at read time; `compact()` cleans them up. Slot- or tombstone-pinned data is preserved via the hardlink. | +| User deleted a slot file | Next insert at that `(kind, pubkey[, d])` wins unopposed. | +| User deleted a tombstone | Enforcement stops for that target — treated as deliberate. | +| Crash mid-insert (canonical written, indexes not) | `scrub()` rebuilds the missing `idx/` entries. | +| Crash mid-rename (.staging tmp leftover) | Cleared on next open. | +| Two `amy` processes write at once | Cross-process flock serialises them. | +| Two threads in one process write at once | Same flock plus per-thread re-entry counter. | +| Disk fills up | Insert throws; the store is consistent at the canonical it wrote (or didn't). Replaceable slot replacement is atomic via `rename(2)`. | + +## Testing + +Tests live under +[`quartz/src/jvmTest/kotlin/.../store/fs/`](../../../../../../jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/): + +| File | Coverage | +|---|---| +| `FsEventStoreTest` | round-trip, sharding, ephemeral, staging cleanup | +| `FsQueryTest` | author / kind / tag drivers, tag OR / tagsAll AND, since/until, limit, count, reopen | +| `FsSlotsTest` | replaceable + addressable winner / eviction / canonical-survives-via-slot | +| `FsDeletionTest` | NIP-09 by id / address, block re-insert, cutoff semantics, non-author no-cascade | +| `FsExpirationTest` | NIP-40 future / past / equal-now / sweep / non-positive | +| `FsVanishTest` | NIP-62 cascade / block / strongest-cutoff-wins / per-relay scoping | +| `FsSearchTest` | tokenizer behaviour, single-token / AND-of-tokens, ordering, reopen | +| `FsMaintenanceTest` | flock, transaction commit + propagated exceptions, re-entrant lock, scrub, compact, two-thread concurrency | +| `FsParityTest` | drive both this store and SQLite with identical streams and assert results match | + +```bash +./gradlew :quartz:jvmTest --tests "com.vitorpamplona.quartz.nip01Core.store.fs.*" +``` + +113 tests, ~5 s on a laptop. + +## Design notes & references + +- Plan documents live in + [`cli/plans/2026-04-24-file-event-store-*.md`](../../../../../../../../cli/plans/) + (overview, pipelines, NIP-by-NIP). +- The CLI integration that makes this the source of truth for `amy` + is documented under "Local event store — the source of truth" in + [`cli/README.md`](../../../../../../../../cli/README.md). +- The SQLite reference implementation that this store achieves + parity with: [`../sqlite/README.md`](../sqlite/README.md).