# File-backed event store for `amy` — Part 1: Overview & Layout **Status:** plan · **Date:** 2026-04-24 · **Part 1 of 3** (see `2026-04-24-file-event-store-pipelines.md`, `2026-04-24-file-event-store-nips.md`) A filesystem-native `IEventStore` for the `cli/` module that matches every feature of the SQLite store (`quartz/.../nip01Core/store/sqlite/`) while tolerating the fact that files may be created or deleted by the user between runs. --- ## Goals 1. **Full feature parity** with `SQLiteEventStore` — same `IEventStore` contract, same NIP semantics (01 replaceable/addressable, 09 deletion, 40 expiration, 45 count, 50 search, 62 vanish, 91 multi- tag AND). 2. **Human-inspectable.** Every event is a JSON file on disk. `ls`, `cat`, `jq`, `grep`, `rsync`, `git`, backup tools — all work. 3. **Tolerates user edits.** If the user deletes an event file, the store converges: dangling index entries become no-ops, tombstones stop enforcing, replaceable slots get rebuilt from `events/`. 4. **JVM-only, no JNI.** No SQLite, no native deps beyond what the CLI already carries. Single-user, single-host. 5. **Small-to-medium scale.** Tens of thousands of events. Not a full Amethyst cache replacement. ## Non-goals - Beating SQLite on throughput for 100k+ event workloads. - Network/replication (rsync is good enough). - Multi-tenant; concurrent writers across machines. - Incremental migration tooling from SQLite (not needed — `amy` has no existing persisted store). --- ## Guiding principles 1. **Filesystem primitives enforce invariants.** Directory-entry uniqueness = `UNIQUE` constraints. `rename(2)` = atomic commit. Hardlink refcount = cascade delete. `chmod 444` = immutable tables. `flock` = transaction serialization. 2. **Hardlinks for all indexes.** An index entry is a second *name* for the canonical event file, never a copy. Deleting any name drops the refcount; when it hits zero the kernel reclaims. 3. **Derived state is rebuildable.** `idx/`, `replaceable/`, `addressable/`, `tombstones/` can be regenerated by walking `events/`. `amy store scrub` does this. 4. **`events/` is the source of truth.** If a file is there, it is part of the store. If it is gone, it is gone — tombstones aside. 5. **No business logic.** Lives next to the SQLite reference store in `quartz/`, sibling pattern. No Nostr-protocol decisions live in `cli/`. --- ## Module placement Source: `quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/` - Sibling to the existing `quartz/src/commonMain/kotlin/.../nip01Core/store/sqlite/` reference implementation. The plan originally proposed `commons/` but the shipped placement is `quartz/jvmMain/`, where every other event- store concern already lives — and quartz already has a `jvmTest` source set so we get JVM-specific tests for free. - JVM-only (uses `java.nio.file`, `FileChannel.lock`, `Files.createLink`). - Consumed by `cli/`. Android keeps `SQLiteEventStore`. Desktop can opt in later if useful. - Tests under `quartz/src/jvmTest/.../store/fs/`. --- ## Directory layout Root: configurable. The CLI uses `/events-store/` (where `` is `--data-dir PATH`, `$AMETHYST_CLI_DATA`, or `./amy`). The plan originally said `events/`; the actual `DataDir.eventsDir` field is `events-store/` to leave the bare name `events/` available for the canonical-events subdirectory inside the store. ``` / ├── .lock # flock target for write serialization ├── .seed # random Long for hash salting; write-once ├── .version # schema version for future migrations │ ├── events///.json # canonical. = first 4 hex of id │ ├── idx/ │ ├── kind//- # hardlink → events/.../.json │ ├── author//- # hardlink │ ├── owner//- # hardlink; gift-wrap recipient │ ├── tag///- # hardlink; single-letter tags │ └── fts// # hardlink; inverted index │ ├── replaceable//.json # hardlink to current winner ├── addressable///.json # hardlink to current winner │ └── tombstones/ ├── id/ # hardlink to the kind-5 event ├── addr/// # hardlink to kind-5 └── vanish/ # hardlink to kind-62 ``` ### Filename conventions - **Event files:** `.json`, where `` is the 64-hex event id. Content is the raw NIP-01 JSON (same bytes the signer produced). - **Timestamp prefix:** zero-padded 10-digit unix seconds (`0001713960000`) so lexicographic sort = chronological sort. - **Hashes:** SHA-256 of the canonical bytes, hex-lowercased. Used for d-tags and arbitrary tag values (sanitizes user content for filesystem safety). - **Owner hash:** `TagNameValueHasher.hash(ownerPubkey)` (MurmurHash3 salted by `.seed`). Matches SQLite's `pubkey_owner_hash` semantics — for GiftWraps it is the recipient's p-tag, not the event pubkey. (`EventIndexesModule.kt:161-166`) ### Sharding - `events///` = 65 536 leaf directories max. Keeps any leaf under ~a few hundred files for 10 M events. Piggybacks on ext4 htree / APFS B-tree / NTFS index. - `idx/*/…/` no sharding; query paths pick one index tree and walk its subdirectory directly (`idx/kind/1/`, `idx/author//`). - Long pubkeys / ids are already 64 hex (32 bytes). No length issues. ### File modes - Event files: `444` (read-only). Matches SQLite's "immutable tables" invariant: triggers block `UPDATE` (`EventIndexesModule.kt:105-111`). We achieve the same at the OS layer. - Directories: `755`. User can still `rm` (write on parent dir only). - Tombstones: `444` (they *are* event files via hardlink). --- ## Feature parity matrix | SQLite feature | File-store mechanism | Enforced by | |---|---|---| | `event_headers` | `events///.json` | Write-then-rename atomicity | | `event_tags` | `idx/tag///-` hardlinks | Set on insert; orphans tolerated | | `event_fts` | `idx/fts//` hardlinks | Tokenize `SearchableEvent.indexableContent()` | | `event_expirations` | `idx/expires_at/-` hardlink | Swept by `deleteExpiredEvents()` | | `event_vanish` | `tombstones/vanish/` hardlink | Checked on insert | | `seeds` | `.seed` file; 8 random bytes | Written once on create | | `UNIQUE(kind,pubkey)` for replaceable | `replaceable//.json` slot | Directory-entry uniqueness + atomic rename | | `UNIQUE(kind,pubkey,d)` for addressable | `addressable///.json` slot | Same | | Reject ephemeral | Code guard before any file write | `if (event.kind.isEphemeral()) return` | | Reject expired on insert | Code guard before any file write | `if (event.isExpired()) throw` | | NIP-09 delete by id | Unlink `events/.../.json` + all hardlinks | Tombstone blocks re-insert | | NIP-09 delete by address | Unlink addressable slot + canonical | Tombstone blocks re-insert | | NIP-09 gift-wrap by p-tag | `idx/owner//` walk | Owner hash matches recipient for GiftWrap | | NIP-62 vanish cascade | Walk `idx/owner//` and unlink | AFTER-insert cascade | | NIP-62 block future | Check `tombstones/vanish/` | Before every insert | | NIP-40 deleteExpired sweep | Walk `idx/expires_at/` until now, unlink | Called by CLI cron | | NIP-45 count | Same planner as query, count results | Streaming | | NIP-50 search | Intersect `idx/fts//` sets | Tokenizer matches Quartz | | NIP-91 tag AND | Sorted-list intersection of `idx/tag///` | Streaming `comm -12` equivalent | | `transaction {}` | `flock(.lock) + stage + commit + unlock` | All writes to staging dir, single rename | | `vacuum()` | `amy store scrub` — walks `events/`, rebuilds `idx/` etc. | Offline maintenance | | `analyse()` | No-op; FS already maintains its own indexes | — | --- ## Public API Implements `com.vitorpamplona.quartz.nip01Core.store.IEventStore` (`quartz/.../store/IEventStore.kt:26-60`) in full. ```kotlin class FsEventStore( root: Path, relayUrl: String, // matches SQLiteEventStore ctor; used for NIP-62 scoping indexingStrategy: IndexingStrategy = DefaultIndexingStrategy, clock: Clock = Clock.systemUTC(), // injectable for tests ) : IEventStore { // IEventStore override fun insert(event: Event) override fun transaction(body: IEventStore.ITransaction.() -> Unit) override fun query(filter: Filter): List override fun query(filters: List): List override fun query(filter: Filter, onEach: (T) -> Unit) override fun query(filters: List, onEach: (T) -> Unit) override fun count(filter: Filter): Int override fun count(filters: List): Int override fun delete(filter: Filter) override fun delete(filters: List) override fun deleteExpiredEvents() override fun close() // FsEventStore extras (parity with SQLiteEventStore) fun delete(id: HexKey): Int fun rawQuery(filter: Filter): List // parses JSON to RawEvent fun rawQuery(filters: List): List fun planQuery(filter: Filter): String // returns human-readable plan fun planQuery(filters: List): String // Maintenance suspend fun scrub() // rebuild all derived state suspend fun compact() // equivalent to vacuum — drops orphans } ``` Signer/event conveniences mirror SQLite's — no behavioural drift. --- ## File-backed counterparts to SQLite pragmas | SQLite pragma | File-store analog | |---|---| | `journal_mode=WAL` | Staging dir `.staging//` + atomic rename on commit | | `synchronous=OFF` | Default = no `fsync` per write; `AMY_FSYNC=1` env opts in | | `cache_size=-32000` | None — rely on OS page cache | | `foreign_keys=ON` | Implicit: hardlinks are the FK; refcount is the cascade | | `BEGIN IMMEDIATE` | `FileChannel.lock(.lock)` exclusive advisory lock | --- ## CLI commands (later; out of scope for this plan) Anticipated but not implemented here: ``` amy store import # from relay / file / stdin amy store export [filter] # to stdout NDJSON amy store query # debug amy store scrub # rebuild derived state amy store compact # drop orphans amy store stat # counts, disk usage ``` Added in a follow-up roadmap row once the store itself is in place. --- ## Open questions (flagged for review before coding) 1. **Tag index granularity.** SQLite hashes `(name, value)` into one `Long`. We could keep the same hash scheme and use the hex of the Murmur hash as the directory name, giving byte-for-byte parity with SQLite's index semantics. Proposed: yes — reuse `TagNameValueHasher`. 2. **Filename collisions on d-tag.** SHA-256 is safe; Murmur is not. Use SHA-256 for d-tag slots specifically. 3. **Relay URL in constructor.** `SQLiteEventStore` takes a `relayUrlIdentifier` for NIP-62 scoping (`shouldVanishFrom`). We mirror that; same semantics. 4. **Empty tombstone semantics.** If a tombstone hardlink is deleted by the user, enforcement stops for that target. Scrub does **not** re-create tombstones from lingering kind-5 events in `events/` — removing the tombstone is treated as an explicit "un-forget" action. Confirm with user. --- Next: pipelines (`2026-04-24-file-event-store-pipelines.md`) — insert, query, delete, transaction, concurrency.