amethyst/cli/plans/2026-04-24-file-event-store-overview.md

File-backed event store for amy — Part 1: Overview & Layout

Status: plan · Date: 2026-04-24 · Part 1 of 3 (see 2026-04-24-file-event-store-pipelines.md, 2026-04-24-file-event-store-nips.md)

A filesystem-native IEventStore for the cli/ module that matches every feature of the SQLite store (quartz/.../nip01Core/store/sqlite/) while tolerating the fact that files may be created or deleted by the user between runs.

---

Goals

1. Full feature parity with SQLiteEventStore — same IEventStore contract, same NIP semantics (01 replaceable/addressable, 09 deletion, 40 expiration, 45 count, 50 search, 62 vanish, 91 multi- tag AND).
2. Human-inspectable. Every event is a JSON file on disk. ls, cat, jq, grep, rsync, git, backup tools — all work.
3. Tolerates user edits. If the user deletes an event file, the store converges: dangling index entries become no-ops, tombstones stop enforcing, replaceable slots get rebuilt from events/.
4. JVM-only, no JNI. No SQLite, no native deps beyond what the CLI already carries. Single-user, single-host.
5. Small-to-medium scale. Tens of thousands of events. Not a full Amethyst cache replacement.

Non-goals

• Beating SQLite on throughput for 100k+ event workloads.
• Network/replication (rsync is good enough).
• Multi-tenant; concurrent writers across machines.
• Incremental migration tooling from SQLite (not needed — amy has no existing persisted store).

---

Guiding principles

1. Filesystem primitives enforce invariants. Directory-entry uniqueness = UNIQUE constraints. rename(2) = atomic commit. Hardlink refcount = cascade delete. chmod 444 = immutable tables. flock = transaction serialization.
2. Hardlinks for all indexes. An index entry is a second name for the canonical event file, never a copy. Deleting any name drops the refcount; when it hits zero the kernel reclaims.
3. Derived state is rebuildable. idx/, replaceable/, addressable/, tombstones/ can be regenerated by walking events/. amy store scrub does this.
4. events/ is the source of truth. If a file is there, it is part of the store. If it is gone, it is gone — tombstones aside.
5. No business logic. Lives next to the SQLite reference store in quartz/, sibling pattern. No Nostr-protocol decisions live in cli/.

---

Module placement

Source: quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/

• Sibling to the existing quartz/src/commonMain/kotlin/.../nip01Core/store/sqlite/ reference implementation. The plan originally proposed commons/ but the shipped placement is quartz/jvmMain/, where every other event- store concern already lives — and quartz already has a jvmTest source set so we get JVM-specific tests for free.
• JVM-only (uses java.nio.file, FileChannel.lock, Files.createLink).
• Consumed by cli/. Android keeps SQLiteEventStore. Desktop can opt in later if useful.
• Tests under quartz/src/jvmTest/.../store/fs/.

---

Directory layout

Root: configurable. The CLI uses <data-dir>/events-store/ (where <data-dir> is --data-dir PATH, $AMETHYST_CLI_DATA, or ./amy). The plan originally said events/; the actual DataDir.eventsDir field is events-store/ to leave the bare name events/ available for the canonical-events subdirectory inside the store.

<root>/
├── .lock                               # flock target for write serialization
├── .seed                               # random Long for hash salting; write-once
├── .version                            # schema version for future migrations
│
├── events/<aa>/<bb>/<id>.json          # canonical. <aa><bb> = first 4 hex of id
│
├── idx/
│   ├── kind/<kind>/<created_at>-<id>          # hardlink → events/.../<id>.json
│   ├── author/<pubkey>/<created_at>-<id>       # hardlink
│   ├── owner/<owner_hash>/<created_at>-<id>    # hardlink; gift-wrap recipient
│   ├── tag/<name>/<hash>/<created_at>-<id>    # hardlink; single-letter tags
│   └── fts/<token>/<id>                        # hardlink; inverted index
│
├── replaceable/<kind>/<pubkey>.json    # hardlink to current winner
├── addressable/<kind>/<pubkey>/<sha256(d)>.json  # hardlink to current winner
│
└── tombstones/
    ├── id/<id>                         # hardlink to the kind-5 event
    ├── addr/<kind>/<pubkey>/<sha256(d)>   # hardlink to kind-5
    └── vanish/<pubkey_hash>            # hardlink to kind-62

Filename conventions

• Event files: <id>.json, where <id> is the 64-hex event id. Content is the raw NIP-01 JSON (same bytes the signer produced).
• Timestamp prefix: zero-padded 10-digit unix seconds (0001713960000) so lexicographic sort = chronological sort.
• Hashes: SHA-256 of the canonical bytes, hex-lowercased. Used for d-tags and arbitrary tag values (sanitizes user content for filesystem safety).
• Owner hash: TagNameValueHasher.hash(ownerPubkey) (MurmurHash3 salted by .seed). Matches SQLite's pubkey_owner_hash semantics — for GiftWraps it is the recipient's p-tag, not the event pubkey. (EventIndexesModule.kt:161-166)

Sharding

• events/<aa>/<bb>/ = 65 536 leaf directories max. Keeps any leaf under ~a few hundred files for 10 M events. Piggybacks on ext4 htree / APFS B-tree / NTFS index.
• idx/*/…/ no sharding; query paths pick one index tree and walk its subdirectory directly (idx/kind/1/, idx/author/<pk>/).
• Long pubkeys / ids are already 64 hex (32 bytes). No length issues.

File modes

• Event files: 444 (read-only). Matches SQLite's "immutable tables" invariant: triggers block UPDATE (EventIndexesModule.kt:105-111). We achieve the same at the OS layer.
• Directories: 755. User can still rm (write on parent dir only).
• Tombstones: 444 (they are event files via hardlink).

---

Feature parity matrix

SQLite feature	File-store mechanism	Enforced by
event_headers	events/<aa>/<bb>/<id>.json	Write-then-rename atomicity
event_tags	idx/tag/<n>/<h>/<ts>-<id> hardlinks	Set on insert; orphans tolerated
event_fts	idx/fts/<token>/<id> hardlinks	Tokenize SearchableEvent.indexableContent()
event_expirations	idx/expires_at/<ts>-<id> hardlink	Swept by deleteExpiredEvents()
event_vanish	tombstones/vanish/<owner_hash> hardlink	Checked on insert
seeds	.seed file; 8 random bytes	Written once on create
UNIQUE(kind,pubkey) for replaceable	replaceable/<k>/<pk>.json slot	Directory-entry uniqueness + atomic rename
UNIQUE(kind,pubkey,d) for addressable	addressable/<k>/<pk>/<h>.json slot	Same
Reject ephemeral	Code guard before any file write	if (event.kind.isEphemeral()) return
Reject expired on insert	Code guard before any file write	if (event.isExpired()) throw
NIP-09 delete by id	Unlink events/.../<id>.json + all hardlinks	Tombstone blocks re-insert
NIP-09 delete by address	Unlink addressable slot + canonical	Tombstone blocks re-insert
NIP-09 gift-wrap by p-tag	idx/owner/<owner_hash>/ walk	Owner hash matches recipient for GiftWrap
NIP-62 vanish cascade	Walk idx/owner/<owner_hash>/ and unlink	AFTER-insert cascade
NIP-62 block future	Check tombstones/vanish/<owner_hash>	Before every insert
NIP-40 deleteExpired sweep	Walk idx/expires_at/ until now, unlink	Called by CLI cron
NIP-45 count	Same planner as query, count results	Streaming
NIP-50 search	Intersect idx/fts/<token>/ sets	Tokenizer matches Quartz
NIP-91 tag AND	Sorted-list intersection of idx/tag/<n>/<h>/	Streaming comm -12 equivalent
transaction {}	flock(.lock) + stage + commit + unlock	All writes to staging dir, single rename
vacuum()	amy store scrub — walks events/, rebuilds idx/ etc.	Offline maintenance
analyse()	No-op; FS already maintains its own indexes	—

---

Public API

Implements com.vitorpamplona.quartz.nip01Core.store.IEventStore (quartz/.../store/IEventStore.kt:26-60) in full.

class FsEventStore(
    root: Path,
    relayUrl: String,                    // matches SQLiteEventStore ctor; used for NIP-62 scoping
    indexingStrategy: IndexingStrategy = DefaultIndexingStrategy,
    clock: Clock = Clock.systemUTC(),    // injectable for tests
) : IEventStore {

    // IEventStore
    override fun insert(event: Event)
    override fun transaction(body: IEventStore.ITransaction.() -> Unit)
    override fun <T : Event> query(filter: Filter): List<T>
    override fun <T : Event> query(filters: List<Filter>): List<T>
    override fun <T : Event> query(filter: Filter, onEach: (T) -> Unit)
    override fun <T : Event> query(filters: List<Filter>, onEach: (T) -> Unit)
    override fun count(filter: Filter): Int
    override fun count(filters: List<Filter>): Int
    override fun delete(filter: Filter)
    override fun delete(filters: List<Filter>)
    override fun deleteExpiredEvents()
    override fun close()

    // FsEventStore extras (parity with SQLiteEventStore)
    fun delete(id: HexKey): Int
    fun rawQuery(filter: Filter): List<RawEvent>         // parses JSON to RawEvent
    fun rawQuery(filters: List<Filter>): List<RawEvent>
    fun planQuery(filter: Filter): String                // returns human-readable plan
    fun planQuery(filters: List<Filter>): String

    // Maintenance
    suspend fun scrub()                                  // rebuild all derived state
    suspend fun compact()                                // equivalent to vacuum — drops orphans
}

Signer/event conveniences mirror SQLite's — no behavioural drift.

---

File-backed counterparts to SQLite pragmas

SQLite pragma	File-store analog
journal_mode=WAL	Staging dir .staging/<uuid>/ + atomic rename on commit
synchronous=OFF	Default = no fsync per write; AMY_FSYNC=1 env opts in
cache_size=-32000	None — rely on OS page cache
foreign_keys=ON	Implicit: hardlinks are the FK; refcount is the cascade
BEGIN IMMEDIATE	FileChannel.lock(.lock) exclusive advisory lock

---

CLI commands (later; out of scope for this plan)

Anticipated but not implemented here:

amy store import <path>            # from relay / file / stdin
amy store export [filter]          # to stdout NDJSON
amy store query <filter-json>      # debug
amy store scrub                    # rebuild derived state
amy store compact                  # drop orphans
amy store stat                     # counts, disk usage

Added in a follow-up roadmap row once the store itself is in place.

---

Open questions (flagged for review before coding)

1. Tag index granularity. SQLite hashes (name, value) into one Long. We could keep the same hash scheme and use the hex of the Murmur hash as the directory name, giving byte-for-byte parity with SQLite's index semantics. Proposed: yes — reuse TagNameValueHasher.
2. Filename collisions on d-tag. SHA-256 is safe; Murmur is not. Use SHA-256 for d-tag slots specifically.
3. Relay URL in constructor. SQLiteEventStore takes a relayUrlIdentifier for NIP-62 scoping (shouldVanishFrom). We mirror that; same semantics.
4. Empty tombstone semantics. If a tombstone hardlink is deleted by the user, enforcement stops for that target. Scrub does not re-create tombstones from lingering kind-5 events in events/ — removing the tombstone is treated as an explicit "un-forget" action. Confirm with user.

---

Next: pipelines (2026-04-24-file-event-store-pipelines.md) — insert, query, delete, transaction, concurrency.