amethyst/cli/plans/2026-04-24-file-event-store-overview.md

# File-backed event store for `amy` — Part 1: Overview & Layout

**Status:** plan · **Date:** 2026-04-24 · **Part 1 of 3**
(see `2026-04-24-file-event-store-pipelines.md`,
`2026-04-24-file-event-store-nips.md`)

A filesystem-native `IEventStore` for the `cli/` module that matches
every feature of the SQLite store
(`quartz/.../nip01Core/store/sqlite/`) while tolerating the fact that
files may be created or deleted by the user between runs.

---

## Goals

1. **Full feature parity** with `SQLiteEventStore` — same `IEventStore`
   contract, same NIP semantics (01 replaceable/addressable, 09
   deletion, 40 expiration, 45 count, 50 search, 62 vanish, 91 multi-
   tag AND).
2. **Human-inspectable.** Every event is a JSON file on disk. `ls`,
   `cat`, `jq`, `grep`, `rsync`, `git`, backup tools — all work.
3. **Tolerates user edits.** If the user deletes an event file, the
   store converges: dangling index entries become no-ops, tombstones
   stop enforcing, replaceable slots get rebuilt from `events/`.
4. **JVM-only, no JNI.** No SQLite, no native deps beyond what the
   CLI already carries. Single-user, single-host.
5. **Small-to-medium scale.** Tens of thousands of events. Not a
   full Amethyst cache replacement.

## Non-goals

- Beating SQLite on throughput for 100k+ event workloads.
- Network/replication (rsync is good enough).
- Multi-tenant; concurrent writers across machines.
- Incremental migration tooling from SQLite (not needed — `amy` has no
  existing persisted store).

---

## Guiding principles

1. **Filesystem primitives enforce invariants.** Directory-entry
   uniqueness = `UNIQUE` constraints. `rename(2)` = atomic commit.
   Hardlink refcount = cascade delete. `chmod 444` = immutable
   tables. `flock` = transaction serialization.
2. **Hardlinks for all indexes.** An index entry is a second *name*
   for the canonical event file, never a copy. Deleting any name
   drops the refcount; when it hits zero the kernel reclaims.
3. **Derived state is rebuildable.** `idx/`, `replaceable/`,
   `addressable/`, `tombstones/` can be regenerated by walking
   `events/`. `amy store scrub` does this.
4. **`events/` is the source of truth.** If a file is there, it is
   part of the store. If it is gone, it is gone — tombstones aside.
5. **No business logic.** Lives next to the SQLite reference store
   in `quartz/`, sibling pattern. No Nostr-protocol decisions live
   in `cli/`.

---

## Module placement

Source: `quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/`

- Sibling to the existing
  `quartz/src/commonMain/kotlin/.../nip01Core/store/sqlite/` reference
  implementation. The plan originally proposed `commons/` but the
  shipped placement is `quartz/jvmMain/`, where every other event-
  store concern already lives — and quartz already has a `jvmTest`
  source set so we get JVM-specific tests for free.
- JVM-only (uses `java.nio.file`, `FileChannel.lock`, `Files.createLink`).
- Consumed by `cli/`. Android keeps `SQLiteEventStore`. Desktop can
  opt in later if useful.
- Tests under `quartz/src/jvmTest/.../store/fs/`.

---

## Directory layout

Root: configurable. The CLI uses `<data-dir>/events-store/` (where
`<data-dir>` is `--data-dir PATH`, `$AMETHYST_CLI_DATA`, or `./amy`).
The plan originally said `events/`; the actual `DataDir.eventsDir`
field is `events-store/` to leave the bare name `events/` available
for the canonical-events subdirectory inside the store.

```
<root>/
├── .lock                               # flock target for write serialization
├── .seed                               # random Long for hash salting; write-once
├── .version                            # schema version for future migrations
│
├── events/<aa>/<bb>/<id>.json          # canonical. <aa><bb> = first 4 hex of id
│
├── idx/
│   ├── kind/<kind>/<created_at>-<id>          # hardlink → events/.../<id>.json
│   ├── author/<pubkey>/<created_at>-<id>       # hardlink
│   ├── owner/<owner_hash>/<created_at>-<id>    # hardlink; gift-wrap recipient
│   ├── tag/<name>/<hash>/<created_at>-<id>    # hardlink; single-letter tags
│   └── fts/<token>/<id>                        # hardlink; inverted index
│
├── replaceable/<kind>/<pubkey>.json    # hardlink to current winner
├── addressable/<kind>/<pubkey>/<sha256(d)>.json  # hardlink to current winner
│
└── tombstones/
    ├── id/<id>                         # hardlink to the kind-5 event
    ├── addr/<kind>/<pubkey>/<sha256(d)>   # hardlink to kind-5
    └── vanish/<pubkey_hash>            # hardlink to kind-62
```

### Filename conventions

- **Event files:** `<id>.json`, where `<id>` is the 64-hex event id.
  Content is the raw NIP-01 JSON (same bytes the signer produced).
- **Timestamp prefix:** zero-padded 10-digit unix seconds
  (`0001713960000`) so lexicographic sort = chronological sort.
- **Hashes:** SHA-256 of the canonical bytes, hex-lowercased. Used
  for d-tags and arbitrary tag values (sanitizes user content for
  filesystem safety).
- **Owner hash:** `TagNameValueHasher.hash(ownerPubkey)` (MurmurHash3
  salted by `.seed`). Matches SQLite's `pubkey_owner_hash` semantics
  — for GiftWraps it is the recipient's p-tag, not the event pubkey.
  (`EventIndexesModule.kt:161-166`)

### Sharding

- `events/<aa>/<bb>/` = 65 536 leaf directories max. Keeps any leaf
  under ~a few hundred files for 10 M events. Piggybacks on ext4
  htree / APFS B-tree / NTFS index.
- `idx/*/…/` no sharding; query paths pick one index tree and walk
  its subdirectory directly (`idx/kind/1/`, `idx/author/<pk>/`).
- Long pubkeys / ids are already 64 hex (32 bytes). No length issues.

### File modes

- Event files: `444` (read-only). Matches SQLite's "immutable tables"
  invariant: triggers block `UPDATE`
  (`EventIndexesModule.kt:105-111`). We achieve the same at the OS
  layer.
- Directories: `755`. User can still `rm` (write on parent dir only).
- Tombstones: `444` (they *are* event files via hardlink).

---

## Feature parity matrix

| SQLite feature | File-store mechanism | Enforced by |
|---|---|---|
| `event_headers` | `events/<aa>/<bb>/<id>.json` | Write-then-rename atomicity |
| `event_tags` | `idx/tag/<n>/<h>/<ts>-<id>` hardlinks | Set on insert; orphans tolerated |
| `event_fts` | `idx/fts/<token>/<id>` hardlinks | Tokenize `SearchableEvent.indexableContent()` |
| `event_expirations` | `idx/expires_at/<ts>-<id>` hardlink | Swept by `deleteExpiredEvents()` |
| `event_vanish` | `tombstones/vanish/<owner_hash>` hardlink | Checked on insert |
| `seeds` | `.seed` file; 8 random bytes | Written once on create |
| `UNIQUE(kind,pubkey)` for replaceable | `replaceable/<k>/<pk>.json` slot | Directory-entry uniqueness + atomic rename |
| `UNIQUE(kind,pubkey,d)` for addressable | `addressable/<k>/<pk>/<h>.json` slot | Same |
| Reject ephemeral | Code guard before any file write | `if (event.kind.isEphemeral()) return` |
| Reject expired on insert | Code guard before any file write | `if (event.isExpired()) throw` |
| NIP-09 delete by id | Unlink `events/.../<id>.json` + all hardlinks | Tombstone blocks re-insert |
| NIP-09 delete by address | Unlink addressable slot + canonical | Tombstone blocks re-insert |
| NIP-09 gift-wrap by p-tag | `idx/owner/<owner_hash>/` walk | Owner hash matches recipient for GiftWrap |
| NIP-62 vanish cascade | Walk `idx/owner/<owner_hash>/` and unlink | AFTER-insert cascade |
| NIP-62 block future | Check `tombstones/vanish/<owner_hash>` | Before every insert |
| NIP-40 deleteExpired sweep | Walk `idx/expires_at/` until now, unlink | Called by CLI cron |
| NIP-45 count | Same planner as query, count results | Streaming |
| NIP-50 search | Intersect `idx/fts/<token>/` sets | Tokenizer matches Quartz |
| NIP-91 tag AND | Sorted-list intersection of `idx/tag/<n>/<h>/` | Streaming `comm -12` equivalent |
| `transaction {}` | `flock(.lock) + stage + commit + unlock` | All writes to staging dir, single rename |
| `vacuum()` | `amy store scrub` — walks `events/`, rebuilds `idx/` etc. | Offline maintenance |
| `analyse()` | No-op; FS already maintains its own indexes | — |

---

## Public API

Implements `com.vitorpamplona.quartz.nip01Core.store.IEventStore`
(`quartz/.../store/IEventStore.kt:26-60`) in full.

```kotlin
class FsEventStore(
    root: Path,
    relayUrl: String,                    // matches SQLiteEventStore ctor; used for NIP-62 scoping
    indexingStrategy: IndexingStrategy = DefaultIndexingStrategy,
    clock: Clock = Clock.systemUTC(),    // injectable for tests
) : IEventStore {

    // IEventStore
    override fun insert(event: Event)
    override fun transaction(body: IEventStore.ITransaction.() -> Unit)
    override fun <T : Event> query(filter: Filter): List<T>
    override fun <T : Event> query(filters: List<Filter>): List<T>
    override fun <T : Event> query(filter: Filter, onEach: (T) -> Unit)
    override fun <T : Event> query(filters: List<Filter>, onEach: (T) -> Unit)
    override fun count(filter: Filter): Int
    override fun count(filters: List<Filter>): Int
    override fun delete(filter: Filter)
    override fun delete(filters: List<Filter>)
    override fun deleteExpiredEvents()
    override fun close()

    // FsEventStore extras (parity with SQLiteEventStore)
    fun delete(id: HexKey): Int
    fun rawQuery(filter: Filter): List<RawEvent>         // parses JSON to RawEvent
    fun rawQuery(filters: List<Filter>): List<RawEvent>
    fun planQuery(filter: Filter): String                // returns human-readable plan
    fun planQuery(filters: List<Filter>): String

    // Maintenance
    suspend fun scrub()                                  // rebuild all derived state
    suspend fun compact()                                // equivalent to vacuum — drops orphans
}
```

Signer/event conveniences mirror SQLite's — no behavioural drift.

---

## File-backed counterparts to SQLite pragmas

| SQLite pragma | File-store analog |
|---|---|
| `journal_mode=WAL` | Staging dir `.staging/<uuid>/` + atomic rename on commit |
| `synchronous=OFF` | Default = no `fsync` per write; `AMY_FSYNC=1` env opts in |
| `cache_size=-32000` | None — rely on OS page cache |
| `foreign_keys=ON` | Implicit: hardlinks are the FK; refcount is the cascade |
| `BEGIN IMMEDIATE` | `FileChannel.lock(.lock)` exclusive advisory lock |

---

## CLI commands (later; out of scope for this plan)

Anticipated but not implemented here:

```
amy store import <path>            # from relay / file / stdin
amy store export [filter]          # to stdout NDJSON
amy store query <filter-json>      # debug
amy store scrub                    # rebuild derived state
amy store compact                  # drop orphans
amy store stat                     # counts, disk usage
```

Added in a follow-up roadmap row once the store itself is in place.

---

## Open questions (flagged for review before coding)

1. **Tag index granularity.** SQLite hashes `(name, value)` into one
   `Long`. We could keep the same hash scheme and use the hex of the
   Murmur hash as the directory name, giving byte-for-byte parity
   with SQLite's index semantics. Proposed: yes — reuse
   `TagNameValueHasher`.
2. **Filename collisions on d-tag.** SHA-256 is safe; Murmur is not.
   Use SHA-256 for d-tag slots specifically.
3. **Relay URL in constructor.** `SQLiteEventStore` takes a
   `relayUrlIdentifier` for NIP-62 scoping (`shouldVanishFrom`). We
   mirror that; same semantics.
4. **Empty tombstone semantics.** If a tombstone hardlink is deleted
   by the user, enforcement stops for that target. Scrub does **not**
   re-create tombstones from lingering kind-5 events in `events/` —
   removing the tombstone is treated as an explicit "un-forget"
   action. Confirm with user.

---

Next: pipelines
(`2026-04-24-file-event-store-pipelines.md`) — insert, query, delete,
transaction, concurrency.