nick/amethyst
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(cli): plan a file-backed event store for amy

Browse Source 
Three-part design doc for an IEventStore backed by a directory tree
instead of SQLite, targeted at the cli/ module.

- overview: goals, directory layout, feature-parity matrix, public API
- pipelines: insert (T1-T8 with crash-safety), query planner, delete,
  transactions, concurrency
- nips: replaceable/addressable slot enforcement via hardlinks +
  atomic rename, NIP-09 tombstones (hardlink to the kind-5), NIP-40
  expirations, NIP-50 FTS via inverted-index hardlinks, NIP-62 vanish
  cascade, tag indexing, seed file, scrub/compact, test strategy,
  10-step rollout
This commit is contained in:
Claude
2026-04-24 20:58:18 +00:00
parent f341bc776e
commit d230c5e86b
3 changed files with 1086 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cli/plans/2026-04-24-file-event-store-nips.md
+500
View File
@@ -0,0 +1,500 @@
# File-backed event store for `amy` — Part 3: NIPs, Enforcement, Scrub, Testing
**Status:** plan · **Date:** 2026-04-24 · **Part 3 of 3**
(see `2026-04-24-file-event-store-overview.md`,
`2026-04-24-file-event-store-pipelines.md`)
Per-NIP behavior, slot enforcement details, tag indexing, scrub, and
test plan.
---
## NIP-01 — Replaceable events (kinds 0, 3, 10000–19999)
### Enforcement
- Slot path: `replaceable/<kind>/<pubkey>.json`
- Slot is a **hardlink** into `events/<aa>/<bb>/<id>.json` — same
inode, two names.
- Directory-entry uniqueness is the `UNIQUE(kind, pubkey)` constraint
(parity with `ReplaceableModule.kt:25-60`).
### Insert flow (inside transaction T2/T5 from Part 2)
```
guard = ReplaceableGuard
if slot exists:
Told = read slot JSON → createdAt
if event.createdAt <= Told: REJECT # blocks older from reinsert
install (at T5):
slotTmp = <slot>.tmp.<rand>
ln events/<new_id>.json slotTmp
Files.move(slotTmp, slot, ATOMIC_MOVE, REPLACE_EXISTING)
# Old slot's inode loses one link. If it had a canonical hardlink too,
# drop it — mirror SQLite trigger that deletes the old row.
if oldId present:
runDeleteLinksFor(oldId) # same code path as delete()
```
### Reading the current version
```
cat replaceable/3/<pubkey>.json # latest kind-3 for pubkey
```
Direct path open. No scan, no MAX — the slot is the winner.
### Edge cases
- **Race between two newer events.** Both pass guard, both try to
install; last `rename` wins atomically. Losing event's canonical
exists orphaned until scrub or until someone queries by its id.
Semantically valid: both were newer than the previous slot.
- **Slot deleted by user.** Next insert of (kind, pubkey) has no
guard and wins. This matches "files come and go" contract.
- **Canonical deleted but slot survives.** Slot still resolves —
hardlink keeps inode alive. Queries work. Scrub re-creates the
canonical from slot.
---
## NIP-01 — Addressable events (kinds 30000–39999)
### Enforcement
- Slot path: `addressable/<kind>/<pubkey>/<sha256(d_tag)>.json`
- SHA-256 of d-tag value avoids filesystem-unsafe characters and
collisions. Parity with `AddressableModule.kt:25-63`.
- Empty d-tag is valid Nostr — hashed as `sha256("")`.
### Flow
Same as replaceable, keyed by (kind, pubkey, d_tag). Guard reads
slot's createdAt; install via `rename REPLACE_EXISTING`; cascade-
unlink old canonical and its index hardlinks.
### Query shortcut
If a filter supplies `kinds`, `authors`, and a d-tag, the planner
bypasses the driver indexes and opens the slot directly. O(1).
---
## NIP-09 — Event deletion (kind 5)
### Deletion event insert
When a kind-5 event is inserted (inside T6 of the pipeline):
```
for target in event.deleteEventIds():
# NIP-09: same author as the deletion
targetPath = events/<aa>/<bb>/<target>.json
if targetPath exists AND read(targetPath).pubkey == event.pubkey:
runDeleteLinksFor(target) # cascade unlink
# Install tombstone (hardlink to the kind-5 event):
tomb = tombstones/id/<target>
ln event_canonical <tomb>.tmp
Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING_IF_NEWER)
for addr in event.deleteAddresses():
hash = sha256(addr.dTag)
slot = addressable/<addr.kind>/<addr.pubkey>/<hash>.json
if slot exists AND read(slot).pubkey == event.pubkey:
Told = read slot.createdAt
if Told <= event.createdAt:
oldId = read slot.id
unlink slot
runDeleteLinksFor(oldId)
tomb = tombstones/addr/<addr.kind>/<addr.pubkey>/<hash>
ln event_canonical <tomb>.tmp
Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING_IF_NEWER)
for replaceable_kind in event.deleteReplaceableKinds():
# kinds 0,3,10000-19999 — no d-tag
slot = replaceable/<kind>/<event.pubkey>.json
similar cascade + tombstone
```
`REPLACE_EXISTING_IF_NEWER` is a helper we implement: if an older
kind-5 already owns the tombstone, and the new one has a later
createdAt, swap it in; otherwise keep the existing. Ensures
tombstone holds the strongest (latest) deletion cutoff, matching
SQLite's "OR" of all deletion events
(`DeletionRequestModule.kt:102-184`).
### Blocking re-insertion
Part of T2 guards:
```
TombstoneIdCheck:
if tombstones/id/<event.id> exists → REJECT
TombstoneAddrCheck (addressable only):
if tombstones/addr/<k>/<pk>/<hash(d)> exists:
Tdel = read tombstone.createdAt
if event.createdAt <= Tdel → REJECT
```
Parity with `DeletionRequestModule.kt:61-75` (BEFORE INSERT trigger
rejecting events whose etag_hash/atag_hash match a kind-5 entry).
### Gift-wrap deletion by p-tag
`EventIndexesModule.kt:161-166` uses `pubkey_owner_hash` that, for
`GiftWrapEvent`, is the recipient's p-tag (not the outer pubkey).
We mirror this: `idx/owner/<owner_hash>/<ts>-<id>` points at every
event where `ownerHash = TagNameValueHasher.hash(recipient)`.
A kind-5 from the recipient deleting by `e`-tag still resolves via
`runDeleteLinksFor(id)`, which walks all hardlinks (including
`idx/owner/`) and unlinks them.
### Security: author check
NIP-09 requires deletions be authored by the same pubkey as the
deleted event. SQLite enforces this via `pubkey_owner_hash` in the
DELETE WHERE clause (`DeletionRequestModule.kt:107-115`). We mirror
by reading the target's JSON and comparing `pubkey` — done inside
`runDeleteLinksFor`.
---
## NIP-40 — Expiration
### Read at insert
`event.expiration()` returns `Long?` (tag value as unix seconds).
```
if exp != null && exp > 0:
if exp <= now: throw ExpiredEventException # T1 guard, mirrors trigger
ln canonical idx/expires_at/<zero_padded(exp)>-<id>
```
Parity with `ExpirationModule.kt:27-95`.
### Sweep
`deleteExpiredEvents()` listed in Part 2. Uses the sorted
`idx/expires_at/` — early termination when `<exp>` ≥ now.
Recommended caller: cron or shell loop.
```sh
while :; do amy store sweep-expired; sleep 900; done
```
Matches the `ExpirationWorker` WorkManager pattern from the SQLite
README.
### Query-time filtering
Belt-and-suspenders: queries also filter out events where
`exp != null && exp <= now`. Covers sweep-missed windows and
clock skew between writer and reader.
---
## NIP-62 — Right to vanish (kind 62)
### Insert
```
if !event.shouldVanishFrom(relayUrl): return # same scoping as SQLite
ownerHash = TagNameValueHasher.hash(event.pubkey)
# Install vanish tombstone (one per pubkey; latest wins)
tomb = tombstones/vanish/<ownerHash>
existing = read tomb if exists
if existing == null || existing.createdAt < event.createdAt:
ln canonical <tomb>.tmp
Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING)
# Cascade: delete every event from this owner with createdAt < tomb.createdAt
for entry in idx/owner/<ownerHash>/:
(ts, id) = parse entry filename
if ts < event.createdAt:
runDeleteLinksFor(id)
```
Parity with `RightToVanishModule.kt:46-71`.
### Blocking future inserts
Part of T2 guards (VanishCheck):
```
tomb = tombstones/vanish/<TagNameValueHasher.hash(event.pubkey)>
if tomb exists:
Tvanish = read tomb.createdAt
if event.createdAt <= Tvanish → REJECT
```
For GiftWraps, owner is the recipient — a vanish request from the
recipient cascades their received gift-wraps too, matching SQLite's
behaviour via `pubkey_owner_hash`.
---
## NIP-45 — Count
`count(filter)` / `count(filters)` runs the exact query plan from
Part 2 and returns the number of surviving entries. Parity with
SQLite `SELECT count(*)`.
Optimisation: if the filter is driver-only (no post-driver code
filter), we just count directory entries — `Files.list().count()` —
without opening any JSON. Linear in the driver's listing size.
---
## NIP-50 — Full-text search
### Tokenisation
Reuse Quartz's `SearchableEvent.indexableContent()` (per
`FullTextSearchModule.kt:66-78`). Tokeniser:
- Lowercase.
- Split on Unicode word boundaries.
- Filter tokens shorter than 3 chars (FTS5 default).
- Keep ASCII hex ids and bech32 identifiers whole (no splitting on
underscores/digits).
Match the Android FTS5 tokeniser behaviour; a shared helper lives
in `commons/` and is called by both backends to guarantee identical
search results.
### Index
```
idx/fts/<token>/<id> # empty hardlink into canonical
```
One empty hardlink per (token, event). For a 1000-word note with
500 unique tokens, 500 hardlinks — each costs one directory entry
(~~50 bytes on ext4), no inode. At scale this is the biggest
directory count; `idx/fts/` alone may hold millions of entries.
Still fine — each token dir stays small.
### Search query
```
tokens = tokenize(filter.search)
streams = tokens.map { idx/fts/<it>/ }
candidates = k-way intersection of streams by <id>
for candidate in candidates:
if passes all other filter predicates:
emit
```
Same semantics as SQLite `MATCH` in FTS5 AND-mode. Matches
`QueryBuilder.kt:590-657`.
### Maintenance
- On insert: write token hardlinks (T4).
- On delete (any path — direct, replaceable replace, vanish,
expiration): `runDeleteLinksFor(id)` enumerates every possible
token and unlinks. We don't store the token list, so we re-derive
it by reading the canonical content *before* unlinking the
canonical. If the canonical is already gone, we fall back to
`find -samefile` to enumerate remaining hardlinks.
### Without an index
`amy store search --raw "keywords"` can fall back to
`ripgrep events/` for forensic use. Always works, even if indexes
are corrupt.
---
## Tag indexing
### Which tags get indexed
Matches `DefaultIndexingStrategy`
(`IndexingStrategy.kt:99-102`): tag arrays with at least 2 elements
where `tag[0].length == 1` (single-letter tag names). Non-single-
letter tags are stored in the JSON and visible to queries but not
reverse-indexed.
### Hash scheme
Reuse `TagNameValueHasher`
(`quartz/.../sqlite/TagNameValueHasher.kt:30-67`). Reads `.seed`
into a `Long` salt identically to SQLite's `seeds` table. The hash
becomes a 16-char hex string that names the directory:
```
idx/tag/<name>/<hex(hash(name, value))>/<ts>-<id>
```
Parity with `event_tags.tag_hash`. d-tag, e-tag, a-tag, p-tag,
owner: see below.
### Special tags
| Tag | Where indexed | Notes |
|---|---|---|
| `d` | Only as part of `addressable/<k>/<pk>/<sha256(d)>.json` slot | Not in `idx/tag/d/` |
| `e` | `idx/tag/e/<hash>/` | Normal |
| `a` | `idx/tag/a/<hash>/` + addressable slot look-aside | Normal |
| `p` | `idx/tag/p/<hash>/` | Recipient for GiftWrap also goes to `idx/owner/` |
| owner (synthetic) | `idx/owner/<owner_hash>/` | Event pubkey for normal events, recipient for GiftWrap — matches `pubkey_owner_hash` (`EventIndexesModule.kt:161-166`) |
---
## Seed file
`.seed` is 8 random bytes generated on store creation. Read into a
`Long` salt for `TagNameValueHasher`, matching SQLite's `seeds` table
(`SeedModule.kt:26-84`).
- Written atomically (tmp-rename) on first open if absent.
- Never updated. Corrupt or mismatched seeds invalidate `idx/` —
scrub rebuilds from canonical.
- Permissions `400`.
---
## Scrub / rebuild
`FsEventStore.scrub()`:
```
1. Lock .lock exclusively.
2. Delete .staging/ leftovers.
3. Enumerate all events/<aa>/<bb>/<id>.json.
4. For each, re-derive:
idx/kind/, idx/author/, idx/owner/, idx/tag/*/,
idx/fts/*/, idx/expires_at/
using the same code as T4. createLink is idempotent — NOACT on
collision (or unlink + recreate).
5. Walk idx/ trees; drop entries whose target inode count is zero
(dangling — canonical was deleted).
6. Rebuild replaceable/ and addressable/ slots from canonicals by
grouping (kind,pubkey[,d]) and picking max createdAt.
7. Keep tombstones as-is. (Removing a tombstone file is a user-
intended action; scrub does not resurrect them.)
8. Release lock.
```
Cost: O(N) disk reads once per scrub. Acceptable as a manual
`amy store scrub` command. Not run automatically.
### `compact()`
Lighter: walk `idx/` only, drop dangling entries. No canonical
reads. O(size of idx trees).
---
## Testing strategy
### Location
`commons/src/jvmTest/kotlin/com/vitorpamplona/commons/store/fs/`
Uses JUnit4 (matches rest of the project). Temp-dir fixture creates
a fresh store per test.
### Test categories
1. **Parity tests.** Drive both `SQLiteEventStore` and `FsEventStore`
with identical event streams and filters; assert query results
match. Existing SQLite tests under
`quartz/src/jvmTest/.../sqlite/` become the golden set. This is
the highest-value tier.
2. **Per-module tests.** One file per enforcement area:
- `ReplaceableSlotTest` — newer wins, older rejected, race
deposes both cleanly.
- `AddressableSlotTest` — same, plus d-tag hash collisions
cannot occur (SHA-256).
- `DeletionTombstoneTest` — by id, by address, by replaceable
kind; re-insert blocked; gift-wrap by owner.
- `VanishTest` — cascade + block, GiftWrap recipient.
- `ExpirationTest` — sweep, early termination, belt-and-
suspenders query filter.
- `SearchTest` — tokeniser parity with SQLite FTS, AND of tokens,
maintenance across replace/delete/expire/vanish.
- `TagAndTest` — NIP-91 intersection, three-tag filters.
- `EphemeralTest` — never written.
- `TransactionTest` — all-or-nothing batch insert, rollback on
exception.
3. **Crash-safety tests.** Inject failures after each pipeline step
(T1–T8) via a testing hook; reopen; assert scrub converges.
4. **Concurrency tests.** Two threads / two processes hitting the
same store: assert no corruption, writes serialise, readers see
consistent snapshots.
5. **External edit tests.** Mid-run, delete a canonical file / a
slot / a tombstone; assert store behaves as spec'd (dangling
index skipped; slot rebuilds; tombstone stops enforcing).
6. **Property tests.** For a random sequence of (insert, delete,
query, vanish, expire, replace) operations, assert the file
store and SQLite store produce identical `query()` results.
Seed = test parameter.
### Running
```
./gradlew :commons:jvmTest --tests "*.store.fs.*"
```
Target: all tests pass before the CLI starts depending on the new
store.
---
## Rollout plan
1. **Step 1 — skeleton.** `FsEventStore` class, directory layout,
`insert` + `query(ids only)` + `delete(id)`. Round-trip test.
2. **Step 2 — indexes.** kind/author/owner/tag hardlinks; query by
kind, author, tag; `count`.
3. **Step 3 — slots.** Replaceable + addressable enforcement +
tests.
4. **Step 4 — tombstones.** NIP-09 deletion by id and address;
block re-insert; gift-wrap via owner.
5. **Step 5 — expirations.** `idx/expires_at/`, `deleteExpiredEvents`.
6. **Step 6 — vanish.** NIP-62 insert + cascade + block.
7. **Step 7 — FTS.** Tokeniser in `commons/`, `idx/fts/`, search
queries.
8. **Step 8 — transactions, concurrency, scrub.**
9. **Step 9 — wire into `amy`.** `Context` gains a `store:
IEventStore` property; default = `FsEventStore` rooted at
`$AMY_HOME/events/`.
10. **Step 10 — parity test matrix** driven by the existing SQLite
golden tests.
Each step is a separate commit. Each is behind `AMY_EXPERIMENTAL_
STORE=1` env guard until step 10 passes.
---
## Out of scope for this plan
- Android integration — stays on SQLite.
- Desktop opt-in — separate plan once CLI is stable.
- Cross-host sync — users can rsync the directory; that's it.
- Encryption at rest — future plan; would wrap each event file.
---
## Summary of plan split
| File | Covers |
|---|---|
| `2026-04-24-file-event-store-overview.md` | Goals, layout, feature matrix, API |
| `2026-04-24-file-event-store-pipelines.md` | Insert, query, delete, transaction, concurrency |
| `2026-04-24-file-event-store-nips.md` (this file) | Per-NIP enforcement, slots, tombstones, FTS, scrub, testing |
cli/plans/2026-04-24-file-event-store-overview.md
+260
View File
@@ -0,0 +1,260 @@
# File-backed event store for `amy` — Part 1: Overview & Layout
**Status:** plan · **Date:** 2026-04-24 · **Part 1 of 3**
(see `2026-04-24-file-event-store-pipelines.md`,
`2026-04-24-file-event-store-nips.md`)
A filesystem-native `IEventStore` for the `cli/` module that matches
every feature of the SQLite store
(`quartz/.../nip01Core/store/sqlite/`) while tolerating the fact that
files may be created or deleted by the user between runs.
---
## Goals
1. **Full feature parity** with `SQLiteEventStore` — same `IEventStore`
contract, same NIP semantics (01 replaceable/addressable, 09
deletion, 40 expiration, 45 count, 50 search, 62 vanish, 91 multi-
tag AND).
2. **Human-inspectable.** Every event is a JSON file on disk. `ls`,
`cat`, `jq`, `grep`, `rsync`, `git`, backup tools — all work.
3. **Tolerates user edits.** If the user deletes an event file, the
store converges: dangling index entries become no-ops, tombstones
stop enforcing, replaceable slots get rebuilt from `events/`.
4. **JVM-only, no JNI.** No SQLite, no native deps beyond what the
CLI already carries. Single-user, single-host.
5. **Small-to-medium scale.** Tens of thousands of events. Not a
full Amethyst cache replacement.
## Non-goals
- Beating SQLite on throughput for 100k+ event workloads.
- Network/replication (rsync is good enough).
- Multi-tenant; concurrent writers across machines.
- Incremental migration tooling from SQLite (not needed — `amy` has no
existing persisted store).
---
## Guiding principles
1. **Filesystem primitives enforce invariants.** Directory-entry
uniqueness = `UNIQUE` constraints. `rename(2)` = atomic commit.
Hardlink refcount = cascade delete. `chmod 444` = immutable
tables. `flock` = transaction serialization.
2. **Hardlinks for all indexes.** An index entry is a second *name*
for the canonical event file, never a copy. Deleting any name
drops the refcount; when it hits zero the kernel reclaims.
3. **Derived state is rebuildable.** `idx/`, `replaceable/`,
`addressable/`, `tombstones/` can be regenerated by walking
`events/`. `amy store scrub` does this.
4. **`events/` is the source of truth.** If a file is there, it is
part of the store. If it is gone, it is gone — tombstones aside.
5. **No business logic.** This is a `commons/` module that the CLI
imports. No Nostr-protocol decisions live in `cli/`.
---
## Module placement
Source: `commons/src/jvmMain/kotlin/com/vitorpamplona/commons/store/fs/`
- JVM-only (uses `java.nio.file`, `FileChannel.lock`, `Files.createLink`).
- Consumed by `cli/`. Android keeps `SQLiteEventStore`. Desktop can
opt in later if useful.
- Tests under `commons/src/jvmTest/.../store/fs/`.
Not in `quartz/` because `quartz/` is protocol-only (no storage).
Not in `cli/` because it contains reusable logic.
---
## Directory layout
Root: configurable, defaults to `$AMY_HOME/events/` (where `$AMY_HOME`
is `~/.amy` by default).
```
<root>/
├── .lock # flock target for write serialization
├── .seed # random Long for hash salting; write-once
├── .version # schema version for future migrations
│
├── events/<aa>/<bb>/<id>.json # canonical. <aa><bb> = first 4 hex of id
│
├── idx/
│ ├── kind/<kind>/<created_at>-<id> # hardlink → events/.../<id>.json
│ ├── author/<pubkey>/<created_at>-<id> # hardlink
│ ├── owner/<owner_hash>/<created_at>-<id> # hardlink; gift-wrap recipient
│ ├── tag/<name>/<hash>/<created_at>-<id> # hardlink; single-letter tags
│ └── fts/<token>/<id> # hardlink; inverted index
│
├── replaceable/<kind>/<pubkey>.json # hardlink to current winner
├── addressable/<kind>/<pubkey>/<sha256(d)>.json # hardlink to current winner
│
└── tombstones/
├── id/<id> # hardlink to the kind-5 event
├── addr/<kind>/<pubkey>/<sha256(d)> # hardlink to kind-5
└── vanish/<pubkey_hash> # hardlink to kind-62
```
### Filename conventions
- **Event files:** `<id>.json`, where `<id>` is the 64-hex event id.
Content is the raw NIP-01 JSON (same bytes the signer produced).
- **Timestamp prefix:** zero-padded 10-digit unix seconds
(`0001713960000`) so lexicographic sort = chronological sort.
- **Hashes:** SHA-256 of the canonical bytes, hex-lowercased. Used
for d-tags and arbitrary tag values (sanitizes user content for
filesystem safety).
- **Owner hash:** `TagNameValueHasher.hash(ownerPubkey)` (MurmurHash3
salted by `.seed`). Matches SQLite's `pubkey_owner_hash` semantics
— for GiftWraps it is the recipient's p-tag, not the event pubkey.
(`EventIndexesModule.kt:161-166`)
### Sharding
- `events/<aa>/<bb>/` = 65 536 leaf directories max. Keeps any leaf
under ~a few hundred files for 10 M events. Piggybacks on ext4
htree / APFS B-tree / NTFS index.
- `idx/*/…/` no sharding; query paths pick one index tree and walk
its subdirectory directly (`idx/kind/1/`, `idx/author/<pk>/`).
- Long pubkeys / ids are already 64 hex (32 bytes). No length issues.
### File modes
- Event files: `444` (read-only). Matches SQLite's "immutable tables"
invariant: triggers block `UPDATE`
(`EventIndexesModule.kt:105-111`). We achieve the same at the OS
layer.
- Directories: `755`. User can still `rm` (write on parent dir only).
- Tombstones: `444` (they *are* event files via hardlink).
---
## Feature parity matrix
| SQLite feature | File-store mechanism | Enforced by |
|---|---|---|
| `event_headers` | `events/<aa>/<bb>/<id>.json` | Write-then-rename atomicity |
| `event_tags` | `idx/tag/<n>/<h>/<ts>-<id>` hardlinks | Set on insert; orphans tolerated |
| `event_fts` | `idx/fts/<token>/<id>` hardlinks | Tokenize `SearchableEvent.indexableContent()` |
| `event_expirations` | `idx/expires_at/<ts>-<id>` hardlink | Swept by `deleteExpiredEvents()` |
| `event_vanish` | `tombstones/vanish/<owner_hash>` hardlink | Checked on insert |
| `seeds` | `.seed` file; 8 random bytes | Written once on create |
| `UNIQUE(kind,pubkey)` for replaceable | `replaceable/<k>/<pk>.json` slot | Directory-entry uniqueness + atomic rename |
| `UNIQUE(kind,pubkey,d)` for addressable | `addressable/<k>/<pk>/<h>.json` slot | Same |
| Reject ephemeral | Code guard before any file write | `if (event.kind.isEphemeral()) return` |
| Reject expired on insert | Code guard before any file write | `if (event.isExpired()) throw` |
| NIP-09 delete by id | Unlink `events/.../<id>.json` + all hardlinks | Tombstone blocks re-insert |
| NIP-09 delete by address | Unlink addressable slot + canonical | Tombstone blocks re-insert |
| NIP-09 gift-wrap by p-tag | `idx/owner/<owner_hash>/` walk | Owner hash matches recipient for GiftWrap |
| NIP-62 vanish cascade | Walk `idx/owner/<owner_hash>/` and unlink | AFTER-insert cascade |
| NIP-62 block future | Check `tombstones/vanish/<owner_hash>` | Before every insert |
| NIP-40 deleteExpired sweep | Walk `idx/expires_at/` until now, unlink | Called by CLI cron |
| NIP-45 count | Same planner as query, count results | Streaming |
| NIP-50 search | Intersect `idx/fts/<token>/` sets | Tokenizer matches Quartz |
| NIP-91 tag AND | Sorted-list intersection of `idx/tag/<n>/<h>/` | Streaming `comm -12` equivalent |
| `transaction {}` | `flock(.lock) + stage + commit + unlock` | All writes to staging dir, single rename |
| `vacuum()` | `amy store scrub` — walks `events/`, rebuilds `idx/` etc. | Offline maintenance |
| `analyse()` | No-op; FS already maintains its own indexes | — |
---
## Public API
Implements `com.vitorpamplona.quartz.nip01Core.store.IEventStore`
(`quartz/.../store/IEventStore.kt:26-60`) in full.
```kotlin
class FsEventStore(
root: Path,
relayUrl: String, // matches SQLiteEventStore ctor; used for NIP-62 scoping
indexingStrategy: IndexingStrategy = DefaultIndexingStrategy,
clock: Clock = Clock.systemUTC(), // injectable for tests
) : IEventStore {
// IEventStore
override fun insert(event: Event)
override fun transaction(body: IEventStore.ITransaction.() -> Unit)
override fun <T : Event> query(filter: Filter): List<T>
override fun <T : Event> query(filters: List<Filter>): List<T>
override fun <T : Event> query(filter: Filter, onEach: (T) -> Unit)
override fun <T : Event> query(filters: List<Filter>, onEach: (T) -> Unit)
override fun count(filter: Filter): Int
override fun count(filters: List<Filter>): Int
override fun delete(filter: Filter)
override fun delete(filters: List<Filter>)
override fun deleteExpiredEvents()
override fun close()
// FsEventStore extras (parity with SQLiteEventStore)
fun delete(id: HexKey): Int
fun rawQuery(filter: Filter): List<RawEvent> // parses JSON to RawEvent
fun rawQuery(filters: List<Filter>): List<RawEvent>
fun planQuery(filter: Filter): String // returns human-readable plan
fun planQuery(filters: List<Filter>): String
// Maintenance
suspend fun scrub() // rebuild all derived state
suspend fun compact() // equivalent to vacuum — drops orphans
}
```
Signer/event conveniences mirror SQLite's — no behavioural drift.
---
## File-backed counterparts to SQLite pragmas
| SQLite pragma | File-store analog |
|---|---|
| `journal_mode=WAL` | Staging dir `.staging/<uuid>/` + atomic rename on commit |
| `synchronous=OFF` | Default = no `fsync` per write; `AMY_FSYNC=1` env opts in |
| `cache_size=-32000` | None — rely on OS page cache |
| `foreign_keys=ON` | Implicit: hardlinks are the FK; refcount is the cascade |
| `BEGIN IMMEDIATE` | `FileChannel.lock(.lock)` exclusive advisory lock |
---
## CLI commands (later; out of scope for this plan)
Anticipated but not implemented here:
```
amy store import <path> # from relay / file / stdin
amy store export [filter] # to stdout NDJSON
amy store query <filter-json> # debug
amy store scrub # rebuild derived state
amy store compact # drop orphans
amy store stat # counts, disk usage
```
Added in a follow-up roadmap row once the store itself is in place.
---
## Open questions (flagged for review before coding)
1. **Tag index granularity.** SQLite hashes `(name, value)` into one
`Long`. We could keep the same hash scheme and use the hex of the
Murmur hash as the directory name, giving byte-for-byte parity
with SQLite's index semantics. Proposed: yes — reuse
`TagNameValueHasher`.
2. **Filename collisions on d-tag.** SHA-256 is safe; Murmur is not.
Use SHA-256 for d-tag slots specifically.
3. **Relay URL in constructor.** `SQLiteEventStore` takes a
`relayUrlIdentifier` for NIP-62 scoping (`shouldVanishFrom`). We
mirror that; same semantics.
4. **Empty tombstone semantics.** If a tombstone hardlink is deleted
by the user, enforcement stops for that target. Scrub does **not**
re-create tombstones from lingering kind-5 events in `events/` —
removing the tombstone is treated as an explicit "un-forget"
action. Confirm with user.
---
Next: pipelines
(`2026-04-24-file-event-store-pipelines.md`) — insert, query, delete,
transaction, concurrency.
cli/plans/2026-04-24-file-event-store-pipelines.md
+326
View File
@@ -0,0 +1,326 @@
# File-backed event store for `amy` — Part 2: Pipelines
**Status:** plan · **Date:** 2026-04-24 · **Part 2 of 3**
(see `2026-04-24-file-event-store-overview.md`,
`2026-04-24-file-event-store-nips.md`)
Covers insert, query, delete, transaction, and concurrency. NIP-
specific enforcement (09/40/50/62) and replaceable/addressable slot
mechanics are in Part 3.
---
## Insert pipeline
Mirrors `SQLiteEventStore.insertEvent()`
(`quartz/.../sqlite/SQLiteEventStore.kt:189-196`) and
`innerInsertEvent()` (lines 178-187). Module ordering is identical
— only the backing representation changes.
### Pre-transaction guards (no files touched)
```
1. if (event.kind.isEphemeral()) return // EphemeralModule
2. if (event.isExpired()) throw ExpiredEventException // ExpirationModule
```
Both match SQLite behaviour exactly — rejected before any I/O.
### Transaction body
Acquire `FileChannel.lock(.lock)` for the whole body. Release only
on COMMIT or ROLLBACK.
```
T1. Stage event file
stagingDir = .staging/<random>/
write stagingDir/event.json (event bytes, fsync if AMY_FSYNC=1)
T2. Pre-insert veto checks (read-only)
• TombstoneIdCheck: tombstones/id/<event.id> exists? → REJECT
• TombstoneAddrCheck: (addressable only) tombstones/addr/<k>/<pk>/<hash(d)>
exists AND Tdel >= event.createdAt? → REJECT
• VanishCheck: tombstones/vanish/<owner_hash> exists AND
Tvanish >= event.createdAt? → REJECT
• ReplaceableGuard: (kinds 0,3,10000-19999) replaceable/<k>/<pk>.json
exists AND its createdAt >= event.createdAt? → REJECT
• AddressableGuard: (kinds 30000-39999) addressable/<k>/<pk>/<hash(d)>.json
exists AND its createdAt >= event.createdAt? → REJECT
Any REJECT → discard staging dir, release lock, return.
T3. Canonical write
canonical = events/<id[0:2]>/<id[2:4]>/<id>.json
Files.move(stagingDir/event.json, canonical, ATOMIC_MOVE)
Files.setLastModifiedTime(canonical, event.createdAt seconds)
Files.setPosixFilePermissions(canonical, r--r--r--)
T4. Derived-index hardlinks (all ln(canonical → X))
• idx/kind/<k>/<ts>-<id>
• idx/author/<pk>/<ts>-<id>
• idx/owner/<owner_hash>/<ts>-<id> # recipient for GiftWrap
• for each indexed tag (name, value):
idx/tag/<name>/<murmur_hex>/<ts>-<id>
• if event.expiration() > 0:
idx/expires_at/<exp_ts>-<id>
• if event implements SearchableEvent:
for each token in tokenize(event.indexableContent()):
idx/fts/<token>/<id>
T5. Replaceable / Addressable slot (if applicable)
slotTmp = <slot>.tmp.<random>
ln canonical slotTmp
Files.move(slotTmp, slot, ATOMIC_MOVE, REPLACE_EXISTING)
# The rename atomically deposes the old winner and installs new.
if oldSlot had different event id:
unlink events/<old_id>.json # drop canonical
walk idx/... and unlink entries for old id # reuses same logic as delete()
T6. Deletion-event side effects (if event.kind == 5)
Apply NIP-09 actions — see Part 3. Executes *inside* the lock.
T7. Vanish-event side effects (if event.kind == 62 && shouldVanishFrom(relay))
Apply NIP-62 cascade — see Part 3.
T8. Commit
Release .lock. Remove emptied staging dir.
```
### Crash safety at every step
| Crash between | Post-crash state | Scrub action |
|---|---|---|
| T1 and T3 | Only staging dir exists | Delete `.staging/` on next open |
| T3 and T4 | Canonical written, no indexes | Rebuild indexes by walking `events/` |
| T4 (partial) | Some indexes written | Rebuild indexes (idempotent) |
| T5 (partial) | Slot tmp hardlink exists | Remove `<slot>.tmp.*` on next open |
| T5 (after rename) | Old winner still has canonical | Slot takes precedence; old canonical GC'd by scrub |
| T6/T7 (partial) | Some targets unlinked, some not | Re-apply kind-5/62 during scrub |
The kernel guarantees `rename(2)` atomicity; `link(2)` either
succeeds fully or fails. All failure modes are forward-recoverable
by scrub — never corrupting.
### Ordering vs SQLite
SQLite runs triggers *before* our code inserts. We run checks
before writing the canonical (T2), then indexes (T4), then slot
enforcement (T5), then kind-5 / kind-62 side effects (T6/T7). Same
overall guarantees, different sequencing because we have no trigger
mechanism.
---
## Transaction API
```kotlin
override fun transaction(body: ITransaction.() -> Unit) {
withLock {
val txn = FsTransaction(this)
try {
txn.body() // accumulates staged events
txn.commit() // all T3..T7 for every staged event, then one unlock
} catch (e: Throwable) {
txn.rollback() // delete staging dirs, do nothing else
throw e
}
}
}
```
Semantics:
- Single `flock` for the whole transaction.
- Staging dirs collected; canonical moves happen at commit time.
- If any event in the batch is rejected by T2, the whole txn aborts.
- Matches SQLite's all-or-nothing behaviour
(`SQLiteConnectionExt.kt:71-81`).
No readers are blocked: they don't hold the lock and see either
pre-commit or post-commit state (atomic rename).
---
## Query pipeline
Matches the planner structure of `QueryBuilder`
(`quartz/.../sqlite/QueryBuilder.kt:110-172`) but emits directory
walks instead of SQL.
### Filter normalisation
Same as SQLite: `filter.toFilterWithDTags()` splits the d-tag out of
the generic tag map, flags `isSimpleQuery` / `isSimpleSearch`.
### Plan selection
For each filter, choose one driver index based on cardinality and
availability:
| Filter contents | Driver index | Rationale |
|---|---|---|
| Only `ids` | direct `events/<aa>/<bb>/<id>.json` opens | O(1) per id |
| `kinds` present, no tags | `idx/kind/<k>/` | Smallest per-kind fanout |
| `authors` present, no kinds, no tags | `idx/author/<pk>/` | Author-scoped |
| `kinds` + `authors` + no tags | Pick the smaller of `idx/kind/` vs `idx/author/` by listing size | Parity with SQLite's `query_by_kind_pubkey_created` choice |
| Any `#tag` filter | `idx/tag/<n>/<h>/` — intersect multiple if AND | Matches SQLite's per-tag INNER JOINs |
| Only `search` | `idx/fts/<token>/` — intersect tokens | Parity with FTS MATCH |
| Only `since`/`until` / `limit` | `idx/kind/*/` or full `events/` walk ordered by mtime | See below |
| `d_tag` present (addressable) | `addressable/<k>/<pk>/<hash(d)>.json` direct open | Slot lookup — O(1) |
Chosen driver produces a **sorted stream of `(createdAt, id)` pairs**
in DESC order (lexicographic sort over zero-padded filenames reversed).
Filenames are `<ts>-<id>`, so `Files.list(dir).sorted(reverseOrder())`
is the plan.
### Secondary filters (post-driver)
Anything not covered by the driver is applied in code, event by
event, after opening the JSON:
- `since <= createdAt <= until` (already a prefix filter on the driver)
- `authors` / `kinds` / `ids` if not the driver
- d-tag match (addressable)
- additional tag filters
- search tokens (if driver wasn't FTS)
- tombstone exclusions (id + address)
- vanish exclusions (by owner_hash)
- expiration exclusions (`createdAt < now < expiration`)
### Tag AND (NIP-91)
For `filter.tagsAll = {#e = [a, b], #p = [c]}`:
```
streamA = idx/tag/e/<hash(a)>/ (sorted by <ts>-<id> DESC)
streamB = idx/tag/e/<hash(b)>/
streamC = idx/tag/p/<hash(c)>/
result = k-way intersection by <id> suffix, preserving DESC order
```
Implemented as a sorted-list intersection (Java `Iterator<String>`
merge). Equivalent to SQLite's INNER JOIN per tag
(`QueryBuilder.kt:434-449`). Bounded memory.
### Multi-filter UNION
`query(filters: List<Filter>)` runs each filter's plan, then merges
the resulting streams in DESC order with dedup by id. Per-filter
`limit` applied before merge, as SQLite does
(`QueryBuilder.kt:577-583`).
### Streaming mode
`query(filter, onEach: (T) -> Unit)` calls `onEach` as each event is
read and parsed. No intermediate list. Matches SQLite's cursor
callback.
### `count()`
Same plan, but open nothing — just count filenames that survive the
post-driver code filters. For filters that require JSON inspection
(tag exact match on non-indexed tags, search tokens), we still open
files; cost scales with matched set size, not store size.
### `planQuery()` output
Plain text, like `EXPLAIN QUERY PLAN`:
```
FILTER 0
DRIVER: idx/kind/1/ (12,400 entries)
POST: authors=[<pk>], since >= 1713960000, limit 100
FILTER 1
DRIVER: idx/tag/e/ab12…/ ∩ idx/tag/p/cd34…/ (est 37 hits)
POST: kinds=[30023], limit 10
MERGE: 2 filters, DESC by created_at, overall limit 110
```
---
## Delete pipeline
### `delete(filter)` / `delete(filters)`
Runs the query pipeline to enumerate matching events, then for each:
```
1. Read canonical, parse tags (needed to know all hardlink paths).
2. Unlink every derived hardlink:
idx/kind/<k>/<ts>-<id>
idx/author/<pk>/<ts>-<id>
idx/owner/<owner_hash>/<ts>-<id>
idx/tag/<n>/<h>/<ts>-<id> (for each indexed tag)
idx/expires_at/<exp>-<id> (if exp present)
idx/fts/<token>/<id> (for each FTS token)
3. If event is the winner of a replaceable/addressable slot:
unlink the slot path.
4. Unlink canonical events/<aa>/<bb>/<id>.json.
5. kernel GC's the inode (no other refs).
```
All inside `flock`. Matches SQLite's
`DELETE FROM event_headers WHERE row_id IN (...)` with FK cascade.
### `delete(id: HexKey): Int`
Direct shortcut. Opens `events/<aa>/<bb>/<id>.json`, runs the same
steps 1-5. Returns `1` on success, `0` if the file didn't exist —
matching the SQLite return value from `changes()`
(`SQLiteEventStore.kt:261-264`).
### `deleteExpiredEvents()`
```
now = clock.now().epochSecond
for entry in idx/expires_at/ where <exp> < now:
delete(entry.id)
```
Optional optimisation: because filenames are `<exp>-<id>`, listing is
already sorted, so we can stop at the first entry with `<exp> >= now`.
---
## Concurrency model
### Single writer, many readers
- Writers hold an exclusive `flock(.lock)` via `FileChannel.tryLock()`.
- Readers take a shared lock only while listing top-level directory
contents that may be mid-rename; most reads hold no lock (kernel-
atomic `open` / `readdir` is enough).
- Two concurrent `amy` invocations serialise cleanly on the flock;
the second one blocks on `FileChannel.lock()` (blocking) or fails
fast with `tryLock()` (configurable).
### Reader–writer interaction
Because every writer mutation is a rename-over-tmp, readers see
either the pre-mutation file or the post-mutation file, never a
torn view. Dangling reads (writer unlinks a file between
`readdir` and `open`) manifest as `NoSuchFileException` and are
treated as "event not in store anymore" — we skip it and keep going.
### Watch / subscribe (future)
`java.nio.file.WatchService` on `events/` gives per-platform
inotify/FSEvents/ReadDirectoryChangesW. Lets a future
`amy store tail` stream newly arriving events to stdout without
polling. Not part of this plan but the layout supports it
natively.
---
## Close / reopen
- `close()` releases any held `FileChannel` and flushes nothing
(writes are already durable as of each rename, modulo the
`AMY_FSYNC` option).
- Reopen is free: the store is stateless. No header page, no WAL
checkpoint. First action after open: scan `.staging/` and delete
leftovers (crash from previous run).
---
Next: NIP specifics and enforcement details
(`2026-04-24-file-event-store-nips.md`).