amethyst/cli/plans/2026-04-24-file-event-store-nips.md

File-backed event store for amy — Part 3: NIPs, Enforcement, Scrub, Testing

Status: plan · Date: 2026-04-24 · Part 3 of 3 (see 2026-04-24-file-event-store-overview.md, 2026-04-24-file-event-store-pipelines.md)

Per-NIP behavior, slot enforcement details, tag indexing, scrub, and test plan.

---

NIP-01 — Replaceable events (kinds 0, 3, 10000–19999)

Enforcement

• Slot path: replaceable/<kind>/<pubkey>.json
• Slot is a hardlink into events/<aa>/<bb>/<id>.json — same inode, two names.
• Directory-entry uniqueness is the UNIQUE(kind, pubkey) constraint (parity with ReplaceableModule.kt:25-60).

Insert flow (inside transaction T2/T5 from Part 2)

guard = ReplaceableGuard
   if slot exists:
       Told = read slot JSON → createdAt
       if event.createdAt <= Told: REJECT   # blocks older from reinsert
install (at T5):
   slotTmp = <slot>.tmp.<rand>
   ln events/<new_id>.json slotTmp
   Files.move(slotTmp, slot, ATOMIC_MOVE, REPLACE_EXISTING)
   # Old slot's inode loses one link. If it had a canonical hardlink too,
   # drop it — mirror SQLite trigger that deletes the old row.
   if oldId present:
       runDeleteLinksFor(oldId)  # same code path as delete()

Reading the current version

cat replaceable/3/<pubkey>.json        # latest kind-3 for pubkey

Direct path open. No scan, no MAX — the slot is the winner.

Edge cases

• Race between two newer events. Both pass guard, both try to install; last rename wins atomically. Losing event's canonical exists orphaned until scrub or until someone queries by its id. Semantically valid: both were newer than the previous slot.
• Slot deleted by user. Next insert of (kind, pubkey) has no guard and wins. This matches "files come and go" contract.
• Canonical deleted but slot survives. Slot still resolves — hardlink keeps inode alive. Queries work. Scrub re-creates the canonical from slot.

---

NIP-01 — Addressable events (kinds 30000–39999)

Enforcement

• Slot path: addressable/<kind>/<pubkey>/<sha256(d_tag)>.json
• SHA-256 of d-tag value avoids filesystem-unsafe characters and collisions. Parity with AddressableModule.kt:25-63.
• Empty d-tag is valid Nostr — hashed as sha256("").

Flow

Same as replaceable, keyed by (kind, pubkey, d_tag). Guard reads slot's createdAt; install via rename REPLACE_EXISTING; cascade- unlink old canonical and its index hardlinks.

Query shortcut

If a filter supplies kinds, authors, and a d-tag, the planner bypasses the driver indexes and opens the slot directly. O(1).

---

NIP-09 — Event deletion (kind 5)

Deletion event insert

When a kind-5 event is inserted (inside T6 of the pipeline):

for target in event.deleteEventIds():
    # NIP-09: same author as the deletion
    targetPath = events/<aa>/<bb>/<target>.json
    if targetPath exists AND read(targetPath).pubkey == event.pubkey:
        runDeleteLinksFor(target)     # cascade unlink

    # Install tombstone (hardlink to the kind-5 event):
    tomb = tombstones/id/<target>
    ln event_canonical <tomb>.tmp
    Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING_IF_NEWER)

for addr in event.deleteAddresses():
    hash = sha256(addr.dTag)
    slot = addressable/<addr.kind>/<addr.pubkey>/<hash>.json
    if slot exists AND read(slot).pubkey == event.pubkey:
        Told = read slot.createdAt
        if Told <= event.createdAt:
            oldId = read slot.id
            unlink slot
            runDeleteLinksFor(oldId)

    tomb = tombstones/addr/<addr.kind>/<addr.pubkey>/<hash>
    ln event_canonical <tomb>.tmp
    Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING_IF_NEWER)

for replaceable_kind in event.deleteReplaceableKinds():
    # kinds 0,3,10000-19999 — no d-tag
    slot = replaceable/<kind>/<event.pubkey>.json
    similar cascade + tombstone

REPLACE_EXISTING_IF_NEWER is a helper we implement: if an older kind-5 already owns the tombstone, and the new one has a later createdAt, swap it in; otherwise keep the existing. Ensures tombstone holds the strongest (latest) deletion cutoff, matching SQLite's "OR" of all deletion events (DeletionRequestModule.kt:102-184).

Blocking re-insertion

Part of T2 guards:

TombstoneIdCheck:
    if tombstones/id/<event.id> exists → REJECT

TombstoneAddrCheck (addressable only):
    if tombstones/addr/<k>/<pk>/<hash(d)> exists:
        Tdel = read tombstone.createdAt
        if event.createdAt <= Tdel → REJECT

Parity with DeletionRequestModule.kt:61-75 (BEFORE INSERT trigger rejecting events whose etag_hash/atag_hash match a kind-5 entry).

Gift-wrap deletion by p-tag

EventIndexesModule.kt:161-166 uses pubkey_owner_hash that, for GiftWrapEvent, is the recipient's p-tag (not the outer pubkey). We mirror this: idx/owner/<owner_hash>/<ts>-<id> points at every event where ownerHash = TagNameValueHasher.hash(recipient).

A kind-5 from the recipient deleting by e-tag still resolves via runDeleteLinksFor(id), which walks all hardlinks (including idx/owner/) and unlinks them.

Security: author check

NIP-09 requires deletions be authored by the same pubkey as the deleted event. SQLite enforces this via pubkey_owner_hash in the DELETE WHERE clause (DeletionRequestModule.kt:107-115). We mirror by reading the target's JSON and comparing pubkey — done inside runDeleteLinksFor.

---

NIP-40 — Expiration

Read at insert

event.expiration() returns Long? (tag value as unix seconds).

if exp != null && exp > 0:
    if exp <= now: throw ExpiredEventException  # T1 guard, mirrors trigger
    ln canonical idx/expires_at/<zero_padded(exp)>-<id>

Parity with ExpirationModule.kt:27-95.

Sweep

deleteExpiredEvents() listed in Part 2. Uses the sorted idx/expires_at/ — early termination when <exp> ≥ now.

Recommended caller: cron or shell loop.

while :; do amy store sweep-expired; sleep 900; done

Matches the ExpirationWorker WorkManager pattern from the SQLite README.

Query-time filtering

Belt-and-suspenders: queries also filter out events where exp != null && exp <= now. Covers sweep-missed windows and clock skew between writer and reader.

---

NIP-62 — Right to vanish (kind 62)

Insert

if !event.shouldVanishFrom(relayUrl): return   # same scoping as SQLite

ownerHash = TagNameValueHasher.hash(event.pubkey)

# Install vanish tombstone (one per pubkey; latest wins)
tomb = tombstones/vanish/<ownerHash>
existing = read tomb if exists
if existing == null || existing.createdAt < event.createdAt:
    ln canonical <tomb>.tmp
    Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING)

# Cascade: delete every event from this owner with createdAt < tomb.createdAt
for entry in idx/owner/<ownerHash>/:
    (ts, id) = parse entry filename
    if ts < event.createdAt:
        runDeleteLinksFor(id)

Parity with RightToVanishModule.kt:46-71.

Blocking future inserts

Part of T2 guards (VanishCheck):

tomb = tombstones/vanish/<TagNameValueHasher.hash(event.pubkey)>
if tomb exists:
    Tvanish = read tomb.createdAt
    if event.createdAt <= Tvanish → REJECT

For GiftWraps, owner is the recipient — a vanish request from the recipient cascades their received gift-wraps too, matching SQLite's behaviour via pubkey_owner_hash.

---

NIP-45 — Count

count(filter) / count(filters) runs the exact query plan from Part 2 and returns the number of surviving entries. Parity with SQLite SELECT count(*).

Optimisation: if the filter is driver-only (no post-driver code filter), we just count directory entries — Files.list().count() — without opening any JSON. Linear in the driver's listing size.

---

NIP-50 — Full-text search

Tokenisation

Reuse Quartz's SearchableEvent.indexableContent() (per FullTextSearchModule.kt:66-78). Tokeniser:

• Lowercase.
• Split on Unicode word boundaries.
• Filter tokens shorter than 3 chars (FTS5 default).
• Keep ASCII hex ids and bech32 identifiers whole (no splitting on underscores/digits).

Match the Android FTS5 tokeniser behaviour; a shared helper lives in commons/ and is called by both backends to guarantee identical search results.

Index

idx/fts/<token>/<id>       # empty hardlink into canonical

One empty hardlink per (token, event). For a 1000-word note with 500 unique tokens, 500 hardlinks — each costs one directory entry (~~50 bytes on ext4), no inode. At scale this is the biggest directory count; idx/fts/ alone may hold millions of entries. Still fine — each token dir stays small.

Search query

tokens = tokenize(filter.search)
streams = tokens.map { idx/fts/<it>/ }
candidates = k-way intersection of streams by <id>
for candidate in candidates:
    if passes all other filter predicates:
        emit

Same semantics as SQLite MATCH in FTS5 AND-mode. Matches QueryBuilder.kt:590-657.

Maintenance

• On insert: write token hardlinks (T4).
• On delete (any path — direct, replaceable replace, vanish, expiration): runDeleteLinksFor(id) enumerates every possible token and unlinks. We don't store the token list, so we re-derive it by reading the canonical content before unlinking the canonical. If the canonical is already gone, we fall back to find -samefile to enumerate remaining hardlinks.

Without an index

amy store search --raw "keywords" can fall back to ripgrep events/ for forensic use. Always works, even if indexes are corrupt.

---

Tag indexing

Which tags get indexed

Matches DefaultIndexingStrategy (IndexingStrategy.kt:99-102): tag arrays with at least 2 elements where tag[0].length == 1 (single-letter tag names). Non-single- letter tags are stored in the JSON and visible to queries but not reverse-indexed.

Hash scheme

Reuse TagNameValueHasher (quartz/.../sqlite/TagNameValueHasher.kt:30-67). Reads .seed into a Long salt identically to SQLite's seeds table. The hash becomes a 16-char hex string that names the directory:

idx/tag/<name>/<hex(hash(name, value))>/<ts>-<id>

Parity with event_tags.tag_hash. d-tag, e-tag, a-tag, p-tag, owner: see below.

Special tags

Tag	Where indexed	Notes
d	Only as part of addressable/<k>/<pk>/<sha256(d)>.json slot	Not in idx/tag/d/
e	idx/tag/e/<hash>/	Normal
a	idx/tag/a/<hash>/ + addressable slot look-aside	Normal
p	idx/tag/p/<hash>/	Recipient for GiftWrap also goes to idx/owner/
owner (synthetic)	idx/owner/<owner_hash>/	Event pubkey for normal events, recipient for GiftWrap — matches pubkey_owner_hash (EventIndexesModule.kt:161-166)

---

Seed file

.seed is 8 random bytes generated on store creation. Read into a Long salt for TagNameValueHasher, matching SQLite's seeds table (SeedModule.kt:26-84).

• Written atomically (tmp-rename) on first open if absent.
• Never updated. Corrupt or mismatched seeds invalidate idx/ — scrub rebuilds from canonical.
• Permissions 400.

---

Scrub / rebuild

FsEventStore.scrub():

1. Lock .lock exclusively.
2. Delete .staging/ leftovers.
3. Enumerate all events/<aa>/<bb>/<id>.json.
4. For each, re-derive:
      idx/kind/, idx/author/, idx/owner/, idx/tag/*/,
      idx/fts/*/, idx/expires_at/
   using the same code as T4. createLink is idempotent — NOACT on
   collision (or unlink + recreate).
5. Walk idx/ trees; drop entries whose target inode count is zero
   (dangling — canonical was deleted).
6. Rebuild replaceable/ and addressable/ slots from canonicals by
   grouping (kind,pubkey[,d]) and picking max createdAt.
7. Keep tombstones as-is. (Removing a tombstone file is a user-
   intended action; scrub does not resurrect them.)
8. Release lock.

Cost: O(N) disk reads once per scrub. Acceptable as a manual amy store scrub command. Not run automatically.

compact()

Lighter: walk idx/ only, drop dangling entries. No canonical reads. O(size of idx trees).

---

Testing strategy

Location

quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/ (sibling to the SQLite reference tests). Uses kotlin.test (matches the rest of the quartz jvmTest source set). Temp-dir fixture creates a fresh store per test.

Test categories

1. Parity tests. Drive both SQLiteEventStore and FsEventStore with identical event streams and filters; assert query results match. Existing SQLite tests under quartz/src/jvmTest/.../sqlite/ become the golden set. This is the highest-value tier.

2. Per-module tests. One file per enforcement area:

   • ReplaceableSlotTest — newer wins, older rejected, race deposes both cleanly.
   • AddressableSlotTest — same, plus d-tag hash collisions cannot occur (SHA-256).
   • DeletionTombstoneTest — by id, by address, by replaceable kind; re-insert blocked; gift-wrap by owner.
   • VanishTest — cascade + block, GiftWrap recipient.
   • ExpirationTest — sweep, early termination, belt-and- suspenders query filter.
   • SearchTest — tokeniser parity with SQLite FTS, AND of tokens, maintenance across replace/delete/expire/vanish.
   • TagAndTest — NIP-91 intersection, three-tag filters.
   • EphemeralTest — never written.
   • TransactionTest — all-or-nothing batch insert, rollback on exception.

3. Crash-safety tests. Inject failures after each pipeline step (T1–T8) via a testing hook; reopen; assert scrub converges.

4. Concurrency tests. Two threads / two processes hitting the same store: assert no corruption, writes serialise, readers see consistent snapshots.

5. External edit tests. Mid-run, delete a canonical file / a slot / a tombstone; assert store behaves as spec'd (dangling index skipped; slot rebuilds; tombstone stops enforcing).

6. Property tests. For a random sequence of (insert, delete, query, vanish, expire, replace) operations, assert the file store and SQLite store produce identical query() results. Seed = test parameter.

Running

./gradlew :commons:jvmTest --tests "*.store.fs.*"

Target: all tests pass before the CLI starts depending on the new store.

---

Rollout plan

1. Step 1 — skeleton. FsEventStore class, directory layout, insert + query(ids only) + delete(id). Round-trip test.
2. Step 2 — indexes. kind/author/owner/tag hardlinks; query by kind, author, tag; count.
3. Step 3 — slots. Replaceable + addressable enforcement + tests.
4. Step 4 — tombstones. NIP-09 deletion by id and address; block re-insert; gift-wrap via owner.
5. Step 5 — expirations. idx/expires_at/, deleteExpiredEvents.
6. Step 6 — vanish. NIP-62 insert + cascade + block.
7. Step 7 — FTS. Tokeniser in commons/, idx/fts/, search queries.
8. Step 8 — transactions, concurrency, scrub.
9. Step 9 — wire into amy. Context gains a store: IEventStore property; default = FsEventStore rooted at $AMY_HOME/events/.
10. Step 10 — parity test matrix driven by the existing SQLite golden tests.

Each step is a separate commit. Each is behind AMY_EXPERIMENTAL_ STORE=1 env guard until step 10 passes.

---

Out of scope for this plan

• Android integration — stays on SQLite.
• Desktop opt-in — separate plan once CLI is stable.
• Cross-host sync — users can rsync the directory; that's it.
• Encryption at rest — future plan; would wrap each event file.

---

Summary of plan split

File	Covers
2026-04-24-file-event-store-overview.md	Goals, layout, feature matrix, API
2026-04-24-file-event-store-pipelines.md	Insert, query, delete, transaction, concurrency
2026-04-24-file-event-store-nips.md (this file)	Per-NIP enforcement, slots, tombstones, FTS, scrub, testing