docs(cli): plan a file-backed event store for amy
Three-part design doc for an IEventStore backed by a directory tree instead of SQLite, targeted at the cli/ module. - overview: goals, directory layout, feature-parity matrix, public API - pipelines: insert (T1-T8 with crash-safety), query planner, delete, transactions, concurrency - nips: replaceable/addressable slot enforcement via hardlinks + atomic rename, NIP-09 tombstones (hardlink to the kind-5), NIP-40 expirations, NIP-50 FTS via inverted-index hardlinks, NIP-62 vanish cascade, tag indexing, seed file, scrub/compact, test strategy, 10-step rollout
This commit is contained in:
Claude
@@ -0,0 +1,500 @@
|
||||
# File-backed event store for `amy` — Part 3: NIPs, Enforcement, Scrub, Testing
|
||||
|
||||
**Status:** plan · **Date:** 2026-04-24 · **Part 3 of 3**
|
||||
(see `2026-04-24-file-event-store-overview.md`,
|
||||
`2026-04-24-file-event-store-pipelines.md`)
|
||||
|
||||
Per-NIP behavior, slot enforcement details, tag indexing, scrub, and
|
||||
test plan.
|
||||
|
||||
---
|
||||
|
||||
## NIP-01 — Replaceable events (kinds 0, 3, 10000–19999)
|
||||
|
||||
### Enforcement
|
||||
|
||||
- Slot path: `replaceable/<kind>/<pubkey>.json`
|
||||
- Slot is a **hardlink** into `events/<aa>/<bb>/<id>.json` — same
|
||||
inode, two names.
|
||||
- Directory-entry uniqueness is the `UNIQUE(kind, pubkey)` constraint
|
||||
(parity with `ReplaceableModule.kt:25-60`).
|
||||
|
||||
### Insert flow (inside transaction T2/T5 from Part 2)
|
||||
|
||||
```
|
||||
guard = ReplaceableGuard
|
||||
if slot exists:
|
||||
Told = read slot JSON → createdAt
|
||||
if event.createdAt <= Told: REJECT # blocks older from reinsert
|
||||
install (at T5):
|
||||
slotTmp = <slot>.tmp.<rand>
|
||||
ln events/<new_id>.json slotTmp
|
||||
Files.move(slotTmp, slot, ATOMIC_MOVE, REPLACE_EXISTING)
|
||||
# Old slot's inode loses one link. If it had a canonical hardlink too,
|
||||
# drop it — mirror SQLite trigger that deletes the old row.
|
||||
if oldId present:
|
||||
runDeleteLinksFor(oldId) # same code path as delete()
|
||||
```
|
||||
|
||||
### Reading the current version
|
||||
|
||||
```
|
||||
cat replaceable/3/<pubkey>.json # latest kind-3 for pubkey
|
||||
```
|
||||
|
||||
Direct path open. No scan, no MAX — the slot is the winner.
|
||||
|
||||
### Edge cases
|
||||
|
||||
- **Race between two newer events.** Both pass guard, both try to
|
||||
install; last `rename` wins atomically. Losing event's canonical
|
||||
exists orphaned until scrub or until someone queries by its id.
|
||||
Semantically valid: both were newer than the previous slot.
|
||||
- **Slot deleted by user.** Next insert of (kind, pubkey) has no
|
||||
guard and wins. This matches "files come and go" contract.
|
||||
- **Canonical deleted but slot survives.** Slot still resolves —
|
||||
hardlink keeps inode alive. Queries work. Scrub re-creates the
|
||||
canonical from slot.
|
||||
|
||||
---
|
||||
|
||||
## NIP-01 — Addressable events (kinds 30000–39999)
|
||||
|
||||
### Enforcement
|
||||
|
||||
- Slot path: `addressable/<kind>/<pubkey>/<sha256(d_tag)>.json`
|
||||
- SHA-256 of d-tag value avoids filesystem-unsafe characters and
|
||||
collisions. Parity with `AddressableModule.kt:25-63`.
|
||||
- Empty d-tag is valid Nostr — hashed as `sha256("")`.
|
||||
|
||||
### Flow
|
||||
|
||||
Same as replaceable, keyed by (kind, pubkey, d_tag). Guard reads
|
||||
slot's createdAt; install via `rename REPLACE_EXISTING`; cascade-
|
||||
unlink old canonical and its index hardlinks.
|
||||
|
||||
### Query shortcut
|
||||
|
||||
If a filter supplies `kinds`, `authors`, and a d-tag, the planner
|
||||
bypasses the driver indexes and opens the slot directly. O(1).
|
||||
|
||||
---
|
||||
|
||||
## NIP-09 — Event deletion (kind 5)
|
||||
|
||||
### Deletion event insert
|
||||
|
||||
When a kind-5 event is inserted (inside T6 of the pipeline):
|
||||
|
||||
```
|
||||
for target in event.deleteEventIds():
|
||||
# NIP-09: same author as the deletion
|
||||
targetPath = events/<aa>/<bb>/<target>.json
|
||||
if targetPath exists AND read(targetPath).pubkey == event.pubkey:
|
||||
runDeleteLinksFor(target) # cascade unlink
|
||||
|
||||
# Install tombstone (hardlink to the kind-5 event):
|
||||
tomb = tombstones/id/<target>
|
||||
ln event_canonical <tomb>.tmp
|
||||
Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING_IF_NEWER)
|
||||
|
||||
for addr in event.deleteAddresses():
|
||||
hash = sha256(addr.dTag)
|
||||
slot = addressable/<addr.kind>/<addr.pubkey>/<hash>.json
|
||||
if slot exists AND read(slot).pubkey == event.pubkey:
|
||||
Told = read slot.createdAt
|
||||
if Told <= event.createdAt:
|
||||
oldId = read slot.id
|
||||
unlink slot
|
||||
runDeleteLinksFor(oldId)
|
||||
|
||||
tomb = tombstones/addr/<addr.kind>/<addr.pubkey>/<hash>
|
||||
ln event_canonical <tomb>.tmp
|
||||
Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING_IF_NEWER)
|
||||
|
||||
for replaceable_kind in event.deleteReplaceableKinds():
|
||||
# kinds 0,3,10000-19999 — no d-tag
|
||||
slot = replaceable/<kind>/<event.pubkey>.json
|
||||
similar cascade + tombstone
|
||||
```
|
||||
|
||||
`REPLACE_EXISTING_IF_NEWER` is a helper we implement: if an older
|
||||
kind-5 already owns the tombstone, and the new one has a later
|
||||
createdAt, swap it in; otherwise keep the existing. Ensures
|
||||
tombstone holds the strongest (latest) deletion cutoff, matching
|
||||
SQLite's "OR" of all deletion events
|
||||
(`DeletionRequestModule.kt:102-184`).
|
||||
|
||||
### Blocking re-insertion
|
||||
|
||||
Part of T2 guards:
|
||||
|
||||
```
|
||||
TombstoneIdCheck:
|
||||
if tombstones/id/<event.id> exists → REJECT
|
||||
|
||||
TombstoneAddrCheck (addressable only):
|
||||
if tombstones/addr/<k>/<pk>/<hash(d)> exists:
|
||||
Tdel = read tombstone.createdAt
|
||||
if event.createdAt <= Tdel → REJECT
|
||||
```
|
||||
|
||||
Parity with `DeletionRequestModule.kt:61-75` (BEFORE INSERT trigger
|
||||
rejecting events whose etag_hash/atag_hash match a kind-5 entry).
|
||||
|
||||
### Gift-wrap deletion by p-tag
|
||||
|
||||
`EventIndexesModule.kt:161-166` uses `pubkey_owner_hash` that, for
|
||||
`GiftWrapEvent`, is the recipient's p-tag (not the outer pubkey).
|
||||
We mirror this: `idx/owner/<owner_hash>/<ts>-<id>` points at every
|
||||
event where `ownerHash = TagNameValueHasher.hash(recipient)`.
|
||||
|
||||
A kind-5 from the recipient deleting by `e`-tag still resolves via
|
||||
`runDeleteLinksFor(id)`, which walks all hardlinks (including
|
||||
`idx/owner/`) and unlinks them.
|
||||
|
||||
### Security: author check
|
||||
|
||||
NIP-09 requires deletions be authored by the same pubkey as the
|
||||
deleted event. SQLite enforces this via `pubkey_owner_hash` in the
|
||||
DELETE WHERE clause (`DeletionRequestModule.kt:107-115`). We mirror
|
||||
by reading the target's JSON and comparing `pubkey` — done inside
|
||||
`runDeleteLinksFor`.
|
||||
|
||||
---
|
||||
|
||||
## NIP-40 — Expiration
|
||||
|
||||
### Read at insert
|
||||
|
||||
`event.expiration()` returns `Long?` (tag value as unix seconds).
|
||||
|
||||
```
|
||||
if exp != null && exp > 0:
|
||||
if exp <= now: throw ExpiredEventException # T1 guard, mirrors trigger
|
||||
ln canonical idx/expires_at/<zero_padded(exp)>-<id>
|
||||
```
|
||||
|
||||
Parity with `ExpirationModule.kt:27-95`.
|
||||
|
||||
### Sweep
|
||||
|
||||
`deleteExpiredEvents()` listed in Part 2. Uses the sorted
|
||||
`idx/expires_at/` — early termination when `<exp>` ≥ now.
|
||||
|
||||
Recommended caller: cron or shell loop.
|
||||
|
||||
```sh
|
||||
while :; do amy store sweep-expired; sleep 900; done
|
||||
```
|
||||
|
||||
Matches the `ExpirationWorker` WorkManager pattern from the SQLite
|
||||
README.
|
||||
|
||||
### Query-time filtering
|
||||
|
||||
Belt-and-suspenders: queries also filter out events where
|
||||
`exp != null && exp <= now`. Covers sweep-missed windows and
|
||||
clock skew between writer and reader.
|
||||
|
||||
---
|
||||
|
||||
## NIP-62 — Right to vanish (kind 62)
|
||||
|
||||
### Insert
|
||||
|
||||
```
|
||||
if !event.shouldVanishFrom(relayUrl): return # same scoping as SQLite
|
||||
|
||||
ownerHash = TagNameValueHasher.hash(event.pubkey)
|
||||
|
||||
# Install vanish tombstone (one per pubkey; latest wins)
|
||||
tomb = tombstones/vanish/<ownerHash>
|
||||
existing = read tomb if exists
|
||||
if existing == null || existing.createdAt < event.createdAt:
|
||||
ln canonical <tomb>.tmp
|
||||
Files.move(<tomb>.tmp, tomb, ATOMIC_MOVE, REPLACE_EXISTING)
|
||||
|
||||
# Cascade: delete every event from this owner with createdAt < tomb.createdAt
|
||||
for entry in idx/owner/<ownerHash>/:
|
||||
(ts, id) = parse entry filename
|
||||
if ts < event.createdAt:
|
||||
runDeleteLinksFor(id)
|
||||
```
|
||||
|
||||
Parity with `RightToVanishModule.kt:46-71`.
|
||||
|
||||
### Blocking future inserts
|
||||
|
||||
Part of T2 guards (VanishCheck):
|
||||
|
||||
```
|
||||
tomb = tombstones/vanish/<TagNameValueHasher.hash(event.pubkey)>
|
||||
if tomb exists:
|
||||
Tvanish = read tomb.createdAt
|
||||
if event.createdAt <= Tvanish → REJECT
|
||||
```
|
||||
|
||||
For GiftWraps, owner is the recipient — a vanish request from the
|
||||
recipient cascades their received gift-wraps too, matching SQLite's
|
||||
behaviour via `pubkey_owner_hash`.
|
||||
|
||||
---
|
||||
|
||||
## NIP-45 — Count
|
||||
|
||||
`count(filter)` / `count(filters)` runs the exact query plan from
|
||||
Part 2 and returns the number of surviving entries. Parity with
|
||||
SQLite `SELECT count(*)`.
|
||||
|
||||
Optimisation: if the filter is driver-only (no post-driver code
|
||||
filter), we just count directory entries — `Files.list().count()` —
|
||||
without opening any JSON. Linear in the driver's listing size.
|
||||
|
||||
---
|
||||
|
||||
## NIP-50 — Full-text search
|
||||
|
||||
### Tokenisation
|
||||
|
||||
Reuse Quartz's `SearchableEvent.indexableContent()` (per
|
||||
`FullTextSearchModule.kt:66-78`). Tokeniser:
|
||||
|
||||
- Lowercase.
|
||||
- Split on Unicode word boundaries.
|
||||
- Filter tokens shorter than 3 chars (FTS5 default).
|
||||
- Keep ASCII hex ids and bech32 identifiers whole (no splitting on
|
||||
underscores/digits).
|
||||
|
||||
Match the Android FTS5 tokeniser behaviour; a shared helper lives
|
||||
in `commons/` and is called by both backends to guarantee identical
|
||||
search results.
|
||||
|
||||
### Index
|
||||
|
||||
```
|
||||
idx/fts/<token>/<id> # empty hardlink into canonical
|
||||
```
|
||||
|
||||
One empty hardlink per (token, event). For a 1000-word note with
|
||||
500 unique tokens, 500 hardlinks — each costs one directory entry
|
||||
(~~50 bytes on ext4), no inode. At scale this is the biggest
|
||||
directory count; `idx/fts/` alone may hold millions of entries.
|
||||
Still fine — each token dir stays small.
|
||||
|
||||
### Search query
|
||||
|
||||
```
|
||||
tokens = tokenize(filter.search)
|
||||
streams = tokens.map { idx/fts/<it>/ }
|
||||
candidates = k-way intersection of streams by <id>
|
||||
for candidate in candidates:
|
||||
if passes all other filter predicates:
|
||||
emit
|
||||
```
|
||||
|
||||
Same semantics as SQLite `MATCH` in FTS5 AND-mode. Matches
|
||||
`QueryBuilder.kt:590-657`.
|
||||
|
||||
### Maintenance
|
||||
|
||||
- On insert: write token hardlinks (T4).
|
||||
- On delete (any path — direct, replaceable replace, vanish,
|
||||
expiration): `runDeleteLinksFor(id)` enumerates every possible
|
||||
token and unlinks. We don't store the token list, so we re-derive
|
||||
it by reading the canonical content *before* unlinking the
|
||||
canonical. If the canonical is already gone, we fall back to
|
||||
`find -samefile` to enumerate remaining hardlinks.
|
||||
|
||||
### Without an index
|
||||
|
||||
`amy store search --raw "keywords"` can fall back to
|
||||
`ripgrep events/` for forensic use. Always works, even if indexes
|
||||
are corrupt.
|
||||
|
||||
---
|
||||
|
||||
## Tag indexing
|
||||
|
||||
### Which tags get indexed
|
||||
|
||||
Matches `DefaultIndexingStrategy`
|
||||
(`IndexingStrategy.kt:99-102`): tag arrays with at least 2 elements
|
||||
where `tag[0].length == 1` (single-letter tag names). Non-single-
|
||||
letter tags are stored in the JSON and visible to queries but not
|
||||
reverse-indexed.
|
||||
|
||||
### Hash scheme
|
||||
|
||||
Reuse `TagNameValueHasher`
|
||||
(`quartz/.../sqlite/TagNameValueHasher.kt:30-67`). Reads `.seed`
|
||||
into a `Long` salt identically to SQLite's `seeds` table. The hash
|
||||
becomes a 16-char hex string that names the directory:
|
||||
|
||||
```
|
||||
idx/tag/<name>/<hex(hash(name, value))>/<ts>-<id>
|
||||
```
|
||||
|
||||
Parity with `event_tags.tag_hash`. d-tag, e-tag, a-tag, p-tag,
|
||||
owner: see below.
|
||||
|
||||
### Special tags
|
||||
|
||||
| Tag | Where indexed | Notes |
|
||||
|---|---|---|
|
||||
| `d` | Only as part of `addressable/<k>/<pk>/<sha256(d)>.json` slot | Not in `idx/tag/d/` |
|
||||
| `e` | `idx/tag/e/<hash>/` | Normal |
|
||||
| `a` | `idx/tag/a/<hash>/` + addressable slot look-aside | Normal |
|
||||
| `p` | `idx/tag/p/<hash>/` | Recipient for GiftWrap also goes to `idx/owner/` |
|
||||
| owner (synthetic) | `idx/owner/<owner_hash>/` | Event pubkey for normal events, recipient for GiftWrap — matches `pubkey_owner_hash` (`EventIndexesModule.kt:161-166`) |
|
||||
|
||||
---
|
||||
|
||||
## Seed file
|
||||
|
||||
`.seed` is 8 random bytes generated on store creation. Read into a
|
||||
`Long` salt for `TagNameValueHasher`, matching SQLite's `seeds` table
|
||||
(`SeedModule.kt:26-84`).
|
||||
|
||||
- Written atomically (tmp-rename) on first open if absent.
|
||||
- Never updated. Corrupt or mismatched seeds invalidate `idx/` —
|
||||
scrub rebuilds from canonical.
|
||||
- Permissions `400`.
|
||||
|
||||
---
|
||||
|
||||
## Scrub / rebuild
|
||||
|
||||
`FsEventStore.scrub()`:
|
||||
|
||||
```
|
||||
1. Lock .lock exclusively.
|
||||
2. Delete .staging/ leftovers.
|
||||
3. Enumerate all events/<aa>/<bb>/<id>.json.
|
||||
4. For each, re-derive:
|
||||
idx/kind/, idx/author/, idx/owner/, idx/tag/*/,
|
||||
idx/fts/*/, idx/expires_at/
|
||||
using the same code as T4. createLink is idempotent — NOACT on
|
||||
collision (or unlink + recreate).
|
||||
5. Walk idx/ trees; drop entries whose target inode count is zero
|
||||
(dangling — canonical was deleted).
|
||||
6. Rebuild replaceable/ and addressable/ slots from canonicals by
|
||||
grouping (kind,pubkey[,d]) and picking max createdAt.
|
||||
7. Keep tombstones as-is. (Removing a tombstone file is a user-
|
||||
intended action; scrub does not resurrect them.)
|
||||
8. Release lock.
|
||||
```
|
||||
|
||||
Cost: O(N) disk reads once per scrub. Acceptable as a manual
|
||||
`amy store scrub` command. Not run automatically.
|
||||
|
||||
### `compact()`
|
||||
|
||||
Lighter: walk `idx/` only, drop dangling entries. No canonical
|
||||
reads. O(size of idx trees).
|
||||
|
||||
---
|
||||
|
||||
## Testing strategy
|
||||
|
||||
### Location
|
||||
|
||||
`commons/src/jvmTest/kotlin/com/vitorpamplona/commons/store/fs/`
|
||||
|
||||
Uses JUnit4 (matches rest of the project). Temp-dir fixture creates
|
||||
a fresh store per test.
|
||||
|
||||
### Test categories
|
||||
|
||||
1. **Parity tests.** Drive both `SQLiteEventStore` and `FsEventStore`
|
||||
with identical event streams and filters; assert query results
|
||||
match. Existing SQLite tests under
|
||||
`quartz/src/jvmTest/.../sqlite/` become the golden set. This is
|
||||
the highest-value tier.
|
||||
|
||||
2. **Per-module tests.** One file per enforcement area:
|
||||
- `ReplaceableSlotTest` — newer wins, older rejected, race
|
||||
deposes both cleanly.
|
||||
- `AddressableSlotTest` — same, plus d-tag hash collisions
|
||||
cannot occur (SHA-256).
|
||||
- `DeletionTombstoneTest` — by id, by address, by replaceable
|
||||
kind; re-insert blocked; gift-wrap by owner.
|
||||
- `VanishTest` — cascade + block, GiftWrap recipient.
|
||||
- `ExpirationTest` — sweep, early termination, belt-and-
|
||||
suspenders query filter.
|
||||
- `SearchTest` — tokeniser parity with SQLite FTS, AND of tokens,
|
||||
maintenance across replace/delete/expire/vanish.
|
||||
- `TagAndTest` — NIP-91 intersection, three-tag filters.
|
||||
- `EphemeralTest` — never written.
|
||||
- `TransactionTest` — all-or-nothing batch insert, rollback on
|
||||
exception.
|
||||
|
||||
3. **Crash-safety tests.** Inject failures after each pipeline step
|
||||
(T1–T8) via a testing hook; reopen; assert scrub converges.
|
||||
|
||||
4. **Concurrency tests.** Two threads / two processes hitting the
|
||||
same store: assert no corruption, writes serialise, readers see
|
||||
consistent snapshots.
|
||||
|
||||
5. **External edit tests.** Mid-run, delete a canonical file / a
|
||||
slot / a tombstone; assert store behaves as spec'd (dangling
|
||||
index skipped; slot rebuilds; tombstone stops enforcing).
|
||||
|
||||
6. **Property tests.** For a random sequence of (insert, delete,
|
||||
query, vanish, expire, replace) operations, assert the file
|
||||
store and SQLite store produce identical `query()` results.
|
||||
Seed = test parameter.
|
||||
|
||||
### Running
|
||||
|
||||
```
|
||||
./gradlew :commons:jvmTest --tests "*.store.fs.*"
|
||||
```
|
||||
|
||||
Target: all tests pass before the CLI starts depending on the new
|
||||
store.
|
||||
|
||||
---
|
||||
|
||||
## Rollout plan
|
||||
|
||||
1. **Step 1 — skeleton.** `FsEventStore` class, directory layout,
|
||||
`insert` + `query(ids only)` + `delete(id)`. Round-trip test.
|
||||
2. **Step 2 — indexes.** kind/author/owner/tag hardlinks; query by
|
||||
kind, author, tag; `count`.
|
||||
3. **Step 3 — slots.** Replaceable + addressable enforcement +
|
||||
tests.
|
||||
4. **Step 4 — tombstones.** NIP-09 deletion by id and address;
|
||||
block re-insert; gift-wrap via owner.
|
||||
5. **Step 5 — expirations.** `idx/expires_at/`, `deleteExpiredEvents`.
|
||||
6. **Step 6 — vanish.** NIP-62 insert + cascade + block.
|
||||
7. **Step 7 — FTS.** Tokeniser in `commons/`, `idx/fts/`, search
|
||||
queries.
|
||||
8. **Step 8 — transactions, concurrency, scrub.**
|
||||
9. **Step 9 — wire into `amy`.** `Context` gains a `store:
|
||||
IEventStore` property; default = `FsEventStore` rooted at
|
||||
`$AMY_HOME/events/`.
|
||||
10. **Step 10 — parity test matrix** driven by the existing SQLite
|
||||
golden tests.
|
||||
|
||||
Each step is a separate commit. Each is behind `AMY_EXPERIMENTAL_
|
||||
STORE=1` env guard until step 10 passes.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope for this plan
|
||||
|
||||
- Android integration — stays on SQLite.
|
||||
- Desktop opt-in — separate plan once CLI is stable.
|
||||
- Cross-host sync — users can rsync the directory; that's it.
|
||||
- Encryption at rest — future plan; would wrap each event file.
|
||||
|
||||
---
|
||||
|
||||
## Summary of plan split
|
||||
|
||||
| File | Covers |
|
||||
|---|---|
|
||||
| `2026-04-24-file-event-store-overview.md` | Goals, layout, feature matrix, API |
|
||||
| `2026-04-24-file-event-store-pipelines.md` | Insert, query, delete, transaction, concurrency |
|
||||
| `2026-04-24-file-event-store-nips.md` (this file) | Per-NIP enforcement, slots, tombstones, FTS, scrub, testing |
|
||||
@@ -0,0 +1,260 @@
|
||||
# File-backed event store for `amy` — Part 1: Overview & Layout
|
||||
|
||||
**Status:** plan · **Date:** 2026-04-24 · **Part 1 of 3**
|
||||
(see `2026-04-24-file-event-store-pipelines.md`,
|
||||
`2026-04-24-file-event-store-nips.md`)
|
||||
|
||||
A filesystem-native `IEventStore` for the `cli/` module that matches
|
||||
every feature of the SQLite store
|
||||
(`quartz/.../nip01Core/store/sqlite/`) while tolerating the fact that
|
||||
files may be created or deleted by the user between runs.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Full feature parity** with `SQLiteEventStore` — same `IEventStore`
|
||||
contract, same NIP semantics (01 replaceable/addressable, 09
|
||||
deletion, 40 expiration, 45 count, 50 search, 62 vanish, 91 multi-
|
||||
tag AND).
|
||||
2. **Human-inspectable.** Every event is a JSON file on disk. `ls`,
|
||||
`cat`, `jq`, `grep`, `rsync`, `git`, backup tools — all work.
|
||||
3. **Tolerates user edits.** If the user deletes an event file, the
|
||||
store converges: dangling index entries become no-ops, tombstones
|
||||
stop enforcing, replaceable slots get rebuilt from `events/`.
|
||||
4. **JVM-only, no JNI.** No SQLite, no native deps beyond what the
|
||||
CLI already carries. Single-user, single-host.
|
||||
5. **Small-to-medium scale.** Tens of thousands of events. Not a
|
||||
full Amethyst cache replacement.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Beating SQLite on throughput for 100k+ event workloads.
|
||||
- Network/replication (rsync is good enough).
|
||||
- Multi-tenant; concurrent writers across machines.
|
||||
- Incremental migration tooling from SQLite (not needed — `amy` has no
|
||||
existing persisted store).
|
||||
|
||||
---
|
||||
|
||||
## Guiding principles
|
||||
|
||||
1. **Filesystem primitives enforce invariants.** Directory-entry
|
||||
uniqueness = `UNIQUE` constraints. `rename(2)` = atomic commit.
|
||||
Hardlink refcount = cascade delete. `chmod 444` = immutable
|
||||
tables. `flock` = transaction serialization.
|
||||
2. **Hardlinks for all indexes.** An index entry is a second *name*
|
||||
for the canonical event file, never a copy. Deleting any name
|
||||
drops the refcount; when it hits zero the kernel reclaims.
|
||||
3. **Derived state is rebuildable.** `idx/`, `replaceable/`,
|
||||
`addressable/`, `tombstones/` can be regenerated by walking
|
||||
`events/`. `amy store scrub` does this.
|
||||
4. **`events/` is the source of truth.** If a file is there, it is
|
||||
part of the store. If it is gone, it is gone — tombstones aside.
|
||||
5. **No business logic.** This is a `commons/` module that the CLI
|
||||
imports. No Nostr-protocol decisions live in `cli/`.
|
||||
|
||||
---
|
||||
|
||||
## Module placement
|
||||
|
||||
Source: `commons/src/jvmMain/kotlin/com/vitorpamplona/commons/store/fs/`
|
||||
|
||||
- JVM-only (uses `java.nio.file`, `FileChannel.lock`, `Files.createLink`).
|
||||
- Consumed by `cli/`. Android keeps `SQLiteEventStore`. Desktop can
|
||||
opt in later if useful.
|
||||
- Tests under `commons/src/jvmTest/.../store/fs/`.
|
||||
|
||||
Not in `quartz/` because `quartz/` is protocol-only (no storage).
|
||||
Not in `cli/` because it contains reusable logic.
|
||||
|
||||
---
|
||||
|
||||
## Directory layout
|
||||
|
||||
Root: configurable, defaults to `$AMY_HOME/events/` (where `$AMY_HOME`
|
||||
is `~/.amy` by default).
|
||||
|
||||
```
|
||||
<root>/
|
||||
├── .lock # flock target for write serialization
|
||||
├── .seed # random Long for hash salting; write-once
|
||||
├── .version # schema version for future migrations
|
||||
│
|
||||
├── events/<aa>/<bb>/<id>.json # canonical. <aa><bb> = first 4 hex of id
|
||||
│
|
||||
├── idx/
|
||||
│ ├── kind/<kind>/<created_at>-<id> # hardlink → events/.../<id>.json
|
||||
│ ├── author/<pubkey>/<created_at>-<id> # hardlink
|
||||
│ ├── owner/<owner_hash>/<created_at>-<id> # hardlink; gift-wrap recipient
|
||||
│ ├── tag/<name>/<hash>/<created_at>-<id> # hardlink; single-letter tags
|
||||
│ └── fts/<token>/<id> # hardlink; inverted index
|
||||
│
|
||||
├── replaceable/<kind>/<pubkey>.json # hardlink to current winner
|
||||
├── addressable/<kind>/<pubkey>/<sha256(d)>.json # hardlink to current winner
|
||||
│
|
||||
└── tombstones/
|
||||
├── id/<id> # hardlink to the kind-5 event
|
||||
├── addr/<kind>/<pubkey>/<sha256(d)> # hardlink to kind-5
|
||||
└── vanish/<pubkey_hash> # hardlink to kind-62
|
||||
```
|
||||
|
||||
### Filename conventions
|
||||
|
||||
- **Event files:** `<id>.json`, where `<id>` is the 64-hex event id.
|
||||
Content is the raw NIP-01 JSON (same bytes the signer produced).
|
||||
- **Timestamp prefix:** zero-padded 10-digit unix seconds
|
||||
(`0001713960000`) so lexicographic sort = chronological sort.
|
||||
- **Hashes:** SHA-256 of the canonical bytes, hex-lowercased. Used
|
||||
for d-tags and arbitrary tag values (sanitizes user content for
|
||||
filesystem safety).
|
||||
- **Owner hash:** `TagNameValueHasher.hash(ownerPubkey)` (MurmurHash3
|
||||
salted by `.seed`). Matches SQLite's `pubkey_owner_hash` semantics
|
||||
— for GiftWraps it is the recipient's p-tag, not the event pubkey.
|
||||
(`EventIndexesModule.kt:161-166`)
|
||||
|
||||
### Sharding
|
||||
|
||||
- `events/<aa>/<bb>/` = 65 536 leaf directories max. Keeps any leaf
|
||||
under ~a few hundred files for 10 M events. Piggybacks on ext4
|
||||
htree / APFS B-tree / NTFS index.
|
||||
- `idx/*/…/` no sharding; query paths pick one index tree and walk
|
||||
its subdirectory directly (`idx/kind/1/`, `idx/author/<pk>/`).
|
||||
- Long pubkeys / ids are already 64 hex (32 bytes). No length issues.
|
||||
|
||||
### File modes
|
||||
|
||||
- Event files: `444` (read-only). Matches SQLite's "immutable tables"
|
||||
invariant: triggers block `UPDATE`
|
||||
(`EventIndexesModule.kt:105-111`). We achieve the same at the OS
|
||||
layer.
|
||||
- Directories: `755`. User can still `rm` (write on parent dir only).
|
||||
- Tombstones: `444` (they *are* event files via hardlink).
|
||||
|
||||
---
|
||||
|
||||
## Feature parity matrix
|
||||
|
||||
| SQLite feature | File-store mechanism | Enforced by |
|
||||
|---|---|---|
|
||||
| `event_headers` | `events/<aa>/<bb>/<id>.json` | Write-then-rename atomicity |
|
||||
| `event_tags` | `idx/tag/<n>/<h>/<ts>-<id>` hardlinks | Set on insert; orphans tolerated |
|
||||
| `event_fts` | `idx/fts/<token>/<id>` hardlinks | Tokenize `SearchableEvent.indexableContent()` |
|
||||
| `event_expirations` | `idx/expires_at/<ts>-<id>` hardlink | Swept by `deleteExpiredEvents()` |
|
||||
| `event_vanish` | `tombstones/vanish/<owner_hash>` hardlink | Checked on insert |
|
||||
| `seeds` | `.seed` file; 8 random bytes | Written once on create |
|
||||
| `UNIQUE(kind,pubkey)` for replaceable | `replaceable/<k>/<pk>.json` slot | Directory-entry uniqueness + atomic rename |
|
||||
| `UNIQUE(kind,pubkey,d)` for addressable | `addressable/<k>/<pk>/<h>.json` slot | Same |
|
||||
| Reject ephemeral | Code guard before any file write | `if (event.kind.isEphemeral()) return` |
|
||||
| Reject expired on insert | Code guard before any file write | `if (event.isExpired()) throw` |
|
||||
| NIP-09 delete by id | Unlink `events/.../<id>.json` + all hardlinks | Tombstone blocks re-insert |
|
||||
| NIP-09 delete by address | Unlink addressable slot + canonical | Tombstone blocks re-insert |
|
||||
| NIP-09 gift-wrap by p-tag | `idx/owner/<owner_hash>/` walk | Owner hash matches recipient for GiftWrap |
|
||||
| NIP-62 vanish cascade | Walk `idx/owner/<owner_hash>/` and unlink | AFTER-insert cascade |
|
||||
| NIP-62 block future | Check `tombstones/vanish/<owner_hash>` | Before every insert |
|
||||
| NIP-40 deleteExpired sweep | Walk `idx/expires_at/` until now, unlink | Called by CLI cron |
|
||||
| NIP-45 count | Same planner as query, count results | Streaming |
|
||||
| NIP-50 search | Intersect `idx/fts/<token>/` sets | Tokenizer matches Quartz |
|
||||
| NIP-91 tag AND | Sorted-list intersection of `idx/tag/<n>/<h>/` | Streaming `comm -12` equivalent |
|
||||
| `transaction {}` | `flock(.lock) + stage + commit + unlock` | All writes to staging dir, single rename |
|
||||
| `vacuum()` | `amy store scrub` — walks `events/`, rebuilds `idx/` etc. | Offline maintenance |
|
||||
| `analyse()` | No-op; FS already maintains its own indexes | — |
|
||||
|
||||
---
|
||||
|
||||
## Public API
|
||||
|
||||
Implements `com.vitorpamplona.quartz.nip01Core.store.IEventStore`
|
||||
(`quartz/.../store/IEventStore.kt:26-60`) in full.
|
||||
|
||||
```kotlin
|
||||
class FsEventStore(
|
||||
root: Path,
|
||||
relayUrl: String, // matches SQLiteEventStore ctor; used for NIP-62 scoping
|
||||
indexingStrategy: IndexingStrategy = DefaultIndexingStrategy,
|
||||
clock: Clock = Clock.systemUTC(), // injectable for tests
|
||||
) : IEventStore {
|
||||
|
||||
// IEventStore
|
||||
override fun insert(event: Event)
|
||||
override fun transaction(body: IEventStore.ITransaction.() -> Unit)
|
||||
override fun <T : Event> query(filter: Filter): List<T>
|
||||
override fun <T : Event> query(filters: List<Filter>): List<T>
|
||||
override fun <T : Event> query(filter: Filter, onEach: (T) -> Unit)
|
||||
override fun <T : Event> query(filters: List<Filter>, onEach: (T) -> Unit)
|
||||
override fun count(filter: Filter): Int
|
||||
override fun count(filters: List<Filter>): Int
|
||||
override fun delete(filter: Filter)
|
||||
override fun delete(filters: List<Filter>)
|
||||
override fun deleteExpiredEvents()
|
||||
override fun close()
|
||||
|
||||
// FsEventStore extras (parity with SQLiteEventStore)
|
||||
fun delete(id: HexKey): Int
|
||||
fun rawQuery(filter: Filter): List<RawEvent> // parses JSON to RawEvent
|
||||
fun rawQuery(filters: List<Filter>): List<RawEvent>
|
||||
fun planQuery(filter: Filter): String // returns human-readable plan
|
||||
fun planQuery(filters: List<Filter>): String
|
||||
|
||||
// Maintenance
|
||||
suspend fun scrub() // rebuild all derived state
|
||||
suspend fun compact() // equivalent to vacuum — drops orphans
|
||||
}
|
||||
```
|
||||
|
||||
Signer/event conveniences mirror SQLite's — no behavioural drift.
|
||||
|
||||
---
|
||||
|
||||
## File-backed counterparts to SQLite pragmas
|
||||
|
||||
| SQLite pragma | File-store analog |
|
||||
|---|---|
|
||||
| `journal_mode=WAL` | Staging dir `.staging/<uuid>/` + atomic rename on commit |
|
||||
| `synchronous=OFF` | Default = no `fsync` per write; `AMY_FSYNC=1` env opts in |
|
||||
| `cache_size=-32000` | None — rely on OS page cache |
|
||||
| `foreign_keys=ON` | Implicit: hardlinks are the FK; refcount is the cascade |
|
||||
| `BEGIN IMMEDIATE` | `FileChannel.lock(.lock)` exclusive advisory lock |
|
||||
|
||||
---
|
||||
|
||||
## CLI commands (later; out of scope for this plan)
|
||||
|
||||
Anticipated but not implemented here:
|
||||
|
||||
```
|
||||
amy store import <path> # from relay / file / stdin
|
||||
amy store export [filter] # to stdout NDJSON
|
||||
amy store query <filter-json> # debug
|
||||
amy store scrub # rebuild derived state
|
||||
amy store compact # drop orphans
|
||||
amy store stat # counts, disk usage
|
||||
```
|
||||
|
||||
Added in a follow-up roadmap row once the store itself is in place.
|
||||
|
||||
---
|
||||
|
||||
## Open questions (flagged for review before coding)
|
||||
|
||||
1. **Tag index granularity.** SQLite hashes `(name, value)` into one
|
||||
`Long`. We could keep the same hash scheme and use the hex of the
|
||||
Murmur hash as the directory name, giving byte-for-byte parity
|
||||
with SQLite's index semantics. Proposed: yes — reuse
|
||||
`TagNameValueHasher`.
|
||||
2. **Filename collisions on d-tag.** SHA-256 is safe; Murmur is not.
|
||||
Use SHA-256 for d-tag slots specifically.
|
||||
3. **Relay URL in constructor.** `SQLiteEventStore` takes a
|
||||
`relayUrlIdentifier` for NIP-62 scoping (`shouldVanishFrom`). We
|
||||
mirror that; same semantics.
|
||||
4. **Empty tombstone semantics.** If a tombstone hardlink is deleted
|
||||
by the user, enforcement stops for that target. Scrub does **not**
|
||||
re-create tombstones from lingering kind-5 events in `events/` —
|
||||
removing the tombstone is treated as an explicit "un-forget"
|
||||
action. Confirm with user.
|
||||
|
||||
---
|
||||
|
||||
Next: pipelines
|
||||
(`2026-04-24-file-event-store-pipelines.md`) — insert, query, delete,
|
||||
transaction, concurrency.
|
||||
@@ -0,0 +1,326 @@
|
||||
# File-backed event store for `amy` — Part 2: Pipelines
|
||||
|
||||
**Status:** plan · **Date:** 2026-04-24 · **Part 2 of 3**
|
||||
(see `2026-04-24-file-event-store-overview.md`,
|
||||
`2026-04-24-file-event-store-nips.md`)
|
||||
|
||||
Covers insert, query, delete, transaction, and concurrency. NIP-
|
||||
specific enforcement (09/40/50/62) and replaceable/addressable slot
|
||||
mechanics are in Part 3.
|
||||
|
||||
---
|
||||
|
||||
## Insert pipeline
|
||||
|
||||
Mirrors `SQLiteEventStore.insertEvent()`
|
||||
(`quartz/.../sqlite/SQLiteEventStore.kt:189-196`) and
|
||||
`innerInsertEvent()` (lines 178-187). Module ordering is identical
|
||||
— only the backing representation changes.
|
||||
|
||||
### Pre-transaction guards (no files touched)
|
||||
|
||||
```
|
||||
1. if (event.kind.isEphemeral()) return // EphemeralModule
|
||||
2. if (event.isExpired()) throw ExpiredEventException // ExpirationModule
|
||||
```
|
||||
|
||||
Both match SQLite behaviour exactly — rejected before any I/O.
|
||||
|
||||
### Transaction body
|
||||
|
||||
Acquire `FileChannel.lock(.lock)` for the whole body. Release only
|
||||
on COMMIT or ROLLBACK.
|
||||
|
||||
```
|
||||
T1. Stage event file
|
||||
stagingDir = .staging/<random>/
|
||||
write stagingDir/event.json (event bytes, fsync if AMY_FSYNC=1)
|
||||
|
||||
T2. Pre-insert veto checks (read-only)
|
||||
• TombstoneIdCheck: tombstones/id/<event.id> exists? → REJECT
|
||||
• TombstoneAddrCheck: (addressable only) tombstones/addr/<k>/<pk>/<hash(d)>
|
||||
exists AND Tdel >= event.createdAt? → REJECT
|
||||
• VanishCheck: tombstones/vanish/<owner_hash> exists AND
|
||||
Tvanish >= event.createdAt? → REJECT
|
||||
• ReplaceableGuard: (kinds 0,3,10000-19999) replaceable/<k>/<pk>.json
|
||||
exists AND its createdAt >= event.createdAt? → REJECT
|
||||
• AddressableGuard: (kinds 30000-39999) addressable/<k>/<pk>/<hash(d)>.json
|
||||
exists AND its createdAt >= event.createdAt? → REJECT
|
||||
Any REJECT → discard staging dir, release lock, return.
|
||||
|
||||
T3. Canonical write
|
||||
canonical = events/<id[0:2]>/<id[2:4]>/<id>.json
|
||||
Files.move(stagingDir/event.json, canonical, ATOMIC_MOVE)
|
||||
Files.setLastModifiedTime(canonical, event.createdAt seconds)
|
||||
Files.setPosixFilePermissions(canonical, r--r--r--)
|
||||
|
||||
T4. Derived-index hardlinks (all ln(canonical → X))
|
||||
• idx/kind/<k>/<ts>-<id>
|
||||
• idx/author/<pk>/<ts>-<id>
|
||||
• idx/owner/<owner_hash>/<ts>-<id> # recipient for GiftWrap
|
||||
• for each indexed tag (name, value):
|
||||
idx/tag/<name>/<murmur_hex>/<ts>-<id>
|
||||
• if event.expiration() > 0:
|
||||
idx/expires_at/<exp_ts>-<id>
|
||||
• if event implements SearchableEvent:
|
||||
for each token in tokenize(event.indexableContent()):
|
||||
idx/fts/<token>/<id>
|
||||
|
||||
T5. Replaceable / Addressable slot (if applicable)
|
||||
slotTmp = <slot>.tmp.<random>
|
||||
ln canonical slotTmp
|
||||
Files.move(slotTmp, slot, ATOMIC_MOVE, REPLACE_EXISTING)
|
||||
# The rename atomically deposes the old winner and installs new.
|
||||
if oldSlot had different event id:
|
||||
unlink events/<old_id>.json # drop canonical
|
||||
walk idx/... and unlink entries for old id # reuses same logic as delete()
|
||||
|
||||
T6. Deletion-event side effects (if event.kind == 5)
|
||||
Apply NIP-09 actions — see Part 3. Executes *inside* the lock.
|
||||
|
||||
T7. Vanish-event side effects (if event.kind == 62 && shouldVanishFrom(relay))
|
||||
Apply NIP-62 cascade — see Part 3.
|
||||
|
||||
T8. Commit
|
||||
Release .lock. Remove emptied staging dir.
|
||||
```
|
||||
|
||||
### Crash safety at every step
|
||||
|
||||
| Crash between | Post-crash state | Scrub action |
|
||||
|---|---|---|
|
||||
| T1 and T3 | Only staging dir exists | Delete `.staging/` on next open |
|
||||
| T3 and T4 | Canonical written, no indexes | Rebuild indexes by walking `events/` |
|
||||
| T4 (partial) | Some indexes written | Rebuild indexes (idempotent) |
|
||||
| T5 (partial) | Slot tmp hardlink exists | Remove `<slot>.tmp.*` on next open |
|
||||
| T5 (after rename) | Old winner still has canonical | Slot takes precedence; old canonical GC'd by scrub |
|
||||
| T6/T7 (partial) | Some targets unlinked, some not | Re-apply kind-5/62 during scrub |
|
||||
|
||||
The kernel guarantees `rename(2)` atomicity; `link(2)` either
|
||||
succeeds fully or fails. All failure modes are forward-recoverable
|
||||
by scrub — never corrupting.
|
||||
|
||||
### Ordering vs SQLite
|
||||
|
||||
SQLite runs triggers *before* our code inserts. We run checks
|
||||
before writing the canonical (T2), then indexes (T4), then slot
|
||||
enforcement (T5), then kind-5 / kind-62 side effects (T6/T7). Same
|
||||
overall guarantees, different sequencing because we have no trigger
|
||||
mechanism.
|
||||
|
||||
---
|
||||
|
||||
## Transaction API
|
||||
|
||||
```kotlin
|
||||
override fun transaction(body: ITransaction.() -> Unit) {
|
||||
withLock {
|
||||
val txn = FsTransaction(this)
|
||||
try {
|
||||
txn.body() // accumulates staged events
|
||||
txn.commit() // all T3..T7 for every staged event, then one unlock
|
||||
} catch (e: Throwable) {
|
||||
txn.rollback() // delete staging dirs, do nothing else
|
||||
throw e
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Semantics:
|
||||
|
||||
- Single `flock` for the whole transaction.
|
||||
- Staging dirs collected; canonical moves happen at commit time.
|
||||
- If any event in the batch is rejected by T2, the whole txn aborts.
|
||||
- Matches SQLite's all-or-nothing behaviour
|
||||
(`SQLiteConnectionExt.kt:71-81`).
|
||||
|
||||
No readers are blocked: they don't hold the lock and see either
|
||||
pre-commit or post-commit state (atomic rename).
|
||||
|
||||
---
|
||||
|
||||
## Query pipeline
|
||||
|
||||
Matches the planner structure of `QueryBuilder`
|
||||
(`quartz/.../sqlite/QueryBuilder.kt:110-172`) but emits directory
|
||||
walks instead of SQL.
|
||||
|
||||
### Filter normalisation
|
||||
|
||||
Same as SQLite: `filter.toFilterWithDTags()` splits the d-tag out of
|
||||
the generic tag map, flags `isSimpleQuery` / `isSimpleSearch`.
|
||||
|
||||
### Plan selection
|
||||
|
||||
For each filter, choose one driver index based on cardinality and
|
||||
availability:
|
||||
|
||||
| Filter contents | Driver index | Rationale |
|
||||
|---|---|---|
|
||||
| Only `ids` | direct `events/<aa>/<bb>/<id>.json` opens | O(1) per id |
|
||||
| `kinds` present, no tags | `idx/kind/<k>/` | Smallest per-kind fanout |
|
||||
| `authors` present, no kinds, no tags | `idx/author/<pk>/` | Author-scoped |
|
||||
| `kinds` + `authors` + no tags | Pick the smaller of `idx/kind/` vs `idx/author/` by listing size | Parity with SQLite's `query_by_kind_pubkey_created` choice |
|
||||
| Any `#tag` filter | `idx/tag/<n>/<h>/` — intersect multiple if AND | Matches SQLite's per-tag INNER JOINs |
|
||||
| Only `search` | `idx/fts/<token>/` — intersect tokens | Parity with FTS MATCH |
|
||||
| Only `since`/`until` / `limit` | `idx/kind/*/` or full `events/` walk ordered by mtime | See below |
|
||||
| `d_tag` present (addressable) | `addressable/<k>/<pk>/<hash(d)>.json` direct open | Slot lookup — O(1) |
|
||||
|
||||
Chosen driver produces a **sorted stream of `(createdAt, id)` pairs**
|
||||
in DESC order (lexicographic sort over zero-padded filenames reversed).
|
||||
Filenames are `<ts>-<id>`, so `Files.list(dir).sorted(reverseOrder())`
|
||||
is the plan.
|
||||
|
||||
### Secondary filters (post-driver)
|
||||
|
||||
Anything not covered by the driver is applied in code, event by
|
||||
event, after opening the JSON:
|
||||
|
||||
- `since <= createdAt <= until` (already a prefix filter on the driver)
|
||||
- `authors` / `kinds` / `ids` if not the driver
|
||||
- d-tag match (addressable)
|
||||
- additional tag filters
|
||||
- search tokens (if driver wasn't FTS)
|
||||
- tombstone exclusions (id + address)
|
||||
- vanish exclusions (by owner_hash)
|
||||
- expiration exclusions (`createdAt < now < expiration`)
|
||||
|
||||
### Tag AND (NIP-91)
|
||||
|
||||
For `filter.tagsAll = {#e = [a, b], #p = [c]}`:
|
||||
|
||||
```
|
||||
streamA = idx/tag/e/<hash(a)>/ (sorted by <ts>-<id> DESC)
|
||||
streamB = idx/tag/e/<hash(b)>/
|
||||
streamC = idx/tag/p/<hash(c)>/
|
||||
result = k-way intersection by <id> suffix, preserving DESC order
|
||||
```
|
||||
|
||||
Implemented as a sorted-list intersection (Java `Iterator<String>`
|
||||
merge). Equivalent to SQLite's INNER JOIN per tag
|
||||
(`QueryBuilder.kt:434-449`). Bounded memory.
|
||||
|
||||
### Multi-filter UNION
|
||||
|
||||
`query(filters: List<Filter>)` runs each filter's plan, then merges
|
||||
the resulting streams in DESC order with dedup by id. Per-filter
|
||||
`limit` applied before merge, as SQLite does
|
||||
(`QueryBuilder.kt:577-583`).
|
||||
|
||||
### Streaming mode
|
||||
|
||||
`query(filter, onEach: (T) -> Unit)` calls `onEach` as each event is
|
||||
read and parsed. No intermediate list. Matches SQLite's cursor
|
||||
callback.
|
||||
|
||||
### `count()`
|
||||
|
||||
Same plan, but open nothing — just count filenames that survive the
|
||||
post-driver code filters. For filters that require JSON inspection
|
||||
(tag exact match on non-indexed tags, search tokens), we still open
|
||||
files; cost scales with matched set size, not store size.
|
||||
|
||||
### `planQuery()` output
|
||||
|
||||
Plain text, like `EXPLAIN QUERY PLAN`:
|
||||
|
||||
```
|
||||
FILTER 0
|
||||
DRIVER: idx/kind/1/ (12,400 entries)
|
||||
POST: authors=[<pk>], since >= 1713960000, limit 100
|
||||
FILTER 1
|
||||
DRIVER: idx/tag/e/ab12…/ ∩ idx/tag/p/cd34…/ (est 37 hits)
|
||||
POST: kinds=[30023], limit 10
|
||||
MERGE: 2 filters, DESC by created_at, overall limit 110
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delete pipeline
|
||||
|
||||
### `delete(filter)` / `delete(filters)`
|
||||
|
||||
Runs the query pipeline to enumerate matching events, then for each:
|
||||
|
||||
```
|
||||
1. Read canonical, parse tags (needed to know all hardlink paths).
|
||||
2. Unlink every derived hardlink:
|
||||
idx/kind/<k>/<ts>-<id>
|
||||
idx/author/<pk>/<ts>-<id>
|
||||
idx/owner/<owner_hash>/<ts>-<id>
|
||||
idx/tag/<n>/<h>/<ts>-<id> (for each indexed tag)
|
||||
idx/expires_at/<exp>-<id> (if exp present)
|
||||
idx/fts/<token>/<id> (for each FTS token)
|
||||
3. If event is the winner of a replaceable/addressable slot:
|
||||
unlink the slot path.
|
||||
4. Unlink canonical events/<aa>/<bb>/<id>.json.
|
||||
5. kernel GC's the inode (no other refs).
|
||||
```
|
||||
|
||||
All inside `flock`. Matches SQLite's
|
||||
`DELETE FROM event_headers WHERE row_id IN (...)` with FK cascade.
|
||||
|
||||
### `delete(id: HexKey): Int`
|
||||
|
||||
Direct shortcut. Opens `events/<aa>/<bb>/<id>.json`, runs the same
|
||||
steps 1-5. Returns `1` on success, `0` if the file didn't exist —
|
||||
matching the SQLite return value from `changes()`
|
||||
(`SQLiteEventStore.kt:261-264`).
|
||||
|
||||
### `deleteExpiredEvents()`
|
||||
|
||||
```
|
||||
now = clock.now().epochSecond
|
||||
for entry in idx/expires_at/ where <exp> < now:
|
||||
delete(entry.id)
|
||||
```
|
||||
|
||||
Optional optimisation: because filenames are `<exp>-<id>`, listing is
|
||||
already sorted, so we can stop at the first entry with `<exp> >= now`.
|
||||
|
||||
---
|
||||
|
||||
## Concurrency model
|
||||
|
||||
### Single writer, many readers
|
||||
|
||||
- Writers hold an exclusive `flock(.lock)` via `FileChannel.tryLock()`.
|
||||
- Readers take a shared lock only while listing top-level directory
|
||||
contents that may be mid-rename; most reads hold no lock (kernel-
|
||||
atomic `open` / `readdir` is enough).
|
||||
- Two concurrent `amy` invocations serialise cleanly on the flock;
|
||||
the second one blocks on `FileChannel.lock()` (blocking) or fails
|
||||
fast with `tryLock()` (configurable).
|
||||
|
||||
### Reader–writer interaction
|
||||
|
||||
Because every writer mutation is a rename-over-tmp, readers see
|
||||
either the pre-mutation file or the post-mutation file, never a
|
||||
torn view. Dangling reads (writer unlinks a file between
|
||||
`readdir` and `open`) manifest as `NoSuchFileException` and are
|
||||
treated as "event not in store anymore" — we skip it and keep going.
|
||||
|
||||
### Watch / subscribe (future)
|
||||
|
||||
`java.nio.file.WatchService` on `events/` gives per-platform
|
||||
inotify/FSEvents/ReadDirectoryChangesW. Lets a future
|
||||
`amy store tail` stream newly arriving events to stdout without
|
||||
polling. Not part of this plan but the layout supports it
|
||||
natively.
|
||||
|
||||
---
|
||||
|
||||
## Close / reopen
|
||||
|
||||
- `close()` releases any held `FileChannel` and flushes nothing
|
||||
(writes are already durable as of each rename, modulo the
|
||||
`AMY_FSYNC` option).
|
||||
- Reopen is free: the store is stateless. No header page, no WAL
|
||||
checkpoint. First action after open: scan `.staging/` and delete
|
||||
leftovers (crash from previous run).
|
||||
|
||||
---
|
||||
|
||||
Next: NIP specifics and enforcement details
|
||||
(`2026-04-24-file-event-store-nips.md`).
|
||||
Reference in New Issue
Block a user