feat(cli): amy store stat/sweep-expired/scrub/compact + e2e cache test

Three changes that go together:

1. Reconcile cli/plans/2026-04-24-file-event-store-{overview,nips}.md
   with shipped reality: code lives in quartz/jvmMain/, not commons/;
   data dir is <data-dir>/events-store/, not <root>/events/.

2. New StoreCommands wired as `amy store …`:
     - stat            → events count, kind histogram, disk bytes,
                         oldest/newest createdAt. Pure read, no Context
                         (skips identity check).
     - sweep-expired   → wraps store.deleteExpiredEvents(); reports
                         {swept, remaining}.
     - scrub           → wraps store.scrub() to rebuild idx/ from
                         canonicals.
     - compact         → wraps store.compact() to drop dangling idx/.
   All four open the FsEventStore directly (no Context, no identity
   needed) — they're store-only operations.

3. New e2e harness at cli/tests/cache/cache-headless.sh that boots a
   local nostr-rs-relay, two amy identities (A + B), and asserts:
     T1 — store stat reports non-empty store after publish-lists +
          profile edit, with kind:0 and kind:10002 present.
     T2 — A's `profile show` is `source: "cache"` by default.
     T3 — `--refresh` forces `source: "relays"`.
     T4 — B's first `profile show <A_NPUB>` is a relay miss; second is
          a cache hit (proves drain populates the local store and
          subsequent reads serve from disk).
     T5 — `relay list` reads URLs back from the local kind:10002 /
          10050 / 10051 events.
     T6 — relays.json no longer exists in either data-dir.
     T7 — store stat / sweep-expired / scrub / compact all run
          without an identity present.

Same pattern as cli/tests/dm/dm-interop-headless.sh — reuses the
nostr-rs-relay infrastructure from cli/tests/marmot/setup.sh.
This commit is contained in:
Claude
2026-04-25 04:10:10 +00:00
parent 99be0b2d16
commit 7bbfb52d87
6 changed files with 495 additions and 13 deletions
@@ -399,10 +399,10 @@ reads. O(size of idx trees).
### Location
`commons/src/jvmTest/kotlin/com/vitorpamplona/commons/store/fs/`
Uses JUnit4 (matches rest of the project). Temp-dir fixture creates
a fresh store per test.
`quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/`
(sibling to the SQLite reference tests). Uses kotlin.test (matches
the rest of the quartz `jvmTest` source set). Temp-dir fixture
creates a fresh store per test.
### Test categories
@@ -51,29 +51,36 @@ files may be created or deleted by the user between runs.
`events/`. `amy store scrub` does this.
4. **`events/` is the source of truth.** If a file is there, it is
part of the store. If it is gone, it is gone — tombstones aside.
5. **No business logic.** This is a `commons/` module that the CLI
imports. No Nostr-protocol decisions live in `cli/`.
5. **No business logic.** Lives next to the SQLite reference store
in `quartz/`, sibling pattern. No Nostr-protocol decisions live
in `cli/`.
---
## Module placement
Source: `commons/src/jvmMain/kotlin/com/vitorpamplona/commons/store/fs/`
Source: `quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/`
- Sibling to the existing
`quartz/src/commonMain/kotlin/.../nip01Core/store/sqlite/` reference
implementation. The plan originally proposed `commons/` but the
shipped placement is `quartz/jvmMain/`, where every other event-
store concern already lives — and quartz already has a `jvmTest`
source set so we get JVM-specific tests for free.
- JVM-only (uses `java.nio.file`, `FileChannel.lock`, `Files.createLink`).
- Consumed by `cli/`. Android keeps `SQLiteEventStore`. Desktop can
opt in later if useful.
- Tests under `commons/src/jvmTest/.../store/fs/`.
Not in `quartz/` because `quartz/` is protocol-only (no storage).
Not in `cli/` because it contains reusable logic.
- Tests under `quartz/src/jvmTest/.../store/fs/`.
---
## Directory layout
Root: configurable, defaults to `$AMY_HOME/events/` (where `$AMY_HOME`
is `~/.amy` by default).
Root: configurable. The CLI uses `<data-dir>/events-store/` (where
`<data-dir>` is `--data-dir PATH`, `$AMETHYST_CLI_DATA`, or `./amy`).
The plan originally said `events/`; the actual `DataDir.eventsDir`
field is `events-store/` to leave the bare name `events/` available
for the canonical-events subdirectory inside the store.
```
<root>/
@@ -139,6 +139,10 @@ private suspend fun dispatch(argv: Array<String>): Int {
Commands.notes(dataDir, tail)
}
"store" -> {
Commands.store(dataDir, tail)
}
else -> {
System.err.println("unknown subcommand: $head")
printUsage()
@@ -270,6 +274,12 @@ private fun printUsage() {
| marmot await epoch GID --min N
|
| marmot reset [--yes] wipe all local MLS/KeyPackage state (destructive)
|
|Local event store (`<data-dir>/events-store/`):
| store stat event count, kind histogram, disk usage
| store sweep-expired delete events past their NIP-40 expiration
| store scrub rebuild idx/ from canonical events (after edits / crashes)
| store compact drop dangling idx entries (canonical gone)
""".trimMargin(),
)
}
@@ -90,4 +90,9 @@ object Commands {
dataDir: DataDir,
tail: Array<String>,
): Int = NotesCommands.dispatch(dataDir, tail)
suspend fun store(
dataDir: DataDir,
tail: Array<String>,
): Int = StoreCommands.dispatch(dataDir, tail)
}
@@ -0,0 +1,208 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.cli.commands
import com.vitorpamplona.amethyst.cli.DataDir
import com.vitorpamplona.amethyst.cli.Json
import com.vitorpamplona.quartz.nip01Core.jackson.JacksonMapper
import com.vitorpamplona.quartz.nip01Core.store.fs.FsEventStore
import java.io.IOException
import java.nio.file.Files
import java.nio.file.Path
import java.util.concurrent.TimeUnit
import kotlin.io.path.exists
/**
* `amy store <stat|sweep-expired|scrub|compact>` — direct introspection
* and maintenance of the file-backed event store at
* `<data-dir>/events-store/`.
*
* - `stat` total event count, kind histogram, disk bytes,
* mtime range — pure read, no relay traffic.
* - `sweep-expired` delete events whose NIP-40 `expiration` tag has
* passed (per the store's own sweep logic). Run
* from cron / scheduler / `amy` periodically.
* - `scrub` rebuild every `idx/` entry from the canonical
* events. Recovers from partial-write crashes or
* external edits.
* - `compact` drop dangling `idx/` entries whose canonical is
* gone. Cheaper than scrub.
*/
object StoreCommands {
suspend fun dispatch(
dataDir: DataDir,
tail: Array<String>,
): Int {
if (tail.isEmpty()) return Json.error("bad_args", "store <stat|sweep-expired|scrub|compact>")
val rest = tail.drop(1).toTypedArray()
return when (tail[0]) {
"stat" -> stat(dataDir)
"sweep-expired" -> sweepExpired(dataDir)
"scrub" -> scrub(dataDir)
"compact" -> compact(dataDir)
else -> Json.error("bad_args", "store ${tail[0]}")
}
}
private fun stat(dataDir: DataDir): Int {
val storeRoot = dataDir.eventsDir.toPath()
if (!storeRoot.exists()) {
Json.writeLine(
mapOf(
"events" to 0,
"by_kind" to emptyMap<String, Long>(),
"disk_bytes" to 0L,
"oldest_at" to null,
"newest_at" to null,
"root" to storeRoot.toAbsolutePath().toString(),
),
)
return 0
}
val eventsRoot = storeRoot.resolve("events")
var count = 0L
var oldest: Long? = null
var newest: Long? = null
if (Files.isDirectory(eventsRoot)) {
Files.walk(eventsRoot).use { stream ->
for (p in stream) {
if (!Files.isRegularFile(p)) continue
if (!p.fileName.toString().endsWith(".json")) continue
count++
val mt =
try {
Files.getLastModifiedTime(p).to(TimeUnit.SECONDS)
} catch (_: IOException) {
continue
}
val o = oldest
if (o == null || mt < o) oldest = mt
val n = newest
if (n == null || mt > n) newest = mt
}
}
}
// Histogram from idx/kind/<k>/ — for a healthy store this is
// exactly one entry per (kind, event), so summing matches `count`.
// Mismatch points at index drift; run `amy store scrub` to fix.
val kindRoot = storeRoot.resolve("idx/kind")
val byKind = sortedMapOf<String, Long>()
if (Files.isDirectory(kindRoot)) {
Files.list(kindRoot).use { stream ->
for (kindDir in stream) {
if (!Files.isDirectory(kindDir)) continue
val n = Files.list(kindDir).use { it.count() }
byKind[kindDir.fileName.toString()] = n
}
}
}
val diskBytes = walkSize(storeRoot)
Json.writeLine(
mapOf(
"events" to count,
"by_kind" to byKind,
"disk_bytes" to diskBytes,
"oldest_at" to oldest,
"newest_at" to newest,
"root" to storeRoot.toAbsolutePath().toString(),
),
)
return 0
}
private fun sweepExpired(dataDir: DataDir): Int =
withStore(dataDir) { store ->
val expiresAtDir = dataDir.eventsDir.toPath().resolve("idx/expires_at")
val before = countEntries(expiresAtDir)
store.deleteExpiredEvents()
val after = countEntries(expiresAtDir)
Json.writeLine(
mapOf(
"swept" to (before - after).coerceAtLeast(0L),
"remaining" to after,
),
)
0
}
private fun scrub(dataDir: DataDir): Int =
withStore(dataDir) { store ->
store.scrub()
Json.writeLine(mapOf("ok" to true))
0
}
private fun compact(dataDir: DataDir): Int =
withStore(dataDir) { store ->
store.compact()
Json.writeLine(mapOf("ok" to true))
0
}
/**
* Maintenance verbs only need the store — not identity, not relays,
* not the signer. Skip [Context.open] (which throws if no identity
* has been bootstrapped) and construct the [FsEventStore] directly
* from [DataDir.eventsDir]. Pretty formatter matches what the rest
* of the CLI uses for inspection-friendly output.
*/
private inline fun withStore(
dataDir: DataDir,
body: (FsEventStore) -> Int,
): Int {
val store =
FsEventStore(
root = dataDir.eventsDir.toPath(),
eventToJson = JacksonMapper::toJsonPretty,
)
try {
return body(store)
} finally {
store.close()
}
}
private fun walkSize(root: Path): Long {
if (!Files.exists(root)) return 0L
var total = 0L
Files.walk(root).use { stream ->
for (p in stream) {
if (!Files.isRegularFile(p)) continue
total +=
try {
Files.size(p)
} catch (_: IOException) {
0L
}
}
}
return total
}
private fun countEntries(dir: Path): Long {
if (!Files.isDirectory(dir)) return 0L
return Files.list(dir).use { it.count() }
}
}
+252
View File
@@ -0,0 +1,252 @@
#!/usr/bin/env bash
#
# cache-headless.sh — verifies the file-backed event store is the
# source of truth for `amy` reads.
#
# Two amy identities (A and B) talk to a local nostr-rs-relay. We
# assert that:
#
# 1. After A runs `amy create`, A's local store contains the bootstrap
# events (kind:0 / 3 / 10002 / 10050 / 10051 …).
# 2. A's `amy profile show` is served from cache (`source: "cache"`)
# — no relay round-trip needed.
# 3. A's `amy profile show --refresh` bypasses the cache and pulls
# from relays (`source: "relays"`).
# 4. B fetches A's profile once over the relay (cache miss), and the
# second invocation serves from B's local store (cache hit).
# 5. `amy store stat` reports the right kind histogram and a non-zero
# event count + disk usage.
# 6. `amy relay list` reads the relay URLs back out of the local
# kind:10002 / 10050 / 10051 events (the pre-relays.json contract).
#
# Usage: ./cache-headless.sh [--port N] [--no-build]
#
set -uo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd -- "$SCRIPT_DIR/../../.." && pwd)"
TESTS_DIR="$(cd -- "$SCRIPT_DIR/.." && pwd)"
STATE_DIR="$SCRIPT_DIR/state-cache-headless"
LOG_DIR="$STATE_DIR/logs"
A_DIR="$STATE_DIR/A"
B_DIR="$STATE_DIR/B"
RUN_TS="$(date +%Y%m%d-%H%M%S)"
LOG_FILE="$LOG_DIR/run-$RUN_TS.log"
RESULTS_FILE="$STATE_DIR/results-$RUN_TS.tsv"
AMY_BIN="$REPO_ROOT/cli/build/install/amy/bin/amy"
# Reuse the relay binary the marmot harness builds.
RELAY_HOST="${RELAY_HOST:-127.0.0.2}"
RELAY_REPO="${RELAY_REPO:-$TESTS_DIR/marmot/state-headless/nostr-rs-relay}"
RELAY_BIN="$RELAY_REPO/target/release/nostr-rs-relay"
RELAY_DATA="$STATE_DIR/relay"
RELAY_PORT="${RELAY_PORT:-8092}"
RELAY_URL="ws://$RELAY_HOST:$RELAY_PORT"
NO_BUILD=0
A_NPUB=""
A_HEX=""
B_NPUB=""
B_HEX=""
while [[ $# -gt 0 ]]; do
case "$1" in
--port) RELAY_PORT="$2"; RELAY_URL="ws://$RELAY_HOST:$RELAY_PORT"; shift ;;
--host) RELAY_HOST="$2"; RELAY_URL="ws://$RELAY_HOST:$RELAY_PORT"; shift ;;
--no-build) NO_BUILD=1 ;;
-h|--help)
sed -n '3,21p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'
exit 0 ;;
*) printf 'unknown flag: %s\n' "$1" >&2; exit 2 ;;
esac
shift
done
mkdir -p "$STATE_DIR" "$LOG_DIR" "$A_DIR" "$B_DIR"
: >"$LOG_FILE"
: >"$RESULTS_FILE"
# shellcheck source=../lib.sh
source "$TESTS_DIR/lib.sh"
# shellcheck source=../marmot/setup.sh — provides start_local_relay / stop_local_relay
source "$TESTS_DIR/marmot/setup.sh"
# shellcheck source=../headless/helpers.sh
source "$TESTS_DIR/headless/helpers.sh"
# Keep the dm setup's preflight (just checks for amy + the relay) but
# define our own identity bootstrap so we don't pull in DM-specific
# wiring.
# shellcheck source=../dm/setup.sh
source "$TESTS_DIR/dm/setup.sh"
amy_b() { "$AMY_BIN" --data-dir "$B_DIR" "$@"; }
# Helper: B-side amy_json. The dm headless's helpers.sh hardcodes A_DIR
# in amy_a; we need a parallel for B without re-sourcing.
amy_b_json() {
local out
if ! out=$(amy_b "$@" 2>>"$LOG_FILE"); then
fail_msg "amy_b $*: exit $? (see $LOG_FILE)"
printf '%s\n' "$out" >>"$LOG_FILE"
return 1
fi
printf '%s' "$out"
}
cleanup() {
local rc=$?
trap - EXIT INT TERM HUP
stop_local_relay
print_summary
exit "$rc"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
trap 'exit 129' HUP
banner "Amethyst event-store cache headless ($RUN_TS)"
preflight_dm
start_local_relay
# Identity A: full bootstrap so the store has kind:0 / 3 / 10002 / …
ensure_identity_for A "$A_DIR"
banner "Bootstrapping A's account (publishes kind:0 + bootstrap events)"
"$AMY_BIN" --data-dir "$A_DIR" relay add "$RELAY_URL" --type all >>"$LOG_FILE" 2>&1
# `amy create` here would mint a *second* identity; A_DIR already has one
# from `init`. Build the bootstrap events ourselves by publishing a
# minimal kind:0 + the relay lists, all of which land in A's local store
# via verifyAndStore.
"$AMY_BIN" --data-dir "$A_DIR" relay publish-lists >>"$LOG_FILE" 2>&1
amy_a profile edit --name "AAA" --about "cache test subject" \
>>"$LOG_FILE" 2>&1 \
|| fail_msg "amy_a profile edit failed"
# Identity B: separate data-dir, separate cache, also pointed at the relay.
ensure_identity_for B "$B_DIR"
"$AMY_BIN" --data-dir "$B_DIR" relay add "$RELAY_URL" --type all >>"$LOG_FILE" 2>&1
"$AMY_BIN" --data-dir "$B_DIR" relay publish-lists >>"$LOG_FILE" 2>&1
# ----------------------------------------------------------------------
# 1. amy store stat reports a non-empty store after bootstrap.
# ----------------------------------------------------------------------
banner "T1 — amy store stat after bootstrap"
T1=$(amy_a store stat) || fail_msg "store stat failed"
T1_EVENTS=$(printf '%s' "$T1" | jq -r '.events')
T1_K0=$(printf '%s' "$T1" | jq -r '.by_kind."0" // 0')
T1_K10002=$(printf '%s' "$T1" | jq -r '.by_kind."10002" // 0')
T1_BYTES=$(printf '%s' "$T1" | jq -r '.disk_bytes')
if [[ "$T1_EVENTS" -gt 0 ]]; then
record_result T1.events pass "$T1_EVENTS event(s) in A's store"
else
record_result T1.events fail "store should be non-empty after publish-lists+profile edit"
fi
if [[ "$T1_K0" -ge 1 ]]; then
record_result T1.kind0 pass "kind:0 present"
else
record_result T1.kind0 fail "no kind:0 in by_kind histogram"
fi
if [[ "$T1_K10002" -ge 1 ]]; then
record_result T1.kind10002 pass "kind:10002 present"
else
record_result T1.kind10002 fail "no kind:10002 in by_kind histogram"
fi
if [[ "$T1_BYTES" -gt 0 ]]; then
record_result T1.disk_bytes pass "$T1_BYTES bytes used"
else
record_result T1.disk_bytes fail "disk_bytes should be > 0"
fi
# ----------------------------------------------------------------------
# 2. profile show on self → source: "cache" (slot lookup, no network).
# ----------------------------------------------------------------------
banner "T2 — A's profile show is served from cache by default"
T2=$(amy_a profile show)
T2_SRC=$(printf '%s' "$T2" | jq -r '.source')
T2_FOUND=$(printf '%s' "$T2" | jq -r '.found')
T2_NAME=$(printf '%s' "$T2" | jq -r '.metadata.name // ""')
assert_eq T2.source "cache" "$T2_SRC" "default reads should hit the local store"
assert_eq T2.found "true" "$T2_FOUND" ""
assert_eq T2.name "AAA" "$T2_NAME" "profile edit must round-trip"
# ----------------------------------------------------------------------
# 3. profile show --refresh forces a relay drain.
# ----------------------------------------------------------------------
banner "T3 — --refresh forces source: relays"
T3=$(amy_a profile show --refresh)
T3_SRC=$(printf '%s' "$T3" | jq -r '.source')
T3_QR=$(printf '%s' "$T3" | jq -r '.queried_relays | length')
assert_eq T3.source "relays" "$T3_SRC" "--refresh must skip cache"
if [[ "$T3_QR" -ge 1 ]]; then
record_result T3.queried_relays pass "$T3_QR relay(s) queried"
else
record_result T3.queried_relays fail "expected at least one queried_relays entry"
fi
# ----------------------------------------------------------------------
# 4. B has not seen A → first profile show is a relay miss; second is a hit.
# ----------------------------------------------------------------------
banner "T4 — B sees A first via relay, then via cache"
T4A=$(amy_b_json profile show "$A_NPUB")
T4A_SRC=$(printf '%s' "$T4A" | jq -r '.source')
T4A_NAME=$(printf '%s' "$T4A" | jq -r '.metadata.name // ""')
assert_eq T4a.source "relays" "$T4A_SRC" "first lookup of a stranger comes from relays"
assert_eq T4a.name "AAA" "$T4A_NAME" "B should resolve A's name on first fetch"
T4B=$(amy_b_json profile show "$A_NPUB")
T4B_SRC=$(printf '%s' "$T4B" | jq -r '.source')
T4B_NAME=$(printf '%s' "$T4B" | jq -r '.metadata.name // ""')
assert_eq T4b.source "cache" "$T4B_SRC" "second lookup of the same stranger must serve from cache"
assert_eq T4b.name "AAA" "$T4B_NAME" "cached metadata must match"
# ----------------------------------------------------------------------
# 5. relay list reads URLs back from the local kind:10002 / 10050 / 10051.
# ----------------------------------------------------------------------
banner "T5 — relay list reads from store"
T5=$(amy_a relay list)
T5_NIP65=$(printf '%s' "$T5" | jq -r '.nip65 | length')
T5_INBOX=$(printf '%s' "$T5" | jq -r '.inbox | length')
T5_KP=$(printf '%s' "$T5" | jq -r '.key_package | length')
if [[ "$T5_NIP65" -ge 1 ]]; then
record_result T5.nip65 pass "$T5_NIP65 nip65 url(s)"
else
record_result T5.nip65 fail "nip65 bucket empty after relay add --type all"
fi
if [[ "$T5_INBOX" -ge 1 ]]; then
record_result T5.inbox pass "$T5_INBOX inbox url(s)"
else
record_result T5.inbox fail "inbox bucket empty after relay add --type all"
fi
if [[ "$T5_KP" -ge 1 ]]; then
record_result T5.key_package pass "$T5_KP key_package url(s)"
else
record_result T5.key_package fail "key_package bucket empty after relay add --type all"
fi
# ----------------------------------------------------------------------
# 6. relays.json must NOT exist anywhere — the events ARE the config.
# ----------------------------------------------------------------------
banner "T6 — relays.json is gone"
if [[ ! -f "$A_DIR/relays.json" && ! -f "$B_DIR/relays.json" ]]; then
record_result T6.no_relays_json pass "neither data-dir contains relays.json"
else
record_result T6.no_relays_json fail "relays.json found — should have been removed"
fi
# ----------------------------------------------------------------------
# 7. Maintenance verbs work without an identity (they only need the store).
# ----------------------------------------------------------------------
banner "T7 — store maintenance verbs (sweep/scrub/compact) without identity"
TMP_NO_ID=$(mktemp -d)
T7_STAT=$(${AMY_BIN} --data-dir "$TMP_NO_ID" store stat)
T7_SWEEP=$(${AMY_BIN} --data-dir "$TMP_NO_ID" store sweep-expired)
T7_SCRUB=$(${AMY_BIN} --data-dir "$TMP_NO_ID" store scrub)
T7_COMPACT=$(${AMY_BIN} --data-dir "$TMP_NO_ID" store compact)
assert_eq T7.stat.events "0" "$(printf '%s' "$T7_STAT" | jq -r '.events')" ""
assert_eq T7.sweep.swept "0" "$(printf '%s' "$T7_SWEEP" | jq -r '.swept')" ""
assert_eq T7.scrub.ok "true" "$(printf '%s' "$T7_SCRUB" | jq -r '.ok')" ""
assert_eq T7.compact.ok "true" "$(printf '%s' "$T7_COMPACT" | jq -r '.ok')" ""
rm -rf "$TMP_NO_ID"