assert_eq is (actual, expected, test_id, note); first version of the
script had test_id and actual swapped, so even passing assertions
showed up as fails with confusing "got T2.source" messages. Now all
21 assertions pass with the right arg order. Each successful assertion
also fires record_result pass so the results table covers them
(previously the tests passed but were invisible — only the explicit
record_result calls showed up).
Verified end-to-end on a real local nostr-rs-relay:
21 passed, 0 failed, 0 skipped (of 21)
Coverage:
T1.* — store stat reports kind histogram + disk usage after publish
T2.* — self profile show is served from cache by default
T3.* — --refresh forces source: relays
T4a/b — B's first lookup of A is a relay miss; second is a cache hit
T5.* — relay list reads URLs back from local kind:10002/10050/10051
T6 — relays.json is gone from both data-dirs
T7.* — store maintenance verbs work without an identity
Three changes that go together:
1. Reconcile cli/plans/2026-04-24-file-event-store-{overview,nips}.md
with shipped reality: code lives in quartz/jvmMain/, not commons/;
data dir is <data-dir>/events-store/, not <root>/events/.
2. New StoreCommands wired as `amy store …`:
- stat → events count, kind histogram, disk bytes,
oldest/newest createdAt. Pure read, no Context
(skips identity check).
- sweep-expired → wraps store.deleteExpiredEvents(); reports
{swept, remaining}.
- scrub → wraps store.scrub() to rebuild idx/ from
canonicals.
- compact → wraps store.compact() to drop dangling idx/.
All four open the FsEventStore directly (no Context, no identity
needed) — they're store-only operations.
3. New e2e harness at cli/tests/cache/cache-headless.sh that boots a
local nostr-rs-relay, two amy identities (A + B), and asserts:
T1 — store stat reports non-empty store after publish-lists +
profile edit, with kind:0 and kind:10002 present.
T2 — A's `profile show` is `source: "cache"` by default.
T3 — `--refresh` forces `source: "relays"`.
T4 — B's first `profile show <A_NPUB>` is a relay miss; second is
a cache hit (proves drain populates the local store and
subsequent reads serve from disk).
T5 — `relay list` reads URLs back from the local kind:10002 /
10050 / 10051 events.
T6 — relays.json no longer exists in either data-dir.
T7 — store stat / sweep-expired / scrub / compact all run
without an identity present.
Same pattern as cli/tests/dm/dm-interop-headless.sh — reuses the
nostr-rs-relay infrastructure from cli/tests/marmot/setup.sh.
Users actually look at <data-dir>/events-store/ files (cat / jq / git
diff), so the CLI now writes them with the InliningTagArrayPrettyPrinter
that quartz already had configured but never invoked. Each event is
indented with 2 spaces, but every tag array stays on a single line —
nice trade-off between human-readable and not-too-tall:
{
"id": "...",
"pubkey": "...",
"created_at": 1700000000,
"kind": 1,
"tags": [
["t","nostr"],
["e","abc...a"],
["alt","quick brown fox"]
],
"content": "hi there",
"sig": "..."
}
Stored bytes are not the canonical NIP-01 form — but verification
re-canonicalises via EventHasher anyway, so format is purely a UX
choice. Compact stays the default for any caller that doesn't opt
in (Android keeps SQLite, generic FsEventStore embedders keep
compact).
- JacksonMapper.toJsonPretty(event): new entry point that uses
writerWithDefaultPrettyPrinter() with the existing inlining printer.
- FsEventStore now takes an `eventToJson: (Event) -> String` callback,
default = Event::toJson (compact). Used in insert.
- Context wires in JacksonMapper::toJsonPretty.
2 new tests in FsEventToJsonTest pin both formats (compact stays
single-line; pretty round-trips). 117 fs tests green.
The local relay configuration was redundant once the event store
became the source of truth: every account already publishes signed
kind:10002 (NIP-65), kind:10050 (DM inbox), and kind:10051
(KeyPackage relays) events, and Context now reads each set straight
out of the local store.
Removed:
- data class RelayConfig (nip65/inbox/keyPackage buckets)
- DataDir.relaysFile / loadRelays() / saveRelays()
- Context.relays parameter — Context.open() no longer reads JSON
- CreateCommand.defaultRelayConfig() helper
- CreateCommand's saveRelays() call (redundant: ctx.publish persists
the bootstrap events into the store anyway)
Context now serves the four relay accessors from the store with sane
fallbacks:
outboxRelays() = relaysOf(self)?.writeRelaysNorm() ?: DefaultNIP65RelaySet
inboxRelays() = dmInboxOf(self)?.relays() ?: DefaultDMRelayList.toSet()
keyPackageRelays() = keyPackageRelaysOf(self)?.relays() ?: outboxRelays()
anyRelays() = union of the three
bootstrapRelays() = anyRelays() ∪ DefaultNIP65RelaySet ∪ DefaultDMRelayList
RelayCommands rewritten:
- `relay add URL --type T` — read existing relay-list event from the
store, append URL, build + sign + ingest a new event of the
matching kind. The replaceable slot mechanism replaces the old
winner atomically.
- `relay list` — dump URLs from the latest event in each bucket.
- `relay publish-lists` — broadcast whichever events are present in
the store; errors out cleanly when none exist (suggests `relay add`
or `create`).
No migration: existing data dirs that still have `relays.json` will
ignore it. Run `amy relay add` or `amy create` to repopulate. Per
discussion this is a clean break, not a backwards-compat shim.
README + DEVELOPMENT updated to reflect the new on-disk layout (no
more `relays.json`; events-store/ is the relay configuration).
Audit found four commands draining replaceable relay-list events
(kind:10050, 10051, 10002) on every invocation, plus one fetching
the user's own kind:3 contact list. All five are now cache-first.
New Context helpers:
- dmInboxOf(pk) → kind:10050 (ChatMessageRelayListEvent)
- keyPackageRelaysOf(pk)→ kind:10051 (KeyPackageRelayListEvent)
- cachedRelayListsOf(pk)→ assembles a RecipientRelayFetcher.Lists
from the local store; returns null if
nothing is cached so callers can fall
back with `?:`.
Wired into:
- FeedCommand.resolveFollowing — replaces a kind:3 drain with
ctx.contactsOf(self).
- DmCommands.resolveDmRelays — `amy dm send` / `dm list`.
- KeyPackageCommands.check.
- AwaitCommands (key-package / member / admin polling loops).
- GroupAddMemberCommand (one cache hit per invitee).
All five sites now go: try cache → if miss, run the existing
RecipientRelayFetcher / drain. The drain itself populates the cache
via verifyAndStore, so subsequent runs are local-only. Replaceable
slot shortcut means each cached read is one stat + one readString,
not a directory walk.
README "Local event store" section gains a list of which commands
are cache-first today.
The first command to consume the local event store as a read source.
Default behaviour: read from the FsEventStore via Context.profileOf().
Pass --refresh to skip the cache and drain from relays (which then
re-populates the store as a side effect, like every other drain).
Output JSON gains a "source" field so scripts/agents can tell whether
the result came from disk or the network:
source = "cache" → served from <data-dir>/events-store/, queried_relays = []
source = "relays" → fresh relay drain, queried_relays = list
Cache hit goes through the planner's slot shortcut from the previous
commit: one stat + one read on replaceable/0/<pubkey>.json, no
directory scan.
profile show + profile edit now have entries in the verb table in
cli/README.md.
`marmot message react "$gid" "$id" "🍕"` was producing a kind:7 whose
inner content was four `U+FFFD` replacement characters instead of the
emoji. Whitenoise's `message_aggregator::emoji_utils` rejected it with
`Invalid reaction content: ����`, and the openmls receiver retried the
event ten times before giving up — by which point the secret tree had
already consumed that generation, so the retry surfaced as a confusing
`Ciphertext generation out of bounds 1 / SecretReuseError`. That last
error was a downstream symptom; the root cause is argv decoding.
Why it happens: JEP 400 (Java 18+) pins `file.encoding` to UTF-8 but
deliberately leaves `sun.jnu.encoding` — the encoding the JVM uses to
decode argv, filenames, and native interop strings — tied to the OS
locale, and `sun.jnu.encoding` is read **before** the JVM parses any
`-D` flag, so it can't be overridden via `applicationDefaultJvmArgs`.
Under POSIX/C locales (no `LANG`/`LC_ALL` set, common on CI runners
and stripped-down containers) `sun.jnu.encoding` ends up at
`ANSI_X3.4-1968` — i.e. ASCII — and every byte > 0x7F in argv lands
as `U+FFFD` before our code ever sees it. The four UTF-8 bytes of 🍕
(F0 9F 8D 95) become four replacement chars, get signed into the kind:7
content, and the test fails on receipt.
The only place we can set the encoding from is the launching shell.
Patch the gradle-generated start scripts (POSIX `amy`, Windows
`amy.bat`) right after `:cli:startScripts`:
* POSIX: if neither `LANG` nor `LC_ALL` is set, `export LANG=C.UTF-8`
before invoking Java. Existing locale settings are respected.
* Windows: `chcp 65001` to switch the console to UTF-8 before Java
runs.
Tested manually: with `LANG=` (sandbox default) `amy` previously sent
`argv[0]=????` for an emoji argument; with the patch it sends `🍕` and
`sun.jnu.encoding=UTF-8`, and whitenoise accepts the kind:7 reaction.
The interop test 09 polling also needed a fix unrelated to the
encoding bug: wn aggregates kind:7 reactions onto the anchor message
under `.reactions.by_emoji.<emoji>` instead of surfacing them as
standalone messages whose `.content` is the emoji, so the previous
`wait_for_message B "$mls_gid" "🍕"` would have failed even with a
correctly-decoded reaction. The new poll inspects each message's
`.reactions.by_emoji` keys for the emoji literal.
Marmot interop score: 15/16 → **16/16** 🎉https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Every event Amy observes — drained from a relay subscription,
unwrapped from a NIP-59 gift wrap, or generated locally for publish —
now flows through Context.verifyAndStore: NIP-01 id + signature check
via Event.verify(), then store.insert(). Bad events are dropped with
a stderr log and never reach command code; persistence failures are
logged but do not propagate, so a broken store never breaks a relay
subscription.
- Context.drain() now persists every received event before surfacing
it to the caller. Existing callers (FeedCommand, DmCommands,
ProfileCommands, Marmot sync) get caching for free.
- Context.publish() persists outbound events too, so the local store
reflects what Amy has done even when every relay rejects.
- Three cache-first read helpers expose the most common lookups
without hitting relays: profileOf(pubKey) → kind:0,
relaysOf(pubKey) → kind:10002, contactsOf(pubKey) → kind:3.
- Class doc on Context spells out the contract; README.md gets a new
"Local event store — the source of truth" section pointing at the
on-disk layout and the design plans.
When wn admin-removes A in interop test 14, A processed the proposal
locally — `tree.removeLeaf(A.leafIndex)` blanked her leaf and shrank
`tree.leafCount` past `myLeafIndex` — and then immediately walked into
BinaryTree.directPath(myLeafIndex, tree.leafCount)
at MlsGroup.processCommitInner. With `myLeafIndex >= leafCount`, the
left-balanced parent walk had no valid stopping point: each recursion
step doubled the candidate parent index, integer-overflowed past 2^31,
and either returned garbage or kept appending to the result list until
the JVM OOM'd. The OOM bubbled up as a `runBlocking` failure that
rolled back the whole commit — so A also stayed locally convinced she
was still a member, and the test timed out waiting for `not_member`.
Three layered fixes so the failure mode can't recur:
* `BinaryTree.parent`/`directPath`/`copath` now `require` an in-range
input. The root and any node ≥ nodeCount used to silently loop or
return garbage; now they throw `IllegalArgumentException` immediately.
This is defense-in-depth — a future caller passing an invalid index
gets a stack trace at the boundary instead of an OOM five frames
deep.
* `MlsGroup.processCommitInner` short-circuits the path-decrypt + epoch
advancement when the proposals in the commit just removed *us*. We
preserve the proposal-side tree mutations so the caller (and the
outer state machine) can observe that we're out, clear pending
proposals + sent keys, and return. Without this short-circuit the
function would derive a bogus all-zero `commit_secret`, fail
`confirmation_tag` verification, throw, and roll the snapshot back —
leaving us still convinced we were a live member.
* `MlsGroupManager.isMember` and a new `MlsGroup.isLocalMember()`
helper now return false once our leaf is null or past `leafCount`.
The post-Remove group entry still lives in `groups` so callers can
inspect the final tree, but every cli command (`group show`,
`message send`, etc.) sees `not_member` and returns the right error.
Also add a comprehensive `BinaryTreeTest` covering non-power-of-2
trees (3, 5, …, 32 leaves) and the boundary cases (root has no
parent, leafIndex ≥ leafCount must throw). The pre-existing tests
only exercised the 4-leaf example from RFC 9420 Appendix C; nothing
hit the `parentInRange` branch, which is exactly where the OOM lived.
Test 14's bash polling needed a small companion fix: it captured
amy's stderr through `2>&1` and fed the result to `jq`, but the
captured stream is interleaved with quartz `Log.d(…)` debug lines,
so the JSON parse always failed. Switch to a `grep` for the
`"error":"not_member"` literal — that signature only appears in
the JSON payload and survives the debug noise. Also tee the
captured output into `$LOG_FILE` so post-mortem logs include each
polling iteration's `[cli] ingest …` traces.
Marmot interop score: 11/16 → 14/16 (tests 9, 15 are unrelated MLS
issues — see follow-up).
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
- DataDir.eventsDir → "<root>/events-store" (avoids colliding with the
marmot file name conventions).
- Context.store: lazy IEventStore over that directory. Lazy so the
many existing commands that don't touch persistent event state pay
zero open cost (no .lock, no seed file). Closed by Context.close()
only when the lazy delegate has actually been initialised — checked
via reflection so close() never accidentally forces it.
Whitenoise-rs ≥ v0.2.x nests the group object one level deeper —
`{"result": {"group": {...}}}` instead of `{"result": {...}}`. The two
metadata-name pollers in tests 07 and 10 still asked for `.result.name`
and silently saw the empty string, so even when wn-side `groups show`
returned the correct new name the polling loops timed out:
07 metadata rename fail B saw name="" not "Interop-02-renamed"
10 concurrent commits fail diverged: A sees "race-from-amethyst", B sees ""
Both queries now also peel a `.group` wrapper when present, leaving the
older bare-`result` shape working too. Re-running the headless harness
flips 07 + 10 from fail → pass; the remaining failures (09, 14, 15) are
unrelated MLS-encryption / Remove-commit issues that need their own fix.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
dm-03 and dm-04 spawn a throwaway "ghost" identity by calling $AMY_BIN
directly (not through amy_a / amy_d), so they missed the plaintext
backend flag added in the previous commit and were failing with
"No TTY and no passphrase source" on headless CI.
https://claude.ai/code/session_01SqdMfLdXvb3GskFLcEj739
identity.json previously stored `privKeyHex` and `nsec` as plaintext fields.
0600 file perms keep other OS users out, but not another app running as the
same user — and that is the threat model the CLI actually cares about.
Introduce a SecretStore indirection:
* identity.json now persists only the public parts plus a typed
`secret: IdentitySecret` envelope (keychain | ncryptsec | plaintext).
* macOS uses `/usr/bin/security` (Keychain ACLs bind the item to the binary
that stored it, so other same-user apps need user consent).
* Linux uses `secret-tool` if a Secret Service is running on the session
D-Bus; the gain there is at-rest encryption while the keyring is locked.
* On any platform without a keychain, auto falls back to NIP-49
(scrypt + XChaCha20) with the passphrase read from --passphrase-file,
then $AMY_PASSPHRASE, then a TTY prompt. Another same-user app can read
the blob but cannot decrypt it without the passphrase.
* `--secret-backend=plaintext` is an explicit opt-in for dev scripts and
the interop test harness.
Legacy identity.json files that still carry top-level privKeyHex/nsec are
read transparently and auto-migrate on the next save.
whoami, `create` / `login` existence checks, and init-re-run now use a
metadata-only load path so they do not trigger a keychain prompt or ask
for a passphrase just to echo the npub.
Tests: cli/tests/ setups wire --secret-backend=plaintext through the amy_a
/ amy_d wrappers so headless CI runs do not stall on a TTY passphrase
prompt.
https://claude.ai/code/session_01SqdMfLdXvb3GskFLcEj739
Three-part design doc for an IEventStore backed by a directory tree
instead of SQLite, targeted at the cli/ module.
- overview: goals, directory layout, feature-parity matrix, public API
- pipelines: insert (T1-T8 with crash-safety), query planner, delete,
transactions, concurrency
- nips: replaceable/addressable slot enforcement via hardlinks +
atomic rename, NIP-09 tombstones (hardlink to the kind-5), NIP-40
expirations, NIP-50 FTS via inverted-index hardlinks, NIP-62 vanish
cascade, tag indexing, seed file, scrub/compact, test strategy,
10-step rollout
Adds a new build-cli matrix to create-release.yml so every tag push also
produces self-contained amy binaries:
- amy-<ver>-macos-x64.tar.gz (macos-13 runner)
- amy-<ver>-macos-arm64.tar.gz (macos-14 runner)
- amy-<ver>-linux-x64.tar.gz (ubuntu-latest)
- amy-<ver>-linux-x64.deb (ubuntu-latest)
- amy-<ver>-linux-x64.rpm (ubuntu-latest)
Each tarball is a flat tree (bin/amy + lib/*.jar + runtime/) with a
jlink'd JDK 21 embedded — no system Java required on the user machine.
The .deb / .rpm install the same tree under /opt/amy/.
Windows is intentionally deferred until cli/ is validated on Windows
(data-dir path handling, file locking on groups/<gid>.mls, identity.json
line endings).
Two follow-ups flagged in comments:
- :commons leaks Compose + Skiko (~40 MB) as transitive deps to :cli.
Tarball lands at ~98 MB instead of the plan's <80 MB target. Size
budget in the workflow is set to 200 MB until :commons is split into
core + ui modules.
- .deb/.rpm install to /opt/amy/bin/amy with no /usr/local/bin symlink.
Users must add to PATH or symlink manually after install.
Wire-up:
- cli/build.gradle.kts gains jlinkRuntime + amyImage (Sync task that
combines installDist output with the jlink runtime + a shell
launcher) + jpackageDeb + jpackageRpm.
- scripts/asset-name.sh gains cli_asset_name() + collect_cli_assets()
alongside the existing desktop collector, preserving the
<product>-<version>-<family>-<arch>.<ext> naming contract.
See cli/plans/2026-04-21-cli-distribution.md for the overall plan.
https://claude.ai/code/session_01Tbh6F7TtEeceb4K3stcUWp
identity.json, relays.json, state.json, MLS group state/retained epochs,
keypackage bundles, and the Marmot message archive were written with
default umask — typically world-readable on Unix. On a shared machine
another OS user could read the stored private key or decrypted chat
history.
Introduce SecureFileIO: overwrites go via a sibling tempfile with POSIX
0600 attrs set at creation time and ATOMIC_MOVE into place, directories
are created 0700, and on upgrade DataDir.init tightens any pre-existing
loose files. Non-POSIX filesystems (Windows) fall back to setReadable/
Writable/Executable to strip other-user access.
This is defense against other OS users. Another app running as the same
user is still in scope — the FileStores doc continues to note that real
deployments MUST encrypt at rest.
https://claude.ai/code/session_01SqdMfLdXvb3GskFLcEj739
Two on-disk layout changes that only affect freshly-created data-dirs (no
migration shim since `amethyst-cli-data` was never tagged or released):
- Default `--data-dir` is now `./amy` instead of `./amethyst-cli-data` —
shorter, matches the binary name, and still overridable via the flag or
`$AMETHYST_CLI_DATA`.
- MLS-only files (`groups/` and `keypackages.bundle`) move under a new
`marmot/` subdirectory so the top-level root stays tidy as more
non-Marmot state lands (notes, profile caches, etc). Existing
top-level files (`identity.json`, `relays.json`, `state.json`) are
unchanged.
README tree diagram and DEVELOPMENT.md test table updated to match.
Moves the top-level `post` and `feed` verbs under a single `notes` group
(`amy notes post`, `amy notes feed`). Keeps the per-event-family grouping
consistent with `profile` (kind:0), `dm` (NIP-17), and `marmot` (MLS).
PostCommand and FeedCommand are unchanged — the new NotesCommands object
is a thin dispatcher that routes `notes post` / `notes feed` into them.
Adds three new top-level verbs to amy that mirror the core social-feed
surface of the Android client:
- `amy profile show [USER]` / `amy profile edit ...` reads and patches the
user's NIP-01 kind:0 metadata. `edit` reuses MetadataEvent.updateFromPast
semantics so unset flags keep prior values and blank values delete the
field, falling back to MetadataEvent.createNew when no kind:0 exists yet.
- `amy post TEXT` publishes a NIP-10 kind:1 short text note to the user's
outbox relays via TextNoteEvent.build.
- `amy feed [--author USER] [--following] [--limit N] [--since|--until TS]`
reads kind:1 notes — own / single-author / contact-list — using
Context.drain with author-scoped filters, dedups by id, and returns the
newest-first window as JSON.
All three keep the thin-assembly-layer rule: no Nostr or business logic in
cli/, every event is built/signed via quartz/ and published through the
existing Context.publish + Context.drain helpers.
Delete() returning false on a still-existing file meant we silently lost
scratch state for MLS group / key-package / message stores. Route the
four call sites through a `deleteOrWarn` helper that logs via the
KMP-safe quartz Log when the file refuses to disappear.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the harness move from tools/marmot-interop/ to cli/tests/,
three doc locations were still describing the old layout:
- cli/DEVELOPMENT.md § Testing — the "round-trip" row pointed at a
non-existent `cli/src/test/resources/scripts/`, and the "interop
with other clients" row claimed it was out of scope. Both now name
`cli/tests/marmot/` and `cli/tests/dm/` with what each covers, and
the interop-script template points at the concrete test files
rather than re-inventing a smaller example.
- cli/README.md — the "For an interop-test script template" pointer
now links to `cli/tests/README.md` alongside the DEVELOPMENT.md
section.
- .claude/skills/amy-expert/SKILL.md — the "Where things live" tree
omitted the new `tests/` subtree entirely. Added it with a
one-line description per suite. Also extended the wire-up checklist
with a step 7: add a harness case when a new verb changes observable
wire behaviour.
No content changes elsewhere; just path corrections.
Two fixes needed to make the DM harness actually pass end-to-end:
1. Bind the loopback relay to 127.0.0.2 instead of 127.0.0.1.
Quartz's `RelayTag.parse` (called from `ChatMessageRelayListEvent.
relays()`) rejects any URL for which `isLocalHost()` returns true —
the substring check matches `"127.0.0.1"` among others. That means a
kind:10050 event whose `relay` tag reads `ws://127.0.0.1:8090/`
silently parses to an empty relay list, and `RecipientRelayFetcher`
reports `dmInbox == []`. Under strict mode `dm send` then correctly
refuses with `no_dm_relays` — hiding an otherwise-valid test setup.
`127.0.0.2` is the cleanest fix: still pure loopback (no network
traffic, no config needed), not matched by `isLocalHost()`, and
Linux binds any IP in 127.0.0.0/8 without setup. The Marmot harness
is unaffected because its tests never depend on parsing their own
kind:10051/10002 relay tags back out — they hit the bootstrap
fallback path.
marmot/setup.sh's `start_local_relay` now reads `${RELAY_HOST:-
127.0.0.1}` so both harnesses can share it; the DM harness sets
`RELAY_HOST=127.0.0.2` by default and exposes `--host` as an escape
hatch.
2. Replace dm-06 cursor-advance check with `--since` filter check.
The original assertion ("second no-flag `dm list` returns 0") was
wrong for the actual design: `filterGiftWrapsToPubkey` always
subtracts a 2-day lookback from `since` (required to catch NIP-59's
randomised `created_at`), so a repeat no-flag query legitimately
re-pulls everything in the last two days. The cursor advances the
scan window for efficiency, not for idempotency. dm-06 now verifies
`--since` actually filters: query with `--since <newest + 2 days +
1 hour>` — after the filter's lookback subtraction, the effective
floor lands past every message, and the list comes back empty.
Also: added `protoc` to the DM preflight checks so the missing-dep
error comes before 4 failed cargo retries.
Result on this machine: 6/6 pass, two clean runs in a row.
Adds two inner-event verbs on top of the existing kind:9 text send path:
amy marmot message react GID TARGET_EVENT_ID EMOJI
amy marmot message delete GID TARGET_EVENT_ID...
Both build an unsigned rumor per MIP-03 (RumorAssembler) and wrap it in
the usual kind:445 outer via MarmotOutboundProcessor — the same shape as
text messages, just a different inner kind. The target event is looked
up in the account's decrypted inner-message log (persisted by the
inbound pipeline) so the NIP-25 / NIP-09 templates can populate the
target's pubkey and kind tags without making the caller re-derive them.
Test 09 previously noted that react round-trip verification was blocked
on a missing CLI verb. Expanded it to react via amy and confirm B sees
the kind:7 surface in its messages stream.
https://claude.ai/code/session_01Unm6uLHGLj9UcBY7hWfJVW
`tools/` is for libraries (arti-build); shell-based interop harnesses
that drive the `amy` binary belong with the CLI module. New layout:
cli/tests/
├── lib.sh # shared logging + result tracking
├── headless/helpers.sh # shared amy_a / amy_json / assertions
├── marmot/ # MLS group-messaging interop (vs whitenoise-rs)
│ ├── marmot-interop.sh
│ ├── marmot-interop-headless.sh
│ ├── setup.sh
│ ├── tests-create.sh
│ ├── tests-manage.sh
│ ├── tests-extras.sh
│ └── patches/
└── dm/ # NIP-17 DM interop (amy ↔ amy)
├── dm-interop-headless.sh
├── setup.sh
└── tests-dm.sh
Per-suite changes:
- Each suite owns its own setup.sh; previously the Marmot setup was at
headless/setup.sh and the DM setup at headless/setup-dm.sh, which
implied shared infra they don't actually share.
- `cli/tests/lib.sh` and `cli/tests/headless/helpers.sh` are the only
shared bits across suites.
- The DM harness still reuses Marmot's `start_local_relay` /
`stop_local_relay` (sourced via `../marmot/setup.sh`) — same relay
binary, no duplication.
- All sourcing paths updated; REPO_ROOT now climbs three levels (was
two when the harness lived under `tools/`); patches/ is now a sibling
of marmot/setup.sh rather than under marmot/headless/.
- `.gitignore` updated to cover both suites' state dirs.
cli/ROADMAP.md and cli/tests/README.md updated to point at the new paths.
No behavioural change — every test runs the same code; only the file
layout and source paths moved.
Mirrors the Android Danger-Zone action: wipes every MLS group, retained
epoch secret, persisted KeyPackage bundle, relay subscription, and the
run-state since-cursors. Does not broadcast any SelfRemove/leave
commits, because the reset path is for recovering from corrupted or
unrecoverable local state where graceful teardown may be impossible.
Requires an explicit --yes to execute. Without --yes it emits a dry-run
JSON listing the groups that would be wiped and exits 2, so scripts
that forget the flag fail loudly instead of nuking state.
https://claude.ai/code/session_01Unm6uLHGLj9UcBY7hWfJVW
Adds `tools/marmot-interop/dm-interop-headless.sh` — a zero-prompt
end-to-end harness that runs two independent `amy` processes against
a loopback nostr-rs-relay and verifies the NIP-17 DM surface.
Six scenarios:
dm-01 text round-trip (kind:14) in both directions
dm-02 `dm list` surfaces prior exchange with `type:text` discriminator
dm-03 strict kind:10050 refuses sends to an inboxless recipient
(surfaces the `no_dm_relays` error JSON)
dm-04 `--allow-fallback` opts into NIP-65 read / bootstrap chain
(asserts a `relay_source: "bootstrap"` or `"nip65_read"` row)
dm-05 file message reference mode (kind:15) — send-file with
--key / --nonce, verify recipient decodes url + key + nonce + mime
dm-06 no-flag `dm list` advances `state.giftWrapSince`; second call
returns an empty `messages` array
Shape matches the existing Marmot harness: reuses `lib.sh` for logging
+ result tracking, reuses `setup.sh` for the nostr-rs-relay lifecycle
(`start_local_relay` / `stop_local_relay`), adds a slim `setup-dm.sh`
preflight (amy + relay only, no whitenoise-rs / Marmot patches), and
defines tests in `tests-dm.sh`.
No new CI job yet — the relay build needs Rust + ~3 minutes on a
cold cache, same constraint as the Marmot harness.
Upload-mode (`dm send-file --file PATH --server URL`) is not scripted
here: it needs a local Blossom server, which is out of scope for this
pass. The shared upload classes are unit-tested on desktop at
`desktopApp/src/jvmTest/kotlin/.../service/upload/`.
`dm send-file` now has two modes:
- **Upload mode** — `dm send-file RECIPIENT --file PATH --server URL`:
generate a fresh AES-GCM cipher, encrypt the file, upload the
ciphertext to the Blossom server, then publish a kind:15 referencing
the returned URL. Auto-detected metadata (encrypted hash + size,
original sha256, mime type, image dimensions, blurhash) is folded
into the kind:15 tags. The response surfaces the encryption key and
nonce so the same encrypted blob can be re-shared without
re-uploading.
- **Reference mode** — the previous shape still works:
`dm send-file RECIPIENT URL --key HEX --nonce HEX [...]`. Useful
when the upload happened elsewhere (Android client, third-party
tool) or for replaying a previous upload.
Mode is selected by the presence of `--file`. Both paths share the
same wrap-and-publish pipeline (`publishWraps`) and respect the strict
NIP-17 §6 relay rule: kind:1059 only reaches the recipient's kind:10050
unless `--allow-fallback` is passed.
Implementation entirely reuses the upload helpers just promoted to
`commons/jvmMain/.../service/upload/`, so no new business logic lives
in `cli/`. The Blossom auth event (kind:24242) is built and signed via
`BlossomAuth.createUploadAuth(... ctx.signer ...)`.
Receive:
- decryptDms now accepts both ChatMessageEvent (kind:14) and
ChatMessageEncryptedFileHeaderEvent (kind:15). The result is a sealed
DecryptedDm hierarchy (TextDm / FileDm) with a `type: "text"|"file"`
discriminator in JSON. File messages emit url, mime_type,
encryption_algorithm, decryption_key, decryption_nonce, hash,
original_hash, size, dim, blurhash. `dm await --match X` searches the
text body for kind:14 and the URL for kind:15.
Send:
- New `dm send-file RECIPIENT URL --key HEX --nonce HEX [...]` verb.
Builds a kind:15 ChatMessageEncryptedFileHeaderEvent via the existing
ChatMessageEncryptedFileHeaderEvent.build + AESGCM(key, nonce), then
publishes through NIP17Factory.createEncryptedFileNIP17. The file is
expected to already be uploaded; carrying the upload itself in `amy`
is the next change.
Spec compliance (NIP-17 §6, "if no kind:10050, shouldn't try"):
- resolveDmRelays now returns kind:10050 only by default. If the
recipient has none, send/send-file exit with `no_dm_relays` and a
message recommending `--allow-fallback`. The fallback chain (kind:10002
read marker → bootstrap) is preserved behind that opt-in flag for
interop tests and brand-new accounts.
Each recipient row in the send response now includes `relay_source`
("kind_10050" / "nip65_read" / "bootstrap") so callers can tell where
the wraps actually went.
Note: Quartz already enforces the NIP-17 §7 step-3 impersonation guard
(seal pubkey is forced over the rumor's claimed pubkey inside
`Rumor.mergeWith`), so no extra check is needed in the CLI.
Three call sites did the same two-layer peel —
kind:1059 → (nip44 decrypt) → kind:13 sealed rumor → (nip44 decrypt) → rumor
Extracted as `GiftWrapEvent.unwrapAndUnsealOrNull(signer)` in
commons/relayClient/nip17Dm/, and collapsed:
- commons/marmot/MarmotIngest.kt — was an inline `when` switch; now a
one-liner. Behaviour unchanged (still passes through if the inner
isn't a seal, matching the previous defensive `else -> inner` branch).
- desktopApp/Main.kt — dropped the nested unwrap/unseal block in the
kind:1059 case of the LocalCache dispatcher.
- cli/DmCommands.kt — replaced the manual chain in `decryptChatMessages`.
Android's DecryptAndIndexProcessor keeps its two separate handlers
(processNewGiftWrap / processNewSealedRumor) because the LocalCache
pipeline re-enters on each peel — that's a different shape and not
touched.
No behaviour change on any platform.
Adds three verbs wired into the top-level dispatch:
- `amy dm send RECIPIENT TEXT` — builds a kind:14 ChatMessageEvent,
runs it through NIP17Factory to seal and gift-wrap, then publishes
each wrap to its recipient's DM inbox (kind:10050, fallback kind:10002
read, final fallback bootstrap). Emits one JSON line with the inner
kind:14 id, per-recipient wrap ids, and per-relay ACK status.
- `amy dm list [--peer NPUB] [--since TS] [--limit N] [--timeout SECS]`
— drains gift wraps addressed to us from our inbox relays (or outbox /
bootstrap fallbacks), unwraps+unseals each one, dedupes by inner id,
and returns them oldest-first. With no flags it advances the
giftWrapSince cursor in state.json (matches the Marmot syncIncoming
convention); with --peer/--since it's a stateless query.
- `amy dm await --peer NPUB --match TEXT [--timeout SECS]` — polls the
same drain on a 2s tick until a DM from NPUB contains TEXT. Timeout
exits 124 like the other await verbs.
All three reuse Quartz entirely (NIP17Factory, GiftWrapEvent,
SealedRumorEvent, RecipientRelayFetcher); the only shared piece we
needed in commons — filterGiftWrapsToPubkey — was extracted in the
previous commit. No business logic leaks into cli/.
Plan: cli/plans/2026-04-23-nip17-dm.md.
Design doc for `amy dm send|list|await`, reusing the existing Quartz
gift-wrap (NIP-17/NIP-59/NIP-44) pipeline. Single file under
cli/plans/; extracts FilterGiftWrapsToPubkey from amethyst/ to commons/
as a prerequisite commit so the CLI can depend on it without pulling
in :amethyst.
The first pass only fixed `group add`. Auditing the rest of the CLI
against MIP-00..03 turned up three more spots where `amy` either
queried the wrong relays or silently skipped a required advertisement:
1. `marmot key-package check <npub>` used `anyRelays()` — i.e. the
inviter's configured relays — so checking for a KeyPackage on a
user who advertises `kind:10051` somewhere we don't know about
always returned `not_found`. Now runs RecipientRelayFetcher against
bootstrap seeds and fetches from the union of (target kind:10051,
target kind:10002 write, bootstrap). Emits `found_on` so callers
can see which relay served the hit.
2. `await key-package <npub>` had the same bug inside the poll loop.
Resolved once up front, then the loop fetches from the target's
advertised relays every tick. Throws an `AwaitTimeout` early if no
relays can be discovered at all, instead of silently polling void.
3. `relay publish-lists` published kind:10002 + kind:10050 but never
kind:10051. Per MIP-00 the KeyPackage Relay List is how other
Marmot clients discover where our KPs live — without it the
`key_package` bucket on disk is invisible to anyone else. Now also
publishes kind:10051; falls back to the NIP-65 set if the bucket
is empty so we never advertise an empty list. (`amy create`
already publishes it via AccountBootstrapEvents.)
Docs: cli/README adds a "Relay routing" section that lists the exact
relay set used for publish vs fetch of every Marmot event kind, plus
the bootstrap-pool definition, so agents + interop-test authors can
reason about cross-user reachability without reading the code.
`amy marmot group add` previously sent the Welcome gift wrap to the
inviter's own kind:10050 (ctx.inboxRelays()) and fetched the invitee's
KeyPackage from only the inviter's configured relays. Two users with
disjoint relay configurations could never successfully marmot each
other — the welcome landed somewhere the invitee never polled.
Mirror the Android Account.addMarmotGroupMember flow in the CLI:
- New RecipientRelayFetcher in quartz/marmot one-shot-drains a target
user's kind:10050, kind:10051 and kind:10002 from a seed relay set.
Replaceable-event semantics: newest created_at per kind wins. Guards
against relays echoing events authored by someone else.
- Context.bootstrapRelays() unions the inviter's configured relays with
AmethystDefaults (DefaultNIP65RelaySet + DefaultDMRelayList) so we can
discover strangers even when nothing overlaps with our own config.
- GroupAddMemberCommand now per-invitee: looks up their relay lists;
passes kind:10051 + kind:10002 write + bootstrap to KeyPackageFetcher;
publishes the welcome to kind:10050 (fallback to NIP-65 read, then
DefaultDMRelayList, then our outbox as belt-and-braces).
Group commits/messages (kind:445) continue to use the group's MIP-01
relay set — those were already correct. Exposes welcome_targets and
key_package_relays in the JSON output so callers can verify routing.
The interop harness prints `[cli] ingest <kind>/<id> via <relay> -> Failure`
for any commit/proposal/app message that quartz can't process. Without
the error string it's impossible to tell `confirmation_tag mismatch` from
`parent_hash invalid` from `commit references unknown proposal` without
reading the Kotlin stack. Append `result.message` for Failure outcomes so
the harness log is self-contained when diagnosing quartz→quartz errors
(the openmls side already has `{e:?}` via the mdk-core patch).
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
Five protocol-level fixes and a batch of harness correctness fixes to get
the headless Marmot/Whitenoise interop harness from 1/13 to 5/13 passing
cleanly, with the remaining failures all rooted in wn's per-account
serial event-processor retry backoff (which drops undecryptable
pre-membership commits after several minutes) rather than amy behaviour.
quartz + commons
----------------
* MarmotGroupData: hold CURRENT_VERSION at 2. mdk-core (the Rust MLS
engine used by whitenoise-rs) strict-rejects v3 payloads with
`ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")`
— our v3 welcomes and GCE commits never apply, so every cross-client
group flow dies at welcome processing. We still parse v3 happily on
the way in; we just don't emit it until mdk publishes the
forward-compat fix MIP-01 mandates.
* MarmotManager.updateGroupMetadata: MERGE extensions instead of
REPLACING. RFC 9420 §12.1.7 says GCE proposals blow away the old
extension list; callers that pass only [marmot_group_data] dropped
[required_capabilities], which peers then reject. Preserve every slot
except the one we're updating.
* MarmotManager.createGroup: new optional `initialMetadata` parameter
that bakes MarmotGroupData into epoch-0 GroupContext.extensions
directly. Without it, creators had to publish a pre-membership
"bootstrap" commit that no later joiner could decrypt — each such
peer then burned their retry budget on an undecryptable kind:445
before seeing the real state. Threaded through MlsGroup.create /
MlsGroupManager.createGroup.
* MarmotManager.mlsGroupIdHex: new translation helper so any code
juggling the MIP-01 nostr_group_id (what amy indexes on) and the
MLS GroupContext groupId (what mdk indexes on) can cross-reference
them without reaching into MlsGroupManager directly.
amethyst module
---------------
* Account.leaveMarmotGroup: self-demote before SelfRemove per MIP-01,
and promote a surviving member to admin first if the caller is the
sole admin (otherwise we'd throw "admin depletion"). Matches the
cli/GroupMembershipCommands.leave flow.
cli (amy)
---------
* Context.syncIncoming: don't advance `giftWrapSince` on empty polls
(so the first-ever sync doesn't bump the cursor past every
past-timestamped wrap we've ever been sent), subtract 2 days lookback
when filtering (NIP-59 randomWithTwoDays gift wraps can have any
createdAt in the last 48h), and only advance `groupSince` for groups
we actually received events for.
* Context.syncIncoming: after ingest, if any Welcome consumed a
KeyPackage, rotate and publish a fresh one immediately. MIP-00
requires this — a KP can only be welcomed once and leaving the
consumed one on relays just means later senders invite us with a
bundle we no longer have private keys for.
* Context.resolveGroupId: accept either nostr_group_id (amy's primary
key) or the MLS GroupContext groupId (what wn emits) on every verb
that takes a group id. Wired through GroupAdd/Remove/Leave/Metadata
/Read, Message send/list, and all the await* verbs so harness
scripts never have to juggle both forms for a single group.
* GroupCreateCommand: bake initial metadata into epoch 0 (see the
quartz change above). Dropped the now-redundant bootstrap commit
publish and tightened the JSON output to include `mls_group_id`.
* GroupMembershipCommands.leave: self-demote admin before SelfRemove;
promote an heir if we're the only admin, otherwise the GCE would
deplete admins and the leave aborts.
* MarmotIngest.ingestGiftWrap: unwrap the sealed-rumor layer. NIP-59
wrap is gift-wrap(kind:1059) → seal(kind:13) → rumor; the old code
only unwrapped once and then checked `inner.kind == 444`, which is
always false because inner is actually the seal. Unseal once more
before the Welcome check. This single fix is what unsticks every
amy-side Welcome ingestion.
wn harness patches + scripts
----------------------------
* whitenoise-defaults-env.patch: honour $WHITENOISE_DISCOVERY_RELAYS in
`Relay::defaults()` (release builds otherwise bake damus.io / primal
/ nos.lol into every new account's NIP-65 / Inbox / KeyPackage
lists, which breaks publishing and prevents the inbox subscription
plane from ever reaching an operational state in a sandbox).
* setup.sh: sleep 2s after amy's initial kind:30443 publish so
nostr-rs-relay has a chance to fsync before wn's first targeted
discovery query. Without it wn's `keys check` races the relay's
WAL flush and intermittently returns NotFound.
* lib.sh: peel wn's `{"result": …}` wrapper in `jq_group_id`,
`wait_for_invite`, `wait_for_message`, `wait_for_member`. Post-v0.2
wn `--json` output nests everything under `.result` (and
`groups invites[]` nests further under `.group.mls_group_id`) —
these helpers were still pattern-matching on the flat shape, so
they returned empty strings for a perfectly good response.
* tests-{create,manage,extras}.sh: track both group IDs per test
(amy's nostr + wn's MLS), pass each CLI the id it understands, and
bump the post-commit wait timeouts to 90–120s so wn's exponential
retry backoff has time to work through the pre-membership commits
it can't decrypt and get to the ones it can.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
Add cli/README.md covering the JSON-output contract, install via
installDist, today's command surface (identity / relays / marmot),
and use from agents + interop tests. Add cli/DEVELOPMENT.md with the
feature-parity matrix vs Amethyst, the extract-from-Android recipe
(amethyst/ -> commons/ -> cli/), command template, EventRenderer
plan shared with Desktop, testing strategy focused on what quartz
and commons don't already cover, distribution matrix across macOS /
Windows / Linux / Nix / Zapstore, and an ordered 10-step roadmap.
https://claude.ai/code/session_01BQ5ZHwa8BAgEQ9zeM4CKhW
Jackson was writing the computed \`hasPrivateKey\` into identity.json
and then refusing to read it back ("Unrecognized field hasPrivateKey")
on the next invocation. All amy commands that reload the data-dir
(\`marmot group create\`, \`marmot key-package publish\`, …) exited 1 at
the Context.open step. Pure boolean getter, no persistence needed.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Move the "defaults a new Amethyst account gets seeded with" into commons
so the UI and amy agree byte-for-byte on what a fresh account looks like.
commons additions:
- commons/defaults/Constants.kt — relay URL constants (moved from amethyst/)
- commons/defaults/AmethystDefaults.kt — DefaultChannels, DefaultNIP65RelaySet,
DefaultNIP65List, DefaultGlobalRelays,
DefaultDMRelayList, DefaultSearchRelayList,
DefaultIndexerRelayList (moved from
amethyst/model/AccountSettings.kt)
- commons/account/AccountBootstrapEvents.kt — data class + bootstrapAccountEvents(signer, name)
that builds the nine events a new Amethyst account publishes: kind:0, 3,
10002, 10050, 10051, 10099, 50, 51, + channel list. One source of truth.
Retrofits:
- AccountSessionManager.createNewAccount now delegates event construction to
the commons helper; it still packages them into AccountSettings for the
Android storage path.
- DefaultSignerPermissions stays in AccountSettings.kt — it uses Android
NIP-55 Permission/CommandType types.
- 19 import updates across amethyst/ to point at the new commons paths.
New CLI verbs:
- amy create [--name NAME]: mint a keypair, write identity.json, seed
relays.json with the Amethyst defaults, publish all nine bootstrap events
to DefaultNIP65RelaySet via NostrClient.publishAndConfirmDetailed.
- amy login KEY [--password X]: accept any identifier form Amethyst's login
screen accepts — nsec1, ncryptsec (+ --password, NIP-49), BIP-39 mnemonic
(NIP-06), npub1, nprofile1, 64-hex pubkey (default) or 64-hex privkey
(with --private), NIP-05 (name@domain.tld, HTTP lookup). Read-only keys
land with privKeyHex=null; Identity now carries that cleanly.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Five extractions so the Amethyst UI and the new amy CLI share one
implementation for each piece of Marmot/identity behaviour instead of
reimplementing it per platform.
quartz additions:
- MarmotGroupData.bootstrap(..) + withMergedRelays(..): factory for the
metadata shape every inviter stamps on a fresh group and the outbox-
merge rule every admin commit carries forward.
- KeyPackageFetcher: (a) pure fetchRelaysFor union of target kind:10051,
target outbox, my outbox; (b) one-shot fetchKeyPackage; (c) publish-
side resolveKeyPackagePublishRelays.
- resolveUserHexOrNull: single entry point for npub / nprofile / 64-hex
/ NIP-05 identifiers. Uses the existing Nip19Parser + Nip05Client
infra so NIP-05 resolution is live over HTTP.
commons additions:
- MarmotManager.leafIndexOf(..): pubkey -> leaf index lookup used by
every removeMember caller.
- MarmotManager.addMember(nostrGroupId, keyPackageEvent, relays):
convenience overload that lifts the base64 decode into the manager.
- MarmotManager.buildTextMessage(..) returning a TextMessageBundle;
optionally persists the outbound inner event (CLI needs persist,
Amethyst relies on LocalCache loopback).
- MarmotIngest.ingest(event): routes kind:1059 / kind:445 through
unwrap -> processWelcome / processGroupEvent -> persist the decrypted
application message. Deliberately platform-agnostic.
Retrofits:
- Account.addMarmotGroupMember and fetchKeyPackageAndAddMember now take
a KeyPackageEvent directly; Account.keyPackagePublishRelays delegates
to KeyPackageFetcher.
- AccountViewModel.updateMarmotGroupMetadata uses bootstrap + merged.
- AccountViewModel.sendMarmotGroupMessage builds the kind:9 template via
buildTextMessage(persistOwn = false) to avoid double-persisting.
- AccountSessionManager's NIP-05 login branch delegates to
resolveUserHexOrNull; no more hand-rolled Nip05Id / nip05Client.get.
- CLI Context exposes nip05Client and requireUserHex; syncIncoming now
calls MarmotManager.ingest; all command sites use KeyPackageFetcher
+ the shared identifier resolver. Npubs util deleted.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
New :cli JVM module producing an `amy` binary that drives Amethyst
functionality headlessly. Identity and relay configuration sit at the
root (`amy init`, `amy whoami`, `amy relay …`); Marmot/MLS lives under
`amy marmot …` so future verbs (dm, feed, profile) can slot in cleanly.
Marmot surface covered:
- key-package publish / check
- group create / list / show / members / admins / add / rename /
promote / demote / remove / leave
- message send / list (kind:9 inner events)
- await key-package / group / member / admin / message / rename / epoch
(non-interactive polling with --timeout; exit 124 on timeout)
Wiring:
- NostrClient + BasicOkHttpWebSocket.Builder, reusing the existing
publishAndConfirmDetailed and fetchFirst accessories
- MarmotManager from :commons with file-backed MlsGroupStateStore,
KeyPackageBundleStore, and MarmotMessageStore (unencrypted — scratch
harness use only)
- each invocation is one-shot: prepare → syncIncoming → run command →
persist → disconnect
All commands emit one JSON object on stdout; diagnostics on stderr.
Designed so shell harnesses can pipe through jq without bespoke parsing.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
davotoula