nestsClient is a KMP module with both JVM and Android targets. Add it
to the desktop matrix's explicit test list so its JVM tests run on all
three desktop OSes alongside quartz/commons/cli/desktopApp.
The merged test-and-build matrix hid Android behind an OS-keyed matrix
leg, so the workflow graph no longer showed a distinct Android node.
Restore visibility by splitting into two parallel jobs after lint:
- build-desktop (matrix: ubuntu/macos/windows): runs JVM tests for the
KMP/JVM modules (quartz, commons, cli, desktopApp) and packages the
native installer per OS.
- test-and-build-android (ubuntu): runs the full test sweep and
assembleBenchmark, uploads APKs and reports.
Trade-off: Ubuntu spins up two parallel runners instead of one, but
wall time is unchanged and Android is once again a visible node.
The GroupEventResult sealed class gained a ProposalStaged variant for
standalone Proposal events that get parked in the pending pool without
advancing the epoch. The when in DecryptAndIndexProcessor.add wasn't
exhaustive, breaking compileFdroidDebugKotlin and compilePlayDebugKotlin.
Logs the result at DEBUG since there's no UI work to do until a later
Commit references the proposal by hash.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The PROPOSAL branch of `MlsGroup.decrypt` previously threw
`IllegalStateException("Standalone PrivateMessage proposals not yet
supported")`. That worked in the openmls-as-Whitenoise topology because
MDK emits SelfRemove as a PublicMessage — but any peer using the
AlwaysCiphertext wire-format policy (RFC 9420 §6.3.2) wraps standalone
proposals in PrivateMessage, and we'd hard-fail on first contact.
Implements the PROPOSAL branch symmetrically with the existing COMMIT
branch:
1. Decode the Proposal struct from PrivateMessageContent (no length
prefix), read signature<V>, drain zero-padding.
2. Restrict to SelfRemove (mirrors `receivePublicMessageProposal`'s
policy — the only standalone proposal type that needs to ride
outside a commit; widening is one-line if interop demands it).
3. Verify the FramedContentTBS signature with `wire_format =
PRIVATE_MESSAGE` against the sender's leaf signature_key.
4. Stage the proposal in `pendingProposals` together with the encoded
AuthenticatedContent (wire_format ‖ FramedContent ‖ signature) so
a subsequent inbound commit folding it in by ProposalRef can
resolve via the §5.2 hash. PrivateMessage AC bytes carry no
membership_tag — auth = signature only.
Adds a symmetric `encryptProposalAsPrivateMessage` helper (internal,
mirrors `encrypt` for application data) so tests can exercise the
inbound path with a real wire frame produced by the same code path
peers use.
2 new tests: round-trip Bob→Alice SelfRemove via Welcome flow with
independent MlsGroup instances, verifies the proposal lands in
Alice's pending pool with AC bytes captured; and rejection of a
non-SelfRemove standalone PrivateMessage proposal (PSK in the test).
Marmot test suite green; marmot-interop-headless 16/16.
Closes audit gap #1.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
idx/tag/<name>/ used to bucket every tag value under a 16-hex Murmur
hash, so `ls idx/tag/p/` showed opaque hex even though pubkey values
are perfectly safe filenames. Now the directory name is:
- the raw tag value, when it's filesystem-safe across Linux/macOS/
Windows: 1..180 bytes, printable ASCII (0x21..0x7e), none of
`/ \ : * ? " < > |`, no leading dot or `_h_`, no trailing dot/space;
- otherwise `_h_<hashHex(murmur)>`. The `_h_` sentinel can never
collide with a raw value (raw values are forbidden from starting
with it), so both forms safely live in the same parent dir.
Common cases keep their raw form and become directly inspectable:
ls idx/tag/p/<your_pubkey>/ — every event that p-tagged you
ls idx/tag/e/<event_id>/ — replies / reactions to an event
ls idx/tag/t/nostr/ — every kind-1 with #nostr
ls idx/tag/k/30023/ — k-tag pointers to articles
ls idx/tag/g/drt3n/ — geohash mentions
Routes through the _h_ bucket: emojis, URLs (slashes), `a`-tags
(colons), free-form `alt` text, anything ≥ 180 bytes, anything that
breaks Windows reserved-name rules. The hash is still seed-salted
Murmur64 (parity with the previous bucket), so collisions are caught
by FilterMatcher post-filter just like before.
Writer (FsIndexer.pathsFor) and reader (FsQueryPlanner.firstTagKey)
both route through FsLayout.tagValueDirName, so they always agree.
Tests: 5 new in FsQueryTest covering raw ASCII tags landing under
named dirs, p-tag pubkeys keeping their raw form, emoji and URL tags
falling back to _h_, and round-trip queries hitting both buckets
correctly. 122 fs tests green.
No migration: existing stores must `amy store scrub` once after the
upgrade — old purely-hashed tag dirs become unreachable, and scrub
rebuilds idx/ from canonicals using the new naming.
Two DoS surfaces in the inbound path:
**#8 — unbounded forward-ratchet on PrivateMessage decrypt.** A
malicious sender (or any peer who can put bytes in the wire frame)
controls the `generation` field of an inbound PrivateMessage. The
SecretTree was happy to fast-forward the sender's application or
handshake ratchet by however many steps that field implied — every
step costing one HKDF-Expand. A single packet with `generation =
2^31` would have pinned a CPU at SHA-256 for minutes. The skipped-key
cache (`MAX_SKIPPED_KEYS = 1000`) bounds memory but NOT compute: it
just stops *caching* past the cap, the ratchet keeps walking.
Adds `MAX_RATCHET_STEPS_PER_CALL = 4096` and rejects any decrypt that
asks for a larger jump from the sender's current head, applied at
both `applicationKeyNonceForGeneration` and
`handshakeKeyNonceForGeneration`. 4096 leaves room for legitimate
catch-up (mobile waking from a long sleep) while bounding the
worst-case per-packet cost to ~4096 SHA-256 invocations.
**#11 — oversized PrivateMessage AAD allocation.** The underlying
[TlsReader] already caps every opaque<V> read at 1 MiB, so the
ciphertext field is bounded — but `authenticated_data` and
`encrypted_sender_data` were also allowed up to 1 MiB even though
both fields legitimately carry a few hundred bytes at most. A
pre-verification frame would force ~3 MiB of allocation per inbound
packet. Tightens both fields to 64 KiB at PrivateMessage decode
(below TlsReader's global cap) so the per-frame floor is predictable.
2 new tests: `secretTree_rejectsRatchetJumpsBeyondCap` exercises the
boundary (4096 OK, larger throws with a "jump too large" message)
and `privateMessage_rejectsOversizedAuthenticatedData` hand-builds
a frame with a 65 KiB AAD field and confirms rejection. Marmot
test suite green; interop 16/16.
Closes audit gaps #8 and #11.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Two RFC 9420 hardening gaps in the inbound path:
**#3 — UpdatePath silent decrypt failures.** When `processCommit` was
handed a commit whose UpdatePath either (a) had no node at our common
ancestor with the sender or (b) had no ciphertext encrypted to a node
in our copath resolution, the path-decrypt block silently fell through
and left `commit_secret = 0`. The downstream confirmation_tag check
caught it and rolled back, but the rollback surfaced as a generic
"ConfirmationTagMismatch" instead of pointing at the real cause —
a tree-shape mismatch between our view and the sender's. Now hard-fail
with a precise error naming the indices involved.
**#6 — parent_hash chain verification on Welcome.** `processWelcome`
was checking the GroupInfo signature and the GroupContext.tree_hash,
but neither gates a forged parent_hash inside a COMMIT-source leaf —
the tree_hash check only proves the ratchet_tree extension matches
the bytes the signer signed, not that the stored parent_hash values
are consistent with the tree shape. A peer that DOES validate
parent_hash (per RFC 9420 §7.9) would reject every commit produced
from such a tree, splitting the group on the next epoch.
Adds `verifyTreeParentHashesForJoin(tree)` as a static-tree
counterpart to `verifyParentHash` (which only handles the
post-UpdatePath dynamic case). For each COMMIT-source leaf, recompute
the parent_hash chain top-down on the leaf's filtered direct path and
compare to the stored value. Returns null on success or a human-
readable mismatch reason. Wired into `processWelcome` right after the
tree_hash check, before any capability or key-schedule work.
Also moves `encodeParentHashInput` into the companion object so the
static and instance variants share one TLS encoder, and adds an
`exportTreeBytes()` test accessor.
2 new tests: trivial single-member tree accepts; a tampered
COMMIT-source parent_hash on a 3-member tree is rejected with a
message naming the offending leaf. Quartz marmot tests green;
marmot-interop-headless 16/16. (Two unrelated NostrClient network
tests fail under the full :quartz:jvmTest run — pre-existing flake,
no MLS code touched.)
Closes audit gaps #3 and #6.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
When a group installs a `required_capabilities` extension (every Marmot
group does, listing MarmotGroupData=0xF2EE, SelfRemove=0x000A, Basic
credential), every member's leaf MUST advertise those types in its own
[Capabilities]. We were validating none of that:
- `applyProposalAdd` checked version + ciphersuite but never matched the
new KP's capabilities against the group's required_capabilities. A
non-conformant member silently joined; the next commit that touched
their leaf got rejected by spec-conformant peers, splitting the group.
- `processWelcome` similarly never checked the joiner's own KP against
the group's required set, nor did it sweep existing members' leaves.
A misconfigured GroupInfo signer could invite us into an incoherent
group whose first commit we'd silently reject forever.
Adds two helpers in `MlsGroup.Companion`:
- `findRequiredCapabilities(extensions)`: decodes the §7.2 struct from
a GroupContext extension list, or returns null if absent.
- `requireCapabilitiesMeetRequirements(caps, req, who)`: throws with a
specific (extensions=, proposals=, credentials=) diff naming the
missing types — turns silent interop breaks into one debuggable line.
Wires the gate in two places:
- `applyProposalAdd` rejects the Add proposal if the new leaf falls
short of the group's required set.
- `processWelcome` rejects the join if either (a) our own KP doesn't
meet the group's requirements or (b) any existing member's leaf
doesn't — the latter catches a malformed GroupInfo at join time.
5 new tests: round-trip decoding, rejection on missing extension /
proposal, acceptance when caps are a superset, and an end-to-end
`addMember` rejection of a hand-crafted KP with SelfRemove stripped
(re-signed so the rejection is from the capability gate, not the
signature check). Quartz test suite green; marmot interop 16/16.
Closes audit gaps #4, #5, #10.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
assert_eq is (actual, expected, test_id, note); first version of the
script had test_id and actual swapped, so even passing assertions
showed up as fails with confusing "got T2.source" messages. Now all
21 assertions pass with the right arg order. Each successful assertion
also fires record_result pass so the results table covers them
(previously the tests passed but were invisible — only the explicit
record_result calls showed up).
Verified end-to-end on a real local nostr-rs-relay:
21 passed, 0 failed, 0 skipped (of 21)
Coverage:
T1.* — store stat reports kind histogram + disk usage after publish
T2.* — self profile show is served from cache by default
T3.* — --refresh forces source: relays
T4a/b — B's first lookup of A is a relay miss; second is a cache hit
T5.* — relay list reads URLs back from local kind:10002/10050/10051
T6 — relays.json is gone from both data-dirs
T7.* — store maintenance verbs work without an identity
Three changes that go together:
1. Reconcile cli/plans/2026-04-24-file-event-store-{overview,nips}.md
with shipped reality: code lives in quartz/jvmMain/, not commons/;
data dir is <data-dir>/events-store/, not <root>/events/.
2. New StoreCommands wired as `amy store …`:
- stat → events count, kind histogram, disk bytes,
oldest/newest createdAt. Pure read, no Context
(skips identity check).
- sweep-expired → wraps store.deleteExpiredEvents(); reports
{swept, remaining}.
- scrub → wraps store.scrub() to rebuild idx/ from
canonicals.
- compact → wraps store.compact() to drop dangling idx/.
All four open the FsEventStore directly (no Context, no identity
needed) — they're store-only operations.
3. New e2e harness at cli/tests/cache/cache-headless.sh that boots a
local nostr-rs-relay, two amy identities (A + B), and asserts:
T1 — store stat reports non-empty store after publish-lists +
profile edit, with kind:0 and kind:10002 present.
T2 — A's `profile show` is `source: "cache"` by default.
T3 — `--refresh` forces `source: "relays"`.
T4 — B's first `profile show <A_NPUB>` is a relay miss; second is
a cache hit (proves drain populates the local store and
subsequent reads serve from disk).
T5 — `relay list` reads URLs back from the local kind:10002 /
10050 / 10051 events.
T6 — relays.json no longer exists in either data-dir.
T7 — store stat / sweep-expired / scrub / compact all run
without an identity present.
Same pattern as cli/tests/dm/dm-interop-headless.sh — reuses the
nostr-rs-relay infrastructure from cli/tests/marmot/setup.sh.
Three audit gaps in the Marmot MLS implementation, fixed together because
they share the same call sites:
1. **MIP-03 inbound authorization gate.** `enforceAuthorizedProposalSet`
only fired for *outbound* commits (it implicitly checked the local
member). A peer could send us a non-admin GroupContextExtensions
rename, a non-admin Remove, or an admin-emptying GCE and we would
silently apply it. `enforceAuthorizedProposalSet` now takes an explicit
`committerLeafIndex` (defaults to `myLeafIndex` for the local case)
and uses `isLeafAdmin(committerLeafIndex)` instead of `isLocalAdmin()`.
`processCommitInner` resolves the proposal list against our pending
pool, then runs both `enforceAuthorizedProposalSet` and
`enforceNoAdminDepletion` against the resolved set before applying.
External commits skip the check (sender has no leaf yet, and §12.4.3.2
already restricts the proposal list).
2. **RFC 9420 §5.3 psk_secret derivation.** The previous
`computePskSecret` HKDF-Extracted bare PSK values with the running
`pskSecret` as salt and ignored `psktype` / `psk_nonce` / index /
count entirely — incompatible with any spec-conformant peer. Per
§5.3 each step is now:
psk_extracted_i = HKDF.Extract(0, psk_i)
psk_input_i = ExpandWithLabel(psk_extracted_i, "derived psk",
PSKLabel(id_i, i, n), Nh)
psk_secret_i = HKDF.Extract(psk_secret_{i-1}, psk_input_i)
`buildPskLabel` encodes the full `PreSharedKeyID || index || count`
struct. Resumption PSKs (`psktype == 2`) reject loudly because
`Proposal.Psk` lacks `(usage, psk_group_id, psk_epoch)` — silently
encoding a broken PSKLabel would diverge from peers without warning.
3. **ProposalRef hash for locally-published proposals.** RFC 9420 §5.2
hashes the encoded AuthenticatedContent, not the bare Proposal.
`buildSelfRemoveProposalMessage` now stages the published proposal in
`pendingProposals` together with the AC bytes (wire_format ‖
FramedContent ‖ FramedContentAuthData), so a subsequent inbound
commit that folds it in by ProposalRef can resolve the hash. Bare-
proposal fallback in `processCommitInner` retained for legacy local
entries that pre-date the capture.
Tests: 7 new cases in `MarmotMipBehaviorTest` covering inbound non-admin
rejection (Add / GCE rename), admin-depletion rejection, AC bytes capture,
PSK empty/single/ordering/resumption-rejected paths. Full quartz JVM test
suite passes; marmot-interop-headless 16/16.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Users actually look at <data-dir>/events-store/ files (cat / jq / git
diff), so the CLI now writes them with the InliningTagArrayPrettyPrinter
that quartz already had configured but never invoked. Each event is
indented with 2 spaces, but every tag array stays on a single line —
nice trade-off between human-readable and not-too-tall:
{
"id": "...",
"pubkey": "...",
"created_at": 1700000000,
"kind": 1,
"tags": [
["t","nostr"],
["e","abc...a"],
["alt","quick brown fox"]
],
"content": "hi there",
"sig": "..."
}
Stored bytes are not the canonical NIP-01 form — but verification
re-canonicalises via EventHasher anyway, so format is purely a UX
choice. Compact stays the default for any caller that doesn't opt
in (Android keeps SQLite, generic FsEventStore embedders keep
compact).
- JacksonMapper.toJsonPretty(event): new entry point that uses
writerWithDefaultPrettyPrinter() with the existing inlining printer.
- FsEventStore now takes an `eventToJson: (Event) -> String` callback,
default = Event::toJson (compact). Used in insert.
- Context wires in JacksonMapper::toJsonPretty.
2 new tests in FsEventToJsonTest pin both formats (compact stays
single-line; pretty round-trips). 117 fs tests green.
The local relay configuration was redundant once the event store
became the source of truth: every account already publishes signed
kind:10002 (NIP-65), kind:10050 (DM inbox), and kind:10051
(KeyPackage relays) events, and Context now reads each set straight
out of the local store.
Removed:
- data class RelayConfig (nip65/inbox/keyPackage buckets)
- DataDir.relaysFile / loadRelays() / saveRelays()
- Context.relays parameter — Context.open() no longer reads JSON
- CreateCommand.defaultRelayConfig() helper
- CreateCommand's saveRelays() call (redundant: ctx.publish persists
the bootstrap events into the store anyway)
Context now serves the four relay accessors from the store with sane
fallbacks:
outboxRelays() = relaysOf(self)?.writeRelaysNorm() ?: DefaultNIP65RelaySet
inboxRelays() = dmInboxOf(self)?.relays() ?: DefaultDMRelayList.toSet()
keyPackageRelays() = keyPackageRelaysOf(self)?.relays() ?: outboxRelays()
anyRelays() = union of the three
bootstrapRelays() = anyRelays() ∪ DefaultNIP65RelaySet ∪ DefaultDMRelayList
RelayCommands rewritten:
- `relay add URL --type T` — read existing relay-list event from the
store, append URL, build + sign + ingest a new event of the
matching kind. The replaceable slot mechanism replaces the old
winner atomically.
- `relay list` — dump URLs from the latest event in each bucket.
- `relay publish-lists` — broadcast whichever events are present in
the store; errors out cleanly when none exist (suggests `relay add`
or `create`).
No migration: existing data dirs that still have `relays.json` will
ignore it. Run `amy relay add` or `amy create` to repopulate. Per
discussion this is a clean break, not a backwards-compat shim.
README + DEVELOPMENT updated to reflect the new on-disk layout (no
more `relays.json`; events-store/ is the relay configuration).
Audit found four commands draining replaceable relay-list events
(kind:10050, 10051, 10002) on every invocation, plus one fetching
the user's own kind:3 contact list. All five are now cache-first.
New Context helpers:
- dmInboxOf(pk) → kind:10050 (ChatMessageRelayListEvent)
- keyPackageRelaysOf(pk)→ kind:10051 (KeyPackageRelayListEvent)
- cachedRelayListsOf(pk)→ assembles a RecipientRelayFetcher.Lists
from the local store; returns null if
nothing is cached so callers can fall
back with `?:`.
Wired into:
- FeedCommand.resolveFollowing — replaces a kind:3 drain with
ctx.contactsOf(self).
- DmCommands.resolveDmRelays — `amy dm send` / `dm list`.
- KeyPackageCommands.check.
- AwaitCommands (key-package / member / admin polling loops).
- GroupAddMemberCommand (one cache hit per invitee).
All five sites now go: try cache → if miss, run the existing
RecipientRelayFetcher / drain. The drain itself populates the cache
via verifyAndStore, so subsequent runs are local-only. Replaceable
slot shortcut means each cached read is one stat + one readString,
not a directory walk.
README "Local event store" section gains a list of which commands
are cache-first today.
The first command to consume the local event store as a read source.
Default behaviour: read from the FsEventStore via Context.profileOf().
Pass --refresh to skip the cache and drain from relays (which then
re-populates the store as a side effect, like every other drain).
Output JSON gains a "source" field so scripts/agents can tell whether
the result came from disk or the network:
source = "cache" → served from <data-dir>/events-store/, queried_relays = []
source = "relays" → fresh relay drain, queried_relays = list
Cache hit goes through the planner's slot shortcut from the previous
commit: one stat + one read on replaceable/0/<pubkey>.json, no
directory scan.
profile show + profile edit now have entries in the verb table in
cli/README.md.
`profileOf(pk)` and `relaysOf(pk)` are the two hottest read patterns
in any Nostr client, both expressed as `Filter(authors=[pk],
kinds=[0|10002])`. Before this change every such query walked
`idx/kind/<k>/`, sorted the listing, and probed candidates — O(N)
in the kind's total event count even though we always wanted exactly
one event.
The planner now intercepts before the directory walk: when a filter
pins us to replaceable / addressable kinds with `authors` (and
`d`-tag for addressables), we read the slot file directly. One
`Files.exists()` + one `readString()` per (kind, author[, d]) triple,
no listing, no sort.
Falls through to the generic walk for anything that doesn't fit
(non-uniqueness kinds, missing d-tag, search clause, etc.). Two
new tests in FsSlotsTest assert the shortcut serves correctly
even when `idx/` has been externally wiped — proving the
shortcut isn't accidentally relying on the index.
115 fs tests green.
Audit cleanup — three correctness-adjacent fixes:
#13 — FsLayout.readOrCreateSeed was TOCTOU-racy for two processes
opening the same fresh directory. Both could pass `!exists()`, both
write their own random bytes, both ATOMIC_MOVE into place. Loser's
seed is forgotten but they already hashed with it, so their future
index entries would be unreachable. Switch to CREATE_NEW: exactly one
process creates the file, the other catches FileAlreadyExistsException
and falls through to read the winner's bytes.
#9 — `catch (_: Throwable)` / `catch (_: Exception)` in FsEventStore,
FsSlots, FsTombstones narrowed to `catch (_: IOException)` plus
`catch (_: JacksonException)` for JSON-parse paths. The old catches
swallowed CancellationException (hides coroutine cancellation),
OutOfMemoryError, and ThreadDeath. Now only the errors we actually
expect on a racing read are suppressed.
#1 — FsLockManager replaced synchronized+ThreadLocal+depth-counter
with a ReentrantLock. Same semantics (cross-process flock + in-process
reentry), cleaner code: the ReentrantLock handles reentry natively
via holdCount, no ThreadLocal, no manual depth bookkeeping. Release
is guarded by `holdCount == 1` on exit. Narrow catches in the
cleanup path to IOException too.
113 fs tests green across all suites.
Two query-planner hotspots called out by the audit:
#2 — mergeDesc used to materialise every driver's full stream into
one ArrayList before sorting, so `limit = 10` against a million-event
store would still load and sort the full million before emitting
anything. Replace with a lazy k-way merge on a max-heap of per-stream
heads. Memory is now O(num streams) instead of O(sum of stream
sizes), and `limit` short-circuits naturally after the first N pops.
walkDir also sorts filenames (cheap strings) instead of Candidate
objects (bigger) — same DESC order because our `<padded_ts>-<id>`
convention makes lex-reverse == chronological DESC.
#3 — ftsDriver used to load every search token's full listing into
HashMap<HexKey, Long> before intersecting, so `search = "the bitcoin"`
against a popular token would spike memory proportional to that
token's size. Switch to smallest-first: count each token dir, drive
the walk with the smallest, and confirm each candidate in the others
via a single `Files.exists()` per (candidate, token). Works because
every FTS hardlink for an event shares the same `<padded_ts>-<id>`
filename, so stat-check is a direct lookup. Memory is now
O(smallest_token_size) — no HashMap for the big tokens.
113 fs tests green, FsSearchTest verifies the ordering, AND semantics,
and limit behaviour that these touch.
The clock injection existed only for NIP-40 expiration tests. A public
constructor parameter that ~every caller ignores is API clutter, so
move it to a subclass seam:
- FsEventStore is now `open class`; no more `clock: () -> Long` param.
- `protected open fun now(): Long = TimeUtils.now()` is the override
point. Production always takes the default; tests subclass.
- FsExpirationTest gains a private `ClockedStore(root, source)` that
overrides `now()`. Semantics unchanged, all 113 fs tests still green.
Public `FsEventStore` ctor is now `(root, indexingStrategy, relay)` —
no behavioural-drift surface that callers have to learn.
`marmot message react "$gid" "$id" "🍕"` was producing a kind:7 whose
inner content was four `U+FFFD` replacement characters instead of the
emoji. Whitenoise's `message_aggregator::emoji_utils` rejected it with
`Invalid reaction content: ����`, and the openmls receiver retried the
event ten times before giving up — by which point the secret tree had
already consumed that generation, so the retry surfaced as a confusing
`Ciphertext generation out of bounds 1 / SecretReuseError`. That last
error was a downstream symptom; the root cause is argv decoding.
Why it happens: JEP 400 (Java 18+) pins `file.encoding` to UTF-8 but
deliberately leaves `sun.jnu.encoding` — the encoding the JVM uses to
decode argv, filenames, and native interop strings — tied to the OS
locale, and `sun.jnu.encoding` is read **before** the JVM parses any
`-D` flag, so it can't be overridden via `applicationDefaultJvmArgs`.
Under POSIX/C locales (no `LANG`/`LC_ALL` set, common on CI runners
and stripped-down containers) `sun.jnu.encoding` ends up at
`ANSI_X3.4-1968` — i.e. ASCII — and every byte > 0x7F in argv lands
as `U+FFFD` before our code ever sees it. The four UTF-8 bytes of 🍕
(F0 9F 8D 95) become four replacement chars, get signed into the kind:7
content, and the test fails on receipt.
The only place we can set the encoding from is the launching shell.
Patch the gradle-generated start scripts (POSIX `amy`, Windows
`amy.bat`) right after `:cli:startScripts`:
* POSIX: if neither `LANG` nor `LC_ALL` is set, `export LANG=C.UTF-8`
before invoking Java. Existing locale settings are respected.
* Windows: `chcp 65001` to switch the console to UTF-8 before Java
runs.
Tested manually: with `LANG=` (sandbox default) `amy` previously sent
`argv[0]=????` for an emoji argument; with the patch it sends `🍕` and
`sun.jnu.encoding=UTF-8`, and whitenoise accepts the kind:7 reaction.
The interop test 09 polling also needed a fix unrelated to the
encoding bug: wn aggregates kind:7 reactions onto the anchor message
under `.reactions.by_emoji.<emoji>` instead of surfacing them as
standalone messages whose `.content` is the emoji, so the previous
`wait_for_message B "$mls_gid" "🍕"` would have failed even with a
correctly-decoded reaction. The new poll inspects each message's
`.reactions.by_emoji` keys for the emoji literal.
Marmot interop score: 15/16 → **16/16** 🎉https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Sibling to the SQLite store's README. Covers:
- what the store is and why (filesystem primitives = invariants:
directory-entry uniqueness IS the UNIQUE constraint, rename(2) IS
the atomic commit, hardlink refcount IS the FK cascade, flock(2)
IS the writer serialisation),
- feature-parity table vs SQLite (NIP-01 replaceable / addressable,
NIP-09 deletion + tombstones, NIP-40 expiration, NIP-45 count,
NIP-50 search, NIP-62 vanish, NIP-91 multi-tag AND),
- on-disk layout with every directory tree,
- filename conventions (sharding, padded-ts entry names, sha256 vs
Murmur, mtime),
- internal architecture (FsLayout / FsLockManager / FsIndexer /
FsSlots / FsTombstones / FsQueryPlanner / FsSearchTokenizer),
- usage examples for insert / query / count / delete / transaction /
expiration sweep / scrub / compact / close,
- failure-mode table (external edits, crashes, multi-process,
full disk) and how the store converges,
- pointer to the 113-test suite under jvmTest and the design plan
documents under cli/plans/.
Two bugs that conspired to break interop test 15 once the test 14 OOM
was fixed:
1. **No receive path for standalone PublicMessage proposals.**
wn/openmls publishes a non-admin's `SelfRemove` as a kind:445 carrying
a `PublicMessage(content_type=PROPOSAL)` envelope and waits for an
admin to fold it into the next commit. Quartz's `MarmotInboundProcessor`
answered every such event with `Error("Standalone proposals not yet
supported")` and dropped it. The admin's subsequent commit then
failed with `Commit references unknown proposal (ref not found in
pending proposals)` because nobody had staged the SelfRemove.
Add `MlsGroup.receivePublicMessageProposal(pubMsg)` that:
- rejects mismatched epoch / group_id / sender,
- reconstructs the FramedContentTBS exactly as the proposer did and
verifies the leaf signature,
- verifies the membership_tag against the current epoch's
membership_key (same threat model as inbound PublicMessage commits),
- decodes the inner Proposal (only `SelfRemove` is accepted today —
other types come bundled in commits' `proposals` lists),
- stages the proposal in `pendingProposals` so a later commit can
resolve its `ProposalRef`.
`processPublicMessage` now routes `ContentType.PROPOSAL` through
that helper and returns a new `GroupEventResult.ProposalStaged`
variant (also surfaced as `MarmotIngestResult.ProposalStaged`),
replacing the old hard-error path.
2. **Wrong `ProposalRef` hash input.** RFC 9420 §5.2 specifies that a
`ProposalRef` hashes the **encoded `AuthenticatedContent`** that
delivered the proposal, not the bare `Proposal` struct. Quartz was
hashing `proposal.toTlsBytes()` only — fine for our local-only
flows where commits inline rather than reference our own pending
proposals, but fatal once we needed to match wn's reference to an
inbound proposal.
Extend `PendingProposal` with an optional `authenticatedContentBytes`
field. The standalone-proposal receive path captures the full
`wire_format || FramedContent || FramedContentAuthData` envelope at
stage time. The reference-resolution code in `processCommitInner`
prefers those bytes when present and falls back to the bare-proposal
hash for locally-proposed entries (which never get referenced
today).
Marmot interop score: 14/16 → 15/16 (test 9 — amy's kind:7 reaction
triggers a `SecretReuseError` on B's wn — is unrelated to this path
and remains for follow-up).
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Replace the inline MessageDigest + hand-rolled hex converter with
quartz.utils.sha256.sha256 + quartz.utils.Hex.encode. Same output
(deterministic algorithm), but we stop duplicating primitives that
already exist in the codebase. One fewer JCA dependency, one fewer
hex-conversion implementation to audit.
All 113 fs tests still green, including FsParityTest vs the SQLite
reference — the d-tag slot paths are byte-identical to before.
Every event Amy observes — drained from a relay subscription,
unwrapped from a NIP-59 gift wrap, or generated locally for publish —
now flows through Context.verifyAndStore: NIP-01 id + signature check
via Event.verify(), then store.insert(). Bad events are dropped with
a stderr log and never reach command code; persistence failures are
logged but do not propagate, so a broken store never breaks a relay
subscription.
- Context.drain() now persists every received event before surfacing
it to the caller. Existing callers (FeedCommand, DmCommands,
ProfileCommands, Marmot sync) get caching for free.
- Context.publish() persists outbound events too, so the local store
reflects what Amy has done even when every relay rejects.
- Three cache-first read helpers expose the most common lookups
without hitting relays: profileOf(pubKey) → kind:0,
relaysOf(pubKey) → kind:10002, contactsOf(pubKey) → kind:3.
- Class doc on Context spells out the contract; README.md gets a new
"Local event store — the source of truth" section pointing at the
on-disk layout and the design plans.
When wn admin-removes A in interop test 14, A processed the proposal
locally — `tree.removeLeaf(A.leafIndex)` blanked her leaf and shrank
`tree.leafCount` past `myLeafIndex` — and then immediately walked into
BinaryTree.directPath(myLeafIndex, tree.leafCount)
at MlsGroup.processCommitInner. With `myLeafIndex >= leafCount`, the
left-balanced parent walk had no valid stopping point: each recursion
step doubled the candidate parent index, integer-overflowed past 2^31,
and either returned garbage or kept appending to the result list until
the JVM OOM'd. The OOM bubbled up as a `runBlocking` failure that
rolled back the whole commit — so A also stayed locally convinced she
was still a member, and the test timed out waiting for `not_member`.
Three layered fixes so the failure mode can't recur:
* `BinaryTree.parent`/`directPath`/`copath` now `require` an in-range
input. The root and any node ≥ nodeCount used to silently loop or
return garbage; now they throw `IllegalArgumentException` immediately.
This is defense-in-depth — a future caller passing an invalid index
gets a stack trace at the boundary instead of an OOM five frames
deep.
* `MlsGroup.processCommitInner` short-circuits the path-decrypt + epoch
advancement when the proposals in the commit just removed *us*. We
preserve the proposal-side tree mutations so the caller (and the
outer state machine) can observe that we're out, clear pending
proposals + sent keys, and return. Without this short-circuit the
function would derive a bogus all-zero `commit_secret`, fail
`confirmation_tag` verification, throw, and roll the snapshot back —
leaving us still convinced we were a live member.
* `MlsGroupManager.isMember` and a new `MlsGroup.isLocalMember()`
helper now return false once our leaf is null or past `leafCount`.
The post-Remove group entry still lives in `groups` so callers can
inspect the final tree, but every cli command (`group show`,
`message send`, etc.) sees `not_member` and returns the right error.
Also add a comprehensive `BinaryTreeTest` covering non-power-of-2
trees (3, 5, …, 32 leaves) and the boundary cases (root has no
parent, leafIndex ≥ leafCount must throw). The pre-existing tests
only exercised the 4-leaf example from RFC 9420 Appendix C; nothing
hit the `parentInRange` branch, which is exactly where the OOM lived.
Test 14's bash polling needed a small companion fix: it captured
amy's stderr through `2>&1` and fed the result to `jq`, but the
captured stream is interleaved with quartz `Log.d(…)` debug lines,
so the JSON parse always failed. Switch to a `grep` for the
`"error":"not_member"` literal — that signature only appears in
the JSON payload and survives the debug noise. Also tee the
captured output into `$LOG_FILE` so post-mortem logs include each
polling iteration's `[cli] ingest …` traces.
Marmot interop score: 11/16 → 14/16 (tests 9, 15 are unrelated MLS
issues — see follow-up).
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Drives both EventStore (SQLite reference) and FsEventStore with
identical event streams and asserts query() result sets match. SQLite
throws on blocked inserts (NIP-09 tombstones, NIP-62 vanish, NIP-40
expiration) while the FS store silently skips — both are observable
"event not persisted", so insertBoth() catches SQLite throws and
parity is judged on the post-insert state.
Coverage: id lookup, kind+author, since/until/limit, single-letter
tag query (OR within key), replaceable winner, replaceable older
rejected, addressable d-tag dedup, deletion by id (incl. block-
reinsert), deletion by address (incl. cutoff semantics), expiration
sweep, NIP-50 single-token search (sticking to ASCII where SQLite's
unicode61 tokenizer agrees with our port), count, multi-filter
union, delete by filter, mixed kitchen-sink scenario.
113 fs tests now green (97 unit + 16 parity).
- DataDir.eventsDir → "<root>/events-store" (avoids colliding with the
marmot file name conventions).
- Context.store: lazy IEventStore over that directory. Lazy so the
many existing commands that don't touch persistent event state pay
zero open cost (no .lock, no seed file). Closed by Context.close()
only when the lazy delegate has actually been initialised — checked
via reflection so close() never accidentally forces it.
Whitenoise-rs ≥ v0.2.x nests the group object one level deeper —
`{"result": {"group": {...}}}` instead of `{"result": {...}}`. The two
metadata-name pollers in tests 07 and 10 still asked for `.result.name`
and silently saw the empty string, so even when wn-side `groups show`
returned the correct new name the polling loops timed out:
07 metadata rename fail B saw name="" not "Interop-02-renamed"
10 concurrent commits fail diverged: A sees "race-from-amethyst", B sees ""
Both queries now also peel a `.group` wrapper when present, leaving the
older bare-`result` shape working too. Re-running the headless harness
flips 07 + 10 from fail → pass; the remaining failures (09, 14, 15) are
unrelated MLS-encryption / Remove-commit issues that need their own fix.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
- FsLockManager: cross-process exclusive flock(.lock) with per-thread
re-entry. withWriteLock { body } acquires once on a fresh thread
and reuses on nested calls — so transaction { insert(); insert() }
doesn't self-deadlock.
- FsEventStore: insert / delete / delete(filter) / delete(filters) /
delete(id) / deleteExpiredEvents / transaction now run under
withWriteLock. The `*Locked` helpers expose the lock-free body for
re-entrant callers (transaction body, vanish/deletion cascades,
expiration sweep). close() releases the lock channel.
- scrub(): wipes idx/ and rebuilds every entry from the canonical
events. Slots, tombstones and seed are left alone — slots can pin
data the canonical pass doesn't see, and tombstone removal is a
deliberate "un-forget" per the design plan.
- compact(): drops dangling idx/ entries whose canonical no longer
exists. Cheap — only touches idx/, never opens a JSON.
Tests: 11 new in FsMaintenanceTest covering lock file presence,
transaction commit / propagated exception with kept-prior-events
semantics, re-entrant lock from inside a transaction, scrub
rebuilding idx + FTS after a manual wipe, scrub preserving the
replaceable slot, compact dropping dangling and leaving valid alone,
close idempotence + reopen, and two-thread concurrent insert
serialisation. 97 fs tests green.
Adds idx/fts/<token>/<ts>-<id> hardlinks and an FTS-driven query path.
- FsSearchTokenizer: lowercase + Unicode-aware split on non letter-or-
digit, matching SQLite FTS5's unicode61 default closely enough that
the same call indexes content and parses queries (any drift cancels).
Tokens capped at 100 chars to keep filenames under FS limits.
- FsLayout: idxFts + ftsEntry / ftsTokenDir helpers; skeleton dir.
- FsIndexer.pathsFor: when event implements SearchableEvent, emits one
hardlink per unique tokenised word — so insert/delete maintenance
rides the existing link/unlink path. Eviction (replaceable swap),
NIP-09 cascade and NIP-62 vanish all clean up FTS for free.
- FsQueryPlanner: when filter.search is non-blank, drives by FTS.
Tokenises the query, walks each idx/fts/<token>/ listing into a
HashMap<id, ts>, and intersects smallest-first (AND across tokens —
matching SQLite FTS5 default MATCH semantics). Output sorted by
createdAt DESC. Other Filter fields (kinds, authors, tags, since /
until) still apply via Filter.match post-filter.
Tests: 16 new in FsSearchTest covering tokenizer (whitespace, case,
unicode, punctuation, empty), index maintenance (entries created,
non-searchable kinds skipped, delete unlinks), and query semantics
(single token, AND of tokens, ordering, limit, kind/author compose,
no-match, blank string ignored, reopen). 86 fs tests green.
Adds tombstones/vanish/<owner_hex>.json hardlinks (one per owner,
strongest cutoff wins) plus the cascade and block-future-insert
semantics from SQLite's RightToVanishModule.
- FsLayout: vanishTombstonePath helper + tombstones/vanish/ skeleton.
- FsTombstones: vanishCutoff / installVanish / clearVanish — install
uses atomic rename-with-REPLACE_EXISTING and only proceeds when the
new kind-62's createdAt is strictly greater than any existing tomb.
- FsEventStore:
- constructor now takes optional relay: NormalizedRelayUrl? to scope
NIP-62 cascades (matches SQLiteEventStore's relay arg).
- isBlockedByTombstone now also rejects events whose owner has an
active vanish with createdAt >= event.createdAt — owner is the
recipient for GiftWrap, matching pubkey_owner_hash semantics.
- processVanish() runs after canonical write: skips if !shouldVanish-
From(relay), installs the tombstone, then walks idx/owner/<hex>/
and deletes every event with ts < vanish.createdAt. The vanish
event itself survives (its ts equals the cutoff).
Tests: 11 new in FsVanishTest — relay-scoped cascade, different-relay
no-op, vanishFromEverywhere, block re-insert + newer-passes, equal
ts blocked, other authors unaffected, stronger cutoff wins, weaker
ignored, tombstone shares inode with kind-62 canonical, kind-62 stays
queryable. 70 fs tests green.
Adds idx/expires_at/<padded_exp>-<id> hardlinks for events with an
expiration tag, an injectable clock so tests can drive time
deterministically, and the deleteExpiredEvents() sweep.
- FsLayout: idxExpiresAt + expirationEntry path helper.
- FsIndexer.pathsFor: emits the expiration entry whenever
event.expiration() > 0, so insert / delete maintain it alongside
the kind / author / owner / tag indexes.
- FsEventStore: pre-insert guard rejects events with exp <= now
(SQLite parity: trigger uses inclusive <=). Constructor takes a
clock function defaulting to TimeUtils.now(). deleteExpiredEvents
walks idx/expires_at, parses filenames, and deletes anything with
exp < now (strict <, matching SQLite's sweep query).
Tests: 8 new in FsExpirationTest — future expiration accepted +
indexed, already-expired-on-insert rejected, exp==now rejected on
insert but kept by sweep, non-positive exp ignored, sweep removes
canonical + index entries, plain events untouched. 59 fs tests green.
Tombstone files under tombstones/id/<id>.json and tombstones/addr/
<kind>/<pubkey>/<sha256(d)>.json, each a hardlink to the kind-5
event that authored the deletion. One source of truth: the tombstone
IS the deletion event, just indexed by target.
- FsTombstones — installs id tombstones unconditionally, installs
addr tombstones with strongest-cutoff-wins semantics (later kind-5
replaces earlier via atomic rename), and exposes hasIdTombstone /
addrTombstoneCutoff for pre-insert checks.
- FsEventStore.insert — pre-insert guard: id tombstone always blocks;
addr tombstone blocks when event.createdAt <= tomb.createdAt, matching
SQLite's reject_deleted_events trigger. Kind-5 inserts trigger a
cascade: for each e-tag target owned by the deletion author, unlink
indexes + slot + canonical; for each a-tag (same pubkey), evict the
slot winner if its createdAt <= deletion.createdAt. Tombstones are
installed for every target regardless, so future re-inserts are
blocked.
Tests: 11 new in FsDeletionTest — delete-by-id, block-reinsert-by-id,
non-author-deletion still installs tombstone but no cascade, cascade
addressable slot, newer-at-deleted-address passes, older blocked,
equal-timestamp blocked, later kind-5 raises cutoff, earlier kind-5
does not lower, deletion event stays queryable, tombstone shares
inode with kind-5 canonical. 51 fs tests green.
Brings the file-backed store up to parity with SQLite's ReplaceableModule
and AddressableModule. A slot is a single hardlink that encodes the
UNIQUE(kind, pubkey[, d-tag]) constraint directly in the directory
layout:
replaceable/<kind>/<pubkey>.json (kinds 0, 3, 10000-19999)
addressable/<kind>/<pubkey>/<sha256(dTag)>.json (kinds 30000-39999)
FsSlots handles the full lifecycle: pre-insert guard (reject if newer or
equal exists), atomic rename-with-REPLACE_EXISTING install, and eviction
of the old winner's canonical + index hardlinks. Because the slot is a
hardlink the event data survives external canonical deletion, matching
the "files come and go" contract in the design plan.
delete(id) also clears the slot when the deleted event is the current
winner, so no orphan slot files linger.
Tests: 14 new in FsSlotsTest — newer wins / older rejected / equal
rejected / eviction unlinks old indexes / empty d-tag / canonical-
deletion survives via hardlink / delete-clears-slot / non-replaceable
events never touch the slot dirs. 40 fs tests pass.
The confirmValueChange parameter on rememberSwipeToDismissBoxState is
deprecated — state changes should not be vetoed via callback. Move the
onStartToEnd action to SwipeToDismissBox.onDismiss and rely on
enableDismissFromEndToStart = false to restrict the anchor set.
https://claude.ai/code/session_01CfYsUSGeuBnYsCqa6FDSPk
Hardlink indexes under idx/ and a minimal query planner bring the
file-backed store up to parity with SQLite on filtered lookups.
- FsLayout — path helpers, .seed file (8 random bytes, salts all hashes),
entry filename format <zero-padded ts>-<id>.
- FsIndexer — on insert creates hardlinks at idx/kind/<k>/, idx/author/
<pk>/, idx/owner/<owner_hex>/, idx/tag/<name>/<hash_hex>/. Owner hash
matches SQLite's pubkey_owner_hash (recipient for GiftWrap). Honours
DefaultIndexingStrategy: single-letter tag names only. On delete
unlinks every known path so the inode can be reclaimed.
- FsQueryPlanner — picks a driver (ids / first tag / kinds / authors /
all kinds) and yields candidates sorted by createdAt DESC. Final
predicate check runs through Filter.match so any driver is
correctness-safe.
- FsEventStore — sets mtime to event.createdAt on write, runs queries
through the planner with post-filter, applies limit, dedupes by id
across multi-filter unions, and unlinks indexes on delete.
Tests: 16 new in FsQueryTest covering order, limit, author / kind /
tag drivers, tag OR within a key, tagsAll AND across keys, non-
single-letter-tag behaviour, since/until, count, index hardlink
maintenance, and reopen persistence. All 26 fs tests green.
davotoula