Test 12 (offline catch-up) was almost passing: the retained-epoch
fallback (c88923a3) brought through the 5 epoch-1 application messages
the user sent before adding C, but 3 epoch-2 messages + the rename
commit still went missing. Only on app restart did they appear — the
relay re-delivered and the replayed arrival order happened to put the
epoch-2-advancing commit before the epoch-2 events.
Root cause: kind:445 events arrive from the relay in subscription
order, not epoch order. When B drives an offline-catch-up sequence
like
msg-1..5 (epoch 1) → add C (epoch 1→2) → msg-6..8 (epoch 2) →
rename (epoch 2→3)
the receiver can easily see {msg-6, msg-7, msg-8, rename, add-C,
msg-1..5}. Everything from the rename onward arrives before the
add-C commit, so the outer ChaCha20-Poly1305 layer fails with
UndecryptableOuterLayer — we don't have the epoch-2 exporter yet.
Quartz correctly reports the failure, but the receiver just logged
"likely from before our join" and dropped the event on the floor.
Nothing retried when the add-C commit eventually landed and the
epoch caught up.
Fix it at the GroupEventHandler level, where we can both see the
original GroupEvent and hook into every CommitProcessed:
- Keep a bounded per-group ArrayDeque<GroupEvent> of events that
failed with UndecryptableOuterLayer. FIFO-evict at 64 entries so
a genuinely pre-join stream can't pin unbounded memory.
- On every CommitProcessed for a group, drain that group's queue
and re-run each event through the normal add() path. A successful
retry may itself emit another CommitProcessed and trigger recursive
draining — that's fine, epochs are strictly monotonic so recursion
depth is bounded by the queue size.
- Concurrent-safe: all queue mutation is guarded by a Mutex.
Ambient discriminator: we can't tell "epoch too old, from before our
join" from "epoch too new, commit hasn't arrived yet" just from
UndecryptableOuterLayer — both report it. Buffering both is fine:
truly-pre-join events stay buffered until FIFO eviction pushes them
out (no correctness problem, just a few dozen bytes of memory for a
bounded time).
Claude
3a2068a792