docs(quic): record retransmit subsystem implementation status

Update the two affected plan docs now that RFC 9002 retransmit is shipped on this branch. quic/plans/2026-05-04-control-frame-retransmit.md: - Mark plan as shipped 2026-05-05; list the 15 commits that landed it (plan + 9 steps + 5 follow-ups + perf + audit). - Document what changed vs the original scope: the deferred follow-ups (STREAM data, CRYPTO, RESET_STREAM/STOP_SENDING/NEW_CONNECTION_ID) all shipped on top of the receive-flow-control core. - Note the binary-search SendBuffer perf optimisation and the audit-driven first-call-wins fix. - Update the cap-workaround status: initialMaxStreamsUni is back to 10_000 (not the 1_000_000 mentioned in the original Why). quic/plans/2026-04-26-quic-stack-status.md: - Phase F downgraded from "partial" to "done (no CC)" — loss detection, RTT estimator, PTO, and per-frame retransmit shipped. - Removed "no STREAM retransmit" / "SendBuffer doesn't retain bytes until ACK" from the deliberately-don't-do list (now do). - Added congestion-control as the new deliberately-don't-do entry. - Crossed out the corresponding deferred-work items; added congestion control as deferred item #8. - Listed the new recovery test files in the test inventory. - Linked to the retransmit plan + implementation log. https://claude.ai/code/session_01PYYez8a6sjiakyjAxsfCEQ
2026-05-05 01:15:29 +00:00
parent 086a9c75dc
commit 70af1953dc
2 changed files with 141 additions and 12 deletions
@@ -21,7 +21,7 @@ tests, 39 test files, five rounds of parallel audit + fix passes.
 | C. Initial + Handshake packets | 2 wk | done | RFC 9001 §A.2/§A.3 vectors pass; ChaCha20 per §A.5 |
 | D. 1-RTT + STREAM | 1 wk | done | Stream offset reassembly, FIN, fuzzed |
 | E. ACK + flow control | 1 wk | done | MAX_DATA / MAX_STREAM_DATA / MAX_STREAMS routing + writer enforcement |
-| F. Loss recovery + congestion control | 1 wk | partial | PTO timer for handshake retries; **no retransmit-on-loss in steady state** (out of scope — see "deferred" below) |
+| F. Loss recovery + congestion control | 1 wk | done (no CC) | RFC 9002 §5/§6 RTT estimator + packet/time-threshold loss detection + PTO + per-frame retransmit shipped 2026-05-05; congestion control still TBD |
 | G. Datagram extension | ½ wk | done | RFC 9221 frames + bounded incoming queue |
 | H. Connection lifecycle | 1 wk | done | CONNECTION_CLOSE, idle timeout, draining/closing, idempotent driver close |
 | I. HTTP/3 | 2 wk | done | Control stream + SETTINGS + GOAWAY (with id-regression check) + duplicate-id rejection |
@@ -61,6 +61,9 @@ quic/
    │   ├── packet/                       ← LongHeaderPacket, ShortHeaderPacket, RetryPacket, peekHeader
    │   ├── qpack/                        ← QpackDecoder, QpackEncoder, QpackHuffman, QpackInteger,
    │   │                                   QpackStaticTable
+    │   ├── connection/recovery/         ← RecoveryToken + SentPacket + QuicLossDetection
+    │   │                                   (RFC 9002 §5/§6 RTT estimator + loss detection +
+    │   │                                   PTO; per-frame retransmit dispatch)
    │   ├── recovery/                     ← AckTracker (with ack-eliciting gating)
    │   ├── stream/                       ← QuicStream, ReceiveBuffer (with FIN-fully-read), SendBuffer, StreamId
    │   ├── tls/                          ← TlsClient state machine + ClientHello/ServerHello/EE/Cert/CV/Finished
@@ -112,14 +115,12 @@ explicit `PermissiveCertificateValidator`; production passes
 - **QPACK dynamic-table inserts on the encoder.** We send literal-only;
  decoder accepts dynamic-table indexed lines.
 - **ECN / anti-amplification limits.** We're a client.
- **Retransmit-on-loss in steady state.** `SendBuffer.takeChunk` releases
-  bytes to the wire and doesn't retain them. The handshake survives via the
-  `Driver.sendLoop` PTO path which re-pulls from CRYPTO send buffers; for
-  STREAM data, a real loss event truncates the stream silently. This is
-  acceptable for MoQ (DATAGRAM-mode audio, plus stream usage is
-  control-plane only) but would be the first item to add for general use.
 - **TLS Key-Update / NewSessionTicket.** Detected and refused (KeyUpdate
  fails the connection rather than silently desynchronising).
+- **Congestion control (NewReno / CUBIC / BBR).** Loss detection is in
+  place (RFC 9002 §5/§6) but no rate-limiting feedback loop reacts to
+  losses; we send as fast as the application provides bytes. Independent
+  follow-up.

 ## Verified interop

@@ -164,7 +165,13 @@ Roughly grouped:
  `TlsTranscriptHashTest`.
 - **WT / HTTP/3:** `CapsuleReaderTest`, `WtPeerStreamDemuxTest`,
  `WtFramingTest`.
- **Recovery:** `AckTrackerCoalescedTest`, `AckTrackerGatingTest`.
+- **Recovery:** `AckTrackerCoalescedTest`, `AckTrackerGatingTest`,
+  `RecoveryTokenTest`, `SentPacketTest`, `ReceiverFlowControlTest` (9
+  cases mirrored from neqo `fc.rs`), `QuicLossDetectionTest`,
+  `PtoTest`, `QuicConnectionRetransmitTest`,
+  `SendBufferRetainUntilAckTest` (14 cases for the rewrite),
+  `StreamRetransmitTest`, `CryptoRetransmitTest`,
+  `ResetStopSendingEmitTest` (7 cases).
 - **Interop:** `InteropRunner` (jvmTest, drives a real socket against a
  Dockerised aioquic; opt-in, not in CI).

@@ -174,9 +181,13 @@ These are the items future audit rounds keep flagging that we've
 consciously not tackled — all confined to the steady-state path that audio
 rooms don't exercise heavily:

-1. **No STREAM retransmit on loss** (audit-4 #10). Acceptable for MoQ
-   datagram audio; would block any heavy stream-based use. ~1 wk to add.
-2. **`SendBuffer` doesn't retain bytes until ACK.** Same scope as #1.
+1. ~~**No STREAM retransmit on loss** (audit-4 #10).~~ **Resolved 2026-05-05** —
+   `SendBuffer` rewritten for retain-until-ACK with three-state range
+   tracking; ACK / loss dispatchers re-queue lost ranges to the
+   retransmit FIFO. Also covers CRYPTO retransmit per encryption level
+   and RESET_STREAM / STOP_SENDING / NEW_CONNECTION_ID retransmit.
+   See [`2026-05-04-control-frame-retransmit.md`](2026-05-04-control-frame-retransmit.md).
+2. ~~**`SendBuffer` doesn't retain bytes until ACK.**~~ Resolved with #1.
 3. **No Initial / Handshake key discard.** RFC 9000 §17.2.2 / RFC 9001 §4.9
   require dropping these after handshake completes; we hold them
   indefinitely. Memory leak per long session.
@@ -191,11 +202,14 @@ rooms don't exercise heavily:
   class` into an interface so the test side can stub. The driver is
   covered indirectly by every pipe-based test plus the live interop
   runner.
+8. **No congestion control.** Loss detection is wired but nothing
+   throttles send rate in response. Independent ~1-2 wk project.

 ## Pointers

 - Original (frozen) plan: `docs/plans/2026-04-22-pure-kotlin-quic-webtransport-plan.md`
 - Audio-rooms NIP draft: `docs/plans/2026-04-22-nip-audio-rooms-draft.md`
 - Completion plan: `nestsClient/plans/2026-04-26-audio-rooms-completion.md`
+- Retransmit plan + implementation log: [`2026-05-04-control-frame-retransmit.md`](2026-05-04-control-frame-retransmit.md)
 - Live interop runner: `quic/src/jvmTest/.../interop/InteropRunner.kt`
 - Audit history: `git log --grep='audit' -- quic/`
@@ -1,6 +1,10 @@
 # Control-frame retransmit for `:quic` — implementation plan

-**Status:** plan, not started.
+**Status:** **shipped 2026-05-05** on branch `claude/fix-nest-audio-display-3chAG`.
+Steps 1–9 landed as planned; the deferred follow-ups (STREAM data, CRYPTO,
+RESET_STREAM / STOP_SENDING / NEW_CONNECTION_ID) all shipped on top, plus an
+audit cleanup pass. See [Implementation log](#implementation-log) at the end
+for the full commit list.

 ## Why

@@ -351,3 +355,114 @@ work for at least a week without regressions.
 - A new `MoqLiteSessionRetransmitTest` simulates packet loss at the moment of `MAX_STREAMS_UNI` emission and confirms audio continues flowing past the threshold.
 - Existing `QuicConnectionWriterTest`, `PeerStreamCreditExtensionTest`, etc. unchanged.
 - No regression in handshake latency or throughput under steady-state.
+
+## Implementation log
+
+Shipped over 13 commits on `claude/fix-nest-audio-display-3chAG`:
+
+| # | Commit | Subject |
+|---|---|---|
+| plan | `c246305` | `docs(quic): plan control-frame retransmit subsystem mirroring neqo` |
+| 1 | `9e6fa3d` | `feat(quic): step 1 of RFC 9002 retransmit — RecoveryToken + SentPacket types` |
+| 2 | `ea15a9a` | `feat(quic): step 2 of RFC 9002 retransmit — record SentPacket per outbound` |
+| 3 | `0ced269` | `feat(quic): step 3 of RFC 9002 retransmit — drain SentPacket on ACK` |
+| 4 | `2928263` | `feat(quic): step 4 of RFC 9002 retransmit — pending* fields + writer drain` |
+| 5 | `1df6441` | `feat(quic): step 5 of RFC 9002 retransmit — loss detection + RTT estimator` |
+| 6 | `15a6bfc` | `feat(quic): step 6 of RFC 9002 retransmit — dispatch lost tokens to pending*` |
+| 7–9 | `c43c951` | `feat(quic): steps 7, 8, 9 of RFC 9002 retransmit — PTO + integration test + revert workaround` |
+| follow-up A | `7f6d908` | `feat(quic): extend RecoveryToken — Stream, Crypto, ResetStream, StopSending, NewConnectionId` |
+| follow-up B | `03cfb31` | `feat(quic): rewrite SendBuffer for retain-until-ACK with markAcked/markLost` |
+| follow-up C | `f623e88` | `feat(quic): wire STREAM data retransmit — token emission + ACK/loss dispatch` |
+| follow-up D | `0c847b4` | `feat(quic): wire CRYPTO retransmit per encryption level` |
+| follow-up E | `996ab39` | `feat(quic): emit RESET_STREAM / STOP_SENDING + per-stream retransmit dispatch` |
+| perf | `303caa8` | `perf(quic): binary-search SendBuffer overlap + insert (O(log N))` |
+| audit | `086a9c7` | `fix(quic): RESET_STREAM/STOP_SENDING first-call-wins + threading contract` |
+
+### What changed vs the plan
+
+The original scope was **only** the receive-side flow-control frames
+(`MAX_STREAMS_UNI/BIDI`, `MAX_DATA`, `MAX_STREAM_DATA`). Once the
+RecoveryToken / SentPacket / loss-detection scaffolding existed, the
+remaining retransmittable frames were a small extension:
+
+- **STREAM data retransmit (B + C).** Required rewriting `SendBuffer`
+  from "release on send" to "retain until ACK", with three logical
+  regions (`in-flight` / `needs retransmit` / `unsent`) tracked as
+  sorted offset ranges. Bytes are released on `markAcked`; lost ranges
+  re-prioritise to the front of `takeChunk` via a FIFO retransmit queue.
+  Removes the "STREAM truncates silently on loss" item from the
+  deferred-work list.
+- **CRYPTO retransmit (D).** Same `SendBuffer` machinery applied
+  per-encryption-level (Initial / Handshake / Application). Closes the
+  handshake reliability gap that previously relied on the driver's PTO
+  re-pull-from-CRYPTO hack.
+- **RESET_STREAM / STOP_SENDING / NEW_CONNECTION_ID emit + retransmit (E).**
+  Public API on `QuicStream` (`resetStream(errorCode)` /
+  `stopSending(errorCode)`); writer drain emits with a `RecoveryToken`;
+  loss dispatcher re-flags per-stream emit-pending bits; ACK dispatcher
+  latches `resetAcked` / `stopSendingAcked` so stale loss tokens don't
+  re-emit. NEW_CONNECTION_ID retransmit drains
+  `QuicConnection.pendingNewConnectionId` (no public emit API yet —
+  `:quic` doesn't rotate connection IDs — but the wiring is in place).
+
+### Performance optimisation
+
+`303caa8` replaced the O(N) full-scan in `SendBuffer.removeOverlap` and
+the O(N) middle-insert in `addToInFlight` with a binary-search-based
+`firstOverlapIndex` + early-exit walk. Hot-path ACK / loss notification
+is now O(log N + k) where k is the number of in-flight ranges actually
+overlapping the ACK range (typically 1).
+
+### Audit follow-up
+
+`086a9c7` cleaned up correctness + threading issues found by re-reading
+the emit commit:
+
+1. `resetStream` / `stopSending` now no-op on the second call. RFC 9000
+   §3.5 pins `finalSize` at first emission; the original "idempotent —
+   second call overwrites with newer error code" claim was wrong (a
+   retransmit after additional `enqueue` would replay with a larger
+   `finalSize`, triggering `FINAL_SIZE_ERROR` on the peer). Two new
+   tests — `resetStream_secondCallIsNoOp_finalSizeFrozen`,
+   `stopSending_secondCallIsNoOp` — lock the contract.
+2. `resetEmitPending`, `resetAcked`, `stopSendingEmitPending`,
+   `stopSendingAcked` are now `@Volatile`. The public emit APIs are
+   callable from any coroutine while the writer / dispatchers read the
+   same fields under `QuicConnection.lock`; volatile gives the
+   cross-thread happens-before, and the first-call-wins gate above
+   eliminates the only multi-writer race.
+3. Stale `SendBuffer` class KDoc claiming O(N) range arithmetic
+   refreshed to reflect the actual O(log N + k) cost.
+4. `removeOverlap`'s bulk-removal comment toned down — it had claimed
+   O(k) per call but `ArrayDeque.removeAt(i)` shifts on every call;
+   actual cost is O(k · (size − end + k)) worst case, fine in practice
+   because k is 1–2 in steady state.
+
+### Test coverage shipped
+
+The 50 planned tests landed plus the follow-up suites:
+
+- `RecoveryTokenTest`, `SentPacketTest` (codec + equality).
+- `ReceiverFlowControlTest` (9 mirrored from neqo's `fc.rs`).
+- `QuicLossDetectionTest` (~15 mirrored from `recovery/mod.rs`).
+- `PtoTest` (~7 mirrored from `recovery/mod.rs` PTO subset).
+- `QuicConnectionRetransmitTest` (integration: lost MAX_STREAMS bump
+  re-emits and lands).
+- `MoqLiteSessionRetransmitTest` (drops a packet at the
+  half-window-threshold MAX_STREAMS_UNI emit; audio keeps flowing).
+- `SendBufferRetainUntilAckTest` (14 cases for the retain-until-ACK
+  rewrite — ack/loss/split/FIN/compaction).
+- `StreamRetransmitTest`, `CryptoRetransmitTest` (token emission + loss
+  re-queues bytes).
+- `ResetStopSendingEmitTest` (7 cases: emit-and-token, retransmit on
+  loss, ack-then-stale-loss-drop, stop-sending emission,
+  NEW_CONNECTION_ID retransmit drain, first-call-wins for both APIs).
+
+### Cap-workaround status
+
+Step 9 of the original plan ("revert `initialMaxStreamsUni` from
+1 000 000 back to a smaller value once retransmit is durable") landed
+in `c43c951`. `QuicConnectionConfig.initialMaxStreamsUni` is now
+`10_000L` — large enough to avoid the moq-rs cliff at startup but
+small enough that the rolling-extension + retransmit path actually
+runs in long sessions. The 1 000 000 emergency value is gone.