feat(nestsClient): listener audio pipeline (Opus decode + AudioTrack)

Phase 3d-1 of the Clubhouse/nests integration. Adds the listener-side audio pipeline that turns a SubscribeHandle's Flow<MoqObject> into audible PCM through Android's MediaCodec + AudioTrack. Encoder / AudioRecord (speaker publishing) lands in Phase 4. commonMain (nestsclient.audio): - `AudioFormat` constants — 48 kHz mono signed-16-bit, 20 ms frames (960 samples), matching the nests Opus profile. - `OpusDecoder` interface — stateful per-track decoder. - `AudioPlayer` interface — start/enqueue/stop, suspend on backpressure. - `AudioRoomPlayer` — wires a Flow<MoqObject> through OpusDecoder into AudioPlayer. Decoder errors are surfaced via an `onError` callback but don't tear down the loop (one bad packet shouldn't kill the room); player errors are fatal. play()/stop() are single-shot per instance, with stop() idempotent and double-release-safe. - `AudioException` with three canonical kinds (DecoderError, DeviceUnavailable, PlaybackFailed). androidMain: - `MediaCodecOpusDecoder` — wraps `MediaCodec("audio/opus")` with the RFC 7845 §5.1 Opus identification header in csd-0 and zeroed pre-skip / seek-pre-roll in csd-1 / csd-2. Drains the output queue per-packet, handles INFO_OUTPUT_FORMAT_CHANGED gracefully. - `AudioTrackPlayer` — wraps `AudioTrack` in MODE_STREAM with USAGE_ VOICE_COMMUNICATION / CONTENT_TYPE_SPEECH so the OS treats audio- room playback like a phone call (call-volume rocker, ducks notifications). Buffer = 4× minimum so the producer can fall behind ~80 ms (roughly the WebTransport datagram jitter on mobile). Tests (commonTest, fake-based): - `AudioRoomPlayerTest` — 7 cases covering happy-path decode + enqueue, decoder failure with continued loop, stop idempotency, double-play rejection, play-after-stop rejection, empty-PCM skip-enqueue, and live channel-backed flow. The real MediaCodec / AudioTrack code is thin glue against Android APIs and is validated manually on device — no Robolectric here, the seams are the Fake* doubles. Next: Phase 3d-2 wires NestsClient end-to-end (HTTP auth → MoqSession listener subscribe → AudioRoomPlayer) plus the Amethyst-side AudioRoomViewModel. After that the only remaining pure-MoQ work is the Kwik handshake (Phase 3b-2). https://claude.ai/code/session_013nVLALALKaHVgHm9u5Cg8D
2026-04-22 07:12:25 +00:00
parent b091b50fe6
commit 6e6fc4cabb
5 changed files with 737 additions and 0 deletions
@@ -0,0 +1,100 @@
+/*
+ * Copyright (c) 2025 Vitor Pamplona
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy of
+ * this software and associated documentation files (the "Software"), to deal in
+ * the Software without restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
+ * Software, and to permit persons to whom the Software is furnished to do so,
+ * subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in all
+ * copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+ * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+ * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
+ * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+ * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+ */
+package com.vitorpamplona.nestsclient.audio
+
+/**
+ * PCM audio format the audio pipeline produces and consumes.
+ *
+ * Listener-only flow runs at 48 kHz mono signed-16-bit, matching the nests
+ * Opus profile (RFC 6716 wideband at the codec's native rate). The whole
+ * pipeline is hardcoded to this format for now — when nests starts varying
+ * codec settings, this becomes a per-room negotiated value out of the
+ * `/api/v1/nests/<room>` response.
+ */
+object AudioFormat {
+    const val SAMPLE_RATE_HZ: Int = 48_000
+    const val CHANNELS: Int = 1
+
+    /** 20 ms at 48 kHz. */
+    const val FRAME_SIZE_SAMPLES: Int = 960
+
+    /** Bytes per PCM 16-bit sample. */
+    const val BYTES_PER_SAMPLE: Int = 2
+}
+
+/**
+ * Decoder for one Opus frame at a time.
+ *
+ * Implementations are stateful — Opus carries forward predictor state across
+ * frames — so a single decoder must be used by a single track for the
+ * lifetime of that subscription. Call [release] when the track ends.
+ */
+interface OpusDecoder {
+    /**
+     * Decode one Opus packet (the bytes of an OBJECT_DATAGRAM payload from
+     * MoQ) into PCM 16-bit signed mono samples. Returns an empty array if
+     * the decoder needs more input before producing output (some codec
+     * pipelines have a ramp-up frame), but never throws on a well-formed
+     * packet.
+     */
+    fun decode(opusPacket: ByteArray): ShortArray
+
+    fun release()
+}
+
+/**
+ * Sink for PCM audio playback. Implementations buffer internally — [enqueue]
+ * may suspend if the device's playback buffer is full.
+ */
+interface AudioPlayer {
+    /** Allocate underlying audio resources and begin playback. */
+    fun start()
+
+    /**
+     * Feed one PCM frame (any length, but typically [AudioFormat.FRAME_SIZE_SAMPLES]
+     * samples) into the playback queue.
+     */
+    suspend fun enqueue(pcm: ShortArray)
+
+    /** Stop playback and release resources. After this, the player is unusable. */
+    fun stop()
+}
+
+/**
+ * Audio-pipeline exception type that lets UI code distinguish between
+ * recoverable codec/IO errors and fatal device-resource failures.
+ */
+class AudioException(
+    val kind: Kind,
+    message: String,
+    cause: Throwable? = null,
+) : RuntimeException(message, cause) {
+    enum class Kind {
+        /** Decoder rejected an Opus packet (corrupted bytes, unsupported config). */
+        DecoderError,
+
+        /** Audio device resource (AudioTrack/AudioRecord) couldn't be allocated. */
+        DeviceUnavailable,
+
+        /** Underlying audio device threw mid-playback. */
+        PlaybackFailed,
+    }
+}
@@ -0,0 +1,112 @@
+/*
+ * Copyright (c) 2025 Vitor Pamplona
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy of
+ * this software and associated documentation files (the "Software"), to deal in
+ * the Software without restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
+ * Software, and to permit persons to whom the Software is furnished to do so,
+ * subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in all
+ * copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+ * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+ * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
+ * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+ * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+ */
+package com.vitorpamplona.nestsclient.audio
+
+import com.vitorpamplona.nestsclient.moq.MoqObject
+import kotlinx.coroutines.CancellationException
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.Job
+import kotlinx.coroutines.flow.Flow
+import kotlinx.coroutines.launch
+
+/**
+ * Bridges a track's `Flow<MoqObject>` (from [com.vitorpamplona.nestsclient.moq.SubscribeHandle.objects])
+ * through an [OpusDecoder] into an [AudioPlayer].
+ *
+ * Single-track. To play multiple speakers in a room, instantiate one
+ * [AudioRoomPlayer] per [com.vitorpamplona.nestsclient.moq.SubscribeHandle];
+ * each owns its own decoder (Opus state is per-track).
+ *
+ * Lifecycle:
+ *   - [play] starts the player and the decode loop. Returns immediately.
+ *   - [stop] cancels the decode loop, stops the player, and releases the
+ *     decoder. Idempotent.
+ */
+class AudioRoomPlayer(
+    private val decoder: OpusDecoder,
+    private val player: AudioPlayer,
+    private val scope: CoroutineScope,
+) {
+    private var job: Job? = null
+    private var stopped = false
+
+    /**
+     * Start consuming [objects] in the background. Each MoQ object's payload
+     * is fed to the Opus decoder; the resulting PCM frame is enqueued to the
+     * player.
+     *
+     * Decoder errors are reported via [onError] but do NOT stop the loop —
+     * one bad packet shouldn't tear down the room. Player errors are fatal.
+     */
+    fun play(
+        objects: Flow<MoqObject>,
+        onError: (AudioException) -> Unit = { /* swallow */ },
+    ) {
+        check(!stopped) { "AudioRoomPlayer already stopped" }
+        check(job == null) { "AudioRoomPlayer.play already called" }
+
+        player.start()
+        job =
+            scope.launch {
+                try {
+                    objects.collect { obj ->
+                        val pcm =
+                            try {
+                                decoder.decode(obj.payload)
+                            } catch (ce: CancellationException) {
+                                throw ce
+                            } catch (t: Throwable) {
+                                onError(
+                                    AudioException(
+                                        AudioException.Kind.DecoderError,
+                                        "Opus decode failed for object ${obj.objectId}",
+                                        t,
+                                    ),
+                                )
+                                return@collect
+                            }
+                        if (pcm.isNotEmpty()) {
+                            player.enqueue(pcm)
+                        }
+                    }
+                } catch (ce: CancellationException) {
+                    throw ce
+                } catch (t: Throwable) {
+                    onError(
+                        AudioException(
+                            AudioException.Kind.PlaybackFailed,
+                            "audio pipeline failed",
+                            t,
+                        ),
+                    )
+                }
+            }
+    }
+
+    /** Stop playback, cancel the decode loop, release the decoder. Idempotent. */
+    fun stop() {
+        if (stopped) return
+        stopped = true
+        job?.cancel()
+        runCatching { player.stop() }
+        runCatching { decoder.release() }
+    }
+}