feat(nestsClient): listener audio pipeline (Opus decode + AudioTrack)

Phase 3d-1 of the Clubhouse/nests integration. Adds the listener-side audio pipeline that turns a SubscribeHandle's Flow<MoqObject> into audible PCM through Android's MediaCodec + AudioTrack. Encoder / AudioRecord (speaker publishing) lands in Phase 4. commonMain (nestsclient.audio): - `AudioFormat` constants — 48 kHz mono signed-16-bit, 20 ms frames (960 samples), matching the nests Opus profile. - `OpusDecoder` interface — stateful per-track decoder. - `AudioPlayer` interface — start/enqueue/stop, suspend on backpressure. - `AudioRoomPlayer` — wires a Flow<MoqObject> through OpusDecoder into AudioPlayer. Decoder errors are surfaced via an `onError` callback but don't tear down the loop (one bad packet shouldn't kill the room); player errors are fatal. play()/stop() are single-shot per instance, with stop() idempotent and double-release-safe. - `AudioException` with three canonical kinds (DecoderError, DeviceUnavailable, PlaybackFailed). androidMain: - `MediaCodecOpusDecoder` — wraps `MediaCodec("audio/opus")` with the RFC 7845 §5.1 Opus identification header in csd-0 and zeroed pre-skip / seek-pre-roll in csd-1 / csd-2. Drains the output queue per-packet, handles INFO_OUTPUT_FORMAT_CHANGED gracefully. - `AudioTrackPlayer` — wraps `AudioTrack` in MODE_STREAM with USAGE_ VOICE_COMMUNICATION / CONTENT_TYPE_SPEECH so the OS treats audio- room playback like a phone call (call-volume rocker, ducks notifications). Buffer = 4× minimum so the producer can fall behind ~80 ms (roughly the WebTransport datagram jitter on mobile). Tests (commonTest, fake-based): - `AudioRoomPlayerTest` — 7 cases covering happy-path decode + enqueue, decoder failure with continued loop, stop idempotency, double-play rejection, play-after-stop rejection, empty-PCM skip-enqueue, and live channel-backed flow. The real MediaCodec / AudioTrack code is thin glue against Android APIs and is validated manually on device — no Robolectric here, the seams are the Fake* doubles. Next: Phase 3d-2 wires NestsClient end-to-end (HTTP auth → MoqSession listener subscribe → AudioRoomPlayer) plus the Amethyst-side AudioRoomViewModel. After that the only remaining pure-MoQ work is the Kwik handshake (Phase 3b-2). https://claude.ai/code/session_013nVLALALKaHVgHm9u5Cg8D
2026-04-22 07:12:25 +00:00
parent b091b50fe6
commit 6e6fc4cabb
5 changed files with 737 additions and 0 deletions
@@ -0,0 +1,136 @@
+/*
+ * Copyright (c) 2025 Vitor Pamplona
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy of
+ * this software and associated documentation files (the "Software"), to deal in
+ * the Software without restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
+ * Software, and to permit persons to whom the Software is furnished to do so,
+ * subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in all
+ * copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+ * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+ * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
+ * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+ * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+ */
+package com.vitorpamplona.nestsclient.audio
+
+import android.media.AudioAttributes
+import android.media.AudioManager
+import android.media.AudioTrack
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.withContext
+import android.media.AudioFormat as AndroidAudioFormat
+
+/**
+ * [AudioPlayer] backed by Android's [AudioTrack] in `MODE_STREAM`. Targets the
+ * voice-call usage stream so the OS treats audio-room playback like a phone
+ * call (volume rocker controls call volume, ducks notifications, etc.).
+ *
+ * Buffer sizing: 4× minimum so the producer can fall behind by ~80 ms before
+ * dropouts, which roughly matches the jitter the WebTransport datagram path
+ * introduces over typical mobile networks.
+ */
+class AudioTrackPlayer(
+    private val usage: Int = AudioAttributes.USAGE_VOICE_COMMUNICATION,
+    private val contentType: Int = AudioAttributes.CONTENT_TYPE_SPEECH,
+) : AudioPlayer {
+    private var track: AudioTrack? = null
+
+    override fun start() {
+        if (track != null) return
+
+        val channelMask =
+            when (AudioFormat.CHANNELS) {
+                1 -> AndroidAudioFormat.CHANNEL_OUT_MONO
+                2 -> AndroidAudioFormat.CHANNEL_OUT_STEREO
+                else -> error("unsupported channel count ${AudioFormat.CHANNELS}")
+            }
+
+        val minBuffer =
+            AudioTrack.getMinBufferSize(
+                AudioFormat.SAMPLE_RATE_HZ,
+                channelMask,
+                AndroidAudioFormat.ENCODING_PCM_16BIT,
+            )
+        if (minBuffer <= 0) {
+            throw AudioException(
+                AudioException.Kind.DeviceUnavailable,
+                "AudioTrack.getMinBufferSize returned $minBuffer for ${AudioFormat.SAMPLE_RATE_HZ} Hz",
+            )
+        }
+        val bufferBytes = minBuffer * 4
+
+        val newTrack =
+            try {
+                AudioTrack
+                    .Builder()
+                    .setAudioAttributes(
+                        AudioAttributes
+                            .Builder()
+                            .setUsage(usage)
+                            .setContentType(contentType)
+                            .build(),
+                    ).setAudioFormat(
+                        AndroidAudioFormat
+                            .Builder()
+                            .setEncoding(AndroidAudioFormat.ENCODING_PCM_16BIT)
+                            .setSampleRate(AudioFormat.SAMPLE_RATE_HZ)
+                            .setChannelMask(channelMask)
+                            .build(),
+                    ).setBufferSizeInBytes(bufferBytes)
+                    .setTransferMode(AudioTrack.MODE_STREAM)
+                    .build()
+            } catch (t: Throwable) {
+                throw AudioException(
+                    AudioException.Kind.DeviceUnavailable,
+                    "Failed to construct AudioTrack",
+                    t,
+                )
+            }
+
+        try {
+            newTrack.play()
+        } catch (t: Throwable) {
+            runCatching { newTrack.release() }
+            throw AudioException(
+                AudioException.Kind.DeviceUnavailable,
+                "AudioTrack.play() rejected start",
+                t,
+            )
+        }
+        track = newTrack
+    }
+
+    override suspend fun enqueue(pcm: ShortArray) {
+        val t = track ?: throw AudioException(AudioException.Kind.PlaybackFailed, "player not started")
+        // AudioTrack.write blocks if the internal buffer is full. Run on IO so
+        // we don't stall a coroutine dispatcher backed by a small thread pool.
+        withContext(Dispatchers.IO) {
+            val written = t.write(pcm, 0, pcm.size, AudioTrack.WRITE_BLOCKING)
+            if (written < 0) {
+                throw AudioException(
+                    AudioException.Kind.PlaybackFailed,
+                    "AudioTrack.write returned error code $written",
+                )
+            }
+        }
+    }
+
+    override fun stop() {
+        val t = track ?: return
+        track = null
+        runCatching { t.pause() }
+        runCatching { t.flush() }
+        runCatching { t.stop() }
+        runCatching { t.release() }
+    }
+
+    @Suppress("unused")
+    val voiceCallUsage: Int get() = AudioManager.STREAM_VOICE_CALL // kept for documentation
+}
@@ -0,0 +1,164 @@
+/*
+ * Copyright (c) 2025 Vitor Pamplona
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy of
+ * this software and associated documentation files (the "Software"), to deal in
+ * the Software without restriction, including without limitation the rights to use,
+ * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
+ * Software, and to permit persons to whom the Software is furnished to do so,
+ * subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in all
+ * copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+ * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+ * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
+ * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+ * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+ */
+package com.vitorpamplona.nestsclient.audio
+
+import android.media.MediaCodec
+import android.media.MediaFormat
+import java.nio.ByteBuffer
+import java.nio.ByteOrder
+
+/**
+ * [OpusDecoder] backed by Android's [MediaCodec] (`audio/opus`, available on
+ * API 21+). One instance per track — Opus carries forward predictor state
+ * across packets, so sharing a decoder across speakers would cause clicks.
+ *
+ * Configuration:
+ *   - 48 kHz mono signed 16-bit PCM output (matches [AudioFormat]).
+ *   - CSD-0: Opus identification header per RFC 7845 §5.1, 19 bytes.
+ *   - CSD-1 / CSD-2: pre-skip + seek pre-roll, both zero (we don't seek).
+ */
+class MediaCodecOpusDecoder : OpusDecoder {
+    private val codec: MediaCodec =
+        try {
+            MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_AUDIO_OPUS).apply {
+                configure(buildFormat(), null, null, 0)
+                start()
+            }
+        } catch (t: Throwable) {
+            throw AudioException(
+                AudioException.Kind.DeviceUnavailable,
+                "Failed to allocate MediaCodec audio/opus decoder",
+                t,
+            )
+        }
+
+    private val bufferInfo = MediaCodec.BufferInfo()
+    private var presentationTimeUs: Long = 0L
+    private var released = false
+
+    override fun decode(opusPacket: ByteArray): ShortArray {
+        check(!released) { "decoder released" }
+
+        val inputIndex = codec.dequeueInputBuffer(DEQUEUE_TIMEOUT_US)
+        if (inputIndex < 0) return ShortArray(0)
+        val inputBuffer =
+            codec.getInputBuffer(inputIndex)
+                ?: throw AudioException(
+                    AudioException.Kind.DecoderError,
+                    "MediaCodec returned null input buffer at index $inputIndex",
+                )
+        inputBuffer.clear()
+        inputBuffer.put(opusPacket)
+        codec.queueInputBuffer(inputIndex, 0, opusPacket.size, presentationTimeUs, 0)
+        // Advance presentation time by one 20 ms frame.
+        presentationTimeUs += FRAME_DURATION_US
+
+        // Drain whatever output is ready right now. A single Opus packet
+        // typically yields exactly one output buffer, but on some devices the
+        // first call returns INFO_OUTPUT_FORMAT_CHANGED before the PCM frame.
+        val collected = ArrayList<Short>(AudioFormat.FRAME_SIZE_SAMPLES)
+        while (true) {
+            val outputIndex = codec.dequeueOutputBuffer(bufferInfo, DEQUEUE_TIMEOUT_US)
+            when {
+                outputIndex >= 0 -> {
+                    val outputBuffer =
+                        codec.getOutputBuffer(outputIndex)
+                            ?: continue
+                    if (bufferInfo.size > 0) {
+                        outputBuffer.position(bufferInfo.offset)
+                        outputBuffer.limit(bufferInfo.offset + bufferInfo.size)
+                        val shorts = outputBuffer.order(ByteOrder.nativeOrder()).asShortBuffer()
+                        val tmp = ShortArray(shorts.remaining())
+                        shorts.get(tmp)
+                        for (s in tmp) collected.add(s)
+                    }
+                    codec.releaseOutputBuffer(outputIndex, false)
+                    if (bufferInfo.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM != 0) break
+                    // No more buffered output for this packet.
+                    if (bufferInfo.size == 0) break
+                    if (collected.size >= AudioFormat.FRAME_SIZE_SAMPLES) break
+                }
+
+                outputIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> {
+                    // The format change carries no audio data; loop to read
+                    // the actual PCM frame.
+                    continue
+                }
+
+                outputIndex == MediaCodec.INFO_TRY_AGAIN_LATER -> {
+                    break
+                }
+
+                else -> {
+                    break
+                }
+            }
+        }
+        return ShortArray(collected.size) { collected[it] }
+    }
+
+    override fun release() {
+        if (released) return
+        released = true
+        runCatching { codec.stop() }
+        runCatching { codec.release() }
+    }
+
+    companion object {
+        private const val DEQUEUE_TIMEOUT_US = 10_000L // 10 ms
+        private const val FRAME_DURATION_US = 20_000L // 20 ms
+
+        private fun buildFormat(): MediaFormat {
+            val format =
+                MediaFormat.createAudioFormat(
+                    MediaFormat.MIMETYPE_AUDIO_OPUS,
+                    AudioFormat.SAMPLE_RATE_HZ,
+                    AudioFormat.CHANNELS,
+                )
+            format.setByteBuffer("csd-0", ByteBuffer.wrap(buildOpusIdHeader()))
+            // Pre-skip + seek pre-roll: both zero, encoded as little-endian
+            // 64-bit nanoseconds per Android's MediaCodec contract.
+            format.setByteBuffer("csd-1", ByteBuffer.wrap(zeroLongLe()))
+            format.setByteBuffer("csd-2", ByteBuffer.wrap(zeroLongLe()))
+            return format
+        }
+
+        private fun buildOpusIdHeader(): ByteArray {
+            // RFC 7845 §5.1 — 19 bytes for mono, mapping family 0.
+            val buf = ByteBuffer.allocate(19).order(ByteOrder.LITTLE_ENDIAN)
+            buf.put("OpusHead".encodeToByteArray()) // 8 bytes magic
+            buf.put(1.toByte()) // version
+            buf.put(AudioFormat.CHANNELS.toByte()) // channel count
+            buf.putShort(0) // pre-skip
+            buf.putInt(AudioFormat.SAMPLE_RATE_HZ) // input sample rate
+            buf.putShort(0) // output gain (Q7.8 dB)
+            buf.put(0.toByte()) // mapping family 0
+            return buf.array()
+        }
+
+        private fun zeroLongLe(): ByteArray =
+            ByteBuffer
+                .allocate(8)
+                .order(ByteOrder.LITTLE_ENDIAN)
+                .putLong(0L)
+                .array()
+    }
+}