nick/amethyst
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(nestsClient): MoQ ANNOUNCE family + Opus encoder + AudioRecord capture

Browse Source 
Phase 4 (codec + audio capture slice). Adds the publisher-side MoQ
control messages and the Opus encode + microphone capture pieces a
speaker needs. The host-grants-speaker UI flow is deferred — that's
multi-screen UX that should be designed before being implemented.

commonMain (nestsclient.moq):
- 5 new MoqMessageType entries with codec round-trip + tests:
  * Announce (0x06): publisher offers a track namespace.
  * AnnounceOk (0x07): subscriber acknowledges.
  * AnnounceError (0x08): subscriber rejects with error code +
    reason phrase.
  * Unannounce (0x09): publisher withdraws a previously-announced
    namespace.
  * SubscribeDone (0x0B): publisher tells the subscriber no more
    objects are coming for this subscription, with stream count and
    reason.

commonMain (nestsclient.audio):
- New `OpusEncoder` interface — symmetric to `OpusDecoder`, one
  instance per outgoing track since Opus state is per-stream.
- New `AudioCapture` interface — `start()`, `readFrame()` returns a
  PCM frame or null when stopped, `stop()` releases the mic.

androidMain (nestsclient.audio):
- `MediaCodecOpusEncoder` — wraps `MediaCodec("audio/opus")` encoder
  variant (API 29+). 48 kHz mono in, 32 kbit/s VBR Opus out, 20 ms
  frames. Drains output queue per encode call.
- `AudioRecordCapture` — wraps `AudioRecord` from
  `MediaRecorder.AudioSource.VOICE_COMMUNICATION` so the platform's
  echo-cancellation + noise-suppression filters apply when available.
  Reads exactly one PCM frame per readFrame() call, retries on
  underrun, throws AudioException(DeviceUnavailable) on permission/
  resource failures.

commonTest:
- `AnnounceCodecTest` — 7 cases covering each message round-trip,
  concatenated decode in sequence, and a guard against accidentally
  reordering MoqMessageType enum codes.

Permission already declared:
- `RECORD_AUDIO` is already in amethyst/AndroidManifest.xml — no
  manifest change needed.

What this does NOT include:
- A publisher loop class (analogous to AudioRoomPlayer for publish
  direction) — can be added when the speak-button wiring lands.
- The host-grants-speaker UI — needs design input.
- STREAM_HEADER_SUBGROUP for stream-based object delivery — datagrams
  cover the listener happy-path; streams add reliability that nests
  may or may not require.

https://claude.ai/code/session_013nVLALALKaHVgHm9u5Cg8D
This commit is contained in:
Claude
2026-04-22 07:33:55 +00:00
parent 64b3367472
commit b62e3dd0ec
6 changed files with 535 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nestsClient/src/androidMain/kotlin/com/vitorpamplona/nestsclient/audio/AudioRecordCapture.kt
+136
View File
@@ -0,0 +1,136 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.nestsclient.audio
import android.media.AudioRecord
import android.media.MediaRecorder
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import android.media.AudioFormat as AndroidAudioFormat
/**
* [AudioCapture] backed by Android's [AudioRecord] from the
* VOICE_COMMUNICATION input source — same source LiveKit, WebRTC, and most
* voice-chat libraries use, so it gets the platform's echo-cancellation and
* noise-suppression filters when available.
*
* **Permission:** the caller is responsible for holding `RECORD_AUDIO` before
* calling [start]; this class will throw [AudioException.Kind.DeviceUnavailable]
* if the OS denies the resource.
*/
class AudioRecordCapture(
private val source: Int = MediaRecorder.AudioSource.VOICE_COMMUNICATION,
) : AudioCapture {
private var record: AudioRecord? = null
private var stopped = false
override fun start() {
check(!stopped) { "capture already stopped" }
if (record != null) return
val channelMask =
when (AudioFormat.CHANNELS) {
1 -> AndroidAudioFormat.CHANNEL_IN_MONO
2 -> AndroidAudioFormat.CHANNEL_IN_STEREO
else -> error("unsupported channel count ${AudioFormat.CHANNELS}")
}
val minBuffer =
AudioRecord.getMinBufferSize(
AudioFormat.SAMPLE_RATE_HZ,
channelMask,
AndroidAudioFormat.ENCODING_PCM_16BIT,
)
if (minBuffer <= 0) {
throw AudioException(
AudioException.Kind.DeviceUnavailable,
"AudioRecord.getMinBufferSize returned $minBuffer for ${AudioFormat.SAMPLE_RATE_HZ} Hz",
)
}
val bufferBytes =
maxOf(minBuffer, AudioFormat.FRAME_SIZE_SAMPLES * AudioFormat.BYTES_PER_SAMPLE * 4)
val rec =
try {
@Suppress("MissingPermission")
AudioRecord(source, AudioFormat.SAMPLE_RATE_HZ, channelMask, AndroidAudioFormat.ENCODING_PCM_16BIT, bufferBytes)
} catch (t: Throwable) {
throw AudioException(
AudioException.Kind.DeviceUnavailable,
"Failed to construct AudioRecord (RECORD_AUDIO permission?)",
t,
)
}
if (rec.state != AudioRecord.STATE_INITIALIZED) {
runCatching { rec.release() }
throw AudioException(
AudioException.Kind.DeviceUnavailable,
"AudioRecord state=${rec.state} after construction (expected INITIALIZED)",
)
}
try {
rec.startRecording()
} catch (t: Throwable) {
runCatching { rec.release() }
throw AudioException(
AudioException.Kind.DeviceUnavailable,
"AudioRecord.startRecording() failed",
t,
)
}
record = rec
}
override suspend fun readFrame(): ShortArray? {
val rec = record ?: return null
val frame = ShortArray(AudioFormat.FRAME_SIZE_SAMPLES)
return withContext(Dispatchers.IO) {
var read = 0
while (read < frame.size) {
if (stopped) return@withContext null
val n = rec.read(frame, read, frame.size - read)
if (n < 0) {
throw AudioException(
AudioException.Kind.PlaybackFailed,
"AudioRecord.read returned error code $n",
)
}
if (n == 0) {
// Underrun — wait briefly and retry. This avoids busy-waiting
// on a slow producer.
kotlinx.coroutines.delay(2)
continue
}
read += n
}
frame
}
}
override fun stop() {
if (stopped) return
stopped = true
val rec = record ?: return
record = null
runCatching { rec.stop() }
runCatching { rec.release() }
}
}
nestsClient/src/androidMain/kotlin/com/vitorpamplona/nestsclient/audio/MediaCodecOpusEncoder.kt
+139
View File
@@ -0,0 +1,139 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.nestsclient.audio
import android.media.MediaCodec
import android.media.MediaCodecInfo
import android.media.MediaFormat
import java.nio.ByteOrder
/**
* [OpusEncoder] backed by Android's [MediaCodec] (`audio/opus`, encoder
* variant available on API 29+ — the decoder works back to API 21 but the
* encoder shipped later). One instance per outgoing track.
*
* Configuration:
* - 48 kHz mono input PCM 16-bit (matches [AudioFormat]).
* - Target bitrate ~32 kbit/s VBR — high-quality wideband speech.
* - 20 ms frames (the encoder requires the input buffer to hold one frame
* at a time for low latency).
*/
class MediaCodecOpusEncoder(
private val targetBitrate: Int = DEFAULT_BITRATE_BPS,
) : OpusEncoder {
private val codec: MediaCodec =
try {
MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_AUDIO_OPUS).apply {
configure(buildFormat(targetBitrate), null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
start()
}
} catch (t: Throwable) {
throw AudioException(
AudioException.Kind.DeviceUnavailable,
"Failed to allocate MediaCodec audio/opus encoder",
t,
)
}
private val bufferInfo = MediaCodec.BufferInfo()
private var presentationTimeUs: Long = 0L
private var released = false
override fun encode(pcm: ShortArray): ByteArray {
check(!released) { "encoder released" }
require(pcm.isNotEmpty()) { "PCM frame must not be empty" }
val inputIndex = codec.dequeueInputBuffer(DEQUEUE_TIMEOUT_US)
if (inputIndex < 0) return ByteArray(0)
val inputBuffer =
codec.getInputBuffer(inputIndex)
?: throw AudioException(
AudioException.Kind.DecoderError,
"MediaCodec returned null input buffer at index $inputIndex",
)
inputBuffer.clear()
inputBuffer.order(ByteOrder.nativeOrder()).asShortBuffer().put(pcm)
val byteCount = pcm.size * AudioFormat.BYTES_PER_SAMPLE
codec.queueInputBuffer(inputIndex, 0, byteCount, presentationTimeUs, 0)
presentationTimeUs += FRAME_DURATION_US
// One PCM frame produces one Opus packet (sometimes after one warmup
// round). Drain the output queue once.
while (true) {
val outputIndex = codec.dequeueOutputBuffer(bufferInfo, DEQUEUE_TIMEOUT_US)
when {
outputIndex >= 0 -> {
val outputBuffer = codec.getOutputBuffer(outputIndex) ?: continue
val opus = ByteArray(bufferInfo.size)
outputBuffer.position(bufferInfo.offset)
outputBuffer.limit(bufferInfo.offset + bufferInfo.size)
outputBuffer.get(opus)
codec.releaseOutputBuffer(outputIndex, false)
if (opus.isNotEmpty()) return opus
}
outputIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> {
continue
}
outputIndex == MediaCodec.INFO_TRY_AGAIN_LATER -> {
return ByteArray(0)
}
else -> {
return ByteArray(0)
}
}
}
@Suppress("UNREACHABLE_CODE")
return ByteArray(0)
}
override fun release() {
if (released) return
released = true
runCatching { codec.stop() }
runCatching { codec.release() }
}
companion object {
const val DEFAULT_BITRATE_BPS: Int = 32_000
private const val DEQUEUE_TIMEOUT_US = 10_000L
private const val FRAME_DURATION_US = 20_000L
private fun buildFormat(bitrate: Int): MediaFormat =
MediaFormat
.createAudioFormat(
MediaFormat.MIMETYPE_AUDIO_OPUS,
AudioFormat.SAMPLE_RATE_HZ,
AudioFormat.CHANNELS,
).apply {
setInteger(MediaFormat.KEY_BIT_RATE, bitrate)
setInteger(MediaFormat.KEY_PCM_ENCODING, android.media.AudioFormat.ENCODING_PCM_16BIT)
// Encoder-side AAC/Opus profile selection: SignalingDelaySamples
// is implicit; nothing else required for mono speech.
setInteger(
MediaFormat.KEY_AAC_PROFILE,
MediaCodecInfo.CodecProfileLevel.AACObjectLC,
)
}
}
}