Files
amethyst/docs/plans/2026-03-16-feat-desktop-media-full-parity-plan.md
T
nrobi144 2a6087af27 feat(media): global media player — persistent playback across navigation
Media playback now survives navigation. A GlobalMediaPlayer singleton owns
VLC players and exposes StateFlows. Composables are thin viewports.
NowPlayingBar has full controls (volume, mute, save, fullscreen).
GlobalFullscreenOverlay renders video fullscreen above all screens.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 12:27:13 +02:00

50 KiB

title, type, status, date, origin, deepened
title type status date origin deepened
feat: Desktop Media — Full Parity feat active 2026-03-16 docs/brainstorms/2026-03-16-desktop-media-brainstorm.md 2026-03-16

Desktop Media — Full Parity

Enhancement Summary

Deepened on: 2026-03-16 Research agents used: Architecture Strategist, Performance Oracle, Security Sentinel, Code Simplicity Reviewer, Pattern Recognition Specialist, Race Condition Reviewer, Coil3 Best Practices Researcher, VLCJ Framework Docs Researcher, Blossom Protocol Researcher, Compose Desktop DnD/Clipboard Researcher, KMP Expect/Actual Pattern Analyzer

Critical Corrections (from original plan)

# Original Assumption Correction
1 metadata-extractor for EXIF stripping READ-ONLY library. Use Apache Commons Imaging ExifRewriter.removeExifMetadata() for lossless JPEG EXIF removal
2 LinkedHashMap.removeEldestEntry for caches NOT thread-safeget() in access-order mode mutates internal linked list. Use existing androidx.collection.LruCache (already KMP dep in commons) or ConcurrentHashMap
3 coil-gif works on JVM desktop Android-onlyAnimatedImageDecoder requires Android API 28+. Use org.jetbrains.skia.Codec for GIF decoding (first frame or animated)
4 VLCJ via SwingPanel Flickering confirmed. Use DirectRendering via CallbackVideoSurface → Skia Bitmap → Compose Image (no SwingPanel needed)
5 Missing kotlinx-coroutines-swing dep Required for Dispatchers.Main.immediate on JVM desktop with Coil3
6 BlossomFetcher "just move" to commons Not trivial — depends on BlossomServerResolver which uses Android LruCache + IRoleBasedHttpClientBuilder. Must abstract resolver interface first
7 UploadOrchestrator "extract" to commons Really a REWRITE — deeply coupled to Context, Uri, R.string, Account, ServerType enum with NIP-95/96/Blossom paths
8 vlc-setup plugin ready for production No confirmed arm64 macOS support. Only tested on Intel Mac (High Sierra). macOS support marked "experimental"
9 Coil3 memory cache "just works" Hard-codes 512MB total memory on non-Android. Must set maxSizeBytes() explicitly using Runtime.getRuntime().maxMemory()
10 MediaUploadResult has no platform deps Hidden dependency on BlurhashWrapper which imports from Android-only BlurHashFetcher.kt

Key Improvements from Research

  1. VLCJ DirectRendering pattern — Complete working code from ComposeVideoPlayer project (no SwingPanel)
  2. Coil3 JVM cookbookPlatformContext.INSTANCE, explicit cache sizing, stable Keyer for custom fetchers
  3. Blossom full spec — BUD-01 through BUD-11 documented with HTTP examples and auth flow
  4. Compose DnD modern APIModifier.dragAndDropTarget (old onExternalDrag removed in 1.8), text/uri-list fallback for Linux
  5. Race condition mitigations — 13 identified (3 CRITICAL), with concrete fixes
  6. Simplicity alternative — Desktop-only code first, extract to commons later (0 users → ship fast)

Overview

Add complete media functionality to Amethyst Desktop: image display, video playback, Blossom upload, drag-drop/paste UX, lightbox, gallery, encrypted DM media, profile gallery, server management, and audio playback. Currently desktop renders zero media — notes show URL text only, no inline images or video.

Problem Statement

Desktop Amethyst is text-only. The NoteCard at desktopApp/ui/note/NoteCard.kt:128-132 renders URLs as blue underlined text via RichTextContent. No AsyncImage, no Coil dependency, no ImageLoader setup. UserAvatar uses coil3.compose.AsyncImage from commons but likely fails silently — no SingletonImageLoader configured for desktop.

Android has 40+ media-related files across upload, display, playback, and gallery. All are tightly coupled to Android APIs (Context, Uri, ContentResolver, BitmapFactory, MediaMetadataRetriever, ExoPlayer).

Proposed Solution

Extract platform-agnostic media logic to commons using a 4-layer architecture, then build desktop-specific UI on top. Blossom-only upload (no NIP-96). Coil3 for images (already KMP). VLCJ with DirectRendering for video.

(see brainstorm: docs/brainstorms/2026-03-16-desktop-media-brainstorm.md)

Simplicity Consideration

The simplicity reviewer recommends a desktop-only-first approach: write desktop code in desktopApp/ without extracting to commons. Extract later when Android migration happens. Rationale: desktop has zero users, so ship fast and iterate. This plan documents the full extraction architecture but implementation should start desktop-only for Phases 0-3, deferring commons extraction to a follow-up PR.

Technical Approach

Architecture: 4-Layer Extraction

Layer 1: commons/commonMain — Pure protocol (BlossomClient, auth, server discovery)
Layer 2: commons/commonMain — expect/actual file operations
Layer 3: commons/{androidMain,jvmMain} — Platform actuals
Layer 4: desktopApp/ and amethyst/ — Platform UI

Research Insight: jvmAndroid Source Set

The codebase already has a jvmAndroid intermediate source set in commons/build.gradle.kts that bridges Android and Desktop JVM code. OkHttp-based HTTP clients work identically on both — place shared JVM code there, not in commonMain (which would try to compile for iOS/web targets).

Key Architectural Decisions

Decision Choice Rationale
Upload protocol Blossom only NIP-96 deprecated (see brainstorm)
Image loading Coil3 3.4.0 Already in project, KMP support
Video playback VLCJ 4.8.x + DirectRendering SwingPanel flickers; DirectRendering uses CallbackVideoSurface → Skia Bitmap
Upload code Desktop-only first, extract later Ship fast, 0 desktop users, extract when Android migrates
Desktop input Drag-drop + paste + file picker Essential desktop UX
EXIF stripping Apache Commons Imaging metadata-extractor is read-only; Commons Imaging does lossless EXIF removal
Video bundling Require system VLC for now vlc-setup plugin has no confirmed arm64 macOS support
GIF decoding org.jetbrains.skia.Codec coil-gif is Android-only; Skia Codec decodes frames
Cache implementation androidx.collection.LruCache Already KMP dep in commons; thread-safe unlike LinkedHashMap

Existing Codebase Inventory

Already shared (ready to use):

Component Location
BlurHashDecoder/Encoder commons/blurhash/
PlatformImage expect/actual commons/blurhash/PlatformImage.kt.android.kt / .jvm.kt
Base64ImagePlatform commons/base64Image/ (JVM uses ImageIO)
RichTextParser (URL → media type) commons/richtext/RichTextParser.kt
MediaContentModels commons/richtext/MediaContentModels.kt
UserAvatar (AsyncImage) commons/ui/components/UserAvatar.kt
GalleryParser commons/richtext/GalleryParser.kt
Blossom protocol events quartz/nipB7Blossom/ (kind 24242, 10063, URI, UploadResult)
NIP-94 FileHeader quartz/nip94FileMetadata/
NIP-92 IMetaTag quartz/nip92IMeta/
NIP-68 PictureEvent quartz/nip68Picture/
NIP-71 VideoEvent/VideoMeta quartz/nip71Video/
ProfileGalleryEntryEvent quartz/experimental/profileGallery/
androidx.collection.LruCache Already in commons dependencies (KMP)

Android-only (needs extraction or desktop equivalent):

Component File Android Deps Action
BlossomUploader amethyst/service/uploads/blossom/BlossomUploader.kt ContentResolver, Uri, Context, MimeTypeMap Rewrite HTTP core for desktop
UploadOrchestrator amethyst/service/uploads/UploadOrchestrator.kt Context, Uri, R.string, Account, ServerType Rewrite — too coupled for extraction
MultiOrchestrator amethyst/service/uploads/MultiOrchestrator.kt Context Defer
MediaUploadResult amethyst/service/uploads/MediaUploadResult.kt Hidden BlurhashWrapper dep Fix dep chain first
MediaCompressor amethyst/service/uploads/MediaCompressor.kt Context, Bitmap, Uri, Compressor expect/actual
BlurhashMetadataCalculator amethyst/service/uploads/BlurhashMetadataCalculator.kt Context, BitmapFactory, MediaMetadataRetriever expect/actual
BlossomServerResolver amethyst/service/uploads/blossom/bud10/BlossomServerResolver.kt LruCache (Android), IRoleBasedHttpClientBuilder Replace LruCache with androidx.collection.LruCache (KMP)
ServerHeadCache amethyst/service/uploads/blossom/bud10/ServerHeadCache.kt None Move (safe)
ImageLoaderSetup amethyst/service/images/ImageLoaderSetup.kt Android SingletonImageLoader, GIF decoder (API 28+) Desktop equivalent
ImageCacheFactory amethyst/service/images/ImageCacheFactory.kt Context for safeCacheDir Desktop paths
BlossomFetcher amethyst/service/images/BlossomFetcher.kt Depends on BlossomServerResolver (Android LruCache) Abstract resolver interface first
BlurHashFetcher amethyst/service/images/BlurHashFetcher.kt toAndroidBitmap() JVM: toBufferedImage()
Base64Fetcher amethyst/service/images/Base64Fetcher.kt toBitmap() JVM: toBufferedImage()
ZoomableContentView amethyst/ui/components/ZoomableContentView.kt Android zoom lib Desktop equivalent
ImageGallery amethyst/ui/components/ImageGallery.kt Pure Compose Extract
MediaAspectRatioCache amethyst/model/MediaAspectRatioCache.kt Android LruCache androidx.collection.LruCache (KMP)

Dependency Changes

desktopApp/build.gradle.kts — add:

// Image loading (Coil3)
implementation(libs.coil.compose)
implementation(libs.coil.okhttp)
implementation(libs.coil.svg)
// NOTE: coil-gif is Android-only, skip it. Use Skia Codec instead.

// Coroutines — required for Coil3 Main dispatcher on JVM desktop
implementation(libs.kotlinx.coroutines.swing)

// Video playback (VLCJ)
implementation("uk.co.caprica:vlcj:4.8.3")

// EXIF stripping (lossless)
implementation("org.apache.commons:commons-imaging:1.0.0-alpha5")

gradle/libs.versions.toml — add:

vlcj = "4.8.3"
commons-imaging = "1.0.0-alpha5"
kotlinx-coroutines-swing = { module = "org.jetbrains.kotlinx:kotlinx-coroutines-swing", version.ref = "kotlinx-coroutines" }

Research Insight: Coil3 JVM Desktop Configuration

// Critical: PlatformContext.INSTANCE (not Android Context)
// Critical: Set memory cache explicitly (Coil3 hard-codes 512MB total on non-Android)
// Critical: Set disk cache to OS-appropriate persistent path (default is temp dir)

fun createDesktopImageLoader(): ImageLoader {
    return ImageLoader.Builder(PlatformContext.INSTANCE)
        .memoryCache {
            MemoryCache.Builder()
                .maxSizeBytes(calculateDesktopMemoryCacheSize())
                .strongReferencesEnabled(true)
                .build()
        }
        .diskCache {
            DiskCache.Builder()
                .directory(getDesktopCacheDir().resolve("amethyst/image_cache").toOkioPath())
                .maxSizeBytes(512L * 1024 * 1024) // 512 MB
                .build()
        }
        .precision(Precision.INEXACT)
        .crossfade(true)
        .components {
            add(SvgDecoder.Factory()) // Works on JVM via Skia SVG
            // add(SkiaGifDecoder.Factory()) // Custom: first-frame GIF via Codec
            // add(BlossomFetcher.Factory(...))
        }
        .build()
}

fun calculateDesktopMemoryCacheSize(): Long {
    val maxMemory = Runtime.getRuntime().maxMemory()
    return (maxMemory * 0.25).toLong().coerceAtMost(512L * 1024 * 1024)
}

fun getDesktopCacheDir(): File {
    val os = System.getProperty("os.name").lowercase()
    return when {
        "mac" in os -> File(System.getProperty("user.home"), "Library/Caches")
        "win" in os -> File(System.getenv("LOCALAPPDATA") ?: System.getProperty("user.home"))
        else -> File(System.getenv("XDG_CACHE_HOME") ?: "${System.getProperty("user.home")}/.cache")
    }
}

Research Insight: Skia GIF Decoding (replaces coil-gif)

// org.jetbrains.skia.Codec provides frame-by-frame GIF decoding
// Skia supports: PNG, JPEG, WebP (static), BMP, ICO, WBMP
// GIF: first frame via Image.makeFromEncoded, full animation via Codec
// Animated WebP: first frame only (same Codec approach for animation)
// HEIC: NOT supported on JVM

class SkiaGifDecoder(private val source: ImageSource) : Decoder {
    override suspend fun decode(): DecodeResult {
        val bytes = source.source().use { it.readByteArray() }
        val data = org.jetbrains.skia.Data.makeFromBytes(bytes)
        val codec = org.jetbrains.skia.Codec.makeFromData(data)
        val bitmap = org.jetbrains.skia.Bitmap()
        bitmap.allocN32Pixels(codec.width, codec.height)
        codec.readPixels(bitmap, 0) // first frame
        bitmap.setImmutable()
        return DecodeResult(image = bitmap.asImage(), isSampled = false)
    }
    class Factory : Decoder.Factory { /* check mimeType == "image/gif" */ }
}

Research Insight: VLCJ DirectRendering Pattern

// NO SwingPanel — renders directly to Skia Bitmap → Compose Image
val callbackVideoSurface = CallbackVideoSurface(
    object : BufferFormatCallback {
        override fun getBufferFormat(w: Int, h: Int): BufferFormat {
            info = ImageInfo.makeN32(w, h, ColorAlphaType.OPAQUE)
            return RV32BufferFormat(w, h)
        }
        override fun allocatedBuffers(buffers: Array<out ByteBuffer>) {
            byteArray = ByteArray(buffers[0].limit())
        }
    },
    object : RenderCallback {
        override fun display(mp: MediaPlayer, bufs: Array<out ByteBuffer>, fmt: BufferFormat?) {
            bufs[0].get(byteArray); bufs[0].rewind()
            val bmp = Bitmap(); bmp.allocPixels(info!!); bmp.installPixels(byteArray)
            imageBitmap = bmp.asComposeImageBitmap() // triggers recomposition
        }
    },
    true, VideoSurfaceAdapters.getVideoSurfaceAdapter()
)
// Then display via: Image(bitmap = imageBitmap, ...)

Performance: 2 frame copies per display frame. ~8MB/frame at 1080p. Adequate for 1-2 players, CPU-intensive for more. New ByteArray per frame required (reusing crashes).

Research Insight: Compose Desktop Drag-and-Drop

// Modern API (1.8+): Modifier.dragAndDropTarget
// Old onExternalDrag was REMOVED in 1.8
// awtTransferable + DataFlavor.javaFileListFlavor for files
// text/uri-list fallback needed for Linux + browser drops
// Cannot inspect file types during drag hover — filter in onDrop

val dropTarget = remember {
    object : DragAndDropTarget {
        override fun onDrop(event: DragAndDropEvent): Boolean {
            val transferable = event.awtTransferable ?: return false
            if (transferable.isDataFlavorSupported(DataFlavor.javaFileListFlavor)) {
                val files = transferable.getTransferData(DataFlavor.javaFileListFlavor) as List<File>
                onFilesDropped(files)
                return true
            }
            return false
        }
    }
}

Clipboard: Compose ClipboardManager is text-only. Use AWT Toolkit.getDefaultToolkit().systemClipboard with DataFlavor.imageFlavor for image paste, DataFlavor.javaFileListFlavor for file paste.


Implementation Phases

Phase 0: Spike — Coil3 + VLCJ Desktop Validation

Goal: Confirm Coil3 loads images on JVM desktop with disk cache. Confirm VLCJ DirectRendering works in Compose.

# Task File Details
0.1 Coil3 spike desktopApp/spike/CoilSpike.kt ImageLoader.Builder(PlatformContext.INSTANCE) with DiskCache (persistent path), MemoryCache (explicit size), load HTTP image via AsyncImage
0.2 VLCJ spike desktopApp/spike/VlcjSpike.kt DirectRendering via CallbackVideoSurface → Skia Bitmap → Compose Image. Test mp4 URL. NO SwingPanel
0.3 Validate SVG spike Test coil-svg + SvgDecoder.Factory() — confirmed KMP-compatible via Skia SVG
0.4 Validate GIF (first frame) spike Test Image.makeFromEncoded(bytes) for static GIF. Optionally test Codec for frame count
0.5 Add kotlinx-coroutines-swing desktopApp/build.gradle.kts Required for Dispatchers.Main.immediate

Gate: If VLCJ DirectRendering drops frames badly or Coil3 disk cache fails, evaluate alternatives (Klarity for video, OkHttp cache interceptor for images). Deliverable: Spike branch with working video + image rendering. Effort: 1-2 days.

Research Insight: Spike Simplification

The simplicity reviewer notes Phase 0 can be a 20-line test in main() — no need for separate spike files. Just add deps, create ImageLoader, call AsyncImage in the existing desktop window.


Phase 1: Image Display Foundation

Goal: Images render inline in desktop notes with blurhash previews.

# Task Module Details
1.1 Add Coil3 deps to desktopApp desktopApp/build.gradle.kts coil-compose, coil-okhttp, coil-svg, kotlinx-coroutines-swing. NOT coil-gif (Android-only)
1.2 Create DesktopImageLoaderSetup desktopApp/service/images/DesktopImageLoaderSetup.kt ImageLoader.Builder(PlatformContext.INSTANCE) with explicit MemoryCache size (25% of JVM heap, max 512MB), DiskCache (OS-appropriate persistent path), OkHttp, SvgDecoder, custom fetchers. Package: com.vitorpamplona.amethyst.desktop.service.images
1.3 Create DesktopImageCacheFactory desktopApp/service/images/DesktopImageCacheFactory.kt macOS: ~/Library/Caches/AmethystDesktop, Linux: $XDG_CACHE_HOME/AmethystDesktop, Windows: %LOCALAPPDATA%/AmethystDesktop/cache. maxSizeBytes(512MB)
1.4 Create desktop BlossomFetcher desktopApp/service/images/DesktopBlossomFetcher.kt Desktop-only first (don't extract to commons yet). Simplified version without IRoleBasedHttpClientBuilder. Uses single OkHttpClient for now. Resolves blossom: URIs via kind 10063 server list
1.5 Create desktop BlurHashFetcher desktopApp/service/images/DesktopBlurHashFetcher.kt Uses PlatformImage.toBufferedImage()BitmapImage(toComposeImageBitmap()). Implement stable Keyer to avoid recomposition flicker
1.6 Create desktop Base64Fetcher desktopApp/service/images/DesktopBase64Fetcher.kt Uses existing Base64ImagePlatform.jvm.kt with ImageIO
1.7 Create MediaAspectRatioCache desktopApp/model/MediaAspectRatioCache.kt Use androidx.collection.LruCache(1000) (already KMP dep). NOT LinkedHashMap (not thread-safe)
1.8 Update NoteCard for inline images desktopApp/ui/note/NoteCard.kt Parse imeta tags from events. When URL matches image extension → AsyncImage with blurhash placeholder. Use RichTextParser media classification
1.9 Initialize ImageLoader in Main.kt desktopApp/Main.kt Call setSingletonImageLoaderFactory at app startup. Use @OptIn(DelicateCoilApi::class) for eager init
1.10 Add SkiaGifDecoder desktopApp/service/images/SkiaGifDecoder.kt Custom Coil3 Decoder using org.jetbrains.skia.Codec for GIF first-frame rendering

Deliverable: Notes with images show blurhash placeholder → full image. UserAvatar also starts working.

Acceptance Criteria:

  • ./gradlew :desktopApp:compileKotlin passes
  • Desktop app shows inline images in feed notes
  • Blurhash placeholders display while loading
  • SVG images render correctly (via SvgDecoder)
  • GIF shows first frame (static display is acceptable for MVP)
  • UserAvatar renders profile pictures
  • Disk cache persists across app restarts (not in temp dir)
  • Android build unbroken: ./gradlew :amethyst:compileDebugKotlin

Research Insight: Coil3 Custom Fetcher Gotchas

  • Use coil3.Uri not android.net.Uri
  • ConnectivityChecker is a no-op on desktop (always returns "connected")
  • Custom fetchers must implement stable Keyer or cache key — otherwise recomposition triggers re-fetch (#2551)
  • PlatformContext.INSTANCE is an empty singleton on JVM — don't cast or use Android methods

Phase 2: Desktop Blossom Upload Client

Goal: Upload media from desktop to Blossom servers.

Approach: Write a desktop-only upload client in desktopApp/. Don't extract to commons yet (simplicity-first). The Android upload path stays unchanged.

# Task Module Details
2.1 Create DesktopBlossomClient desktopApp/service/upload/DesktopBlossomClient.kt HTTP PUT /upload, /mirror, /media, DELETE. Takes File + metadata. Returns BlossomUploadResult (from quartz). Uses OkHttp. Package: com.vitorpamplona.amethyst.desktop.service.upload
2.2 Create DesktopBlossomAuthHelper desktopApp/service/upload/DesktopBlossomAuthHelper.kt Creates kind 24242 auth events via quartz BlossomAuthorizationEvent, base64-encodes for Authorization: Nostr <b64> header. Include t, x, expiration, server tags per BUD-11
2.3 Create DesktopServerDiscovery desktopApp/service/upload/DesktopServerDiscovery.kt Queries kind 10063 from relays, resolves server list, caches with androidx.collection.LruCache
2.4 Create DesktopMediaMetadata desktopApp/service/upload/DesktopMediaMetadata.kt ImageIO.read(file) for dimensions, BlurHashEncoder (from commons) for blurhash, file size + SHA-256
2.5 Create DesktopMediaCompressor desktopApp/service/upload/DesktopMediaCompressor.kt JPEG: ImageWriteParam.compressionQuality. EXIF strip: Apache Commons Imaging ExifRewriter.removeExifMetadata(). No re-encoding for EXIF removal
2.6 Create DesktopUploadOrchestrator desktopApp/service/upload/DesktopUploadOrchestrator.kt Coordinate: strip EXIF → compute metadata → auth → upload. Accept File directly (not Uri). Blossom-only (no NIP-95/96)
2.7 Create DesktopMediaUploadTracker desktopApp/service/upload/DesktopMediaUploadTracker.kt Simple mutableStateOf(isUploading: Boolean) for MVP (matches existing Android's 47-line tracker). StateFlow upgrade later
2.8 BUD-06 preflight check desktopApp/service/upload/ServerHeadCache.kt HEAD /upload with X-Content-Type, X-Content-Length, X-SHA-256 headers. Cache results
2.9 Unit tests desktopApp/src/jvmTest/ Mock OkHttp responses. Test upload, auth header format (base64 of kind 24242), SHA-256 computation, EXIF stripping

Deliverable: ./gradlew :desktopApp:jvmTest passes. Desktop can upload files to Blossom.

Acceptance Criteria:

  • BlossomClient uploads file to test server (integration test or mock)
  • Auth headers correctly signed (kind 24242 with t=upload, x=<hash>, expiration, server)
  • SHA-256 computed correctly for test files
  • EXIF stripped losslessly from JPEG test image
  • BUD-06 preflight HEAD works
  • Android build unbroken

Research Insight: Blossom Protocol Details

Upload flow per BUD-02:

  1. Compute SHA-256 of exact file bytes
  2. Create kind 24242 event: t=upload, x=<sha256>, expiration=+1hr, server=<domain>
  3. Base64-encode (standard base64 works despite spec saying base64url)
  4. PUT /upload with Authorization: Nostr <b64>, Content-Type, Content-Length, X-SHA-256
  5. Response: BlobDescriptor JSON with url, sha256, size, type, uploaded

/upload vs /media (BUD-05):

  • /upload — server MUST NOT modify blob. Hash preserved.
  • /media — server MAY optimize (strip EXIF, compress). Returned hash differs. Primal uses this exclusively.
  • Recommendation: Use /upload first (simpler). Add /media support later for trusted servers.

Auth token security (BUD-11):

  • Always include server tag to prevent token replay across servers
  • Unscoped delete tokens can be replayed — always scope delete operations
  • Include x tag (blob hash) for upload/delete operations

Research Insight: Race Conditions in Upload

CRITICAL: Upload scope cancellation on navigation. If user navigates away during upload, the coroutine scope gets cancelled. Mitigation: launch uploads in a GlobalScope or applicationScope that survives navigation, with a reference in the tracker.

HIGH: DnD during active upload. User drops new files while upload in progress. Mitigation: queue uploads, don't replace in-flight uploads.


Phase 3: Desktop Upload UX

Goal: Upload media from desktop via file picker, drag-drop, and clipboard paste.

# Task Module Details
3.1 Desktop file picker desktopApp/ui/media/DesktopFilePicker.kt java.awt.FileDialog (native look) or JFileChooser with media type filters. Multi-select support
3.2 Drag-and-drop handler desktopApp/ui/media/DragDropHandler.kt Modifier.dragAndDropTarget (modern API, 1.8+). event.awtTransferable + DataFlavor.javaFileListFlavor. Include text/uri-list fallback for Linux. Visual drop zone indicator via onEntered/onExited callbacks
3.3 Clipboard paste handler desktopApp/ui/media/ClipboardPasteHandler.kt AWT Toolkit.getDefaultToolkit().systemClipboard. Check DataFlavor.imageFlavor (→ save BufferedImage to temp file → upload) and DataFlavor.javaFileListFlavor (→ direct file upload)
3.4 Upload dialog composable desktopApp/ui/media/UploadDialog.kt Progress bar per file, server selector (kind 10063 list), alt text input, cancel button
3.5 Upload preview thumbnails desktopApp/ui/media/UploadPreview.kt Thumbnail generation from local file using ImageIO
3.6 Wire upload to ComposeNoteDialog desktopApp/ui/ComposeNoteDialog.kt Add "Attach media" button. Connect to file picker + drag-drop. On upload complete → insert imeta tag + URL into note
3.7 Media button in note composer desktopApp/ui/ComposeNoteDialog.kt IconButton (Attach) → opens file picker. Shows attached files with remove option

Deliverable: Drop/paste/pick file → preview → upload → note publishes with embedded image.

Acceptance Criteria:

  • File picker opens, filters by media type, multi-select works
  • Drag file onto compose area shows drop zone highlight
  • Clipboard paste (Ctrl+V / Cmd+V) detects images and files
  • Upload progress shows per-file
  • Server selection from kind 10063 list
  • Alt text saved in imeta tag
  • Published note contains correct imeta tags
  • Cancel upload works mid-flight

Research Insight: DnD Platform Quirks

Platform Behavior Note
macOS Finder javaFileListFlavor works Aliases resolve to alias file (use canonicalFile)
Windows Explorer javaFileListFlavor works Stable
Linux Nautilus javaFileListFlavor works Some WMs only provide text/uri-list
Browser drag Usually text/uri-list Not javaFileListFlavor — handle separately
Cannot inspect file types during drag hover Only know flavor (files vs text), not extensions Filter in onDrop, show generic "drop files here" during hover

Phase 4: Video Playback (VLCJ DirectRendering)

Goal: Videos play inline in desktop notes.

# Task Module Details
4.1 Add VLCJ deps desktopApp/build.gradle.kts uk.co.caprica:vlcj:4.8.3. No vlc-setup plugin — require system VLC for now (arm64 macOS not confirmed for plugin)
4.2 DesktopVideoPlayer composable desktopApp/ui/media/DesktopVideoPlayer.kt DirectRendering: CallbackVideoSurfaceRV32BufferFormatByteBuffer.get(byteArray)Skia Bitmap.installPixels()asComposeImageBitmap() → Compose Image. Lazy init player
4.3 Video controls overlay desktopApp/ui/media/VideoControls.kt Play/pause, seek bar, volume slider, fullscreen toggle, time display. Auto-hide after 2s. Compose overlay on top of Compose Image (no z-order issues since no SwingPanel)
4.4 Video in note rendering desktopApp/ui/note/NoteCard.kt When URL matches video extension → show thumbnail + play button. On click → load VLCJ player
4.5 Video upload support desktopApp/ui/media/UploadDialog.kt Accept video files in upload flow. No transcoding — upload as-is
4.6 Video thumbnail generation desktopApp/service/media/VideoThumbnailGenerator.kt Use VLCJ snapshots().get(320, 0) on a dedicated player. Play → seek to 2s → pause → snapshot → stop
4.7 VLCJ player pool desktopApp/service/media/VlcjPlayerPool.kt Single MediaPlayerFactory, pool of 2-3 reusable EmbeddedMediaPlayer instances. Reuse players (media().play(newMrl)) instead of create/destroy. Strong references to prevent GC crash

Deliverable: Video posts play inline. Controls work. Resource-efficient.

Acceptance Criteria:

  • mp4, webm URLs play inline (m3u8 if VLC supports)
  • Play/pause, seek, volume controls functional
  • Fullscreen toggle works
  • Video pauses when scrolled out of view
  • Player instances pooled (max 3 active)
  • Graceful fallback if VLC init fails (show URL link)
  • VLC not installed → show "Install VLC" prompt with download link

Research Insight: VLCJ Lifecycle Critical Rules

  1. Never let player instances get garbage collected — native callbacks crash JVM if Java object is collected
  2. Keep strong references to MediaPlayerFactory, EmbeddedMediaPlayer, and video surfaces
  3. Reuse players — one factory, pool of players. Change media with media().play(newMrl)
  4. Release order: player first, then factory
  5. macOS: only CallbackVideoSurface works — no heavyweight AWT components since Java 7
  6. Audio-only: Use MediaPlayerFactory("--no-video") for audio tracks

Research Insight: Race Conditions in Video

CRITICAL: VLCJ use-after-release segfault. If DisposableEffect.onDispose releases player while a RenderCallback.display() is executing on the VLC thread → native crash. Mitigation: state machine per player slot. Set RELEASING state, wait for in-flight render to complete, then release.

CRITICAL: Rapid navigation between notes with video. User scrolls fast → player allocated → scrolled away before mediaPlayerReady event → player released during initialization. Mitigation: debounce player allocation (300ms visibility threshold before allocating).


Goal: Full-screen media viewing with zoom and gallery navigation.

# Task Module Details
5.1 Lightbox overlay desktopApp/ui/media/LightboxOverlay.kt Full-window overlay with semi-transparent backdrop. Click outside to close
5.2 Zoom + pan desktopApp/ui/media/ZoomableImage.kt Mouse wheel zoom, click-drag pan. graphicsLayer with scaleX/Y + translationX/Y. Double-click to reset
5.3 Gallery carousel desktopApp/ui/media/GalleryCarousel.kt Multi-image navigation with left/right buttons and indicator dots
5.4 Save to disk desktopApp/ui/media/SaveMediaAction.kt Button → JFileChooser save dialog. Download URL → write to chosen path
5.5 Keyboard shortcuts desktopApp/ui/media/LightboxOverlay.kt Esc=close, Left/Right=navigate, +/-=zoom, Ctrl+S=save, Space=play/pause (video)
5.6 Extract ImageGallery layout logic amethyst/ui/components/ImageGallery.ktcommons/ The 1-5+ image grid layout is pure Compose — extract to commons for reuse

Deliverable: Click image → fullscreen lightbox with zoom. Multi-image carousel. Keyboard navigation.

Acceptance Criteria:

  • Click any inline image opens lightbox
  • Mouse wheel zooms in/out smoothly
  • Click-drag pans when zoomed
  • Arrow keys navigate gallery
  • Esc closes lightbox
  • Save downloads file to disk
  • Video plays in lightbox with controls

Research Insight: Race Condition — Lightbox Double-Click

MEDIUM: User double-clicks an image. First click opens lightbox, second click registers as "click outside" → closes immediately. Mitigation: 200ms debounce on lightbox open/close transitions.


Phase 6: Encrypted Media (NIP-17 DM Files)

Goal: Send and receive encrypted files in DMs.

# Task Module Details
6.1 Verify encryption is shared quartz/nip17Dm/files/ ChatMessageEncryptedFileHeaderEvent (kind 15) already in commonMain with AESGCM cipher from quartz utils. Should work on desktop as-is
6.2 Encrypted upload flow desktopApp/ui/chat/ Pick file → encrypt with NIP-44 → Blossom upload → create ChatMessageEncryptedFileHeaderEvent
6.3 Encrypted download + display desktopApp/ui/chat/ Receive encrypted file event → download from Blossom → decrypt → display/save
6.4 Chat file upload UI desktopApp/ui/chat/ChatFileUploader.kt Similar to Phase 3 upload dialog but in DM context. Show encryption indicator

Deliverable: Desktop DM users can send/receive encrypted images and files.

Acceptance Criteria:

  • Send image in DM from desktop, visible on Android
  • Receive encrypted image from Android, displays on desktop
  • Non-participants cannot view encrypted media
  • Upload progress shown in chat compose

Goal: View and create picture-first posts (kind 20). Profile gallery tab.

# Task Module Details
7.1 Picture event display desktopApp/ui/note/PictureDisplay.kt Kind 20 renderer. Image-first layout with title + description below. Uses imeta tags for multi-image
7.2 Profile gallery tab desktopApp/ui/profile/GalleryTab.kt Grid layout of user's picture posts. Lazy grid with thumbnails
7.3 Picture post composer desktopApp/ui/media/PicturePostComposer.kt Create kind 20 events. Multi-image upload + imeta + title + description
7.4 Gallery entry event support desktopApp/ui/profile/GalleryTab.kt Read ProfileGalleryEntryEvent (kind 1163) for curated galleries

Deliverable: Profile shows gallery tab with image grid. Users can create picture posts.

Acceptance Criteria:

  • Profile screen shows Gallery tab
  • Gallery grid loads thumbnails with blurhash
  • Click thumbnail opens lightbox
  • Can create kind 20 picture post from desktop
  • Picture post visible on Android Amethyst

Phase 8: Media Server Management

Goal: UI for managing Blossom server list (kind 10063).

# Task Module Details
8.1 Server list settings screen desktopApp/ui/settings/MediaServerSettings.kt View/add/remove servers. Drag to reorder
8.2 Server status checking desktopApp/service/media/ServerHealthCheck.kt HEAD request to verify availability. Show green/red indicator
8.3 Default server selection settings UI Mark preferred upload server. Persist in Preferences
8.4 Publish kind 10063 Uses quartz BlossomServersEvent Update on relays when user modifies list

Deliverable: Settings page to manage Blossom servers.

Acceptance Criteria:

  • Server list loads from kind 10063 event
  • Add/remove servers updates list
  • Health check shows server status
  • Changes published to relays
  • Upload dialog reflects server list

Phase 9: Audio Playback

Goal: Play audio tracks (MP3, OGG, FLAC, WAV) inline in notes.

# Task Module Details
9.1 Audio player composable desktopApp/ui/media/AudioPlayer.kt VLCJ with MediaPlayerFactory("--no-video"). Play/pause + progress bar + time. No video surface needed
9.2 Audio in note rendering desktopApp/ui/note/NoteCard.kt URL matches audio extension → show inline audio player
9.3 Waveform visualization (optional) desktopApp/ui/media/AudioWaveform.kt Simple amplitude bars. Low priority — skip if VLCJ doesn't expose PCM

Deliverable: Audio files play inline in notes.

Acceptance Criteria:

  • MP3, OGG, FLAC URLs show audio player
  • Play/pause and seek work
  • Multiple audio players on screen don't conflict

Phase Dependency Graph

Phase 0 (Spike) ──→ Phase 1 (Images) ─────┬──→ Phase 5 (Lightbox)
                                            │
                     Phase 2 (Upload) ──────┼──→ Phase 3 (Desktop UX) ──→ Phase 6 (Encrypted)
                                            │
                                            ├──→ Phase 7 (Gallery)
                                            │
                                            └──→ Phase 8 (Server Mgmt)

                     Phase 4 (Video) ────────────→ Phase 9 (Audio)

Independent: Phase 0 first. Then Phase 1, 2, 4 can run in parallel.
Ship Phases 0-3 as first PR. Phases 4-9 are incremental follow-ups.

System-Wide Impact

Interaction Graph

  • Image display: NoteCard → RichTextParser → MediaContentModels → Coil3 ImageLoader → custom Fetchers → OkHttp → Blossom servers
  • Upload: ComposeNoteDialog → DesktopFilePicker/DragDrop/Clipboard → DesktopMediaCompressor → DesktopBlossomClient → OkHttp → Blossom server → imeta tag → relay broadcast
  • Auth chain: DesktopUploadOrchestrator → DesktopBlossomAuthHelper → quartz BlossomAuthorizationEvent → Account.signer → kind 24242 → base64 header

Error & Failure Propagation

Error Source Handling
Coil3 load failure Show blurhash if available, else placeholder icon. Log error
Blossom upload network error Retry with exponential backoff (3 attempts). Show error in upload dialog
VLCJ init failure (VLC not installed) Show "Install VLC" prompt with download link. Fall back to URL link
SHA-256 mismatch after upload Re-upload. Warn user if persistent
Server unreachable (BUD-06 preflight) Skip server, try next in list
Encrypted media decrypt failure Show "Unable to decrypt" placeholder
EXIF strip failure Upload anyway with warning (privacy risk)

Race Condition Mitigations (from review)

Severity Issue Mitigation
CRITICAL LinkedHashMap concurrent corruption Use androidx.collection.LruCache (thread-safe) or ConcurrentHashMap
CRITICAL VLCJ use-after-release segfault State machine per player: IDLELOADINGPLAYINGRELEASING. Guard RenderCallback.display() with state check
CRITICAL Upload scope cancelled on navigation Launch uploads in application-scoped coroutine (survives navigation)
HIGH Stale blurhash/image flash in lazy list Stable Keyer for Coil3 custom fetchers
HIGH DnD during active upload Queue uploads, don't replace in-flight
HIGH Non-atomic progress + state update Use MutableStateFlow<UploadState> (single source of truth, atomic update)
HIGH Pause-after-dispose (VLCJ) Check player state before calling controls().pause()
MEDIUM Stale server cache TTL on server discovery cache (5 min)
MEDIUM Lightbox double-click 200ms debounce on open/close
MEDIUM Gallery rapid navigation Debounce image load requests
MEDIUM StateFlow progress frequency Throttle upload progress updates to 100ms intervals
LOW Clipboard + DnD concurrent Serialize input handling (queue)
LOW Video control timer overlap Single timer job for auto-hide

State Lifecycle Risks

Risk Mitigation
Upload in progress + user navigates away Application-scoped coroutine. Upload continues in background. Show notification on completion
VLCJ player leak DisposableEffect with state machine release. Player pool enforces max count. Strong references prevent GC crash
Coil disk cache corruption Coil3 handles internally. Worst case: re-download
Partial upload (network cut) SHA-256 pre-computed. Server rejects incomplete uploads. Client retries

API Surface Parity

Interface Android Desktop Same?
BlossomClient Android-specific Desktop-specific Different (for now — extract to commons later)
UploadOrchestrator Android-specific Desktop-specific Different (for now)
ImageLoader fetchers Android-specific Desktop-specific Different (for now — same Coil3 API though)
VideoPlayer ExoPlayer VLCJ DirectRendering Different
FilePicker ActivityResultContracts JFileChooser / FileDialog Different
MediaCompressor Zelory + MediaCodec ImageIO + Commons Imaging Different

Integration Test Scenarios

  1. Upload round-trip: Desktop uploads image → published note with imeta → other client sees image via Blossom URL
  2. Cross-platform encrypted DM: Android sends encrypted image → Desktop receives, decrypts, displays
  3. Blossom fallback: Primary server down → resolver tries secondary server from kind 10063
  4. Gallery consistency: Create kind 20 on desktop → visible in Android profile gallery
  5. Video lifecycle: Scroll feed with 5 video notes → only 2-3 VLCJ players active → no OOM

Alternative Approaches Considered

Approach Why Rejected
NIP-96 + Blossom NIP-96 deprecated. Double protocol = double maintenance (see brainstorm)
JavaFX MediaView for video Adds JavaFX runtime (~50MB), conflicts with Compose Desktop rendering
SwingPanel for VLCJ Confirmed flickering + always-on-top z-order issues. DirectRendering avoids entirely
Extract to commons immediately Over-engineering for 0 desktop users. Ship desktop-only first, extract when needed
FFmpeg JNI for video Complex native binding. VLCJ wraps VLC which bundles FFmpeg internally
Ktor instead of OkHttp OkHttp already used everywhere. Both are JVM. No benefit to switching
metadata-extractor for EXIF strip Read-only library. Only reads EXIF, cannot remove it
LinkedHashMap for caches Not thread-safe in access-order mode. get() mutates internal state
Klarity (FFmpeg-based) Interesting alternative (65 stars, Feb 2026 update), but small community. Worth prototyping as VLCJ backup

Risk Analysis & Mitigation

Risk Probability Impact Mitigation
VLCJ DirectRendering CPU overhead at 1080p Medium Medium ~8MB/frame copy. Adequate for 1-2 players. Pool limits exposure
VLC not installed on user system High High Show clear "Install VLC" prompt. Link to VLC download. Future: bundle VLC
vlc-setup plugin no arm64 macOS Confirmed Medium Don't use plugin for now. Require system VLC. Revisit when plugin matures
Coil3 JVM edge cases Low Medium Coil 3.4.0 is mature. Explicit cache sizing avoids defaults. SvgDecoder confirmed working
Commons extraction breaks Android N/A (deferred) N/A Desktop-only approach for Phases 0-3 eliminates this risk
Apache Commons Imaging alpha status Medium Low Only using EXIF strip (well-tested path). Fallback: skip EXIF strip for MVP
Scope creep across 9 phases High High Ship Phases 0-3 as first PR. Later phases are incremental
Compose DnD experimental API changes Medium Low Pin Compose version. AWT DropTarget fallback available

Acceptance Criteria

Functional Requirements

  • Images display inline in desktop feed notes with blurhash placeholders
  • Upload works via file picker, drag-drop, and clipboard paste
  • Videos play inline with controls (play/pause, seek, volume)
  • Lightbox opens on image click with zoom and gallery navigation
  • Encrypted media works in DMs (send + receive)
  • Profile gallery shows picture posts
  • Blossom server management UI in settings
  • Audio plays inline in notes

Non-Functional Requirements

  • Image load time < 2s for typical photos on broadband
  • Upload shows progress in real-time
  • Max 3 active video players (prevent OOM)
  • Disk cache respects 512MB limit
  • EXIF stripped before upload (privacy)
  • Android build never broken (desktop-only approach)

Quality Gates

  • ./gradlew :desktopApp:compileKotlin passes
  • ./gradlew :desktopApp:jvmTest passes (upload unit tests)
  • ./gradlew :amethyst:compileDebugKotlin passes (no Android breakage)
  • ./gradlew spotlessApply passes
  • Manual test: full upload → display round-trip on desktop

Key Files (New or Modified)

New Files

File Phase
desktopApp/.../service/images/DesktopImageLoaderSetup.kt 1
desktopApp/.../service/images/DesktopImageCacheFactory.kt 1
desktopApp/.../service/images/DesktopBlossomFetcher.kt 1
desktopApp/.../service/images/DesktopBlurHashFetcher.kt 1
desktopApp/.../service/images/DesktopBase64Fetcher.kt 1
desktopApp/.../service/images/SkiaGifDecoder.kt 1
desktopApp/.../model/MediaAspectRatioCache.kt 1
desktopApp/.../service/upload/DesktopBlossomClient.kt 2
desktopApp/.../service/upload/DesktopBlossomAuthHelper.kt 2
desktopApp/.../service/upload/DesktopServerDiscovery.kt 2
desktopApp/.../service/upload/DesktopMediaMetadata.kt 2
desktopApp/.../service/upload/DesktopMediaCompressor.kt 2
desktopApp/.../service/upload/DesktopUploadOrchestrator.kt 2
desktopApp/.../service/upload/DesktopMediaUploadTracker.kt 2
desktopApp/.../ui/media/DesktopFilePicker.kt 3
desktopApp/.../ui/media/DragDropHandler.kt 3
desktopApp/.../ui/media/ClipboardPasteHandler.kt 3
desktopApp/.../ui/media/UploadDialog.kt 3
desktopApp/.../ui/media/UploadPreview.kt 3
desktopApp/.../ui/media/DesktopVideoPlayer.kt 4
desktopApp/.../ui/media/VideoControls.kt 4
desktopApp/.../service/media/VlcjPlayerPool.kt 4
desktopApp/.../service/media/VideoThumbnailGenerator.kt 4
desktopApp/.../ui/media/LightboxOverlay.kt 5
desktopApp/.../ui/media/ZoomableImage.kt 5
desktopApp/.../ui/media/GalleryCarousel.kt 5
desktopApp/.../ui/media/AudioPlayer.kt 9

Modified Files

File Phase Change
desktopApp/build.gradle.kts 0,1,4 Add Coil3, kotlinx-coroutines-swing, VLCJ, Commons Imaging deps
gradle/libs.versions.toml 0,1,4 Add new version entries
desktopApp/ui/note/NoteCard.kt 1,4,9 Add inline media rendering
desktopApp/ui/ComposeNoteDialog.kt 3 Add media attach button + upload flow
desktopApp/Main.kt 1 Initialize ImageLoader

Sources & References

Origin

Internal References

  • Existing shared UI analysis: docs/shared-ui-analysis.md
  • PlatformImage expect/actual pattern: commons/blurhash/PlatformImage.kt
  • Android ImageLoader setup: amethyst/service/images/ImageLoaderSetup.kt:25-91
  • Android BlossomUploader: amethyst/service/uploads/blossom/BlossomUploader.kt
  • NoteCard (current text-only): desktopApp/ui/note/NoteCard.kt:128-132
  • ComposeNoteDialog (current text-only): desktopApp/ui/ComposeNoteDialog.kt
  • Blossom protocol research: docs/brainstorms/2026-03-16-blossom-protocol-research.md

External References

Answered Questions (from original plan)

# Question Answer
1 Does coil3-gif work on JVM desktop? No. Android-only (AnimatedImageDecoder requires API 28+). Use org.jetbrains.skia.Codec for GIF decoding
2 Does vlc-setup support arm64 macOS? Unconfirmed. Only Intel Mac (High Sierra) tested. macOS marked "experimental". Require system VLC for now
3 Does awtTransferable deliver files on macOS? Yes. DataFlavor.javaFileListFlavor works with Finder on JDK 17+. Include text/uri-list fallback for Linux
4 Is LinkedHashMap.removeEldestEntry sufficient? No. Not thread-safe in access-order mode. Use androidx.collection.LruCache (already KMP dep) or ConcurrentHashMap
5 Is Coil3 SvgDecoder KMP? Yes. Works on JVM via org.jetbrains.skia.svg.SVGDOM. Add SvgDecoder.Factory() to ImageLoader components
6 Is UserAvatar rendering on desktop? Likely failing silently — no SingletonImageLoader configured. Will work after Phase 1 ImageLoader init

Remaining Unanswered Questions

  1. Does VLCJ DirectRendering achieve acceptable frame rate at 1080p on Apple Silicon?
  2. Can org.jetbrains.skia.Codec handle all animated WebP variants reliably on JVM?
  3. Is VLC 3.0.21 universal binary reliable via JNA on arm64 macOS with JDK 17+?
  4. Is there a GPU-accelerated path for VLCJ → Skia Bitmap transfer (avoid CPU copy)?
  5. Does Modifier.dragAndDropTarget work on Linux Wayland in Compose 1.10.x?
  6. Can Klarity handle streaming URLs (HLS, DASH) as VLCJ alternative?
  7. Does Coil3 @ExperimentalCoilApi NetworkFetcher constructor stay stable across versions?