Skip to content

feat(audio): Add ComfyUI workflows for Stable Audio Open 3.0 & YuE music generationΒ #12

Description

@spelech

🎯 Objective & Overview

Enable text-to-sound-effects (SFX), ambient audio, and full musical song generation by creating ComfyUI workflow presets for Stable Audio Open 3.0 and YuE (乐).


πŸ“‚ Files & Components to Touch

  1. Workflow Presets (Workflows/Audio/):
    • Workflows/Audio/stable_audio_open_sfx.json (Generates up to 47s stereo SFX / music clips)
    • Workflows/Audio/yue_full_song.json (Dual-track lyrics-to-music generation)
  2. Endpoints (Endpoints/WorkflowEndpoints.cs):
    • Add GET /api/audio/workflows listing available audio generation workflow templates.
    • Add POST /api/audio/generate queuing audio workflow execution on ComfyUI.
    • Add GET /api/audio/files listing generated .wav / .flac / .mp3 output files from output_audio/.
  3. UI Player (LocalLLMServerManager.Shared/Views/Controls/EngineStudioTabControl.axaml):
    • Add audio waveform visualizer or embedded HTML5 <audio controls> player for previewing generated tracks.

πŸ”Œ API Contract

POST /api/audio/generate

{
  "workflowId": "stable_audio_open_sfx",
  "prompt": "Cyberpunk atmospheric ambient drone, heavy synthesizer, cinematic low end, 48kHz stereo",
  "negativePrompt": "low quality, harsh distortion",
  "durationSeconds": 30,
  "seed": -1
}

Response: 200 OK

{
  "promptId": "f9e8d7c6-b5a4-...",
  "status": "queued",
  "wsUrl": "ws://127.0.0.1:8188/ws"
}

πŸ› οΈ Step-by-Step Implementation Guide

  1. Create and validate ComfyUI workflow JSONs using ComfyUI-Stable-Audio or native audio nodes.
  2. Implement substitution and queuing in WorkflowEndpoints.cs.
  3. Save generated audio outputs into wwwroot/output_audio/ and serve them statically.
  4. Add audio player controls to the Studio UI.

πŸ§ͺ Verification

  • dotnet build
  • dotnet test --filter Category!=LiveExternal
  • npm run lint & npx tsc --noEmit

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions