Skip to content
CommanderTurtlePublic

About

Multimedia Frontend Wrappers for AI tooling

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

mm-tools

mm-tools is a pruned, local-first multimedia workstation monorepo. It contains the runtime source, per-project setup entrypoints, and browser/CLI frontends used by this tool suite. It does not contain checkpoints, virtual environments, caches, generated media, or training/evaluation material.

Root-authored mm-tools code is distributed under the root AGPLv3 LICENSE. Imported upstream attributions and revisions are recorded in the commit ledger.

The all-in-one frontend for local multimedia inference. Cutting out the cloud.

feelsyogaman

Runtime inference is local. Network access is needed only while installing or downloading the model artifacts required.

This repo was created for protecting open-source runtimes under a documentable heirarchy with AGPL.

What you actually have to do

Directory Runtimes
vocals/tts, vocals/stt, vocals/singing Speech synthesis, transcription, singing
music Composition, generation, score transcription
motion-sim Character animation and motion reconstruction
sculpting Mesh generation and editing
world-gen World generation
visual-modeling Image understanding and decomposition

translate, ideogram, img2svg, video-compact, models, workflows, and tests remain at the root. GIF/AVIF conversion lives in video-compact/gif-avif.

Install the host tools required by the projects you plan to run:

  • uv for Python environments and packages
  • ffmpeg and ffprobe for audio/video projects
  • bun - sandwich optional
  • Rust/Cargo for img2svg and the optional CrisperWhisper launcher
  • a GPU or CPU, and likely a monitor

Then download the checkpoints once (the dependencies):

Due to the size of the model weights. Make sure you're sitting on a speedy drive.

cd ~ && git clone https://github.com/CommanderTurtle/mm-tools multimedia && cd multimedia
cd models
uv venv --python 3.12.10 --seed --managed-python
source .venv/bin/activate
uv pip install -U huggingface_hub hf_transfer gdown==5.2.0

uv run download_models.py
# Choose workers first, then select project bundles or press Enter for all.
# Non-interactive full install: uv run download_models.py all --yes --workers 24
deactivate # when done

The downloader is resumable. Run it again after an interrupted transfer. New source-import bundles are named 4d, lingbot, lingbot-5090, fire3d, aukspeech, sculpting, worldsculpt, nvidia-sim, v2v, yue2, and motionbricks, flowhmr, optional flowhmr-physics, and physag3d; use uv run download_models.py --list for the complete current menu. The source imports, single-GPU adapters, setup/start contracts, and private browser studios are included here; model artifacts remain external and resumable. LingBot deliberately separates its shared encoder/VAE bundle from the official 1.3B single-5090 DiT and does not retain the unusable 14B DiT.

Model downloads remain governed by each model's native license and any access terms accepted for their usage. ARDY uses an ungated, pre-merged INT8 LLM2Vec encoder that preserves the exact Llama 3 + MNTP + supervised-adapter pipeline; the original gated BF16 assembly is not downloaded. 4DAnyone's neutral SMPL-X body model remains a separate manual download. See the model inventory for every included artifact, output path, and explicit exclusion. The ability to self-host a tool rivaling Adobe in 2026 was the purpose for this repo.

Assuming one has a nice GPU, you can host a wide variety of tools. Emulating the top-tier closed source software out there, solely with open-source runtimes. It wouldn't be without the community that building something like this would be possible.

~16-20gb vram recommended for most tools. The largest quality-first music paths benefit from 24-32 GB. Each project has its own upper bound, and none is authoritative if you run them one at a time. (only ~8-9gb for LongCat)

So. What can you do?

Voice Cloning

2-longcat.mp4

Translating

compatible with vox

translate

  • an in-house built translation pipeline with native web-ui. Compatibile with vox system-audio driver. (This exact fork)

Speech-to-Text

compatible with vox

whisper.mp4

Motion authoring

Motion authoring: MotionBricks studio provides SOMA and Unitree G1 inbetweening, Smart Primitive authoring, scene editing and interactive character preview through the pinned native runtime.

FlowHMR studio reconstructs SMPL-H motion from reference video, with person selection, reusable intermediates and an optional side-by-side PHC humanoid rollout.

PhysAg3D studio builds articulated objects from images through separately saved TRELLIS generation and OMP/Blender/MuJoCo stages, with part controls, motion playback and portable asset bundles.

Idol studio turns images into Gaussian splats, with the original Space presets and sampling controls, an orbit viewer and PLY export.

SceneBloom studio generates furnished rooms with local vision, saved stages, spatial inspection and textured GLB export.

Hy-Nomas studio generates continuous worlds with text, image starts and recorded camera motion, rollback, saved state and MP4 exports.

Video Compression

videosmaller

Video to GIF

2-video-to-gif-avif.mp4

because who says ffmpeg is fine in cli?

Background Removal (Locally Hosted) -> AI Perfect SVG

view what a combined workflow looks like:

2-ideogram+img2svg-compact.1.mp4

Redesign's asset-separation (qwen layered):

2-redesign.mp4

Music Tools (generation, singing, orchestration, OMR, and transcription)

2-muscriptor.mp4

musvit

Update 8-20, added minimax music custom stdio

(uses minimized, stripped comfyUI pipeline directly, but custom WebUI)

minimax.mp4

Have a 5090?

startupsh

Run all at once.

The install sequence is simple for each project. Requiring WSL if on Windows.

Jump into a project folder, run its setup once, and start it. Setup creates the ignored .env and isolated .venv automatically.

cd ~/multimedia/PROJECT
./setupwithuv.sh
./startwithuv.sh

Vox - Native Audio/Translate Routing for this repo.

See other projects by me!

Agent Infrastructure Projects:

https://github.com/CommanderTurtle/persephone just my integration of own internal config + continuity with omp. gpu & local-only for just about everything so far

  • Diogenes - hosting (linux, service manager), maintained fork of Odysseus for backends and control center — this can install and integrate the following if not already installed:

  • sandwich - node package management/auditing for all these package.jsons

  • libriarian - wikifier (makes wikis, dreaming agents)

  • retrieval - skill labrador that fetches necessary skills from archive on disk, rather than bloating agent prompt with full list

  • persephone - adds omp (pi agent) gateway layer

  • leetcoder - allows hermes to puppeteer pi’s for small coding tasks (delegation)

Lots of ideas from stablyai/orca, which does all these things, and where i got a lot of inspiration for the ecosystem

Still WIP. Doing lots of editing as I continue development. Especially with Diogenes automatically setting up my config upon install

Web-Development:

  • orc - a multisite/multiframework F# engine for modular websites in py, ts, c#, js, and ruby.
  • reactor - a lightweight rust-poller that allows for a live fsharp repo in 7+ languages. Side-hosts vite dev server, zensical serve, jekyll serve, netdocs serving (C#), and more with preview
  • tools - just some lightweight tools that help making maintaining websites easy in modular languages

Libraries:

Apps:

extra

For Python projects, choose the setup entrypoint in that project only. No root setup wrapper exists.

Ideogram / ObjectClear

Local Ideogram 4 generation source plus the fuzzy-mask editing browser: ObjectClear, BiRefNet, OpenCV, and optional private-Comfy Ideogram masked editing.

cd ~/multimedia/ideogram
./setupwithuv.sh
./startwithuv.sh                             # http://127.0.0.1:8174

The standalone Ideogram generator is local_generate.py; add src to PYTHONPATH or install the local package when using that lane. The editor loads Ideogram only when explicitly selected. Its private engine reuses bundled Comfy code (not the Minimax service), supports local caption drafting and configurable crop processing, and preserves the original image outside the feathered mask. See Ideogram architecture and setup.

img2svg

Native VTracer-based raster-to-SVG CLI and browser studio.

cd ~/multimedia/img2svg
./setupwithrust
./startwithrust                           # http://127.0.0.1:417

LongCat

Local LongCat voice synthesis and cloning. The HTTP service and browser workbench own independent model lifecycles.

cd ~/multimedia/vocals/tts/longcat
./setupwithuv.sh
./starthttp.sh                            # machine API: :8230
./startwithuv.sh                          # browser UI: :8231

ACE-Step

Native ACE-Step 1.5 music generation, editing, repainting, extension, and audio-to-audio controls.

cd ~/multimedia/music/acestep
./setupwithuv.sh
./startwithuv.sh                          # http://127.0.0.1:8250

The browser process preloads the native XL-SFT DiT and 4B 5 Hz language model: the quality-first ACE combination for a 24 GB+ workstation. It reads the shared native ACE-Step model tree directly and releases all weights on Ctrl+C.

MiniMax Music 3

Standalone MiniMax Music 3 text-to-song production desk using the official graph: full FP16 DiT, pruned INT8 text encoder, DAV decoder, real CFG/top-k/sampler controls, and an animated lyrics-and-spectrum performance view. It does not require a separately running ComfyUI.

cd ~/multimedia/music/minimax
./setupwithuv.sh
./startwithuv.sh                          # http://127.0.0.1:8254

The browser owns a private loopback song engine. Its separate Prompt Guide tab uses the bundled Comfy CLIPLoader(type=krea2) → TextGenerate path with the local SergiusFlavius/Qwen3-VL-4B-Instruct-heretic-NVFP4 checkpoint, but begins fully off and unloaded; enabling it starts a second loopback-only engine. A Guided Brief Lab adds one-click recipes plus expandable genre, instrumentation, key/mode, BPM, meter, groove, harmony, vocal, form, production, and listening-context vocabulary. It creates plain-English copypasta without invoking a model or silently changing controls. Guide results remain detached, with manual copy controls for MiniMax's Global Metadata, Vocal Details, Arrangement, and tuning notes. Load models and Unload control song-model residency explicitly, while Ctrl+C unloads all weights and closes the web app plus both private engines. Links in the studio hand covers/editing to ACE-Step, loops to Foundation-1, and audio reverse engineering to MuScriptor.

Stable Audio Foundation

Foundation-1 text-to-audio with the enhanced Stable Audio Gradio controls, local T5 conditioning, MIDI analysis, and optional TorchAO controls.

cd ~/multimedia/music/stableaudio
./setupwithuv.sh
./startwithuv.sh                          # http://127.0.0.1:8251

SymphonyGen

Two-stage symbolic composition: generate harmony MIDI, then orchestrate a MIDI sketch with baseline or GRPO checkpoints.

cd ~/multimedia/music/symphony
./setupwithuv.sh
./startwithuv.sh                          # http://127.0.0.1:8252

The web process is model-free while idle. Each generation request owns and releases its model in a short-lived worker.

VocalRender

Prompt-conditioned local singing synthesis with the highest-quality Pro weights only, explicit load/unload controls, browser playback, and WAV downloads.

cd ~/multimedia/vocals/singing/vocalrender
./setupwithuv.sh
./startwithuv.sh                          # http://127.0.0.1:8253

MuScriptor

Audio-to-MIDI/score transcription with a browser piano roll.

cd ~/multimedia/music/muscriptor
./setupwithuv.sh
./startwithuv.sh                             # http://127.0.0.1:8222

Single-file CLI:

./startwithuv.sh INPUT.mp3

MuSViT

Sheet-music image/PDF encoding and SMT-backed beKern/MIDI/SVG conversion.

cd ~/multimedia/music/musvit
./setupwithuv.sh
./startwithuv.sh                             # browser studio: :8223
./startwithuv.sh SCORE_IMAGE_OR_PDF          # direct CLI

Use musvit_embed.py directly for the embedding lane.

Unlimited OCR 2

Unlimited OCR 2 studio parses ordered PDF/image pages into a structured document with layout overlays, tables, equations and figure crops. Select unlimited-ocr2 in the downloader; serve on :8287.

Pixel Wizard

Pixel Wizard uses WeVisOne-VL-4B for grounding, detection, counting, pointing, OCR, captions, segmentation and alpha matting. Select pixel-wizard in the downloader; serve on :8288.

ReDesign

Flat-image decomposition into editable layers using local detection, segmentation, inpainting, OCR, and Qwen Image Layered components.

cd ~/multimedia/visual-modeling/redesign
./setupwithuv.sh
./startwithuv.sh                             # http://127.0.0.1:8173

Review .env before launch. Native Diffusers is the primary image lane; an existing private ComfyUI listener is optional.

Translate

Local EraX text translation, language classification, and a shared INT4 multimodal lane for image explanation, OCR translation, custom visual prompts, and spoken-language arbitration.

cd ~/multimedia/translate
./setupwithuv.sh
./starthttp.sh                            # http://127.0.0.1:8176
./startwithuv.sh                          # standalone browser UI: :8177

The setup prompt accepts GPU or CPU; TRANSLATE_ACCELERATOR remains available for noninteractive runs. The browser UI owns an independent model lifecycle. Its explicit external-load button can attach to an already-loaded starthttp.sh instance, but it never falls back between them automatically.

Video Compact

Private VideoSmaller-style FFmpeg compression.

cd ~/multimedia/video-compact
./setupwithuv.sh
./startwithuv.sh                             # http://127.0.0.1:8240

Video to GIF/AVIF

Local trim/crop/resize and animated GIF or AVIF conversion.

cd ~/multimedia/video-compact/gif-avif
./setupwithuv.sh
./startwithuv.sh                             # http://127.0.0.1:8241

The host FFmpeg build must expose the requested GIF/AVIF encoder and AVIF muxer.

Nemotron

Local Nemotron 3.5 transcription, speaker timelines, karaoke, subtitles and audio questions.

uv run models/download_models.py nemotron
cd vocals/stt/nemotron
./setupwithuv.sh
./startwithuv.sh                             # http://127.0.0.1:8289

Native NVFP4/FP8 on Blackwell; microphone, media files and explicit URL imports, word seeking, CPU vocal separation, saved conversations and timed exports.

Audio8

Local Realistic Air speech, reference voices and paragraph takes.

uv run models/download_models.py audio8
cd vocals/tts/audio8
./setupwithuv.sh
./startwithuv.sh                             # http://127.0.0.1:8290

Native BF16 and 44.1 kHz codec, microphone/file references, waveform seeking, saved sessions, WAV/FLAC and codec-token exports.

Ming3

Local voice design, cloning, multilingual delivery and multi-speaker podcasts.

uv run models/download_models.py ming3-0.5b  # dense, ~3 GB
uv run models/download_models.py ming3-tiny  # optional MoE + podcast, ~18 GB
cd vocals/tts/ming3
./setupwithuv.sh
./startwithuv.sh                             # http://127.0.0.1:8291

Separate native BF16 environments, reference recording, director controls, vocal tags, saved takes, waveform seeking and WAV/FLAC exports.

CrisperWhisper

Local CrisperWhisper 2.0 transcription in normalized or literal mode. The HTTP service and browser workbench use separate ports and model lifecycles.

cd ~/multimedia/vocals/stt/whisper
./setupwithuv.sh
./starthttp.sh                            # machine API: :8172
./startwithuv.sh                          # browser UI: :8173

ReDesign also defaults to 8173; change CW2_UI_PORT when both browser services run together.

Underdog Bark

English transcription with independent Bark 3.0, Plus, Clean and Pro loaders. Whisper-style studio, microphone recording, transcript comparison and TXT/JSON export.

uv run models/download_models.py underdog-pro  # also: underdog-base, underdog-plus, underdog-clean
cd vocals/stt/underdog
./setupwithuv.sh
./startwithuv.sh                           # studio: :8293
./starthttp.sh                             # optional API: :8292, BARK_MODEL in .env

Port map

Port Runtime
417 img2svg
8172 CrisperWhisper HTTP
8173 CrisperWhisper UI or ReDesign
8174 Ideogram/ObjectClear editor
8176 Translate
8177 Translate standalone UI
8221 MotionBricks character studio
8222 MuScriptor
8223 MuSViT local score studio
8230 LongCat HTTP
8231 LongCat UI
8240 Video Compact
8241 Video to GIF/AVIF
8250 ACE-Step 1.5
8251 Stable Audio Foundation-1
8252 SymphonyGen local studio
8253 VocalRender local studio
8254 MiniMax Music 3 local studio
8281 FlowHMR reference-to-motion studio
8282 PhysAg3D staged asset studio
8283 Idol Gaussian studio
8284 Hy3D mesh studio
8285 SceneBloom scene studio
8286 Hy-Nomas world studio
8287 Unlimited OCR 2 document studio
8288 Pixel Wizard vision studio
8289 Nemotron transcription studio
8290 Audio8 speech studio
8291 Ming3 speech and podcast studio
8292 Underdog Bark HTTP
8293 Underdog Bark studio

The new studios default to loopback; existing services retain their configured private-LAN interfaces. Keep network-exposed services behind the host firewall or set the supported bearer token where provided.

Update 9-17, new, local creative studios

The current expansion uses one shared, dependency-light product surface with project-native adapters. Every studio has responsive dark/light themes, searchable task cards, complete native controls, presets and drafts, raw streaming uploads, a persistent single-GPU queue, cancellation, SSE progress, model load/unload health, an output library, side-by-side comparison, and API examples. There are no hosted assets, analytics, CORS opt-ins, or automatic model downloads at runtime.

Project Canonical surface Default port VRAM (approx)
AuK 16 speech generation, cloning, editing, emotion, timing, and nonverbal workflows; Vox-compatible speech route 8260 ~10 GB
4DAnyone monocular 4D human reconstruction and reusable result inspection 8261 ~14 GB
TRELLIS.2 + Pixal3D image/multiview sculpting, local refinement, and interactive GLB inspection 8262 ~20 GB
WorldSculpt complete multi-object scene reconstruction and scene inspection 8263 ~18 GB
ID-V2V identity-preserving transfer, normal/depth guidance, and reusable preprocessing 8265 ~22 GB
ARDY + SOMA-X text-to-motion, motion inspection, parametric bodies, and hands 8266 ~8 GB
LingBot World V2 1.3B single-5090 world generation and visual camera-path authoring 8267 32 GB
Fire3D RGB/RGBD scene reconstruction, native-release evaluation, and GLB inspection 8268 ~20 GB
YuE2 composition, planning, continuation/cover, score realization/editing, transcription, resume, and VAE decode 8270 ~24 GB

Update 11-12, more local creative studios

This expansion folds thirteen additional runtimes into the same shared, dependency-light product surface with project-native adapters: motion repair and video-to-character animation, geometry, scene, and world generation, optical character recognition and natural-language image editing, plus speech synthesis, transcription, and podcast generation. Every studio keeps responsive dark/light themes, searchable task cards, complete native controls, presets and drafts, raw streaming uploads, the persistent single-GPU queue, cancellation, SSE progress, model load/unload health, an output library, side-by-side comparison, and API examples. Pinned upstream source snapshots now ride in-tree, so a fresh checkout restores any runtime with dependency installation alone; only checkpoint bytes route through the shared downloader.

Project Canonical surface Default port VRAM (approx)
Audio8 speech synthesis 8179 ~8 GB
MotionBricks motion repair, inpainting, outpainting, and trajectory steering 8272 ~10 GB
FlowHMR video and reference to animated character meshes 8273 ~12 GB
PhysAg3D physics-grounded 3D human scenes 8275 ~16 GB
Idol video to 3D character generation and inspection 8276 ~22 GB
Hy3D image and video to high-fidelity 3D geometry 8277 ~22 GB
SceneBloom procedural 3D scene generation with Blender-MCP refinement 8278 ~12 GB
Hy-Nomas interactive generated worlds with first-person play 8279 ~28 GB
Unlimited OCR 2 document and scene text extraction 8280 ~12 GB
Pixel Wizard natural-language image editing 8281 ~24 GB
Nemotron long-form transcription 8282 ~6 GB
Ming3 speech synthesis and podcast generation 8291 ~16 GB
Underdog Bark speech behind plain HTTP and studio surfaces 8292/8293 ~6 GB

More?

Trellis & Pixal3D - 3d modeling/scuplting pipelines

sculpting-1-compact.mp4

YuE2

The only Yue2 frontend with stem splitting!

yuesplitter

Live Viewer Exporting

  • (Share with Friends via app.shel.sh/make & a.shel.sh superlinks, entirely local)

Another:

https://a.shel.sh/#media:rZ9jBas~g*mK]Q-()SB$IFDo]Xl-b5y[P,/@f/FwenhdLmh~Aaw

About

Multimedia Frontend Wrappers for AI tooling

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages