Add Stable Fast 3D example (native MLX single-image-to-3D-mesh) - #1444
Open
bahaehmimdi wants to merge 1 commit into
Open
Add Stable Fast 3D example (native MLX single-image-to-3D-mesh)#1444bahaehmimdi wants to merge 1 commit into
bahaehmimdi wants to merge 1 commit into
Conversation
Native MLX port of Stability AI's stable-fast-3d: camera embedder, DINOv2 tokenizer with AdaLN modulation, two-stream transformer backbone, post-processor, and triplane decoder, plus MLX/numpy mesh finishing (marching-tetrahedra isosurface, UV unwrap, texture baking). No PyTorch. Weights and the fixed tetrahedra grid are fetched at runtime (from Hugging Face and the upstream repo respectively), not redistributed here. Note: this folder carries its own LICENSE (Stability AI Community License) as an explicit exception to the repo-wide MIT default, per NOTICE.md, since the underlying model's license is non-sublicensable. Flagged in the tracking issue (ml-explore#1443) before this PR for maintainer visibility. Validated end-to-end against a live run: ~9.9s neural pass + mesh finishing on an M1 Pro, producing a valid textured GLB (11422 verts, 17520 faces) from a test image. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1443.
Adds a
stable_fast_3d/example: Stable Fast 3D (single-image -> textured 3D mesh), reimplemented natively in MLX. No PyTorch anywhere in the pipeline.Why
The official PyTorch implementation is slow on Apple Silicon via MPS (~7-9 min/generation on an M1 Pro, full model reload every call, no Mac-specific optimization). This port brings that down to ~12.4s end-to-end (~34-44x faster), with both the neural forward pass and mesh finishing (UV unwrap, texture baking, previously PyTorch/CPU) running in 100% MLX/numpy. See
stable_fast_3d/README.mdfor the full speed table and validation methodology.This is the one thing I want explicit maintainer sign-off on.
stable_fast_3d/is a Derivative Work of Stability AI'sstable-fast-3dsource (module-by-module hand-port, not an independent reimplementation from the paper alone), so it's covered by the Stability AI Community License, which is non-sublicensable. I can't relicense it under this repo's root MIT license perCONTRIBUTING.md.So this folder carries its own
LICENSE(Stability AI Community License text) and aNOTICE.mdexplaining the exception, exactly as flagged in #1443 before opening this PR. No model weights are redistributed —weights.pydownloads and converts the checkpoint from the gatedstabilityai/stable-fast-3dHF repo at runtime (same gated-access pattern as the official repo), and downloads one small fixed data file (160_tets.npz, a geometric grid definition, not a trained weight) from the upstream GitHub repo.If a per-folder license exception isn't something you're willing to carry in this repo, totally understand — happy to close this and just leave the standalone repo linked from #1443 instead.
What's included
stable_fast_3d/stable_fast_3d/: camera embedder, DINOv2 tokenizer + AdaLN modulation, two-stream transformer backbone, post-processor, triplane decoder (camera_embedder.py,dinov2.py,backbone.py,post_processor.py,decoder.py,bicubic.py), plus mesh finishing (isosurface.py,uv_unwrap.py,texture_baker.py), tied together inpipeline.py'sStableFast3Dclass.image_to_3d.py: CLI entry point, mirrors the style of other examples here (e.g.stable_diffusion/txt2image.py).black+isort --profile=blackperCONTRIBUTING.md.Testing
Ran the actual package end-to-end (not just reviewed the diff): loaded the real checkpoint, generated a mesh from a test image, produced a valid textured GLB (11,422 verts / 17,520 faces) in 9.9s on an M1 Pro. Each neural module was separately validated against the original PyTorch model's intermediate tensors to float32 tolerance during development (max-abs-diff < 1e-4 throughout) — full validation scripts and history in the standalone repo linked in
NOTICE.md, kept out of this PR to keep it focused on the example itself.Related: standalone development repo at https://github.com/bahaehmimdi/stable-fast-3d-mlx, upstream issue at Stability-AI/stable-fast-3d#89.