EdenLive: Real-Time Generative Rendering from Any Visual Reference
Project Page · Code · Models
Zehuan Huang1,*, Tianxing Xu1,*, Chunli Peng1,*, Cheng Zhang1,*, Zeyi Zheng1,†, Yifei Song1,†, Zekai Gu1,†, Zhipeng Yu1, Peng Wang1,‡, Yanpei Cao1,✉
* Equal contribution · † Intern at VAST · ‡ Project lead · ✉ Corresponding author
1VAST · Project Eden
TL;DR:
- For interactive world models, we decouple state evolution (maintained by coding agents or engines) and appearance (rendered by EdenLive).
- EdenLive renders reference videos to enhanced videos in real time at 24 fps or higher, even on a consumer GPU (RTX 5090).
edenlive_demo_A_audio_720p.mp4
EdenLive is a real-time generative renderer that treats the reference video as the interface between an interactive program and neural visual synthesis: the upstream program executes the rules and maintains the world state, whereas EdenLive renders the resulting reference video, guided by a target first frame and a text prompt, into detailed frames.
The project page demonstrates over 24 FPS, minute-scale generation, and 480P/720P output. The 5B model supports single-GPU deployment at 720P.
We have open-sourced:
- 5B and 14B model weights, for both bidirectional and few-step (chunk-AR) inference.
- Real-time streaming inference code.
-
game-prototype-turboskill that turns a text prompt into a coarse playable game. - A frontend to try the full prompt-to-game pipeline.
One Python environment runs both the offline inference in edenlive/ and the
real-time service in realtime/.
| Verified version | |
|---|---|
| OS / GPU | Linux x86-64, NVIDIA GPUs with 80 GB (CUDA 13 driver) |
| Python | 3.10 |
| PyTorch | 2.14.0.dev20260717+cu130 nightly (realtime/requirements.txt pins 2.13.0, cu130), torchvision 0.28.0 |
| diffusers / transformers / accelerate | 0.39.0 / 5.12.1 / 1.0.1 |
| decord / pandas / pyarrow | 0.6.0 / 2.3.3 / 25.0.1 |
| imageio / imageio-ffmpeg | 2.37.4 / 0.6.0 |
| Node.js | 22 (only to fetch three.js for the game renders) |
git clone https://github.com/VAST-AI-Research/EdenLive.git EdenLive
cd EdenLive
conda create -n edenlive python=3.10 -y
conda activate edenlive
# 1. PyTorch for a CUDA 13 driver (use .../whl/cu128 for CUDA 12.8).
pip install torch==2.13.0 torchvision==0.28.0 --index-url https://download.pytorch.org/whl/cu130
# 2. The real-time service and its pinned dependencies.
cd realtime
pip install --no-deps -e ./flashdreams -e ./apps/cam2v -e ./integrations_v2/eden
pip install -r requirements.txt
python -m playwright install chromium
npm install --prefix games
cd ..
# 3. Offline inference and video I/O dependencies.
pip install av==17.1.0 decord==0.6.0 pandas==2.3.3 pyarrow==25.0.1 imageio==2.37.4 imageio-ffmpeg==0.6.0
pip install --no-deps -e .The default 5B bidirectional profile uses FA4 temporal text
attention and requires SM90, BF16, flash-attn-4==4.0.0b31, and
nvidia-cutlass-dsl==4.7.1. Use --no-deps when installing kernels into an
existing environment to preserve PyTorch. See docs/zh/wan22-fa4.md.
hf download VAST-AI/EdenLive --local-dir ./models/EdenLiveEverything loads from one directory, models/EdenLive under the repository root by
default (about 210 GB in total).
models/EdenLive/
dit-14b/ diffusion_pytorch_model.safetensors, config.json 14B few-step (chunk-AR) DiT
dit-14b-bidir/ diffusion_pytorch_model.safetensors, config.json 14B bidirectional DiT
dit-5b/ diffusion_pytorch_model.safetensors, config.json 5B few-step (chunk-AR) DiT
dit-5b-bidir/ diffusion_pytorch_model.safetensors, config.json 5B bidirectional DiT
vae-14b/ Wan2.1_VAE.pth, config.json
vae-5b/ Wan2.2_VAE.pth, config.json
decoder-14b/ lighttaew2_1.pth TAEHV decoder (real-time only)
decoder-5b/ lighttaew2_2.pth TAEHV decoder (real-time only)
scheduler-14b/ scheduler_config.json offline only
scheduler-5b/ scheduler_config.json offline only
umt5/ text_encoder/, tokenizer/ UMT5 text encoder
clip/ image_encoder/, image_processor/ CLIP, 14B I2V only
vision-encoder/ backbone/, projection.safetensors proxy-DINO encoder
flux-c2f/ Diffusers FLUX tree + c2f_lora.safetensors workbench first frame (real-time only)
The default weight directory is models/EdenLive, relative to the working directory.
To specify another directory:
python -m edenlive.scripts.infer --config ... model.pretrained_dir=/data/EdenLiveThe real-time service reads the same tree through MODEL_ROOT in
realtime/deploy.conf.
Two proxy recordings ship with the repo under examples/samples/, with a manifest that
points at them:
examples/samples/
manifest.csv id, prompt, proxy_video, first_frame
cryo-crossing/ cover.png, first-frame.png, prompts.json, prompts.txt, recording.mp4
pulse-skirmish/ same five files
recording.mp4 is the greybox proxy (1280x704), and first-frame.png is its first frame.
cover.png is a photoreal render of that frame; the manifest's first_frame points to this file.
prompts.json contains staticPrompt and timestamped dynamicPrompts;
prompts.txt is the corresponding text version.
| Mode | Wan2.1-I2V-14B @ 480p | Wan2.2-TI2V-5B @ 720p |
|---|---|---|
| Bidirectional | wan21_14b_480p_bidir.yaml |
wan22_5b_720p_bidir.yaml |
| Few-step (chunk-AR) | wan21_14b_480p_few_step.yaml |
wan22_5b_720p_few_step.yaml |
Bidirectional inference: all frames are denoised jointly.
torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
--config edenlive/configs/wan21_14b_480p_bidir.yaml \
runtime.sequence_parallel.size=2python -m edenlive.scripts.infer \
--config edenlive/configs/wan22_5b_720p_bidir.yamlFew-step inference: chunk-autoregressive sampling, 4 steps per chunk.
torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
--config edenlive/configs/wan21_14b_480p_few_step.yaml \
runtime.sequence_parallel.size=2python -m edenlive.scripts.infer \
--config edenlive/configs/wan22_5b_720p_few_step.yamlReal-time service: uses game frames as input and streams video over WebRTC.
Configure the weight directory, service addresses, ports, and GPU count in
realtime/deploy.conf.
5B uses one GPU by default. Set NPROC for multi-GPU inference.
Set TURN_HOST (the relay address reachable by the browser), TURN_IFACE
(the relay interface), and TURN_SECRET (your own password) in
realtime/deploy.conf. To run existing games, execute on the GPU host:
cd realtime
apt-get install coturn # First-time installation
bash turn_server.sh # Start TURN first
bash start.sh 14b # Model service; or use 5bThe workbench — a game library, a chat that builds new games from a text description,
and a play window — needs all four services. Its first-frame renderer (the FLUX worker)
runs FLUX.2 [klein] 9B under the FLUX Non-Commercial License, so the workbench is
for non-commercial use only, and that license requires content filtering on the
images it generates (see License). Playing registered games with start.sh
does not load FLUX.
The bridge host requires the claude CLI, claude auth login, and first-run trust confirmation.
Game generation uses --dangerously-skip-permissions.
For separate hosts, copy the bridge's .workbench/token to the GPU host's realtime/
directory and set BRIDGE_HOST; leave it empty for a single-host deployment.
cd realtime
# 1. TURN relay, GPU host; skip if already running
bash turn_server.sh
# 2. Workbench bridge, Claude CLI host
bash workbench_bridge.sh
# 3. FLUX first-frame service, GPU host
bash workbench_flux.sh
# 4. Model service, GPU host
bash launch.sh
# Stop services on this host
bash stop.shOpen http://<host>:8089 (APP_PORT in deploy.conf).
| Model | Application slug |
|---|---|
| 5B | game2v-eden-world-v2-5b-taehv |
| 14B | game2v-eden-world-v2-14b-taehv |
The real-time service uses the TAEHV decoder. See realtime/README.md
for the service topology and developer documentation.
The default profiles read a recording manifest (CSV/JSONL/JSON/parquet), one
recording per request. Each directory contains recording.mp4, cover.png,
and prompts.json; text comes from staticPrompt and dynamicPrompts:
id,proxy_video,first_frame
shot_001,shot_001/recording.mp4,shot_001/cover.pngpython -m edenlive.scripts.infer --config CONFIG.yaml \
data.dataset.params.path=/data/my_manifest.csv \
data.dataset.params.root=/dataLightX2V provides accelerated inference for EdenLive's 14B and 5B models in both bidirectional and four-step autoregressive modes. Optimizations include low-bit quantization, efficient kernels and operator fusion, KV cache optimizations, torch.compile, CUDA Graphs, and streaming LightTAE decoding.
Real-time inference on a single RTX 5090: EdenLive 5B achieves approximately 24.7 FPS inference throughput at 1280×704, meeting the throughput requirement for real-time 24 FPS playback.
See the EdenLive deployment guide for environment setup, model conversion, quantization, and launch commands.
@misc{edenlive2026,
title = {EdenLive: Real-Time Generative Rendering from Any Visual Reference},
author = {Zehuan Huang and Tianxing Xu and Chunli Peng and Cheng Zhang and Zeyi Zheng and Yifei Song and Zekai Gu and Zhipeng Yu and Peng Wang and Yanpei Cao},
year = {2026},
url = {https://vast-eden.ai/?p=edenlive}
}Built on Wan2.1 / Wan2.2, 🤗 Diffusers, DINOv3, FLUX.2 [klein] by Black Forest Labs, and the LightX2V LightTAE decoders. The game renders use Three.js, and the workbench's game-generation skill builds on GameBlocks.
The real-time streaming service under realtime/ builds on
NVIDIA FlashDreams.
First-party code is licensed under Apache 2.0 — see LICENSE and NOTICE.
realtime/ is also Apache 2.0 and keeps its own LICENSE,
NOTICE and THIRD-PARTY-NOTICES.
The vendored html-to-image bundle and GameBlocks-derived skill modules retain
their MIT licenses and original copyright notices; per-file scope is recorded
in REUSE.toml.
The model weights are distributed separately and are not all Apache 2.0:
| Component | License |
|---|---|
dit-*, vae-*, umt5/, scheduler-*, decoder-* |
Apache 2.0 |
clip/ (LAION CLIP ViT-H/14 / OpenCLIP) |
MIT |
vision-encoder/ (DINOv3 and derivatives) |
DINOv3 License |
flux-c2f/ (FLUX.2 [klein] 9B plus our LoRA; workbench first frame only) |
FLUX Non-Commercial License — non-commercial, non-production use only |
Each component ships its own license and notice files alongside the weights; see the model repository's README for details.