🗺️ StoryMapDiffusion

Turn city data into AI-generated maps you can explore.

3D Rendering · Dataset Preparation · LoRA Training · Deep Zoom · AI Curator

Demo · Setup · Dashboard · Deployment

StoryMapDiffusion aims to make it easy to create maps of real places in a variety of visual styles. Choose a geographic area, use or train a LoRA for your preferred style, and generate a map you can pan, zoom, and explore, with AI-assisted stories tied to its landmarks. The dashboard brings data collection, 3D rendering, dataset preparation, image generation, and map export into one workflow.

Choose your area and visual style. The workflow supports different geographic areas with suitable 3D coverage and compatible LoRAs. For demonstration, the examples below use part of Tokyo, including Tokyo Tower and the surrounding neighborhoods.

Explore the demo below, then set up the dashboard to create your own map.

✨ Explore the demo

🗺️ Open the interactive demo — drag to pan, scroll to zoom, and explore the map in your browser.

Demo inference: the map contains 3,531 AI-generated tiles (14,124 quadrants), produced on NVIDIA A100 80GB GPUs rented through Vast.ai and Modal, with the full pipeline on the GPU, 4-step Lightning + LoRA fusion/compilation, and a configured batch size of 8. Total runtime: 16.5 hours. Total cost: US$39.74.

Explore the map	Discover its stories
Deep Zoom · pan, zoom, minimap, fullscreen	AI Curator · landmark pins, information cards, Explore

🎬 Map walkthroughs

From a map overview to street-level detail

Select a landmark and open Explore

🚀 Setup and launch

Requires Node.js ≥ 22.12 and Python ≥ 3.10. The dashboard does not need a local GPU; training and inference can run on Modal or a separate GPU server. Large map datasets are not included in a clone, so a fresh installation will not contain the demo map.

1. Set up the environment

cd StoryMapDiffusion
npm install
python -m venv .venv

If your system uses python3, substitute it for python. Activate the environment once in your current terminal:

Terminal	Command
Linux / macOS (Bash, Zsh)	source .venv/bin/activate
Windows PowerShell	.\.venv\Scripts\Activate.ps1
Windows Command Prompt	.venv\Scripts\activate.bat

After activation, the remaining commands work across operating systems:

python -m pip install -e ".[dzi,curator]" "pyvips[binary]"

pyvips[binary] installs pyvips and a bundled libvips library, usually without a separate download or PATH changes on supported Windows, Linux, and macOS systems. If no compatible binary is available, see the alternative libvips installation options.

2. Create .env and enable the features you need

Copy .env.example to .env in the repository root using your file manager or IDE. Keep your existing file if you have already configured it. Fill in only the settings you need:

Task	Requirements
View an existing map	No API key or GPU required
Capture a new area	CESIUM_ION_TOKEN for the default Cesium Ion source, or GOOGLE_KEY when using the Google provider
Train / deploy on Modal	Run the two authentication commands below once
Use an existing GPU server	INFERENCE_ENDPOINT=http://127.0.0.1:8000 or your server URL
Generate Curator text with an LLM	LLM_API_KEY, LLM_MODEL_NAME, and LLM_BASE_URL from the same provider

For Modal:

python -m pip install modal
python -m modal token new

Choose the GPU, LoRA, and execution mode in the dashboard when training or deploying. To run your own server on RunPod/Vast.ai, see the external GPU guide.

3. Open the dashboard

node --env-file=.env scripts/dashboard.mjs

Open http://localhost:3000. Keep the terminal running. Restart the dashboard after editing .env to load the new configuration.

Already have a map? Put model_map.dzi and its model_map_files/ directory in output/, then open Map viewer → Available maps. Keep the accompanying plan and .curator.json files if available. Viewing a map does not require rerunning capture, training, or inference.

🧩 Create a map with the dashboard

Select an area → Capture → Review → Choose/train a LoRA → Run inference → Build → Explore. If you already have a suitable LoRA, skip dataset preparation and training. The illustrated UI walkthrough explains each control for first-time users.

1. Select an area and capture it

Prepare a .geojson file containing the area boundary as a Polygon or MultiPolygon, using WGS84 coordinates ([longitude, latitude]). Draw an area on geojson.io, obtain OpenStreetMap boundaries through Overpass Turbo, or download them from a local open-data portal. Put the file in districts/ and keep only the areas you want to process. GeoJSON defines the boundary; the area still needs 3D coverage from your configured provider.

Review Capture settings, then click Regenerate Grid.

Open Capture controls → Start capture and monitor the grid and logs. Use Retry failed tiles for failed captures.

Next, open Render review → Rescan, inspect flagged images, and delete confirmed faulty renders so they can be captured again.

2. Choose a style or train your own LoRA

Using an existing LoRA: open Map inference → Cloud LoRA → Refresh checkpoints to select weights on Modal. For local files, open Upload local LoRA, select the .safetensors file and JSON metadata, then click Push to Modal Volume.

Training your own style: open Dataset, upload style images whose filenames contain coordinates matching the source renders, check Matched Pairs, choose a prompt, and click Generate Dataset.

In LoRA Training, choose the GPU, epochs, learning rate, and rank, then click Start training. Checkpoints are saved automatically; Download weights downloads the final adapter to your machine.

3. Generate images and inspect the results

In Map inference, select the GPU/LoRA and click Deploy on Modal, or enter your own server endpoint. Try one tile with Spot-Check, then choose Entire map / Custom bounds → Start Batch Inference. Inspect compares a source image with an existing result; Stop Pipeline stops the current run.

Inference option	When to use it
Lightning	Generate images in 4 steps; turn off for standard mode
Fuse LoRA + compile	Optimize longer runs; the first request needs warm-up
GPU microbatch	Start small and increase when VRAM allows; separate from images per request
Offload / VAE tiling	Reduce VRAM pressure, potentially at the cost of speed

Redeploy after changing the LoRA or execution mode. For a custom endpoint, configure these settings on the GPU server; see the mode and environment variable reference.

4. Build and open the map

Open Map viewer → Build generated map to assemble AI outputs, or Build rendered map to view the original 3D renders. Choose a map under Available maps, drag to pan, scroll to zoom, and enable Grid to inspect tile coordinates. Rebuild after generating or editing tiles to update the map.

5. Explore with AI Curator

Click the sparkle icon in the viewer, select a green pin to read about a landmark, then visit another suggested place.

When an Explore invitation appears, open it to read about the connection; click Return to continue exploring.

Maps created through the full workflow include the coordinate, camera, and layout metadata Curator needs. Open the map and enable AI Curator; the dashboard starts the local service and uses that metadata to position pins, without manual coordinate configuration.

🖥️ Use your own cloud GPU: Vast.ai, RunPod, and more

Run the inference server on a rented GPU machine, then connect from the dashboard or CLI through an SSH tunnel. Source renders and generated results stay on your local machine; the GPU server runs the model and stores the LoRA. A Modal deployment is not required.

Set up the GPU server and connect your local client

1. Set up SSH — local machine and provider web terminal

Rent a Linux instance with an NVIDIA GPU, CUDA, and Python 3.10+. Get the user, IP address, and SSH port from the provider's connection page; replace GPU_HOST, SSH_PORT, and root below with the actual values. Choose a direct TCP SSH connection that supports SCP and port forwarding.

On your local machine, create an SSH key if you do not already have one, then copy the public key:

ssh-keygen -t ed25519 -C "isometric-map-gpu"
cat ~/.ssh/id_ed25519.pub

Add the public key to the provider's SSH configuration. To add it to a running instance, open the GPU instance's web terminal and run the following, replacing YOUR_PUBLIC_KEY with your actual key:

mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "ssh-ed25519 YOUR_PUBLIC_KEY isometric-map-gpu" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
cat ~/.ssh/authorized_keys

Connect from your local machine:

ssh -p SSH_PORT -i ~/.ssh/id_ed25519 root@GPU_HOST

This workflow uses SSH keys and does not require enabling password login. If you are prompted for a password, check your key and connection details in the detailed SSH guide.

2. Install the environment — GPU server

In the SSH terminal on the GPU server, clone the repository if needed:

mkdir -p /workspace
cd /workspace
git clone https://github.com/CallmeTruong/StoryMapDiffusion.git
cd /workspace/StoryMapDiffusion
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cloud.txt
python -m pip install -e .
nvidia-smi
python -c "import torch; print('CUDA:', torch.cuda.is_available())"
mkdir -p lora

Keep the server and client on the same repository revision. If your directory is /StoryMapDiffusion, update the commands and LORA_PATH accordingly. If your system uses python3, use that command to create the environment.

3. Upload the LoRA — local machine

Open another local terminal in the repository root:

scp -P SSH_PORT -i ~/.ssh/id_ed25519 output/models/pytorch_lora_weights.safetensors root@GPU_HOST:/workspace/StoryMapDiffusion/lora/

SCP uses uppercase -P; SSH uses lowercase -p. Keep the filename and set LORA_WEIGHT_NAME to match it. Renaming it to diffusion_pytorch_lora.safetensors is unnecessary.

4. Configure and start the server — GPU server

Create .env in the repository root on the GPU server. This example targets an 80 GB GPU. If .env already exists, edit the relevant lines instead: cat > .env overwrites the file.

cd /workspace/StoryMapDiffusion
cat > .env << 'EOF'
BASE_MODEL=Qwen/Qwen-Image-Edit-2511
LIGHTNING_ENABLED=1
INFERENCE_STEPS=4
INFERENCE_GUIDANCE=1.0
INFERENCE_TRUE_CFG=1.0
OFFLOAD_MODE=gpu
DTYPE=bfloat16
MAX_BATCH_SIZE=8
TENSOR_BATCH_SIZE=2
LORA_PATH=/workspace/StoryMapDiffusion/lora
LORA_WEIGHT_NAME=pytorch_lora_weights.safetensors
LORA_WEIGHT=1.0
FUSE_COMPILE_ENABLED=0
ENABLE_VAE_TILING_OPT=1
EOF

GPUs with less than 70 GiB of CUDA-visible total VRAM must use OFFLOAD_MODE=model or sequential; start with TENSOR_BATCH_SIZE=1. If you are not using a custom LoRA, set LORA_PATH= and skip the upload step.

Start the server in the background from the GPU terminal:

source .venv/bin/activate
nohup python -m uvicorn inference.server.main:app --host 127.0.0.1 --port 8000 --workers 1 > /tmp/inference.log 2>&1 &
echo "Server PID: $!"
tail -f /tmp/inference.log

Ctrl+C exits the log viewer while the server keeps running. Wait for the model to load, then check curl http://127.0.0.1:8000/health for model_loaded: true. If the logs report a missing hf_transfer package, run python -m pip install hf_transfer in this environment and restart.

To restart after editing .env, stop the inference process, wait for VRAM to be released, then rerun the nohup command:

pkill -f '[u]vicorn inference.server.main:app' || true

After a successful small run on an A100 80GB, try TENSOR_BATCH_SIZE=8, FUSE_COMPILE_ENABLED=1, and ENABLE_VAE_TILING_OPT=0. If you encounter an OOM error, reduce the microbatch size and re-enable tiling. See the mode and batch explanation.

5. Open an SSH tunnel — local machine

Keep this command running in a separate local terminal:

ssh -p SSH_PORT -i ~/.ssh/id_ed25519 -L 8000:127.0.0.1:8000 -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -N root@GPU_HOST

It is normal for the tunnel not to open a shell. Check it from another local terminal:

python -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8000/health').read().decode())"

If local port 8000 is in use, change the forwarding option to -L 18000:127.0.0.1:8000 and use http://127.0.0.1:18000 as the endpoint. With a tunnel, the server only needs to bind to 127.0.0.1; the inference port does not need to be exposed to the Internet.

6. Run inference — local client

Activate the local Python environment from Setup and run from the repository root containing output/renders/. These single-line commands work in Bash/Zsh and PowerShell:

python -m inference.scripts.run_inference_pipeline --renders output/renders --output model_generate --endpoint http://127.0.0.1:8000 --batch-size 8 --steps 4 --resume -v

To reduce the request batch size and network payload:

python -m inference.scripts.run_inference_pipeline --renders output/renders --output model_generate --endpoint http://127.0.0.1:8000 --batch-size 4 --jpeg-quality 85 --steps 4 --resume

--resume continues missing work in the same output directory. --batch-size controls images per request, separately from GPU TENSOR_BATCH_SIZE. Results are saved in model_generate/ on your local machine.

Using the dashboard instead of the CLI: enter http://127.0.0.1:8000 as the endpoint in Map inference and run normally; do not click Deploy on Modal. Keep the tunnel open throughout the run. When finished, stop/terminate the instance through your provider; closing SSH does not stop the rented machine.

🌐 Share your map

After building the DZI, package and preview the website:

python scripts/prepare_web_demo.py
python -m http.server 8088 --bind 127.0.0.1 --directory web-demo

Open http://localhost:8088. Upload web-demo/ to a static host to share the pannable, zoomable map. The static version does not currently integrate the Curator backend.

📚 Further reading

Topic	Documentation
Illustrated dashboard walkthrough	UI workflow
External GPUs, SSH, VAE tiling, offloading, and batching	Remote inference
4-step Lightning and LoRA fusion / compilation	Lightning mode · Fuse LoRA + compile
3D Tiles, capture, and cloud training	Tiles & cloud crawler
GeoJSON, CLI, checkpoints, deployment, and troubleshooting	Advanced operations
Curator across multiple maps	Curator multimap

🔗 References and acknowledgments

Open-source projects and documentation related to StoryMapDiffusion's approach:

Repository / model	Relevance
ModelTC / LightX2V-Qwen-Image-Lightning	Qwen Image distillation and acceleration. This project's Lightning mode uses the 4-step Qwen-Image-Edit-2511-Lightning adapter.
cannoneyed / isometric-nyc	The main reference for building AI-generated isometric pixel-art maps of real places.
FlyMyAI / flymyai-lora-trainer	A reference for Qwen-Image LoRA training; the repository provides a text-to-image trainer.
Qwen / Qwen-Image-Edit-2511	Model card for the base model used by this project's image-editing and LoRA pipeline.

Technical documentation:

• Diffusers — QwenImage: Qwen Image/Edit pipelines and usage.
• Diffusers — Merge LoRAs: merge adapter weights into the model with fuse_lora().
• Diffusers — Accelerate inference: inference optimization and regional compilation, relevant to Fuse LoRA + compile.
• Modal — Pricing: GPU, CPU, RAM, and storage rates for deployment cost estimates.
• CesiumJS: 3D Tiles, cameras, and geographic visualization for capture.
• OpenSeadragon: high-resolution image viewing, zoom/pan, and Deep Zoom.
• pyvips / libvips: large-image processing and map image pyramids.
• MediaWiki Action API: Wikipedia content queries for Curator landmark information.