Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[submodule "third_party/embodiinfer"]
path = third_party/embodiinfer
url = https://github.com/BUAA-CI-LAB/EmbodiInfer.git
branch = main
url = https://github.com/yufoo1/EmbodiInfer.git
branch = feat/activevln-go2-serving
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ lists robot, simulator, and model integrations.
Recipes use the shared `embodirun example` CLI; `examples/run.sh` remains a
compatible entrypoint for source checkouts.

[Go2 + ActiveVLN: single-chunk, continuous and RTC execution](docs/en/demos/go2-activevln.md) · [Deployment](docs/en/go2-activevln.md).

## Why EmbodiRun

| Feature | What you can do |
Expand Down
2 changes: 2 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@
Recipe 使用统一的 `embodirun example` 命令行入口;源码 checkout 仍可使用
兼容的 `examples/run.sh`。

[Go2 + ActiveVLN:单轮、连续与 RTC 真机执行](docs/zh/demos/go2-activevln.md) · [部署指南](docs/zh/go2-activevln.md)。

## 为什么选择 EmbodiRun

| 特点 | 使用方式 |
Expand Down
26 changes: 26 additions & 0 deletions configs/activevln/go2-orin-nx-jp5-eager.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"inference_url": "http://127.0.0.1:18767",
"robot_id": "go2",
"mode": "chunk",
"overlap": {
"lead_s": 1.5,
"max_capture_uncertainty_s": 0.15
},
"inference_token_env": "ACTIVEVLN_BENCHMARK_TOKEN",
"inference_timeout_s": 300.0,
"camera_timeout_s": 5.0,
"control_timeout_s": 0.5,
"session": {
"linear_speed_mps": 0.25,
"yaw_rate_rps": 0.5,
"control_hz": 10.0,
"max_chunk_s": 9.0,
"max_runtime_s": 120.0,
"max_chunks": 100,
"capture_timeout_s": 3.0,
"max_plan_age_s": 5.0,
"max_control_gap_s": 0.5
},
"robot_url": "http://127.0.0.1:8080",
"robot_token_env": "GO2_API_TOKEN"
}
26 changes: 26 additions & 0 deletions configs/activevln/go2-orin-nx-onboard.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"inference_url": "http://127.0.0.1:18767",
"robot_id": "go2",
"mode": "overlap",
"overlap": {
"lead_s": 1.5,
"max_capture_uncertainty_s": 0.15
},
"inference_token_env": "ACTIVEVLN_BENCHMARK_TOKEN",
"inference_timeout_s": 60.0,
"camera_timeout_s": 2.0,
"control_timeout_s": 0.5,
"session": {
"linear_speed_mps": 0.25,
"yaw_rate_rps": 0.5,
"control_hz": 10.0,
"max_chunk_s": 9.0,
"max_runtime_s": 120.0,
"max_chunks": 100,
"capture_timeout_s": 3.0,
"max_plan_age_s": 5.0,
"max_control_gap_s": 0.5
},
"robot_url": "http://127.0.0.1:8080",
"robot_token_env": "GO2_API_TOKEN"
}
26 changes: 26 additions & 0 deletions configs/activevln/go2-orin-nx.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"inference_url": "",
"robot_id": "go2",
"mode": "overlap",
"overlap": {
"lead_s": 1.5,
"max_capture_uncertainty_s": 0.15
},
"inference_token_env": "ACTIVEVLN_BENCHMARK_TOKEN",
"inference_timeout_s": 60.0,
"camera_timeout_s": 2.0,
"control_timeout_s": 0.5,
"session": {
"linear_speed_mps": 0.25,
"yaw_rate_rps": 0.5,
"control_hz": 10.0,
"max_chunk_s": 9.0,
"max_runtime_s": 120.0,
"max_chunks": 100,
"capture_timeout_s": 3.0,
"max_plan_age_s": 5.0,
"max_control_gap_s": 0.5
},
"robot_url": "",
"robot_token_env": "GO2_API_TOKEN"
}
9 changes: 9 additions & 0 deletions configs/activevln/infer-jp5-eager.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"policy_kwargs": {
"attention": "eager",
"do_sample": false,
"max_new_tokens": 512,
"max_context": 32768,
"repetition_penalty": 1.05
}
}
5 changes: 5 additions & 0 deletions configs/go2/owner-dry-run.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"server": {"host": "127.0.0.1", "port": 8080, "token_env": "GO2_API_TOKEN"},
"control": {"mode": "dry-run"},
"camera": null
}
62 changes: 62 additions & 0 deletions docs/en/demos/go2-activevln.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# ActiveVLN on a physical Go2

EmbodiRun connects the Go2 camera and hardware owner to EmbodiInfer's ActiveVLN
HTTP service. The same instruction can execute one chunk, continue through
multiple chunks, or overlap the next inference with current motion.

<video controls muted playsinline preload="metadata" width="720"
poster="https://raw.githubusercontent.com/BUAA-CI-LAB/misc/main/embodirun/v0.1/activevln_go2/go2_activevln_3x.jpg">
<source src="https://raw.githubusercontent.com/BUAA-CI-LAB/misc/main/embodirun/v0.1/activevln_go2/go2_activevln_3x.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>

[3× version (9 seconds)](https://raw.githubusercontent.com/BUAA-CI-LAB/misc/main/embodirun/v0.1/activevln_go2/go2_activevln_3x.mp4) · [Deployment guide](../go2-activevln.md)

The demo is 9 seconds at 3× playback speed. The three cases use:

| Case | CLI options | Behavior |
|---|---|---|
| Chunk 1 | `--mode chunk --max-chunks 1` | Observe, infer and execute once |
| Chunk 100 | `--mode chunk --max-chunks 100` | Repeat synchronous turns until STOP or a task bound |
| RTC + Chunk 100 | `--mode overlap --max-chunks 100 --overlap-lead-s 1.5` | Begin the next inference during the current motion chunk |

Here 1 and 100 are the maximum number of **chunks per instruction**, not the
number of actions returned by a single inference. Each model response contains
at most three actions. The video re-typesets CLI options and log excerpts for
readability; they are not a live terminal or synchronized telemetry.

## Recorded setup and limits

The footage was recorded on 2026-10-29 with a Go2, onboard Orin NX, D435I RGB,
and ActiveVLN `Arvil/Qwen2.5-VL-3B_rl_r2r_4000`
(revision `160987313e3e869705f42400d1b8f28177044518`). The JP5 eager service
uses BF16-converted weights on JetPack 5.1.1 / CUDA 11.4. The deployment guide
separates this setup from the newer optimized benchmark environment.

The RTC segment retains the first run's turn, forward motion, natural pause,
and following forward motion. It excludes the later manual head-return sequence.
The static timeline illustrates **1.493 seconds of overlap** and **1.861 seconds
of boundary waiting** from that run's log. This demonstrates execution modes;
it is not a controlled comparison of success rates or a general speedup claim.
The three clips are different tasks or takes, rather than matched trials.

## Run the modes

After preparing the owner and inference service with the deployment guide:

```bash
CONFIG=configs/activevln/go2-orin-nx-jp5-eager.json
embodirun go2 activevln plan --config "$CONFIG"
embodirun go2 activevln check --config "$CONFIG"
embodirun go2 activevln interactive --config "$CONFIG" --mode chunk --max-chunks 1 --execute
embodirun go2 activevln interactive --config "$CONFIG" --mode chunk --max-chunks 100 --execute
embodirun go2 activevln interactive --config "$CONFIG" --mode overlap --max-chunks 100 \
--overlap-lead-s 1.5 --execute
```

Run one mode at a time. The owner must be deliberately armed by the operator;
omitting `--execute` performs inference without motion. Overlap requires the
service's execution-prefix acknowledgement capability. STOP and the configured
runtime/chunk bounds end the task; exhausting a bound is not a success result.

[Reproduce the demos](../examples.md) · [Support matrix](../support-matrix.md)
176 changes: 176 additions & 0 deletions docs/en/go2-activevln.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
# Go2 / ActiveVLN on Orin NX

Use this Go2 integration with the pinned EmbodiInfer ActiveVLN serving adapter.
The entrypoint is `embodirun go2 activevln`. The [physical demo](demos/go2-activevln.md)
shows synchronous chunk execution and execution-prefix-aware overlap on Go2.

Start with `configs/activevln/go2-orin-nx-jp5-eager.json` for the recorded JP5
eager service. `go2-orin-nx-onboard.json` describes the separate optimized
service profile. Both use loopback inference (18767) and one hardware owner
(8080). `plan` is offline, `check` is read-only, and `run` without `--execute`
performs one inference without motion.

The checkpoint revision remains `160987313e3e869705f42400d1b8f28177044518`.
The standalone download and BF16 conversion scripts preserve source weights
and write checksums. Preparation uses its own Python 3.10+ project and lock at
`scripts/activevln-preparation`; it installs neither Torch nor MicroDuck and
does not change the Host or inference environments. With uv 0.12.x, run these
commands from the repository root:

```bash
uv sync --project scripts/activevln-preparation --frozen
uv run --project scripts/activevln-preparation --frozen hf auth login
uv run --project scripts/activevln-preparation --frozen \
python scripts/download_navigation_models.py \
--repo Arvil/Qwen2.5-VL-3B_rl_r2r_4000 \
--revision 160987313e3e869705f42400d1b8f28177044518 \
--output ../models/activevln-r2r --workers 4 --authenticated
uv run --project scripts/activevln-preparation --frozen \
python scripts/prepare_activevln_bf16.py \
../models/activevln-r2r ../models/activevln-r2r-bf16
```

Conversion requires verified checksums for the config and every indexed shard.
It excludes unverified files, preserves integer tensors and records nearest-even
BF16 conversion. A final `SHA256SUMS` is written only after complete conversion.

The optimized benchmark NX launcher also requires its pinned recorded R2R warmup assets and
256-episode selection manifest. Prepare these assets separately; the JP5 eager profile does not use them. INT8 needs
an additional matching calibration file. The reference NX environment is L4T
R36.4.7 / CUDA 12.6, while the inspected robot uses R35.3.1 / CUDA 11.4; deployment targets the existing JetPack 5.1.1 system. The fresh compute environment
at `/home/unitree/embodirun-envs/compute-jp5` now uses checksummed ARM64/cp310
PyTorch 2.2.0 and torchvision 0.17.2+c1d70fe wheels. CUDA 11.4 / cuDNN 8.6, BF16
GPU operations and camera inference have passed. Source compilation
was stopped. This eager path does not enable the optimized benchmark profile.

## Unified hardware owner

`integrations/go2_owner` is an independently installed Python 3.8+ package. It
owns SDK2 control and camera capture with one lifecycle, HTTP endpoint and token.
Host clients remain in `robots/unitree/go2`; shared mobile-base contracts live in
`robots/mobile_base`; execution loops live in `application/navigation`. Neither
hardware nor client code imports the model service.

The deployed control environment is a fresh uv environment at
`/home/unitree/embodirun-envs/control-jp5`, using Python 3.10.20, privately built
CycloneDDS 0.10.5 and device-built `pyrealsense2==2.55.1+go2.jp5`. The inspected
PyPI ARM64 RealSense wheels require newer glibc despite their manylinux2014 tags.
The environment's `python` wrapper selects its private native libraries and can
serve as both `EDGE_PYTHON` and `HOST_PYTHON`; no Conda/container environment is reused.
CLI install/start/status/stop, live SDK state and D435I RGB/RGB-D observations
have been checked on the Orin with motion readiness disabled. Camera-to-model CLI inference and the physical execution modes shown in the linked demo have been recorded.

Generate configuration on the Orin with its vendor Python, without streaming:

```bash
/usr/bin/python3 scripts/inspect_go2_onboard.py --camera --write-owner-config go2-owner.local.json
export PYTHONPATH="$PWD/integrations/go2_owner/src:$HOME/unitree_sdk2_python"
/usr/bin/python3 -m embodirun_go2_owner --config go2-owner.local.json --validate
/usr/bin/python3 -m embodirun_go2_owner --config go2-owner.local.json
```

Check the generated DDS interface/library paths. The file uses measured camera
serial and depth scale; multiple cameras require `--serial`. Existing files are
never overwritten. Motion readiness is an explicit `--operator-ready` startup
flag; it is never inherited from environment or configuration.

ActiveVLN uses fresh RGB at `/observation/rgb.json` without requiring depth.
`/observation.json` retains matched RGB-D validation. All camera and control routes
share one credential. Capture/HTTP workers do not hold the motor control lock;
shutdown stops motion before joining capture workers.

Remote deployment is model-independent:

```bash
embodirun go2 edge --host "$GO2_HOST" --username "$GO2_USER" --ask-password install
embodirun go2 edge --host "$GO2_HOST" --username "$GO2_USER" --ask-password start \
--config /path/to/go2-owner.local.json --python "$EDGE_PYTHON" \
--sdk-pythonpath /home/unitree/unitree_sdk2_python --supervisor-python "$HOST_PYTHON"
```

`EDGE_PYTHON` is the remote vendor SDK Python (3.8+); `HOST_PYTHON` is the remote
supervisor Python (3.10+). `status` and `stop` take the same `--supervisor-python`.
Install uploads only the owner and standalone supervisor. One service ID, `go2`,
replaces independent camera/control services. A running PID does not imply camera
or DDS readiness; use `activevln check` after starting the inference service.

The old `go2 activevln edge` and `embodirun-go2-streamvln` commands forward to this
CLI, but old `--service camera/control` flags and the two agent module entrypoints
are removed. Stop old processes before migrating. A non-loopback bind requires a
32-character-or-longer `GO2_API_TOKEN`. Navigation's `robot_url` supplies both
camera/control endpoints; explicit legacy URLs must agree when `robot_url` is set.
Single-shot `run` still disarms at the end of attempted motion. Use `interactive`
below to accept successive tasks without restarting a healthy owner. Emergency
stop and motion faults still require an intentional restart with operator readiness.
The model service remains separate.

## Repeated instructions and longer tasks

The inference service already keeps model weights loaded. The `interactive` CLI
keeps the terminal open, creates a fresh recurrent session for each instruction,
and closes that session before accepting the next instruction. Steps within one
instruction retain navigation history. This requires updated EmbodiRun **and**
`integrations/go2_owner`; the CLI rejects an old owner before enabling motion.

With the model and owner services running, use the control environment:

```bash
source /home/unitree/embodirun-envs/control-jp5/environment.sh
embodirun go2 activevln interactive \
--config configs/activevln/go2-orin-nx-jp5-eager.json \
--max-chunks 100 --max-runtime-s 600 --execute
```

```text
activevln> Move to the open area to the right of the yellow pillar.
... observation / inference / execution repeats ...
Task ended: stop (... chunks). Enter the next instruction.
activevln> Go to the door and stop.
activevln> /quit
```

Each model chunk contains at most three actions. `max_chunks=1` in the initial
first-motion configuration limited the whole instruction to one chunk; the CLI
overrides above allow up to 100 chunks or 600 seconds per instruction. Model STOP
ends the task earlier; reaching a bound does not mean the destination was reached.
Runtime bounds prevent further motion; an in-flight inference/cleanup may delay
returning to the prompt. Model context limits still apply. The 9-second chunk,
speed, heartbeat, and 5-second plan-freshness bounds remain enforced.

Ctrl+C **during a task** stops its lease, drains pending inference, clears that
task's model history and returns to the prompt. Ctrl+C at the prompt, EOF, or
`/quit` exits the CLI. Services remain running. Commands are read between tasks;
use Ctrl+C to interrupt an executing task. Omitting `--execute` makes each entered
instruction perform one inference with no motion writes.

Normal completion and Ctrl+C use an authenticated, action-ID-scoped lease release.
It stops only this task's stream and preserves already-enabled readiness after a
confirmed healthy stop. It cannot re-arm, cancel someone else's action, or hide a
failed lease. `/v1/stop`, SIGTERM during motion, and task faults after motion still
use disarming stop. Operational failures exit the CLI without retrying or re-arming.
The linked demo records physical chunk and overlap execution; it does not
establish navigation reliability.

## Verified JP5 eager service

From the repository directory on the Orin:

```bash
/home/unitree/embodirun-envs/compute-jp5/run embodiinfer-http-serve \
--policy activevln --checkpoint /home/unitree/models/activevln-r2r-bf16 \
--adapter-config configs/activevln/infer-jp5-eager.json \
--device cuda:0 --dtype auto --no-cuda-graph --max-batch 1 \
--host 127.0.0.1 --port 18767 --max-sessions 1 --max-images 1
```

Use the control environment for `embodirun go2 activevln check/run` with
`configs/activevln/go2-orin-nx-jp5-eager.json`. This synchronous chunk profile
allows a 300-second inference request timeout while retaining the five-second
execution freshness limit. Leave `ACTIVEVLN_BENCHMARK_TOKEN` unset for the
local server above, which has no token configured.

The 2026-10-29 stationary D435I RGB test completed one CLI inference with about
7.72 seconds first-request round-trip latency and three proposed 15-degree right
turns. `executed_actions=0`; no motion was enabled. This does not establish
steady-state performance or closed-loop navigation. Its first-request latency
exceeds the default execution freshness limit.
10 changes: 8 additions & 2 deletions docs/en/support-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ service.
**✓ Franka FR3** — adapter and π0.5 binding.

**◐ ARX5** — DM0.5 binding.<br>
**◐ Unitree Go2** — StreamVLN navigation.<br>
**◐ Unitree Go2** — StreamVLN; ActiveVLN async/chunk CLI for onboard Orin NX (recorded physical execution).<br>
**◐ XLeRobot** — external hardware owner; dual-arm π0.5 recipe.

[Explore robot adapters](https://github.com/BUAA-CI-LAB/EmbodiRun/tree/main/src/embodirun/robots)
Expand Down Expand Up @@ -99,10 +99,16 @@ service.
| Bi-SO-101 | π0.5, 12-dimensional dual-arm policy | [Dual-arm guide](pi05-bi-so101.md) |
| Franka FR3 | π0.5 binding | [Adapter](https://github.com/BUAA-CI-LAB/EmbodiRun/tree/main/src/embodirun/robots/franka/fr3) |
| ARX5 | DM0.5 binding | [Binding](https://github.com/BUAA-CI-LAB/EmbodiRun/tree/main/src/embodirun/bindings/arx/x5/dm05) |
| Unitree Go2 | StreamVLN navigation | [Robot integration](https://github.com/BUAA-CI-LAB/EmbodiRun/tree/main/src/embodirun/robots/unitree/go2) |
| Unitree Go2 | StreamVLN; ActiveVLN / Orin NX, RealSense, batch=1 | [CLI and configuration](https://github.com/BUAA-CI-LAB/EmbodiRun/blob/main/examples/README.md#go2--onboard-orin-nx-activevln) |
| XLeRobot | External hardware owner; `lerobot.xlerobot.pi05` for two arms | [Recipe](xlerobot-snack-delivery.md) / [owner package](https://github.com/BUAA-CI-LAB/EmbodiRun/blob/main/integrations/xlerobot_owner/README.md) |
| SO-101 (wired teleoperation) | Multi-leader UDP fan-out, follower-side episode collection | [Integration package](https://github.com/BUAA-CI-LAB/EmbodiRun/blob/main/integrations/so101_wired_teleop/README.md) |

The Go2 ActiveVLN CLI targets the frozen optimized R2R checkpoint service
(`Arvil/Qwen2.5-VL-3B_rl_r2r_4000`, revision `160987313e3e869705f42400d1b8f28177044518`).
`embodirun go2 activevln run --config LOCAL_CONFIG --instruction TEXT` is
observation/inference only; `--execute` enables the bounded motion path.
Python 3.10+; [recorded physical chunk/overlap execution](demos/go2-activevln.md) on JP5 Orin NX. See the linked recipe for service setup.

SO-101 deployments need calibrated arms, camera mappings, and a checkpoint
trained for the selected binding. FR3 needs its robot SDK; ARX5 needs the
vendor motor driver. Source links are starting points for custom integration.
Expand Down
Loading
Loading