Skip to content

rig: install the shared ROS 2 package set from reiserlab/rig-nodes (extraction, LAB-338) - #274

Merged
5 commits merged into
mainfrom
feat/rig-nodes-extraction
Oct 28, 2026
Merged

5 commits merged into
mainfrom
feat/rig-nodes-extraction

Conversation

@mbreiser

@mbreiser mbreiser commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Extraction step of the rig-software decision record (docs/development/rig-software-repos-and-runners.md §3.3 / §7 / §9, LAB-338, LAB-303): the shared ROS 2 infrastructure under rig/ moved with history to reiserlab/rig-nodes and is installed here as the pinned package set rig_nodes + rig_interfaces 0.2.1 from that repository's conda channel. rig/ keeps the arena's own code: the Studio gateway (bridge_node.py), the behavior_v2 converter (mcap_convert.py, _rows.py), the composition (rig.py, pixi.toml), the arena visualization (viz_node.py, the arena half of foxglove_json.py, foxglove/), the Studio emulator and the harness.

Moved (removed here): rig/interfaces/rig_interfaces, rig/channel, rig/config (the Zenoh configs now ship inside the rig_nodes package), rig/nodes/{_common, fictrac_udp_adapter, kinefly_udp_adapter, kinefly_sim, bias_node, bias_sim, bias_cmd, scanimage_node, scanimage_cmd, recorder, mcap_index, graph_dump, build_interfaces}.py, rig/tests/si_json_mock.py, rig/sizing/, the three benches, rig/docs/{shm-bench, typed-distribution-decision, typed-windows}.md, fictrac-bridge/fictrac_sim.py (+ its tests → rig-nodes tests/test_fictrac_sim.py; the closed-loop-report half stays as tests/test-closed-loop-report.py), .github/workflows/rig-interfaces.yml (→ rig-nodes package-set.yml).

Rewired: rig.py composes with rig_nodes.supervise (process groups, port preflight, Child) and starts device nodes as python -m rig_nodes.<module>; bridge_node.py imports rig_nodes._common / recorder / mcap_index and writes the package-set versions into hello_ack and the /arena/session record (record §7.7); mcap_convert.py imports the CDR decoder from rig_nodes.cdr; _rows.py re-exports the FicTrac line grammar from rig_nodes.fictrac; foxglove_json.py is the arena enrich wrapper around the generic rewriter; the harness drives the installed nodes. pixi run sim at the root delegates to the rig/ workspace.

rig/pixi.toml: channel https://raw.githubusercontent.com/reiserlab/rig-nodes/v0.2.1/channel; rig_nodes ==0.2.1; ros2-distro-mutex 0.19.* + Python 3.12 pinned explicitly (the set the packages were built against; [system-requirements] macos = "14.0" so osx-arm64 resolves the same set instead of 0.16); environments default (String topics, as before), typed (+ rig_interfaces ==0.2.1, same solve group — the 0.2.0 build's mixed pins had forced a separate one), viz, and dev (rig-nodes main built from source through git source dependencies pinned in the lock — the no-release developer path of record §7 item 6; the first version used path dependencies on a sibling checkout, which made pixi install fail on every machine without one — fixed 2026-10-27 in 31011dd, verified with the sibling hidden and the network blocked: default/typed install offline, lock unchanged). The former typed-prebuilt is typed; the former source-build typed lives in the rig-nodes checkout.

Not moved (doubtful stays): everything the inventory marks "stays"; _rows.py beyond the parser; the arena half of foxglove_json.py. scanimage_frames_node / SiPM / bias_health follow when #261, #266 and #269 land.

Docs: record §9 completion note, CLAUDE.md boundaries paragraph, orientation skill (rig-nodes = current location), rig/README.md banner, WINDOWS-TEST.md 1b/7b, fictrac-bridge/README.md.

Tier

next candidate — the Studio is untouched; this changes the rig-side workspace only. Coordinate with #269 (okey-cokey): it edits rig.py, foxglove_json.py, viz_node.py and adds bias_health.py; recommended order is this PR first, then #269 rebases (its bias_health.py goes to rig-nodes as the record lists it, the dashboard code stays).

Release notes

none (no user-visible change; rig PCs: cd rig && pixi install re-locks onto the rig-nodes channel — the Studio's hello_ack now carries packages).

Test plan

  • pixi run test (root): 71 files; tests/test-closed-loop-report.py 27/27 replaces the simulator half of test-fictrac-sim.py (39/39 in rig-nodes)
  • cd rig && pixi install from the channel (osx-arm64, local index of the Mac-built packages): rig_nodes 0.2.1, mutex 0.19.0, Python 3.12; pixi install -e typed → rig_interfaces 0.2.1, pixi run -e typed check-typed usable
  • cd rig && pixi run test (the LAB-303 harness): 59/59 String + 62/62 typed against the packages INSTALLED from the tagged channel (2026-10-27, after purging a pixi package-cache entry that a local source build had left under the release's filename — the manifest comment names the gotcha)
  • pixi install --locked + -e typed with NO sibling rig-nodes checkout and the network blocked (pixi git cache removed, environments removed first): both succeed, pixi.lock byte-identical; pixi install -e dev builds from the rig-nodes git source and its Zenoh config URIs resolve
  • pixi run test-viz
  • pixi lock against the tagged channel v0.2.1 (three platforms); pixi install --locked + -e typed from the real URL, check-typed usable
  • Windows rig: pixi install --locked + harness (the %CONDA_PREFIX% activation paths are untested here)

LAB-338, LAB-303.

🤖 Generated with Claude Code

cshl-2026 and others added 3 commits October 26, 2026 20:37
…eep gateway, composition, viz (LAB-338)

Extraction step of docs/development/rig-software-repos-and-runners.md §3.3 / §7 / §9. The shared
infrastructure under rig/ moved with history to reiserlab/rig-nodes (package set 0.2.1:
rig_interfaces + rig_nodes) and is installed here from that repository's channel.

Removed (moved): rig/interfaces, rig/channel, rig/config (the Zenoh configs ship inside rig_nodes),
rig/nodes/{_common, fictrac_udp_adapter, kinefly_udp_adapter, kinefly_sim, bias_node, bias_sim,
bias_cmd, scanimage_node, scanimage_cmd, recorder, mcap_index, graph_dump, build_interfaces}.py,
rig/tests/si_json_mock.py, rig/sizing, the benches, three rig/docs notes, fictrac-bridge/fictrac_sim.py
(its tests → rig-nodes; the closed-loop-report half stays as tests/test-closed-loop-report.py),
.github/workflows/rig-interfaces.yml (→ rig-nodes package-set.yml).

Rewired: rig.py composes with rig_nodes.supervise and starts nodes as `python -m rig_nodes.<module>`;
bridge_node imports rig_nodes._common / recorder / mcap_index and records the package-set versions
(hello_ack + /arena/session `packages`); mcap_convert imports the CDR decoder from rig_nodes.cdr;
_rows re-exports the FicTrac grammar from rig_nodes.fictrac; foxglove_json is the arena `enrich`
wrapper; the harness drives the installed nodes; root `pixi run sim` delegates to rig/.

rig/pixi.toml: channel raw.githubusercontent.com/reiserlab/rig-nodes/v0.2.1/channel, rig_nodes ==0.2.1,
ros2-distro-mutex 0.19 + Python 3.12 pinned, macOS 14 floor; environments default / typed (+
rig_interfaces, same solve group) / viz / dev (sibling ../../rig-nodes path deps). pixi.lock is
re-generated against the tagged channel in the next commit.

Verified on osx-arm64 against the installed packages (local index of the CI-equivalent build):
harness 59/59 (String) and 62/62 (typed), test-viz, root suite.

Docs: record §9 completion note, CLAUDE.md boundaries, orientation skill, rig/README banner,
WINDOWS-TEST 1b/7b, fictrac-bridge/README.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…x 0.19 + Python 3.12, all three platforms)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, PR #274)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser added a commit that referenced this pull request Oct 27, 2026
…d on the installed rig_nodes package (LAB-303, LAB-338)

The shared half of rig/ moved to reiserlab/rig-nodes (PR #274); this merge re-expresses the
okey-cokey additions through the installed package instead of the deleted local modules:

* rig.py — the --no-fictrac / --stimulus / --stimulus-args options and the stimulus Child on top of
  the rig_nodes.supervise composition (fictrac via `python -m rig_nodes.fictrac_udp_adapter` only
  when not --no-fictrac; the udp 60000 preflight skipped the same way).
* foxglove_json.py — the arena wrapper around rig_nodes.foxglove_json.convert keeps #274's
  `enrich` hook (frame derivations, FicTrac columns) and adds the --stimulus dashboard replay as the
  rewriter's `derive` hook (whole-stream → new topics, foxglove.CompressedImage by well-known
  schema name); dashboard_replay.run_streams is unchanged. Needs rig-nodes PR #1 (derive hook).
* bias_health.py — moved to rig-nodes (rig_nodes.bias_health, rig-nodes PR #1); viz_node.py,
  dashboard_replay.py and the tests import it from there, test_stimulus's bias_health section
  moved with it (tests/test_pure.py in rig-nodes).
* viz_node.py / stimulus_node.py / test_stimulus_live.py — `_common` → `rig_nodes._common`,
  the launcher primitives → `rig_nodes.supervise`.

Until the next rig-nodes package set ships, run with `cd rig && pixi install -e dev` (a sibling
../../rig-nodes checkout on feat/bias-health or later) — the released 0.2.1 has no bias_health /
derive. Tests (dev env): test-stimulus 50/50, test-viz, `pixi run test`; root `pixi run test` 72/72.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mbreiser

Copy link
Copy Markdown
Contributor Author

Mac QC of this branch (2026-10-27, ad98b2a, worktree install from the v0.2.1 channel)

Independent rerun of the test plan plus a live run with real FicTrac (Fictrac-MacOS build/macos-opencv5, sample/sample.mp4 at 20 fps, UDP → rig-fictrac-adapter :60000).

Check Result
cd rig && pixi run test (String) 59/59 PASS, 54 s
pixi run -e typed test 62/62 PASS (ArenaFrame in the bag)
Live graph pixi run rig, Studio emulator, 2 epochs, no bias, start_frame 37 12/12 PASS; both epochs open on 37 (tare + start position correct with real heading data)
Live graph (typed), emulator, sine bias 45°/0.5 Hz 12/12 PASS; bag typed=True, /arena/frame = rig_interfaces/msg/ArenaFrame ×210
Bag → converter identical to the live file (bridge's own check) and pixi run convert offline output byte-identical to the live file; log_export content == live file
check_runlog_readers.js on the export PASS (396 lines, 210 FicTrac rows, 159 echoes, 0 parse / normalize errors; replay parser 2 epochs)
closed-loop-report.py 2 epochs found, idx mismatch 0.00 % on real FicTrac data
hello_ack packages: {rig_nodes: 0.2.1, rig_interfaces: 0.2.1}, typed: true, connectors bias + scanimage
Real Studio (arena_studio.html served from this branch, Chrome) → Run view bridge Connect connected, FicTrac 20 Hz, 692 frames received, scope traces live, no console errors

Not covered here: Windows rig/ install, any arena controller (no Web Serial run, so applied = 0 in the browser check and runlog-check.py reports the expected "no trial_quality / stream_schema" for an emulator run).

Two observations, both pre-existing in bridge.py (not introduced by this PR):

  1. Bias clock vs first frame. With constant 90°/s the emulator's "epoch opened on start_frame" assertion fails at 20 fps: the first applied index is start_frame + 2 because the bias angle accumulates from the epoch config, and the first FicTrac frame arrives up to 50 ms later. The 50 Hz simulator hides it. Either anchor the bias waveform's t0 to the first post-tare frame (matches "the epoch opens on frame_index") or give the emulator a frame-rate-aware tolerance.
  2. Scope spikes on FicTrac restart. Looping the 10 s clip restarts FicTrac; the Studio's live turning / forward traces spike (±10⁶) at each restart because the derivative does not guard the counter / timestamp reset the logger already records as ft_reset. Cosmetic.

…ng path (fixes `pixi install` on rig PCs)

`cd rig && pixi install` failed on any machine without a `../../rig-nodes` checkout
(ww18, 2026-10-27): pixi validates every environment's source PATHS before installing
ANY environment, so the `dev` feature's path dependencies broke the default install.
`pixi install --locked` reported "lock-file not up-to-date", `pixi install -e default`
"source path … could not be found".

The `dev` feature now uses git source dependencies on reiserlab/rig-nodes `main`
(`subdirectory = src/rig_nodes` / `src/rig_interfaces`), pinned to one commit in
pixi.lock (75b9986 = the v0.2.1 release commit without the channel payload). Git sources
are fetched only when `dev` itself is installed. Verified on the Mac with the sibling
hidden and the network blocked (dead proxy, pixi git cache removed, environments
removed first): `pixi install --locked` and `pixi install --locked -e typed` succeed and
leave pixi.lock byte-identical. Lock diff is confined to the dev block (the two source
lines per platform) and to the source packages' build records (uv 0.13.5 → 0.13.6).

Dev Zenoh config URIs: with a git source the package is a real (non-editable) install,
so the `$CONDA_PREFIX/…/site-packages/rig_nodes/config/*.json5` URIs resolve in `dev`
too (they did not with the editable path build — LAB-338 note).

Also found: a source build of the same version as the release lands in pixi's package
cache under the release's filename and shadows a later channel install on that machine
(my default env held the editable 25-file stub while conda-meta showed the channel URL
and sha). After purging that entry: default 59/59, typed 62/62 harness checks against
the genuine channel packages. The manifest comment, CLAUDE.md, rig/README.md,
WINDOWS-TEST.md and the record (§7 item 6, §9) say how to point `dev` at a local
checkout without committing it and name the cache gotcha.

LAB-338, LAB-303.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mbreiser

Copy link
Copy Markdown
Contributor Author

QC — webDisplayTools PR #274 (feat/rig-nodes-extraction) on reiserlab-ww18 — 2026-10-27

PR: #274 — rig/ installs the shared package set from the
reiserlab/rig-nodes channel (tag v0.2.1) instead of carrying the device nodes.

Host: reiserlab-ww18 · Microsoft Windows 11 Enterprise 10.0.26200 · pixi 0.80.0 · git 2.55.0.windows.3
Started: 2026-10-27 10:07 ET · finished ~11:00 ET

Summary

step result
1 checkout ad98b2a PASS
2 pixi install FAIL as shipped (dev env path deps need ..\rig-nodes; known, fix in progress) → with the sibling clone workaround pixi install --locked PASS, 10 s (warm cache), 1.6 GB, 398 pkgs; rig_nodes 0.2.1, mutex 0.19.0, Python 3.12.16
3 pixi install --locked -e typed + check-typed PASS, 5 s, +rig_interfaces 0.2.1 (+ rosidl-default-runtime) only, no compiler; importable + serializes
4 pixi run test PASS 59 / 59, 58 s
5 graph, real FicTrac graph PASS; real FicTrac NOT DONE — the FLIR camera failed (Spinnaker −1011 frame timeout, then no USB enumeration); hardware, not the PR
5b graph, model-fly sim PASS — /fictrac/raw 51 Hz → /arena/frame 50 Hz
6 Production Studio v0.98 ↔ bridge node PASS — connect + config, two full runs of bench_closed_loop_2x10, 4/4 steps, 0 errors
7 export / bag PASS — bag closed + zstd-indexed at run end, metadata.yaml present, export identical to the live file: True, /arena/session carries packages: {rig_nodes: 0.2.1, rig_interfaces: null}
7 closed-loop-report report says 100 % idx mismatch — script gap (ignores bridge-3.4 start_frame); recomputed with it: 0.0 % both epochs, both runs; epochs open on frames 25 / 75
7 runlog-check run 1 FAIL (telemetry ring off in this browser); run 2 (ring on) FAIL only from ring backlog on the first drain — in-run controller 0x70 = host 0x70 = 800 exactly; 0 failed trials
8 main comparison skipped (time)

arena run: DONE (2 runs, G6 2x10, model-fly simulator). Real-FicTrac run: NOT DONE.
Verdict for #274 on this PC: the rig_nodes 0.2.1 channel packages work end to end on Windows (install, typed, harness,
graph, recorder, converter, export, Studio); the one blocker is the step-2 install failure without a sibling checkout.

1. Checkout

git fetch origin && git checkout feat/rig-nodes-extraction && git log --oneline -1
ad98b2a docs: record §9 — extraction state (rig-nodes CI green, v0.2.1 tagged, PR #274)

Result: PASS. (rig/.pixi did not exist before → the install below is a fresh env.)
Untracked files left in the checkout from earlier work (not part of this QC): .agents/skills/arena-power/,
.claude/skills/arena-power/, persist.json, power-logs/, tests/test-netio-power.py.

2. pixi install (fresh default env) — FAIL (blocker)

cd rig
pixi install          # timed with date +%s around it

Exits rc=1 after 0 s — nothing downloaded. Verbatim (after the manifest warning below):

Error:   × failed to solve requirements of environment 'dev' for platform 'win-64'
  ├─▶   × failed to resolve source package 'rig_interfaces' (at '../../rig-
  │     │ nodes/src/rig_interfaces')
  │
  ╰─▶   × failed to discover a valid project manifest, the source path 'C:
        │ \Lab\GitHub\rig-nodes\src\rig_interfaces' could not be found

Diagnosis: pixi install (pixi 0.80.0) checks the lock for EVERY environment before installing the default
one. pixi.lock contains the dev environment, whose rig_nodes / rig_interfaces are conda_source
path entries @ ../../rig-nodes/src/...; with no sibling C:\Lab\GitHub\rig-nodes checkout (a rig PC
normally has none) pixi cannot read the source manifests and aborts — even though default itself only
needs the v0.2.1 channel. So as shipped, the plain rig install requires a rig-nodes checkout beside
webDisplayTools, which is what the extraction was meant to remove. pixi.lock was not modified
(git status rig/ clean); an empty rig\.pixi dir was created.

Also printed (warning only, harmless today):

 WARN Encountered 1 warning while parsing the manifest:
  ⚠ the `[system-requirements]` table is deprecated in favor of virtual
  │ packages on `platforms`
  help: e.g. platforms = [{ name = "my-osx-arm64", platform = "osx-arm64",
        macos = "14.0" }, "win-64", "linux-64"]

Quirk #1 / workaround (known #274 bug, fix in progress in another session): clone rig-nodes next to the
checkout so the dev path check is satisfied; default and typed still install from the channel and build nothing.

git clone --branch v0.2.1 --depth 1 https://github.com/reiserlab/rig-nodes ..\rig-nodes
#   → C:\Lab\GitHub\rig-nodes @ e69ac22 "release: package set v0.2.1 (channel payload from run 40427967719)"
cd rig
pixi install --locked

2b. pixi install --locked (with the sibling clone) — PASS

  • rc=0, 10 s wall ("The default environment has been installed."). The env is new (rig\.pixi did not
    exist), but the packages came from this PC's warm rattler cache (C:\Users\reiserm\AppData\Local\rattler\cache,
    8.4 GB, filled by earlier ros2-poc installs) — so 10 s is NOT a cold-download time.
  • Env size: rig\.pixi\envs\default = 1.6 GB, 398 packages. pixi.lock unchanged (git clean).
  • Versions (pixi list):
package version build channel
rig_nodes 0.2.1 pyh4616a5c_0 raw.githubusercontent.com/reiserlab/rig-nodes/v0.2.1/channel
ros2-distro-mutex 0.19.0 jazzy_26 robostack-jazzy
python 3.12.16 hb12b558_0_cpython conda-forge
ros-jazzy-ros-base 0.11.0 h9490d1a_25 robostack-jazzy
ros-jazzy-rmw-zenoh-cpp 0.2.11 h9490d1a_25 robostack-jazzy
ros-jazzy-rosbag2-py / -rosbag2-storage-mcap 0.26.11 h9490d1a_25 robostack-jazzy
mcap 1.6.0 pyhd8ed1ab_0 conda-forge
websockets 17.3 py312h5247d40_0 conda-forge
pyzmq 27.2.0 py312h343a6d4_2 conda-forge

Mutex 0.19.x + Python 3.12: as required.

  • Windows activation: ZENOH_ROUTER_CONFIG_URI expands to
    C:\Lab\GitHub\webDisplayTools\rig\.pixi\envs\default\Lib\site-packages\rig_nodes\config\zenoh-router.json5;
    router + session json5 both exist; import rig_nodes → __version__ 0.2.1 from the env's site-packages.
  • Every pixi command prints the [system-requirements] deprecation warning (noise only).

3. pixi install --locked -e typed + check-typed — PASS

  • rc=0, 5 s wall (warm cache), env rig\.pixi\envs\typed = 1.6 GB, 400 packages. Versus default it adds
    exactly two packages: rig_interfaces 0.2.1 (ha41a7fa_0, rig-nodes v0.2.1 channel) and
    ros-jazzy-rosidl-default-runtime; everything else identical (same solve group → one RoboStack set,
    mutex 0.19.0, Python 3.12.16). No compiler package (cxx-compiler/vs20xx absent; cmake 4.4.4 is in
    both envs as a ros-base dependency, not a build step). Nothing was built.
  • pixi run -e typed check-typed (2 s):
    {"install_prefix": null, "built": false, "usable": true, "importable": true,
     "msgs": ["FicTracFrame", "ArenaFrame", "ConnectorReply"], "serializes": true}
    
  • Odd, not reproduced: my very first pixi install --locked -e typed returned Error: × unknown environment 'typed'
    (rc=1, 0 s) although pixi info in rig\ lists typed; an immediate retry with the identical command installed
    fine. Most likely my shell was not in rig\ for that one call (the root workspace has no typed env). Treat as
    operator noise unless someone else sees it.

4. pixi run test -- --keep (acceptance harness, default env, no hardware) — PASS

  • 59 / 59 checks passed, rc=0, 58 s wall. (WINDOWS-TEST.md still says "46 / 46" — the suite has grown; doc is stale.)
  • info typed topics: False (['std_msgs/msg/String']) — expected in the default env.
  • Crash phase (bridge node killed mid-run with TerminateProcess):
    • crashsafe profile: NDJSON 617 lines / 157 frames (no torn line); bag --recover → 611 records / 156 frames
      (torn: true) — 1 frame lost, same as the Mac. Restarted bridge recovers the unclosed bag on log_export
      (recovered:true), recovered bag whole (metadata.yaml + chunk index, .torn.bak kept).
    • default profile: NDJSON 618 lines / 157 frames; bag unrecoverable, as designed —
      [convert] ValueError: no session-open record on /arena/session (not a bridge-node bag, or the open record was lost)
      (rosbag2 defaults buffer a 768 KiB chunk + message cache until close). Reported as info, not a failure.
  • After the harness: no python/zenoh processes left, nothing listening on 7447 / 8765 / 60000 (TIME_WAIT only).
  • Artifacts kept in C:\Users\reiserm\AppData\Local\Temp\rig-test-5onlh4oz; summary.md + full log copied to
    C:\qc\harness-pr274\.

5. Graph with real FicTrac — graph PASS, FicTrac camera BLOCKED (not a #274 issue)

pixi run rig started 10:23:34 ET (background, log kept):

[rig] zenoh router listening on 127.0.0.1:7447
[rig] up: router, fictrac, bridge  —  Studio → ws://localhost:8765   (Ctrl-C to stop)
[fictrac ] ... [fictrac_udp_adapter]: UDP 127.0.0.1:60000 → /fictrac/raw (std_msgs/String JSON, reliable)
[bridge  ] [bridge-node] Windows clocks: precise wall clock for every stamp; 1 ms system timer granted
[bridge  ] [ws] serving ws://127.0.0.1:8765  (bridge-node 3.4 · ... rmw=rmw_zenoh_cpp, in=/fictrac/raw, typed=False,
           record=mcap/crashsafe, export=converted, frames=200, coupling=1, deg_per_frame=1.8, log=on-demand, level=behavior_v2)

Up in ~3 s. Bridge also prints two zenoh config deprecation WARNs (routing.router.peers_failover_brokering,
routing.peer.mode/linkstate "deprecated and have no effect") — from the json5 shipped in rig_nodes 0.2.1; cosmetic.

pixi run graph -- --text (5 s): nodes /arena_bridge, /fictrac_udp_adapter; /fictrac/raw adapter → bridge;
bridge publishes /arena/frame, /bias/cmd, /scanimage/cmd, subscribes /bias/*, /kinefly/raw, /scanimage/*
(all 0 Hz before FicTrac). PASS.

FicTrac (2.1.2, Spinnaker 4.4.0.246, cshl\rig05_vfov28_2026-09-22\config.txt, src_fn 0, UDP 127.0.0.1:60000):

  • Quirk remove G5 #2: launched as build\Release\fictrac.exe config.txt from Git Bash →
    error while loading shared libraries: api-ms-win-crt-locale-l1-1-0.dll: cannot open shared object file: No such file or directory (rc 127).
    Fix: launch via C:\Lab\GitHub\fictrac\build\Release\fictrac.bat config.txt (puts the fictrac pixi env's
    Library\bin + Spinnaker\bin64\vs2015 on PATH) — same as W11 on 2026-10-15.

  • Quirk Standardize CI/CD Node.js version to v24 #3 (blocking): with the .bat, twice (10:26:07, 10:26:29): PGR camera initialised (720x540 @ -1.000 fps)!,
    Opening UDP connection to 127.0.0.1:60000, then after ~1 s
    Error grabbing frame! Error was: Spinnaker: Failed waiting for EventData on NEW_BUFFER_DATA event. (GenTL error code: -1011) [-1011]
    and exit -1073740791 (0xC0000409). The camera opens but delivers no frame. Adapter stays at 0 lines published (0.0/s), 0 datagrams.
    No other camera program running (only the graph's python processes). Awaiting a camera check by Michael.

  • 10:3x: SpinView (Michael) also showed no frames from the camera → camera unplugged/replugged. FicTrac retry
    10:32:27: Error! Could not find any connected PGR cameras! / Error! Could not open input frame source (0)! (rc 0).
    Windows PnP: the FLIR camera (USB3 Vision Camera, USB\VID_1E10&PID_4000\21205235) is listed with
    Present = False — after the replug the OS does not enumerate it at all. Hardware/USB issue on this PC's
    camera, independent of the PR (FicTrac/Spinnaker never reach the graph).

  • After moving to another USB port + cable (Michael): PnP still Present = False for USB\VID_1E10&PID_4000\21205235,
    no new/unknown USB device and no USB/PnP events in the System log for the last 20 min. FicTrac retry: again
    Error! Could not find any connected PGR cameras!. The camera's rear status LED is blinking (Michael), so it has power
    but never shows up on the bus.

  • 11:01 ET retry (after the runs): still Present = False, last arrival still 10:33:19. FicTrac again
    Error! Could not find any connected PGR cameras!. The only USB arrivals since 10:33 are a Genesys hub
    (VID_05E3&PID_0625 / PID_0610) and the G6_Arena controller (VID_16C0&PID_0483, COM4), both at 10:37.

Real-FicTrac part of step 5: NOT DONE (camera hardware down; Michael's decision 10:4x: continue on the simulator).

5b. Graph with the FicTrac simulator (model fly) — PASS

Stopped the real-FicTrac graph (TaskStop; no run/bag was open, nothing lingered, 7447/8765/60000 free), then:

pixi run rig -- --sim --sim-args "--proto udp --model fly --kp 2 --seed 1 --rate 50"

[rig] up: router, fictrac, bridge, sim — Studio → ws://localhost:8765; sim model fly: kp=2 kv=0 tau=0.1s noise=30°/s saccades 0.5/s×45°, UDP → 127.0.0.1:60000. pixi run graph -- --text: nodes /arena_bridge, /fictrac_udp_adapter;
/fictrac/raw 51.0 Hz (adapter → bridge), /arena/frame 50.0 Hz (bridge). The sim's own ws client connected
to the bridge (client connected … (total 1)). PASS.

  • Camera history from Windows (Get-PnpDeviceProperty USB\VID_1E10&PID_4000\21205235): driver Teledyne 2.8.3.16 (unchanged);
    LastArrivalDate 10:33:19, LastRemovalDate 10:33:35 — after the first replug it enumerated and dropped off the bus 16 s
    later; after the second replug (other port/cable) it never arrived. PC up since 2026-10-13 22:23; KB5124007/KB5126052/
    KB5129195 installed 10-13/10-14 (after that boot). Next steps proposed: reboot after the QC run, then full camera power
    cycle, then SpinView on another PC to split camera vs PC.

6. Studio (Production, v0.98) ↔ bridge node

  • Bridge log on Studio connect: [ws] client connected ('127.0.0.1', 57874) (total 2) then
    [cfg] applied {'deg_per_frame': 1.8, 'coupling': 1.0, 'offset': 0.0, 'fictrac_port': 60000} — Studio handshake + config push accepted.

  • Arena attached and free → connected (firmware 12bba7e9 2x10 2026-09-30 detached freerun sdfast, SD SanDisk SD64G 63.83 GB exFAT 128 KiB clusters), protocol protocols/validation/bench_closed_loop_2x10.yaml (unchanged by the PR),
    Run experiment at 10:47:14 ET, Commit directly OFF. run_metadata.tool_version = "Arena Studio v0.98", rig_id rig5-mr,
    log_format behavior_v2 (acked).

  • Bridge during the run (verbatim):

    [log] writing to C:\Lab\GitHub\webDisplayTools\rig\runlogs\arena-log-20261027-104714-183.jsonl (behavior_v2)
    [record] C:\Lab\GitHub\webDisplayTools\rig\runlogs\arena-log-20261027-104714-183.bag (crashsafe, typed=False; opened in 22 ms)
    [cfg] applied {... 'coupling': -1.0, 'offset': 45.0, 'frames': 200, 'start_frame': 25, ... 'epoch': True}    # epoch A
    [cfg] applied {... 'coupling': -1.0, 'offset': 135.0, 'frames': 200, 'start_frame': 75, ... 'epoch': True}   # epoch B
    [record] closed C:\Lab\GitHub\webDisplayTools\rig\runlogs\arena-log-20261027-104714-183.bag: 1951 records, 67949 B on disk
    

    The bag closed at run end by itself (Studio's log close), before any export.

  • Runner summary: completed: true, aborted: false, steps: 4, errors: 0, skipped: 0, fault: null, stopAcked: true.

  • Cosmetic: 3× [log] arena_command kept verbatim: head '0d 08 03 24 00 00 00 19 …' not spaced lowercase hex bytes — the
    Studio truncates long trialParams heads with "…", which the compact a row cannot hold; the echo is written verbatim
    (lossless, by design, capped at 3 warnings). Same on main's bridge.py.

  • arena run: DONE (on the model-fly simulator, not real FicTrac).

7. Export + reports

pixi run export (1 s):

[export] arena-log-20261027-104714-183.jsonl: 1952 lines, 91776 bytes → C:\Lab\GitHub\webDisplayTools\rig\runlogs\exports\arena-log-20261027-104714-183.jsonl
[bridge] [convert] ...bag → 1952 lines in 22.8 ms; identical to the live file: True
[bridge] [log] export → arena-log-20261027-104714-183.jsonl (91776 chars)

Bag: rig\runlogs\arena-log-20261027-104714-183.bag\ = …bag_0.mcap (67 949 B, 1 chunk, zstd, indexed) +
metadata.yaml present (storage mcap, message_count 1951). Topics: /arena/browser 835, /arena/frame 1104,
/arena/event 10, /arena/session 2. Live shadow NDJSON 93 728 B; bag ≈ 72 % of it. PASS.
PR-specific: the /arena/session open record carries "packages":{"rig_nodes":"0.2.1","rig_interfaces":null}
(null is right in the default env). PASS.

pixi run python scripts/closed-loop-report.py <export> — 1104 FicTrac rows · 2 epochs (from runner fictracApply):

# condition dur s coupling · °/frame frames fly net turn ° fly ω mean / rms °/s feature mean|az| ° frontal idx mismatch (report)
1 epoch_a_frame25 8.02 k −1 · 1.8°/f 200 −96.8 −12.13 / 170.6 140.2 12 % 100.00 %
2 epoch_b_frame75 8.0 k −1 · 1.8°/f 200 −37.4 −4.68 / 186.8 171.9 0 % 100.00 %

The 100 % is a report-script gap, not a bridge fault: closed-loop-report.py (last changed in #211, before
bridge 3.4's start_frame) predicts round((coupling·Δh + offset + b)/pitch) with offset from the browser's config
echo, which is 0; it never adds start_frame × deg_per_frame (the bridge applied offset 45° / 135°). Recomputed with
the same formula plus that offset (scratch script, unwrapped Δh since each epoch's heading_tare):
epoch A 0.0 % mismatch (400 rows, opens on 25, 26, 26 …), epoch B 0.0 % (400 rows, opens on 75, 76, 78 …).
So both epochs opened on their frame_index and the mapping is exact. Affects main too → follow-up for the script.

pixi run python scripts/runlog-check.py <export> — FAIL (expected for this run, not the PR):

FAIL rig/runlogs/exports/arena-log-20261027-104714-183.jsonl
      firmware: 12bba7e9 2x10 2026-09-30 detached freerun sdfast
      sd_card: SanDisk SD64G 8.5 sn fb50043b (2024-04) 63.83 GB SDHC/SDXC exFAT 128 KiB clusters
      cc_0x70_ok: 0
      a_ok: 801
      ! no stream_schema
      ! no trial_quality event (export ordering / finalize did not run)
0 / 1 run files complete

Cause: run_metadata has "telemetry":"off" — the Studio's controller-telemetry drainer was off in this browser
(Debug ▾ telemetry toggle), so no cc/cf/cs rows, no stream_schema, no trial_quality. /arena/browser has no
rows message either, so the bridge node was never offered any. Re-run with telemetry ON to exercise that path.

7b. Run 2 with the telemetry ring ON (Debug ▾ → Telemetry ring) — PASS for the PR's path

Same protocol, 10:55:40 ET. Bridge: [record] closed …arena-log-20261027-105540-020.bag: 2583 records, 118947 B on disk;
[convert] … → 7240 lines in 40.2 ms; identical to the live file: True; bag has metadata.yaml. run_metadata.telemetry
= ring-10hz; stream_schema + trial_quality present; runner completed: true, errors 0, skipped 0, stopAcked: true.
So the Studio → rows → bridge node → bag → converted export path works with the rig_nodes 0.2.1 recorder.

closed-loop-report: 1105 FicTrac rows, epochs 8.01 s / 8.02 s, fly net turn −134.4° / −44.7°, idx mismatch (report) 100 % / 100 %
— same start_frame gap as run 1; recomputed with offset = start_frame × 1.8: 0.0 % / 0.0 % (401 rows each; opens on 25 and 75).

runlog-check:

FAIL rig/runlogs/exports/arena-log-20261027-105540-020.jsonl
      firmware: 12bba7e9 2x10 2026-09-30 detached freerun sdfast
      verdicts: {'pass': 19, 'fail': 0, 'unknown': 3, 'open': 0}
      cc_0x70_ok: 3385
      a_ok: 800
      fictrac_per_cmd: 0.33
      ! 3 trial(s) unknown (coverage) — see reasons in the event
      ! controller-accepted 0x70 3385 vs host-accepted 800 differ by 2585 (a drain gap if a > cc; a second run in the same file if cc > a)
      ! FicTrac rows / commands = 0.33 (< 0.8: behaviour stream gaps)

Explained, not a PR fault: telemetry was switched on mid-session, so this run's first drain carried the ring's
backlog (controller t_us of the cc rows spans 972 s → 3770 s, ~47 min, for a 22 s run; trial_quality's trial 1 =
507 s). Splitting the cc rows at the log's start: 800 accepted 0x70 inside the run = a_ok 800 exactly, 2585 before it.
With 800, FicTrac rows / commands = 1.38. The 3 unknown verdicts are coverage on the backlog-padded trials. A third run
would come out clean; skipped to save time. Firmware verdicts: 0 fail.

8. Stop + main comparison

  • Graph stopped 10:58 ET (supervisor terminated from the session after both runs were closed and exported; no run open,
    so no bag at risk). Nothing left behind: no python/zenoh processes, nothing listening on 7447 / 8765 / 60000.
  • Run logs, bags and exports archived to C:\qc\runs-pr274\ (1.1 MB): arena-log-20261027-104714-183 (run 1, telemetry
    off) and arena-log-20261027-105540-020 (run 2, telemetry ring on), each .jsonl + .bag\ + exports\…jsonl.
  • main comparison: NOT DONE (skipped to finish the gate; optional per the plan).
  • Checkout left on main @ 71c7a60 "Release 2026-10-26 (Release 2026-10-26: Arena Studio v0.98 #271)". Local main is 2 commits behind origin/main
    (a8ef775 / 92142a5, docs only, docs: remove the pilot sign-off gate between cleanup and extraction (rig-software decision record) #272); not pulled. Heads-up: rig\.pixi\ (the PR branch's default + typed envs) and
    rig\runlogs\ stay in the working tree (gitignored); the next pixi install on main's rig\ re-syncs that env to
    main's lock. C:\Lab\GitHub\rig-nodes (v0.2.1 shallow clone) is left in place for the PR branch's workaround.

Follow-ups noticed (not this PR)

  1. scripts/closed-loop-report.py does not apply the config's start_frame (bridge 3.4 offset = start_frame × deg_per_frame),
    so every start_frame epoch reports 100 % idx mismatch. Same on main. Fix: PR closed-loop-report: apply bridge-3.4 start_frame in the idx check #278
    (closed-loop-report: apply bridge-3.4 start_frame in the idx check #278, Next candidate; these logs → 0.00 %).
  2. runlog-check after switching the telemetry ring on mid-session counts the ring backlog as this run's 0x70s.
    2b. Studio Debug ▾ "Telemetry ring: on/off" does not show its state → Studio: Debug ▾ telemetry ring toggle doesn't show whether it's on or off #277
  3. rig\WINDOWS-TEST.md still says the harness passes 46/46; it is 59 now.
  4. rig_nodes 0.2.1's zenoh json5 triggers two deprecation WARNs on every node start (routing.router.peers_failover_brokering,
    routing.peer.mode/linkstate); and every pixi command warns [system-requirements] is deprecated.

SETUP.md row

(a) Host-table row

host role OS / hardware installed tested (date · commit/release) result
reiserlab-ww18 arena rig PC (G6 2x10 fly-on-ball; FicTrac FLIR/Spinnaker camera) Windows 11 Enterprise 10.0.26200; Dell Pro Max Tower T2 FCT2250, Intel Core Ultra 9 285K, 63 GB RAM, NVIDIA RTX PRO 4500 Blackwell pixi 0.80.0; git 2.55.0.windows.3; webDisplayTools rig\ default + typed envs (rig_nodes 0.2.1, rig_interfaces 0.2.1, ros2-distro-mutex 0.19.0, Python 3.12.16); FicTrac 2.1.2 (pixi-built, Spinnaker 4.4.0.246); Chrome; C:\Lab\GitHub\rig-nodes v0.2.1 clone (workaround) 2026-10-27 · webDisplayTools #274 ad98b2a (feat/rig-nodes-extraction) on rig-nodes v0.2.1 install (with workaround), typed, harness 59/59, graph, 2 Studio v0.98 arena runs on the FicTrac sim: PASS; real FicTrac not done (camera hardware)

(b) Machine quirks hit tonight

  • pixi install in rig\ fails without a sibling rig-nodes checkout —
    Error: × failed to solve requirements of environment 'dev' for platform 'win-64' /
    × failed to resolve source package 'rig_interfaces' (at '../../rig-nodes/src/rig_interfaces') /
    × failed to discover a valid project manifest, the source path 'C:\Lab\GitHub\rig-nodes\src\rig_interfaces' could not be found.
    Fix (workaround until rig: install the shared ROS 2 package set from reiserlab/rig-nodes (extraction, LAB-338) #274's fix lands): git clone --branch v0.2.1 --depth 1 https://github.com/reiserlab/rig-nodes ..\rig-nodes
    next to webDisplayTools, then cd rig; pixi install --locked.
  • FicTrac started directly from Git Bash cannot find its DLLs —
    C:/Lab/GitHub/fictrac/build/Release/fictrac.exe: error while loading shared libraries: api-ms-win-crt-locale-l1-1-0.dll: cannot open shared object file: No such file or directory.
    Fix: start it with C:\Lab\GitHub\fictrac\build\Release\fictrac.bat config.txt (puts the fictrac pixi env's
    Library\bin and C:\Program Files\Teledyne\Spinnaker\bin64\vs2015 on PATH), from the config's directory.
  • FLIR camera (USB3 Vision, USB\VID_1E10&PID_4000\21205235) opened but delivered no frames —
    Error grabbing frame! Error was: Spinnaker: Failed waiting for EventData on NEW_BUFFER_DATA event. (GenTL error code: -1011) [-1011],
    then FicTrac exits 0xC0000409; SpinView also showed no frames. After a replug: arrived 10:33:19, removed 10:33:35;
    after another port + cable it never enumerates (Error! Could not find any connected PGR cameras!, PnP Present = False),
    rear LED blinking. Not fixed tonight. Next: reboot the PC (up since 2026-10-13, three updates installed after that
    boot), full camera power-down ≥ 30 s (USB + GPIO), then SpinView on another PC to split camera vs PC.
  • No system Python on PATH — Python was not found; run without arguments to install from the Microsoft Store…
    (the Store alias). Fix: use pixi run python … inside a workspace; nothing in the rig flow needs a system Python.
  • Production Studio's telemetry ring was off in this browser — run_metadata.telemetry: "off", so runlog-check reports
    ! no stream_schema / ! no trial_quality event. Fix: Debug ▾ → Telemetry ring: on (remembered per browser);
    the first run after enabling it carries the ring backlog, so judge the second run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mbreiser

Copy link
Copy Markdown
Contributor Author

ww18 QC follow-up (2026-10-27 afternoon → 10-28): new head ca9f5c4 + real FicTrac

This follows the first QC comment.

Install fix confirmed. I renamed C:\Lab\GitHub\rig-nodes away and ran a plain cd rig; pixi install at ca9f5c4: exit 0, 13 s, "The default environment has been installed.", rig_nodes 0.2.1 from the v0.2.1 channel. The step-2 blocker is fixed.

New finding: that install rewrites rig/pixi.lock on pixi 0.80.0: 1095 lines, 554 insertions(+), 541 deletions(-).

  • The rig-nodes pin is unchanged (#75b99867…).
  • pixi renames the osx-arm64 platform to osx-arm64-macos-14-0, adds virtual-packages: blocks and recomputes every conda_source hash. That looks like the [system-requirements] macos = "14.0" table being migrated into the newer lock format, so the committed lock was probably written by a different pixi version.
  • Effect on a rig: the checkout is dirty after pixi install, and the next git checkout refuses with Your local changes to the following files would be overwritten by checkout: rig/pixi.lock.
  • Possible fixes: regenerate the lock with pixi 0.80.0 and move the requirement onto platforms as the warning suggests, or pin the pixi version (requires-pixi).
  • The diff is kept on ww18 at C:\qc\_pixi_lock_rewrite_ca9f5c4.diff.

Camera. A PC reboot cleared the FLIR/USB fault from the first comment; the camera itself was fine. The camera had moved during the replugs, so I shifted the FicTrac ball ROI down 14 px, choosing the shift with the lowest FicTrac match error: median 6.0e3, against 5.2e3 on the well-tracked 10-15 run.

Graph + real FicTrac.

  • /fictrac/raw at 97 Hz and /arena/frame at 96 Hz.
  • It ran unattended for 24.5 h (8505543 lines published (96.4/s), 8505543 datagrams) with the bridge still serving.

Arena runs on real FicTrac: DONE, PASS ×2. Production Studio v0.98, bench_closed_loop_2x10.yaml, telemetry ring on, ball turned by hand. The second run had a live Foxglove dashboard connected.

run bag records · export idx mismatch (#278 report) runlog-check controller 0x70 = host 0x70
16:04 4151 · identical to live: True 0.00 % / 0.00 % (opens @25 / @75) OK, 4 pass / 0 fail / 0 unknown 1544 = 1544
16:11 + Foxglove 4147 · identical to live: True 0.00 % / 0.00 % OK, 4 pass / 0 fail / 0 unknown 1544 = 1544

Both bags were closed, zstd-indexed, and have metadata.yaml. The /arena/session record carries "packages":{"rig_nodes":"0.2.1","rig_interfaces":null}.

Timing (from the export's host clock): no measurable cost from Foxglove. Both runs had:

  • FicTrac at 96.5 Hz with 0 frames skipped.
  • Inter-arrival p50 / p99 / max of 10 / 13 / 15 ms, and 0 gaps over 30 ms.
  • 0x70 round trip p50 / p99 / max of 1 / 3 / 6 ms.

Outside this PR: pixi run rig -- --foxglove on Windows.

  • foxglove_bridge exits with Process exited with failure 3221225781 (0xC0000135, DLL not found).
  • The robostack ros2-foxglove-bridge 3.6.0 package puts foxglove.dll and foxglove_cpp_shared.dll in Library\lib, which is not on PATH.
  • Starting it by hand with <rig>\.pixi\envs\viz\Library\lib prepended to PATH works.

Verdict for #274 on ww18: it works end to end on Windows, including real FicTrac. The one thing left is the pixi.lock rewrite above.

The full QC file on ww18 (C:\qc\ww18-2026-10-27.md) has a new section 5c with these results, and its "SETUP.md row" section has a new Foxglove quirk. Archives are in C:\qc\runs-pr274\.

🤖 Generated with Claude Code

@mbreiser mbreiser closed this pull request by merging all changes into main in 8d5b952 Oct 28, 2026
@mbreiser
mbreiser deleted the feat/rig-nodes-extraction branch October 28, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants