Skip to content

fix(sharpa-avatar): backport lifecycle and health fixes - #1381

Open
zihanjiang321 wants to merge 5 commits into
NVIDIA:release/1.6.xfrom
zihanjiang321:fix/sharpa-avatar-health-release-1.6
Open

zihanjiang321 wants to merge 5 commits into
NVIDIA:release/1.6.xfrom
zihanjiang321:fix/sharpa-avatar-health-release-1.6

Conversation

@zihanjiang321

Copy link
Copy Markdown
Contributor

Fixes #1319.

Summary

This revision addresses all actionable review feedback raised on #1263, #1264, and the current PR.

Rather than continuing to add isolated health checks, it revises the Sharpa Avatar plugin lifecycle around three separate concepts:

  • Transport — whether the USB dongle is physically present and usable.
  • Physical gloves — whether each glove exists in the Avatar SDK inventory and is producing current ingress.
  • Dataset outputs — whether HUMAN, RAW, and ROBOT output streams are independently available and advancing.

This separation allows the plugin to distinguish dongle removal, glove power/link loss, stale data, dataset setup failures, and runtime SDK errors without conflating them into a single connected flag.

The implementation remains pinned to the production Avatar SDK 1.7.3.

Review feedback addressed

This revision covers the previous and current review comments, including:

  • Refreshing and rebinding XDev hand sources when devices appear after startup or receive new XDev IDs.
  • Resetting schema sample state when tensorCollectionId changes, even if the collection occupies the same list index.
  • Preserving independent health for HUMAN, RAW, ROBOT, and haptic-only configurations.
  • Reporting non-NO_VALID_DATA_RETURNED SDK failures as device errors instead of generic data staleness.
  • Avoiding publication of fresh-looking cached SDK health after an inventory refresh failure.
  • Keeping physical dongle detection independent from SDK inventory refresh.
  • Reporting incomplete Linux USB enumeration as UNKNOWN rather than disconnected.
  • Exercising all configured datasets and haptics across connect, disconnect, and reconnect transitions.

Lifecycle changes

The plugin lifecycle has been revised so that resources are associated with the hardware or data generation that owns them:

  • USB transport state is monitored independently from glove state.
  • Left and right gloves have independent lifecycle state and recovery.
  • Dataset publishers are created per glove and per data generation, and are withdrawn when their source becomes stale or disappears.
  • Dataset setup failures are isolated per side and retried without stopping the other glove.
  • Shared SDK callbacks use generation-scoped lifetime tokens rather than retaining raw plugin pointers.
  • Long-running consumers periodically refresh XDev topology and recover when the plugin or gloves come online after the consumer has already started.
  • Health collection and publication failures remain isolated from the tracking data path.

Haptics

Haptic handling now follows the same per-glove lifecycle:

  • Stale commands are stopped by the watchdog.
  • Glove reset, disappearance, plugin shutdown, and destruction explicitly request zero vibration.
  • A glove coming back online does not replay a command from the previous device generation.
  • Failed SDK haptic calls use a short retry backoff, preventing vendor log flooding while still allowing automatic recovery.
  • Haptic-only mode retains meaningful per-glove health through its internal liveness probe.

Health reporting

The plugin publishes health for:

  • sharpa_avatar/transport/dongle
  • sharpa_avatar/glove/left
  • sharpa_avatar/glove/right
  • HUMAN hand outputs
  • RAW joint outputs
  • ROBOT joint outputs

Physical USB presence is checked separately from the SDK transport state because Avatar SDK 1.7.3 may continue to report CONNECTED after the dongle has been physically removed. Glove inventory, ingress freshness, dataset progress, and the latest SDK error are then evaluated independently.

No new Viser health UI is introduced. Applications can consume the existing TeleopSession.get_status() / get_device_status() interface, while the Sharpa example can print health transitions in the terminal.

Validation

Validated with the production Avatar SDK 1.7.3 and physical wireless gloves:

  • Plugin startup with both gloves already online.
  • Plugin and Viser startup before hardware becomes available.
  • Independent left-glove power-off and recovery.
  • Dongle unplug and replug while the plugin remains running.
  • Automatic recovery of HUMAN, RAW, and ROBOT outputs.
  • Haptic operation before and after reconnect.
  • Immediate haptic stop on stale input and plugin shutdown.
  • Haptic failure retry throttling while the dongle is absent.
  • Long-running Viser/TeleopSession recovery without restarting the consumer.
  • Clean SIGINT shutdown.

During dongle removal, the Viser and CloudXR session intentionally remain alive while the hands disappear. The hands return automatically after the transport and gloves recover.

Signed-off-by: zihan <zihan.jiang@sharpa.com>
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@lotusl-code lotusl-code removed their assignment Oct 29, 2026
Signed-off-by: zihan <zihan.jiang@sharpa.com>
Signed-off-by: zihan <zihan.jiang@sharpa.com>
Signed-off-by: zihan <zihan.jiang@sharpa.com>
@zihanjiang321

Copy link
Copy Markdown
Contributor Author

Follow-up review fixes

Backported the two follow-up fixes from #1347 in fa3aa8215:

  • Recreate the XDev list on each topology refresh and compare its generation.
    Existing trackers remain active until the new generation is fully bound, and
    an initial XDev-list creation failure is now retryable.
  • Keep haptic commands pending until the SDK successfully delivers them.
    Failed commands are retried after backoff, while a newer command received
    during backoff replaces the older pending command.

Regression coverage includes:

  • recovery after initial XDev-list creation failure;
  • rebinding unchanged XDev IDs from a new generation;
  • retrying a failed haptic command;
  • retaining the newest command received during retry backoff.

Signed-off-by: zihan <zihan.jiang@sharpa.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants