Skip to content

CI flake: MPI_Init dies in UCX on some Azure runners (mana_0 verbs transport) #528

Description

@lmoresi

Twice in two days (PR #526 first run, PR #527 first run) the test job died at the import smoke test, before any underworld3 code loaded:

ib_iface.c:1316 UCX ERROR mana_0: iface ... failed to create UD QP ... Operation not supported
ucp_worker.c:1415 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error
MPIDI_UCX_init_worker(86): ucx function returned with failed status

UCX is trying an unsupported verbs transport on the runner's Azure MANA network interface; whether a run flakes depends on which VM type the job lands on. Rerun on a fresh runner passes (measured, #526).

Suggested immunisation: pin the transport in the workflow env for the test job, e.g.

env:
  UCX_TLS: tcp,self,sm

(or UCX_NET_DEVICES=lo for the serial import smoke alone). Filed so the reruns stop being folklore.

Underworld development team with AI support from Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions