Skip to content

Podman port - #206

Open
rountree wants to merge 8 commits into
llnl:develfrom
rountree:podman-port
Open

rountree wants to merge 8 commits into
llnl:develfrom
rountree:podman-port

Conversation

@rountree

@rountree rountree commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Work in progress.

Enables containerization similar to Docker+GitHub using rootless podman.

@nchaimov
nchaimov self-requested a review August 7, 2026 17:51
[podman-port] Reconstructs serial build

spindle-podman build serial
completed with
    ==========================================
    Build complete!
    ==========================================

    Image: spindle-serial-ubuntu

    Next steps:
      1. Run regular tests: ./scripts/podman/test-spindle-serial.sh
      2. Run crash tests: ./scripts/podman/test-spindle-serial-crash.sh
      3. Or manually: podman run --rm -it spindle-serial-ubuntu

    ✓ Build completed in 300s
    ⚠  Build time (300s) exceeded baseline (250s)

Copies from podman-port-old (serial build)
    spindle-podman
    common.sh
    build-spindle-serial.sh
    Dockerfile.podman
    entrypoint.sh.podman
    .dockerignore

Updates .gitignore with
    .podman-state.json
    .podman-metrics.log
    .podman-logs
    .*.swp
The check is here because of an earlier bug that kept adding
build results to the same tarball; it eventually exceeded
96 GB.  Slow build times were the first tip-off that something
was amiss.
Added
    test-spindle-serial.sh

Works:
    ALL TESTS PASSED

    ==========================================
    ✓ All tests passed!
    ==========================================

    Cleaning up container...
    spindlenode
    ✓ Cleanup complete

    ✓ All tests passed in 68s
Working:  spindle-podman build slurm-base
    ==========================================
    Build complete!
    ==========================================

    Image: spindle-slurm-base

    Next step: Build the testing image
      ./scripts/podman/build-spindle-slurm-srun.sh

    ✓ Build completed in 580s
    ⚠  Build time (580s) exceeded baseline (450s)

Added
build-spindle-slurm-base.sh
Dockerfile.podman
build_mpich.sh.podman
Adds registry (targets.sh) describing each build target, expected
completion time, etc.  Allows for code duplication removal.

Adds `--dry-run` to `spindle-podman build x` to allow verification
that nothing in the build process has changed without needed a
full build.

Bugfixes:

1. logs no longer flags every later log in a session once one has errors
2. The grep -c ... || echo 0 pattern is fixed, so it can't write 0\n0 into the state JSON.
3. cleanup and defenestrate no longer delete `/tmp/spindle*` on the host.

Adds:

targets.sh
Adds `spindle-podman test slurm-srun`: a 4-worker Slurm cluster (MariaDB,
slurmdbd, slurmctld) in rootless podman, running the Spindle testsuite
(138 s end to end).

New targets and commands: slurm-srun and mariadb (pulled) targets; save,
load, env, diagnose; per-command help and typical workflows.

Findings that shaped the design (details in STATE.md):
- netavark starts aardvark-dns through `systemd-run --user`, which fails
  without /run/user/<uid>; the cluster helper starts aardvark-dns itself.
- Spindle's configure picks its srun launch mode from `srun --version`,
  which fails without a slurm.conf; the image now ships a stub so Spindle
  gets --external-launcher.  A build guard fails the image if it does not.
- podman state follows XDG_RUNTIME_DIR, which moves when a login session
  appears; spindle-podman pins it.

Also: hang detection with automatic snapshot (redacted), readiness polling
instead of sleeps, per-target build baselines, pulled-image targets, and
removal of the per-target build scripts.
…Slurm flavors

sshd needs uid/gid 65534, which rootless podman does not map by default, so
ssh logins were reset.  The target adds --uidmap/--gidmap for 65534 to the
Slurm containers (T_RUN_ARGS, from LLNL's container gotchas page).
`diagnose ssh` shows it: login fails by default, works with the maps.

Also: test-spindle-slurm.sh replaces the srun-only script; build guard shared
as check_launch_mode.sh.podman (checks srun or rsh mode).
…dresses

Adds `spindle-podman test slurm-plugin` (122 s): a Slurm cluster with
Spindle loaded as a SPANK plugin.

Spindle fix (src/slurm_plugin/plugin_utils.c): getHostAddrSinfo ran
`sinfo -O NodeAddr`, whose column sinfo truncates to 20 characters.  A node
named slurm-plugin-1-node-1 came back as "slurm-plugin-1-node-", the front
end could not resolve it, and the first test hung.  Ask for `NodeAddr:255`.
CI's names are short, so it never showed.  (Fully qualified hostnames over
20 characters would hit it anywhere.)

Podman side:
- plugstack.conf is chmod 644: salloc/srun read it as the job user.
- Cluster names for the plugin target use a short stem (T_CLUSTER_NAME) and
  the test script refuses names past 20 characters (T_NODE_NAME_MAX), so the
  bug cannot silently return; both lines can go once the fix is verified.
- check_launch_mode.sh.podman gains a "plugin" mode (macro and installed
  libspindleslurm.so).

Adds spindle-podman exec: runs podman exec with the pinned runtime directory, so it works from any shell.

This branch is waiting to be deployed

1 waiting deployment
Spindle CI — 8cd78ef5 Waiting Nov 14, 2026 by rountree via Testsuite (Serial, Ubuntu, arm64) #312
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant