Ephemeral GitHub Actions runners on Proxmox VE. Each job runs in its own VM or LXC container, a linked clone of a template, destroyed when the job ends. Short jobs can instead run in pods: one long-lived VM that runs one throwaway container per job.
Mayfly is one static binary and a set of systemd units on the Proxmox host. It requires a GitHub App, a thin-provisioned storage pool (LVM-thin, ZFS or Ceph) and a network bridge.
On the Proxmox host, as root:
curl -fsSL https://github.com/variiuz/mayfly/releases/latest/download/install.sh | bash
The installer downloads the latest release, verifies it against SHA256SUMS, installs mayfly and its
units and creates /etc/mayfly. It starts no unit and touches no guest. Run it again to update;
configuration, slots and running jobs are kept.
-
Create a GitHub App without a webhook. Permissions: Self-hosted runners: read and write for organization runners, or Administration: read and write for repository runners. Install it on the organization or repository, generate a private key and copy it to the host:
install -m 600 my-app.private-key.pem /etc/mayfly/app.pem -
In
/etc/mayfly/pool.env, setAPP_ID, eitherORG=<org>orREPO=<owner>/<name>, andSTORAGE(thin pool),BRIDGEandVLANfor job guests. All other options have defaults; see pool.env.example. -
In
/etc/mayfly/slots.conf, add one line per runner slot: name, VMID, labels, vCPU, RAM, disk. See slots.conf.example. -
Check the host, build a template and start the slots:
mayfly check # App, storage and network mayfly template build # Ubuntu 24.04 with runner, Docker, gh; 20 to 40 min journalctl -fu mayfly-template-build mayfly template promote 9097 # VMID printed by the build mayfly enable # starts all slots and the gc timer mayfly status -
Target the slot's labels in a workflow:
runs-on: [self-hosted, linux]
A setup is working when:
- Runners register with the base labels plus the slot's labels and are removed after their job.
ACTIONS_RUNNER_INPUT_JITCONFIGis not present in a job's environment.mayfly statusand thin pool usage stay stable over a day of jobs.mayfly statsafter a day shows no stuck guests and no errors.
| Command | Does |
|---|---|
mayfly check |
checks App key, storage, network and template; changes nothing |
mayfly template build|canary|promote|list |
see Templates |
mayfly enable [slot ...] |
starts slots (all when none is named) and the gc timer |
mayfly disable [slot ...] |
stops slots; running jobs finish first |
mayfly status |
each slot's service, guest, job and runner state on GitHub (online, busy, offline, none); pool and RAM |
mayfly stats [hours] |
per-slot counts from the journal, last 24 hours by default |
mayfly gc |
removes stale runner registrations, orphan guests and old snippets |
mayfly update |
see Updates |
mayfly help, mayfly version |
Logs: journalctl -u 'mayfly-slot@*' -f.
mayfly stats counts guests destroyed, jobs run, guests replaced idle (MAX_IDLE_HOURS) or stuck
(runner not online), jobs cut off (MAX_JOB_HOURS, the backstop, a full pool), errors (clone, start,
GitHub registration) and the median life of a guest that ran a job. Pods count their VMs and errors
only.
- The slot requests a single-job (JIT) runner config from GitHub using the App. The App key stays on the host.
- The config is written to a cloud-init snippet (mode 0600). The template is cloned to the slot's VMID with the slot's vCPU, RAM (no balloon, no swap), disk, network and firewall settings, and started. The snippet is deleted once the VM is running.
- In the guest,
mayfly-jobreads and deletes the config, runs the job asrunner(no sudo) and powers off. If the runner is not online on GitHub within 10 minutes, the slot replaces the guest, runs thevm-stuckhook and backs off (2 min per repeat, at most 30 min). - The slot destroys the VM and its disk, removes the runner registration if it still exists, and starts the next clone.
Host settings, documented in pool.env.example. The file is read as plain
KEY=value lines, optionally quoted, and never run as shell: $VAR, $(...) and other commands are
refused, and mayfly check lists such lines.
One line per slot:
# name vmid labels vcpu ram_gb disk_gb env template [key=value ...]
build-1 9002 linux 8 16 60 cache current
small-1 9011 small 2 4 30 no current max_job_hours=1
pods-1 9041 small 12 36 120 no current kind=pod containers=6
labelsare appended toRUNNER_LABELS_BASE(self-hosted,Linux,X64).envis a comma-separated list of job environments, orno. Each is a file/etc/mayfly/env/<name>.envofKEY=valuelines (mode 0600), passed to the job through cloud-init and deleted from the guest after it is read. Release slots should not use a cache that PR jobs can write to.templateiscurrent, ornextfor canary slots that use the template under test.key=valuefields overridepool.envfor that slot:kind=vm|pod|lxc,containers=N,template=<vmid>(pin a template or use another profile),bridge=,vlan=,cpu=,disk_cache=,max_job_hours=,max_idle_hours=.
A template consists of the core (runner, Docker, gh, single-job service) and a profile. A profile is a directory with:
provision.sh, run as root in the template guest from/opt/mayfly-profile(readable forrunner, removed when the template is sealed);template.env, with versions andPREPULL_IMAGES.
The shipped default profile installs build-essential, pkg-config, OpenSSL headers and shellcheck.
Profiles in /etc/mayfly/profiles/<name>/ take precedence over shipped profiles of the same name.
PROFILE in pool.env sets the default; mayfly template build <name> builds another. Slots can use
templates of different profiles via template=<vmid>.
Executables in HOOK_DIR (/etc/mayfly/hooks.d) are called with:
vm-up <slot> <vmid> <runner>vm-gone <slot> <vmid> <minutes>vm-stuck <slot> <vmid> <runner>(the runner was not online on GitHub within 10 min)pod-up <slot> <vmid> <containers>pod-gone <slot> <vmid> <minutes>
Each hook has a 30 s timeout. Failures are logged and otherwise ignored.
ORGregisters organization runners inRUNNER_GROUP_ID;REPO=owner/nameregisters repository runners.- For GitHub Enterprise Server, set
API_URLtohttps://<host>/api/v3. - Runners are named
<RUNNER_PREFIX>-<slot>-<epoch>.mayfly gcremoves only offline registrations with that prefix and no matching guest.
| Command | Does |
|---|---|
mayfly template build [profile] |
builds into a free template VMID; --lxc builds a container template |
mayfly template canary <vmid> |
points slots marked next at it |
mayfly template promote <vmid> |
switches all slots at their next job |
mayfly template list |
lists templates |
- The build runs in the systemd unit
mayfly-template-build, so it survives the SSH session; one build runs at a time.--foregroundruns it in the terminal instead. - The previous template is kept; roll back with
mayfly template promote <old>. canaryandpromoteaccept only templates built bymayfly template build(descriptionprofile <name>);mayfly checkreports a current or next template that was not.- The template disk size is
TEMPLATE_DISKGB. Set it to the largest slot disk: clones keep the template size and are resized only whendisk_gbis larger, because growing a disk makes ext4 in the guest zero-fill new inode tables (about 32 MB per GB added). Unwritten space in a thin pool is not allocated.
For jobs of one to two minutes, the host CPU spent on qm, pvesh and LVM calls for a per-job VM
exceeds the job itself. A slot with kind=pod containers=N is a long-lived VM, cloned from the same
template, that runs N containers, each for a single job. The slot's vCPU, RAM and disk apply to the
whole pod; each container is limited to vcpu/N CPUs and ram/N GB.
- Runner configs: every 15 s the host queries the guest agent for idle container loops, creates one JIT
config per idle loop, and passes each with
qm guest exec ... mayfly-pod put. The guest stores it in/run/mayfly-pod/spool(mode 0600); an idle loop takes it and starts a container with it as an environment variable. The App key stays on the host. - Image: at boot, the guest imports its root filesystem (excluding Docker data, logs and keys) as the
image
mayfly-pod:local, so containers have the template's tools. This takes a few minutes per pod. - Isolation: containers run as
runner, unprivileged, with all capabilities dropped,no-new-privileges, no Docker socket, a separate network namespace without container-to-container traffic, CPU, memory and pid limits, and scratch space removed with the container. Jobs that need a Docker daemon or service containers cannot run in pods. - Recycling: the host drains, destroys and reclones a pod when its VM stops, when the guest agent has
not responded for 5 min, after
POD_MAX_AGE_HOURS(24, plus 0 to 3.5 h offset by VMID), and when its template changes. Draining lets running jobs finish withinMAX_JOB_HOURS;systemctl stopdrains as well. - Startup: a pod that is not ready within 30 min is replaced; each repeat waits longer before the next clone (2 min per repeat, at most 30 min).
To move back from pods to VMs, restore the VM slots in slots.conf, mayfly enable them and
mayfly disable the pod slots.
A slot with kind=lxc behaves like a VM slot but clones an unprivileged LXC container template: one
container per job, destroyed afterwards. Containers start in about a second and cost the host less than
a VM boot. Docker works inside.
ct-1 9051 small 2 4 20 no current kind=lxc
- Template:
mayfly template build --lxc [profile]creates a container from the newestLXC_OSTEMPLATEimage (ubuntu-24.04-standard, downloaded withpveam), unprivileged withnestingandkeyctlenabled, and provisions it with the same core and profile as a VM template.canaryandpromotedetect container templates and setNEXT_LXC_TEMPLATE_VMID/LXC_TEMPLATE_VMID. VM and container templates shareTEMPLATE_VMIDS; allocate five VMIDs when using both. - Runner config: containers have no cloud-init. After boot, the host writes the JIT config and job
environment into the container via
lxc-attachstdin (not to the host disk or a command line) and startsmayfly-job. The App key stays on the host. - Limits:
vcpuandram_gbset cores and memory (no swap);disk_gbsets the rootfs size, grown only if larger than the template's. The per-guest firewall applies.cpu=anddisk_cache=are ignored. - Isolation: containers share the host kernel. An unprivileged container with user namespaces is a weaker boundary than a VM; a kernel vulnerability reachable from inside affects the host. Use LXC slots for trusted code (for example, private repositories of your own organization) and VM slots for untrusted code such as pull requests from forks.
- Storage: linked clones require LVM-thin, ZFS or Ceph. Docker overlay2 in a container on ZFS requires ZFS 2.2 or later.
- Scope: Mayfly stops or destroys only guests that have the tag
TAG(mayfly), a name starting withNAME_PREFIX(ci-) and a VMID inVMID_MIN..VMID_MAX. During a migration from another tool, guests with a tag inLEGACY_TAGSand aci-name are included. No other guests are touched. - Capacity: no new job guest is created while the thin pool is above
MAX_POOL_PERCENT(70) or while it would leave less thanMIN_FREE_RAM_GB(8) for the host. The slot logs the reason. - Time: a job running longer than
MAX_JOB_HOURS(3) is stopped and its guest destroyed. A guest without a job forMAX_IDLE_HOURS(2) is replaced. - Full pool: all job guests are destroyed while the pool is above
POOL_PANIC_PERCENT(85). - Firewall (
FIREWALL=yes, per guest): no inbound traffic; outbound only DNS toGATEWAY,CACHE_IP,ALLOW_OUT, and destinations outsideBLOCK_NETS(private ranges, Tailscale, link-local). Job guests reach the internet but not each other or the LAN. This requires the cluster firewall to be enabled (mayfly checkreports it). Before enabling the cluster firewall, allow SSH and port 8006 from your admin network: its default input policy drops all other traffic to the host.
For hosts without a VLAN for job guests:
host/pve-on-debian.shinstalls Proxmox on Debian 13 (two stages, each followed by a reboot).host/nat-bridge.shaddsvmbr1(10.66.0.0/24, or setBRIDGE=andNET=) with dnsmasq, forwarding and nftables isolation (mayfly-nat.service).- Set
BRIDGE=vmbr1,VLAN=andGATEWAY=10.66.0.1inpool.env.
Use bare metal: on nested Proxmox, fsync latency breaks file-lock and timing tests, even with
DISK_CACHE=unsafe.
mayfly update # latest release
mayfly update --check # only say whether a newer release exists
mayfly update --version 0.3.1 # that release (also an older one, or one that was skipped)
mayfly update --rollback # restore previous/ (the version before the last update)
mayfly update:
- downloads
mayfly-<version>.tar.gzandSHA256SUMSfrom the GitHub release (no token) and checks the checksum; - runs the new
mayfly version; - saves the installed files in
/usr/local/share/mayfly/previous/; - runs the new
mayfly install --from <release>; - runs the new
mayfly check. If the old one passed and the new one fails, everything is restored, the version is skipped and the exit code is 1.
Running slots are not restarted and no job is cancelled. A slot switches to the new command by itself
once its current guest is gone: a VM or container slot after its job (or after MAX_IDLE_HOURS without
one), a pod slot when its VM is recycled (POD_MAX_AGE_HOURS). The slot logs mayfly was updated.
Up to 0.9.0 a Bash script, mayfly.sh, ran the slots, or with engine=rust / SLOT_ENGINE=rust the
binary did. Now the binary always does, with the same calls and the same log lines. Updating needs
nothing from you:
- The script is removed.
mayfly update --rollbackbrings it back with the old version. - Slots that are running are restarted once, because loops of those versions cannot switch by themselves. A restart drains: an idle guest is replaced, a job in progress finishes first.
engine=inslots.confandSLOT_ENGINEinpool.envdo nothing.mayfly checklists them until you remove them.
mayfly-update.timer runs mayfly update --auto daily, up to an hour late. The first install that has
the timer switches it on. Log: journalctl -u mayfly-update.
UPDATE in pool.env sets what --auto does; mayfly update by hand ignores it.
UPDATE |
--auto installs |
|---|---|
auto (default) |
any newer release |
patch |
only a newer patch of the installed major.minor |
off |
nothing; the timer is not switched on either |
--auto never installs a release whose notes have a BREAKING CHANGES section, nor one with such a
release between it and the installed version. It logs that, records last_result=breaking and leaves
the update to mayfly update by hand (--check says so too). It also waits when the release list
cannot be read.
/var/lib/mayfly/update.state holds last_attempt, last_result (ok, up-to-date, rolled-back,
failed, breaking) and skipped=<version>. A skipped version is not installed again by the timer or
by a plain mayfly update; mayfly update --version <version> forces it. mayfly update --rollback
skips the version it removed. mayfly status ends with the installed version and the last attempt.
| File | Installed to | Purpose |
|---|---|---|
mayfly |
/usr/local/sbin |
the command |
mayfly-slot@.service |
systemd | one instance per slot in slots.conf: mayfly slot <name> |
mayfly-gc.{service,timer} |
systemd | every 15 min: mayfly gc |
mayfly-update.{service,timer} |
systemd | daily: mayfly update --auto |
pool.env.example, slots.conf.example |
/etc/mayfly/ |
host settings and slots, with all options documented |
template/ |
/usr/local/share/mayfly, then the template guest |
core provisioning (runner, Docker, gh), single-job service, pod script |
profiles/<name>/ |
/usr/local/share/mayfly/profiles, then the template guest |
additional toolchains, libraries and images |
host/pve-on-debian.sh, host/nat-bridge.sh |
new host, once | Proxmox on Debian; NAT bridge for job guests |
Tests and other installs can move the paths: MAYFLY_SBIN, MAYFLY_SHARE, MAYFLY_SYSTEMD_DIR,
MAYFLY_CONF_DIR, MAYFLY_STATE_DIR, MAYFLY_TMP (default /var/tmp), MAYFLY_RELEASE_API and
MAYFLY_REPO_URL.
Releases are managed by release-please. Merging the
release PR tags the version and attaches mayfly-<version>.tar.gz, the binary on its own as
mayfly-<version>-x86_64-linux, a version-pinned install.sh and SHA256SUMS.
To install a specific version:
curl -fsSL https://github.com/variiuz/mayfly/releases/download/v0.1.0/install.sh | bash
To install from a checkout:
scp -r mayfly root@HOST:/opt/mayfly-src && ssh root@HOST bash /opt/mayfly-src/install.sh
The binary is built with cargo when the host has it; otherwise the released binary of the checkout's
version is downloaded and checked against SHA256SUMS. install.sh only finds and checks the files;
mayfly install [--no-timer] [--from <dir>] does the rest (<dir> is a release or checkout directory;
by default the directory of the binary).
Tests run without Proxmox.
| Command | Checks |
|---|---|
cargo test |
every command against a fake Proxmox host and a fake GitHub |
bash tests/e2e.sh |
the built binary running a slot of each kind against stub commands |
bash tests/update.sh <dist> |
an older release updating itself to this build, and rolling back |
tests/e2e.sh compares every call, file, request and log line of a run with tests/golden/. Those
recordings were made from the Bash loop of 0.9.0, so they hold the binary to what that loop did. After
an intended change, UPDATE_GOLDEN=1 bash tests/e2e.sh records them anew; review the diff.