Add Ansible for the pullab_cloud AWS poller - #268
Open
nuclearcat wants to merge 1 commit into
Open
Conversation
The KernelCI pull-labs AWS integration (kernelci/pullab_cloud) was set
up by hand on kci-aws and has been running since June from a detached
root screen session looping loop.sh, with the pipeline configs and
wrapper scripts untracked on the host and the API/KCIDB token inlined in
prod-amd64.sh / prod-arm64.sh.
Nothing about it was written down, sorry for that.
So i capture the deployment (and modified things nicely, so some need
redeploy, like script in screen now proper systemd service).
- roles/pullab_cloud provisions the host (python3.11, awscli-2, docker,
checkout, editable install) and replaces the screen loop with
pullab-poller@{amd64,arm64} oneshot units on 300s timers. Same
cadence, but it survives reboots, logs to the journal, bounds a wedged
cycle with TimeoutStartSec, and reads the token from a 0600
environment file instead of an executable script. The two
architectures now run in parallel rather than back-to-back; they have
disjoint platform filters and separate cursor files, so they do not
interact.
- config.json.j2 reproduces config-{amd64,arm64}-prod.json. The rendered
output was diffed against the live files and is identical apart from
the free-text comment field, where the deployed amd64 config describes
itself as the arm64 variant.
- roles/pullab_aws_infra declares the eu-west-2 resources: the
kernelci-results bucket, kernelci-ecr, kernelci-cluster and the two
CloudWatch log groups. Additive only, and a no-op unless run with
pullab_infra_apply=true. The VM IAM role is deliberately left out
because pullab_cloud recreates it on every run.
- Secrets stay out of the tree: group_vars/vault.yml is gitignored and
only an example is committed. Both the poller token and the AWS keys
are asserted before the role touches the host, so a run either has
what it needs or stops on the control node having changed nothing. The
AWS assert is conditional on pullab_manage_aws_credentials, which
keeps the IAM instance-profile path usable with no static keys.
- This repo is public, so identifying detail is kept out of it too. The
AWS account id lives in the vault alongside the credentials, with
aws-infra.yml asserting on it before its first API call so a stray
AWS_PROFILE cannot target the wrong account. The inventory carries no
address: the host name matches an ~/.ssh/config alias, and anyone
without one sets ansible_host locally.
Two things in the deployed config look meaningful but are not, and are
documented so nobody acts on them, but they might become TODO after
rethinking:
- external_storage names "kernelci-storage", which is the KernelCI
storage service (Azure-backed, behind storage.kernelci.org), not an S3
bucket, and no such bucket exists. The field is only read through
s3.copy_object/list_objects_v2, so it could never have addressed that
service. Kernel artifacts reach the VM another way entirely: every
test type maps to the url-kernel-boot vm-test, which curls
artifacts.kernel/modules/rootfs from the job definition inside the VM.
Across 39k+ run logs the only test to reach the external-requirements
path is url-kernel-boot (1604 times), whose external_requirements.json
is all-false. It is latent, not broken; creating an S3 bucket by that
name would only add a second unrelated thing sharing it.
- test_config.vms is a template, not the job list. pull_labs_translate
replaces test_config wholesale per job and keeps only role_name, so
the simple-unixbench / unixbench-kernel-regression entries never run.
What jobs inherit from the file is the VM shape. For the same reason
poll_interval_sec is dead under --once; the real cadence was the
sleep.
Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
nuclearcat
force-pushed
the
pullab-cloud
branch
from
August 31, 2026 05:40
cf2b910 to
419af96
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The KernelCI pull-labs AWS integration (kernelci/pullab_cloud) was set up by hand on kci-aws and has been running since June from a detached root screen session looping loop.sh, with the pipeline configs and wrapper scripts untracked on the host and the API/KCIDB token inlined in prod-amd64.sh / prod-arm64.sh.
Nothing about it was written down, sorry for that.
So i capture the deployment (and modified things nicely, so some need redeploy, like script in screen now proper systemd service).
roles/pullab_cloud provisions the host (python3.11, awscli-2, docker, checkout, editable install) and replaces the screen loop with pullab-poller@{amd64,arm64} oneshot units on 300s timers. Same cadence, but it survives reboots, logs to the journal, bounds a wedged cycle with TimeoutStartSec, and reads the token from a 0600 environment file instead of an executable script. The two architectures now run in parallel rather than back-to-back; they have disjoint platform filters and separate cursor files, so they do not interact.
config.json.j2 reproduces config-{amd64,arm64}-prod.json. The rendered output was diffed against the live files and is identical apart from the free-text comment field, where the deployed amd64 config describes itself as the arm64 variant.
roles/pullab_aws_infra declares the eu-west-2 resources: the kernelci-results bucket, kernelci-ecr, kernelci-cluster and the two CloudWatch log groups. Additive only, and a no-op unless run with pullab_infra_apply=true. The VM IAM role is deliberately left out because pullab_cloud recreates it on every run.
Secrets stay out of the tree: group_vars/vault.yml is gitignored and only an example is committed. Both the poller token and the AWS keys are asserted before the role touches the host, so a run either has what it needs or stops on the control node having changed nothing. The AWS assert is conditional on pullab_manage_aws_credentials, which keeps the IAM instance-profile path usable with no static keys.
This repo is public, so identifying detail is kept out of it too. The AWS account id lives in the vault alongside the credentials, with aws-infra.yml asserting on it before its first API call so a stray AWS_PROFILE cannot target the wrong account. The inventory carries no address: the host name matches an ~/.ssh/config alias, and anyone without one sets ansible_host locally.
Two things in the deployed config look meaningful but are not, and are documented so nobody acts on them, but they might become TODO after rethinking:
external_storage names "kernelci-storage", which is the KernelCI storage service (Azure-backed, behind storage.kernelci.org), not an S3 bucket, and no such bucket exists. The field is only read through s3.copy_object/list_objects_v2, so it could never have addressed that service. Kernel artifacts reach the VM another way entirely: every test type maps to the url-kernel-boot vm-test, which curls artifacts.kernel/modules/rootfs from the job definition inside the VM. Across 39k+ run logs the only test to reach the external-requirements path is url-kernel-boot (1604 times), whose external_requirements.json is all-false. It is latent, not broken; creating an S3 bucket by that name would only add a second unrelated thing sharing it.
test_config.vms is a template, not the job list. pull_labs_translate replaces test_config wholesale per job and keeps only role_name, so the simple-unixbench / unixbench-kernel-regression entries never run. What jobs inherit from the file is the VM shape. For the same reason poll_interval_sec is dead under --once; the real cadence was the sleep.