Skip to content

Add Ansible for the pullab_cloud AWS poller - #268

Open
nuclearcat wants to merge 1 commit into
kernelci:mainfrom
nuclearcat:pullab-cloud
Open

Add Ansible for the pullab_cloud AWS poller#268
nuclearcat wants to merge 1 commit into
kernelci:mainfrom
nuclearcat:pullab-cloud

Conversation

@nuclearcat

Copy link
Copy Markdown
Member

The KernelCI pull-labs AWS integration (kernelci/pullab_cloud) was set up by hand on kci-aws and has been running since June from a detached root screen session looping loop.sh, with the pipeline configs and wrapper scripts untracked on the host and the API/KCIDB token inlined in prod-amd64.sh / prod-arm64.sh.
Nothing about it was written down, sorry for that.

So i capture the deployment (and modified things nicely, so some need redeploy, like script in screen now proper systemd service).

  • roles/pullab_cloud provisions the host (python3.11, awscli-2, docker, checkout, editable install) and replaces the screen loop with pullab-poller@{amd64,arm64} oneshot units on 300s timers. Same cadence, but it survives reboots, logs to the journal, bounds a wedged cycle with TimeoutStartSec, and reads the token from a 0600 environment file instead of an executable script. The two architectures now run in parallel rather than back-to-back; they have disjoint platform filters and separate cursor files, so they do not interact.

  • config.json.j2 reproduces config-{amd64,arm64}-prod.json. The rendered output was diffed against the live files and is identical apart from the free-text comment field, where the deployed amd64 config describes itself as the arm64 variant.

  • roles/pullab_aws_infra declares the eu-west-2 resources: the kernelci-results bucket, kernelci-ecr, kernelci-cluster and the two CloudWatch log groups. Additive only, and a no-op unless run with pullab_infra_apply=true. The VM IAM role is deliberately left out because pullab_cloud recreates it on every run.

  • Secrets stay out of the tree: group_vars/vault.yml is gitignored and only an example is committed. Both the poller token and the AWS keys are asserted before the role touches the host, so a run either has what it needs or stops on the control node having changed nothing. The AWS assert is conditional on pullab_manage_aws_credentials, which keeps the IAM instance-profile path usable with no static keys.

  • This repo is public, so identifying detail is kept out of it too. The AWS account id lives in the vault alongside the credentials, with aws-infra.yml asserting on it before its first API call so a stray AWS_PROFILE cannot target the wrong account. The inventory carries no address: the host name matches an ~/.ssh/config alias, and anyone without one sets ansible_host locally.

Two things in the deployed config look meaningful but are not, and are documented so nobody acts on them, but they might become TODO after rethinking:

  • external_storage names "kernelci-storage", which is the KernelCI storage service (Azure-backed, behind storage.kernelci.org), not an S3 bucket, and no such bucket exists. The field is only read through s3.copy_object/list_objects_v2, so it could never have addressed that service. Kernel artifacts reach the VM another way entirely: every test type maps to the url-kernel-boot vm-test, which curls artifacts.kernel/modules/rootfs from the job definition inside the VM. Across 39k+ run logs the only test to reach the external-requirements path is url-kernel-boot (1604 times), whose external_requirements.json is all-false. It is latent, not broken; creating an S3 bucket by that name would only add a second unrelated thing sharing it.

  • test_config.vms is a template, not the job list. pull_labs_translate replaces test_config wholesale per job and keeps only role_name, so the simple-unixbench / unixbench-kernel-regression entries never run. What jobs inherit from the file is the VM shape. For the same reason poll_interval_sec is dead under --once; the real cadence was the sleep.

The KernelCI pull-labs AWS integration (kernelci/pullab_cloud) was set
up by hand on kci-aws and has been running since June from a detached
root screen session looping loop.sh, with the pipeline configs and
wrapper scripts untracked on the host and the API/KCIDB token inlined in
prod-amd64.sh / prod-arm64.sh.
Nothing about it was written down, sorry for that.

So i capture the deployment (and modified things nicely, so some need
redeploy, like script in screen now proper systemd service).

- roles/pullab_cloud provisions the host (python3.11, awscli-2, docker,
  checkout, editable install) and replaces the screen loop with
  pullab-poller@{amd64,arm64} oneshot units on 300s timers. Same
  cadence, but it survives reboots, logs to the journal, bounds a wedged
  cycle with TimeoutStartSec, and reads the token from a 0600
  environment file instead of an executable script. The two
  architectures now run in parallel rather than back-to-back; they have
  disjoint platform filters and separate cursor files, so they do not
  interact.

- config.json.j2 reproduces config-{amd64,arm64}-prod.json. The rendered
  output was diffed against the live files and is identical apart from
  the free-text comment field, where the deployed amd64 config describes
  itself as the arm64 variant.

- roles/pullab_aws_infra declares the eu-west-2 resources: the
  kernelci-results bucket, kernelci-ecr, kernelci-cluster and the two
  CloudWatch log groups. Additive only, and a no-op unless run with
  pullab_infra_apply=true. The VM IAM role is deliberately left out
  because pullab_cloud recreates it on every run.

- Secrets stay out of the tree: group_vars/vault.yml is gitignored and
  only an example is committed. Both the poller token and the AWS keys
  are asserted before the role touches the host, so a run either has
  what it needs or stops on the control node having changed nothing. The
  AWS assert is conditional on pullab_manage_aws_credentials, which
  keeps the IAM instance-profile path usable with no static keys.

- This repo is public, so identifying detail is kept out of it too. The
  AWS account id lives in the vault alongside the credentials, with
  aws-infra.yml asserting on it before its first API call so a stray
  AWS_PROFILE cannot target the wrong account. The inventory carries no
  address: the host name matches an ~/.ssh/config alias, and anyone
  without one sets ansible_host locally.

Two things in the deployed config look meaningful but are not, and are
documented so nobody acts on them, but they might become TODO after
rethinking:

- external_storage names "kernelci-storage", which is the KernelCI
  storage service (Azure-backed, behind storage.kernelci.org), not an S3
  bucket, and no such bucket exists. The field is only read through
  s3.copy_object/list_objects_v2, so it could never have addressed that
  service. Kernel artifacts reach the VM another way entirely: every
  test type maps to the url-kernel-boot vm-test, which curls
  artifacts.kernel/modules/rootfs from the job definition inside the VM.
  Across 39k+ run logs the only test to reach the external-requirements
  path is url-kernel-boot (1604 times), whose external_requirements.json
  is all-false. It is latent, not broken; creating an S3 bucket by that
  name would only add a second unrelated thing sharing it.

- test_config.vms is a template, not the job list. pull_labs_translate
  replaces test_config wholesale per job and keeps only role_name, so
  the simple-unixbench / unixbench-kernel-regression entries never run.
  What jobs inherit from the file is the VM shape. For the same reason
  poll_interval_sec is dead under --once; the real cadence was the
  sleep.

Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant