diff --git a/playbooks/pullab-cloud/.gitignore b/playbooks/pullab-cloud/.gitignore new file mode 100644 index 0000000..b805b35 --- /dev/null +++ b/playbooks/pullab-cloud/.gitignore @@ -0,0 +1,2 @@ +group_vars/vault.yml +*.retry diff --git a/playbooks/pullab-cloud/README.md b/playbooks/pullab-cloud/README.md new file mode 100644 index 0000000..bb5a47f --- /dev/null +++ b/playbooks/pullab-cloud/README.md @@ -0,0 +1,155 @@ +# pullab_cloud — KernelCI pull-labs on AWS + +Ansible for [`kernelci/pullab_cloud`](https://github.com/kernelci/pullab_cloud): +the poller that consumes `pull-labs-aws-ec2` jobs from the KernelCI API, runs +them on EC2 VMs, and reports results back. + +This codifies a deployment that was set up by hand. It was written from the +live state of `kci-aws` as of 2026-08-31; the "As deployed today" section below +records what was actually found, including the parts that are not good. + +## Layout + +| Path | Purpose | +| --- | --- | +| `site.yml` | Provision the poller host over SSH | +| `aws-infra.yml` | Create/verify AWS resources (runs locally against the API) | +| `group_vars/all.yml` | Every resource name, region and tuning knob | +| `group_vars/vault.yml.example` | Template for the secrets, to be vaulted | +| `roles/pullab_cloud/` | Packages, checkout, config, systemd units | +| `roles/pullab_aws_infra/` | S3 / ECR / ECS / CloudWatch bootstrap | +| `requirements.yml` | Galaxy collections needed by `aws-infra.yml` | + +## Usage + +```sh +cp group_vars/vault.yml.example group_vars/vault.yml +$EDITOR group_vars/vault.yml +ansible-vault encrypt group_vars/vault.yml + +# always dry-run first +ansible-playbook site.yml --check --diff --ask-vault-pass +ansible-playbook site.yml --ask-vault-pass + +# AWS resources: no-op unless explicitly enabled +ansible-playbook aws-infra.yml --check --diff +ansible-playbook aws-infra.yml -e pullab_infra_apply=true +``` + +To deploy new code, add `-e pullab_update_checkout=true` — otherwise the +checkout is left alone so a routine run never pulls `main` from under a +running poll cycle. + +`aws-infra.yml` needs the AWS collections; `site.yml` does not: + +```sh +ansible-galaxy collection install -r requirements.yml +``` + +## As deployed today (2026-08-31) + +**Poller host** — `kci-aws`, a `t3.micro` in +**us-east-1**, Amazon Linux 2023, user `ec2-user`. Note the split: the host is +in us-east-1, everything it drives is in **eu-west-2**. That works only because +the pipeline config carries an explicit `region`; `/root/.aws/config` says +`us-east-1`. + +**How it runs** — a detached root `screen` session (`929013.pts-1`) running +`/root/pullab_cloud/loop.sh`, alive since early June: + +```sh +while true; do + ./prod-amd64.sh + ./prod-arm64.sh + sleep 300 +done +``` + +Each `prod-*.sh` is a one-liner that inlines `UNIFIED_TOKEN=` and calls +`pull_labs_poller --config examples/aws/config--prod.json --once`. + +`site.yml` replaces this with `pullab-poller@amd64` / `pullab-poller@arm64` +oneshot services driven by 300s timers — same cadence, but restarts on reboot, +logs to the journal, and keeps the token out of an executable script. + +**Not in git** — `loop.sh`, `prod-amd64.sh`, `prod-arm64.sh`, +`config-amd64-prod.json` and `config-arm64-prod.json` are all untracked in the +checkout on the host, as is a local modification to `config.json`. The two prod +configs are reproduced by `roles/pullab_cloud/templates/config.json.j2`. + +**Python** — `python3.11` (AL2023's `python3` is 3.9 and unsupported). +`kernel-ci-cloud-labs` is pip-installed editable into the *system* python3.11, +not into the `.venv` that also exists in the checkout; the scripts additionally +set `PYTHONPATH=src`. Both are reproduced as-is. + +### AWS resources (eu-west-2) + +| Kind | Name | Notes | +| --- | --- | --- | +| S3 | `kernelci-results` | Results, prefix `results/` | +| S3 | `kernelci-storage` | Named in `external_storage`, but no such bucket exists — see "How kernel artifacts actually arrive" | +| ECR | `kernelci-ecr` | Fargate test image | +| ECS | `kernelci-cluster` | Task family `kernelci-task`, 1024 CPU / 2048 MB | +| IAM | `kernelci-ecs-role` (+ instance profile) | Recreated on every run by the tool | +| Logs | `/ecs/kernelci-task` | 7 day retention | +| Logs | `/ec2/kernelci-vms` | 3 day retention | + +A parallel `kernel-ci-exampleuser-*` set (bucket, cluster, ECR repo, IAM role, +log groups) also exists from the default `--prefix kernel-ci-$USER-` in the +upstream quickstart. It is not used by the prod pollers and is left alone here. + +**Test matrix** — `test_config.vms` in the configs is a *template, not the +job list*. `pull_labs_translate.py` rewrites `test_config` wholesale for every +job (`"vms": [vm_entry]`), carrying over only `role_name`, so the +`simple-unixbench` / `unixbench-kernel-regression` entries never actually run. +What each job inherits from the file is the shape: 40 GB root, 3600s max +runtime, `c5a.4xlarge` on amd64 and `c7g.4xlarge` on arm64, both on the latest +AL2023 AMI resolved via SSM. + +**How kernel artifacts actually arrive** — not over S3. Every test type +(`baseline`, `ltp`, `unixbench`, and anything unrecognised) maps to the +`url-kernel-boot` vm-test, which receives `artifacts.kernel`, +`artifacts.modules` and `artifacts.rootfs` from the job definition as +`KERNEL_URL` / `MODULES_URL` / `ROOTFS_URL` and `curl`s them **inside the VM**. +Those URLs point at the KernelCI storage server (the Azure-backed service in +`kernelci/kernelci-storage`, behind storage.kernelci.org) — nothing to do with +AWS. + +This is why the `external_storage` bucket does not exist: the S3-bucket field +was filled in with the *name of the KernelCI storage service*, which is a +different thing that happens to share the name. `external_storage` is only ever +read through `s3.copy_object` / `list_objects_v2`, so it could not reach an HTTP +service even if it were meant to. Across all 39k+ run logs the only test that +ever reaches that code path is `url-kernel-boot` (1604 times), whose +`external_requirements.json` is all-`false` — so the bucket is never contacted. +Leave it alone; do not create an S3 bucket by that name. + +## Known problems this does not fix + +These are recorded deliberately — each needs a decision, not a playbook. + +1. **Root-account access keys.** `/root/.aws/credentials` holds keys for + `arn:aws:iam:::root`. Every VM, bucket and role in the account + is reachable with them. Replace with a scoped IAM user, or attach an + instance profile to the poller host and set + `pullab_manage_aws_credentials: false`. +2. **Plaintext JWT on disk.** The `UNIFIED_TOKEN` in `prod-*.sh` is committed + to no repo but sits world-readable-by-root in an executable, and is visible + in `ps` while a poll runs. It expires 2031. It should be rotated once the + vault-managed environment file is in use. +3. **`external_storage` is inert and misleadingly named.** It points at + `kernelci-storage`, which is the KernelCI storage *service*, not an S3 + bucket (see above). Nothing reads it today, so this is latent, not broken. + It only starts to matter if someone wants the kernel-install tests + (`unixbench-kernel-regression`, `example-kernel-reboot-test`, + `simple-source-reboot`) — those need real RPMs under `kernel-rpms/binary/` + in a genuine S3 bucket, which should then be given a name that does not + collide with the storage service. The IAM policy also grants S3 access to + `arn:aws:s3:::kernelci-storage*`, which is equally inert. +4. **Per-run log groups never expire.** `/ec2/kernelci-vms/run_*` groups are + created without retention and there are thousands of them. Retention is only + set on the two parent groups. +5. **39k+ log directories** under `/root/pullab_cloud/logs/` on a `t3.micro`. + No rotation. +6. **Single point of failure.** One `t3.micro`, one screen session, no + monitoring. The systemd units help; alerting does not exist. diff --git a/playbooks/pullab-cloud/ansible.cfg b/playbooks/pullab-cloud/ansible.cfg new file mode 100644 index 0000000..6e581cd --- /dev/null +++ b/playbooks/pullab-cloud/ansible.cfg @@ -0,0 +1,4 @@ +[defaults] +inventory = inventory.yaml +host_key_checking = True +callback_result_format = yaml diff --git a/playbooks/pullab-cloud/aws-infra.yml b/playbooks/pullab-cloud/aws-infra.yml new file mode 100644 index 0000000..03d4a34 --- /dev/null +++ b/playbooks/pullab-cloud/aws-infra.yml @@ -0,0 +1,15 @@ +# Create/verify the AWS resources the pullab_cloud pipeline runs against. +# +# ansible-playbook -i inventory.yaml aws-infra.yml +# +# Runs locally against the AWS API, not over SSH. Idempotent and additive: it +# never deletes buckets, images or log groups. The IAM role for the test VMs is +# deliberately NOT managed here — pullab_cloud recreates it on every run +# ("force_recreate_roles": true in the pipeline config), so Ansible would only +# fight it. Check with --check --diff first. +- name: pullab_cloud AWS infrastructure + hosts: localhost + connection: local + gather_facts: false + roles: + - pullab_aws_infra diff --git a/playbooks/pullab-cloud/group_vars/all.yml b/playbooks/pullab-cloud/group_vars/all.yml new file mode 100644 index 0000000..d90ed28 --- /dev/null +++ b/playbooks/pullab-cloud/group_vars/all.yml @@ -0,0 +1,74 @@ +--- +# Non-secret description of the running pullab_cloud deployment. +# Captured from the kci-aws poller host on 2026-08-31. + +# --- AWS account / regions ------------------------------------------------ +# The poller host runs in us-east-1; every resource it drives lives in +# eu-west-2. Nothing relies on the host's default region, because the pipeline +# config carries an explicit "region" key. +# +# pullab_aws_account_id is deliberately not here - it lives in the gitignored +# vault, since this repo is public. aws-infra.yml asserts against it. +pullab_aws_region: eu-west-2 +pullab_host_region: us-east-1 + +# --- Resource names (must match the deployed pipeline) -------------------- +pullab_results_bucket: kernelci-results +pullab_results_prefix: results +# Inert. This names the KernelCI storage *service*, not an S3 bucket, and no +# bucket by this name exists. Only the kernel-install tests would read it, and +# none of them run. Kernel artifacts reach the VM by HTTP from the job +# definition instead. Do not create a bucket to match it - see README. +pullab_external_bucket: kernelci-storage +pullab_ecr_repository: kernelci-ecr +pullab_ecs_cluster: kernelci-cluster +pullab_ecs_task_family: kernelci-task +pullab_ecs_container_name: kernelci-app +pullab_ecs_task_cpu: "1024" +pullab_ecs_task_memory: "2048" +pullab_vm_role_name: kernelci-ecs-role + +pullab_log_groups: + - name: /ecs/kernelci-task + retention_days: 7 + - name: /ec2/kernelci-vms + retention_days: 3 +pullab_metrics_namespace: kernelci-metrics + +# --- KernelCI / KCIDB wiring --------------------------------------------- +pullab_api_base_uri: https://api.kernelci.org/latest +pullab_runtime_name: pull-labs-aws-ec2 +pullab_kcidb_submit_url: https://db.kernelci.org/submit +pullab_kcidb_origin: pull_labs_aws_ec2 +pullab_poll_interval_sec: 30 + +# --- Source --------------------------------------------------------------- +pullab_repo: https://github.com/kernelci/pullab_cloud.git +pullab_version: main +pullab_install_dir: /root/pullab_cloud + +# --- Poller instances ----------------------------------------------------- +# One systemd service+timer per architecture. Both share runtime_name; the +# platforms allowlist is what splits the job stream between them. +pullab_pollers: + - name: amd64 + platforms: ["aws-ec2-x86_64"] + ami_id: "resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64" + instance_type: c5a.4xlarge + - name: arm64 + platforms: ["aws-ec2-arm64"] + ami_id: "resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64" + instance_type: c7g.4xlarge + +# Template only: pull_labs_translate rewrites test_config per job, so these +# entries never run. They still define the VM shape each job inherits. +pullab_tests: + - simple-unixbench + - unixbench-kernel-regression +pullab_vm_root_volume_size: 40 +pullab_vm_max_runtime: 3600 +pullab_test_id: test-all-tests + +# How often each poller does one --once cycle. Matches the 300s sleep in the +# hand-written loop.sh this replaces. +pullab_poll_period_sec: 300 diff --git a/playbooks/pullab-cloud/group_vars/vault.yml.example b/playbooks/pullab-cloud/group_vars/vault.yml.example new file mode 100644 index 0000000..eda0e1c --- /dev/null +++ b/playbooks/pullab-cloud/group_vars/vault.yml.example @@ -0,0 +1,28 @@ +--- +# Copy to group_vars/vault.yml and encrypt: +# +# ansible-vault encrypt group_vars/vault.yml +# +# group_vars/vault.yml is gitignored. Never commit these in the clear — the +# values currently deployed were pasted straight into prod-amd64.sh / +# prod-arm64.sh on the host and should be treated as exposed. + +# JWT used for both the KernelCI API and KCIDB submission. Minted by +# tools/unified-token-generator.py for the lab's bot account, with origin +# pull_labs_aws_ec2. +pullab_unified_token: "eyJ..." + +# Credentials written to /root/.aws/credentials on the poller host. +# +# WARNING: the keys deployed today are the account's ROOT access keys +# (sts:GetCallerIdentity returns arn:aws:iam:::root). Replace them +# with a dedicated IAM user — or better, attach an instance profile to the +# poller host and set pullab_manage_aws_credentials: false so no static keys +# exist at all. +pullab_aws_access_key_id: "AKIA..." +pullab_aws_secret_access_key: "..." + +# Checked by aws-infra.yml before it touches anything, so a stray AWS_PROFILE +# cannot point the run at the wrong account. Kept out of group_vars/all.yml +# because this repo is public. +pullab_aws_account_id: "123456789012" diff --git a/playbooks/pullab-cloud/inventory.yaml b/playbooks/pullab-cloud/inventory.yaml new file mode 100644 index 0000000..06d49dc --- /dev/null +++ b/playbooks/pullab-cloud/inventory.yaml @@ -0,0 +1,15 @@ +# Inventory for the pullab_cloud (KernelCI pull-labs on AWS) poller host. +# +# The poller host itself is a small always-on EC2 instance; the actual test +# VMs it spawns live in a different region (see pullab_aws_region). +all: + children: + pullab_poller: + hosts: + # t3.micro in us-east-1. No ansible_host on purpose: the name matches + # the `Host kci-aws` alias in ~/.ssh/config, so the address is not + # published here. Set ansible_host locally if you have no such alias. + kci-aws: + ansible_port: 22 + ansible_user: ec2-user + ansible_become: true diff --git a/playbooks/pullab-cloud/requirements.yml b/playbooks/pullab-cloud/requirements.yml new file mode 100644 index 0000000..397c24a --- /dev/null +++ b/playbooks/pullab-cloud/requirements.yml @@ -0,0 +1,9 @@ +--- +# ansible-galaxy collection install -r requirements.yml +# +# Only aws-infra.yml needs these; site.yml runs on ansible.builtin alone. +collections: + - name: amazon.aws + version: ">=6.0.0" + - name: community.aws + version: ">=6.0.0" diff --git a/playbooks/pullab-cloud/roles/pullab_aws_infra/defaults/main.yml b/playbooks/pullab-cloud/roles/pullab_aws_infra/defaults/main.yml new file mode 100644 index 0000000..b81db37 --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_aws_infra/defaults/main.yml @@ -0,0 +1,4 @@ +--- +# Refuse to touch anything unless explicitly asked. AWS resources here are +# shared and long-lived; a stray run should be a no-op, not a surprise. +pullab_infra_apply: false diff --git a/playbooks/pullab-cloud/roles/pullab_aws_infra/tasks/main.yml b/playbooks/pullab-cloud/roles/pullab_aws_infra/tasks/main.yml new file mode 100644 index 0000000..3d4025b --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_aws_infra/tasks/main.yml @@ -0,0 +1,87 @@ +--- +# Idempotent, additive bootstrap of the eu-west-2 resources the pipeline uses. +# Nothing here deletes; teardown is `kernel-ci-cloud-runner aws cleanup`. +# +# Run with --check --diff first, then -e pullab_infra_apply=true to apply. + +- name: Confirm the caller meant to touch AWS + ansible.builtin.assert: + that: pullab_infra_apply | bool + fail_msg: >- + Refusing to modify AWS resources. Re-run with -e pullab_infra_apply=true + (and preferably --check --diff first). + +# Lives in the gitignored vault, not group_vars/all.yml, because this repo is +# public. Check it is present before spending an API call. +- name: Assert the expected account id is known + ansible.builtin.assert: + that: + - pullab_aws_account_id | default('') | length > 0 + fail_msg: >- + pullab_aws_account_id is not set. Add it to group_vars/vault.yml (see + vault.yml.example) so this playbook can confirm it is talking to the + right AWS account before it creates anything. + +- name: Verify we are pointed at the expected account + amazon.aws.aws_caller_info: + region: "{{ pullab_aws_region }}" + register: pullab_caller + +- name: Assert account id + ansible.builtin.assert: + that: pullab_caller.account | string == pullab_aws_account_id | string + fail_msg: >- + Connected to AWS account {{ pullab_caller.account }}, expected + {{ pullab_aws_account_id }}. Check your AWS_PROFILE. + +# The credentials currently on the poller host are root-account keys. Flag it +# rather than fail: rotating them is a separate, deliberate change. +- name: Warn if running as the account root + ansible.builtin.debug: + msg: >- + Authenticated as the account root ({{ pullab_caller.arn }}). Root access + keys should be replaced with a dedicated IAM user or an instance profile. + when: "':root' in pullab_caller.arn" + +# --- S3 ------------------------------------------------------------------- +- name: Create results bucket + amazon.aws.s3_bucket: + name: "{{ pullab_results_bucket }}" + region: "{{ pullab_aws_region }}" + state: present + public_access: + block_public_acls: true + block_public_policy: true + ignore_public_acls: true + restrict_public_buckets: true + +# --- ECR ------------------------------------------------------------------ +- name: Create ECR repository for the test image + community.aws.ecs_ecr: + name: "{{ pullab_ecr_repository }}" + region: "{{ pullab_aws_region }}" + scan_on_push: false + state: present + +# --- ECS ------------------------------------------------------------------ +# The task definition itself is registered by pullab_cloud at run time from the +# "ecs" block of the pipeline config; only the cluster is managed here. +- name: Create ECS cluster + community.aws.ecs_cluster: + name: "{{ pullab_ecs_cluster }}" + region: "{{ pullab_aws_region }}" + state: present + +# --- CloudWatch ----------------------------------------------------------- +# pullab_cloud creates a per-run log group under /ec2/kernelci-vms/ +# with no retention set, so those accumulate indefinitely. Only the two parent +# groups are declared here; pruning the per-run ones is a separate job. +- name: Create log groups with retention + amazon.aws.cloudwatchlogs_log_group: + log_group_name: "{{ item.name }}" + retention: "{{ item.retention_days }}" + region: "{{ pullab_aws_region }}" + state: present + loop: "{{ pullab_log_groups }}" + loop_control: + label: "{{ item.name }}" diff --git a/playbooks/pullab-cloud/roles/pullab_cloud/defaults/main.yml b/playbooks/pullab-cloud/roles/pullab_cloud/defaults/main.yml new file mode 100644 index 0000000..a6bf2df --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_cloud/defaults/main.yml @@ -0,0 +1,12 @@ +--- +# Manage /root/.aws/credentials from the vault. Set to false once the host has +# an IAM instance profile, so no static keys are written at all. +pullab_manage_aws_credentials: true + +# Update the checkout to pullab_version on every run. Off by default so a run +# never pulls new code from under a poller cycle unexpectedly; enable with +# -e pullab_update_checkout=true when you actually mean to deploy. +pullab_update_checkout: false + +pullab_config_dir: /etc/pullab_cloud +pullab_python: /usr/bin/python3.11 diff --git a/playbooks/pullab-cloud/roles/pullab_cloud/handlers/main.yml b/playbooks/pullab-cloud/roles/pullab_cloud/handlers/main.yml new file mode 100644 index 0000000..98142a4 --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_cloud/handlers/main.yml @@ -0,0 +1,12 @@ +--- +- name: Reload systemd + ansible.builtin.systemd_service: + daemon_reload: true + +- name: Restart pullab pollers + ansible.builtin.systemd_service: + name: "pullab-poller@{{ item.name }}.timer" + state: restarted + loop: "{{ pullab_pollers }}" + loop_control: + label: "{{ item.name }}" diff --git a/playbooks/pullab-cloud/roles/pullab_cloud/tasks/main.yml b/playbooks/pullab-cloud/roles/pullab_cloud/tasks/main.yml new file mode 100644 index 0000000..b724be4 --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_cloud/tasks/main.yml @@ -0,0 +1,211 @@ +--- +- name: Assert we are on Amazon Linux 2023 + ansible.builtin.assert: + that: + - ansible_distribution == 'Amazon' + - ansible_distribution_major_version == '2023' + fail_msg: >- + pullab_cloud is packaged for Amazon Linux 2023 (python3.11 and awscli-2 + come from the default repos there). + +# The systemd unit hard-requires /etc/pullab_cloud/env (EnvironmentFile= with +# no leading '-'). Fail here rather than silently skipping the env file and +# leaving units that only fail once systemd tries to start them. +- name: Assert the poller token is available + ansible.builtin.assert: + that: + - pullab_unified_token is defined + - pullab_unified_token | default('') | length > 0 + fail_msg: >- + pullab_unified_token is not set. Copy group_vars/vault.yml.example to + group_vars/vault.yml, fill it in, `ansible-vault encrypt` it, and re-run + with --ask-vault-pass. + +# Skipped when the host uses an IAM instance profile instead of static keys. +- name: Assert AWS credentials are available + ansible.builtin.assert: + that: + - pullab_aws_access_key_id | default('') | length > 0 + - pullab_aws_secret_access_key | default('') | length > 0 + fail_msg: >- + pullab_aws_access_key_id / pullab_aws_secret_access_key are not set. + Add them to group_vars/vault.yml, or set + pullab_manage_aws_credentials: false if this host authenticates with an + IAM instance profile. + when: pullab_manage_aws_credentials | bool + +# --- packages ------------------------------------------------------------- +# AL2023 ships python3.9 as python3, which pullab_cloud does not support, so +# python3.11 is installed and invoked explicitly everywhere. +- name: Install system packages + ansible.builtin.dnf: + name: + - python3.11 + - python3.11-pip + - python3.11-devel + - git + - make + - gcc + - awscli-2 + - docker + state: present + +# Only needed to build and push the Fargate test image +# (dockerfiles/aws/test.dockerfile); the poller itself does not use it. +- name: Enable docker + ansible.builtin.systemd_service: + name: docker + enabled: true + state: started + +# --- source --------------------------------------------------------------- +- name: Clone pullab_cloud + ansible.builtin.git: + repo: "{{ pullab_repo }}" + dest: "{{ pullab_install_dir }}" + version: "{{ pullab_version }}" + update: "{{ pullab_update_checkout }}" + register: pullab_checkout + +# `pip install -e` reports changed on every run, so it is gated. Gating on the +# checkout alone is not enough: with pullab_update_checkout false the clone only +# changes once, so a run that failed after the clone would never install. +- name: Check whether pullab_cloud is installed + ansible.builtin.command: /usr/bin/pip3.11 show kernel-ci-cloud-labs + register: pullab_installed + changed_when: false + failed_when: false + +- name: Install pullab_cloud into the system python3.11 + ansible.builtin.pip: + name: "{{ pullab_install_dir }}" + editable: true + executable: /usr/bin/pip3.11 + when: pullab_checkout.changed or pullab_installed.rc != 0 + +# --- AWS credentials ------------------------------------------------------ +- name: Create /root/.aws + ansible.builtin.file: + path: /root/.aws + state: directory + owner: root + group: root + mode: "0700" + when: pullab_manage_aws_credentials | bool + +- name: Write AWS config + ansible.builtin.copy: + dest: /root/.aws/config + owner: root + group: root + mode: "0600" + content: | + [default] + region = {{ pullab_host_region }} + when: pullab_manage_aws_credentials | bool + +- name: Write AWS credentials + ansible.builtin.copy: + dest: /root/.aws/credentials + owner: root + group: root + mode: "0600" + content: | + [default] + aws_access_key_id = {{ pullab_aws_access_key_id }} + aws_secret_access_key = {{ pullab_aws_secret_access_key }} + no_log: true + when: pullab_manage_aws_credentials | bool + +# --- configuration -------------------------------------------------------- +- name: Create config directory + ansible.builtin.file: + path: "{{ pullab_config_dir }}" + state: directory + owner: root + group: root + mode: "0750" + +# Replaces the UNIFIED_TOKEN=... prefix that was inlined in prod-*.sh. +- name: Write poller environment file + ansible.builtin.template: + src: pullab-poller.env.j2 + dest: "{{ pullab_config_dir }}/env" + owner: root + group: root + mode: "0600" + no_log: true + notify: Restart pullab pollers + +- name: Write per-architecture pipeline configs + ansible.builtin.template: + src: config.json.j2 + dest: "{{ pullab_install_dir }}/examples/aws/config-{{ poller.name }}-prod.json" + owner: root + group: root + mode: "0644" + validate: "{{ pullab_python }} -m json.tool %s" + loop: "{{ pullab_pollers }}" + loop_control: + loop_var: poller + label: "{{ poller.name }}" + notify: Restart pullab pollers + +# --- systemd -------------------------------------------------------------- +# Upstream ran this as `screen -dmS ... ./loop.sh`, an unsupervised root shell +# looping `--once` over both architectures with a 300s sleep. The units below +# are the same cycle under systemd: survives reboots, logs to the journal, and +# does not silently vanish when the screen session dies. +- name: Install poller service unit + ansible.builtin.template: + src: pullab-poller@.service.j2 + dest: /etc/systemd/system/pullab-poller@.service + owner: root + group: root + mode: "0644" + notify: + - Reload systemd + - Restart pullab pollers + +- name: Install poller timer unit + ansible.builtin.template: + src: pullab-poller@.timer.j2 + dest: /etc/systemd/system/pullab-poller@.timer + owner: root + group: root + mode: "0644" + notify: + - Reload systemd + - Restart pullab pollers + +- name: Flush handlers before enabling timers + ansible.builtin.meta: flush_handlers + +- name: Enable and start poller timers + ansible.builtin.systemd_service: + name: "pullab-poller@{{ item.name }}.timer" + enabled: true + state: started + loop: "{{ pullab_pollers }}" + loop_control: + label: "{{ item.name }}" + +# --- migration off the hand-rolled screen session ------------------------- +# Not automated: killing the running loop is a judgement call, and the old +# scripts hold the only copy of a token that is not yet rotated. +- name: Check for the legacy screen-based loop + ansible.builtin.command: pgrep -f 'loop\.sh' + register: pullab_legacy_loop + changed_when: false + failed_when: false + +- name: Warn about the legacy loop still running + ansible.builtin.debug: + msg: >- + The old screen session running loop.sh is still alive (PIDs: + {{ pullab_legacy_loop.stdout_lines | join(', ') }}). Stop it by hand once + the systemd timers are confirmed working, otherwise both will poll: + attach with `screen -r` as root, Ctrl-C, then remove + {{ pullab_install_dir }}/loop.sh and prod-*.sh - they contain a + plaintext JWT. + when: pullab_legacy_loop.rc == 0 diff --git a/playbooks/pullab-cloud/roles/pullab_cloud/templates/config.json.j2 b/playbooks/pullab-cloud/roles/pullab_cloud/templates/config.json.j2 new file mode 100644 index 0000000..431ecf3 --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_cloud/templates/config.json.j2 @@ -0,0 +1,236 @@ +{# Managed by Ansible - playbooks/pullab-cloud. + Renders examples/aws/config--prod.json for one architecture. + `poller` is the loop item from pullab_pollers. #} +{ + "provider": "aws", + "region": "{{ pullab_aws_region }}", + "storage": { + "type": "s3", + "bucket": "{{ pullab_results_bucket }}", + "results_prefix": "{{ pullab_results_prefix }}" + }, + "external_storage": { + "type": "s3", + "bucket": "{{ pullab_external_bucket }}" + }, + "force_recreate_roles": true, + "auth_credentials": { + "auth_provider": "aws" + }, + "roles": { + "{{ pullab_vm_role_name }}": { + "description": "ECS task execution role with EC2 and SSM access", + "trust_policy": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": [ + "ecs-tasks.amazonaws.com", + "ec2.amazonaws.com" + ] + }, + "Action": "sts:AssumeRole" + } + ] + }, + "policies": [ + "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy", + "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore", + "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" + ], + "inline_policies": { + "AllowPassRole": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam::*:role/{{ pullab_vm_role_name }}", + "Condition": { + "StringEquals": { + "iam:PassedToService": "ec2.amazonaws.com" + } + } + } + ] + }, + "AllowEC2": { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "EC2RunInstances", + "Effect": "Allow", + "Action": [ + "ec2:RunInstances", + "ec2:CreateTags" + ], + "Resource": "*" + }, + { + "Sid": "EC2TerminateOwnInstances", + "Effect": "Allow", + "Action": "ec2:TerminateInstances", + "Resource": "arn:aws:ec2:*:*:instance/*", + "Condition": { + "StringLike": { + "aws:ResourceTag/run_prefix": "run_*" + } + } + }, + { + "Sid": "EC2Describe", + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeImages", + "ec2:DescribeSubnets", + "ec2:DescribeSecurityGroups", + "ec2:DescribeVpcs", + "ec2:GetConsoleOutput" + ], + "Resource": "*" + } + ] + }, + "AllowS3Access": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::{{ pullab_results_bucket }}", + "arn:aws:s3:::{{ pullab_results_bucket }}/*", + "arn:aws:s3:::{{ pullab_results_bucket }}-*", + "arn:aws:s3:::{{ pullab_results_bucket }}-*/*", + "arn:aws:s3:::{{ pullab_external_bucket }}", + "arn:aws:s3:::{{ pullab_external_bucket }}/*" + ] + } + ] + }, + "AllowSSMCommands": { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "SSMSendCommandDocument", + "Effect": "Allow", + "Action": "ssm:SendCommand", + "Resource": "arn:aws:ssm:*::document/AWS-RunShellScript" + }, + { + "Sid": "SSMSendCommandInstances", + "Effect": "Allow", + "Action": "ssm:SendCommand", + "Resource": "arn:aws:ec2:*:*:instance/*", + "Condition": { + "StringLike": { + "ssm:resourceTag/run_prefix": "run_*" + } + } + }, + { + "Sid": "SSMManageCommands", + "Effect": "Allow", + "Action": [ + "ssm:GetCommandInvocation", + "ssm:CancelCommand" + ], + "Resource": "*" + }, + { + "Sid": "SSMDescribeAndResolve", + "Effect": "Allow", + "Action": [ + "ssm:DescribeInstanceInformation", + "ssm:GetParameter" + ], + "Resource": "*" + } + ] + }, + "AllowIAMInstanceProfile": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "iam:CreateInstanceProfile", + "iam:DeleteInstanceProfile", + "iam:AddRoleToInstanceProfile", + "iam:RemoveRoleFromInstanceProfile", + "iam:GetInstanceProfile" + ], + "Resource": "arn:aws:iam::*:instance-profile/{{ pullab_vm_role_name }}" + } + ] + } + } + } + }, + "ecr": { + "repository": "{{ pullab_ecr_repository }}", + "scan_on_push": false + }, + "docker": { + "dockerfile": "dockerfiles/aws/test.dockerfile", + "build_context": ".", + "force_rebuild": true + }, + "ecs": { + "cluster": "{{ pullab_ecs_cluster }}", + "task_definition": { + "family": "{{ pullab_ecs_task_family }}", + "cpu": "{{ pullab_ecs_task_cpu }}", + "memory": "{{ pullab_ecs_task_memory }}", + "container_name": "{{ pullab_ecs_container_name }}", + "region": "{{ pullab_aws_region }}" + } + }, + "cloudwatch": { + "log_groups": { +{% for lg in pullab_log_groups %} + "{{ lg.name }}": { + "retention_days": {{ lg.retention_days }} + }{{ "," if not loop.last }} +{% endfor %} + }, + "metrics_namespace": "{{ pullab_metrics_namespace }}" + }, + "kernelci": { + "api_base_uri": "{{ pullab_api_base_uri }}", + "api_token": null, + "runtime_name": "{{ pullab_runtime_name }}", + "platforms": {{ poller.platforms | to_json }}, + "poll_interval_sec": {{ pullab_poll_interval_sec }}, + "cursor_file": "/tmp/pullab_cloud_cursor_{{ poller.name }}_prod.json", + "kcidb_submit_url": "{{ pullab_kcidb_submit_url }}", + "kcidb_origin": "{{ pullab_kcidb_origin }}", + "kcidb_jwt": null, + "comment": "Both architectures share runtime_name '{{ pullab_runtime_name }}'; the platforms filter is what splits the job stream. api_token/kcidb_jwt stay null and are injected at runtime via UNIFIED_TOKEN." + }, + "test_config": { + "test_id": "{{ pullab_test_id }}", + "role_name": "{{ pullab_vm_role_name }}", + "vms": [ +{% for test in pullab_tests %} + { + "ami_id": "{{ poller.ami_id }}", + "instance_type": "{{ poller.instance_type }}", + "root_volume_size": {{ pullab_vm_root_volume_size }}, + "max_runtime": {{ pullab_vm_max_runtime }}, + "test": [ + "{{ test }}" + ], + "min_count": 1 + }{{ "," if not loop.last }} +{% endfor %} + ] + } +} diff --git a/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller.env.j2 b/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller.env.j2 new file mode 100644 index 0000000..c851c2f --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller.env.j2 @@ -0,0 +1,6 @@ +# Managed by Ansible - playbooks/pullab-cloud +# +# UNIFIED_TOKEN is the shared fallback for both KERNELCI_API_TOKEN and +# KCIDB_JWT; the config files leave api_token/kcidb_jwt null on purpose. +UNIFIED_TOKEN={{ pullab_unified_token }} +PYTHONPATH=src diff --git a/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller@.service.j2 b/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller@.service.j2 new file mode 100644 index 0000000..5e7fb53 --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller@.service.j2 @@ -0,0 +1,18 @@ +# Managed by Ansible - playbooks/pullab-cloud +# %i is the poller name (amd64, arm64), matching +# examples/aws/config-%i-prod.json. +[Unit] +Description=KernelCI pull-lab AWS poller (%i) +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=root +WorkingDirectory={{ pullab_install_dir }} +EnvironmentFile={{ pullab_config_dir }}/env +ExecStart={{ pullab_python }} -m kernel_ci_cloud_labs.pull_labs_poller \ + --config {{ pullab_install_dir }}/examples/aws/config-%i-prod.json --once +# A poll cycle spawns EC2 VMs and waits for them; max_runtime is +# {{ pullab_vm_max_runtime }}s per VM, so allow generous headroom. +TimeoutStartSec={{ (pullab_vm_max_runtime | int) * 3 }} diff --git a/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller@.timer.j2 b/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller@.timer.j2 new file mode 100644 index 0000000..ca20f42 --- /dev/null +++ b/playbooks/pullab-cloud/roles/pullab_cloud/templates/pullab-poller@.timer.j2 @@ -0,0 +1,14 @@ +# Managed by Ansible - playbooks/pullab-cloud +[Unit] +Description=KernelCI pull-lab AWS poller timer (%i) + +[Timer] +# Same cadence as the sleep in the loop.sh this replaces. Measured from the end +# of the previous run, so cycles never overlap. +OnBootSec=60 +OnUnitInactiveSec={{ pullab_poll_period_sec }} +AccuracySec=10 +Unit=pullab-poller@%i.service + +[Install] +WantedBy=timers.target diff --git a/playbooks/pullab-cloud/site.yml b/playbooks/pullab-cloud/site.yml new file mode 100644 index 0000000..55b02f8 --- /dev/null +++ b/playbooks/pullab-cloud/site.yml @@ -0,0 +1,15 @@ +# Provision the pullab_cloud poller host. +# +# ansible-playbook -i inventory.yaml site.yml +# +# Secrets (UNIFIED_TOKEN, AWS keys) come from group_vars/vault.yml — see +# group_vars/vault.yml.example. The role asserts on them before it touches the +# host, so a run without them stops on the control node having changed nothing. +# Set pullab_manage_aws_credentials=false if the host authenticates with an IAM +# instance profile and has no static keys to place. +- name: pullab_cloud poller + hosts: pullab_poller + gather_facts: true + become: true + roles: + - pullab_cloud