From 6f782439e4120e94d7afefaec436dd097a4cfef1 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 4 Sep 2026 15:17:44 +0200 Subject: [PATCH 1/7] security(kubescape): opt longhorn-system into the controller-scope baseline context MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four namespaces opted into add-baseline-context-optin-* carry only `baseline-context`, which gates three rules that all match `kinds: Pod`. Those mutate at admission, while Kubescape reads the STORED SPEC of the controller — so the opt-in cannot move C-0211 no matter how many namespaces receive it. The controller-scope rules that can move it (add-baseline-context-optin- controllers*, matching Deployment/StatefulSet/DaemonSet on CREATE and UPDATE) gate on a second label, `baseline-context-controllers`, which is applied in no namespace anywhere in this repository. This adds the first one. Measured on prod 2026-09-04, separating the two surfaces: - kubescape: its pods pick the fields up as they recreate (7 of 15, then 8 of 16 eleven minutes later, the same pods carrying both) while 0 of its 6 controllers ever do. The pod figures are volatile by construction and only their direction is the evidence; the controller figures are the stable ones. - longhorn-system: all 11 workloads carry neither, though opted in since 09-02. - velero reads 2/2 only because its HelmRelease supplies the fields at source, so the one namespace that looked healthy could not have revealed the gap. longhorn-system first: its 11 workloads are genuinely scanned (Deployment:6 + DaemonSet:5 workloadconfigurationscan objects), all 11 miss both fields, and none sets fsGroup, so fsGroupChangePolicy is inert at runtime here. kubescape is deliberately not next — the operator's excludeNamespaces drops its workloads, which hold no workload-kind scan objects, so mutating them cannot move C-0211. Exercised with the Kyverno CLI against the real longhorn-ui Deployment in both states. With the label: fsGroupChangePolicy=OnRootMismatch and the container's seLinuxOptions.level=s0 are added while runAsUser/runAsGroup/runAsNonRoot/ seccompProfile survive untouched. Without it the output is byte-identical to the input. Part of #3239 Co-Authored-By: Claude Opus 5 (1M context) --- .../controllers/longhorn/namespace.yaml | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/k8s/providers/hetzner/infrastructure/controllers/longhorn/namespace.yaml b/k8s/providers/hetzner/infrastructure/controllers/longhorn/namespace.yaml index 55cd41a82..02be88d28 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/longhorn/namespace.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/longhorn/namespace.yaml @@ -42,3 +42,30 @@ metadata: # The rules are CREATE-only, so running pods are unaffected until they are # next recreated. pod-security.devantler.tech/baseline-context: enabled + # + # 🔴 The label above is NOT sufficient to move C-0211, and that is why this + # second one exists. add-baseline-context-optin-namespaces / -pod-selinux-level / + # -containers all match `kinds: Pod`, so they mutate pods at ADMISSION while + # Kubescape scans the STORED SPEC of the controller. Measured on prod 2026-09-04: + # kubescape's pods pick both fields up as they are recreated — 7 of 15, then 8 of 16 + # eleven minutes later, the same pods carrying both — while 0 of its 6 Deployments + # and DaemonSets ever do. Here all 11 workloads still carry neither, despite this + # namespace having been opted in since 2026-09-02. The pod figures are volatile by + # construction and only their direction is the evidence; the controller figures are + # the stable ones, and they are the surface the scanner reads. velero reads 2/2 only + # because its HelmRelease supplies the fields at the source, not because a mutation + # reached it — so velero could not have revealed this gap. + # + # add-baseline-context-optin-controllers* match Deployment/StatefulSet/DaemonSet on + # CREATE *and* UPDATE, so Flux's next reconcile rewrites the stored spec without + # anything needing to be deleted. They inject the same two fields through the same + # +(key) anchors and touch no user, group or privilege field. + # + # First namespace opted in at controller scope. longhorn-system is the right place + # to start: its 11 workloads are genuinely scanned (the namespace holds Deployment:6 + # and DaemonSet:5 workloadconfigurationscan objects), all 11 miss both fields, and + # none sets fsGroup — so the change is inert at runtime here, exactly as the pod + # rules already are. kubescape is deliberately NOT next: the operator's own + # excludeNamespaces drops its workloads, which hold no workload-kind scan objects at + # all, so mutating them cannot move the control. + pod-security.devantler.tech/baseline-context-controllers: enabled From 777c5fc79320cf13c73f46d590521d8c1bf1bb02 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 4 Sep 2026 15:20:22 +0200 Subject: [PATCH 2/7] docs(security): correct the C-0211 sizing note's claim about the opt-in rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The note said "all three of those rules match `kinds: Pod`", which reads as the opt-in mechanism being structurally unable to reach the stored spec. That was true when written and is not now: add-baseline-context-optin-controllers* match Deployment/StatefulSet/DaemonSet on CREATE and UPDATE. What is actually true is narrower and more useful — those controller rules gate on a SECOND label, `baseline-context-controllers`, which was applied in no namespace at all. Left as it stood, the note would tell the next reader the opt-in can never move C-0211, when the accurate statement is that the half that can move it had never been switched on anywhere. Comment-only: rendering k8s/bases/infrastructure/cluster-security-exceptions before and after is byte-identical at 962 lines. Part of #3239 Co-Authored-By: Claude Opus 5 (1M context) --- .../pod-security-mutations-unscoped.yaml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml index 473a549de..81a61478a 100644 --- a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml +++ b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml @@ -99,9 +99,15 @@ # # The remaining 29 split observability 15, longhorn-system 12, flux-system 2. # observability and longhorn-system are already labelled into -# add-baseline-context-optin-*, but all three of those rules match `kinds: Pod`, so -# they mutate at admission and leave the stored spec Kubescape reads untouched — which -# is why the opt-in alone moves none of those 27. Template-level changes are what moved +# add-baseline-context-optin-*, but that label gates only the three `kinds: Pod` +# rules, so they mutate at admission and leave the stored spec Kubescape reads +# untouched — which is why the opt-in alone moves none of those 27. The +# controller-scope rules that DO reach the stored spec are +# add-baseline-context-optin-controllers* (Deployment/StatefulSet/DaemonSet on +# CREATE and UPDATE), and they gate on a SECOND label, +# `baseline-context-controllers`, which was applied in no namespace at all until +# longhorn-system opted in — re-measure this split once that has reconciled. +# Template-level changes are what moved # velero and flux-system. flux-system's remaining two are flux-operator and # tofu-controller, the latter retired and pending removal in #3480. # From f734359be160a63db4b964bb236d885fcf8b7d07 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 4 Sep 2026 15:32:52 +0200 Subject: [PATCH 3/7] Revert "security(kubescape): opt longhorn-system into the controller-scope baseline context" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit 6f782439. The controller-scope opt-in is not a label edit. tests/add-baseline-context pins `expected_controllers_optin=""` deliberately, and the procedure beside the rules in add-security-context.yaml requires, per namespace: the label, a template annotation UPDATE per existing Deployment/StatefulSet/DaemonSet/CronJob (the label alone changes nothing already stored — helm-controller reapplies only on a chart or values change and excludes /spec/template from drift correction), a read-back of the stored specs, a watch of every owner's write rate for the reconcile loop the same comment warns about, and only then the recorded evidence the gate expects. That flip restarts Longhorn's manager, CSI and UI — the cluster's storage system — and step 3 needs sustained observation. It is a mutating prod operation and this repository's prod access is read-only, so it is not an unattended change. The gate caught this correctly. The measurement stands and is recorded on #3239; only the premature label is withdrawn. The doc correction in the following commit is independent and holds. Co-Authored-By: Claude Opus 5 (1M context) --- .../controllers/longhorn/namespace.yaml | 27 ------------------- 1 file changed, 27 deletions(-) diff --git a/k8s/providers/hetzner/infrastructure/controllers/longhorn/namespace.yaml b/k8s/providers/hetzner/infrastructure/controllers/longhorn/namespace.yaml index 02be88d28..55cd41a82 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/longhorn/namespace.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/longhorn/namespace.yaml @@ -42,30 +42,3 @@ metadata: # The rules are CREATE-only, so running pods are unaffected until they are # next recreated. pod-security.devantler.tech/baseline-context: enabled - # - # 🔴 The label above is NOT sufficient to move C-0211, and that is why this - # second one exists. add-baseline-context-optin-namespaces / -pod-selinux-level / - # -containers all match `kinds: Pod`, so they mutate pods at ADMISSION while - # Kubescape scans the STORED SPEC of the controller. Measured on prod 2026-09-04: - # kubescape's pods pick both fields up as they are recreated — 7 of 15, then 8 of 16 - # eleven minutes later, the same pods carrying both — while 0 of its 6 Deployments - # and DaemonSets ever do. Here all 11 workloads still carry neither, despite this - # namespace having been opted in since 2026-09-02. The pod figures are volatile by - # construction and only their direction is the evidence; the controller figures are - # the stable ones, and they are the surface the scanner reads. velero reads 2/2 only - # because its HelmRelease supplies the fields at the source, not because a mutation - # reached it — so velero could not have revealed this gap. - # - # add-baseline-context-optin-controllers* match Deployment/StatefulSet/DaemonSet on - # CREATE *and* UPDATE, so Flux's next reconcile rewrites the stored spec without - # anything needing to be deleted. They inject the same two fields through the same - # +(key) anchors and touch no user, group or privilege field. - # - # First namespace opted in at controller scope. longhorn-system is the right place - # to start: its 11 workloads are genuinely scanned (the namespace holds Deployment:6 - # and DaemonSet:5 workloadconfigurationscan objects), all 11 miss both fields, and - # none sets fsGroup — so the change is inert at runtime here, exactly as the pod - # rules already are. kubescape is deliberately NOT next: the operator's own - # excludeNamespaces drops its workloads, which hold no workload-kind scan objects at - # all, so mutating them cannot move the control. - pod-security.devantler.tech/baseline-context-controllers: enabled From 9677735249c1e838bbd612e4b284bf25585832ab Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 4 Sep 2026 15:33:33 +0200 Subject: [PATCH 4/7] docs(security): say why the controller-scope opt-in is empty, not that it is absent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows the revert. The previous wording implied the second label was an oversight about to be corrected here. It is not: tests/add-baseline-context pins the empty inventory on purpose, and a namespace joins only together with the measured post-rollout read-back its flip procedure produces. So the accurate statement is that the 27 are waiting on a per-namespace rollout, not on a mechanism that cannot reach them — which is the opposite of what the original sentence ("all three of those rules match `kinds: Pod`") told a reader, and also not the "nobody switched it on" reading my first attempt suggested. Comment-only: rendering cluster-security-exceptions before and after is byte-identical. Part of #3239 Co-Authored-By: Claude Opus 5 (1M context) --- .../pod-security-mutations-unscoped.yaml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml index 81a61478a..398818b8a 100644 --- a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml +++ b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml @@ -104,12 +104,12 @@ # untouched — which is why the opt-in alone moves none of those 27. The # controller-scope rules that DO reach the stored spec are # add-baseline-context-optin-controllers* (Deployment/StatefulSet/DaemonSet on -# CREATE and UPDATE), and they gate on a SECOND label, -# `baseline-context-controllers`, which was applied in no namespace at all until -# longhorn-system opted in — re-measure this split once that has reconciled. -# Template-level changes are what moved -# velero and flux-system. flux-system's remaining two are flux-operator and -# tofu-controller, the latter retired and pending removal in #3480. +# CREATE and UPDATE), gated on a SECOND label, `baseline-context-controllers`, +# which is deliberately applied in NO namespace — tests/add-baseline-context pins +# that empty inventory, and a namespace joins only together with the measured +# post-rollout read-back its documented flip procedure produces. So the 27 are +# waiting on that per-namespace rollout, not on a mechanism that cannot reach +# them. Template-level changes are what moved # # The per-workload table and the remediation split are on #3217. These counts are # prod-only and the local overlay opts in to a different namespace set, so re-measure From 7c6db99f7fe330532151ef8991663c2316e8ad67 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 4 Sep 2026 16:11:10 +0200 Subject: [PATCH 5/7] docs(security): restore the sentence and issue reference the rewrite truncated The previous commit's rewrite of this comment block ended mid-sentence at "Template-level changes are what moved", dropping both its object (velero and flux-system) and the flux-system breakdown that named tofu-controller and its open removal issue #3480. Restores the dropped text verbatim. Comment-only: the rendered output is byte-identical before and after, verified by rendering the base directory both ways (31314 B each) with a sensitivity control proving the comparison detects a semantic change. Co-Authored-By: Claude Opus 5 (1M context) --- .../pod-security-mutations-unscoped.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml index 398818b8a..e2d89dfee 100644 --- a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml +++ b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml @@ -109,7 +109,9 @@ # that empty inventory, and a namespace joins only together with the measured # post-rollout read-back its documented flip procedure produces. So the 27 are # waiting on that per-namespace rollout, not on a mechanism that cannot reach -# them. Template-level changes are what moved +# them. Template-level changes are what moved velero and flux-system. flux-system's +# remaining two are flux-operator and tofu-controller, the latter retired and pending +# removal in #3480. # # The per-workload table and the remediation split are on #3217. These counts are # prod-only and the local overlay opts in to a different namespace set, so re-measure From adf9609f668e276014078ecb4afef8dee1862225 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 4 Sep 2026 16:33:05 +0200 Subject: [PATCH 6/7] docs(security): name the CronJob rule family in the rollout note The note identified only add-baseline-context-optin-controllers* as the rules that reach the stored spec, but the policy handles CronJobs through a separate add-baseline-context-optin-cronjobs* family (CronJob on CREATE and UPDATE), and seven of the residual workloads this note sizes are CronJobs. Planning the rollout from the note as written would have left those seven and their read-back checks out. Names both families and says why both matter. Raised by Codex review at 7c6db99f (P2); verified against the policy before accepting. Comment-only: the rendered output is byte-identical. Co-Authored-By: Claude Opus 5 (1M context) --- .../pod-security-mutations-unscoped.yaml | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml index e2d89dfee..01d19e0c7 100644 --- a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml +++ b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml @@ -101,11 +101,15 @@ # observability and longhorn-system are already labelled into # add-baseline-context-optin-*, but that label gates only the three `kinds: Pod` # rules, so they mutate at admission and leave the stored spec Kubescape reads -# untouched — which is why the opt-in alone moves none of those 27. The -# controller-scope rules that DO reach the stored spec are -# add-baseline-context-optin-controllers* (Deployment/StatefulSet/DaemonSet on -# CREATE and UPDATE), gated on a SECOND label, `baseline-context-controllers`, -# which is deliberately applied in NO namespace — tests/add-baseline-context pins +# untouched — which is why the opt-in alone moves none of those 27. The rules that +# DO reach the stored spec are TWO families, both gated on a SECOND label, +# `baseline-context-controllers`: add-baseline-context-optin-controllers* +# (Deployment/StatefulSet/DaemonSet on CREATE and UPDATE) and +# add-baseline-context-optin-cronjobs* (CronJob on CREATE and UPDATE). Both are +# load-bearing for this rollout — seven of the residual workloads are CronJobs, so +# planning only the controller family would silently leave those out along with +# their read-back checks. That label is deliberately applied in NO namespace — +# tests/add-baseline-context pins # that empty inventory, and a namespace joins only together with the measured # post-rollout read-back its documented flip procedure produces. So the 27 are # waiting on that per-namespace rollout, not on a mechanism that cannot reach From e6000eca59fe200825ca1fc3ade52083cbc0cb5a Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 4 Sep 2026 17:27:55 +0200 Subject: [PATCH 7/7] docs(security): name the Job case in the controller rule family add-baseline-context-optin-controllers* also matches Job, on CREATE only. The rollout note described the family as Deployment/StatefulSet/DaemonSet, presenting it as narrower than it is. Comment-only. Co-Authored-By: Claude Opus 5 (1M context) --- .../pod-security-mutations-unscoped.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml index 01d19e0c7..c7cef8fa7 100644 --- a/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml +++ b/k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yaml @@ -104,7 +104,7 @@ # untouched — which is why the opt-in alone moves none of those 27. The rules that # DO reach the stored spec are TWO families, both gated on a SECOND label, # `baseline-context-controllers`: add-baseline-context-optin-controllers* -# (Deployment/StatefulSet/DaemonSet on CREATE and UPDATE) and +# (Deployment/StatefulSet/DaemonSet on CREATE and UPDATE, Job on CREATE) and # add-baseline-context-optin-cronjobs* (CronJob on CREATE and UPDATE). Both are # load-bearing for this rollout — seven of the residual workloads are CronJobs, so # planning only the controller family would silently leave those out along with