[istio] Add interactive istioctl debug via d8 network istio by skurbatov · Pull Request #454 · deckhouse/deckhouse-cli

skurbatov · 2026-08-24T14:58:39Z

Summary

d8 network istio starts an interactive istioctl debug session: it applies the namespaced RBAC, pulls the platform debug image, and attaches to a one-shot pod — the same workflow operators previously ran by hand with d8 k.

Problem

Debugging Istio required applying a ServiceAccount in the debug namespace plus a Role/RoleBinding in the target namespace (pods get/list, pods/portforward create), then d8 k run --rm -it with --overrides for the SA
The debug image had to be read from ConfigMap d8-system/debug-container on every run
The two namespaces, leftover pods, and --rm cleanup were easy to get wrong; none of this lived under d8 network

Fix

New leaf command d8 network istio next to cni-migration
Resolves the image from d8-system/debug-container (data.image) unless --image is set
Idempotently creates ServiceAccount istioctl-debug in --namespace, and Role/RoleBinding istioctl-debug in --target-namespace (defaults to --namespace)
Runs pod istioctl-debug (restartPolicy: Never, stdin/TTY, default command bash), waits until Running, attaches, deletes the pod on exit; RBAC is left in place for reuse
Replaces a leftover pod of the same name; fails fast on ErrImagePull / ImagePullBackOff / CrashLoopBackOff
Resets SIGINT/SIGTERM so the first Ctrl-C ends the attach session instead of being swallowed by the d8 root handler

Before / After

Before: apply three RBAC manifests, IMG="$(d8 k -n d8-system get cm debug-container -o jsonpath='{.data.image}')", then d8 k run istioctl-debug --rm -it --overrides=... -- bash.

After: d8 network istio -n <debug-namespace> --target-namespace <target-namespace> — same SA/Role/RoleBinding and the same debug pod, without the manual kubectl steps.

Tests

TestEnsureRBACCreatesObjects — SA in the debug namespace, Role rules and RoleBinding subject/ref in the target namespace
TestEnsureRBACIsIdempotentAndUpdatesRules — second apply restores Role rules if they were cleared
TestResolveDebugImage — --image override, ConfigMap lookup, missing CM / empty image key
TestBuildDebugPod — SA, automount token, Never restart, stdin/TTY
TestCreateDebugPodReplacesLeftover — leftover pod is replaced
TestWaitForPodRunning / TestWaitForPodRunningImagePullError — Running vs terminal wait reason
TestNewCommandFlags — namespace, target-namespace, image, kubeconfig, context

Notes

Interactive attach is not covered by the fake clientset tests; end-to-end attach against a live cluster was not run in this environment.

Signed-off-by: Sergey Kurbatov <sergey.kurbatov@flant.com>

Glitchy-Sheep

Reviewed the new d8 network istio command. 6 inline comments below: the first four are confirmed by mechanism (signal handling vs cleanup, one-shot mode, KUBECONFIG list, concurrent sessions), the last two depend on the cluster setup (RBAC scope for istioctl, image pull / PSS) and are worth checking on a real DKP cluster.

Glitchy-Sheep · 2026-08-27T10:37:23Z

+		RunE: func(cmd *cobra.Command, args []string) error {
+			// Match kubectl: first SIGINT/SIGTERM must stop the attach session
+			// instead of being swallowed by the d8 root graceful handler.
+			signal.Reset(syscall.SIGINT, syscall.SIGTERM)


signal.Reset disables the root graceful handler - the only thing that cancels cmd.Context() on SIGINT/SIGTERM. With the default signal action restored, Ctrl+C kills the process on the spot: the deferred deletePod never runs and the debug pod (with its SA token mounted) leaks. Ctrl+C during the up-to-2-minute image pull wait is the easiest way to hit this.

The kubectl.go code this mirrors has no cleanup defer, so the reset is harmless there. Here the whole path is ctx-aware: without the reset, cancel aborts the wait/stream and the defer (built on context.Background() for exactly this) deletes the pod. Suggest dropping the reset.

Glitchy-Sheep · 2026-08-27T10:37:23Z

+		switch pod.Status.Phase {
+		case corev1.PodRunning:
+			return true, nil
+		case corev1.PodFailed, corev1.PodSucceeded:


The one-shot form istio -- command from Use does not survive the poll-then-attach design:

a fast command (istioctl version) drives the pod to Succeeded within the 1s poll interval; waitForPodRunning returns that as an error, and the deferred delete removes the pod before its logs can be read - the output is lost;

even when attach wins the race, everything printed before the stream opened is not replayed;

attach, unlike exec, carries no exit status, so StreamWithContext returns nil even when the command failed and d8 exits 0 (root.go already maps ExitCode() errors to the process exit code).

kubectl run --attach falls back to printing container logs for exactly this race. If one-shot is out of scope for now, dropping [-- command ...] from Use may be the cheaper fix.

Glitchy-Sheep · 2026-08-27T10:37:23Z

+	}
+
+	ns, _, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
+		&clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeconfigPath},


kubeconfigNamespace puts the raw flag value into ExplicitPath, but the flag default is $KUBECONFIG verbatim, which may be a colon-separated list. SetupK8sClientSet splits it (clientset.go:53), this function does not: with KUBECONFIG=~/.kube/config:~/.kube/other the client builds fine, then the command dies with "no such file or directory" on the literal path config:other.

transport.KubeconfigNamespace (internal/snapshot/transport/kubeconfig.go) already implements the correct loading rules and can be reused here. It also makes the ns == "" fallback unnecessary: clientcmd already returns "default".

Glitchy-Sheep · 2026-08-27T10:37:23Z

+}
+
+func createDebugPod(ctx context.Context, kube kubernetes.Interface, namespace, image string, command []string) (*corev1.Pod, error) {
+	if existing, err := kube.CoreV1().Pods(namespace).Get(ctx, resourceName, metav1.GetOptions{}); err == nil {


The fixed pod name plus unconditional "leftover" deletion makes every run destructive to a concurrent session:

a second run force-deletes a live Running pod someone is attached to;

the first session's deferred delete-by-name (no UID precondition) then kills the second session's fresh pod;

createOrUpdateRoleBinding overwrites Subjects, so a run with a different --namespace against the same --target-namespace silently revokes the other session's RBAC mid-flight.

Suggest GenerateName per run with the app.kubernetes.io/managed-by label as the ownership marker, and garbage-collecting only terminal-phase pods.

Glitchy-Sheep · 2026-08-27T10:37:23Z

+	return nil
+}
+
+func istioctlDebugRules() []rbacv1.PolicyRule {


These rules cover only the target namespace, but istioctl's core commands need the control plane: istioctl proxy-status discovers istiod pods and port-forwards to them in d8-istio, which this RBAC never grants -> Forbidden. proxy-config against pods in the target namespace will work; if proxy-status / xds diagnostics are part of the intended scope, one --target-namespace cannot cover both. Worth checking on a real cluster before merge.

I’ve set up the minimum RBAC grants; we’ll add more as needed.

Glitchy-Sheep · 2026-08-27T10:37:23Z

+			ServiceAccountName:           resourceName,
+			AutomountServiceAccountToken: ptr.To(true),
+			RestartPolicy:                corev1.RestartPolicyNever,
+			Containers: []corev1.Container{{


The pod spec has no imagePullSecrets, while the default image from d8-system/debug-container lives in the private Deckhouse registry whose pull secret exists only in d8-* namespaces. Unless nodes carry containerd auth for that registry, d8 network istio -n <user-ns> without --image ends in ImagePullBackOff. Related: there is no securityContext either, so a namespace enforcing the restricted PSS rejects the pod at admission. Both are worth a check on a real DKP cluster.

At the moment, we are accepting this problem. In the future, we will consider how to fix it.

Signed-off-by: Sergey Kurbatov <sergey.kurbatov@flant.com>

skurbatov self-assigned this Aug 24, 2026

skurbatov added the enhancement New feature or request label Aug 24, 2026

skurbatov requested a review from ldmonster as a code owner August 24, 2026 14:58

skurbatov marked this pull request as draft August 24, 2026 14:59

[istio] Add interactive istioctl debug via d8 network istio

7d75fb0

Signed-off-by: Sergey Kurbatov <sergey.kurbatov@flant.com>

skurbatov force-pushed the istio_debug branch from 9b6537b to 7d75fb0 Compare August 24, 2026 14:59

Fix linter

18ff1f8

Signed-off-by: Sergey Kurbatov <sergey.kurbatov@flant.com>

skurbatov marked this pull request as ready for review August 24, 2026 16:15

skurbatov requested a review from apolovov August 24, 2026 16:16

ldmonster requested a review from Glitchy-Sheep August 26, 2026 09:14

apolovov previously approved these changes Aug 27, 2026

View reviewed changes

Glitchy-Sheep reviewed Aug 27, 2026

View reviewed changes

Correcting comments for the code

0936d9f

Signed-off-by: Sergey Kurbatov <sergey.kurbatov@flant.com>

skurbatov dismissed apolovov’s stale review via 0936d9f August 27, 2026 12:09

skurbatov added 2 commits August 27, 2026 16:40

Add access to istio ns

c905073

Signed-off-by: Sergey Kurbatov <sergey.kurbatov@flant.com>

fix

70679bc

Signed-off-by: Sergey Kurbatov <sergey.kurbatov@flant.com>

skurbatov requested review from Glitchy-Sheep and apolovov August 27, 2026 15:54

Navigation Menu

Uh oh!

[istio] Add interactive istioctl debug via d8 network istio - #454

[istio] Add interactive istioctl debug via d8 network istio#454
skurbatov wants to merge 5 commits into
mainfrom
istio_debug

skurbatov commented Aug 24, 2026

Uh oh!

Glitchy-Sheep left a comment

Uh oh!

Glitchy-Sheep Aug 27, 2026

Uh oh!

skurbatov Aug 27, 2026

Uh oh!

Glitchy-Sheep Aug 27, 2026

Uh oh!

skurbatov Aug 27, 2026

Uh oh!

Glitchy-Sheep Aug 27, 2026

Uh oh!

skurbatov Aug 27, 2026

Uh oh!

Glitchy-Sheep Aug 27, 2026

Uh oh!

skurbatov Aug 27, 2026

Uh oh!

Glitchy-Sheep Aug 27, 2026

Uh oh!

skurbatov Aug 27, 2026

Uh oh!

Glitchy-Sheep Aug 27, 2026

Uh oh!

skurbatov Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

skurbatov commented Aug 24, 2026

Summary

Problem

Fix

Before / After

Tests

Notes

Uh oh!

Glitchy-Sheep left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants