Repository navigation
fix(workflows): cap create and destroy steps at 2h across all workflows #1945
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
fa412f4
d2a37cb
be95325
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -70,6 +70,8 @@ spec: | |
| template: destroy | ||
|
|
||
| - name: create | ||
| activeDeadlineSeconds: 7200 | ||
| timeout: 2h | ||
| outputs: | ||
| artifacts: | ||
| - name: kubeconfig | ||
|
|
@@ -208,6 +210,7 @@ spec: | |
| suspend: {} | ||
|
|
||
| - name: destroy | ||
| activeDeadlineSeconds: 7200 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bound pending time for destroy and create steps. The active deadline does not cover unscheduled pods.
Add a pending-time bound to these steps, or document why the controller already enforces one. The Argo field reference documents
|
||
| container: | ||
| image: quay.io/stackrox-io/ci:automation-flavors-openshift-4-{{ .Chart.Annotations.automationFlavorsVersion }} | ||
| imagePullPolicy: Always | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bound the create step at 2 hours from node start, including Running time.
The
timeout: 2hsetting fails a node only while it isPending. Argo discards the deadline it computes for other phases. Kubelet countsactiveDeadlineSeconds: 7200from pod start, so time spent Pending does not reduce it. A pod that starts at about 119 minutes can run for another 2 hours. The create step can then last about 4 hours, not the intended 2. The same pair of fields appears in the other create templates, includingworkflow-eks.yaml,workflow-gke-default.yaml,workflow-openshift-rosa-hcp.yaml,workflow-openshift-4-demo.yaml, andworkflow-qa-demo.yaml.The create step must end no later than 2 hours after its start, whether the pod is Pending or Running. Choose the enforcement mechanism for that total limit. Apply the same limit to every create template that uses this pair.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Verified against the deployed controller (v4.0.2) — this scenario cannot occur. While
operator.godoes discard the computed deadline for non-Pending nodes (no controller-side enforcement for running pods), the same deadline is applied to the pod spec at creation time inworkflow/controller/workflowpod.go:The deadline is
node.StartedAt + timeout, and it only ever lowers the pod'sactiveDeadlineSeconds. A create pod created after ~119 minutes of Pending getsactiveDeadlineSeconds ≈ 60s, not 7200 — kubelet then enforces the remaining budget. Total elapsed time from node start is bounded at 2h whether the pod is Pending or Running, so the ~4h scenario doesn't arise. No change needed.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Analysis chain
Script executed:
Length of output: 22655
web_search executed:
For Kubernetes Pods, when does pod.status.startTime start the spec.activeDeadlineSeconds clock, especially when a Pod remains Pending and unscheduled? Consult official Kubernetes documentation or source.Length of output: 12411
Script executed:
Length of output: 39067
@tommartensenYou’re right that Argo deducts time spent waiting before pod creation. My earlier comment did not distinguish that case from a pod waiting to be scheduled.The distinction matters here:
createWorkflowPodreturns an existing pod without recalculating its deadline. Kubernetes measuresactiveDeadlineSecondsfrom when the kubelet acknowledges the pod, not from pod creation (Pod API).A pod created promptly but scheduled after 119 minutes can therefore still have nearly two hours of active deadline left. Once the node is Running, Argo’s template timeout check does not fail it.
I haven’t reproduced this on the deployed controller. I’m not marking the finding as addressed.
You are interacting with an AI system.