Repository navigation
Stop reprocessing jobs that can never progress, and skip resets that … - #465
Merged
Merged
Conversation
…do nothing. Every run looked at ~49 old jobs with closed bugs, ~24 minutes of a run that was getting close to its 75 minute limit. They were jobs with the obsolete RELINQUISHED status, which we log and skip, and jobs whose try pushes Treeherder no longer has results for. Mark both done once their bug is closed (the try ones only when over a week old, since a new push has no results for a few minutes). And after each job we reset the checkout, ~20 seconds, even when the job never touched it; skip that when it's clean and still at the push's revision.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Removed Treeherder pushes raise before the age/closed-bug cleanup, leaving old jobs with closed bugs retrying instead of being cleaned up.
1 open issue
What changed in this PR
This PR reduces wasted work in the vendoring runner by retiring stalled jobs with closed bugs and avoiding unnecessary checkout resets.
Changes:
- Mark obsolete-status jobs and qualifying old jobs without try results as done when their bugs are closed.
- Skip full resets when the checkout is clean and already at the expected revision.
- Add regression tests and update the existing-job workflow diagrams.
| File | Description |
|---|---|
tests/dead_jobs.py |
Tests job retirement conditions and checkout reset behavior. |
test.py |
Registers the new regression test suite. |
tasktypes/vendoring.py |
Adds stalled-job cleanup and a clean-checkout reset shortcut. |
docs/existing_job.md |
Documents the new job retirement paths. |
Review effort: Balanced
Treeherder eventually deletes old pushes, and then get_job_details raised, after retrying, out of processing the library, so the job was never cleaned up and the library's new versions weren't processed. A missing push now raises TryPushNotFoundError, which isn't retried, and is handled as a push with no jobs. Push health is only requested once the jobs are complete, since it's never needed before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…do nothing.
Every run looked at ~49 old jobs with closed bugs, ~24 minutes of a run that was getting close to its 75 minute limit. They were jobs with the obsolete RELINQUISHED status, which we log and skip, and jobs whose try pushes Treeherder no longer has results for. Mark both done once their bug is closed (the try ones only when over a week old, since a new push has no results for a few minutes). And after each job we reset the checkout, ~20 seconds, even when the job never touched it; skip that when it's clean and still at the push's revision.