Skip to content

CHORE: Flaky Setup - Gate benchmark and baseline steps on job success - #724

Draft
Gaurav Sharma (bewithgaurav) wants to merge 1 commit into
mainfrom
bewithgaurav/gate-benchmark-steps-on-success
Draft

CHORE: Flaky Setup - Gate benchmark and baseline steps on job success#724
Gaurav Sharma (bewithgaurav) wants to merge 1 commit into
mainfrom
bewithgaurav/gate-benchmark-steps-on-success

Conversation

@bewithgaurav

@bewithgaurav Gaurav Sharma (bewithgaurav) commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Work Item / Issue Reference

ADO Work Item: Fixed AB#47384


Summary

the benchmark and baseline steps keep running after SQL Server setup has already failed, so a dead job still burns 20 minutes benchmarking against a database that does not exist.

build 167929, macOS x86_64 SQL2025:

Build + start SQL Server (Colima boot & SQL setup...)   failed               15.6 min
Install mssql_py_core from NuGet wheels                 skipped
Uninstall system ODBC Driver before running tests       skipped
Run pytest with coverage                                skipped
Download and restore AdventureWorks2022 database        succeeded             0.4 min
Download macOS baseline from main                       succeeded             0.0 min
Prepare macOS baseline file                             succeeded             0.0 min
Run performance benchmarks on macOS SQL2025             succeededWithIssues  20.2 min

pytest was correctly skipped because it relies on the default step condition. the four that ran override it:

condition: or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025'))

that only asks which SQL version the leg is. writing an explicit condition replaces the implicit succeeded() rather than adding to it, so nothing upstream is checked. Publish macOS benchmark baseline already had this right with and(succeeded(), ...).

each condition now starts with succeeded(). the windows leg has the same four steps with the same pattern, and its SQL setup steps carry no continueOnError, so a failure there propagates the same way. eight sites in total across pytestonwindows and PytestOnMacOS.

the benchmark step also carries timeoutInMinutes: 20 and continueOnError: true, so when it times out the job reports succeededWithIssues instead of failing. the macOS SQL2022 leg in that same build finished succeededWithIssues for exactly that reason. this does not change that behaviour, it just stops the step running when there is nothing to benchmark against.

related to AB#47311. that one raised the macOS job timeout to 90 minutes, this one stops a failed setup from eating 20 minutes of the budget.

the AdventureWorks restore, baseline download, baseline prep and benchmark steps only checked which SQL version the leg was running, never whether anything upstream still worked. an explicit condition replaces the implicit succeeded() rather than adding to it, so these kept running after SQL Server setup had already failed. build 167929 spent 20.2 minutes benchmarking against a database that did not exist on a job that was already dead. each condition now starts with succeeded(), matching what the publish-baseline step already did. applies to the windows and macos legs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the pr-size: small Minimal code update label Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-size: small Minimal code update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants