Skip to content

Bound transient Windows virtual environment removal retries - #2019

Closed
Stella Huang (StellaHuang95) wants to merge 2 commits into
microsoft:mainfrom
StellaHuang95:stellahuang-microsoft-ci-failure-analysis
Closed

Stella Huang (StellaHuang95) wants to merge 2 commits into
microsoft:mainfrom
StellaHuang95:stellahuang-microsoft-ci-failure-analysis

Conversation

@StellaHuang95

@StellaHuang95 Stella Huang (StellaHuang95) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Replace the test-only isolation revision with a conservative production fix for short Windows sharing failures. Restore the original in-workspace environment-creation integration tests unchanged, including their workspace-association coverage.
  • Continue using fs-extra.remove; retry only EPERM/EBUSY on Windows, around the entire deletion operation. Allow at most five attempts, with 100ms backoff and one monotonic 500ms retry-eligibility window. Do not enable native recursive per-entry retries.
  • Capture the root directory's bigint device/inode identity and check it before each retry. Stop if the path names a replacement directory; if identity is unavailable, retain the original single-attempt behavior. Retry inside the already-validated removal because partial deletion can remove pyvenv.cfg.
  • Preserve confirmation, validation, localized error reporting, and untracking only after successful deletion. Non-Windows behavior is unchanged. Do not abandon or race an active filesystem operation, retry the public API, or add Mocha retries.

Behavior and limits

The window limits when another deletion attempt may start; it is not a hard 500ms timeout for filesystem I/O. A slow first attempt is not retried if it exhausts the window, and a slow successful deletion is still awaited. Fast persistent Windows sharing failures can take a few hundred milliseconds longer to report. The directory check prevents retrying an observed replacement, but does not make concurrent filesystem mutation atomic.

This avoids the earlier native-recursive retry policy whose waits multiplied across a directory tree. Controlled Windows checks with real Python virtual environments reproduced the failure mechanism: a temporary executable lock failed with the original helper (19ms) and succeeded with this helper (127ms); a persistent lock failed with both (6ms versus 442ms). A slower deeply nested locked executable received no additional retries (about 2.4s versus 2.1s). Replacement-directory and junction-target checks preserved unrelated files.

The original CI logs do not identify the filesystem error or lock-holding process, so this is a verified recovery for a reproduced mechanism, not a claim that a specific Python probe caused those CI failures. The original creation and interpreter-resolution operations are awaited before deletion.

Relates to #2012 and #2015.

Validation

  • npm run lint, npm run compile-tests, and npm run unittest: latest full unit run passed with 3,580 passing and 9 pending tests.
  • The focused discovery/removal suites passed with 56 passing and 2 platform-specific pending tests. A separate full run hit two existing 1-second discovery-test deadlines in unchanged initialization code; those tests do not call removal, and no test timeouts or retries were changed.
  • npm run compile passed after the rebase; focused removal/orchestration tests passed with 36 passing and 2 platform-specific pending tests.
  • The original three workspace lifecycle integration tests passed repeatedly in fresh real VS Code hosts, including the final production bundle, with --forbid-pending --fail-zero and no Mocha retries.
  • Fifteen deterministic fake-timer cases cover transient and persistent locks, partial deletion, exhausted budgets, delayed timers/metadata reads, in-flight success, replacement/missing directories, unavailable identities, non-Windows behavior, non-sharing errors, validation, and untracking failures. A control using the original helper fails the relevant new recovery/budget cases.
  • Rebased onto the latest fetched main before pushing. The net diff contains only venvUtils.ts and its removal unit tests; the workspace integration file matches main.
  • Both rebased PR commits are GPG-signed: 521daed1 and 92f0ac0f. CI results for this pushed revision must be assessed separately; this PR has not been merged.

@StellaHuang95 Stella Huang (StellaHuang95) added the bug Issue identified by VS Code Team member as probable bug label Nov 13, 2026
@rchiodo

Rich Chiodo (rchiodo) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

@rchiodo

Rich Chiodo (rchiodo) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Reviewed commit: 15ba12d6438cbee3e83a3268413e09ce0673f291

Result: 🔴 could-not-verify

Verification details

Verification: Automated verification is best effort. The PR author remains responsible for testing.

Summary: [unavailable] Container verification could not start and local execution was not authorized for this PR HEAD: Podman machine 'pyrx-automation' is not running.

Test runs: none recorded.

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added the review-auto:approved Automated review: no blocking findings (approval posted). label Nov 13, 2026
@StellaHuang95
Stella Huang (StellaHuang95) force-pushed the stellahuang-microsoft-ci-failure-analysis branch from 254306f to 15ba12d Compare November 13, 2026 00:59
@StellaHuang95 Stella Huang (StellaHuang95) changed the title Retry transient Windows virtual environment removal failures Isolate virtual environment creation integration tests Nov 13, 2026
Create real environments in an owned temporary directory outside the open workspace, avoiding activation-script watcher refreshes during deletion. Compare discovered environments by their manager and normalized prefix. Preserve the actual API removal and discovery assertions; retry only owned fixture teardown. Restore production deletion behavior unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retry only Windows EPERM/EBUSY failures around the entire deletion, with a short shared eligibility window and directory identity checks. Preserve the original removal primitive, confirmation, validation, untracking order, and workspace integration coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95
Stella Huang (StellaHuang95) force-pushed the stellahuang-microsoft-ci-failure-analysis branch from 15ba12d to 92f0ac0 Compare November 13, 2026 18:33
@StellaHuang95 Stella Huang (StellaHuang95) changed the title Isolate virtual environment creation integration tests Bound transient Windows virtual environment removal retries Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Issue identified by VS Code Team member as probable bug review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants