Skip to content

Add report-only packaged release compatibility checks - #26280

Open
Stella Huang (StellaHuang95) wants to merge 5 commits into
microsoft:mainfrom
StellaHuang95:stellahuang95-core-release-readiness
Open

Stella Huang (StellaHuang95) wants to merge 5 commits into
microsoft:mainfrom
StellaHuang95:stellahuang95-core-release-readiness

Conversation

@StellaHuang95

@StellaHuang95 Stella Huang (StellaHuang95) commented Nov 6, 2026 •

Copy link
Copy Markdown

Summary

Add Windows x64 and Linux x64 compatibility checks for the actual packaged Python extension, paired with a fixed released Python Environments extension. The goal is to catch packaging and cross-extension integration issues before release by verifying discovery, interpreter switching, and real Python execution with Environments explicitly enabled and disabled.

Existing source, native, and packaged smoke tests remain in place. This closes a specific coverage gap: the current Run Python File packaged smoke test explicitly disables Environments, and its execution case skips Windows. It does not imply that all integration coverage was missing or that the released packages are incompatible.

Before and after

Area Existing behavior Added coverage
Packaged execution The Run Python File smoke case exercises the legacy route and skips execution on Windows. Separate Windows/Linux checks discover two real venvs, select and switch between them, and execute through Core's normal command in both modes.
Candidate identity CI builds platform-specific VSIX artifacts. The new jobs consume the original artifact from that same run and retain its version, hash, and producer/source identity; they do not rebuild or substitute a released Core.
Runtime evidence Package contents or terminal creation alone cannot prove which interpreter or discovery backend ran. Check the actual Python executable, environment prefix, and fresh execution token, plus the observed PET supplier or supported legacy discovery backend.

Current flows with this change

Flow What runs
GitHub PR/CI Existing build-vsix jobs -> new Windows/Linux candidate compatibility jobs -> retained reports and raw logs. Each candidate is paired with the pinned released Environments package. Existing build and test jobs are unchanged.
Harness maintenance When relevant tooling changes, a separate workflow runs guard tests, packages the reusable harness, and exercises a pinned released Core/Environments pair on Windows/Linux. This is a control for the harness, not a substitute for candidate testing.
Azure release reporting A new, separately queued report pipeline consumes an exact existing Build artifact and pinned partner inputs from the same Azure project. It is opt-in and pre-sign, with no signing or publishing steps. It is not inserted into the existing production release graph.

The fixed controls currently use Environments 1.42.0 and VS Code 1.110.0; the harness-control pair also uses Core 2026.10.0. Missing or mismatched candidate inputs fail explicitly rather than falling back to that released pair.

What the checks verify

  • Core and Environments load from the inspected VSIX contents, not product source checkouts. Only the small independent controller is a development extension.
  • Each mode has its own process, workspace, profile, and extension directory. Explicit settings are written before activation and checked afterward, including the supported false opt-out.
  • Both fixture venvs must be discovered before selection. Core's normal python.execInTerminal command must produce fresh output matching the selected sys.executable and sys.prefix; opening a terminal is not a pass.
  • Enabled execution must delegate through the real Environments API. When PET runs, its actual startup path and hash must match the expected installed supplier. Disabled mode may legitimately use legacy JavaScript discovery, which requires positive backend evidence rather than an invented PET identity.
  • Reports distinguish assertion failures, infrastructure failures, and cases that did not run. Original package hashes are rechecked; isolated profiles, credential filtering, bounded archive extraction, and owned-process cleanup keep the test independent of a developer installation.

The reusable harness also supports the bundled-PET to Core-PET fallback upgrade used by the companion Environments PR #1994. Upgrade phases use normal isolated windows with disk-backed storage, verify independent storage continuity, and read the saved interpreter selection before any reselection. Product state is not copied or reseeded. This upgrade is not an additional default Core candidate job.

Scope and release impact

There are no production runtime, integration-default, opt-out, dependency, PET-bundling, or existing build/sign/publish changes in this PR. The harness is excluded from the shipped VSIX. No branch-protection or publication gate is enabled.

GitHub candidate checks can fail CI; the separate Azure consumer is deliberately report-only. CI-built package results are not final signed-publication certification. Actual Azure artifact permissions/hosted acceptance, a post-Sign hook, and reconciliation with the exact signed/published bytes remain rollout work.

Coverage is bounded to the pinned pairs and local venv workflow. The current bundled-Environments pair can use Environments' PET when enabled and JavaScript discovery when disabled, so it does not guarantee execution of the candidate Core's own PET in every run.

Runner inputs, reports, and release handoff details are documented in build/releaseCompatibility/README.md.

@StellaHuang95 Stella Huang (StellaHuang95) added no-changelog No news entry required skip package*.json package.json and package-lock.json don't both need updating skip-issue-check labels Nov 6, 2026
@StellaHuang95
Stella Huang (StellaHuang95) force-pushed the stellahuang95-core-release-readiness branch from 36931ab to b698286 Compare November 11, 2026 22:52
@StellaHuang95
Stella Huang (StellaHuang95) marked this pull request as ready for review November 11, 2026 22:53
@StellaHuang95
Stella Huang (StellaHuang95) force-pushed the stellahuang95-core-release-readiness branch from b698286 to 06cca70 Compare November 13, 2026 18:18
@rchiodo

Rich Chiodo (rchiodo) commented Nov 13, 2026 •

Copy link
Copy Markdown

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

@rchiodo

Rich Chiodo (rchiodo) commented Nov 13, 2026 •

Copy link
Copy Markdown

Reviewed commit: 837ed28fc9457941113449e8e2f89c926870f102

Result: 🔴 could-not-verify

Verification details

Verification: Automated verification is best effort. The PR author remains responsible for testing.

Summary: [unavailable] Container verification could not start and local execution was not authorized for this PR HEAD: C:\Program Files\RedHat\Podman\podman.EXE --connection pyrx-automation failed: stderr: Error: unable to start container "89d90b700e7aaa0e060af07bfb22beefb11c8c01c5a9262464bcb99cb3880890": crun: open `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

Test runs: none recorded.

@StellaHuang95
Stella Huang (StellaHuang95) force-pushed the stellahuang95-core-release-readiness branch from 06cca70 to 837ed28 Compare November 13, 2026 21:21

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added the review-auto:approved Automated review: no blocking findings (approval posted). label Nov 13, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95
Stella Huang (StellaHuang95) force-pushed the stellahuang95-core-release-readiness branch from 837ed28 to 2af7f47 Compare November 13, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog No news entry required review-auto:approved Automated review: no blocking findings (approval posted). skip package*.json package.json and package-lock.json don't both need updating skip-issue-check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants