From 0a21e2e292cf525cd966f43dc94fde887d5f8484 Mon Sep 17 00:00:00 2001 From: Pavel Feldman Date: Mon, 9 Nov 2026 17:22:58 -0800 Subject: [PATCH] chore(bots): hand triage, review, roll and release to the Playwright bot --- .claude/skills/playwright-triage/SKILL.md | 88 ++----- .github/workflows/bots.yml | 69 ----- .github/workflows/release.yml | 216 ++------------- .github/workflows/review.yml | 33 +++ .github/workflows/revision.yml | 138 ---------- .github/workflows/roll.yml | 305 ++-------------------- .github/workflows/triage.yml | 234 ++--------------- CLAUDE.md => AGENTS.md | 16 +- 8 files changed, 126 insertions(+), 973 deletions(-) delete mode 100644 .github/workflows/bots.yml create mode 100644 .github/workflows/review.yml delete mode 100644 .github/workflows/revision.yml rename CLAUDE.md => AGENTS.md (61%) diff --git a/.claude/skills/playwright-triage/SKILL.md b/.claude/skills/playwright-triage/SKILL.md index 7a9e9c7..9c8550d 100644 --- a/.claude/skills/playwright-triage/SKILL.md +++ b/.claude/skills/playwright-triage/SKILL.md @@ -1,18 +1,10 @@ --- name: playwright-triage -description: Triage and fix one microsoft/playwright-cli issue, or do a maintainer's task, unattended inside the Issue triage GitHub Actions workflow (.github/workflows/triage.yml), which a needs-triage, needs-triage-mac or needs-triage-win label starts, or a maintainer starts by hand with an issue, a task or both. Reproduces the issue on the runner, then leaves a verified fix proposal for the workflow to open as a draft PR, in this repo or in microsoft/playwright where the source of the CLI is, or an investigation note for it to post on the issue. Only for that workflow. +description: Triage and fix an issue in microsoft/playwright-cli. Where the code of the CLI is, how to reproduce on tip-of-tree in microsoft/playwright, where a fix goes, how to run a published version, and what not to run from a PR of a third party. --- # Playwright CLI Issue Triage -You run unattended in the [Issue triage](../../../.github/workflows/triage.yml) workflow of -`microsoft/playwright-cli`. First read the shared guide, `$AGENT_GUIDES_DIR/triage.md`, a copy of -[microsoft/playwright/.github/actions/run-agent/guides/triage.md](https://github.com/microsoft/playwright/blob/main/.github/actions/run-agent/guides/triage.md), -and then [CLAUDE.md](../../../CLAUDE.md), both in full. The guide says how to triage an issue in any -Playwright repo. CLAUDE.md says how to build, test and lint this one. This file covers what neither -has: where the code is, the checkouts, how to run a published version, and the outcome that this -repo adds, a fix in `microsoft/playwright`. - ## Where the code is This repo is the npm package `@playwright/cli`: `playwright-cli.js` with its update check, @@ -27,72 +19,50 @@ Fix a bug here only when the cause is in a file that this repo owns. `skills/pla copy of `packages/playwright-core/src/tools/skills/playwright-cli/`, which `scripts/update.js` refreshes on a roll, so its fix is upstream too. -## The checkouts +## Reproduce on tip-of-tree -`$REPO_DIR` is the checkout of `microsoft/playwright-cli` that the guide describes. `npm ci` ran in -it, and the browsers of the pinned version are installed. The CLI uses Google Chrome from the -runner image by default. When it is missing, install it with `npx playwright install chrome`. The -clone is shallow: run `git fetch --unshallow --tags origin` in it -before you need its history. +The pinned alpha is older than tip-of-tree. `npm view playwright-core@ gitHead` prints the +upstream commit that an alpha was built from. -`$PLAYWRIGHT_DIR` is a clone of `microsoft/playwright` at the tip of `main`, next to `$REPO_DIR`. -`npm ci` and `npm run build` ran in it, and Chromium, Firefox and WebKit are installed. It is -tip-of-tree for this repo: reproduce there, as a test in its `tests/mcp/cli-*.spec.ts`, with the -`cli` fixture of `tests/mcp/cli-fixtures.ts`. `npm run test-playwright-cli` runs those tests. Read -its `AGENTS.md` in full before you write the test. It says how to build, test and lint it. The -clone has the full history, without file contents until git needs them, so `git log` and -`git tag --contains` work there. +Reproduce in a checkout of `microsoft/playwright` at the tip of `main`, next to this repo, in +`../playwright`. Write the repro as a test in its `tests/mcp/cli-*.spec.ts`, with the `cli` fixture +of `tests/mcp/cli-fixtures.ts`. `npm run test-playwright-cli` runs those tests. Read its `AGENTS.md` +in full before you write the test. It says how to build, test and lint it. -The pinned version is older than tip-of-tree. `npm view playwright-core@ gitHead` prints the -upstream commit that an alpha was built from. +The CLI uses Google Chrome by default. When it is missing, install it with +`npx playwright install chrome`. ## Fix in microsoft/playwright -When the fix is in `$PLAYWRIGHT_DIR`, follow the Fix section of the guide there, in place of -`$REPO_DIR`: create the branch `fix-` from `main` in `$PLAYWRIGHT_DIR`, put the regression test -in its `tests/mcp/cli-*.spec.ts`, build and lint as its `AGENTS.md` says, and commit in its -convention, for example `fix(cli): ...`. Then leave the proposal in -`$OUTPUT_DIR/upstream-fix/` in place of `$OUTPUT_DIR/fix/`: - -- `fix.patch`: the output of `git diff --binary main HEAD` in `$PLAYWRIGHT_DIR`. -- `pr-body.md`: as the guide says. The `Fixes` line has the full URL of the issue, - `Fixes https://github.com/microsoft/playwright-cli/issues/`, so that the PR closes the issue in - this repo. - -Add the `pull-request` call for `microsoft/playwright` last, as the guide says: - -```json -{ - "call": "pull-request", - "repo": "microsoft/playwright", - "branch": "triage/playwright-cli-/-", - "title": "fix(cli): description", - "patch": "upstream-fix/fix.patch", - "bodyFile": "upstream-fix/pr-body.md", - "commitBody": "Fixes: https://github.com/microsoft/playwright-cli/issues/" -} -``` +When the cause is upstream, fix it in `../playwright`: create the branch `fix-` from `main`, put +the regression test in its `tests/mcp/cli-*.spec.ts`, build and lint as its `AGENTS.md` says, and +commit in its convention, for example `fix(cli): ...`. The PR there says +`Refs https://github.com/microsoft/playwright-cli/issues/`: the issue stays open until this repo +rolls to the fix. Open one PR, here or in `microsoft/playwright`, not both. Do not change +`package.json` here in the same change. -The publish job checks out `microsoft/playwright` at the commit of `$PLAYWRIGHT_DIR` and opens the -draft PR there. The revision workflow of that repo addresses the review. Make one `pull-request` -call: in this repo or in `microsoft/playwright`, not both. Do not change -`package.json` here to pick up the fix. It reaches `@playwright/cli` with the next roll. +After the upstream PR merges, the fix reaches this repo with a roll. Check that `playwright@next` +has it: +`git -C ../playwright merge-base --is-ancestor `. If +it has the fix, roll this repo to it, as [roll.md](../dev/roll.md) says, and add a test here when +the tests of this repo can cover the issue. That PR says `Fixes `. If it does not have +the fix yet, the next alpha brings it. **Check the linked PRs in both repos.** A PR in `microsoft/playwright` links this issue with its full URL or as `microsoft/playwright-cli#`. Search there too: `repo:microsoft/playwright is:pr "playwright-cli/issues/"` and `repo:microsoft/playwright is:pr "playwright-cli#"`. A PR there from the team or from -`microsoft-playwright-automation` is a team PR. Verify a PR there in `$PLAYWRIGHT_DIR`. +`microsoft-playwright-automation` is a team PR. Verify a PR there in `../playwright`. **Already fixed** means that tip-of-tree passes and the pinned version fails. The fix reaches users with the next roll. Find the upstream PR with -`git -C $PLAYWRIGHT_DIR log ..main -- packages/playwright-core/src/tools`, -and name it in the note. +`git -C ../playwright log ..main -- packages/playwright-core/src/tools`, +and name it. ## Published versions -To run a published version, for example the version of the reporter or to bisect, create -`$SCRATCH_DIR/playwright-cli-//` and run there: +To run a published version, for example the version of the reporter or to bisect, create an empty +directory outside the repo for each version, and run there: ```bash npm init -y @@ -107,10 +77,10 @@ that no daemon stays open. `npm view @playwright/cli@ dependencies.playwright-core` prints the alpha of a release. Match a regression to upstream commits with -`git -C $PLAYWRIGHT_DIR log .. -- packages/playwright-core/src/tools`. +`git -C ../playwright log .. -- packages/playwright-core/src/tools`. ## PRs from third parties Do not run a PR in this repo that changes `package.json`, `package-lock.json`, `scripts/`, `.github/`, `.azure-pipelines/` or `.claude/`. For a PR in `microsoft/playwright`, the list in -`$PLAYWRIGHT_DIR/.claude/skills/playwright-triage/SKILL.md` applies. +`../playwright/.claude/skills/playwright-triage/SKILL.md` applies. diff --git a/.github/workflows/bots.yml b/.github/workflows/bots.yml deleted file mode 100644 index 52dfa01..0000000 --- a/.github/workflows/bots.yml +++ /dev/null @@ -1,69 +0,0 @@ -# The Playwright bot runs in microsoft/devops.playwright.dev. This workflow hands it the request -# when a maintainer applies a label: agent-triage, agent-implement or agent-task on an issue starts -# issue.yml there, and agent-review on a pull request starts pr.yml. Nothing else of the bot lives -# in this repo. Apply the label again to start another run. -# -# The same file is in every Playwright repo, as .github/workflows/bots.yml. Its source is -# bots/shim.yml in microsoft/devops.playwright.dev, and README.md there describes the flows. -name: Bots - -on: - issues: - types: [labeled] - # Runs from the default branch, for PRs from forks too. It never checks out the PR. - pull_request_target: - types: [labeled] - -permissions: {} - -jobs: - issue: - name: Hand the issue to the bot - if: >- - github.repository_owner == 'microsoft' && github.event_name == 'issues' && ( - github.event.label.name == 'agent-triage' || - github.event.label.name == 'agent-implement' || - github.event.label.name == 'agent-task') - runs-on: ubuntu-latest - timeout-minutes: 5 - steps: - - uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 - id: app-token - with: - client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }} - private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }} - owner: microsoft - repositories: devops.playwright.dev - permission-actions: write - - name: Start issue.yml in microsoft/devops.playwright.dev - env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} - NUMBER: ${{ github.event.issue.number }} - LABEL: ${{ github.event.label.name }} - run: >- - gh workflow run issue.yml --repo microsoft/devops.playwright.dev --ref main - -f repo="$GITHUB_REPOSITORY" -f issue="$NUMBER" -f label="$LABEL" - - pr: - name: Hand the PR to the bot - if: >- - github.repository_owner == 'microsoft' && github.event_name == 'pull_request_target' && - github.event.label.name == 'agent-review' - runs-on: ubuntu-latest - timeout-minutes: 5 - steps: - - uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 - id: app-token - with: - client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }} - private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }} - owner: microsoft - repositories: devops.playwright.dev - permission-actions: write - - name: Start pr.yml in microsoft/devops.playwright.dev - env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} - NUMBER: ${{ github.event.pull_request.number }} - run: >- - gh workflow run pr.yml --repo microsoft/devops.playwright.dev --ref main - -f repo="$GITHUB_REPOSITORY" -f pr="$NUMBER" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ba2483b..46b55f4 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,215 +1,33 @@ -# Prepares a release. Copilot CLI writes the release notes, as .claude/skills/dev/release.md says, -# without write credentials. Fresh jobs bump the version, open the "chore: mark v" pull -# request with the notes as its description using the app, so CI runs on it, and comment on the -# issue. A maintainer copies the notes into the release draft. -name: Prepare release +# Starts the Playwright bot with the task below. The bot does it with the skills of this repo, and +# opens the pull request as the app. Start it from the Actions tab. +name: "Agent: prepare release" on: - # Apply needs-release to an issue; re-apply it to write the notes again. - issues: - types: [labeled] workflow_dispatch: - inputs: - issue: - description: "Issue to report on; leave empty for none" - type: string - required: false - default: "" - version: - description: "Version to release; leave empty for the next patch version" - type: string - required: false - default: "" permissions: {} jobs: start: - name: Find the version - if: >- - github.repository == 'microsoft/playwright-cli' && - (github.event_name == 'workflow_dispatch' || github.event.label.name == 'needs-release') + name: Hand the task to the bot + if: github.repository == 'microsoft/playwright-cli' runs-on: ubuntu-latest - permissions: - contents: read - issues: write - outputs: - version: ${{ steps.version.outputs.version }} - comment: ${{ steps.post.outputs.id }} - env: - ISSUE: ${{ github.event.issue.number || inputs.issue }} - VERSION: ${{ inputs.version }} - GH_TOKEN: ${{ github.token }} - GH_REPO: ${{ github.repository }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - - name: Find the next version - id: version - run: | - current=$(node -p "require('./package.json').version") - version=${VERSION:-$(node -p 'const [major, minor, patch] = process.argv[1].split("."); `${major}.${minor}.${+patch + 1}`' "$current")} - [[ "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || { echo "invalid version: $version" >&2; exit 1; } - echo "Releasing v$version; package.json has $current" >> "$GITHUB_STEP_SUMMARY" - echo "version=$version" >> "$GITHUB_OUTPUT" - - name: Post the run link - id: post - if: env.ISSUE != '' - env: - NEXT: ${{ steps.version.outputs.version }} - run: | - [[ "$ISSUE" =~ ^[1-9][0-9]*$ ]] || { echo "issue must be a positive integer" >&2; exit 1; } - body=$(printf "Hi, I'm the Playwright bot. I started to write the release notes of v%s: [agent run](%s)\n" "$NEXT" "$RUN_URL") - id=$(gh api "repos/$GH_REPO/issues/$ISSUE/comments" -f body="$body" --jq .node_id) - echo "id=$id" >> "$GITHUB_OUTPUT" - - notes: - name: Write the release notes of v${{ needs.start.outputs.version }} - needs: start - runs-on: ubuntu-latest - timeout-minutes: 90 - concurrency: - group: release - cancel-in-progress: false - defaults: - run: - shell: bash - permissions: - contents: read - copilot-requests: write - env: - VERSION: ${{ needs.start.outputs.version }} - outputs: - has_notes: ${{ steps.agent.outputs.has-result }} - steps: - # The full history, for the previous release. No credentials in the checkout. - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - fetch-depth: 0 - persist-credentials: false - - name: Record the directories - run: | - echo "OUTPUT_DIR=$RUNNER_TEMP/release" >> "$GITHUB_ENV" - echo "SCRATCH_DIR=$RUNNER_TEMP/scratch" >> "$GITHUB_ENV" - echo "REPO_DIR=$GITHUB_WORKSPACE" >> "$GITHUB_ENV" - echo "PLAYWRIGHT_DIR=$(dirname "$GITHUB_WORKSPACE")/playwright" >> "$GITHUB_ENV" - mkdir -p "$RUNNER_TEMP/release" "$RUNNER_TEMP/scratch" - - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 - with: - node-version: '24' - - name: Clone microsoft/playwright and install the Copilot CLI - run: | - git clone --filter=blob:none https://github.com/microsoft/playwright.git "$PLAYWRIGHT_DIR" - npm install -g @github/copilot - - name: Write the notes with Copilot CLI - id: agent - uses: microsoft/playwright/.github/actions/run-agent@d1518e3e3fca90352c4390814943cced496fd7e4 # main - with: - prompt: | - Write the release notes of @playwright/cli v${{ env.VERSION }}. Follow steps 2 to 7 of .claude/skills/dev/release.md. Read the skill in full before you start. - - The workflow does the other steps. A later job does step 1, the version bump, and step 8: it commits, pushes and opens the pull request with your notes as its description. The notes cover the Playwright version that package.json on main pins. Do not change, commit or push files in the checkout, and do not run gh. - - `~/code/playwright` in the skill is $PLAYWRIGHT_DIR here: a clone of microsoft/playwright at the tip of main, with the full history. The checkout of this repo has the full history too. `gh` is not authenticated: read pull requests and issues with the GitHub MCP tools. - - Nobody watches the run: do not ask questions. - - Keep $OUTPUT_DIR/summary.md up to date as you work: the commit window, what you kept, what you dropped and why. Draft the notes in $SCRATCH_DIR. When they are done, write them to $OUTPUT_DIR/release-notes.md, in place of RELEASE_NOTES_v.md, last, in one go. - output-dir: ${{ env.OUTPUT_DIR }} - result-file: release-notes.md - artifact-name: release - github-tools: true - - publish: - name: Open the pull request - needs: [start, notes] - # release-notes.md is written last, so finished notes are published even when the job failed later. - if: ${{ !cancelled() && needs.notes.outputs.has_notes == 'true' }} - runs-on: ubuntu-latest - outputs: - pr: ${{ steps.pr.outputs.url }} - env: - VERSION: ${{ needs.start.outputs.version }} - TITLE: "chore: mark v${{ needs.start.outputs.version }}" - BRANCH: release/v${{ needs.start.outputs.version }} + timeout-minutes: 5 steps: - uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 id: app-token with: client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }} private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }} - # Pushes as the app, so CI runs on the pull request. - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - token: ${{ steps.app-token.outputs.token }} - - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 - with: - node-version: '24' - - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 - with: - name: release - path: ${{ runner.temp }}/release - # Step 1 of the skill. Only the notes come from the agent, as the description. - - name: Bump the version and push the branch - run: | - npm version "$VERSION" --no-git-tag-version --ignore-scripts --allow-same-version - git config user.name "microsoft-playwright-automation[bot]" - git config user.email "203992400+microsoft-playwright-automation[bot]@users.noreply.github.com" - git checkout -b "$BRANCH" - git commit -am "$TITLE" - git push --force origin "$BRANCH" - # The description is the notes verbatim, for a maintainer to copy into the release draft. - - name: Open the pull request - id: pr - uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 - env: - BASE: ${{ github.ref_name }} - with: - github-token: ${{ steps.app-token.outputs.token }} - script: | - const fs = require('fs'); - const { BASE, BRANCH, TITLE, RUNNER_TEMP } = process.env; - const body = fs.readFileSync(`${RUNNER_TEMP}/release/release-notes.md`, 'utf8').trim(); - if (!body) - throw new Error('release-notes.md is empty'); - const { owner, repo } = context.repo; - const { data: open } = await github.rest.pulls.list({ owner, repo, head: `${owner}:${BRANCH}`, state: 'open' }); - let pr = open[0]; - if (pr) - await github.rest.pulls.update({ owner, repo, pull_number: pr.number, body }); - else - ({ data: pr } = await github.rest.pulls.create({ owner, repo, head: BRANCH, base: BASE, title: TITLE, body })); - core.setOutput('url', pr.html_url); - await core.summary.addRaw(`Pull request: ${pr.html_url}\n`).write(); - - comment: - name: Comment on the issue - needs: [start, notes, publish] - if: ${{ !cancelled() && needs.start.result == 'success' && (github.event_name == 'issues' || inputs.issue != '') }} - runs-on: ubuntu-latest - permissions: - issues: write - steps: - - name: Post the result + owner: microsoft + repositories: devops.playwright.dev + permission-actions: write + - name: Start the bot env: - GH_TOKEN: ${{ github.token }} - GH_REPO: ${{ github.repository }} - ISSUE: ${{ github.event.issue.number || inputs.issue }} - VERSION: ${{ needs.start.outputs.version }} - PR_URL: ${{ needs.publish.outputs.pr }} - START_COMMENT: ${{ needs.start.outputs.comment }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: | - if [[ -n "$PR_URL" ]]; then - text="I opened the release PR of v$VERSION. Its description has the release notes, ready to copy into the release draft: $PR_URL" - else - text="I couldn't prepare the release of v$VERSION. The agent run has the details." - fi - printf "Hi, I'm the Playwright bot. %s\n\nPrepared by the Playwright bot - [agent run](%s)\n" "$text" "$RUN_URL" > "$RUNNER_TEMP/comment.md" - gh issue comment "$ISSUE" --body-file "$RUNNER_TEMP/comment.md" - if [[ -n "$START_COMMENT" ]]; then - gh api graphql -f query='mutation($id: ID!) { minimizeComment(input: { subjectId: $id, classifier: OUTDATED }) { clientMutationId } }' -f id="$START_COMMENT" \ - || echo "::warning::Could not hide the comment with the run link" - fi + GH_TOKEN: ${{ steps.app-token.outputs.token }} + TASK: >- + Prepare the release of the next patch version, as .claude/skills/dev/release.md says. + The notes cover the Playwright version that main pins. + run: >- + gh workflow run task.yml --repo microsoft/devops.playwright.dev --ref main + -f repo="$GITHUB_REPOSITORY" -f task="$TASK" diff --git a/.github/workflows/review.yml b/.github/workflows/review.yml new file mode 100644 index 0000000..56f9ec9 --- /dev/null +++ b/.github/workflows/review.yml @@ -0,0 +1,33 @@ +# Hands the pull request to the Playwright bot when a maintainer applies agent-review. Nothing +# else of the bot lives in this repo. Apply the label again to start another run. +name: "Agent: review" + +on: + # Runs from the default branch, for PRs from forks too. It never checks out the PR. + pull_request_target: + types: [labeled] + +permissions: {} + +jobs: + pr: + name: Hand the PR to the bot + if: github.repository_owner == 'microsoft' && github.event.label.name == 'agent-review' + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 + id: app-token + with: + client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }} + private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }} + owner: microsoft + repositories: devops.playwright.dev + permission-actions: write + - name: Start the bot + env: + GH_TOKEN: ${{ steps.app-token.outputs.token }} + NUMBER: ${{ github.event.pull_request.number }} + run: >- + gh workflow run pr.yml --repo microsoft/devops.playwright.dev --ref main + -f repo="$GITHUB_REPOSITORY" -f pr="$NUMBER" diff --git a/.github/workflows/revision.yml b/.github/workflows/revision.yml deleted file mode 100644 index 1d8f5b6..0000000 --- a/.github/workflows/revision.yml +++ /dev/null @@ -1,138 +0,0 @@ -# Copilot addresses the maintainers' review comments on a draft PR that the triage workflow opened -# in this repo, without write credentials, on the PR rebased on main. The agent leaves the calls to -# make on GitHub in calls.json. A fresh job checks them and makes them using the app: it replaces -# the PR's branch with the rebased commits and the new one, so CI runs on it, and replies in the -# review threads. The PRs that the -# triage workflow opens in microsoft/playwright are revised there. The steps that are the same for -# every Playwright repo are actions in microsoft/playwright, and the shared guide they carry says how -# to revise; the prompt below has what is specific to this repo. -name: PR revision - -on: - # Apply needs-revision, needs-revision-mac or needs-revision-win; re-apply it to trigger another pass. - # pull_request_target always runs this file from main; the jobs only accept the triage app's own PRs. - pull_request_target: - types: [labeled] - -permissions: {} - -jobs: - prepare: - name: Collect the maintainers' comments - if: >- - github.repository == 'microsoft/playwright-cli' && - contains(fromJSON('["needs-revision","needs-revision-mac","needs-revision-win"]'), github.event.label.name) && - github.event.pull_request.state == 'open' && - github.event.pull_request.user.login == 'microsoft-playwright-automation[bot]' && - github.event.pull_request.head.repo.full_name == github.repository - runs-on: ubuntu-latest - permissions: - issues: read - pull-requests: write - env: - PLATFORM: ${{ fromJSON('{"needs-revision":"linux","needs-revision-mac":"mac","needs-revision-win":"windows"}')[github.event.label.name] }} - outputs: - head: ${{ steps.collect.outputs.head }} - base: ${{ steps.collect.outputs.base }} - branch: ${{ steps.collect.outputs.branch }} - platform: ${{ env.PLATFORM }} - steps: - - name: Collect the PR and the maintainers' comments - id: collect - uses: microsoft/playwright/.github/actions/collect-review@d1518e3e3fca90352c4390814943cced496fd7e4 # main - with: - pull-request: ${{ github.event.pull_request.number }} - - name: Post the run link - env: - GH_TOKEN: ${{ github.token }} - PR: ${{ github.event.pull_request.number }} - PLATFORM_NAME: ${{ fromJSON('{"linux":"Linux","mac":"macOS","windows":"Windows"}')[env.PLATFORM] }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: | - body=$(printf "Hi, I'm the Playwright bot. I started to address the review comments on %s: [agent run](%s)\n" "$PLATFORM_NAME" "$RUN_URL") - gh pr comment "$PR" --body "$body" -R "$GITHUB_REPOSITORY" || echo "::warning::Could not post the run link" - - revise: - name: Revise PR ${{ github.event.pull_request.number }} (${{ needs.prepare.outputs.platform }}) - needs: prepare - runs-on: ${{ fromJSON('{"linux":"playwright-x64-ubuntu24-64-core","mac":"macos-15-xlarge","windows":"playwright-x64-server2025-64-core"}')[needs.prepare.outputs.platform] }} - timeout-minutes: 240 - concurrency: - group: revision-${{ github.event.pull_request.number }} - cancel-in-progress: false - defaults: - run: - shell: bash - env: - PR: ${{ github.event.pull_request.number }} - HEAD_SHA: ${{ needs.prepare.outputs.head }} - BASE_SHA: ${{ needs.prepare.outputs.base }} - TRIAGE_PLATFORM: ${{ needs.prepare.outputs.platform }} - permissions: - copilot-requests: write - outputs: - has_revision: ${{ steps.agent.outputs.has-result }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: ${{ needs.prepare.outputs.head }} - persist-credentials: false - - name: Rebase on main - uses: microsoft/playwright/.github/actions/rebase-pr@d1518e3e3fca90352c4390814943cced496fd7e4 # main - with: - onto: ${{ needs.prepare.outputs.base }} - - name: Record the directories - run: | - echo "OUTPUT_DIR=${RUNNER_TEMP//\\//}/revision" >> "$GITHUB_ENV" - echo "REVIEW_DIR=${RUNNER_TEMP//\\//}/review" >> "$GITHUB_ENV" - echo "SCRATCH_DIR=${RUNNER_TEMP//\\//}/scratch" >> "$GITHUB_ENV" - echo "REPO_DIR=${GITHUB_WORKSPACE//\\//}" >> "$GITHUB_ENV" - mkdir -p "${RUNNER_TEMP//\\//}/revision" "${RUNNER_TEMP//\\//}/scratch" - - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 - with: - name: review - path: ${{ runner.temp }}/review - - name: Set up the runner - uses: ./.github/actions/setup-triage-runner - - name: Revise with Copilot CLI - id: agent - # No GitHub tools and no URL access: the agent reads the PR only from the review the prepare job filtered. - uses: microsoft/playwright/.github/actions/run-agent@d1518e3e3fca90352c4390814943cced496fd7e4 # main - with: - prompt: | - Follow the shared guide revision.md and this repo's CLAUDE.md. Read both in full before starting. $PLAYWRIGHT_DIR is a clone of microsoft/playwright at the tip of main, where the source of the CLI is, built; read it, but do not edit, commit or switch branches in it. - - Pull request: https://github.com/microsoft/playwright-cli/pull/${{ github.event.pull_request.number }} - output-dir: ${{ env.OUTPUT_DIR }} - result-file: calls.json - artifact-name: revision - exec: ${{ env.CI_ENV_EXEC }} - - push: - name: Push the revision - needs: [prepare, revise] - # calls.json is written when the revision is done, so it is pushed even if the job failed later. - if: ${{ !cancelled() && needs.revise.outputs.has_revision == 'true' }} - runs-on: ubuntu-latest - steps: - - uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 - id: app-token - with: - client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }} - private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }} - permission-contents: write - permission-pull-requests: write - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: ${{ needs.prepare.outputs.head }} - token: ${{ steps.app-token.outputs.token }} - - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 - with: - name: revision - path: ${{ runner.temp }}/revision - # Never execute code or scripts from the artifact in the app job. The calls are data. - - name: Check and make the calls - uses: microsoft/playwright/.github/actions/publish-calls@d1518e3e3fca90352c4390814943cced496fd7e4 # main - with: - token: ${{ steps.app-token.outputs.token }} - dir: ${{ runner.temp }}/revision diff --git a/.github/workflows/roll.yml b/.github/workflows/roll.yml index d8d4d84..4690011 100644 --- a/.github/workflows/roll.yml +++ b/.github/workflows/roll.yml @@ -1,302 +1,35 @@ -# Rolls Playwright to the latest alpha, playwright@next, and opens a pull request. The roll job bumps -# the pins and refreshes the skill, then Copilot CLI updates the README, runs the tests and fixes what -# the roll breaks, as .claude/skills/dev/roll.md says, without write credentials. Fresh jobs open the -# pull request using the app, so CI runs on it, and comment on the issue. -name: Roll Playwright +# Starts the Playwright bot with the task below. The bot does it with the skills of this repo, and +# opens the pull request as the app. Start it from the Actions tab. +name: "Agent: roll Playwright" on: - # Apply needs-roll to an issue; re-apply it to roll again. - issues: - types: [labeled] workflow_dispatch: - inputs: - issue: - description: "Issue to report on; leave empty for none" - type: string - required: false - default: "" - version: - description: "Playwright version to roll to; leave empty for playwright@next" - type: string - required: false - default: "" permissions: {} jobs: start: - name: Find the version - if: >- - github.repository == 'microsoft/playwright-cli' && - (github.event_name == 'workflow_dispatch' || github.event.label.name == 'needs-roll') + name: Hand the task to the bot + if: github.repository == 'microsoft/playwright-cli' runs-on: ubuntu-latest - permissions: - contents: read - issues: write - outputs: - old: ${{ steps.versions.outputs.old }} - new: ${{ steps.versions.outputs.new }} - old_commit: ${{ steps.versions.outputs.old_commit }} - new_commit: ${{ steps.versions.outputs.new_commit }} - comment: ${{ steps.post.outputs.id }} - env: - ISSUE: ${{ github.event.issue.number || inputs.issue }} - VERSION: ${{ inputs.version }} - GH_TOKEN: ${{ github.token }} - GH_REPO: ${{ github.repository }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - - name: Read the pinned and the new version - id: versions - run: | - old=$(node -p "require('./package.json').dependencies['playwright-core']") - new=${VERSION:-$(npm view playwright@next version)} - [[ "$new" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] || { echo "invalid version: $new" >&2; exit 1; } - new_commit=$(npm view "playwright-core@$new" gitHead) - [[ "$new_commit" =~ ^[0-9a-f]{40}$ ]] || { echo "playwright-core@$new has no gitHead" >&2; exit 1; } - old_commit=$(npm view "playwright-core@$old" gitHead || true) - echo "Pinned $old ($old_commit), rolling to $new ($new_commit)" >> "$GITHUB_STEP_SUMMARY" - { - echo "old=$old" - echo "new=$new" - echo "old_commit=$old_commit" - echo "new_commit=$new_commit" - } >> "$GITHUB_OUTPUT" - - name: Post the run link - id: post - if: env.ISSUE != '' - env: - NEW: ${{ steps.versions.outputs.new }} - run: | - [[ "$ISSUE" =~ ^[1-9][0-9]*$ ]] || { echo "issue must be a positive integer" >&2; exit 1; } - body=$(printf "Hi, I'm the Playwright bot. I started to roll Playwright to %s: [agent run](%s)\n" "$NEW" "$RUN_URL") - id=$(gh api "repos/$GH_REPO/issues/$ISSUE/comments" -f body="$body" --jq .node_id) - echo "id=$id" >> "$GITHUB_OUTPUT" - - roll: - name: Roll to ${{ needs.start.outputs.new }} - needs: start - if: needs.start.outputs.new != needs.start.outputs.old - runs-on: ubuntu-latest - timeout-minutes: 120 - concurrency: - group: roll - cancel-in-progress: false - defaults: - run: - shell: bash - permissions: - contents: read - copilot-requests: write - env: - OLD: ${{ needs.start.outputs.old }} - NEW: ${{ needs.start.outputs.new }} - OLD_COMMIT: ${{ needs.start.outputs.old_commit }} - NEW_COMMIT: ${{ needs.start.outputs.new_commit }} - outputs: - has_changes: ${{ steps.save.outputs.has_changes }} - complete: ${{ steps.agent.outputs.outcome == 'success' && steps.check.outcome == 'success' && steps.save.outputs.passed == 'true' }} - steps: - # No credentials in the checkout: the agent must not push. - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - - name: Record the directories - run: | - echo "OUTPUT_DIR=$RUNNER_TEMP/roll" >> "$GITHUB_ENV" - echo "SCRATCH_DIR=$RUNNER_TEMP/scratch" >> "$GITHUB_ENV" - echo "REPO_DIR=$GITHUB_WORKSPACE" >> "$GITHUB_ENV" - echo "PLAYWRIGHT_DIR=$(dirname "$GITHUB_WORKSPACE")/playwright" >> "$GITHUB_ENV" - mkdir -p "$RUNNER_TEMP/roll" "$RUNNER_TEMP/scratch" - - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 - with: - node-version: '24' - - name: Clone microsoft/playwright at the new version - run: | - git clone --filter=blob:none --no-checkout https://github.com/microsoft/playwright.git "$PLAYWRIGHT_DIR" - git -C "$PLAYWRIGHT_DIR" checkout -q "$NEW_COMMIT" - # Steps 1 to 3 of .claude/skills/dev/roll.md. - - name: Bump the pins and refresh the skill - run: | - npm pkg set "dependencies.playwright=$NEW" "dependencies.playwright-core=$NEW" "devDependencies.@playwright/test=$NEW" - npm install - node scripts/update.js - - name: Install the Copilot CLI - run: npm install -g @github/copilot - - name: Roll with Copilot CLI - id: agent - uses: microsoft/playwright/.github/actions/run-agent@d1518e3e3fca90352c4390814943cced496fd7e4 # main - with: - prompt: | - Roll Playwright in @playwright/cli from ${{ env.OLD }} to ${{ env.NEW }}. Follow .claude/skills/dev/roll.md, and read CLAUDE.md for how to test. - - The workflow already did steps 1 to 3 of the skill: package.json pins ${{ env.NEW }}, `npm install` updated package-lock.json, and `node scripts/update.js` refreshed skills/playwright-cli/. $PLAYWRIGHT_DIR is a clone of microsoft/playwright at ${{ env.NEW_COMMIT }}, the commit that the new version was built from; the old version was built from ${{ env.OLD_COMMIT }}. Start with the README update: `git diff -- skills/` shows what changed in the skill. - - When a test fails, find out why before you change anything. `git -C $PLAYWRIGHT_DIR log ${{ env.OLD_COMMIT }}..${{ env.NEW_COMMIT }} -- packages/playwright-core/src/tools tests/mcp` lists the upstream changes. When an upstream change is intended, update the test in this repo to the new behavior. When it is a regression, do not hide it: leave the test as it is, and describe the regression and the upstream commit that caused it. Change only README.md and files in skills/ and tests/: the workflow rejects changes to other files, except the pins in package.json and package-lock.json. - - The last step is for the workflow: do not create a branch, commit, push or run gh. A later job commits your changes and opens the pull request. Nobody watches the run: do not ask questions. - - Keep $OUTPUT_DIR/summary.md up to date as you work. When you finish, write the pull request description to $OUTPUT_DIR/pr-body.md: what changed in the skill and the README, the test results, and for each test that you changed, why, with the upstream commit. Then write $OUTPUT_DIR/roll.json last: {"passed": true} when `npm test` passes, {"passed": false} otherwise. - output-dir: ${{ env.OUTPUT_DIR }} - result-file: roll.json - artifact-name: roll - # The skill says to check that playwright-core rolled: the CLI runs it, and the tests use @playwright/test. - - name: Check the roll - id: check - continue-on-error: true - run: | - node -e ' - const pkg = require("./package.json"); - const pins = [pkg.dependencies.playwright, pkg.dependencies["playwright-core"], pkg.devDependencies["@playwright/test"], require("playwright-core/package.json").version]; - if (pins.some(pin => pin !== process.argv[1])) - throw new Error(`package.json pins and node_modules have ${pins.join(", ")}`); - ' "$NEW" - node playwright-cli.js --help > /dev/null - - name: Save the changes - id: save - run: | - git add -A - # Against the checked-out commit, in case the agent committed anyway. - git diff --cached --binary "$GITHUB_SHA" > "$OUTPUT_DIR/roll.patch" - if [[ -s "$OUTPUT_DIR/roll.patch" ]]; then - echo "has_changes=true" >> "$GITHUB_OUTPUT" - else - echo "has_changes=false" >> "$GITHUB_OUTPUT" - fi - passed=false - if [[ -f "$OUTPUT_DIR/roll.json" ]]; then - passed=$(node -p 'JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")).passed === true' "$OUTPUT_DIR/roll.json" || echo false) - fi - echo "passed=$passed" >> "$GITHUB_OUTPUT" - { - echo '```' - git diff --cached --stat "$GITHUB_SHA" - echo '```' - } >> "$GITHUB_STEP_SUMMARY" - - name: Upload the roll - if: always() - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 - with: - name: roll - path: ${{ env.OUTPUT_DIR }} - overwrite: true - if-no-files-found: warn - - publish: - name: Open the pull request - needs: [start, roll] - if: ${{ !cancelled() && needs.roll.outputs.has_changes == 'true' }} - runs-on: ubuntu-latest - outputs: - pr: ${{ steps.pr.outputs.url }} - env: - TITLE: "chore: roll Playwright to ${{ needs.start.outputs.new }}" - BRANCH: roll/${{ needs.start.outputs.new }} + timeout-minutes: 5 steps: - uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 id: app-token with: client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }} private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }} - # Pushes as the app, so CI runs on the pull request. - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - token: ${{ steps.app-token.outputs.token }} - - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 - with: - name: roll - path: ${{ runner.temp }}/roll - # Never execute code or scripts from the roll in the app job. - - name: Push the branch - run: | - git apply --index "$RUNNER_TEMP/roll/roll.patch" - # A roll changes only what a roll changes. --no-renames lists a renamed file under its old - # name too. - git diff --cached --name-only --no-renames > "$RUNNER_TEMP/changed.txt" - if grep -v -E '^(package\.json|package-lock\.json|README\.md)$|^(skills|tests)/' "$RUNNER_TEMP/changed.txt"; then - echo "The roll changes files outside package.json, package-lock.json, README.md, skills/ and tests/." >&2 - exit 1 - fi - if grep -E '(^|/)(AGENTS|CLAUDE)\.md$' "$RUNNER_TEMP/changed.txt"; then - echo "The roll changes agent instructions." >&2 - exit 1 - fi - if git diff --cached --summary | grep ' mode 120000 '; then - echo "The roll adds a symbolic link." >&2 - exit 1 - fi - git config user.name "microsoft-playwright-automation[bot]" - git config user.email "203992400+microsoft-playwright-automation[bot]@users.noreply.github.com" - git checkout -b "$BRANCH" - git commit -m "$TITLE" - git push --force origin "$BRANCH" - - name: Open the pull request - id: pr - uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 - env: - BASE: ${{ github.ref_name }} - COMPLETE: ${{ needs.roll.outputs.complete }} - OLD_COMMIT: ${{ needs.start.outputs.old_commit }} - NEW_COMMIT: ${{ needs.start.outputs.new_commit }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - with: - github-token: ${{ steps.app-token.outputs.token }} - script: | - const fs = require('fs'); - const { BASE, BRANCH, TITLE, COMPLETE, OLD_COMMIT, NEW_COMMIT, RUN_URL, RUNNER_TEMP } = process.env; - const file = `${RUNNER_TEMP}/roll/pr-body.md`; - const parts = [fs.existsSync(file) ? fs.readFileSync(file, 'utf8').trim() : 'The agent wrote no description.']; - if (COMPLETE !== 'true') - parts.push(`**The roll is not complete:** the tests, the agent or the checks failed. See the [roll run](${RUN_URL}).`); - if (OLD_COMMIT) - parts.push(`Upstream changes: https://github.com/microsoft/playwright/compare/${OLD_COMMIT}...${NEW_COMMIT}`); - parts.push(`Rolled by the [roll workflow](${RUN_URL}).`); - const body = parts.join('\n\n'); - const { owner, repo } = context.repo; - const { data: open } = await github.rest.pulls.list({ owner, repo, head: `${owner}:${BRANCH}`, state: 'open' }); - let pr = open[0]; - if (pr) - await github.rest.pulls.update({ owner, repo, pull_number: pr.number, body }); - else - ({ data: pr } = await github.rest.pulls.create({ owner, repo, head: BRANCH, base: BASE, title: TITLE, body, draft: COMPLETE !== 'true' })); - core.setOutput('url', pr.html_url); - await core.summary.addRaw(`Pull request: ${pr.html_url}\n`).write(); - - comment: - name: Comment on the issue - needs: [start, roll, publish] - if: ${{ !cancelled() && needs.start.result == 'success' && (github.event_name == 'issues' || inputs.issue != '') }} - runs-on: ubuntu-latest - permissions: - issues: write - steps: - - name: Post the result + owner: microsoft + repositories: devops.playwright.dev + permission-actions: write + - name: Start the bot env: - GH_TOKEN: ${{ github.token }} - GH_REPO: ${{ github.repository }} - ISSUE: ${{ github.event.issue.number || inputs.issue }} - NEW: ${{ needs.start.outputs.new }} - ROLL_RESULT: ${{ needs.roll.result }} - COMPLETE: ${{ needs.roll.outputs.complete }} - PR_URL: ${{ needs.publish.outputs.pr }} - START_COMMENT: ${{ needs.start.outputs.comment }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: | - if [[ "$ROLL_RESULT" == "skipped" ]]; then - text="main already pins Playwright $NEW. There is nothing to roll." - elif [[ -n "$PR_URL" && "$COMPLETE" == "true" ]]; then - text="I rolled Playwright to $NEW, and the tests pass: $PR_URL" - elif [[ -n "$PR_URL" ]]; then - text="I rolled Playwright to $NEW, but the roll is not complete: the tests, the agent or the checks failed. The pull request says what is left: $PR_URL" - else - text="I couldn't roll Playwright to $NEW. The agent run has the details." - fi - printf "Hi, I'm the Playwright bot. %s\n\nRolled by the Playwright bot - [agent run](%s)\n" "$text" "$RUN_URL" > "$RUNNER_TEMP/comment.md" - gh issue comment "$ISSUE" --body-file "$RUNNER_TEMP/comment.md" - if [[ -n "$START_COMMENT" ]]; then - gh api graphql -f query='mutation($id: ID!) { minimizeComment(input: { subjectId: $id, classifier: OUTDATED }) { clientMutationId } }' -f id="$START_COMMENT" \ - || echo "::warning::Could not hide the comment with the run link" - fi + GH_TOKEN: ${{ steps.app-token.outputs.token }} + TASK: >- + Roll Playwright to the latest alpha, playwright@next, as .claude/skills/dev/roll.md + says. When a test fails, find the upstream change that broke it. For an intended + change, update the test. For a regression, leave the test as it is, and describe the + regression and its upstream commit in the PR. + run: >- + gh workflow run task.yml --repo microsoft/devops.playwright.dev --ref main + -f repo="$GITHUB_REPOSITORY" -f task="$TASK" diff --git a/.github/workflows/triage.yml b/.github/workflows/triage.yml index 399d794..aa39150 100644 --- a/.github/workflows/triage.yml +++ b/.github/workflows/triage.yml @@ -1,163 +1,24 @@ -# Copilot triages and fixes an issue, or works on a task given by hand, without write credentials. -# The agent leaves the calls to make on GitHub in calls.json. A fresh job posts the run link on the -# issue first. After the agent, a fresh job checks the calls and makes them using the app: it opens -# the fix as a draft PR, so CI runs on it, and comments on the issue. The source of the CLI is in -# microsoft/playwright, so most fixes open there, and that repo's revision workflow addresses the -# review. The steps that are the same for every Playwright repo are actions in microsoft/playwright. -# The skill in .claude/skills/playwright-triage/ has what is specific to this repo. -name: Issue triage +# Hands the issue to the Playwright bot when a maintainer applies agent-triage, agent-implement or +# agent-task. Nothing else of the bot lives in this repo. Apply the label again to start another +# run. +name: "Agent: triage" on: - # Apply needs-triage, needs-triage-mac or needs-triage-win; re-apply it to trigger another pass. issues: types: [labeled] - workflow_dispatch: - inputs: - issue: - description: "Issue number to triage; leave empty to work on the task alone" - type: string - required: false - default: "" - platform: - description: "Platform to triage on" - type: choice - required: true - default: linux - options: - - linux - - mac - - windows - task: - description: "Task to direct the triage, such as what to focus on; the whole job when there is no issue" - type: string - required: false - default: "" permissions: {} jobs: - start: - name: Post the run link on the issue - # The triage job's condition, for runs that have an issue. + issue: + name: Hand the issue to the bot if: >- - github.repository == 'microsoft/playwright-cli' && - ((github.event_name == 'workflow_dispatch' && inputs.issue != '') || - contains(fromJSON('["needs-triage","needs-triage-mac","needs-triage-win"]'), github.event.label.name)) - runs-on: ubuntu-latest - permissions: - issues: write - outputs: - comment: ${{ steps.post.outputs.id }} - env: - ISSUE: ${{ github.event.issue.number || inputs.issue }} - PLATFORM_NAME: ${{ fromJSON('{"linux":"Linux","mac":"macOS","windows":"Windows","needs-triage":"Linux","needs-triage-mac":"macOS","needs-triage-win":"Windows"}')[inputs.platform || github.event.label.name] }} - GH_TOKEN: ${{ github.token }} - GH_REPO: ${{ github.repository }} - WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - steps: - - name: Post the run link - id: post - run: | - [[ "$ISSUE" =~ ^[1-9][0-9]*$ ]] || { echo "issue must be a positive integer" >&2; exit 1; } - body=$(printf "Hi, I'm the Playwright bot. I started to triage this issue on %s: [agent run](%s)\n" "$PLATFORM_NAME" "$WORKFLOW_URL") - id=$(gh api "repos/$GH_REPO/issues/$ISSUE/comments" -f body="$body" --jq .node_id) - echo "id=$id" >> "$GITHUB_OUTPUT" - - triage: - name: Triage ${{ (github.event.issue.number || inputs.issue) && format('issue {0}', github.event.issue.number || inputs.issue) || 'task' }} (${{ inputs.platform || github.event.label.name }}) - if: >- - github.repository == 'microsoft/playwright-cli' && - (github.event_name == 'workflow_dispatch' || - contains(fromJSON('["needs-triage","needs-triage-mac","needs-triage-win"]'), github.event.label.name)) - # Falls back to a runner name for unrelated labels, where the job is skipped anyway. - runs-on: ${{ fromJSON('{"linux":"playwright-x64-ubuntu24-64-core","mac":"macos-15-xlarge","windows":"playwright-x64-server2025-64-core"}')[inputs.platform || fromJSON('{"needs-triage":"linux","needs-triage-mac":"mac","needs-triage-win":"windows"}')[github.event.label.name]] || 'ubuntu-latest' }} - timeout-minutes: 240 - concurrency: - group: triage-${{ github.event.issue.number || inputs.issue || github.run_id }} - cancel-in-progress: false - defaults: - run: - shell: bash - env: - ISSUE: ${{ github.event.issue.number || inputs.issue }} - TRIAGE_PLATFORM: ${{ inputs.platform || fromJSON('{"needs-triage":"linux","needs-triage-mac":"mac","needs-triage-win":"windows"}')[github.event.label.name] }} - TASK: ${{ inputs.task }} - permissions: - copilot-requests: write - outputs: - base: ${{ steps.base.outputs.sha }} - upstream_base: ${{ steps.upstream-base.outputs.sha }} - setup: ${{ steps.setup.outcome }} - agent: ${{ steps.agent.outputs.outcome }} - has_upstream_fix: ${{ steps.upstream-fix.outputs.has-result }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: main - persist-credentials: false - - name: Record the base and the directories - id: base - run: | - if [[ -n "$ISSUE" ]]; then - [[ "$ISSUE" =~ ^[1-9][0-9]*$ ]] || { echo "issue must be a positive integer" >&2; exit 1; } - elif [[ -z "${TASK//[[:space:]]/}" ]]; then - echo "give an issue, a task or both" >&2; exit 1 - fi - echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" - echo "OUTPUT_DIR=${RUNNER_TEMP//\\//}/triage" >> "$GITHUB_ENV" - echo "SCRATCH_DIR=${RUNNER_TEMP//\\//}/scratch" >> "$GITHUB_ENV" - echo "REPO_DIR=${GITHUB_WORKSPACE//\\//}" >> "$GITHUB_ENV" - mkdir -p "${RUNNER_TEMP//\\//}/triage" "${RUNNER_TEMP//\\//}/scratch" - - name: Set up the runner - id: setup - uses: ./.github/actions/setup-triage-runner - - name: Record the base of microsoft/playwright - id: upstream-base - run: echo "sha=$(git -C "$PLAYWRIGHT_DIR" rev-parse HEAD)" >> "$GITHUB_OUTPUT" - - name: Write the prompt - id: prompt - run: | - delimiter="PROMPT_$RANDOM$RANDOM" - { - echo "prompt<<$delimiter" - echo "Follow .claude/skills/playwright-triage/SKILL.md. Read it, the guide and the CLAUDE.md" - echo "it points to in full before starting." - if [[ -n "$ISSUE" ]]; then - printf '\nIssue: https://github.com/%s/issues/%s\n' "$GITHUB_REPOSITORY" "$ISSUE" - fi - if [[ -n "$TASK" ]]; then - printf '\nTask: %s\n' "$TASK" - fi - echo "$delimiter" - } >> "$GITHUB_OUTPUT" - - name: Triage with Copilot CLI - id: agent - uses: microsoft/playwright/.github/actions/run-agent@d1518e3e3fca90352c4390814943cced496fd7e4 # main - with: - prompt: ${{ steps.prompt.outputs.prompt }} - output-dir: ${{ env.OUTPUT_DIR }} - result-file: calls.json - artifact-name: triage - github-tools: true - exec: ${{ env.CI_ENV_EXEC }} - # The publish job checks out microsoft/playwright only for a pull request there. - - name: Check the fix in microsoft/playwright - id: upstream-fix - run: | - if node -e 'const calls = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8")); process.exit(calls.some(c => c.call === "pull-request" && String(c.repo).toLowerCase() === "microsoft/playwright") ? 0 : 1)' "$OUTPUT_DIR/calls.json" 2> /dev/null; then - echo "has-result=true" >> "$GITHUB_OUTPUT" - echo "Fix proposed in microsoft/playwright" >> "$GITHUB_STEP_SUMMARY" - else - echo "has-result=false" >> "$GITHUB_OUTPUT" - fi - - publish: - name: Publish the results - needs: [start, triage] - # Runs after every triage run, also a failed one: without a comment from the agent, the comment - # says why the bot could not finish. - if: ${{ !cancelled() && needs.triage.result != 'skipped' }} + github.repository_owner == 'microsoft' && ( + github.event.label.name == 'agent-triage' || + github.event.label.name == 'agent-implement' || + github.event.label.name == 'agent-task') runs-on: ubuntu-latest + timeout-minutes: 5 steps: - uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 id: app-token @@ -165,70 +26,13 @@ jobs: client-id: ${{ vars.PLAYWRIGHT_APP_CLIENT_ID }} private-key: ${{ secrets.PLAYWRIGHT_PRIVATE_KEY }} owner: microsoft - repositories: playwright-cli,playwright - permission-contents: write - permission-issues: write - permission-pull-requests: write - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: ${{ needs.triage.outputs.base }} - token: ${{ steps.app-token.outputs.token }} - # A pull request in microsoft/playwright goes to its checkout in playwright/, at the commit that - # the agent worked on. - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - if: needs.triage.outputs.has_upstream_fix == 'true' - with: - repository: microsoft/playwright - ref: ${{ needs.triage.outputs.upstream_base }} - path: playwright - token: ${{ steps.app-token.outputs.token }} - - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 - # A failed triage run may not have uploaded anything. - continue-on-error: true - with: - name: triage - path: ${{ runner.temp }}/triage - # Without a comment from the agent on the issue, for example after a failed run, the issue gets - # one that says that the bot could not finish. - - name: Add a comment when the agent left none - if: github.event.issue.number || inputs.issue - env: - CALLS: ${{ runner.temp }}/triage/calls.json - ISSUE: ${{ github.event.issue.number || inputs.issue }} - SETUP: ${{ needs.triage.outputs.setup }} - AGENT: ${{ needs.triage.outputs.agent }} - FOOTER: Triaged on ${{ fromJSON('{"linux":"Linux","mac":"macOS","windows":"Windows","needs-triage":"Linux","needs-triage-mac":"macOS","needs-triage-win":"Windows"}')[inputs.platform || github.event.label.name] }} by the Playwright bot - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: | - mkdir -p "$(dirname "$CALLS")" - if jq -e --argjson issue "$ISSUE" 'any(.[]; .call == "comment" and .issue == $issue)' "$CALLS" > /dev/null 2>&1; then - exit 0 - fi - if [[ "$SETUP" == "failure" ]]; then - reason="setting up the runner failed" - elif [[ "$AGENT" == "failure" ]]; then - reason="the Copilot session failed or ran out of budget" - else - reason="the session ended without a comment" - fi - body=$(printf "Hi, I'm the Playwright bot. I couldn't finish triaging this issue: %s. The agent run has the details.\n\n%s - [agent run](%s)\n" "$reason" "$FOOTER" "$RUN_URL") - comment=$(jq -n --arg repo "$GITHUB_REPOSITORY" --argjson issue "$ISSUE" --arg body "$body" '{call: "comment", repo: $repo, issue: $issue, body: $body}') - # A calls.json that is not an array is replaced: its calls would be rejected anyway. - jq --argjson comment "$comment" '. + [$comment]' "$CALLS" > "$CALLS.tmp" 2> /dev/null \ - || jq -n --argjson comment "$comment" '[$comment]' > "$CALLS.tmp" - mv "$CALLS.tmp" "$CALLS" - # Never execute code or scripts from the artifact in the app job. The calls are data. - - name: Check and make the calls - id: publish - uses: microsoft/playwright/.github/actions/publish-calls@d1518e3e3fca90352c4390814943cced496fd7e4 # main - with: - token: ${{ steps.app-token.outputs.token }} - dir: ${{ runner.temp }}/triage - - name: Hide the comment with the run link - if: ${{ !cancelled() && steps.publish.outputs.comment == 'true' && needs.start.outputs.comment != '' }} + repositories: devops.playwright.dev + permission-actions: write + - name: Start the bot env: GH_TOKEN: ${{ steps.app-token.outputs.token }} - ID: ${{ needs.start.outputs.comment }} - run: | - gh api graphql -f query='mutation($id: ID!) { minimizeComment(input: { subjectId: $id, classifier: OUTDATED }) { clientMutationId } }' -f id="$ID" \ - || echo "::warning::Could not hide the comment with the run link" + NUMBER: ${{ github.event.issue.number }} + LABEL: ${{ github.event.label.name }} + run: >- + gh workflow run issue.yml --repo microsoft/devops.playwright.dev --ref main + -f repo="$GITHUB_REPOSITORY" -f issue="$NUMBER" -f label="$LABEL" diff --git a/CLAUDE.md b/AGENTS.md similarity index 61% rename from CLAUDE.md rename to AGENTS.md index 889fc9f..0930d71 100644 --- a/CLAUDE.md +++ b/AGENTS.md @@ -54,12 +54,14 @@ See [.claude/skills/dev/roll.md](.claude/skills/dev/roll.md) and [.claude/skills ## Bots -Labels start GitHub Actions workflows that run the Copilot CLI and open pull requests as the -`microsoft-playwright-automation` app: +The Playwright bot does the work below. It opens pull requests as the +`microsoft-playwright-automation` app. [triage.yml](.github/workflows/triage.yml) and +[review.yml](.github/workflows/review.yml) hand it the labels: -- `needs-triage`, `needs-triage-mac`, `needs-triage-win` on an issue: [triage.yml](.github/workflows/triage.yml) reproduces and fixes it, with the skill in [.claude/skills/playwright-triage/](.claude/skills/playwright-triage/SKILL.md). A fix in `microsoft/playwright` opens as a draft PR there. -- `needs-revision`, `needs-revision-mac`, `needs-revision-win` on a PR that triage opened here: [revision.yml](.github/workflows/revision.yml) addresses the maintainers' review comments. -- `needs-roll` on an issue: [roll.yml](.github/workflows/roll.yml) rolls Playwright to `playwright@next`, as the roll skill says. -- `needs-release` on an issue: [release.yml](.github/workflows/release.yml) opens the `chore: mark v` PR with the release notes as its description, as the release skill says. Copy the notes into the release draft. +- `agent-triage` on an issue: finds the cause and fixes the issue if it can, with the skill in [.claude/skills/playwright-triage/](.claude/skills/playwright-triage/SKILL.md). A fix in `microsoft/playwright` opens as a draft PR there. +- `agent-implement` on an issue: implements what the issue and the maintainers' comments say. +- `agent-task` on an issue: does what the issue asks, with the skills of this repo. +- `agent-review` on a PR: revises the PR, as the maintainers' review comments say. +- [roll.yml](.github/workflows/roll.yml) and [release.yml](.github/workflows/release.yml), from the Actions tab: they hand a task to the bot. Roll Playwright to `playwright@next`, as the roll skill says. Or open the `chore: mark v` PR with the release notes as its description, as the release skill says. Copy the notes into the release draft. -Apply a label again to start another run. [bots.md](https://github.com/microsoft/playwright/blob/main/.github/workflows/bots.md) in `microsoft/playwright` describes triage and revision. +Apply a label again to start another run.