diff --git a/.github/actions/plugin-check-report/README.md b/.github/actions/plugin-check-report/README.md new file mode 100644 index 0000000..afd715d --- /dev/null +++ b/.github/actions/plugin-check-report/README.md @@ -0,0 +1,363 @@ +# plugin-check-report + +Compares a [WordPress.org Plugin Check](https://github.com/WordPress/plugin-check) results file with +a baseline: the number of errors and warnings per rule, kept in a file the repository commits, +`.plugin-check-baseline.json`. It writes a short body for the pull request comment, and a new +baseline to copy into the repository. + +Most repositories do not use this action directly. They call the reusable workflow +[`.github/workflows/plugin-check.yml`](../../workflows/plugin-check.yml), which builds the plugin, +runs [wordpress/plugin-check-action](https://github.com/WordPress/plugin-check-action), runs this +action and posts the comment. A caller to copy is in +[`templates/workflows/plugin-check.yml`](../../../templates/workflows/plugin-check.yml): + +```yaml +jobs: + plugin-check: + permissions: + contents: read + issues: write + pull-requests: write + uses: stellarwp/plugin-toolbox/.github/workflows/plugin-check.yml@main + with: + plugin-slug: my-plugin +``` + +## What the repository needs + +- A `.puprc` whose `build` makes the plugin as it ships, and whose `zip_name` equals `plugin-slug`. + The zip's top folder is named after `zip_name`, and Plugin Check takes the slug from the folder + name. +- Nothing for pup itself. The workflow downloads it in the shared [zip-name](../zip-name/) and [release-zip](../release-zip/) actions, so the + repository needs no copy of `pup.phar` and no `pup` Composer script. Set the `pup-version` input + to the release the repository's own release build uses, so the check sees the same files. +- Ignore rules that keep dev-only files out of the zip: a `.distignore`, `export-ignore` lines in + `.gitattributes` (pup reads both), or pup's default ignore list. +- A baseline file, `.plugin-check-baseline.json` at the root. Without it the check still works, but + slower: see [The baseline](#the-baseline). Keep it out of the zip: Plugin Check flags hidden files, + and pup's default ignore list already drops them. + +The JavaScript toolchain comes from files the repository already has: `.nvmrc` sets up Node, +`package-lock.json` turns on the npm cache, and `bun.lock` or `bun.lockb` sets up bun. + +## How the plugin is built + +By default, with pup, the same QA build the release zip makes, from the root of the checkout (the +`source/` folder of the job). Two shared actions do it: [zip-name](../zip-name/) reads the version +and names the zip (`production: no`, `prerelease: dev`), and [release-zip](../release-zip/) runs +`pup build` and `pup package ` with `check: no` and `i18n: no`, since this workflow needs +neither. The workflow then unpacks the zip into `$PLUGIN_CHECK_BUILD_DIR`. + +One pup release (the `pup-version` input) reads the version, builds, packages and names the zip, so +the name looked for is always the name the package step made. + +So `.puprc` and the ignore files decide what is checked. If the release needs a step the check should +see too, such as making the POT file, put it in `.puprc`'s `build`, not in the workflow. + +### `build-command`, for a repository without pup + +`build-command` replaces the pup steps. Prefer `.puprc` when the repository has one: a custom +command is a second place that defines the build, and the two drift apart. + +The contract: + +- It runs with bash, from the root of the checkout, with `-e` and `pipefail`. +- PHP, Node or bun are set up before it, as for pup. The `composer-token` and `package-token` + secrets are in its environment. +- It must leave the plugin, as it ships, in `$PLUGIN_CHECK_BUILD_DIR//`, with the main + plugin file inside: `.php`, or `main-file` when the name differs. + +After the build, pup or not, the workflow checks that `$PLUGIN_CHECK_BUILD_DIR//` +exists, and fails naming what it found instead. A folder, not a zip, because the Plugin Check action +takes a folder and its slug from the folder name; asking for a zip would only add an unzip and a +guess at the top folder. + +```yaml + with: + plugin-slug: my-plugin + build-command: | + composer install --no-dev + npm ci && npm run build + mkdir -p "$PLUGIN_CHECK_BUILD_DIR/my-plugin" + rsync -a --exclude-from=.distignore ./ "$PLUGIN_CHECK_BUILD_DIR/my-plugin/" +``` + +## The baseline + +The baseline is `.plugin-check-baseline.json`, a file the repository commits: + +```json +{ + "errors": { + "WordPress.Security.EscapeOutput.OutputNotEscaped": 12 + }, + "warnings": { + "WordPress.DB.DirectDatabaseQuery.DirectQuery": 3 + } +} +``` + +It holds a count for each rule code, and no file names or line numbers, so it only changes when a +rule gains or loses an error somewhere, not when code moves. Only a rule with more errors than the +file allows fails the check. + +A pull request is compared with the file in its own checkout, the pull request merged into its +target. So a pull request that fixes errors lowers the file in the same diff, and one that adds an +error on purpose raises it in the same diff, where a reviewer sees it. The comment says "This pull +request changes the baseline file" when it does. The tradeoff: a pull request can raise a count to +hide a new error. The change is in the diff, in a file with an obvious name. To require a reviewer +for it, add a line such as `/.plugin-check-baseline.json @my-org/leads` to CODEOWNERS. + +A push is compared with the file at that commit, so a merge that adds errors fails on the target +branch, where the whole team sees it. + +### Making and updating the file + +Every run uploads the `plugin-check-report` artifact, with a `baseline.json` made from that run's +results. To start the file, or to lower it after errors were fixed, download that artifact and copy +the file. The job summary shows the command with the run number filled in: + +```bash +gh run download --repo my-org/my-plugin --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json .plugin-check-baseline.json +``` + +Or locally, from the plugin's root, with the results of a local Plugin Check run: + +```bash +node .github/actions/plugin-check-report/plugin-check-report.js update results.txt +``` + +The workflow never commits the file itself. That would need a write token on every pull request, +could not push to a fork, and would change the baseline without the person who owns the change. + +When error counts go down and the file is not updated, the check passes with a notice and the +comment asks for the update. Slack in the file lets a later pull request add errors back, so it is +worth lowering the file when errors are fixed. If two pull requests change the same count, git +reports a conflict in the file: merge the target branch, then make the file again from the new run +instead of merging the numbers by hand. + +### With no baseline file + +The check is slower, not weaker: + +- **A pull request** (in `baseline` mode) builds and checks the target branch's commit too, in a + second job that runs in parallel with the first, and compares with those results. The comment's + column is then "Target branch", and a line says there is no baseline file. If the target branch + cannot be built or checked, the check fails with a message that says so: it never passes without a + baseline. The cost is a second build and Plugin Check run, so the job minutes double, though the + wait does not. +- **A push** has no target to compare with. The counts are reported and the check passes with a + warning that says how to start the file. +- **`strict` mode** needs no baseline, so it builds nothing extra and says nothing. + +Commit the file to stop the second build. + +### A file that is not valid + +A file that is not JSON (a merge conflict left markers in it, say) or not the shape above fails the +check with an error annotation that names the file and says what is wrong. The comment then says +Plugin Check could not run. Replace it with the `baseline.json` of the run's artifact. + +## Modes + +- `baseline` (default): fail only when a rule has more errors than the baseline. More warnings, or + fewer of either, only add a notice. The totals can go down while one rule goes up, so the check + looks at each rule. +- `strict`: fail on any error. The comment still compares with the baseline. Switch to it once a + plugin reaches zero errors; a new plugin with zero errors should start there. + +## Excluding checks + +Some Plugin Check checks do not fit a plugin. Name them in a `.plugin-check.json` at the root of +the repository: + +```json +{ + "exclude-checks": ["late_escaping", "i18n_usage"] +} +``` + +One check name per entry, with no spaces or commas. Other keys are ignored, so the file can hold +more settings later. A missing file, a missing key or an empty list excludes nothing. A file that is +not valid JSON, or whose entries are not check names, fails the run in seconds, before the build starts, +with a message that says what to fix. + +The workflow's `exclude-checks` input does the same, one check per line. **When the input names any +check, it replaces the file** (the file is not even read). When it is empty, the file is used. + +The same list is available for a local run. This prints the flag `wp plugin check` takes, or +nothing when no check is excluded: + +```bash +node .github/actions/plugin-check-report/plugin-check-report.js exclude-checks +# --exclude-checks=late_escaping,i18n_usage + +npx wp-env run cli wp plugin check my-plugin --format=json \ + $(node .github/actions/plugin-check-report/plugin-check-report.js exclude-checks) > results.txt +``` + +It reads `.plugin-check.json` in the current folder, or the file named after it. The environment +variable `EXCLUDE_CHECKS` plays the part of the workflow input. + +## The comment + +One comment per pull request, replaced on every run, found by a hidden first line: +``. It says "in progress" while the check runs, then shows the +result, the totals, up to five rules that went up, and links to the run and the artifact. It never +grows past 15 lines. Pull requests from forks get no comment: their token cannot write. + +The workflow writes it with the [pr-comment](../pr-comment/) action. The comment is found by its +first line, not by its author. + +One line under the table says where the baseline came from when that matters: that the baseline is +the target branch's results because there is no baseline file, or that the pull request changes the +baseline file. When an error count went down, the "Rules that went down" line asks for the file to be +updated. The commands are in the job summary, not in the comment. + +The action builds the body. Under the status line it puts a "Check Finished" badge with the time in +Eastern Time (`06:08 PM EDT` in summer, `EST` in winter), made by `finishBadge()` in +`plugin-check-report.js`, so no caller passes one. When the build or the check broke and there was +nothing to compare, the action writes a short body instead, "Plugin Check could not run" with the +same badge and a link to the run, so "in progress" never stays on the pull request. A body the +compare wrote is never replaced. + +## The check name + +A job in a called workflow reports as ` / `. With the template's job id, +the check to require in branch protection is `plugin-check / Plugin Check`. + +## Inputs of the workflow + +| Input | Default | What it does | +|---|---|---| +| `plugin-slug` | required | The plugin's folder name. Equals `zip_name` in `.puprc` | +| `main-file` | `.php` | The main plugin file, checked after the build | +| `build-command` | empty: pup | A bash command that builds the plugin instead of pup. See [`build-command`](#build-command-for-a-repository-without-pup) | +| `pup-version` | `2.0.0` | The pup release that builds the plugin, or `latest`. Use the release the repository's release build uses | +| `runs-on` | `ubuntu-latest` | Runner of the job that builds and checks, of runner version 2.336.0 or newer (the job uses `$/`) | +| `php-version` | `7.4` | PHP that builds `vendor/`. Use the oldest the plugin supports | +| `exclude-checks` | empty | Checks to skip, one per line. Replaces `.plugin-check.json` when it names any. See [Excluding checks](#excluding-checks) | +| `cache-paths` | empty | Build caches kept between runs, one path per line, relative to the repository | +| `mode` | `baseline` | `baseline` or `strict`. See [Modes](#modes) | +| `submodules` | `false` | Passed to `actions/checkout` | + +| Secret | Becomes, in the build step only | +|---|---| +| `composer-token` | `COMPOSER_AUTH` with a `github-oauth` entry | +| `package-token` | `NODE_AUTH_TOKEN`, `BUN_AUTH_TOKEN` and `GITHUB_TOKEN` | + +The secrets never reach the job's environment, so the third-party Plugin Check step that runs later +in the same job does not see them there. + +### Permissions the caller grants + +```yaml +permissions: + contents: read # check out the repository, and read the baseline file + issues: write # the sticky comment + pull-requests: write # the sticky comment +``` + +A called workflow can only lower these. If the caller grants less than a job asks for, GitHub does +not start the run. The job that runs the Plugin Check action lowers itself to `contents: read`. +A caller from an earlier version that still grants `actions: read` works: a caller may give more +than a job asks for. + +## Inputs of the action + +| Input | Required | Default | What it does | +|---|---|---|---| +| `results` | yes | | The results file. A missing file fails the compare | +| `comment-file` | no | none | Where to write the comment body | +| `mode` | no | `baseline` | `baseline` or `strict` | +| `comment-marker` | no | `` | First line of the comment body | +| `baseline-file` | no | `.plugin-check-baseline.json` | The committed baseline. A relative path is read from the working directory | +| `base-results` | no | none | Results of the target branch, used as the baseline when there is no baseline file. Not used in `strict` mode | +| `base-required` | no | `false` | `true` when the caller checked the target branch: a missing `base-results` file then fails the compare | +| `baseline-changed` | no | `false` | `true` when the pull request changes the baseline file; the comment says so | + +| Output | What it is | +|---|---| +| `artifact-url` | The `plugin-check-report` artifact of this run | + +## Running it locally + +You need Node 20 or newer. For a pull request's baseline, check out its branch: the file is in it. + +1. Build the plugin and run Plugin Check in a local WordPress, for example with wp-env. The excluded + checks come from the same `.plugin-check.json` as in CI: + + ```bash + npx wp-env start + npx wp-env run cli wp plugin check my-plugin --format=json \ + $(node .github/actions/plugin-check-report/plugin-check-report.js exclude-checks) > results.txt + ``` + + Keep only the command's output in the file. A notice from wp-env at the top fails the compare + with "not Plugin Check output", on purpose. + +2. Compare, from the plugin's root, where `.plugin-check-baseline.json` is: + + ```bash + node .github/actions/plugin-check-report/plugin-check-report.js compare --comment-file comment.md results.txt + ``` + + Exit 0 means the check passes. `comment.md` holds what the pull request would show. Add + `PLUGIN_CHECK_MODE=strict` to see the strict result, or `PLUGIN_CHECK_BASELINE=path` to use + another file. With no baseline file, the counts are shown and nothing is compared. + +3. To make or update the file from these results: + + ```bash + node .github/actions/plugin-check-report/plugin-check-report.js update results.txt + ``` + + `update` is short for `compare --update`. It writes `.plugin-check-baseline.json` in the current + folder, with the same text as the `baseline.json` of the CI artifact. + +## Differences from the earlier bash scripts + +The first version of this action was two bash scripts. The JavaScript that replaced them gives the +same results on every input Plugin Check can produce. These differences are on purpose, and only +show with a hand-edited baseline or an odd results file, except the first two: + +- The mode `ratchet` is now called `baseline`. The old word is refused, and the message says what + to use. +- The `workflow-file`, `baseline-wait`, `token` and `baseline-url` pieces are gone, and so is the + "older run" status and the "Target branch" / "Previous run" column names. The baseline is a + committed file now, not the artifact of an earlier run, so nothing is looked up or waited for. + The `fetch` command is gone with them. +- The texts for a missing baseline, for error counts that went down and for an invalid baseline + file are new, and the job summary has the commands to update the file. These are the only + expected files under `fixtures/expected/` that no longer come from the bash script. +- The `badge-url` input is gone. The action makes the "Check Finished" badge itself, and also writes + the "could not run" body that a shell step in the workflow wrote before. Both look as they did. +- A rule with an empty code is counted like any other. Bash misread its row and showed wrong totals. +- A count in the baseline that is not a whole number (such as `1.5`) fails the compare with exit + status 1. Bash printed an arithmetic error and passed with zero totals. A count written as `1.0` + or `1e2` is read as the whole number it is. + +- A baseline count above 9007199254740991 (2^53 - 1) fails the compare with exit status 5 and says + so. JavaScript cannot hold a larger number exactly, so a comment would show another number. Bash + compared them exactly. + +## How it is tested + +The work is in `plugin-check-report.js`: small functions that take plain values, so a test calls +them directly. `tests/actions/plugin-check-report/` has four files, run with `node --test`: + +- `compare.test.js` runs the compare in-process on the sample results files in `fixtures/`, and + checks the comment, the job summary and the annotations against the exact files in + `fixtures/expected/`. Most of those files were written by the original bash script, so they pin + that the JavaScript prints what it printed; the ones listed under "Differences" were rewritten for + the committed baseline. It also covers the missing file, the target branch as a baseline, the + invalid file and the command line. +- `action.test.js` calls the step functions the way `actions/github-script` does. +- `badge.test.js` checks the badge URL against what the old shell step printed, from the real + `date` command, for both sides of both daylight saving changes, midnight, noon and the end of a + year. It also covers the "could not run" body. +- `exclude.test.js` covers the config file and the input, every way the file can be wrong, and the + command line. + +No `jq` or `bash` is needed. diff --git a/.github/actions/plugin-check-report/action.yml b/.github/actions/plugin-check-report/action.yml new file mode 100644 index 0000000..64ac08f --- /dev/null +++ b/.github/actions/plugin-check-report/action.yml @@ -0,0 +1,150 @@ +# Compares a WordPress.org Plugin Check results file with a baseline: the counts per rule in a file +# the repository commits, `.plugin-check-baseline.json`. Only a rule with more errors than the +# baseline fails (baseline mode), or any error at all (strict mode). It also saves this run's counts +# as a new baseline, to copy into the repository, and writes the short body for the pull request +# comment. +# +# It does not run Plugin Check and it does not post the comment. The reusable workflow +# .github/workflows/plugin-check.yml does both, so this action stays usable with a results file +# from anywhere. It does write the comment's body, with the "Check Finished" badge, and a body that +# says Plugin Check could not run when there was nothing to compare. +# +# The work is in one JavaScript file next to this file, plugin-check-report.js. Each step below is +# an actions/github-script step that requires it by path, so there is nothing to build: the file in +# the repository is the file that runs. The same file runs from a command line; see the README. +name: Plugin Check report +description: Compares Plugin Check results with the repository's baseline file, and saves a new baseline + +inputs: + results: + description: >- + Path of the results file wordpress/plugin-check-action writes, `plugin-check-results.txt`. A + missing file fails the compare step: that is how a broken build or a broken check shows up. + required: true + comment-file: + description: >- + Where to write the short Markdown body for the pull request comment. Empty writes none. The + file is only written when the compare runs, so a caller can tell "no result" from a result. + required: false + default: '' + mode: + description: >- + `baseline`: fail only when a rule has more errors than the baseline. `strict`: fail on any + error, baseline or not. + required: false + default: 'baseline' + comment-marker: + description: >- + Hidden first line of the comment body, which the caller searches for to find its own + comment. Empty uses ``. + required: false + default: '' + baseline-file: + description: >- + Path of the baseline file the repository commits: `{"errors": {code: n}, "warnings": {code: n}}`. + A relative path is read from the working directory. + required: false + default: '.plugin-check-baseline.json' + base-results: + description: >- + Path of the Plugin Check results of the pull request's target branch. Used as the baseline when + there is no baseline file, so a pull request is still compared with something. Empty when + nothing was checked on the target branch. Not used in `strict` mode. + required: false + default: '' + base-required: + description: >- + `true` when the caller checked the target branch because there is no baseline file. A missing + `base-results` file then fails the compare, so a target branch that could not be checked never + passes as "no baseline". + required: false + default: 'false' + baseline-changed: + description: >- + `true` when the pull request changes the baseline file. The comment then says so, because the + pull request is compared with its own change and a reviewer should look at the file's diff. + required: false + default: 'false' + +outputs: + artifact-url: + description: 'Link to the plugin-check-report artifact this run uploaded. Empty when there was nothing to upload.' + value: ${{ steps.artifact.outputs.artifact-url }} + +runs: + using: 'composite' + steps: + # One folder per run under RUNNER_TEMP, cleared first, so a second use of the action in the same + # job never reads the first one's files. + - name: Set file paths + id: paths + uses: actions/github-script@v9 + env: + # github-script's require() resolves against the workspace of the repository being built, + # not this action's directory, so the path has to be passed in. + ACTION_PATH: ${{ github.action_path }} + INPUT_COMMENT_MARKER: ${{ inputs.comment-marker }} + with: + script: | + const { runPaths } = require(`${process.env.ACTION_PATH}/plugin-check-report.js`) + process.exitCode = runPaths({ core, env: process.env }) + + # A baseline made from this run, uploaded even when the compare below fails. It is the easy way + # to update the committed file: download the artifact and copy its baseline.json. + - name: Write a new baseline + uses: actions/github-script@v9 + env: + ACTION_PATH: ${{ github.action_path }} + RESULTS: ${{ inputs.results }} + REPORT_DIR: ${{ steps.paths.outputs.report }} + with: + script: | + const { runUpdate } = require(`${process.env.ACTION_PATH}/plugin-check-report.js`) + process.exitCode = runUpdate({ core, env: process.env }) + + # The comment links to this artifact: the results, and the new baseline.json. + - name: Upload the report + id: artifact + uses: actions/upload-artifact@v7 + with: + name: plugin-check-report + path: ${{ steps.paths.outputs.report }} + if-no-files-found: ignore + + # Fails the step when the mode says so. Also writes the comment body and the same text to the + # job summary. The step ends with an exit status, not an extra error annotation: the messages + # above it already say what is wrong. + - name: Compare with the baseline + uses: actions/github-script@v9 + env: + ACTION_PATH: ${{ github.action_path }} + RESULTS: ${{ inputs.results }} + COMMENT_FILE: ${{ inputs.comment-file }} + PLUGIN_CHECK_MODE: ${{ inputs.mode }} + PLUGIN_CHECK_BASELINE: ${{ inputs.baseline-file }} + PLUGIN_CHECK_BASE_RESULTS: ${{ inputs.base-results }} + PLUGIN_CHECK_BASE_REQUIRED: ${{ inputs.base-required }} + PLUGIN_CHECK_BASELINE_CHANGED: ${{ inputs.baseline-changed }} + COMMENT_MARKER: ${{ steps.paths.outputs.marker }} + ARTIFACT_URL: ${{ steps.artifact.outputs.artifact-url }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + with: + script: | + const { runCompare } = require(`${process.env.ACTION_PATH}/plugin-check-report.js`) + process.exitCode = runCompare({ core, env: process.env }) + + # When the build or the check broke, the compare wrote no body. Say so, instead of leaving "in + # progress" behind. Runs after a failed step too, but not when the run was cancelled. A body the + # compare wrote is left as it is. + - name: Write the comment if the compare could not run + if: ${{ !cancelled() && inputs.comment-file != '' }} + uses: actions/github-script@v9 + env: + ACTION_PATH: ${{ github.action_path }} + COMMENT_FILE: ${{ inputs.comment-file }} + COMMENT_MARKER: ${{ inputs.comment-marker || format('', github.repository) }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + with: + script: | + const { runCouldNotRun } = require(`${process.env.ACTION_PATH}/plugin-check-report.js`) + process.exitCode = runCouldNotRun({ core, env: process.env }) diff --git a/.github/actions/plugin-check-report/exclude-checks/action.yml b/.github/actions/plugin-check-report/exclude-checks/action.yml new file mode 100644 index 0000000..6986686 --- /dev/null +++ b/.github/actions/plugin-check-report/exclude-checks/action.yml @@ -0,0 +1,46 @@ +# Works out which Plugin Check checks to skip: the workflow's `exclude-checks` input when it names +# any, else the `exclude-checks` list in the repository's .plugin-check.json. The result is in the +# format wordpress/plugin-check-action takes for its own `exclude-checks` input: one check per +# line. A bad file fails the step with a message that says what to fix. +# +# The work is in plugin-check-report.js, one folder up, which also prints the same list as the +# `--exclude-checks=a,b` flag of `wp plugin check` for a local run. See the README of +# plugin-check-report. +name: Plugin Check excluded checks +description: Lists the checks to skip, from the exclude-checks input or the repository's .plugin-check.json + +inputs: + exclude-checks: + description: >- + Checks to skip, one per line. When it names any check, it replaces the file: the file is not + even read. + required: false + default: '' + config-file: + description: >- + Path of the repository's config file, relative to the workspace. A missing file means no + checks are skipped. + required: false + default: '.plugin-check.json' + +outputs: + checks: + description: 'The checks to skip, one per line. Empty when there are none.' + value: ${{ steps.list.outputs.checks }} + +runs: + using: 'composite' + steps: + - name: List the excluded checks + id: list + uses: actions/github-script@v9 + env: + # github-script's require() resolves against the workspace of the repository being built, + # not this action's directory, so the path has to be passed in. + ACTION_PATH: ${{ github.action_path }} + EXCLUDE_CHECKS: ${{ inputs.exclude-checks }} + CONFIG_FILE: ${{ inputs.config-file }} + with: + script: | + const { runExcludeChecks } = require(`${process.env.ACTION_PATH}/../plugin-check-report.js`) + process.exitCode = runExcludeChecks({ core, env: process.env }) diff --git a/.github/actions/plugin-check-report/plugin-check-report.js b/.github/actions/plugin-check-report/plugin-check-report.js new file mode 100644 index 0000000..3460a5b --- /dev/null +++ b/.github/actions/plugin-check-report/plugin-check-report.js @@ -0,0 +1,1466 @@ +'use strict' + +/** + * Compares WordPress.org Plugin Check results with a baseline file. + * + * The baseline is the count of errors and warnings per rule code, kept in a file the repository + * commits, `.plugin-check-baseline.json`. A pull request is compared with the file in its own + * checkout, so a change to the baseline shows in the diff. + * + * One job: + * compare Counts the results, compares them with the baseline file, writes the pull request + * comment and the job summary, and says whether the check passes. With `--update` it only + * writes the baseline file for this run. + * + * The small functions are pure, so a test can call them with plain values. The same code runs in + * three places: the action's steps (`runPaths`, `runUpdate`, `runCompare`, through + * actions/github-script), a command line (`node plugin-check-report.js compare ...`), and the tests. + * + * Every function that decides an exit status returns it: 0 passes, 1 is a failed check or a missing + * file, another number is a file that could not be read as it should be. A step sets + * `process.exitCode` from it, so the step fails without github-script adding a second error. + */ + +const fs = require('node:fs') +const path = require('node:path') + +/** Title of every annotation. */ +const TITLE = 'Plugin Check' + +/** The baseline file a repository commits, at its root. */ +const BASELINE_FILE = '.plugin-check-baseline.json' + +/** The pull request comment must stay short. Everything else is behind the links. */ +const MAX_COMMENT_LINES = 15 +const MAX_RULES_LISTED = 5 + +/** The first unexpected line of a results file is shown cut to this many bytes. */ +const MAX_UNEXPECTED_BYTES = 200 + +const MODES = ['baseline', 'strict'] +const DEFAULT_MODE = 'baseline' + +/** Modes that were renamed, so the message can say what to use now. */ +const RENAMED_MODES = { ratchet: 'baseline' } + +/** A line of Plugin Check output: a file name, a JSON array of its results, or the clean message. */ +const KNOWN_LINE = /^(FILE: |\[|Success:)/ +const OUTPUT_LINE = /^(FILE: |Success:)/ +/** Blank as `grep '[[:space:]]'` reads it: ASCII white space only. */ +const BLANK_LINE = /^[ \t\n\v\f\r]*$/ + +/** + * A failure the person running the check has to fix: the message is shown, the step fails with the + * exit code, and no stack trace helps. + */ +class CheckFailure extends Error { + /** + * @param {string} message What went wrong, in plain words. + * @param {number} exitCode The exit status for the step. + * @param {boolean} annotate Show the message as an error annotation, not only in the log. + */ + constructor(message, exitCode = 1, annotate = false) { + super(message) + this.name = 'CheckFailure' + this.exitCode = exitCode + this.annotate = annotate + } +} + +/** + * Orders two strings by Unicode code point, as `jq` sorts them. JavaScript's default order compares + * UTF-16 units, which puts an emoji before U+FFFF; UTF-8 bytes sort by code point. + * + * @param {string} a First string. + * @param {string} b Second string. + * + * @returns {number} Negative, zero or positive. + */ +function compareCodePoints(a, b) { + return Buffer.compare(Buffer.from(a), Buffer.from(b)) +} + +/** + * Cuts a string to a number of UTF-8 bytes. A character cut in the middle shows as U+FFFD. + * + * @param {string} text The text. + * @param {number} bytes The most bytes to keep. + * + * @returns {string} The text, or its first bytes. + */ +function cutBytes(text, bytes) { + const buffer = Buffer.from(text) + + return buffer.length <= bytes ? text : buffer.subarray(0, bytes).toString('utf8') +} + +/** + * Splits a run of JSON values written one after another, as `jq` reads its input: whitespace + * between values, any number of them. + * + * @param {string} text The text. + * + * @throws {CheckFailure} When it is not JSON. + * + * @returns {Array} The values, in order. + */ +function readJsonValues(text) { + const values = [] + let position = 0 + + while (true) { + while (position < text.length && /\s/.test(text[position])) { + position++ + } + if (position >= text.length) { + return values + } + + const end = endOfJsonValue(text, position) + try { + values.push(JSON.parse(text.slice(position, end))) + } catch (error) { + throw new CheckFailure(`jq: parse error: ${error.message}`, 2) + } + position = end + } +} + +/** + * Finds where the JSON value that starts at a position ends. + * + * @param {string} text The text. + * @param {number} start Index of the value's first character. + * + * @throws {CheckFailure} When the value never ends. + * + * @returns {number} Index just after the value. + */ +function endOfJsonValue(text, start) { + const first = text[start] + + if (first === '[' || first === '{') { + let depth = 0 + let inString = false + + for (let index = start; index < text.length; index++) { + const char = text[index] + + if (inString) { + if (char === '\\') { + index++ + } else if (char === '"') { + inString = false + } + } else if (char === '"') { + inString = true + } else if (char === '[' || char === '{') { + depth++ + } else if ((char === ']' || char === '}') && --depth === 0) { + return index + 1 + } + } + + throw new CheckFailure('jq: parse error: Unfinished JSON term at EOF', 2) + } + + const scalar = /^("(?:[^"\\]|\\.)*"|[^\s[\]{}",:]+)/.exec(text.slice(start)) + if (!scalar) { + throw new CheckFailure('jq: parse error: Invalid JSON', 2) + } + + return start + scalar[0].length +} + +/** + * Why a results file cannot be Plugin Check's output, or null when it can. + * + * Anything else in the file (an HTML database error, a PHP fatal) means the check itself broke. + * Without this it would count as "no results" and pass. A clean run prints "Success: ..." and a run + * with results prints "FILE: ..." lines; an empty file means the command failed with its error on + * stderr, which the results file does not hold. + * + * @param {string} text The results file. + * + * @returns {string|null} The message, or null. + */ +function checkResultsText(text) { + const lines = text.split('\n') + const unexpected = lines.find((line) => !KNOWN_LINE.test(line) && !BLANK_LINE.test(line)) + + if (unexpected !== undefined) { + return `The results file is not Plugin Check output, so the check did not run. First unexpected line: ${cutBytes(unexpected, MAX_UNEXPECTED_BYTES)}` + } + + if (!lines.some((line) => OUTPUT_LINE.test(line))) { + return 'The results file has no Plugin Check output, so the check did not run.' + } + + return null +} + +/** + * The results of a file: every line that starts with `[` holds a JSON array of one file's results. + * + * @param {string} text The results file. + * + * @throws {CheckFailure} When a line is not JSON, or does not hold what Plugin Check prints. + * + * @returns {object[]} All results, in file order. + */ +function parseResults(text) { + const results = [] + + for (const line of text.split('\n')) { + if (!line.startsWith('[')) { + continue + } + + let values + try { + values = readJsonValues(line) + } catch (error) { + throw new CheckFailure(error.message, 5) + } + + for (const value of values) { + if (!Array.isArray(value)) { + throw new CheckFailure('A line of the results file does not hold a list of results.', 5) + } + for (const item of value) { + results.push(item) + } + } + } + + return results +} + +/** + * Counts the ERROR and WARNING results per rule code. + * + * A `null` item is skipped, as jq skips it. Any other item that is not an object, and any counted + * result whose code is not a string, fails: a rule with no name cannot be compared. + * + * @param {object[]} results From parseResults. + * + * @throws {CheckFailure} For an item or a code of the wrong type. + * + * @returns {{errors: Map, warnings: Map}} Counts, codes sorted by + * Unicode code point. + */ +function countPerCode(results) { + for (const item of results) { + if (item !== null && (typeof item !== 'object' || Array.isArray(item))) { + throw new CheckFailure(`A Plugin Check result is not an object: ${JSON.stringify(item)}`, 5) + } + } + + const count = (type) => { + const counts = new Map() + + for (const item of results) { + if (item === null || item.type !== type) { + continue + } + if (typeof item.code !== 'string') { + throw new CheckFailure(`A Plugin Check ${type} result has no rule code: ${JSON.stringify(item)}`, 5) + } + counts.set(item.code, (counts.get(item.code) ?? 0) + 1) + } + + return new Map([...counts].sort(([a], [b]) => compareCodePoints(a, b))) + } + + return { errors: count('ERROR'), warnings: count('WARNING') } +} + +/** + * The text of a baseline file: sorted keys, two-space indent, one final newline, and DEL written as + * \u007f, exactly as `jq -S` prints it. Built by hand, because an object puts keys such as "10" + * before "9" whatever order they were added in. + * + * @param {{errors: Map, warnings: Map}} counts From countPerCode. + * + * @returns {string} The file's text. + */ +function formatBaseline(counts) { + const kind = (map) => { + if (map.size === 0) { + return '{}' + } + + const lines = [...map].map(([code, count]) => ` ${JSON.stringify(code).replace(/\x7f/g, '\\u007f')}: ${count}`) + + return `{\n${lines.join(',\n')}\n }` + } + + return `{\n "errors": ${kind(counts.errors)},\n "warnings": ${kind(counts.warnings)}\n}\n` +} + +/** + * Reads a baseline file. + * + * An empty file, or one that holds `null`, is an empty baseline. A kind that is `null` or missing + * is empty. A count that is `null` or `false` is zero. Anything that cannot be compared fails. + * + * @param {string} text The baseline file. + * + * @throws {CheckFailure} When it is not a baseline. + * + * @returns {{errors: Map, warnings: Map}} Counts. + */ +function readBaseline(text) { + const empty = { errors: new Map(), warnings: new Map() } + const [value] = readJsonValues(text) + + if (value === undefined || value === null) { + return empty + } + + const invalid = (reason) => new CheckFailure(`The baseline is not valid: ${reason}`, 5) + + if (typeof value !== 'object' || Array.isArray(value)) { + throw invalid('it must be an object with "errors" and "warnings".') + } + + const counts = {} + for (const kind of ['errors', 'warnings']) { + const entries = value[kind] + counts[kind] = new Map() + + if (entries === undefined || entries === null) { + continue + } + if (typeof entries !== 'object' || Array.isArray(entries)) { + throw invalid(`"${kind}" must be an object of counts per rule code.`) + } + + for (const [code, count] of Object.entries(entries)) { + if (count === null || count === false) { + counts[kind].set(code, 0) + } else if (Number.isSafeInteger(count)) { + counts[kind].set(code, count) + } else if (Number.isInteger(count)) { + // JavaScript cannot hold it exactly, so a table or a message would show another number. + throw new CheckFailure( + `The baseline is not valid: the count of ${displayCode(code)} in "${kind}" is more than ${Number.MAX_SAFE_INTEGER}, which cannot be compared exactly.`, + 5 + ) + } else { + throw new CheckFailure( + `The baseline is not valid: the count of ${displayCode(code)} in "${kind}" must be a whole number.`, + typeof count === 'number' ? 1 : 5 + ) + } + } + } + + return counts +} + +/** + * A rule code as it is shown in a message: a backslash, a tab, a newline and a carriage return are + * written out, as jq's @tsv writes them. + * + * @param {string} code The rule code. + * + * @returns {string} The code, safe to show on one line. + */ +function displayCode(code) { + return code.replace(/\\/g, '\\\\').replace(/\t/g, '\\t').replace(/\n/g, '\\n').replace(/\r/g, '\\r') +} + +/** + * One row for each rule code found in either set of counts, for errors and then warnings. + * + * Rules that went up come first, errors before warnings, the biggest increase first; the rest keep + * their order by kind and code. The sort is stable, as jq's is, so the order is the same every time. + * + * @param {object} now This run's counts. + * @param {object} base The baseline's counts. + * + * @returns {{kind: string, code: string, base: number, now: number}[]} The rows. + */ +function buildRows(now, base) { + const rows = [] + + for (const kind of ['errors', 'warnings']) { + const codes = [...new Set([...now[kind].keys(), ...base[kind].keys()])].sort(compareCodePoints) + + for (const code of codes) { + rows.push({ kind, code, base: base[kind].get(code) ?? 0, now: now[kind].get(code) ?? 0 }) + } + } + + return rows.sort( + (a, b) => + (a.now > a.base ? 0 : 1) - (b.now > b.base ? 0 : 1) || + (a.kind === 'errors' ? 0 : 1) - (b.kind === 'errors' ? 0 : 1) || + -(a.now - a.base) - -(b.now - b.base) + ) +} + +/** + * Judges the rows: which messages to print, the totals, and whether the check fails. + * + * @param {object} input + * @param {object[]} input.rows From buildRows. + * @param {string} input.mode `baseline` or `strict`. + * @param {boolean} input.noBaseline There was nothing to compare with: only the totals count. + * @param {string} input.baselineName The baseline file's name, for the hint to update it. + * @param {boolean} input.fromFile The baseline is the committed file, which can be lowered. + * + * @returns {object} `messages` (`{level, text}` in print order), `totals`, `wentUp`, `wentDown`, + * `errorsDown` (how many rules have fewer errors), `errorsUp`, `upList` and `failed`. + */ +function evaluate({ rows, mode, noBaseline, baselineName = BASELINE_FILE, fromFile = true }) { + const totals = { errorsBase: 0, errorsNow: 0, warningsBase: 0, warningsNow: 0 } + const messages = [] + let failed = false + let wentDown = 0 + let errorsDown = 0 + let wentUp = 0 + let errorsUp = 0 + let upList = '' + + for (const { kind, code, base: allowed, now: found } of rows) { + let label = '' + + if (kind === 'errors') { + totals.errorsBase += allowed + totals.errorsNow += found + } else { + totals.warningsBase += allowed + totals.warningsNow += found + label = ' warning' + } + + // With no baseline, only the totals mean anything. + if (noBaseline) { + continue + } + + const shown = displayCode(code) + + if (found > allowed) { + wentUp++ + if (wentUp <= MAX_RULES_LISTED) { + upList += `${upList ? ', ' : ''}\`${shown}\`${label} (+${found - allowed})` + } + + if (kind !== 'errors') { + messages.push({ + level: 'notice', + text: `${shown} has ${found} warnings, the baseline has ${allowed}. Warnings do not fail the check.`, + }) + continue + } + + errorsUp++ + failed = true + messages.push({ + level: 'error', + text: `${shown} has ${found} errors, the baseline has ${allowed}. The full list is in the plugin-check-results artifact.`, + }) + } else if (found < allowed) { + wentDown++ + if (kind === 'errors') { + errorsDown++ + } + messages.push({ level: 'notice', text: `${shown} (${kind}) went down from ${allowed} to ${found}.` }) + } + } + + // Fewer errors than the file says leaves room for them to come back unnoticed. Strict mode does + // not use the file, so it asks for nothing. + if (mode !== 'strict' && fromFile && errorsDown > 0) { + messages.push({ + level: 'notice', + text: `Error counts went down. Lower the baseline so they cannot come back: commit the baseline.json from this run's plugin-check-report artifact as ${baselineName}.`, + }) + } + + // Strict mode ignores the baseline for pass or fail: any error fails. The counts and the table + // still compare with the baseline, so the comment shows what changed. + if (mode === 'strict') { + failed = totals.errorsNow > 0 + if (failed) { + messages.push({ level: 'error', text: `${totals.errorsNow} errors found. Strict mode fails on any error.` }) + } + } + + return { messages, totals, wentUp, wentDown, errorsDown, errorsUp, upList, failed } +} + +/** + * "+3", "-2" or "±0". + * + * @param {number} now This run's count. + * @param {number} base The baseline's count. + * + * @returns {string} The change. + */ +function change(now, base) { + const diff = now - base + + if (diff > 0) { + return `+${diff}` + } + + return diff < 0 ? String(diff) : '±0' +} + +/** + * The status line at the top of the comment. + * + * The totals can go down while a rule goes up, so the status names the rules, not the totals. + * + * @param {object} input + * @param {string} input.mode `baseline` or `strict`. + * @param {boolean} input.failed The check fails. + * @param {boolean} input.noBaseline Nothing was compared. + * @param {number} input.errorsUp How many rules have more errors. + * @param {number} input.errorsNow This run's error total. + * + * @returns {string} The line. + */ +function statusLine({ mode, failed, noBaseline, errorsUp, errorsNow }) { + if (mode === 'strict' && failed) { + return `❌ Plugin Check failed: ${errorsNow} errors, and strict mode allows none` + } + if (mode === 'strict') { + return '✅ Plugin Check passed: no errors' + } + if (noBaseline) { + return '⚠️ Plugin Check ran, but there is no baseline file yet' + } + if (failed) { + return errorsUp === 1 + ? '❌ Plugin Check failed: 1 rule has more errors than the baseline allows' + : `❌ Plugin Check failed: ${errorsUp} rules have more errors than the baseline allows` + } + + return '✅ Plugin Check passed' +} + +/** + * The line of links at the bottom of the comment. + * + * @param {object} input + * @param {string} input.artifactUrl Link to the uploaded results and the fresh baseline, or ''. + * @param {string} input.runUrl Link to the workflow run, or ''. + * @param {string} input.resultsPath The results file, shown for a run on a developer's machine. + * + * @returns {string} The line. + */ +function linksLine({ artifactUrl, runUrl, resultsPath }) { + if (artifactUrl) { + return `📎 [Full results and new baseline](${artifactUrl})${runUrl ? ` · [Run](${runUrl})` : ''}` + } + + return runUrl ? `📎 [Run](${runUrl})` : `📎 Local run, see ${resultsPath}` +} + +/** + * The body of the pull request comment. + * + * @param {object} input + * @param {string} input.marker Hidden first line. + * @param {string} input.status From statusLine. + * @param {string} input.badge Image URL shown under the status line, or ''. + * @param {string} input.column Name of the baseline column, with its link when it has one. + * @param {boolean} input.noBaseline Leave the baseline and change columns out. + * @param {object} input.totals From evaluate. + * @param {number} input.wentUp How many rules went up. + * @param {number} input.wentDown How many rules went down. + * @param {boolean} input.updateHint An error count went down: ask for the baseline file to be updated. + * @param {string} input.note One line under the table, such as where the baseline came from, or ''. + * @param {string} input.upList The first rules that went up, ready to print. + * @param {string} input.links From linksLine. + * + * @returns {string} The body, cut to MAX_COMMENT_LINES lines, with no final line break. + */ +function buildComment({ marker, status, badge, column, noBaseline, totals, wentUp, wentDown, updateHint = false, note = '', upList, links }) { + let body = `${marker}\n${status}\n\n` + + if (badge) { + body += `![Plugin Check finished](${badge})\n\n` + } + + if (noBaseline) { + body += '| | Now |\n|:--|--:|\n' + body += `| Errors | ${totals.errorsNow} |\n` + body += `| Warnings | ${totals.warningsNow} |\n\n` + } else { + body += `| | Now | ${column} | Change |\n|:--|--:|--:|:--|\n` + body += `| Errors | ${totals.errorsNow} | ${totals.errorsBase} | ${change(totals.errorsNow, totals.errorsBase)} |\n` + body += `| Warnings | ${totals.warningsNow} | ${totals.warningsBase} | ${change(totals.warningsNow, totals.warningsBase)} |\n\n` + } + + if (note) { + body += `${note}\n` + } + if (wentUp > MAX_RULES_LISTED) { + body += `Went up (top ${MAX_RULES_LISTED} of ${wentUp}): ${upList}\n` + } else if (wentUp > 0) { + body += `Went up: ${upList}\n` + } + if (wentDown > 0) { + body += `Rules that went down: ${wentDown}.${updateHint ? ' Update the baseline file, see the job summary.' : ''}\n` + } + // A blank line separates the notes above from the links, when there are notes. + if (wentUp > 0 || wentDown > 0 || note) { + body += '\n' + } + body += links + + // Hard cap: whatever happens above, the comment never grows past MAX_COMMENT_LINES. + return body.split('\n').slice(0, MAX_COMMENT_LINES).join('\n').replace(/\n+$/, '') +} + +/** + * The one line under the table that says where the baseline came from, when that is worth saying. + * + * @param {object} input + * @param {string} input.source `file`, `target` or `none`. + * @param {boolean} input.changed The pull request changes the baseline file, so the comparison is + * with its own change. A reviewer should look at the diff of the file. + * + * @returns {string} The line, or ''. + */ +function baselineNote({ source, changed }) { + if (source === 'target') { + return "ℹ️ There is no baseline file, so the baseline is the target branch's results. See the job summary to add one." + } + + return source === 'file' && changed ? 'ℹ️ This pull request changes the baseline file.' : '' +} + +/** + * Reads the baseline file, and names it when it cannot be used. + * + * Plugin Check reports it as an error annotation: a committed file can be broken by a merge conflict + * or by hand, and the author has to fix it. + * + * @param {string} file The path. + * @param {string} name The file's name, as the messages show it. + * + * @throws {CheckFailure} When it is not a baseline. + * + * @returns {{errors: Map, warnings: Map}} Counts. + */ +function readBaselineFile(file, name) { + try { + return readBaseline(fs.readFileSync(file, 'utf8')) + } catch (error) { + if (!(error instanceof CheckFailure)) { + throw error + } + + const reason = error.message + .replace(/^The baseline is not valid: /, '') + .replace(/^jq: parse error: /, 'it is not JSON: ') + + // A parser's message has no full stop of its own, and the sentence after it needs one. + const stop = /[.!?]$/.test(reason) ? '' : '.' + + throw new CheckFailure( + `The baseline file ${name} is not valid: ${reason}${stop} Fix it, or replace it with the baseline.json from this run's plugin-check-report artifact.`, + error.exitCode, + true + ) + } +} + +/** + * The target branch's results as a baseline, for a pull request in a repository with no baseline + * file. The workflow checked the target branch's commit in the same run, so its counts are what the + * pull request is compared with. + * + * @param {object} env PLUGIN_CHECK_BASE_RESULTS (the results file) and PLUGIN_CHECK_BASE_REQUIRED. + * @param {string} name The baseline file's name, for the message. + * + * @throws {CheckFailure} When the results are missing or are not Plugin Check output. Never passes + * without a baseline, because the workflow asked for one. + * + * @returns {{errors: Map, warnings: Map}} Counts. + */ +function readTargetResults(env, name) { + const file = env.PLUGIN_CHECK_BASE_RESULTS || '' + + if (!file || !isFile(file)) { + throw new CheckFailure( + `The target branch could not be checked, so there is nothing to compare with. Look at the job that checks the target branch in this run. To skip that check, commit the baseline.json from this run's plugin-check-report artifact as ${name}.`, + 1, + true + ) + } + + const text = fs.readFileSync(file, 'utf8') + const refusal = checkResultsText(text) + if (refusal) { + throw new CheckFailure(`The results of the target branch cannot be used. ${refusal}`, 1, true) + } + + return countPerCode(parseResults(text)) +} + +/** + * The part of the job summary that says how to change the baseline file, or '' when there is nothing + * to do. The comment stays short; this is where the commands go. + * + * @param {object} input + * @param {string} input.mode `baseline` or `strict`. + * @param {boolean} input.noFile There is no baseline file. + * @param {object} input.verdict From evaluate. + * @param {string} input.baselineName The baseline file's name. + * @param {object} input.env GITHUB_RUN_ID and GITHUB_REPOSITORY, for the real command. + * + * @returns {string} Markdown with a final line break, or ''. + */ +function updateSection({ mode, noFile, verdict, baselineName, env }) { + if (mode === 'strict') { + return '' + } + + const lines = [] + if (noFile) { + lines.push(`There is no ${baselineName} yet. To start one, add the file from this run:`) + } else if (verdict.errorsDown > 0) { + lines.push(`Error counts went down. To lower the baseline, replace ${baselineName} with the file from this run:`) + } else if (verdict.failed) { + lines.push( + `If the new errors are on purpose, raise their counts in ${baselineName} in this pull request, so reviewers see it. The file from this run has the new counts:` + ) + } else { + return '' + } + + const run = env.GITHUB_RUN_ID || '' + const repo = env.GITHUB_REPOSITORY || '/' + + return [ + '', + ...lines, + '', + '```bash', + `gh run download ${run} --repo ${repo} --name plugin-check-report --dir /tmp/plugin-check`, + `cp /tmp/plugin-check/baseline.json ${baselineName}`, + '```', + '', + ].join('\n') +} + +/** + * Compares a results file with a baseline, or writes the baseline for the results. + * + * @param {object} input + * @param {string[]} input.args Arguments: `--update` and `--comment-file ` in any order, then + * the results file. + * @param {object} input.env Environment. PLUGIN_CHECK_MODE (`baseline`, the default, or `strict`), + * COMMENT_MARKER, PLUGIN_CHECK_BASELINE (default: .plugin-check-baseline.json + * in the current folder), PLUGIN_CHECK_BASE_RESULTS (the target branch's results, + * used as the baseline when there is no file) and + * PLUGIN_CHECK_BASE_REQUIRED (`true`: they must exist), + * PLUGIN_CHECK_BASELINE_CHANGED (`true`: this pull request changes the + * baseline file), COMMENT_BADGE, + * RUN_URL, ARTIFACT_URL, GITHUB_STEP_SUMMARY. + * @param {object} input.log Where messages go: `out`, `err`, `error`, `warning`, `notice`. + * + * @returns {number} The exit status: 0 passes, 1 fails the check, another number is a file that + * could not be read as it should be. + */ +function compare({ args, env, log }) { + try { + return compareOrThrow({ args, env, log }) + } catch (error) { + if (error instanceof CheckFailure) { + // A baseline file that cannot be read is the author's to fix, so it shows in the checks panel. + if (error.annotate) { + log.error(error.message) + } else { + log.err(error.message) + } + + return error.exitCode + } + + throw error + } +} + +/** + * The body of compare(), which lets a CheckFailure through. + * + * @see compare + * + * @param {object} input The same input as compare(). + * + * @returns {number} The exit status. + */ +function compareOrThrow({ args, env, log }) { + // Checked before anything else, so a typo in the caller never runs as the default. + const mode = env.PLUGIN_CHECK_MODE || DEFAULT_MODE + if (!MODES.includes(mode)) { + const renamed = RENAMED_MODES[mode] ? ` '${mode}' is now called '${RENAMED_MODES[mode]}'.` : '' + log.error(`Unknown mode '${mode}'. Use 'baseline' or 'strict'.${renamed}`) + + return 1 + } + + let update = false + let commentFile = '' + let rest = [...args] + while (rest.length > 0) { + if (rest[0] === '--update') { + update = true + rest = rest.slice(1) + } else if (rest[0] === '--comment-file') { + if (!rest[1]) { + log.err('--comment-file needs a path') + + return 1 + } + commentFile = rest[1] + rest = rest.slice(2) + } else { + break + } + } + + const resultsPath = rest[0] + if (!resultsPath) { + log.err('Usage: compare [--update | --comment-file ] ') + + return 1 + } + + // The default is the committed file in the current folder, so `compare --update` from the plugin + // root writes it in place. + const baselinePath = env.PLUGIN_CHECK_BASELINE || path.resolve(BASELINE_FILE) + const baselineName = path.basename(baselinePath) + + if (!isFile(resultsPath)) { + log.error(`Results file not found: ${resultsPath}`) + + return 1 + } + + const text = fs.readFileSync(resultsPath, 'utf8') + const refusal = checkResultsText(text) + if (refusal) { + log.error(refusal) + + return 1 + } + + const now = countPerCode(parseResults(text)) + + if (update) { + writeText(baselinePath, formatBaseline(now), 'write') + log.out(`Wrote ${baselinePath}: ${now.errors.size} error codes, ${now.warnings.size} warning codes`) + + return 0 + } + + // Where the baseline comes from: the committed file, else (baseline mode, pull request) the target + // branch's results, else nothing. With no baseline at all, the counts are reported and the check + // passes: failing would block a push before anyone has a result to make the file from. + const hasFile = isFile(baselinePath) + let source = hasFile ? 'file' : 'none' + let base = { errors: new Map(), warnings: new Map() } + + if (hasFile) { + base = readBaselineFile(baselinePath, baselineName) + } else if (mode === 'strict') { + // Strict mode does not use a baseline, so it has nothing to say and builds nothing. + log.out('No baseline file. Strict mode does not need one.') + } else if (env.PLUGIN_CHECK_BASE_RESULTS || env.PLUGIN_CHECK_BASE_REQUIRED === 'true') { + base = readTargetResults(env, baselineName) + source = 'target' + log.warning( + `No baseline file at ${baselineName}, so the target branch was checked too, and its results are the baseline for this run. That makes the run slower. To stop it, commit the baseline.json from this run's plugin-check-report artifact as ${baselineName}.` + ) + } else { + log.warning( + `No baseline file at ${baselineName}, so nothing was compared. To start one, commit the baseline.json from this run's plugin-check-report artifact as ${baselineName}.` + ) + } + + const noBaseline = source === 'none' + const verdict = evaluate({ rows: buildRows(now, base), mode, noBaseline, baselineName, fromFile: hasFile }) + + for (const { level, text: message } of verdict.messages) { + log[level](message) + } + + let runUrl = env.RUN_URL || '' + if (!runUrl && env.GITHUB_RUN_ID) { + for (const name of ['GITHUB_SERVER_URL', 'GITHUB_REPOSITORY']) { + if (env[name] === undefined) { + log.err(`${name}: unbound variable`) + + return 1 + } + } + runUrl = `${env.GITHUB_SERVER_URL}/${env.GITHUB_REPOSITORY}/actions/runs/${env.GITHUB_RUN_ID}` + } + + const body = buildComment({ + marker: env.COMMENT_MARKER || '', + status: statusLine({ mode, noBaseline, ...verdict, errorsNow: verdict.totals.errorsNow }), + badge: env.COMMENT_BADGE || '', + column: source === 'target' ? 'Target branch' : 'Baseline', + noBaseline, + totals: verdict.totals, + wentUp: verdict.wentUp, + wentDown: verdict.wentDown, + updateHint: mode !== 'strict' && hasFile && verdict.errorsDown > 0, + note: baselineNote({ source, changed: env.PLUGIN_CHECK_BASELINE_CHANGED === 'true' }), + upList: verdict.upList, + links: linksLine({ artifactUrl: env.ARTIFACT_URL || '', runUrl, resultsPath }), + }) + + if (commentFile) { + writeText(commentFile, `${body}\n`, 'write') + } + if (env.GITHUB_STEP_SUMMARY) { + const how = updateSection({ mode, noFile: !hasFile, verdict, baselineName, env }) + writeText(env.GITHUB_STEP_SUMMARY, `${body}\n${how}`, 'append') + } + + if (verdict.failed) { + log.out( + mode === 'strict' + ? 'Plugin Check found errors, and strict mode allows none.' + : 'Plugin Check found more errors than the baseline allows.' + ) + + return 1 + } + + log.out( + noBaseline + ? 'Plugin Check ran with no baseline, so nothing was compared.' + : 'Plugin Check error counts are within the baseline.' + ) + + return 0 +} + +/** + * Writes a file, or adds to it. A folder that does not exist fails the step with the reason. + * + * @param {string} file The path. + * @param {string} text What to write. + * @param {string} how `write` replaces the file, `append` adds to it. + * + * @throws {CheckFailure} When the file cannot be written. + */ +function writeText(file, text, how) { + try { + if (how === 'append') { + fs.appendFileSync(file, text) + } else { + fs.writeFileSync(file, text) + } + } catch (error) { + throw new CheckFailure(`Could not write ${file}: ${error.message}`, 1) + } +} + +/** + * Whether a path is a file, following links, as `[ -f ]` does. + * + * @param {string} file The path. + * + * @returns {boolean} True for a file. + */ +function isFile(file) { + try { + return fs.statSync(file).isFile() + } catch { + return false + } +} + +/** + * Where messages go when the code runs from a command line: annotations as the `::error` lines a + * runner would turn into annotations, and plain text on the two streams. + * + * @param {{out: Function, err: Function}} io Writes one line to standard output or standard error. + * + * @returns {object} The log. + */ +function commandLog(io) { + const annotation = (level) => (message) => io.out(`::${level} title=${TITLE}::${message}`) + + return { + out: io.out, + err: io.err, + error: annotation('error'), + warning: annotation('warning'), + notice: annotation('notice'), + } +} + +/** + * Where messages go inside an actions/github-script step. + * + * @param {object} core The toolkit github-script passes in. + * + * @returns {object} The log. + */ +function actionLog(core) { + const annotate = (method) => (message) => core[method](message, { title: TITLE }) + + return { + out: (line) => core.info(line), + err: (line) => core.info(line), + error: annotate('error'), + warning: annotate('warning'), + notice: annotate('notice'), + } +} + +/** + * The first step: where this run keeps its files. + * + * One folder per run under RUNNER_TEMP, cleared first, so a second use of the action in the same + * job never reads the first one's files. + * + * @param {object} input + * @param {object} input.core The toolkit github-script passes in. + * @param {object} input.env The environment: RUNNER_TEMP, GITHUB_REPOSITORY and INPUT_COMMENT_MARKER. + * + * @returns {number} 0. + */ +function runPaths({ core, env }) { + const dir = path.join(env.RUNNER_TEMP, 'plugin-check-report') + fs.rmSync(dir, { recursive: true, force: true }) + fs.mkdirSync(path.join(dir, 'report'), { recursive: true }) + + core.setOutput('report', path.join(dir, 'report')) + core.setOutput('marker', env.INPUT_COMMENT_MARKER || ``) + + return 0 +} + +/** + * The second step: a baseline made from this run, kept even when the compare fails. Later runs + * download it as their baseline. + * + * @param {object} input + * @param {object} input.core The toolkit. + * @param {object} input.env RESULTS and REPORT_DIR. + * + * @returns {number} The exit status. A results file that is not Plugin Check output fails here too, + * with the same message, and then nothing is uploaded. + */ +function runUpdate({ core, env }) { + if (!isFile(env.RESULTS ?? '')) { + return 0 + } + + fs.copyFileSync(env.RESULTS, path.join(env.REPORT_DIR, path.basename(env.RESULTS))) + + return compare({ + args: ['--update', env.RESULTS], + env: { ...env, PLUGIN_CHECK_BASELINE: path.join(env.REPORT_DIR, 'baseline.json') }, + log: actionLog(core), + }) +} + +/** + * The last step: fails the step when the mode says so. Also writes the comment body and the same + * text to the job summary. + * + * @param {object} input + * @param {object} input.core The toolkit. + * @param {object} input.env RESULTS, COMMENT_FILE, and the variables compare() reads. + * + * @returns {number} The exit status. + */ +function runCompare({ core, env, now }) { + const args = env.COMMENT_FILE ? ['--comment-file', env.COMMENT_FILE] : [] + + // The comment always shows the finish badge, so the action makes it: no caller passes one. + return compare({ + args: [...args, env.RESULTS ?? ''], + env: { ...env, COMMENT_BADGE: finishBadge(now) }, + log: actionLog(core), + }) +} + +/** The time zone of the time on the badges. */ +const BADGE_TIME_ZONE = 'America/New_York' + +/** + * The time on a badge, like `11/03 06:08 PM EST`: month and day, a 12-hour clock, and the zone's + * short name, which is EST in winter and EDT in summer. It is what `date '+%m/%d %I:%M %p %Z'` prints + * with `LC_ALL=C` and `TZ=America/New_York`. + * + * Tested for the default zone only. A zone whose short name Intl writes as `GMT+1` would need the + * `+` and the `-` of a shields.io badge escaped. + * + * @param {Date} now The moment. + * @param {string} timeZone An IANA time zone. + * + * @returns {string} The time, not yet escaped for a URL. + */ +function badgeStamp(now, timeZone) { + const parts = Object.fromEntries( + new Intl.DateTimeFormat('en-US', { + timeZone, + month: '2-digit', + day: '2-digit', + hour: '2-digit', + minute: '2-digit', + hour12: true, + timeZoneName: 'short', + }) + .formatToParts(now) + .map(({ type, value }) => [type, value]) + ) + + return `${parts.month}/${parts.day} ${parts.hour}:${parts.minute} ${parts.dayPeriod} ${parts.timeZoneName}` +} + +/** + * The "Check Finished" badge shown under the status line of the comment: a shields.io image with + * the time the check finished. Only a space and a slash are escaped, as the shell step this + * replaced did, so the colon of the time stays as it is. + * + * The workflow's "Check Started" badge in the `start` job is made by a shell step of the same + * shape. That job has no checkout to run this file from. + * + * @param {Date} now The moment. Default: now. + * @param {string} timeZone An IANA time zone. Default: America/New_York. + * + * @returns {string} The image URL. + */ +function finishBadge(now = new Date(), timeZone = BADGE_TIME_ZONE) { + const stamp = badgeStamp(now, timeZone).replace(/ /g, '%20').replace(/\//g, '%2F') + + return `https://img.shields.io/badge/Check%20Finished-${stamp}-darkgreen` +} + +/** + * The comment for a run that could not compare anything: the build or the check broke before the + * results existed, or the compare refused its settings. + * + * @param {object} input + * @param {string} input.marker The hidden first line. + * @param {string} input.badge The badge URL. + * @param {string} input.runUrl The link to the workflow run. + * + * @returns {string} The Markdown body, with a final line break. + */ +function couldNotRunComment({ marker, badge, runUrl }) { + return `${marker}\n❌ Plugin Check could not run\n\n![Plugin Check finished](${badge})\n\n📎 [Run](${runUrl})\n` +} + +/** + * The step after the compare: when the compare wrote no comment body, says so, instead of leaving + * "in progress" on the pull request. A body that exists and is not empty is left alone, whatever + * the verdict was. + * + * @param {object} input + * @param {object} input.core The toolkit. + * @param {object} input.env COMMENT_FILE, COMMENT_MARKER, RUN_URL and GITHUB_REPOSITORY. + * @param {Date} input.now Optional clock. + * + * @returns {number} 0, or 1 when the file cannot be written. + */ +function runCouldNotRun({ core, env, now }) { + const file = env.COMMENT_FILE ?? '' + if (!file) { + return 0 + } + + try { + if (fs.statSync(file).size > 0) { + return 0 + } + } catch { + // No file: write it. + } + + const marker = env.COMMENT_MARKER || `` + try { + writeText(file, couldNotRunComment({ marker, badge: finishBadge(now), runUrl: env.RUN_URL ?? '' }), 'write') + } catch (error) { + core.error(error.message, { title: TITLE }) + + return 1 + } + core.info('The compare wrote no comment, so the comment says Plugin Check could not run.') + + return 0 +} + +/** The file a repository keeps its excluded checks in, at its root. */ +const EXCLUDE_FILE = '.plugin-check.json' + +/** + * The checks to skip, from the workflow input: one per line. A comma works too, because the Plugin + * Check action turns the lines into a comma list. Blank lines and repeats are dropped. + * + * @param {string} text The input. + * + * @returns {string[]} The check names, in order. + */ +function parseExcludeInput(text) { + return [...new Set(text.split(/[\r\n,]+/).map((line) => line.trim()).filter(Boolean))] +} + +/** + * The checks to skip, from the repository's `.plugin-check.json`: + * `{"exclude-checks": ["check_a", "check_b"]}`. Other keys are ignored, so the file can grow. A + * missing `exclude-checks` key means none. Repeats are dropped. + * + * @param {string} text The file's contents. + * @param {string} name What to call the file in a message. + * + * @returns {string[]} The check names, in order. + * + * @throws {CheckFailure} When the file cannot be used. The message says what to fix. + */ +function parseExcludeFile(text, name = EXCLUDE_FILE) { + let data + try { + data = JSON.parse(text) + } catch (error) { + throw new CheckFailure(`${name} is not valid JSON: ${error.message}`, 1) + } + + if (data === null || typeof data !== 'object' || Array.isArray(data)) { + throw new CheckFailure(`${name} must hold a JSON object, like {"exclude-checks": ["check_name"]}.`, 1) + } + + const list = data['exclude-checks'] + if (list === undefined) { + return [] + } + if (!Array.isArray(list)) { + throw new CheckFailure(`"exclude-checks" in ${name} must be a list of check names, like ["check_name"].`, 1) + } + + const checks = [] + for (const entry of list) { + const check = typeof entry === 'string' ? entry.trim() : '' + if (!check || /[\s,]/.test(check)) { + throw new CheckFailure( + `"exclude-checks" in ${name} has ${JSON.stringify(entry)}, which is not a check name. Each entry is one text without spaces or commas.`, + 1 + ) + } + checks.push(check) + } + + return [...new Set(checks)] +} + +/** + * Which checks to skip. The workflow input wins when it names any: then the file is not even read. + * Otherwise the file is used. With neither, none. + * + * @param {object} input + * @param {string} input.input The workflow input. + * @param {string|null} input.fileText The file's contents, or null when there is no file. + * @param {string} input.fileName What to call the file in a message. + * + * @returns {{checks: string[], source: string}} The check names, and where they came from: `input`, + * `file` or `none`. + * + * @throws {CheckFailure} When the file is used and cannot be. + */ +function excludedChecks({ input = '', fileText = null, fileName = EXCLUDE_FILE }) { + const fromInput = parseExcludeInput(input) + if (fromInput.length > 0) { + return { checks: fromInput, source: 'input' } + } + if (fileText === null) { + return { checks: [], source: 'none' } + } + + const checks = parseExcludeFile(fileText, fileName) + + return { checks, source: checks.length > 0 ? 'file' : 'none' } +} + +/** + * The file's contents, or null when there is no such file. + * + * @param {string} file The path. + * + * @returns {string|null} The text. + * + * @throws {CheckFailure} When the file exists but cannot be read. + */ +function readIfFile(file) { + if (!file || !isFile(file)) { + return null + } + + try { + return fs.readFileSync(file, 'utf8') + } catch (error) { + throw new CheckFailure(`Could not read ${file}: ${error.message}`, 1) + } +} + +/** + * The step before the Plugin Check action: works out which checks to skip, from the workflow input + * or else the repository's `.plugin-check.json`. + * + * Sets the output `checks`: one name per line, the format the Plugin Check action takes. + * + * @param {object} input + * @param {object} input.core The toolkit. + * @param {object} input.env EXCLUDE_CHECKS (the workflow input) and CONFIG_FILE (the path). + * + * @returns {number} 0, or 1 when the file cannot be used. + */ +function runExcludeChecks({ core, env }) { + const file = env.CONFIG_FILE ?? '' + + try { + const { checks, source } = excludedChecks({ + input: env.EXCLUDE_CHECKS ?? '', + // Not read when the input names checks: the input replaces the file. + fileText: parseExcludeInput(env.EXCLUDE_CHECKS ?? '').length > 0 ? null : readIfFile(file), + fileName: path.basename(file) || EXCLUDE_FILE, + }) + + core.setOutput('checks', checks.join('\n')) + core.info( + source === 'none' + ? 'No checks are excluded.' + : `Excluding ${checks.length} check(s) from the ${source === 'input' ? 'exclude-checks input' : path.basename(file)}: ${checks.join(', ')}` + ) + + return 0 + } catch (error) { + if (error instanceof CheckFailure) { + core.error(error.message, { title: TITLE }) + + return error.exitCode + } + + throw error + } +} + +/** + * The command line: `compare`, `update` and `exclude-checks`. + * + * @param {string[]} argv The arguments after the script name. + * @param {object} env The environment, default process.env. + * @param {object} io `out` and `err`, each writing one line. Default: the process streams. + * + * @returns {Promise} The exit status. + */ +async function main( + argv, + env = process.env, + io = { + out: (line) => process.stdout.write(`${line}\n`), + err: (line) => process.stderr.write(`${line}\n`), + } +) { + const [command, ...args] = argv + const log = commandLog(io) + + if (command === 'compare') { + return compare({ args, env, log }) + } + + // `update ` is `compare --update `: writes the baseline file for the results. + if (command === 'update') { + return compare({ args: ['--update', ...args], env, log }) + } + + // Prints the flag `wp plugin check` takes, so a local run skips the same checks as CI: + // `wp plugin check my-plugin $(node plugin-check-report.js exclude-checks)`. Prints nothing when + // no check is excluded. EXCLUDE_CHECKS, when set, replaces the file, as the workflow input does. + if (command === 'exclude-checks') { + const file = args[0] ?? EXCLUDE_FILE + + try { + const input = env.EXCLUDE_CHECKS ?? '' + const { checks } = excludedChecks({ + input, + fileText: parseExcludeInput(input).length > 0 ? null : readIfFile(file), + fileName: file, + }) + if (checks.length > 0) { + io.out(`--exclude-checks=${checks.join(',')}`) + } + + return 0 + } catch (error) { + if (error instanceof CheckFailure) { + log.err(error.message) + + return error.exitCode + } + + throw error + } + } + + io.err('Usage: plugin-check-report.js compare|update|exclude-checks ...') + + return 2 +} + +if (require.main === module) { + main(process.argv.slice(2)).then((status) => { + process.exitCode = status + }) +} + +module.exports = { + CheckFailure, + MODES, + actionLog, + BADGE_TIME_ZONE, + BASELINE_FILE, + EXCLUDE_FILE, + badgeStamp, + buildComment, + buildRows, + change, + checkResultsText, + commandLog, + compare, + compareCodePoints, + countPerCode, + couldNotRunComment, + cutBytes, + displayCode, + evaluate, + excludedChecks, + finishBadge, + formatBaseline, + linksLine, + main, + parseExcludeFile, + parseExcludeInput, + parseResults, + readBaseline, + baselineNote, + readBaselineFile, + readTargetResults, + readIfFile, + readJsonValues, + runCompare, + runCouldNotRun, + runExcludeChecks, + runPaths, + runUpdate, + statusLine, + updateSection, +} diff --git a/.github/workflows/plugin-check.yml b/.github/workflows/plugin-check.yml new file mode 100644 index 0000000..508ba87 --- /dev/null +++ b/.github/workflows/plugin-check.yml @@ -0,0 +1,559 @@ +name: Plugin Check + +# Runs WordPress.org's Plugin Check against the built plugin: the same files the release zip ships, +# so tests, dotfiles, node_modules and other dev-only files cannot cause false positives. +# +# A reusable workflow. A plugin repository calls it from a thin workflow of its own; copy +# templates/workflows/plugin-check.yml. +# +# The plugin is built with pup by default (`pup build`, then `pup package`), so the repository's +# .puprc and ignore files decide what is checked, the same as the release. pup comes from the shared +# zip-name and release-zip actions, so the repository does not need a copy of it or a Composer +# script for it. A repository without pup passes `build-command` instead; see the plugin-check-report README. +# +# Four jobs, so the third-party action never holds a token that can write. The comments are written +# by .github/actions/pr-comment, which calls the GitHub API itself: +# start posts "in progress" on the pull request. +# plan lists the excluded checks, and decides whether the target branch is checked too. +# check builds the plugin and runs wordpress/plugin-check-action. Read-only token. It runs for +# the pull request ("head") and, when the repository has no baseline file, for the commit +# the pull request targets ("base"), in parallel. +# report compares the results with the baseline and replaces the comment with the result. +# +# The baseline is the repository's `.plugin-check-baseline.json`, read from the pull request's own +# checkout. Without that file, a pull request is compared with the target branch's results instead, +# which costs a second build and check; commit the file to stop that. +# +# The action also tries to post its own comment. With a read-only token that call fails (it logs +# "Failed to post PR comment" and carries on), so the only comment is ours. +# +# The caller has to grant every permission a job below asks for: a called workflow can lower the +# token's permissions, never raise them, and GitHub refuses to start the run when a job asks for +# more than the caller gave. `concurrency` is the caller's too: here it would be evaluated with the +# caller's context and could clash with the caller's own group. +# +# Run it locally: see "Running it locally" in .github/actions/plugin-check-report/README.md. In +# short, `wp plugin check --format=json > results.txt` in a local WordPress, then +# `node .github/actions/plugin-check-report/plugin-check-report.js compare results.txt`. + +on: + workflow_call: + inputs: + plugin-slug: + description: >- + The plugin's folder name, e.g. `my-plugin`. It has to equal the top folder of the zip pup + builds (`zip_name` in .puprc), because Plugin Check takes the slug from the folder name. + required: true + type: string + main-file: + description: 'The main plugin file inside the plugin folder. Empty means `.php`.' + required: false + type: string + default: '' + build-command: + description: >- + Empty (the default) builds with pup. Otherwise a bash command that runs in the root of + the checkout instead of pup, and leaves the plugin as it ships in + "$PLUGIN_CHECK_BUILD_DIR//". Prefer .puprc when the repository has one: this + is a second place that defines the build. + required: false + type: string + default: '' + pup-version: + description: >- + The pup release that builds the plugin, e.g. `2.0.0`, or `latest`. Use the release the + repository's own release build uses, so the check sees the same files. Not used with + `build-command`. + required: false + type: string + default: '2.0.0' + runs-on: + description: >- + Runner label for the job that builds and checks, of runner version 2.336.0 or newer: the + job uses the `$/` reference. The two small jobs run on ubuntu-latest. + required: false + type: string + default: 'ubuntu-latest' + php-version: + description: >- + PHP that builds vendor/. Use the oldest version the plugin supports, as its release does, + so Composer resolves dependencies that still run there. + required: false + type: string + default: '7.4' + exclude-checks: + description: >- + Checks to skip, one per line. When it names any, it replaces the `exclude-checks` list in + the repository's `.plugin-check.json`; when empty, that file is used. See the + plugin-check-report README. + required: false + type: string + default: '' + cache-paths: + description: >- + Build caches to keep between runs, one path per line, relative to the repository root, + e.g. `.cache/webpack`. Empty keeps none. A path inside node_modules does not survive: + pup's build reinstalls it. + required: false + type: string + default: '' + mode: + description: >- + `baseline`: fail only when a rule has more errors than the baseline. `strict`: fail on any + error. Not the `strict` input of wordpress/plugin-check-action, which means something else. + required: false + type: string + default: 'baseline' + submodules: + description: 'Passed to actions/checkout: `false`, `true` or `recursive`.' + required: false + type: string + default: 'false' + secrets: + composer-token: + description: >- + GitHub token for private Composer packages. Becomes COMPOSER_AUTH for the build step + only. + required: false + package-token: + description: >- + Token for private npm or bun packages. Becomes NODE_AUTH_TOKEN, BUN_AUTH_TOKEN and + GITHUB_TOKEN for the build step only. + required: false + +env: + # Hidden line that marks the sticky pull request comment. The report action writes it as the + # first line of the body, and the pr-comment action searches for it. Named after the repository, so + # two plugin repositories never share a marker. + COMMENT_MARKER: '' + # Only pull requests from this repository get a comment: a fork's token is read-only. + COMMENT: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository }} + +jobs: + start: + name: Plugin Check start + if: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + issues: write + pull-requests: write + steps: + # One comment per pull request, found by a hidden marker line and replaced on every run. The + # badge time is Eastern Time: %Z prints EST in winter and EDT in summer. + - name: Make the start badge + id: badge + run: | + stamp=$(LC_ALL=C TZ='America/New_York' date '+%m/%d %I:%M %p %Z') + stamp=${stamp// /%20} + stamp=${stamp//\//%2F} + echo "url=https://img.shields.io/badge/Check%20Started-${stamp}-blue" >> "$GITHUB_OUTPUT" + + # A failed API call here must not stop the check. + - name: Post "in progress" + continue-on-error: true + uses: $/.github/actions/pr-comment + with: + marker: ${{ env.COMMENT_MARKER }} + body: | + ${{ env.COMMENT_MARKER }} + ⏳ Plugin Check in progress + + ![Plugin Check started](${{ steps.badge.outputs.url }}) + + # Works out, from the repository's files, what the check job needs to know. Cheap: a sparse + # checkout of two files. A bad .plugin-check.json fails here in seconds, before any build. + plan: + name: Plugin Check plan + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read + outputs: + matrix: ${{ steps.target.outputs.matrix }} + base: ${{ steps.target.outputs.base }} + checks: ${{ steps.exclude.outputs.checks }} + steps: + - uses: actions/checkout@v7 + with: + path: source + sparse-checkout: | + .plugin-check.json + .plugin-check-baseline.json + sparse-checkout-cone-mode: false + persist-credentials: false + + # The checks to skip: the `exclude-checks` input when it names any, else the list in the + # repository's .plugin-check.json. Like the report job, this needs a runner of version 2.336.0 + # or newer for `$/`; the two small jobs run on ubuntu-latest, which has it. + - name: List the excluded checks + id: exclude + uses: $/.github/actions/plugin-check-report/exclude-checks + with: + exclude-checks: ${{ inputs.exclude-checks }} + config-file: ${{ github.workspace }}/source/.plugin-check.json + + # A pull request in baseline mode, in a repository with no baseline file, is compared with the + # target branch, so that commit is built and checked too. Strict mode needs no baseline, and a + # push has no target to compare with. + - name: Decide whether to check the target branch + id: target + env: + EVENT: ${{ github.event_name }} + MODE: ${{ inputs.mode }} + run: | + if [ "$EVENT" = pull_request ] && [ "$MODE" != strict ] && [ ! -f source/.plugin-check-baseline.json ]; then + echo 'matrix={"target":["head","base"]}' >> "$GITHUB_OUTPUT" + echo 'base=true' >> "$GITHUB_OUTPUT" + echo "::notice title=Plugin Check::There is no .plugin-check-baseline.json, so the target branch is built and checked too. Commit the baseline.json from the plugin-check-report artifact of this run to stop that." + else + echo 'matrix={"target":["head"]}' >> "$GITHUB_OUTPUT" + echo 'base=false' >> "$GITHUB_OUTPUT" + fi + + check: + name: Plugin Check run (${{ matrix.target }}) + needs: plan + runs-on: ${{ inputs.runs-on }} + + strategy: + # The pull request and its target branch are separate checks: one failing must not stop the other. + fail-fast: false + matrix: ${{ fromJSON(needs.plan.outputs.matrix) }} + + # Build, WordPress start and the check together take about 15 minutes for a large plugin. The cap is + # for a container that stops answering. + timeout-minutes: 30 + + # Read-only on purpose: this job runs a third-party action. Its own attempt to post a comment + # fails with this token, which is what keeps the pull request to one comment. + permissions: + contents: read + + env: + PLUGIN_CHECK_BUILD_DIR: ${{ github.workspace }}/build + SLUG: ${{ inputs.plugin-slug }} + MAIN_FILE: ${{ inputs.main-file || format('{0}.php', inputs.plugin-slug) }} + + steps: + # The repo goes into source/, so the workspace root has no .wp-env.json. A repo's own file + # mounts the repo root as a plugin, and the action would merge into it. + - uses: actions/checkout@v7 + with: + path: source + # "base" is the commit the pull request targets, as it is without the pull request. + ref: ${{ matrix.target == 'base' && github.event.pull_request.base.sha || '' }} + submodules: ${{ inputs.submodules }} + # Nothing here pushes over git, and the job goes on to run third-party code. + persist-credentials: false + + # The action's wp-env pulls these images when it starts, and pulling them is part of the + # time that step takes. Started now, in the background, the download overlaps with the + # build. The images are the ones wp-env's generated files name (FROM wordpress..., FROM + # wordpress:cli..., image: mariadb:lts); if a later wp-env names others, these pulls are just + # wasted, never wrong. A failed pull only means wp-env pulls the image itself, so this step + # cannot fail the job. + - name: Start pulling the wp-env Docker images + continue-on-error: true + env: + IMAGES: wordpress wordpress:cli mariadb:lts + run: | + ( + set +e + start=$(date +%s) + for image in $IMAGES; do + docker pull --quiet "$image" || echo "Could not pull $image, wp-env will pull it." & + done + wait + echo "Pulled in $(( $(date +%s) - start )) seconds." + touch "$RUNNER_TEMP/docker-pull.done" + ) > "$RUNNER_TEMP/docker-pull.log" 2>&1 < /dev/null & + + - name: Set Up PHP + uses: shivammathur/setup-php@v2 + with: + php-version: ${{ inputs.php-version }} + coverage: none + + - name: Find the Composer cache + id: composer-cache + working-directory: source + run: echo "dir=$(composer config cache-files-dir)" >> "$GITHUB_OUTPUT" + + - name: Cache dependencies + uses: actions/cache@v6 + with: + path: ${{ steps.composer-cache.outputs.dir }} + key: composer-${{ hashFiles('source/composer.lock') }} + + # The JavaScript toolchain is picked from the files the repository already has, not from an + # input: .nvmrc for Node, package-lock.json for the npm cache, bun.lock or bun.lockb for bun. + - name: Set Up Node + if: ${{ hashFiles('source/.nvmrc') != '' }} + uses: actions/setup-node@v7 + with: + node-version-file: source/.nvmrc + cache: ${{ hashFiles('source/package-lock.json') != '' && 'npm' || '' }} + cache-dependency-path: source/package-lock.json + + - name: Set Up Bun + if: ${{ hashFiles('source/bun.lock', 'source/bun.lockb') != '' }} + uses: oven-sh/setup-bun@v2 + with: + bun-version: latest + + # actions/cache takes paths relative to the workspace, and the repository sits in source/. + # A path that leaves the checkout is refused: it would cache files this job did not build. + - name: List the cache paths + id: cache-paths + if: ${{ inputs.cache-paths != '' }} + env: + CACHE_PATHS: ${{ inputs.cache-paths }} + run: | + { + echo 'paths<> "$GITHUB_OUTPUT" + + # Each commit saves a new entry; the restore key picks the newest one for the same lockfile. + # A stale cache must be safe for the tool that owns it (webpack checks every entry against + # the files). + - name: Restore the build cache + id: build-cache + if: ${{ inputs.cache-paths != '' }} + uses: actions/cache/restore@v6 + with: + path: ${{ steps.cache-paths.outputs.paths }} + key: plugin-check-${{ runner.os }}-${{ hashFiles('source/package-lock.json', 'source/bun.lock', 'source/bun.lockb') }}-${{ github.sha }} + restore-keys: | + plugin-check-${{ runner.os }}-${{ hashFiles('source/package-lock.json', 'source/bun.lock', 'source/bun.lockb') }}- + + # pup builds and packages the plugin, so .puprc and the ignore files decide what is checked, + # the same as the release. The version and the zip name come from the zip-name action, the zip + # from release-zip: the same QA build the zip workflow makes, without `pup check` and `pup i18n`, + # which this workflow does not need. pup runs on the PHP set up above. The repository is in + # source/, so both actions run there. + - name: Name the zip + id: zip-name + if: ${{ inputs.build-command == '' }} + uses: $/.github/actions/zip-name + with: + production: 'no' + prerelease: dev + pup-version: ${{ inputs.pup-version }} + working-directory: source + + # The secrets reach this step only, never the job, so the third-party action later in this + # job does not see them in its environment. The build runs with them in its environment, as + # pup build needs them to install private packages. + - name: Build the plugin with pup + id: zip + if: ${{ inputs.build-command == '' }} + uses: $/.github/actions/release-zip + with: + version: ${{ steps.zip-name.outputs.version }} + production: 'no' + check: 'no' + i18n: 'no' + pup-version: ${{ inputs.pup-version }} + working-directory: source + env: + COMPOSER_AUTH: >- + ${{ secrets.composer-token != '' && format('{{"github-oauth": {{"github.com": "{0}"}}}}', secrets.composer-token) || '' }} + NODE_AUTH_TOKEN: ${{ secrets.package-token }} + BUN_AUTH_TOKEN: ${{ secrets.package-token }} + GITHUB_TOKEN: ${{ secrets.package-token != '' && secrets.package-token || github.token }} + + - name: Build the plugin with build-command + if: ${{ inputs.build-command != '' }} + working-directory: source + env: + BUILD_COMMAND: ${{ inputs.build-command }} + PACKAGE_TOKEN: ${{ secrets.package-token }} + COMPOSER_TOKEN: ${{ secrets.composer-token }} + run: | + if [ -n "$COMPOSER_TOKEN" ]; then + COMPOSER_AUTH=$(jq -cn --arg token "$COMPOSER_TOKEN" '{"github-oauth": {"github.com": $token}}') + export COMPOSER_AUTH + fi + if [ -n "$PACKAGE_TOKEN" ]; then + export NODE_AUTH_TOKEN="$PACKAGE_TOKEN" BUN_AUTH_TOKEN="$PACKAGE_TOKEN" GITHUB_TOKEN="$PACKAGE_TOKEN" + fi + unset COMPOSER_TOKEN PACKAGE_TOKEN + mkdir -p "$PLUGIN_CHECK_BUILD_DIR" + echo "Building with build-command." + bash -eo pipefail -c "$BUILD_COMMAND" + + - name: Unpack the zip + if: ${{ inputs.build-command == '' }} + env: + ZIP_PATH: ${{ steps.zip.outputs.zip-path }} + run: | + mkdir -p "$PLUGIN_CHECK_BUILD_DIR" + unzip -q "$ZIP_PATH" -d "$PLUGIN_CHECK_BUILD_DIR" + + # The one contract every build has to meet, pup or not: a folder named after the slug, with + # the main plugin file in it. Plugin Check takes the slug from the folder name, so a wrong + # folder would check nothing useful and still look green. + - name: Check the built plugin + env: + CACHE_PATHS: ${{ inputs.cache-paths }} + run: | + dir="$PLUGIN_CHECK_BUILD_DIR/$SLUG" + if [ ! -f "$dir/$MAIN_FILE" ]; then + echo "::error title=Plugin Check::The build did not leave $SLUG/$MAIN_FILE in $PLUGIN_CHECK_BUILD_DIR. Found: $(find "$PLUGIN_CHECK_BUILD_DIR" -mindepth 1 -maxdepth 1 -printf '%f ')" + exit 1 + fi + # A cache that ends up in the plugin would be checked, and shipped by the release. + while IFS= read -r line; do + line=$(echo "$line" | xargs) + [ -n "$line" ] || continue + if [ -e "$dir/$line" ]; then + echo "::error title=Plugin Check::The cache path $line is inside the built plugin. Add it to .distignore or to .gitattributes with export-ignore." + exit 1 + fi + done <<< "$CACHE_PATHS" + echo "Checking $dir ($(find "$dir" -type f | wc -l | xargs) files)." + + # Saved now, before the third-party action below runs, so nothing it does can end up in a + # cache that later runs restore. Skipped when this exact commit already saved one, and for the + # target branch, which only reads the cache. + - name: Save the build cache + if: ${{ matrix.target == 'head' && inputs.cache-paths != '' && steps.build-cache.outputs.cache-hit != 'true' }} + uses: actions/cache/save@v6 + with: + path: ${{ steps.cache-paths.outputs.paths }} + key: ${{ steps.build-cache.outputs.cache-primary-key }} + + # Waits for the pulls started earlier, so wp-env does not start a second download of an + # image that is still coming, and prints how long they took. Gives up after two minutes and + # lets wp-env carry on; never fails the job. + - name: Wait for the Docker image pulls + continue-on-error: true + timeout-minutes: 3 + run: | + for _ in $(seq 120); do + [ -f "$RUNNER_TEMP/docker-pull.done" ] && break + sleep 1 + done + cat "$RUNNER_TEMP/docker-pull.log" 2>/dev/null || true + + # Fails on any error, which is expected while a plugin still has known problems: the report + # job decides pass or fail from the counts. The action writes its results to + # $RUNNER_TEMP/plugin-check-results.txt and uploads them as the plugin-check-results + # artifact. Its `slug` input is passed for the target branch only: it renames that artifact to + # plugin-check-results-, so the two do not clash. + - name: Run Plugin Check + continue-on-error: true + uses: wordpress/plugin-check-action@v1 + with: + build-dir: ${{ github.workspace }}/build/${{ inputs.plugin-slug }} + exclude-checks: ${{ needs.plan.outputs.checks }} + # The target branch's results go into an artifact of their own. The slug is the folder the + # plugin is in anyway, so the checks see the same thing; only the artifact's name changes. + slug: ${{ matrix.target == 'base' && inputs.plugin-slug || '' }} + + report: + name: Plugin Check + needs: [start, plan, check] + # "start" is skipped for forks and non-pull-request runs, which would skip this job too. + if: ${{ !cancelled() }} + runs-on: ubuntu-latest + timeout-minutes: 10 + + # Writes one sticky comment on the pull request, and reads the repository's baseline file. The + # scripts come with the action. + permissions: + contents: read + issues: write + pull-requests: write + + steps: + # Only the baseline file, nothing else of the repository. It is JSON that the compare reads + # and never runs. The checkout is the pull request's merge commit, so a pull request that + # changes the file is compared with its own change, which a reviewer sees in the diff. + - name: Check out the baseline file + uses: actions/checkout@v7 + with: + path: source + sparse-checkout: .plugin-check-baseline.json + sparse-checkout-cone-mode: false + # The merge commit and its two parents, to see whether the pull request changes the file. + fetch-depth: 2 + persist-credentials: false + + # The comment says so when a pull request changes the baseline: it is compared with its own + # change, which a reviewer should look at. + - name: Check whether the pull request changes the baseline file + id: changed + if: ${{ github.event_name == 'pull_request' }} + working-directory: source + run: | + if git diff --quiet HEAD^1 HEAD -- .plugin-check-baseline.json; then + echo 'changed=false' >> "$GITHUB_OUTPUT" + else + echo 'changed=true' >> "$GITHUB_OUTPUT" + fi + + - name: Set file paths + run: | + { + echo "RESULTS=$RUNNER_TEMP/results/plugin-check-results.txt" + echo "BASE_RESULTS=$RUNNER_TEMP/base-results/plugin-check-results.txt" + echo "COMMENT_BODY=$RUNNER_TEMP/plugin-check-comment.md" + } >> "$GITHUB_ENV" + + # The action uploaded its results under this name. Missing when the build or the check broke + # before then: the compare reports that. + - name: Download the results + continue-on-error: true + uses: actions/download-artifact@v8 + with: + name: plugin-check-results + path: ${{ runner.temp }}/results + + # The target branch's results, when the plan said to check it. A missing artifact is not an + # error here: the compare fails with a message that says the target could not be checked. + - name: Download the target branch's results + if: ${{ needs.plan.outputs.base == 'true' }} + continue-on-error: true + uses: actions/download-artifact@v8 + with: + name: plugin-check-results-${{ inputs.plugin-slug }} + path: ${{ runner.temp }}/base-results + + # `$/` is this repository at the commit that runs this workflow, so the action is always the + # same revision as the workflow, whichever ref the caller used, and it needs no checkout. It + # needs a runner of version 2.336.0 or newer; this job runs on ubuntu-latest. Not ./, which + # in a called workflow resolves against the caller's repository. + # + # The action also writes the "Check Finished" badge into the body, and, when the build or the + # check broke and there was nothing to compare, a body that says Plugin Check could not run, + # so "in progress" never stays on the pull request. + - name: Compare with the baseline + uses: $/.github/actions/plugin-check-report + with: + results: ${{ env.RESULTS }} + comment-file: ${{ env.COMMENT_BODY }} + comment-marker: ${{ env.COMMENT_MARKER }} + mode: ${{ inputs.mode }} + baseline-file: ${{ github.workspace }}/source/.plugin-check-baseline.json + base-results: ${{ needs.plan.outputs.base == 'true' && env.BASE_RESULTS || '' }} + base-required: ${{ needs.plan.outputs.base == 'true' }} + baseline-changed: ${{ steps.changed.outputs.changed == 'true' }} + + - name: Post the result + if: ${{ !cancelled() && env.COMMENT == 'true' }} + continue-on-error: true + uses: $/.github/actions/pr-comment + with: + marker: ${{ env.COMMENT_MARKER }} + body-file: ${{ env.COMMENT_BODY }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 0450a2e..c0870c1 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -3,7 +3,7 @@ name: Tests # This repo's own checks. A bug here reaches every repo that pins a tag from it, so the logic in the # actions is unit tested rather than only exercised by a real release. # -# Run it locally: `npm install` once, then `node --test`. +# Run it locally: `npm install` once, then `node --test`. For the lint job, `actionlint`. on: push: @@ -40,3 +40,25 @@ jobs: # finds every *.test.js under the repository on all of them. - name: Run the unit tests run: node --test + + # The workflows and templates never run in this repository the way a plugin runs them, so a typo + # in one would only show up in someone else's CI. actionlint also runs shellcheck on every run: + # block it finds. + lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + with: + persist-credentials: false + + # Pinned, so a new actionlint rule cannot fail a pull request that did not change anything. + # Two things are ignored because GitHub accepts them and actionlint 1.7.12 does not know them + # yet: `queue` in `concurrency` (release-tags.yml uses it), and a `uses: $/path` reference to + # this repository (the reusable workflows use it for their own actions). Only that exact shape + # is ignored, so any other action reference that lacks a ref still fails. + - name: Run actionlint + run: | + bash <(curl -sSfL https://raw.githubusercontent.com/rhysd/actionlint/v1.7.12/scripts/download-actionlint.bash) 1.7.12 + ./actionlint -ignore 'unexpected key "queue" for "concurrency" section' \ + -ignore 'specifying action "\$/[^"]*" in invalid format because ref is missing' \ + .github/workflows/*.yml templates/workflows/*.yml diff --git a/.gitignore b/.gitignore index 04f3b70..ef9bf5d 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,6 @@ npm-shrinkwrap.json bun.lock bun.lockb yarn.lock + +# plugin-check-report.js fetch writes the baseline here when run by hand (see the plugin-check-report README). +.github/actions/plugin-check-report/baseline.json diff --git a/templates/workflows/plugin-check.yml b/templates/workflows/plugin-check.yml new file mode 100644 index 0000000..49d4194 --- /dev/null +++ b/templates/workflows/plugin-check.yml @@ -0,0 +1,43 @@ +name: Plugin Check + +# Runs WordPress.org's Plugin Check on the plugin as it ships, and compares the error counts with +# the baseline the repository commits, `.plugin-check-baseline.json`. +# +# A file to copy into .github/workflows/ of a plugin repository, under any name. The work is in +# stellarwp/plugin-toolbox/.github/workflows/plugin-check.yml. The repository needs a .puprc whose +# zip_name equals plugin-slug; pup itself is installed by the workflow. See the plugin-check-report +# action's README. +# +# Checks that do not fit the plugin go in a .plugin-check.json at the repository root, as +# {"exclude-checks": ["check_name"]}; or pass the `exclude-checks` input, which replaces the file. +# +# Run it locally: `wp plugin check --format=json > results.txt` in a local WordPress, then +# the compare command on that file. See "Running it locally" in the action's README. + +on: + pull_request: + # The branches pull requests target. A push there keeps them honest after a merge. + push: + branches: [main] + workflow_dispatch: + +# A new push to a pull request cancels the run still going for the older commit. +concurrency: + group: plugin-check-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + plugin-check: + # The most any job of the shared workflow asks for. A called workflow cannot go above this. + permissions: + contents: read # check out the repository + issues: write # the sticky pull request comment + pull-requests: write # the sticky pull request comment + uses: stellarwp/plugin-toolbox/.github/workflows/plugin-check.yml@main + with: + plugin-slug: my-plugin + # The pup release your release build uses. It builds the plugin for the check. + pup-version: '2.0.0' diff --git a/tests/actions/plugin-check-report/action.test.js b/tests/actions/plugin-check-report/action.test.js new file mode 100644 index 0000000..acdda74 --- /dev/null +++ b/tests/actions/plugin-check-report/action.test.js @@ -0,0 +1,155 @@ +'use strict' + +const { describe, it } = require('node:test') +const assert = require('node:assert') +const fs = require('node:fs') +const os = require('node:os') +const path = require('node:path') + +const report = require('../../../.github/actions/plugin-check-report/plugin-check-report.js') +const { actionsFor } = require('../../support/actions.js') + +const FIXTURES = path.join(__dirname, 'fixtures') +const REPO = 'example-org/my-plugin' + +const temp = (prefix) => fs.mkdtempSync(path.join(os.tmpdir(), `plugin-check-${prefix}-`)) + +describe('plugin-check-report action steps', () => { + describe('runPaths', () => { + it('makes a clean folder for the run', async (t) => { + const { core, outputs } = await actionsFor(t) + const tempDir = temp('paths') + fs.mkdirSync(path.join(tempDir, 'plugin-check-report', 'report'), { recursive: true }) + fs.writeFileSync(path.join(tempDir, 'plugin-check-report', 'report', 'old.txt'), 'from an earlier use') + + const status = report.runPaths({ + core, + env: { + RUNNER_TEMP: tempDir, + GITHUB_REPOSITORY: REPO, + }, + }) + + assert.equal(status, 0) + assert.deepEqual(outputs(), { + report: path.join(tempDir, 'plugin-check-report', 'report'), + marker: ``, + }) + assert.deepEqual(fs.readdirSync(path.join(tempDir, 'plugin-check-report', 'report')), []) + }) + + it('uses the marker the caller gives', async (t) => { + const { core, outputs } = await actionsFor(t) + + report.runPaths({ + core, + env: { RUNNER_TEMP: temp('marker'), GITHUB_REPOSITORY: REPO, INPUT_COMMENT_MARKER: '' }, + }) + + assert.equal(outputs().marker, '') + }) + }) + + describe('runUpdate', () => { + it('writes the baseline of this run next to a copy of the results', async (t) => { + const { core, logged } = await actionsFor(t) + const reportDir = temp('report') + + const status = report.runUpdate({ + core, + env: { RESULTS: path.join(FIXTURES, 'errors-up.txt'), REPORT_DIR: reportDir }, + }) + + assert.equal(status, 0) + assert.ok(fs.existsSync(path.join(reportDir, 'errors-up.txt'))) + assert.deepEqual(JSON.parse(fs.readFileSync(path.join(reportDir, 'baseline.json'), 'utf8')).warnings, { + 'WordPress.DB.DirectDatabaseQuery.DirectQuery': 1, + }) + assert.match(logged(), /Wrote .*baseline\.json: 2 error codes, 1 warning codes/) + }) + + it('does nothing when there is no results file', async (t) => { + const { core } = await actionsFor(t) + const reportDir = temp('report') + + assert.equal(report.runUpdate({ core, env: { RESULTS: '/nonexistent/results.txt', REPORT_DIR: reportDir } }), 0) + assert.deepEqual(fs.readdirSync(reportDir), []) + }) + + it('fails on a results file that is not Plugin Check output, as the compare does', async (t) => { + const { core, logged } = await actionsFor(t) + + const status = report.runUpdate({ + core, + env: { RESULTS: path.join(FIXTURES, 'garbage.txt'), REPORT_DIR: temp('report') }, + }) + + assert.equal(status, 1) + assert.match(logged(), /::error title=Plugin Check::The results file is not Plugin Check output/) + }) + }) + + describe('runCompare', () => { + const base = () => ({ + RESULTS: path.join(FIXTURES, 'errors-up.txt'), + PLUGIN_CHECK_BASELINE: path.join(FIXTURES, 'baseline.json'), + RUN_URL: 'https://example.test/run/1', + }) + + it('fails the step with the annotation, the comment and the same text in the job summary', async (t) => { + const { core, logged, summary } = await actionsFor(t) + const commentFile = path.join(temp('compare'), 'comment.md') + + const status = report.runCompare({ core, env: { ...base(), COMMENT_FILE: commentFile, GITHUB_STEP_SUMMARY: process.env.GITHUB_STEP_SUMMARY } }) + + assert.equal(status, 1) + assert.match(logged(), /::error title=Plugin Check::WordPress\.Security\.EscapeOutput\.OutputNotEscaped has 3 errors, the baseline has 2\./) + assert.match(logged(), /Plugin Check found more errors than the baseline allows\./) + const comment = fs.readFileSync(commentFile, 'utf8') + assert.match(comment, /❌ Plugin Check failed: 1 rule has more errors than the baseline allows/) + assert.ok(summary().startsWith(comment), 'the summary is the comment, and then how to update the baseline') + }) + + it('passes, and writes no comment file when none is asked for', async (t) => { + const { core, logged } = await actionsFor(t) + + const status = report.runCompare({ core, env: { ...base(), RESULTS: path.join(FIXTURES, 'same-as-baseline.txt'), COMMENT_FILE: '' } }) + + assert.equal(status, 0) + assert.match(logged(), /Plugin Check error counts are within the baseline\./) + }) + + it('fails when the results file is missing, with no comment', async (t) => { + const { core, logged } = await actionsFor(t) + const commentFile = path.join(temp('compare'), 'comment.md') + + const status = report.runCompare({ core, env: { ...base(), RESULTS: '/nonexistent/results.txt', COMMENT_FILE: commentFile } }) + + assert.equal(status, 1) + assert.match(logged(), /::error title=Plugin Check::Results file not found: \/nonexistent\/results\.txt/) + assert.equal(fs.existsSync(commentFile), false) + }) + + it('refuses the old mode name ratchet with a hint', async (t) => { + const { core, logged } = await actionsFor(t) + + const status = report.runCompare({ core, env: { ...base(), PLUGIN_CHECK_MODE: 'ratchet' } }) + + assert.equal(status, 1) + assert.match(logged(), /::error title=Plugin Check::Unknown mode 'ratchet'\. Use 'baseline' or 'strict'\. 'ratchet' is now called 'baseline'\./) + }) + + it('escapes a percent sign in a message, as the runner reads annotations', async (t) => { + const { core, logged } = await actionsFor(t) + const results = path.join(temp('compare'), 'results.txt') + fs.writeFileSync(results, 'FILE: a.php\n[{"type":"ERROR","code":"100%done"}]\n') + + report.runCompare({ core, env: { ...base(), RESULTS: results, PLUGIN_CHECK_BASELINE: '/nonexistent/baseline.json' } }) + + assert.match(logged(), /::warning title=Plugin Check::No baseline file at baseline\.json, so nothing was compared\./) + const failing = report.runCompare({ core, env: { ...base(), RESULTS: results, PLUGIN_CHECK_BASELINE: path.join(FIXTURES, 'baseline.json') } }) + assert.equal(failing, 1) + assert.match(logged(), /::error title=Plugin Check::100%25done has 1 errors/) + }) + }) +}) diff --git a/tests/actions/plugin-check-report/badge.test.js b/tests/actions/plugin-check-report/badge.test.js new file mode 100644 index 0000000..f58ca14 --- /dev/null +++ b/tests/actions/plugin-check-report/badge.test.js @@ -0,0 +1,162 @@ +'use strict' + +const { describe, it } = require('node:test') +const assert = require('node:assert') +const fs = require('node:fs') +const os = require('node:os') +const path = require('node:path') + +const report = require('../../../.github/actions/plugin-check-report/plugin-check-report.js') +const { actionsFor } = require('../../support/actions.js') + +const URL_START = 'https://img.shields.io/badge/Check%20Finished-' + +/** + * What the shell step this replaced printed for each moment, from the real `date` command: + * + * stamp=$(LC_ALL=C TZ='America/New_York' date -r '+%m/%d %I:%M %p %Z') + * stamp=${stamp// /%20}; stamp=${stamp//\//%2F} + * echo "https://img.shields.io/badge/Check%20Finished-${stamp}-darkgreen" + * + * The moments are the ones where a clock formatter goes wrong: both sides of both daylight saving + * changes, the repeated hour of the fall change, midnight, noon and the end of a year. + */ +const GOLDEN = [ + ['winter, noon-ish', '2026-01-15T17:08:00Z', '01%2F15%2012:08%20PM%20EST'], + ['summer', '2026-07-04T22:05:00Z', '07%2F04%2006:05%20PM%20EDT'], + ['just before spring forward', '2026-03-08T06:59:59Z', '03%2F08%2001:59%20AM%20EST'], + ['spring forward', '2026-03-08T07:00:00Z', '03%2F08%2003:00%20AM%20EDT'], + ['repeated hour, first pass', '2026-11-01T05:30:00Z', '11%2F01%2001:30%20AM%20EDT'], + ['just before fall back', '2026-11-01T05:59:59Z', '11%2F01%2001:59%20AM%20EDT'], + ['fall back', '2026-11-01T06:00:00Z', '11%2F01%2001:00%20AM%20EST'], + ['repeated hour, second pass', '2026-11-01T06:30:00Z', '11%2F01%2001:30%20AM%20EST'], + ['midnight', '2026-01-15T05:00:00Z', '01%2F15%2012:00%20AM%20EST'], + ['noon', '2026-01-15T17:00:00Z', '01%2F15%2012:00%20PM%20EST'], + ['the last minute of a year', '2027-01-01T04:59:00Z', '12%2F31%2011:59%20PM%20EST'], +] + +describe('finishBadge', () => { + for (const [name, moment, stamp] of GOLDEN) { + it(`makes the URL the shell step made: ${name}`, () => { + assert.equal(report.finishBadge(new Date(moment)), `${URL_START}${stamp}-darkgreen`) + }) + } + + it('uses America/New_York unless told otherwise', () => { + assert.equal(report.BADGE_TIME_ZONE, 'America/New_York') + assert.equal(report.finishBadge(new Date('2026-07-04T22:05:00Z')), report.finishBadge(new Date('2026-07-04T22:05:00Z'), 'America/New_York')) + }) + + it('reads the clock when no moment is given', () => { + assert.match(report.finishBadge(), /^https:\/\/img\.shields\.io\/badge\/Check%20Finished-\d\d%2F\d\d%20\d\d:\d\d%20[AP]M%20E[SD]T-darkgreen$/) + }) + + it('escapes only the spaces and the slash, as the shell did', () => { + const url = report.finishBadge(new Date('2026-07-04T22:05:00Z')) + + assert.doesNotMatch(url, / |\/\d/) + assert.match(url, /06:05%20PM/) + }) +}) + +describe('the badge in the comment', () => { + const FIXTURES = path.join(__dirname, 'fixtures') + const env = (extra = {}) => ({ + RESULTS: path.join(FIXTURES, 'same-as-baseline.txt'), + PLUGIN_CHECK_BASELINE: path.join(FIXTURES, 'baseline.json'), + RUN_URL: 'https://example.test/run/1', + ...extra, + }) + + it('shows the finish badge under the status line, whatever the caller passed', async (t) => { + const { core } = await actionsFor(t) + const commentFile = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-badge-')), 'comment.md') + + const status = report.runCompare({ + core, + env: env({ COMMENT_FILE: commentFile, COMMENT_BADGE: 'https://example.test/other.svg' }), + now: new Date('2026-07-04T22:05:00Z'), + }) + + assert.equal(status, 0) + const lines = fs.readFileSync(commentFile, 'utf8').split('\n') + assert.match(lines[1], /^✅ Plugin Check passed/) + assert.equal(lines[3], `![Plugin Check finished](${URL_START}07%2F04%2006:05%20PM%20EDT-darkgreen)`) + }) +}) + +describe('runCouldNotRun', () => { + const temp = () => fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-fallback-')) + const env = (file, extra = {}) => ({ + COMMENT_FILE: file, + COMMENT_MARKER: '', + RUN_URL: 'https://github.com/example-org/my-plugin/actions/runs/7', + ...extra, + }) + + it('writes the comment when the compare wrote none', async (t) => { + const { core } = await actionsFor(t) + const file = path.join(temp(), 'comment.md') + + const status = report.runCouldNotRun({ core, env: env(file), now: new Date('2026-07-04T22:05:00Z') }) + + assert.equal(status, 0) + assert.equal( + fs.readFileSync(file, 'utf8'), + [ + '', + '❌ Plugin Check could not run', + '', + `![Plugin Check finished](${URL_START}07%2F04%2006:05%20PM%20EDT-darkgreen)`, + '', + '📎 [Run](https://github.com/example-org/my-plugin/actions/runs/7)', + '', + ].join('\n') + ) + }) + + it('writes it over an empty file, as `[ ! -s file ]` did', async (t) => { + const { core } = await actionsFor(t) + const file = path.join(temp(), 'comment.md') + fs.writeFileSync(file, '') + + report.runCouldNotRun({ core, env: env(file) }) + + assert.match(fs.readFileSync(file, 'utf8'), /❌ Plugin Check could not run/) + }) + + it('leaves a comment the compare wrote', async (t) => { + const { core } = await actionsFor(t) + const file = path.join(temp(), 'comment.md') + fs.writeFileSync(file, '\n❌ Plugin Check failed\n') + + const status = report.runCouldNotRun({ core, env: env(file) }) + + assert.equal(status, 0) + assert.equal(fs.readFileSync(file, 'utf8'), '\n❌ Plugin Check failed\n') + }) + + it('does nothing when no comment is asked for', async (t) => { + const { core } = await actionsFor(t) + + assert.equal(report.runCouldNotRun({ core, env: env('') }), 0) + }) + + it('names the marker after the repository when none is given', async (t) => { + const { core } = await actionsFor(t) + const file = path.join(temp(), 'comment.md') + + report.runCouldNotRun({ core, env: env(file, { COMMENT_MARKER: '', GITHUB_REPOSITORY: 'example-org/my-plugin' }) }) + + assert.match(fs.readFileSync(file, 'utf8'), /^\n/) + }) + + it('fails the step, naming the file, when it cannot be written', async (t) => { + const { core, logged } = await actionsFor(t) + + const status = report.runCouldNotRun({ core, env: env('/nonexistent-folder/comment.md') }) + + assert.equal(status, 1) + assert.match(logged(), /::error title=Plugin Check::Could not write \/nonexistent-folder\/comment\.md/) + }) +}) diff --git a/tests/actions/plugin-check-report/compare.test.js b/tests/actions/plugin-check-report/compare.test.js new file mode 100644 index 0000000..5598cbf --- /dev/null +++ b/tests/actions/plugin-check-report/compare.test.js @@ -0,0 +1,823 @@ +'use strict' + +const { describe, it } = require('node:test') +const assert = require('node:assert') +const fs = require('node:fs') +const os = require('node:os') +const path = require('node:path') + +const report = require('../../../.github/actions/plugin-check-report/plugin-check-report.js') +const { CASES, expectedFor, runCase } = require('./golden.js') + +const FIXTURES = path.join(__dirname, 'fixtures') +const BASELINE = path.join(FIXTURES, 'baseline.json') + +const ESCAPE = 'WordPress.Security.EscapeOutput.OutputNotEscaped' +const DIRECT_QUERY = 'WordPress.DB.DirectDatabaseQuery.DirectQuery' + +/** + * Runs the compare in this process, with an environment built from scratch rather than copied from + * process.env. In CI the test runner has GITHUB_STEP_SUMMARY, GITHUB_RUN_ID and friends set, and the + * compare would append its summary to this repository's own job and link to this run. Messages go + * to captured lines, never to the runner's log, so a passing test run does not annotate itself. + * + * @param {string[]} args Arguments of the compare, usually a results file. + * @param {object} env The environment to add. + * + * @returns {{status: number, stdout: string, stderr: string}} The exit status and both streams. + */ +function execute(args, env) { + const stdout = [] + const stderr = [] + const log = report.commandLog({ out: (line) => stdout.push(`${line}\n`), err: (line) => stderr.push(`${line}\n`) }) + + return { status: report.compare({ args, env, log }), stdout: stdout.join(''), stderr: stderr.join('') } +} + +/** + * Runs the compare the way the action does, and returns what it printed and wrote. + * + * @param {string[]} args Arguments after `--comment-file`, usually a results file. + * @param {object} env Extra environment, such as PLUGIN_CHECK_BASELINE. + * + * @returns {{status: number, stdout: string, stderr: string, comment: string, summary: string}} + * The exit status, both streams, the comment file and the job summary it wrote. + */ +function compareWith(args, env = {}) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-compare-')) + const comment = path.join(dir, 'comment.md') + const summary = path.join(dir, 'summary.md') + + const result = execute(['--comment-file', comment, ...args], { + GITHUB_STEP_SUMMARY: summary, + // A path that does not exist, so a test that wants no baseline gets none, instead of a + // baseline.json some earlier local run left next to the script. + PLUGIN_CHECK_BASELINE: path.join(dir, 'no-baseline.json'), + ...env, + }) + + const read = (file) => (fs.existsSync(file) ? fs.readFileSync(file, 'utf8') : '') + + return { ...result, comment: read(comment), summary: read(summary) } +} + +/** + * The path of one of the sample results files. + * + * @param {string} name File name under fixtures/. + * + * @returns {string} The absolute path. + */ +function fixture(name) { + return path.join(FIXTURES, name) +} + +/** + * A results file with the lines given, in a temporary folder. + * + * @param {string} text The file's text. + * + * @returns {string} The path of the file. + */ +function resultsFile(text) { + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-results-')), 'results.txt') + fs.writeFileSync(file, text) + + return file +} + +/** + * A results file with the results given as one file's JSON line. + * + * @param {object[]} results The results. + * + * @returns {string} The path of the file. + */ +function resultsOf(results) { + return resultsFile(`FILE: plugin.php\n${JSON.stringify(results)}\n`) +} + +/** + * A results file with one new ERROR code per entry, for the tests about long lists. + * + * @param {number} codes How many different codes to write. + * + * @returns {string} The path of the file. + */ +function resultsWithNewCodes(codes) { + return resultsOf( + Array.from({ length: codes }, (_, index) => ({ + line: index + 1, + column: 1, + type: 'ERROR', + code: `Test.Rule.Number${index + 1}`, + message: 'A rule added for the test.', + docs: '', + })) + ) +} + +/** + * A baseline file with the text given, in a temporary folder. + * + * @param {string|object} content The file's text, or a value to write as JSON. + * + * @returns {string} The path of the file. + */ +function baselineFile(content) { + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-baseline-')), 'baseline.json') + fs.writeFileSync(file, typeof content === 'string' ? content : JSON.stringify(content)) + + return file +} + +describe('plugin-check-report compare, pinned output', () => { + for (const spec of CASES) { + it(`prints what the original script printed for ${spec.name}`, () => { + assert.deepStrictEqual(runCase(spec, execute), expectedFor(spec.name)) + }) + } +}) + +describe('plugin-check-report compare', () => { + describe('results it refuses', () => { + it('fails when the results file is missing', () => { + const run = compareWith([fixture('does-not-exist.txt')]) + + assert.equal(run.status, 1) + assert.match(run.stdout, /::error title=Plugin Check::Results file not found/) + assert.equal(run.comment, '', 'no comment body, so the workflow writes "could not run"') + }) + + it('fails when the results file is not Plugin Check output', () => { + const run = compareWith([fixture('garbage.txt')]) + + assert.equal(run.status, 1) + assert.match(run.stdout, /is not Plugin Check output.*First unexpected line: /) + }) + + it('fails when the results file is empty', () => { + const run = compareWith([fixture('empty.txt')]) + + assert.equal(run.status, 1) + assert.match(run.stdout, /has no Plugin Check output/) + }) + + it('fails on a mode it does not know, before reading anything', () => { + const run = compareWith([fixture('clean.txt')], { PLUGIN_CHECK_MODE: 'strcit' }) + + assert.equal(run.status, 1) + assert.match(run.stdout, /Unknown mode 'strcit'/) + }) + + it('refuses the old name ratchet, and says what to use', () => { + const run = compareWith([fixture('clean.txt')], { PLUGIN_CHECK_MODE: 'ratchet' }) + + assert.equal(run.status, 1) + assert.match( + run.stdout, + /::error title=Plugin Check::Unknown mode 'ratchet'\. Use 'baseline' or 'strict'\. 'ratchet' is now called 'baseline'\./ + ) + assert.equal(run.comment, '') + }) + + it('runs baseline mode when the mode is not set, and when it is empty', () => { + for (const env of [{}, { PLUGIN_CHECK_MODE: '' }, { PLUGIN_CHECK_MODE: 'baseline' }]) { + const run = compareWith([fixture('errors-up.txt')], { ...env, PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 1, JSON.stringify(env)) + assert.match(run.comment, /rule has more errors than the baseline allows/) + } + }) + + it('fails on a mode it does not know, even with a missing results file', () => { + const run = compareWith([fixture('does-not-exist.txt')], { PLUGIN_CHECK_MODE: 'strcit' }) + + assert.match(run.stdout, /Unknown mode/) + assert.doesNotMatch(run.stdout, /Results file not found/) + }) + + it('fails with a usage message when no results file is given', () => { + const run = execute([], {}) + + assert.equal(run.status, 1) + assert.match(run.stderr, /Usage/) + }) + + it('fails when --comment-file has no path', () => { + for (const args of [['--comment-file'], ['--comment-file', '']]) { + const run = execute(args, {}) + + assert.equal(run.status, 1, args.join(' ')) + assert.match(run.stderr, /--comment-file needs a path/) + } + }) + + it('cuts the first unexpected line to 200 bytes', () => { + const run = compareWith([resultsFile(`${'x'.repeat(300)}\n`)]) + + assert.equal(run.status, 1) + assert.match(run.stdout, new RegExp(`First unexpected line: x{200}\\n`)) + }) + + it('does not count a line of white space as unexpected', () => { + const run = compareWith([resultsFile('FILE: plugin.php\n \t \n[]\n\n')]) + + assert.equal(run.status, 0) + }) + + it('fails when a results line is not JSON', () => { + const run = compareWith([resultsFile('FILE: plugin.php\n[{"type":"ERROR"\n')]) + + assert.equal(run.status, 5) + assert.match(run.stderr, /parse error/) + assert.equal(run.comment, '') + }) + + it('fails when a counted result has no rule code', () => { + for (const code of [undefined, null, 5, true, { a: 1 }]) { + const run = compareWith([resultsOf([{ type: 'ERROR', code }])]) + + assert.equal(run.status, 5, String(code)) + assert.match(run.stderr, /has no rule code/) + } + }) + + it('fails when a result is not an object, but skips null', () => { + for (const item of [1, 'x', [1], true]) { + assert.equal(compareWith([resultsOf([item])]).status, 5, JSON.stringify(item)) + } + + assert.equal(compareWith([resultsOf([null])]).status, 0) + }) + + it('ignores results that are neither ERROR nor WARNING', () => { + const run = compareWith([resultsOf([{ type: 'INFO' }, { type: 'error', code: 'a' }])]) + + assert.equal(run.status, 0) + assert.match(run.comment, /\| Errors {3}\| 0 \|/) + }) + + it('reads several lists written on one line', () => { + const run = compareWith([resultsFile('FILE: a.php\n[{"type":"ERROR","code":"a"}] [{"type":"ERROR","code":"b"}]\n')], { + PLUGIN_CHECK_MODE: 'strict', + }) + + assert.match(run.stdout, /2 errors found/) + }) + }) + + describe('--update', () => { + it('writes the counts per rule code', () => { + const out = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-update-')), 'baseline.json') + const run = compareWith(['--update', fixture('errors-up.txt')], { PLUGIN_CHECK_BASELINE: out }) + + assert.equal(run.status, 0) + assert.deepEqual(JSON.parse(fs.readFileSync(out, 'utf8')), { + errors: { + [ESCAPE]: 3, + 'WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents': 1, + }, + warnings: { [DIRECT_QUERY]: 1 }, + }) + }) + + it('writes empty counts for a clean run', () => { + const out = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-update-')), 'baseline.json') + const run = compareWith(['--update', fixture('clean.txt')], { PLUGIN_CHECK_BASELINE: out }) + + assert.equal(run.status, 0) + assert.deepEqual(JSON.parse(fs.readFileSync(out, 'utf8')), { errors: {}, warnings: {} }) + }) + + it('writes no comment and no summary', () => { + const out = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-update-')), 'baseline.json') + const run = compareWith(['--update', fixture('errors-up.txt')], { PLUGIN_CHECK_BASELINE: out }) + + assert.equal(run.comment, '') + assert.equal(run.summary, '') + }) + + it('fails, with the reason, when the folder of the file does not exist', () => { + const run = compareWith(['--update', fixture('clean.txt')], { PLUGIN_CHECK_BASELINE: '/nonexistent/folder/baseline.json' }) + + assert.equal(run.status, 1) + assert.match(run.stderr, /Could not write \/nonexistent\/folder\/baseline\.json/) + }) + }) + + describe('compare in baseline mode', () => { + it('passes a clean run with no baseline, with a warning and the one-column table', () => { + const run = compareWith([fixture('clean.txt')]) + + assert.equal(run.status, 0) + assert.match( + run.stdout, + /::warning title=Plugin Check::No baseline file at no-baseline\.json, so nothing was compared\. To start one, commit the baseline\.json from this run's plugin-check-report artifact as no-baseline\.json\./ + ) + assert.match(run.comment, /⚠️ Plugin Check ran, but there is no baseline file yet/) + assert.match(run.comment, /\| Errors {3}\| 0 \|/) + assert.doesNotMatch(run.comment, /Change/) + }) + + it('says nothing about a missing baseline file in strict mode, which does not use one', () => { + const run = compareWith([fixture('clean.txt')], { PLUGIN_CHECK_MODE: 'strict' }) + + assert.equal(run.status, 0) + assert.doesNotMatch(run.stdout, /::warning/) + assert.match(run.stdout, /No baseline file\. Strict mode does not need one\./) + assert.match(run.comment, /✅ Plugin Check passed: no errors/) + assert.equal(run.summary, run.comment, 'nothing to update, so the summary is the comment') + }) + + it('asks for the baseline file to be lowered when an error count went down', () => { + const run = compareWith([fixture('errors-down.txt')], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 0) + assert.match( + run.stdout, + /::notice title=Plugin Check::Error counts went down\. Lower the baseline so they cannot come back: commit the baseline\.json from this run's plugin-check-report artifact as baseline\.json\./ + ) + assert.match(run.comment, /Rules that went down: \d+\. Update the baseline file, see the job summary\./) + }) + + it('does not ask for an update when only warnings went down, or in strict mode', () => { + const warningsDown = baselineFile({ ...JSON.parse(fs.readFileSync(BASELINE, 'utf8')), warnings: { [DIRECT_QUERY]: 9 } }) + const quiet = compareWith([fixture('same-as-baseline.txt')], { PLUGIN_CHECK_BASELINE: warningsDown }) + + assert.match(quiet.comment, /Rules that went down: 1\.\n/) + assert.doesNotMatch(quiet.stdout, /Error counts went down/) + assert.equal(quiet.summary, quiet.comment) + + const strict = compareWith([fixture('errors-down.txt')], { PLUGIN_CHECK_BASELINE: BASELINE, PLUGIN_CHECK_MODE: 'strict' }) + + assert.doesNotMatch(strict.stdout, /Error counts went down/) + assert.doesNotMatch(strict.comment, /Update the baseline file/) + }) + + it('puts the command to update the baseline in the job summary, not in the comment', () => { + const run = compareWith([fixture('errors-down.txt')], { + PLUGIN_CHECK_BASELINE: BASELINE, + GITHUB_RUN_ID: '42', + GITHUB_REPOSITORY: 'example-org/my-plugin', + GITHUB_SERVER_URL: 'https://github.com', + }) + + assert.ok(run.summary.startsWith(run.comment)) + assert.match(run.summary, /Error counts went down\. To lower the baseline, replace baseline\.json with the file from this run:/) + assert.match(run.summary, /gh run download 42 --repo example-org\/my-plugin --name plugin-check-report --dir \/tmp\/plugin-check/) + assert.match(run.summary, /cp \/tmp\/plugin-check\/baseline\.json baseline\.json/) + assert.doesNotMatch(run.comment, /gh run download/) + }) + + it('tells the author how to raise a count on purpose when the check fails', () => { + const run = compareWith([fixture('errors-up.txt')], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 1) + assert.match(run.summary, /raise their counts in baseline\.json in this pull request, so reviewers see it/) + assert.match(run.summary, /gh run download --repo \/ /, 'placeholders outside a run') + }) + + it('fails with the file named, as an error annotation, when the baseline file is not valid', () => { + const cases = { + 'not JSON': ['{"errors": ', /it is not JSON: /], + 'a merge conflict': ['<<<<<<< HEAD\n{"errors":{}}\n=======\n{"errors":{"a":1}}\n>>>>>>> branch\n', /it is not JSON: /], + 'the wrong shape': ['[1]', /it must be an object with "errors" and "warnings"\./], + } + + for (const [what, [content, reason]] of Object.entries(cases)) { + const run = compareWith([fixture('clean.txt')], { PLUGIN_CHECK_BASELINE: baselineFile(content) }) + + assert.notEqual(run.status, 0, what) + assert.match(run.stdout, /::error title=Plugin Check::The baseline file baseline\.json is not valid: /, what) + assert.match(run.stdout, reason, what) + assert.match(run.stdout, /[.!?] Fix it, or replace it with the baseline\.json from this run's plugin-check-report artifact\./, what) + assert.equal(run.comment, '', what) + } + }) + + it('passes when every count matches the baseline', () => { + const run = compareWith([fixture('same-as-baseline.txt')], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 0) + assert.match(run.comment, /✅ Plugin Check passed/) + assert.match(run.comment, /\| Errors {3}\| 3 \| 3 \| ±0 \|/) + }) + + it('fails when a rule has more errors than the baseline', () => { + const run = compareWith([fixture('errors-up.txt')], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 1) + assert.match(run.stdout, new RegExp(`::error title=Plugin Check::${ESCAPE} has 3 errors, the baseline has 2`)) + assert.match(run.comment, /❌ Plugin Check failed: 1 rule has more errors than the baseline allows/) + assert.match(run.comment, new RegExp(`Went up: \`${ESCAPE}\` \\(\\+1\\)`)) + }) + + it('fails when a rule goes up even though the total stays the same', () => { + const run = compareWith([fixture('rule-up-total-same.txt')], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 1) + assert.match(run.comment, /\| Errors {3}\| 3 \| 3 \| ±0 \|/) + assert.match(run.comment, /Rules that went down: 1\./) + }) + + it('passes with a notice when errors go down', () => { + const run = compareWith([fixture('errors-down.txt')], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 0) + assert.match(run.stdout, new RegExp(`::notice title=Plugin Check::${ESCAPE} \\(errors\\) went down from 2 to 1`)) + assert.match(run.comment, /\| Errors {3}\| 2 \| 3 \| -1 \|/) + }) + + it('passes with a notice when only warnings go up', () => { + const run = compareWith([fixture('warnings-up.txt')], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 0) + assert.match(run.stdout, /Warnings do not fail the check/) + assert.match(run.comment, new RegExp(`Went up: \`${DIRECT_QUERY}\` warning \\(\\+2\\)`)) + }) + + it('treats an empty baseline file, and null, as no counts at all', () => { + for (const content of ['', 'null', ' \n']) { + const run = compareWith([fixture('errors-up.txt')], { PLUGIN_CHECK_BASELINE: baselineFile(content) }) + + assert.equal(run.status, 1, JSON.stringify(content)) + assert.match(run.stdout, /has 3 errors, the baseline has 0/) + } + }) + + it('treats a missing kind, and a null or false count, as zero', () => { + const run = compareWith([fixture('errors-up.txt')], { + PLUGIN_CHECK_BASELINE: baselineFile({ errors: { [ESCAPE]: null } }), + }) + + assert.match(run.stdout, /has 3 errors, the baseline has 0/) + }) + + it('reads only the first value of a baseline file', () => { + const run = compareWith([fixture('same-as-baseline.txt')], { + PLUGIN_CHECK_BASELINE: baselineFile(`${fs.readFileSync(BASELINE, 'utf8')}\n{"errors":{"x":9}}`), + }) + + assert.equal(run.status, 0) + }) + + it('fails, with a reason, on a baseline that cannot be compared', () => { + const cases = [ + ['not json', 2], + ['[]', 5], + ['5', 5], + ['{"errors":5}', 5], + ['{"errors":[]}', 5], + ['{"errors":false}', 5], + ['{"errors":{"a":"2"}}', 5], + ['{"errors":{"a":true}}', 5], + ['{"errors":{"a":1.5}}', 1], + ] + + for (const [content, status] of cases) { + const run = compareWith([resultsOf([{ type: 'ERROR', code: 'a' }])], { PLUGIN_CHECK_BASELINE: baselineFile(content) }) + + assert.equal(run.status, status, content) + assert.equal(run.comment, '', content) + } + }) + + it('fails on a count too large to compare exactly, instead of printing another number', () => { + for (const content of ['{"errors":{"a":9007199254740993}}', '{"warnings":{"a":1e300}}', '{"errors":{"a":-9007199254740993}}']) { + const run = compareWith([resultsOf([{ type: 'ERROR', code: 'a' }])], { PLUGIN_CHECK_BASELINE: baselineFile(content) }) + + assert.equal(run.status, 5, content) + assert.match(run.stdout, /::error title=Plugin Check::The baseline file baseline\.json is not valid: .*more than 9007199254740991, which cannot be compared exactly/) + assert.equal(run.comment, '', content) + } + + const largest = compareWith([resultsOf([{ type: 'ERROR', code: 'a' }])], { + PLUGIN_CHECK_BASELINE: baselineFile('{"errors":{"a":9007199254740991}}'), + }) + assert.equal(largest.status, 0, 'the largest exact number still works') + assert.match(largest.comment, /\| 1 \| 9007199254740991 \| -9007199254740990 \|/) + }) + + it('counts a rule with an empty code like any other', () => { + const run = compareWith([resultsOf([{ type: 'ERROR', code: '' }, { type: 'ERROR', code: '' }])], { + PLUGIN_CHECK_BASELINE: baselineFile({ errors: { '': 1 }, warnings: {} }), + }) + + assert.equal(run.status, 1) + assert.match(run.comment, /\| Errors {3}\| 2 \| 1 \| \+1 \|/) + }) + + it('shows a rule code on one line, with a tab or a new line written out', () => { + const run = compareWith([resultsOf([{ type: 'ERROR', code: 'a\tb\\c' }])], { PLUGIN_CHECK_BASELINE: baselineFile({}) }) + + assert.match(run.stdout, /::error title=Plugin Check::a\\tb\\\\c has 1 errors/) + }) + + it('takes the links from GITHUB_RUN_ID when RUN_URL is not set', () => { + const run = compareWith([fixture('clean.txt')], { + GITHUB_RUN_ID: '42', + GITHUB_SERVER_URL: 'https://github.com', + GITHUB_REPOSITORY: 'example-org/my-plugin', + }) + + assert.match(run.comment, /📎 \[Run\]\(https:\/\/github\.com\/example-org\/my-plugin\/actions\/runs\/42\)/) + }) + + it('shows the results file for a run on a developer machine', () => { + const run = compareWith([fixture('clean.txt')]) + + assert.match(run.comment, /📎 Local run, see .*clean\.txt/) + }) + }) + + describe('compare in strict mode', () => { + it('fails on any error, even with no baseline', () => { + const run = compareWith([fixture('errors-down.txt')], { PLUGIN_CHECK_MODE: 'strict' }) + + assert.equal(run.status, 1) + assert.match(run.stdout, /2 errors found\. Strict mode fails on any error/) + assert.match(run.comment, /❌ Plugin Check failed: 2 errors, and strict mode allows none/) + }) + + it('fails on errors that went down, which baseline mode would pass', () => { + const run = compareWith([fixture('errors-down.txt')], { + PLUGIN_CHECK_MODE: 'strict', + PLUGIN_CHECK_BASELINE: BASELINE, + }) + + assert.equal(run.status, 1) + }) + + it('passes a clean run', () => { + const run = compareWith([fixture('clean.txt')], { PLUGIN_CHECK_MODE: 'strict' }) + + assert.equal(run.status, 0) + assert.match(run.comment, /✅ Plugin Check passed: no errors/) + }) + + it('passes a run with warnings only', () => { + const lines = fs + .readFileSync(fixture('warnings-up.txt'), 'utf8') + .split('\n') + .map((line) => { + if (!line.startsWith('[')) { + return line + } + + return JSON.stringify(JSON.parse(line).filter((result) => result.type === 'WARNING')) + }) + + const run = compareWith([resultsFile(lines.join('\n'))], { PLUGIN_CHECK_MODE: 'strict' }) + + assert.equal(run.status, 0) + }) + + it('prints the per-rule messages before the strict one', () => { + const run = compareWith([fixture('errors-up.txt')], { PLUGIN_CHECK_MODE: 'strict', PLUGIN_CHECK_BASELINE: BASELINE }) + const lines = run.stdout.split('\n') + + assert.match(lines[0], /has 3 errors, the baseline has 2/) + assert.match(lines[1], /4 errors found\. Strict mode/) + }) + }) + + describe('the comment', () => { + it('starts with the default marker', () => { + const run = compareWith([fixture('clean.txt')]) + + assert.equal(run.comment.split('\n')[0], '') + }) + + it('starts with the marker the workflow passes', () => { + const marker = '' + const run = compareWith([fixture('clean.txt')], { COMMENT_MARKER: marker }) + + assert.equal(run.comment.split('\n')[0], marker) + }) + + it('lists at most five rules and never grows past fifteen lines', () => { + const run = compareWith([resultsWithNewCodes(7)], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.status, 1) + assert.match(run.comment, /Went up \(top 5 of 7\)/) + assert.doesNotMatch(run.comment, /Number6/) + assert.ok(run.comment.trimEnd().split('\n').length <= 15) + }) + + it('cuts a comment to fifteen lines when a value holds line breaks, and drops the empty end', () => { + const marker = `${'\n'.repeat(20)}end` + const run = compareWith([fixture('clean.txt')], { COMMENT_MARKER: marker }) + + // The first fifteen lines are all empty, so nothing is left once the empty end is dropped. + assert.equal(run.comment, '\n') + }) + + it('writes the same text to the job summary', () => { + const run = compareWith([fixture('same-as-baseline.txt')], { PLUGIN_CHECK_BASELINE: BASELINE }) + + assert.equal(run.summary, run.comment) + }) + + it('writes a badge under the status line', () => { + const run = compareWith([fixture('clean.txt')], { COMMENT_BADGE: 'https://example.test/badge.svg' }) + + assert.match(run.comment, /\n!\[Plugin Check finished\]\(https:\/\/example\.test\/badge\.svg\)\n/) + }) + + it('fails, with the reason, when the comment file cannot be written', () => { + const run = execute(['--comment-file', '/nonexistent/folder/comment.md', fixture('clean.txt')], { RUN_URL: 'u' }) + + assert.equal(run.status, 1) + assert.match(run.stderr, /Could not write \/nonexistent\/folder\/comment\.md/) + }) + }) +}) + +describe('compare with no baseline file, in a pull request', () => { + // The workflow checked the target branch's commit in the same run, and passes its results. + const baseResults = () => fixture('same-as-baseline.txt') + + it('compares with the target branch and says so', () => { + const run = compareWith([fixture('errors-up.txt')], { PLUGIN_CHECK_BASE_RESULTS: baseResults(), PLUGIN_CHECK_BASE_REQUIRED: 'true' }) + + assert.equal(run.status, 1) + assert.match( + run.stdout, + /::warning title=Plugin Check::No baseline file at no-baseline\.json, so the target branch was checked too, and its results are the baseline for this run\. That makes the run slower\. To stop it, commit the baseline\.json from this run's plugin-check-report artifact as no-baseline\.json\./ + ) + assert.match(run.stdout, new RegExp(`::error title=Plugin Check::${ESCAPE} has 3 errors, the baseline has 2`)) + assert.match(run.comment, /\| Now \| Target branch \| Change \|/) + assert.match(run.comment, /ℹ️ There is no baseline file, so the baseline is the target branch's results\. See the job summary to add one\./) + assert.match(run.summary, /There is no no-baseline\.json yet\. To start one, add the file from this run:/) + }) + + it('passes when the pull request has no more errors than the target branch', () => { + const run = compareWith([fixture('same-as-baseline.txt')], { PLUGIN_CHECK_BASE_RESULTS: baseResults(), PLUGIN_CHECK_BASE_REQUIRED: 'true' }) + + assert.equal(run.status, 0) + assert.match(run.comment, /✅ Plugin Check passed/) + assert.match(run.comment, /\| Errors {3}\| 3 \| 3 \| ±0 \|/) + }) + + it('does not ask to lower a baseline file that does not exist', () => { + const run = compareWith([fixture('errors-down.txt')], { PLUGIN_CHECK_BASE_RESULTS: baseResults(), PLUGIN_CHECK_BASE_REQUIRED: 'true' }) + + assert.equal(run.status, 0) + assert.match(run.stdout, /went down from/, 'the per-rule notices stay') + assert.doesNotMatch(run.stdout, /Error counts went down/) + assert.doesNotMatch(run.comment, /Update the baseline file/) + }) + + it('keeps the comment within fifteen lines, links last, with the note', () => { + const run = compareWith([resultsWithNewCodes(7)], { PLUGIN_CHECK_BASE_RESULTS: baseResults(), PLUGIN_CHECK_BASE_REQUIRED: 'true' }) + const lines = run.comment.replace(/\n$/, '').split('\n') + + assert.ok(lines.length <= 15, `${lines.length} lines`) + assert.match(lines.at(-1), /^📎 /) + assert.ok(lines.some((line) => line.startsWith('ℹ️ There is no baseline file'))) + }) + + it('fails, never passes, when the target branch could not be checked', () => { + for (const env of [{}, { PLUGIN_CHECK_BASE_RESULTS: '/nonexistent/base-results.txt' }]) { + const run = compareWith([fixture('clean.txt')], { PLUGIN_CHECK_BASE_REQUIRED: 'true', ...env }) + + assert.equal(run.status, 1) + assert.match(run.stdout, /::error title=Plugin Check::The target branch could not be checked, so there is nothing to compare with\./) + assert.equal(run.comment, '', 'no comment body, so the workflow writes "could not run"') + } + }) + + it('fails when the results of the target branch are not Plugin Check output', () => { + const run = compareWith([fixture('clean.txt')], { PLUGIN_CHECK_BASE_RESULTS: fixture('garbage.txt'), PLUGIN_CHECK_BASE_REQUIRED: 'true' }) + + assert.equal(run.status, 1) + assert.match(run.stdout, /::error title=Plugin Check::The results of the target branch cannot be used\. The results file is not Plugin Check output/) + }) + + it('uses the baseline file when there is one, and ignores the target branch', () => { + const run = compareWith([fixture('errors-up.txt')], { + PLUGIN_CHECK_BASELINE: BASELINE, + PLUGIN_CHECK_BASE_RESULTS: fixture('garbage.txt'), + PLUGIN_CHECK_BASE_REQUIRED: 'true', + }) + + assert.equal(run.status, 1) + assert.match(run.comment, /\| Now \| Baseline \| Change \|/) + assert.doesNotMatch(run.stdout, /target branch was checked/) + }) + + it('leaves strict mode alone, which needs no baseline', () => { + const run = compareWith([fixture('clean.txt')], { + PLUGIN_CHECK_MODE: 'strict', + PLUGIN_CHECK_BASE_RESULTS: fixture('garbage.txt'), + PLUGIN_CHECK_BASE_REQUIRED: 'true', + }) + + assert.equal(run.status, 0) + assert.match(run.stdout, /No baseline file\. Strict mode does not need one\./) + assert.doesNotMatch(run.stdout, /target branch/) + }) +}) + +describe('the note about a pull request that changes the baseline file', () => { + it('says so under the table, and keeps the comment within fifteen lines', () => { + const run = compareWith([resultsWithNewCodes(7)], { PLUGIN_CHECK_BASELINE: BASELINE, PLUGIN_CHECK_BASELINE_CHANGED: 'true' }) + const lines = run.comment.replace(/\n$/, '').split('\n') + + assert.ok(lines.includes('ℹ️ This pull request changes the baseline file.')) + assert.ok(lines.length <= 15, `${lines.length} lines`) + assert.match(lines.at(-1), /^📎 /) + }) + + it('says nothing when the file is not changed, or does not exist', () => { + const same = compareWith([fixture('same-as-baseline.txt')], { PLUGIN_CHECK_BASELINE: BASELINE, PLUGIN_CHECK_BASELINE_CHANGED: 'false' }) + const none = compareWith([fixture('clean.txt')], { PLUGIN_CHECK_BASELINE_CHANGED: 'true' }) + + assert.doesNotMatch(same.comment, /changes the baseline file/) + assert.doesNotMatch(none.comment, /changes the baseline file/) + }) +}) + +describe('the baseline file text', () => { + it('sorts codes by Unicode code point, not by UTF-16 unit', () => { + const counts = report.countPerCode([ + { type: 'ERROR', code: '😀' }, + { type: 'ERROR', code: '￿' }, + { type: 'ERROR', code: '10' }, + { type: 'ERROR', code: '9' }, + ]) + + assert.deepEqual([...counts.errors.keys()], ['10', '9', '￿', '😀']) + }) + + it('writes DEL as \\u007f and ends with one new line', () => { + const text = report.formatBaseline(report.countPerCode([{ type: 'ERROR', code: 'a\x7fb' }])) + + assert.equal(text, '{\n "errors": {\n "a\\u007fb": 1\n },\n "warnings": {}\n}\n') + }) + + it('keeps a code such as __proto__ as a code', () => { + const counts = report.countPerCode([{ type: 'WARNING', code: '__proto__' }]) + + assert.equal(report.formatBaseline(counts), '{\n "errors": {},\n "warnings": {\n "__proto__": 1\n }\n}\n') + }) +}) + +describe('the compare command line', () => { + /** + * Runs the command line in a new folder, so the default baseline path lands there. + * + * @param {string[]} argv The arguments after the script name. + * + * @returns {Promise<{status: number, lines: string[], dir: string}>} The status, what it printed and the folder. + */ + async function inNewFolder(argv) { + const dir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-cli-'))) + const before = process.cwd() + const lines = [] + + process.chdir(dir) + try { + const status = await report.main(argv, {}, { out: (line) => lines.push(line), err: (line) => lines.push(line) }) + + return { status, lines, dir } + } finally { + process.chdir(before) + } + } + + const pinned = () => fs.readFileSync(path.join(FIXTURES, 'expected', 'update-errors-up.json'), 'utf8') + + it('writes .plugin-check-baseline.json in the current folder by default', async () => { + const { status, lines, dir } = await inNewFolder(['compare', '--update', fixture('errors-up.txt')]) + + assert.equal(status, 0) + assert.equal(fs.readFileSync(path.join(dir, '.plugin-check-baseline.json'), 'utf8'), pinned()) + assert.match(lines.join('\n'), /Wrote .*\.plugin-check-baseline\.json: 2 error codes, 1 warning codes/) + }) + + it('takes `update` as a short name for `compare --update`', async () => { + const { status, dir } = await inNewFolder(['update', fixture('errors-up.txt')]) + + assert.equal(status, 0) + assert.equal(fs.readFileSync(path.join(dir, '.plugin-check-baseline.json'), 'utf8'), pinned()) + }) + + it('compares with the file it wrote', async () => { + const dir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-cli-'))) + fs.writeFileSync(path.join(dir, '.plugin-check-baseline.json'), pinned()) + const before = process.cwd() + const lines = [] + + process.chdir(dir) + try { + const status = await report.main(['compare', fixture('errors-up.txt')], {}, { out: (line) => lines.push(line), err: (line) => lines.push(line) }) + + assert.equal(status, 0) + assert.match(lines.join('\n'), /within the baseline/) + } finally { + process.chdir(before) + } + }) +}) diff --git a/tests/actions/plugin-check-report/exclude.test.js b/tests/actions/plugin-check-report/exclude.test.js new file mode 100644 index 0000000..70601e6 --- /dev/null +++ b/tests/actions/plugin-check-report/exclude.test.js @@ -0,0 +1,241 @@ +'use strict' + +const { describe, it } = require('node:test') +const assert = require('node:assert') +const fs = require('node:fs') +const os = require('node:os') +const path = require('node:path') + +const report = require('../../../.github/actions/plugin-check-report/plugin-check-report.js') +const { actionsFor } = require('../../support/actions.js') + +const temp = () => fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-exclude-')) + +/** + * A repository root holding a config file. + * + * @param {string} text What the file holds. + * + * @returns {string} The path of the file. + */ +function configFile(text) { + const file = path.join(temp(), '.plugin-check.json') + fs.writeFileSync(file, text) + + return file +} + +describe('parseExcludeInput', () => { + it('reads one check per line', () => { + assert.deepEqual(report.parseExcludeInput('late_escaping\ni18n_usage\n'), ['late_escaping', 'i18n_usage']) + }) + + it('trims, drops blank lines and repeats, and reads Windows line breaks', () => { + assert.deepEqual(report.parseExcludeInput(' a \r\n\r\n\n b\na\n'), ['a', 'b']) + }) + + it('reads a comma list too, which the Plugin Check action turns its lines into', () => { + assert.deepEqual(report.parseExcludeInput('a,b\nc'), ['a', 'b', 'c']) + }) + + it('is empty for an empty or blank input', () => { + assert.deepEqual(report.parseExcludeInput(''), []) + assert.deepEqual(report.parseExcludeInput(' \n\t\n'), []) + }) +}) + +describe('parseExcludeFile', () => { + it('reads the list', () => { + assert.deepEqual(report.parseExcludeFile('{"exclude-checks": ["late_escaping", "i18n_usage"]}'), ['late_escaping', 'i18n_usage']) + }) + + it('trims entries and drops repeats, keeping the first order', () => { + assert.deepEqual(report.parseExcludeFile('{"exclude-checks": [" b ", "a", "b"]}'), ['b', 'a']) + }) + + it('means none when the key is missing or the list is empty', () => { + assert.deepEqual(report.parseExcludeFile('{}'), []) + assert.deepEqual(report.parseExcludeFile('{"exclude-checks": []}'), []) + }) + + it('ignores other keys, so the file can grow', () => { + assert.deepEqual(report.parseExcludeFile('{"mode": "strict", "exclude-checks": ["a"]}'), ['a']) + }) + + const refused = [ + ['text that is not JSON', 'late_escaping', /^\.plugin-check\.json is not valid JSON: /], + ['an empty file', '', /^\.plugin-check\.json is not valid JSON: /], + ['a list at the top', '["a"]', /^\.plugin-check\.json must hold a JSON object, like \{"exclude-checks": \["check_name"\]\}\.$/], + ['null', 'null', /must hold a JSON object/], + ['a number', '3', /must hold a JSON object/], + ['a text instead of a list', '{"exclude-checks": "a,b"}', /^"exclude-checks" in \.plugin-check\.json must be a list of check names, like \["check_name"\]\.$/], + ['null instead of a list', '{"exclude-checks": null}', /must be a list of check names/], + ['a number in the list', '{"exclude-checks": [1]}', /has 1, which is not a check name/], + ['an empty entry', '{"exclude-checks": ["a", ""]}', /has "", which is not a check name/], + ['a blank entry', '{"exclude-checks": [" "]}', /has " ", which is not a check name/], + ['a space inside an entry', '{"exclude-checks": ["late escaping"]}', /has "late escaping", which is not a check name/], + ['a comma inside an entry', '{"exclude-checks": ["a,b"]}', /has "a,b", which is not a check name/], + ['a line break inside an entry', '{"exclude-checks": ["a\\nb"]}', /which is not a check name/], + ['a nested list', '{"exclude-checks": [["a"]]}', /which is not a check name/], + ] + for (const [name, text, message] of refused) { + it(`fails on ${name}`, () => { + assert.throws( + () => report.parseExcludeFile(text), + (error) => error instanceof report.CheckFailure && error.exitCode === 1 && message.test(error.message) + ) + }) + } + + it('names the file in the message', () => { + assert.throws( + () => report.parseExcludeFile('{', 'checks.json'), + (error) => error.message.startsWith('checks.json is not valid JSON: ') + ) + }) +}) + +describe('excludedChecks', () => { + it('uses the input when it names checks, and does not read the file', () => { + const result = report.excludedChecks({ input: 'a\nb', fileText: 'this is not JSON' }) + + assert.deepEqual(result, { checks: ['a', 'b'], source: 'input' }) + }) + + it('uses the file when the input is empty or blank', () => { + const fileText = '{"exclude-checks": ["from_file"]}' + + assert.deepEqual(report.excludedChecks({ input: '', fileText }), { checks: ['from_file'], source: 'file' }) + assert.deepEqual(report.excludedChecks({ input: ' \n', fileText }), { checks: ['from_file'], source: 'file' }) + }) + + it('excludes nothing with neither', () => { + assert.deepEqual(report.excludedChecks({}), { checks: [], source: 'none' }) + assert.deepEqual(report.excludedChecks({ input: '', fileText: null }), { checks: [], source: 'none' }) + }) + + it('says none for a file with an empty list', () => { + assert.deepEqual(report.excludedChecks({ fileText: '{"exclude-checks": []}' }), { checks: [], source: 'none' }) + }) + + it('fails on a bad file when the input is empty', () => { + assert.throws(() => report.excludedChecks({ input: '', fileText: '{' }), /is not valid JSON/) + }) +}) + +describe('runExcludeChecks', () => { + it('outputs the file\'s checks one per line', async (t) => { + const { core, outputs, logged } = await actionsFor(t) + const file = configFile('{"exclude-checks": ["late_escaping", "i18n_usage"]}') + + const status = report.runExcludeChecks({ core, env: { EXCLUDE_CHECKS: '', CONFIG_FILE: file } }) + + assert.equal(status, 0) + assert.equal(outputs().checks, 'late_escaping\ni18n_usage') + assert.match(logged(), /Excluding 2 check\(s\) from the \.plugin-check\.json: late_escaping, i18n_usage/) + }) + + it('lets the input replace the file, and leaves a bad file alone', async (t) => { + const { core, outputs, logged } = await actionsFor(t) + const file = configFile('this is not JSON') + + const status = report.runExcludeChecks({ core, env: { EXCLUDE_CHECKS: 'from_input\n', CONFIG_FILE: file } }) + + assert.equal(status, 0) + assert.equal(outputs().checks, 'from_input') + assert.match(logged(), /from the exclude-checks input: from_input/) + }) + + it('outputs nothing when there is no input and no file', async (t) => { + const { core, outputs, logged } = await actionsFor(t) + + const status = report.runExcludeChecks({ core, env: { EXCLUDE_CHECKS: '', CONFIG_FILE: path.join(temp(), '.plugin-check.json') } }) + + assert.equal(status, 0) + assert.equal(outputs().checks, '') + assert.match(logged(), /No checks are excluded\./) + }) + + it('fails the step with the message, as an annotation, when the file is bad', async (t) => { + const { core, logged } = await actionsFor(t) + const file = configFile('{"exclude-checks": "late_escaping"}') + + const status = report.runExcludeChecks({ core, env: { EXCLUDE_CHECKS: '', CONFIG_FILE: file } }) + + assert.equal(status, 1) + assert.match(logged(), /::error title=Plugin Check::"exclude-checks" in \.plugin-check\.json must be a list of check names/) + }) + + it('fails the step when the file cannot be read', async (t) => { + // Root reads a file with no permissions, so there is nothing to fail there. + if (process.getuid?.() === 0) { + return t.skip('root can read any file') + } + const { core, logged } = await actionsFor(t) + const file = configFile('{}') + fs.chmodSync(file, 0) + + const status = report.runExcludeChecks({ core, env: { EXCLUDE_CHECKS: '', CONFIG_FILE: file } }) + + fs.chmodSync(file, 0o600) + assert.equal(status, 1) + assert.match(logged(), /::error title=Plugin Check::Could not read /) + }) +}) + +describe('exclude-checks on the command line', () => { + const run = async (args, env = {}) => { + const out = [] + const err = [] + const status = await report.main(['exclude-checks', ...args], env, { out: (line) => out.push(line), err: (line) => err.push(line) }) + + return { status, out, err } + } + + it('prints the flag wp plugin check takes', async () => { + const file = configFile('{"exclude-checks": ["late_escaping", "i18n_usage"]}') + + assert.deepEqual(await run([file]), { status: 0, out: ['--exclude-checks=late_escaping,i18n_usage'], err: [] }) + }) + + it('prints nothing when no check is excluded', async () => { + assert.deepEqual(await run([path.join(temp(), '.plugin-check.json')]), { status: 0, out: [], err: [] }) + }) + + it('lets EXCLUDE_CHECKS replace the file, as the workflow input does', async () => { + const file = configFile('{"exclude-checks": ["from_file"]}') + + assert.deepEqual(await run([file], { EXCLUDE_CHECKS: 'a\nb' }), { status: 0, out: ['--exclude-checks=a,b'], err: [] }) + }) + + it('reads .plugin-check.json in the current folder by default', async () => { + const folder = temp() + fs.writeFileSync(path.join(folder, '.plugin-check.json'), '{"exclude-checks": ["from_cwd"]}') + const before = process.cwd() + process.chdir(folder) + try { + assert.deepEqual(await run([]), { status: 0, out: ['--exclude-checks=from_cwd'], err: [] }) + } finally { + process.chdir(before) + } + }) + + it('fails with the message on standard error when the file is bad', async () => { + const file = configFile('{') + + const { status, out, err } = await run([file]) + + assert.equal(status, 1) + assert.deepEqual(out, []) + assert.match(err[0], new RegExp(`^${file.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')} is not valid JSON`)) + }) + + it('lists the command in the usage line', async () => { + const err = [] + + const status = await report.main(['nope'], {}, { out: () => {}, err: (line) => err.push(line) }) + + assert.equal(status, 2) + assert.match(err[0], /compare\|update\|exclude-checks/) + }) +}) diff --git a/tests/actions/plugin-check-report/fixtures/baseline.json b/tests/actions/plugin-check-report/fixtures/baseline.json new file mode 100644 index 0000000..532809e --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/baseline.json @@ -0,0 +1,9 @@ +{ + "errors": { + "WordPress.Security.EscapeOutput.OutputNotEscaped": 2, + "WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents": 1 + }, + "warnings": { + "WordPress.DB.DirectDatabaseQuery.DirectQuery": 1 + } +} diff --git a/tests/actions/plugin-check-report/fixtures/clean.txt b/tests/actions/plugin-check-report/fixtures/clean.txt new file mode 100644 index 0000000..ad6c4f9 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/clean.txt @@ -0,0 +1 @@ +Success: Checks complete. No errors found. diff --git a/tests/actions/plugin-check-report/fixtures/empty.txt b/tests/actions/plugin-check-report/fixtures/empty.txt new file mode 100644 index 0000000..e69de29 diff --git a/tests/actions/plugin-check-report/fixtures/errors-down.txt b/tests/actions/plugin-check-report/fixtures/errors-down.txt new file mode 100644 index 0000000..fe81472 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/errors-down.txt @@ -0,0 +1,4 @@ +FILE: my-plugin.php +[{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""}] +FILE: src/Forms/Form.php +[{"line":40,"column":5,"type":"ERROR","code":"WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents","message":"file_get_contents() is discouraged.","docs":""},{"line":88,"column":9,"type":"WARNING","code":"WordPress.DB.DirectDatabaseQuery.DirectQuery","message":"Use of a direct database call is discouraged.","docs":""}] diff --git a/tests/actions/plugin-check-report/fixtures/errors-up.txt b/tests/actions/plugin-check-report/fixtures/errors-up.txt new file mode 100644 index 0000000..49ae2cc --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/errors-up.txt @@ -0,0 +1,4 @@ +FILE: my-plugin.php +[{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""},{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""},{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""}] +FILE: src/Forms/Form.php +[{"line":40,"column":5,"type":"ERROR","code":"WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents","message":"file_get_contents() is discouraged.","docs":""},{"line":88,"column":9,"type":"WARNING","code":"WordPress.DB.DirectDatabaseQuery.DirectQuery","message":"Use of a direct database call is discouraged.","docs":""}] diff --git a/tests/actions/plugin-check-report/fixtures/expected/artifact-only.md b/tests/actions/plugin-check-report/fixtures/expected/artifact-only.md new file mode 100644 index 0000000..324adff --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/artifact-only.md @@ -0,0 +1,9 @@ + +⚠️ Plugin Check ran, but there is no baseline file yet + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 0 | + +📎 [Full results and new baseline](https://github.com/o/r/actions/runs/1/artifacts/2) diff --git a/tests/actions/plugin-check-report/fixtures/expected/artifact-only.out b/tests/actions/plugin-check-report/fixtures/expected/artifact-only.out new file mode 100644 index 0000000..fae3456 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/artifact-only.out @@ -0,0 +1,2 @@ +::warning title=Plugin Check::No baseline file at baseline.json, so nothing was compared. To start one, commit the baseline.json from this run's plugin-check-report artifact as baseline.json. +Plugin Check ran with no baseline, so nothing was compared. diff --git a/tests/actions/plugin-check-report/fixtures/expected/artifact-only.summary b/tests/actions/plugin-check-report/fixtures/expected/artifact-only.summary new file mode 100644 index 0000000..092a5b2 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/artifact-only.summary @@ -0,0 +1,16 @@ + +⚠️ Plugin Check ran, but there is no baseline file yet + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 0 | + +📎 [Full results and new baseline](https://github.com/o/r/actions/runs/1/artifacts/2) + +There is no baseline.json yet. To start one, add the file from this run: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.md b/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.md new file mode 100644 index 0000000..c9247b3 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.md @@ -0,0 +1,13 @@ + +❌ Plugin Check failed: 1 rule has more errors than the baseline allows + +![Plugin Check finished](https://img.shields.io/badge/Check%20Finished-now-darkgreen) + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 4 | 3 | +1 | +| Warnings | 1 | 1 | ±0 | + +Went up: `WordPress.Security.EscapeOutput.OutputNotEscaped` (+1) + +📎 [Full results and new baseline](https://github.com/o/r/actions/runs/1/artifacts/2) · [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.out b/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.out new file mode 100644 index 0000000..90bf791 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.out @@ -0,0 +1,2 @@ +::error title=Plugin Check::WordPress.Security.EscapeOutput.OutputNotEscaped has 3 errors, the baseline has 2. The full list is in the plugin-check-results artifact. +Plugin Check found more errors than the baseline allows. diff --git a/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.summary b/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.summary new file mode 100644 index 0000000..4375d48 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/badge-and-artifact.summary @@ -0,0 +1,20 @@ + +❌ Plugin Check failed: 1 rule has more errors than the baseline allows + +![Plugin Check finished](https://img.shields.io/badge/Check%20Finished-now-darkgreen) + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 4 | 3 | +1 | +| Warnings | 1 | 1 | ±0 | + +Went up: `WordPress.Security.EscapeOutput.OutputNotEscaped` (+1) + +📎 [Full results and new baseline](https://github.com/o/r/actions/runs/1/artifacts/2) · [Run](https://example.test/actions/runs/1) + +If the new errors are on purpose, raise their counts in baseline.json in this pull request, so reviewers see it. The file from this run has the new counts: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.md b/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.md new file mode 100644 index 0000000..a465c6b --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.md @@ -0,0 +1,9 @@ + +⚠️ Plugin Check ran, but there is no baseline file yet + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 0 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.out b/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.out new file mode 100644 index 0000000..fae3456 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.out @@ -0,0 +1,2 @@ +::warning title=Plugin Check::No baseline file at baseline.json, so nothing was compared. To start one, commit the baseline.json from this run's plugin-check-report artifact as baseline.json. +Plugin Check ran with no baseline, so nothing was compared. diff --git a/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.summary b/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.summary new file mode 100644 index 0000000..8d892fd --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/clean-no-baseline.summary @@ -0,0 +1,16 @@ + +⚠️ Plugin Check ran, but there is no baseline file yet + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 0 | + +📎 [Run](https://example.test/actions/runs/1) + +There is no baseline.json yet. To start one, add the file from this run: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/errors-down.md b/tests/actions/plugin-check-report/fixtures/expected/errors-down.md new file mode 100644 index 0000000..d8cc887 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/errors-down.md @@ -0,0 +1,11 @@ + +✅ Plugin Check passed + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 2 | 3 | -1 | +| Warnings | 1 | 1 | ±0 | + +Rules that went down: 1. Update the baseline file, see the job summary. + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/errors-down.out b/tests/actions/plugin-check-report/fixtures/expected/errors-down.out new file mode 100644 index 0000000..ebb31c4 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/errors-down.out @@ -0,0 +1,3 @@ +::notice title=Plugin Check::WordPress.Security.EscapeOutput.OutputNotEscaped (errors) went down from 2 to 1. +::notice title=Plugin Check::Error counts went down. Lower the baseline so they cannot come back: commit the baseline.json from this run's plugin-check-report artifact as baseline.json. +Plugin Check error counts are within the baseline. diff --git a/tests/actions/plugin-check-report/fixtures/expected/errors-down.summary b/tests/actions/plugin-check-report/fixtures/expected/errors-down.summary new file mode 100644 index 0000000..ec490bc --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/errors-down.summary @@ -0,0 +1,18 @@ + +✅ Plugin Check passed + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 2 | 3 | -1 | +| Warnings | 1 | 1 | ±0 | + +Rules that went down: 1. Update the baseline file, see the job summary. + +📎 [Run](https://example.test/actions/runs/1) + +Error counts went down. To lower the baseline, replace baseline.json with the file from this run: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/errors-up.md b/tests/actions/plugin-check-report/fixtures/expected/errors-up.md new file mode 100644 index 0000000..c5e77f1 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/errors-up.md @@ -0,0 +1,11 @@ + +❌ Plugin Check failed: 1 rule has more errors than the baseline allows + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 4 | 3 | +1 | +| Warnings | 1 | 1 | ±0 | + +Went up: `WordPress.Security.EscapeOutput.OutputNotEscaped` (+1) + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/errors-up.out b/tests/actions/plugin-check-report/fixtures/expected/errors-up.out new file mode 100644 index 0000000..90bf791 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/errors-up.out @@ -0,0 +1,2 @@ +::error title=Plugin Check::WordPress.Security.EscapeOutput.OutputNotEscaped has 3 errors, the baseline has 2. The full list is in the plugin-check-results artifact. +Plugin Check found more errors than the baseline allows. diff --git a/tests/actions/plugin-check-report/fixtures/expected/errors-up.summary b/tests/actions/plugin-check-report/fixtures/expected/errors-up.summary new file mode 100644 index 0000000..8c8c95f --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/errors-up.summary @@ -0,0 +1,18 @@ + +❌ Plugin Check failed: 1 rule has more errors than the baseline allows + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 4 | 3 | +1 | +| Warnings | 1 | 1 | ±0 | + +Went up: `WordPress.Security.EscapeOutput.OutputNotEscaped` (+1) + +📎 [Run](https://example.test/actions/runs/1) + +If the new errors are on purpose, raise their counts in baseline.json in this pull request, so reviewers see it. The file from this run has the new counts: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.md b/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.md new file mode 100644 index 0000000..fa5abf2 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.md @@ -0,0 +1,15 @@ +line 1 +line 2 +line 3 +line 4 +line 5 +line 6 +line 7 +line 8 +line 9 +line 10 +line 11 +line 12 +line 13 +line 14 +line 15 diff --git a/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.out b/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.out new file mode 100644 index 0000000..90bf791 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.out @@ -0,0 +1,2 @@ +::error title=Plugin Check::WordPress.Security.EscapeOutput.OutputNotEscaped has 3 errors, the baseline has 2. The full list is in the plugin-check-results artifact. +Plugin Check found more errors than the baseline allows. diff --git a/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.summary b/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.summary new file mode 100644 index 0000000..8a706aa --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/marker-with-line-breaks.summary @@ -0,0 +1,22 @@ +line 1 +line 2 +line 3 +line 4 +line 5 +line 6 +line 7 +line 8 +line 9 +line 10 +line 11 +line 12 +line 13 +line 14 +line 15 + +If the new errors are on purpose, raise their counts in baseline.json in this pull request, so reviewers see it. The file from this run has the new counts: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/marker.md b/tests/actions/plugin-check-report/fixtures/expected/marker.md new file mode 100644 index 0000000..2be0d1f --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/marker.md @@ -0,0 +1,9 @@ + +⚠️ Plugin Check ran, but there is no baseline file yet + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 0 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/marker.out b/tests/actions/plugin-check-report/fixtures/expected/marker.out new file mode 100644 index 0000000..fae3456 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/marker.out @@ -0,0 +1,2 @@ +::warning title=Plugin Check::No baseline file at baseline.json, so nothing was compared. To start one, commit the baseline.json from this run's plugin-check-report artifact as baseline.json. +Plugin Check ran with no baseline, so nothing was compared. diff --git a/tests/actions/plugin-check-report/fixtures/expected/marker.summary b/tests/actions/plugin-check-report/fixtures/expected/marker.summary new file mode 100644 index 0000000..8137bfd --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/marker.summary @@ -0,0 +1,16 @@ + +⚠️ Plugin Check ran, but there is no baseline file yet + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 0 | + +📎 [Run](https://example.test/actions/runs/1) + +There is no baseline.json yet. To start one, add the file from this run: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/results-empty.out b/tests/actions/plugin-check-report/fixtures/expected/results-empty.out new file mode 100644 index 0000000..b79b7db --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/results-empty.out @@ -0,0 +1 @@ +::error title=Plugin Check::The results file has no Plugin Check output, so the check did not run. diff --git a/tests/actions/plugin-check-report/fixtures/expected/results-garbage.out b/tests/actions/plugin-check-report/fixtures/expected/results-garbage.out new file mode 100644 index 0000000..ec39073 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/results-garbage.out @@ -0,0 +1 @@ +::error title=Plugin Check::The results file is not Plugin Check output, so the check did not run. First unexpected line: diff --git a/tests/actions/plugin-check-report/fixtures/expected/results-missing.out b/tests/actions/plugin-check-report/fixtures/expected/results-missing.out new file mode 100644 index 0000000..1286ae1 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/results-missing.out @@ -0,0 +1 @@ +::error title=Plugin Check::Results file not found: /nonexistent/results.txt diff --git a/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.md b/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.md new file mode 100644 index 0000000..99a03fa --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.md @@ -0,0 +1,12 @@ + +❌ Plugin Check failed: 1 rule has more errors than the baseline allows + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 3 | 3 | ±0 | +| Warnings | 1 | 1 | ±0 | + +Went up: `WordPress.Security.EscapeOutput.OutputNotEscaped` (+1) +Rules that went down: 1. Update the baseline file, see the job summary. + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.out b/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.out new file mode 100644 index 0000000..caaeea3 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.out @@ -0,0 +1,4 @@ +::error title=Plugin Check::WordPress.Security.EscapeOutput.OutputNotEscaped has 3 errors, the baseline has 2. The full list is in the plugin-check-results artifact. +::notice title=Plugin Check::WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents (errors) went down from 1 to 0. +::notice title=Plugin Check::Error counts went down. Lower the baseline so they cannot come back: commit the baseline.json from this run's plugin-check-report artifact as baseline.json. +Plugin Check found more errors than the baseline allows. diff --git a/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.summary b/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.summary new file mode 100644 index 0000000..2386d65 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/rule-up-total-same.summary @@ -0,0 +1,19 @@ + +❌ Plugin Check failed: 1 rule has more errors than the baseline allows + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 3 | 3 | ±0 | +| Warnings | 1 | 1 | ±0 | + +Went up: `WordPress.Security.EscapeOutput.OutputNotEscaped` (+1) +Rules that went down: 1. Update the baseline file, see the job summary. + +📎 [Run](https://example.test/actions/runs/1) + +Error counts went down. To lower the baseline, replace baseline.json with the file from this run: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.md b/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.md new file mode 100644 index 0000000..c965444 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.md @@ -0,0 +1,9 @@ + +✅ Plugin Check passed + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 3 | 3 | ±0 | +| Warnings | 1 | 1 | ±0 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.out b/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.out new file mode 100644 index 0000000..0d8c877 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.out @@ -0,0 +1 @@ +Plugin Check error counts are within the baseline. diff --git a/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.summary b/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.summary new file mode 100644 index 0000000..c965444 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/same-as-baseline.summary @@ -0,0 +1,9 @@ + +✅ Plugin Check passed + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 3 | 3 | ±0 | +| Warnings | 1 | 1 | ±0 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.md b/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.md new file mode 100644 index 0000000..c407f25 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.md @@ -0,0 +1,12 @@ + +❌ Plugin Check failed: 7 rules have more errors than the baseline allows + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 7 | 3 | +4 | +| Warnings | 0 | 1 | -1 | + +Went up (top 5 of 7): `Test.Rule.Number1` (+1), `Test.Rule.Number2` (+1), `Test.Rule.Number3` (+1), `Test.Rule.Number4` (+1), `Test.Rule.Number5` (+1) +Rules that went down: 3. Update the baseline file, see the job summary. + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.out b/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.out new file mode 100644 index 0000000..fd225d1 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.out @@ -0,0 +1,12 @@ +::error title=Plugin Check::Test.Rule.Number1 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number2 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number3 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number4 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number5 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number6 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number7 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::notice title=Plugin Check::WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents (errors) went down from 1 to 0. +::notice title=Plugin Check::WordPress.Security.EscapeOutput.OutputNotEscaped (errors) went down from 2 to 0. +::notice title=Plugin Check::WordPress.DB.DirectDatabaseQuery.DirectQuery (warnings) went down from 1 to 0. +::notice title=Plugin Check::Error counts went down. Lower the baseline so they cannot come back: commit the baseline.json from this run's plugin-check-report artifact as baseline.json. +Plugin Check found more errors than the baseline allows. diff --git a/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.summary b/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.summary new file mode 100644 index 0000000..3f2176c --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/seven-new-codes.summary @@ -0,0 +1,19 @@ + +❌ Plugin Check failed: 7 rules have more errors than the baseline allows + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 7 | 3 | +4 | +| Warnings | 0 | 1 | -1 | + +Went up (top 5 of 7): `Test.Rule.Number1` (+1), `Test.Rule.Number2` (+1), `Test.Rule.Number3` (+1), `Test.Rule.Number4` (+1), `Test.Rule.Number5` (+1) +Rules that went down: 3. Update the baseline file, see the job summary. + +📎 [Run](https://example.test/actions/runs/1) + +Error counts went down. To lower the baseline, replace baseline.json with the file from this run: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.md b/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.md new file mode 100644 index 0000000..131efef --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.md @@ -0,0 +1,12 @@ + +❌ Plugin Check failed: 6 rules have more errors than the baseline allows + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 6 | 3 | +3 | +| Warnings | 0 | 1 | -1 | + +Went up (top 5 of 6): `Test.Rule.Number1` (+1), `Test.Rule.Number2` (+1), `Test.Rule.Number3` (+1), `Test.Rule.Number4` (+1), `Test.Rule.Number5` (+1) +Rules that went down: 3. Update the baseline file, see the job summary. + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.out b/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.out new file mode 100644 index 0000000..ecd7d0f --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.out @@ -0,0 +1,11 @@ +::error title=Plugin Check::Test.Rule.Number1 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number2 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number3 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number4 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number5 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::error title=Plugin Check::Test.Rule.Number6 has 1 errors, the baseline has 0. The full list is in the plugin-check-results artifact. +::notice title=Plugin Check::WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents (errors) went down from 1 to 0. +::notice title=Plugin Check::WordPress.Security.EscapeOutput.OutputNotEscaped (errors) went down from 2 to 0. +::notice title=Plugin Check::WordPress.DB.DirectDatabaseQuery.DirectQuery (warnings) went down from 1 to 0. +::notice title=Plugin Check::Error counts went down. Lower the baseline so they cannot come back: commit the baseline.json from this run's plugin-check-report artifact as baseline.json. +Plugin Check found more errors than the baseline allows. diff --git a/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.summary b/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.summary new file mode 100644 index 0000000..11d5606 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/six-new-codes.summary @@ -0,0 +1,19 @@ + +❌ Plugin Check failed: 6 rules have more errors than the baseline allows + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 6 | 3 | +3 | +| Warnings | 0 | 1 | -1 | + +Went up (top 5 of 6): `Test.Rule.Number1` (+1), `Test.Rule.Number2` (+1), `Test.Rule.Number3` (+1), `Test.Rule.Number4` (+1), `Test.Rule.Number5` (+1) +Rules that went down: 3. Update the baseline file, see the job summary. + +📎 [Run](https://example.test/actions/runs/1) + +Error counts went down. To lower the baseline, replace baseline.json with the file from this run: + +```bash +gh run download --repo / --name plugin-check-report --dir /tmp/plugin-check +cp /tmp/plugin-check/baseline.json baseline.json +``` diff --git a/tests/actions/plugin-check-report/fixtures/expected/status.json b/tests/actions/plugin-check-report/fixtures/expected/status.json new file mode 100644 index 0000000..45e1c09 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/status.json @@ -0,0 +1,25 @@ +{ + "clean-no-baseline": 0, + "same-as-baseline": 0, + "errors-up": 1, + "rule-up-total-same": 1, + "errors-down": 0, + "warnings-up": 0, + "strict-no-baseline": 1, + "strict-errors-down": 1, + "strict-clean": 0, + "strict-warnings-only": 0, + "seven-new-codes": 1, + "six-new-codes": 1, + "badge-and-artifact": 1, + "artifact-only": 0, + "marker": 0, + "marker-with-line-breaks": 1, + "results-missing": 1, + "results-garbage": 1, + "results-empty": 1, + "unknown-mode": 1, + "update-errors-up": 0, + "update-clean": 0, + "update-garbage": 1 +} diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-clean.md b/tests/actions/plugin-check-report/fixtures/expected/strict-clean.md new file mode 100644 index 0000000..f6d8e1f --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-clean.md @@ -0,0 +1,9 @@ + +✅ Plugin Check passed: no errors + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 0 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-clean.out b/tests/actions/plugin-check-report/fixtures/expected/strict-clean.out new file mode 100644 index 0000000..afbd6dd --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-clean.out @@ -0,0 +1,2 @@ +No baseline file. Strict mode does not need one. +Plugin Check ran with no baseline, so nothing was compared. diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-clean.summary b/tests/actions/plugin-check-report/fixtures/expected/strict-clean.summary new file mode 100644 index 0000000..f6d8e1f --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-clean.summary @@ -0,0 +1,9 @@ + +✅ Plugin Check passed: no errors + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 0 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.md b/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.md new file mode 100644 index 0000000..a8f2c38 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.md @@ -0,0 +1,11 @@ + +❌ Plugin Check failed: 2 errors, and strict mode allows none + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 2 | 3 | -1 | +| Warnings | 1 | 1 | ±0 | + +Rules that went down: 1. + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.out b/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.out new file mode 100644 index 0000000..46ba4e0 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.out @@ -0,0 +1,3 @@ +::notice title=Plugin Check::WordPress.Security.EscapeOutput.OutputNotEscaped (errors) went down from 2 to 1. +::error title=Plugin Check::2 errors found. Strict mode fails on any error. +Plugin Check found errors, and strict mode allows none. diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.summary b/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.summary new file mode 100644 index 0000000..a8f2c38 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-errors-down.summary @@ -0,0 +1,11 @@ + +❌ Plugin Check failed: 2 errors, and strict mode allows none + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 2 | 3 | -1 | +| Warnings | 1 | 1 | ±0 | + +Rules that went down: 1. + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.md b/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.md new file mode 100644 index 0000000..4fd50c5 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.md @@ -0,0 +1,9 @@ + +❌ Plugin Check failed: 2 errors, and strict mode allows none + +| | Now | +|:--|--:| +| Errors | 2 | +| Warnings | 1 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.out b/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.out new file mode 100644 index 0000000..ee0d0f0 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.out @@ -0,0 +1,3 @@ +No baseline file. Strict mode does not need one. +::error title=Plugin Check::2 errors found. Strict mode fails on any error. +Plugin Check found errors, and strict mode allows none. diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.summary b/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.summary new file mode 100644 index 0000000..4fd50c5 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-no-baseline.summary @@ -0,0 +1,9 @@ + +❌ Plugin Check failed: 2 errors, and strict mode allows none + +| | Now | +|:--|--:| +| Errors | 2 | +| Warnings | 1 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.md b/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.md new file mode 100644 index 0000000..d2368c1 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.md @@ -0,0 +1,9 @@ + +✅ Plugin Check passed: no errors + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 3 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.out b/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.out new file mode 100644 index 0000000..afbd6dd --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.out @@ -0,0 +1,2 @@ +No baseline file. Strict mode does not need one. +Plugin Check ran with no baseline, so nothing was compared. diff --git a/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.summary b/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.summary new file mode 100644 index 0000000..d2368c1 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/strict-warnings-only.summary @@ -0,0 +1,9 @@ + +✅ Plugin Check passed: no errors + +| | Now | +|:--|--:| +| Errors | 0 | +| Warnings | 3 | + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/unknown-mode.out b/tests/actions/plugin-check-report/fixtures/expected/unknown-mode.out new file mode 100644 index 0000000..0ff583b --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/unknown-mode.out @@ -0,0 +1 @@ +::error title=Plugin Check::Unknown mode 'strcit'. Use 'baseline' or 'strict'. diff --git a/tests/actions/plugin-check-report/fixtures/expected/update-clean.json b/tests/actions/plugin-check-report/fixtures/expected/update-clean.json new file mode 100644 index 0000000..75fdfd5 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/update-clean.json @@ -0,0 +1,4 @@ +{ + "errors": {}, + "warnings": {} +} diff --git a/tests/actions/plugin-check-report/fixtures/expected/update-clean.out b/tests/actions/plugin-check-report/fixtures/expected/update-clean.out new file mode 100644 index 0000000..1c5ea8d --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/update-clean.out @@ -0,0 +1 @@ +Wrote /baseline.json: 0 error codes, 0 warning codes diff --git a/tests/actions/plugin-check-report/fixtures/expected/update-errors-up.json b/tests/actions/plugin-check-report/fixtures/expected/update-errors-up.json new file mode 100644 index 0000000..5344a3d --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/update-errors-up.json @@ -0,0 +1,9 @@ +{ + "errors": { + "WordPress.Security.EscapeOutput.OutputNotEscaped": 3, + "WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents": 1 + }, + "warnings": { + "WordPress.DB.DirectDatabaseQuery.DirectQuery": 1 + } +} diff --git a/tests/actions/plugin-check-report/fixtures/expected/update-errors-up.out b/tests/actions/plugin-check-report/fixtures/expected/update-errors-up.out new file mode 100644 index 0000000..4240312 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/update-errors-up.out @@ -0,0 +1 @@ +Wrote /baseline.json: 2 error codes, 1 warning codes diff --git a/tests/actions/plugin-check-report/fixtures/expected/update-garbage.out b/tests/actions/plugin-check-report/fixtures/expected/update-garbage.out new file mode 100644 index 0000000..ec39073 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/update-garbage.out @@ -0,0 +1 @@ +::error title=Plugin Check::The results file is not Plugin Check output, so the check did not run. First unexpected line: diff --git a/tests/actions/plugin-check-report/fixtures/expected/warnings-up.md b/tests/actions/plugin-check-report/fixtures/expected/warnings-up.md new file mode 100644 index 0000000..5d8ef4f --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/warnings-up.md @@ -0,0 +1,11 @@ + +✅ Plugin Check passed + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 3 | 3 | ±0 | +| Warnings | 3 | 1 | +2 | + +Went up: `WordPress.DB.DirectDatabaseQuery.DirectQuery` warning (+2) + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/expected/warnings-up.out b/tests/actions/plugin-check-report/fixtures/expected/warnings-up.out new file mode 100644 index 0000000..2211d5d --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/warnings-up.out @@ -0,0 +1,2 @@ +::notice title=Plugin Check::WordPress.DB.DirectDatabaseQuery.DirectQuery has 3 warnings, the baseline has 1. Warnings do not fail the check. +Plugin Check error counts are within the baseline. diff --git a/tests/actions/plugin-check-report/fixtures/expected/warnings-up.summary b/tests/actions/plugin-check-report/fixtures/expected/warnings-up.summary new file mode 100644 index 0000000..5d8ef4f --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/expected/warnings-up.summary @@ -0,0 +1,11 @@ + +✅ Plugin Check passed + +| | Now | Baseline | Change | +|:--|--:|--:|:--| +| Errors | 3 | 3 | ±0 | +| Warnings | 3 | 1 | +2 | + +Went up: `WordPress.DB.DirectDatabaseQuery.DirectQuery` warning (+2) + +📎 [Run](https://example.test/actions/runs/1) diff --git a/tests/actions/plugin-check-report/fixtures/garbage.txt b/tests/actions/plugin-check-report/fixtures/garbage.txt new file mode 100644 index 0000000..c31d675 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/garbage.txt @@ -0,0 +1,3 @@ + +Database Error +

Error establishing a database connection

diff --git a/tests/actions/plugin-check-report/fixtures/rule-up-total-same.txt b/tests/actions/plugin-check-report/fixtures/rule-up-total-same.txt new file mode 100644 index 0000000..2efabff --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/rule-up-total-same.txt @@ -0,0 +1,4 @@ +FILE: my-plugin.php +[{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""},{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""},{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""}] +FILE: src/Forms/Form.php +[{"line":88,"column":9,"type":"WARNING","code":"WordPress.DB.DirectDatabaseQuery.DirectQuery","message":"Use of a direct database call is discouraged.","docs":""}] diff --git a/tests/actions/plugin-check-report/fixtures/same-as-baseline.txt b/tests/actions/plugin-check-report/fixtures/same-as-baseline.txt new file mode 100644 index 0000000..b2ba8a2 --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/same-as-baseline.txt @@ -0,0 +1,4 @@ +FILE: my-plugin.php +[{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""},{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""}] +FILE: src/Forms/Form.php +[{"line":40,"column":5,"type":"ERROR","code":"WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents","message":"file_get_contents() is discouraged.","docs":""},{"line":88,"column":9,"type":"WARNING","code":"WordPress.DB.DirectDatabaseQuery.DirectQuery","message":"Use of a direct database call is discouraged.","docs":""}] diff --git a/tests/actions/plugin-check-report/fixtures/warnings-up.txt b/tests/actions/plugin-check-report/fixtures/warnings-up.txt new file mode 100644 index 0000000..4b9f1cc --- /dev/null +++ b/tests/actions/plugin-check-report/fixtures/warnings-up.txt @@ -0,0 +1,4 @@ +FILE: my-plugin.php +[{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""},{"line":12,"column":10,"type":"ERROR","code":"WordPress.Security.EscapeOutput.OutputNotEscaped","message":"All output should be run through an escaping function.","docs":""}] +FILE: src/Forms/Form.php +[{"line":40,"column":5,"type":"ERROR","code":"WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents","message":"file_get_contents() is discouraged.","docs":""},{"line":88,"column":9,"type":"WARNING","code":"WordPress.DB.DirectDatabaseQuery.DirectQuery","message":"Use of a direct database call is discouraged.","docs":""},{"line":88,"column":9,"type":"WARNING","code":"WordPress.DB.DirectDatabaseQuery.DirectQuery","message":"Use of a direct database call is discouraged.","docs":""},{"line":88,"column":9,"type":"WARNING","code":"WordPress.DB.DirectDatabaseQuery.DirectQuery","message":"Use of a direct database call is discouraged.","docs":""}] diff --git a/tests/actions/plugin-check-report/golden.js b/tests/actions/plugin-check-report/golden.js new file mode 100644 index 0000000..1823152 --- /dev/null +++ b/tests/actions/plugin-check-report/golden.js @@ -0,0 +1,186 @@ +'use strict' + +/** + * The cases whose exact output is pinned in fixtures/expected/. + * + * The files there were written by the original ratchet.sh. Every implementation of the compare has + * to print the same status, the same stdout, the same comment, the same job summary and, for + * --update, the same baseline file, byte for byte. A case is described once here and run by a + * `execute` function, so the same list checks the bash script and the JavaScript that replaces it. + */ + +const fs = require('node:fs') +const os = require('node:os') +const path = require('node:path') + +const FIXTURES = path.join(__dirname, 'fixtures') +const EXPECTED = path.join(FIXTURES, 'expected') + +/** A path that never exists, so a case without a baseline gets none and prints a fixed path. */ +const NO_BASELINE = '/nonexistent/baseline.json' + +/** + * A results file with one new ERROR code per entry. + * + * @param {number} codes How many different codes to write. + * + * @returns {string} The file's text. + */ +function newCodes(codes) { + const results = Array.from({ length: codes }, (_, index) => ({ + line: index + 1, + column: 1, + type: 'ERROR', + code: `Test.Rule.Number${index + 1}`, + message: 'A rule added for the test.', + docs: '', + })) + + return `FILE: plugin.php\n${JSON.stringify(results)}\n` +} + +/** + * The warnings-up sample without its ERROR results. + * + * @returns {string} The file's text. + */ +function warningsOnly() { + const text = fs.readFileSync(path.join(FIXTURES, 'warnings-up.txt'), 'utf8') + + return text + .split('\n') + .map((line) => { + if (!line.startsWith('[')) { + return line + } + + return JSON.stringify(JSON.parse(line).filter((result) => result.type === 'WARNING')) + }) + .join('\n') +} + +const RUN = { RUN_URL: 'https://example.test/actions/runs/1' } +const BASELINE = 'baseline.json' + +/** + * Each case: a name, the results (a fixture name, or `text` to write), the baseline fixture to use, + * extra arguments, and the environment. + */ +const CASES = [ + { name: 'clean-no-baseline', results: 'clean.txt', env: RUN }, + { name: 'same-as-baseline', results: 'same-as-baseline.txt', baseline: BASELINE, env: RUN }, + { name: 'errors-up', results: 'errors-up.txt', baseline: BASELINE, env: RUN }, + { name: 'rule-up-total-same', results: 'rule-up-total-same.txt', baseline: BASELINE, env: RUN }, + { name: 'errors-down', results: 'errors-down.txt', baseline: BASELINE, env: RUN }, + { name: 'warnings-up', results: 'warnings-up.txt', baseline: BASELINE, env: RUN }, + { name: 'strict-no-baseline', results: 'errors-down.txt', env: { ...RUN, PLUGIN_CHECK_MODE: 'strict' } }, + { + name: 'strict-errors-down', + results: 'errors-down.txt', + baseline: BASELINE, + env: { ...RUN, PLUGIN_CHECK_MODE: 'strict' }, + }, + { name: 'strict-clean', results: 'clean.txt', env: { ...RUN, PLUGIN_CHECK_MODE: 'strict' } }, + { name: 'strict-warnings-only', text: warningsOnly, env: { ...RUN, PLUGIN_CHECK_MODE: 'strict' } }, + { name: 'seven-new-codes', text: () => newCodes(7), baseline: BASELINE, env: RUN }, + { name: 'six-new-codes', text: () => newCodes(6), baseline: BASELINE, env: RUN }, + { + name: 'badge-and-artifact', + results: 'errors-up.txt', + baseline: BASELINE, + env: { + ...RUN, + COMMENT_BADGE: 'https://img.shields.io/badge/Check%20Finished-now-darkgreen', + ARTIFACT_URL: 'https://github.com/o/r/actions/runs/1/artifacts/2', + }, + }, + { name: 'artifact-only', results: 'clean.txt', env: { ARTIFACT_URL: 'https://github.com/o/r/actions/runs/1/artifacts/2' } }, + { + name: 'marker', + results: 'clean.txt', + env: { ...RUN, COMMENT_MARKER: '' }, + }, + { + name: 'marker-with-line-breaks', + results: 'errors-up.txt', + baseline: BASELINE, + env: { ...RUN, COMMENT_MARKER: Array.from({ length: 20 }, (_, index) => `line ${index + 1}`).join('\n') }, + }, + { name: 'results-missing', results: false, args: ['/nonexistent/results.txt'], env: RUN }, + { name: 'results-garbage', results: 'garbage.txt', env: RUN }, + { name: 'results-empty', results: 'empty.txt', env: RUN }, + { name: 'unknown-mode', results: 'clean.txt', env: { ...RUN, PLUGIN_CHECK_MODE: 'strcit' } }, + { name: 'update-errors-up', results: 'errors-up.txt', update: true, env: RUN }, + { name: 'update-clean', results: 'clean.txt', update: true, env: RUN }, + { name: 'update-garbage', results: 'garbage.txt', update: true, env: RUN }, +] + +/** + * Runs one case. + * + * @param {object} spec A case from CASES. + * @param {Function} execute `(args, env) => {status, stdout}`. Runs the script under test with the + * arguments and environment given. + * + * @returns {{status: number, stdout: string, comment: string, summary: string, baseline: string}} + * What the script did. The temporary folder in stdout is written ``. + */ +function runCase(spec, execute) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-check-golden-')) + const comment = path.join(dir, 'comment.md') + const summary = path.join(dir, 'summary.md') + const written = path.join(dir, 'baseline.json') + + let results + if (spec.text) { + results = path.join(dir, 'results.txt') + fs.writeFileSync(results, spec.text()) + } else { + results = spec.args ? spec.args[0] : path.join(FIXTURES, spec.results) + } + + const env = { + GITHUB_STEP_SUMMARY: summary, + PLUGIN_CHECK_BASELINE: spec.update ? written : spec.baseline ? path.join(FIXTURES, spec.baseline) : NO_BASELINE, + ...spec.env, + } + const args = spec.update ? ['--update', results] : ['--comment-file', comment, results] + const result = execute(args, env) + + const read = (file) => (fs.existsSync(file) ? fs.readFileSync(file, 'utf8') : '') + + return { + status: result.status, + stdout: result.stdout.split(dir).join(''), + comment: read(comment), + summary: read(summary), + baseline: read(written), + } +} + +/** + * The pinned files of a case, as written by the original script. + * + * @param {string} name A case name. + * + * @returns {{status: number, stdout: string, comment: string, summary: string, baseline: string}} + * The expected result. + */ +function expectedFor(name) { + const read = (extension) => { + const file = path.join(EXPECTED, `${name}.${extension}`) + + return fs.existsSync(file) ? fs.readFileSync(file, 'utf8') : '' + } + const statuses = JSON.parse(fs.readFileSync(path.join(EXPECTED, 'status.json'), 'utf8')) + + return { + status: statuses[name], + stdout: read('out'), + comment: read('md'), + summary: read('summary'), + baseline: read('json'), + } +} + +module.exports = { CASES, FIXTURES, expectedFor, runCase }