Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,24 @@ jobs:

steps:
- name: Checkout repository
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
fetch-depth: 0

# The repo's package manager is npm (package-lock.json, no yarn.lock).
# borales/actions-yarn@v2.1.0 ran a docker image with no yarn in it
# ("sh: yarn: not found"), so this check failed on every PR.
- name: Set up Node
uses: actions/setup-node@v2
uses: actions/setup-node@v4
with:
node-version: 14
node-version: 22
cache: npm

- name: Install Dependencies
uses: borales/actions-yarn@v2.1.0
with:
cmd: install
run: npm ci

- name: Build Project
uses: borales/actions-yarn@v2.1.0
with:
cmd: build
run: npm run build

- name: Check commits messages
uses: wagoid/commitlint-github-action@v5
105 changes: 105 additions & 0 deletions .github/workflows/playwright-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
name: "@keploy/playwright publish"

# Publishes @keploy/playwright to npm from a tag.
#
# To release X.Y.Z: set "version": "X.Y.Z" in playwright/package.json on main
# (a pull request, so the dry run below checks it), then push the tag
# playwright-vX.Y.Z on that commit. The rows run on the tagged commit first;
# the publish runs only when every leg passes, and refuses a tag that does not
# name package.json's version (scripts/release-version.mjs). npm itself
# refuses a version it already has.
#
# Push playwright-vX.Y.Z as a plain git tag, never as a GitHub release: a
# published GitHub release also starts any workflow of this repository that
# runs on releases (the root @keploy/sdk release workflow, #117), which would
# try to publish that package under this tag's name. playwright/README.md,
# "Releasing", says the same.
#
# On a pull request the same package is published as a dry run, with no
# token, so a package that would ship without its build shows before a tag.
on:
push:
tags: ["playwright-v*"]
pull_request:
paths:
- "playwright/**"
- ".github/workflows/playwright-publish.yml"
- ".github/workflows/playwright-sdk.yml"

permissions:
contents: read

jobs:
rows:
if: github.event_name == 'push'
uses: ./.github/workflows/playwright-sdk.yml

publish:
name: publish to npm
if: github.event_name == 'push'
needs: rows
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # npm --provenance signs the package with this run's identity
defaults:
run:
shell: bash # -eo pipefail: a failed command before a pipe fails the step
working-directory: playwright
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
# Writes the token line for registry.npmjs.org and pins the @keploy
# scope to it (@keploy:registry), in the runner's own npmrc.
registry-url: https://registry.npmjs.org
scope: "@keploy"
- name: The tag names package.json's version
run: node scripts/release-version.mjs "$GITHUB_REF_NAME"
- name: The npm token is set
env:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
run: test -n "$NPM_TOKEN" || { echo "::error::the NPM_TOKEN secret is not set; refusing to publish"; exit 1; }
- name: The @keploy scope publishes to registry.npmjs.org
# npm sends a scoped package to its scope's registry, whatever
# --registry says. Read the setting npm will use, with every npmrc in
# it (a project one included), and refuse any other.
run: |
got=$(npm config get @keploy:registry)
test "$got" = https://registry.npmjs.org/ || { echo "::error::@keploy:registry is '$got', not https://registry.npmjs.org/; refusing to publish"; exit 1; }
- run: npm ci
- name: The token authenticates
# Fails here, named, on a token that is expired, revoked or pasted
# with a stray character, before anything is uploaded.
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
run: npm whoami --registry https://registry.npmjs.org/
- name: Publish
# prepublishOnly builds dist/ first.
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
run: npm publish --access public --provenance

dry-run:
name: publish dry run
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
defaults:
run:
shell: bash # -eo pipefail: a failed command before a pipe fails the step
working-directory: playwright
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm ci
- name: Publish, dry run
run: npm publish --dry-run --access public 2>&1 | tee "$RUNNER_TEMP/publish.log"
- name: The package carries its build
# A fresh checkout has no dist/; only prepublishOnly puts it there.
run: |
for f in dist/index.js dist/index.d.ts package.json README.md; do
grep -qE "npm notice [0-9.]+[kMG]?B +$f\$" "$RUNNER_TEMP/publish.log" || { echo "::error::the package would ship without $f"; exit 1; }
done
38 changes: 38 additions & 0 deletions .github/workflows/playwright-sdk.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: "@keploy/playwright"

on:
pull_request:
paths:
- "playwright/**"
- ".github/workflows/playwright-sdk.yml"
push:
branches: [main]
paths:
- "playwright/**"
- ".github/workflows/playwright-sdk.yml"
# playwright-publish.yml runs these rows on a release tag before it publishes.
workflow_call:

jobs:
rows:
name: build and rows (Node ${{ matrix.node }})
runs-on: ubuntu-latest
strategy:
# One leg failing must not cancel the others: each Node version's
# result is its own.
fail-fast: false
matrix:
# Playwright 1.62 (the rows' version) needs Node.js 20 or later, and
# so does this package (package.json engines).
node: [20, 22, 24]
defaults:
run:
working-directory: playwright
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
- run: npm ci
- run: npx playwright install --with-deps chromium
- run: npm test
3 changes: 3 additions & 0 deletions playwright/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
node_modules
dist
test-results
65 changes: 65 additions & 0 deletions playwright/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# @keploy/playwright

Joins a Playwright suite to keploy's Universal Mocker. `keploy mock record` records the calls each test's browser and app server make; `keploy mock replay` answers them from the recording, so the suite runs without its backends. keploy judges every recorded test, and lists each test it cannot replay with a reason.

This package tells keploy where each test begins and ends. Through it, keploy skips a test it set aside and fails a test whose calls changed. It speaks keploy's scope protocol v1, the same one go-sdk's `keploytest` speaks, and decides nothing itself.

## Use

```sh
npm install --save-dev @keploy/playwright
```

It needs Node.js 20 or later. Its rows run on Node.js 20, 22 and 24.

Import `test` and `expect` from this package instead of `@playwright/test`. A fixture file that extends `test` extends this one:

```ts
import { test, expect } from '@keploy/playwright';

test('lists the apps', async ({ page }) => {
await page.goto('/apps');
await expect(page.getByRole('row')).toHaveCount(3);
});
```

Run the suite under keploy with `KEPLOY_PROVEN=1`, one worker and no retries:

```sh
KEPLOY_PROVEN=1 keploy mock record -c "npx playwright test --workers=1 --retries=0"
KEPLOY_PROVEN=1 keploy mock replay -c "npx playwright test --workers=1 --retries=0"
```

`KEPLOY_PROVEN=1` is keploy's switch for its proven model. keploy speaks this package's wire, scope protocol v1 (the claim, begin, end and given), only under it. Without it, keploy answers the worker's claim with a plain 404, and every test of the worker fails with `keploy: this keploy agent answered 404 to POST /agent/instance, so this worker cannot claim keploy's run: ...`, which names the switch.

keploy's first release runs one test process at a time. A second worker that is still alive while another runs is refused by keploy, and its tests fail. Turn retries off: keploy sets aside a test it cannot prove, with its reason, and fails a test whose calls changed, and a retry changes neither.

## Without keploy

With `KEPLOY_MOCK_AGENT` unset or `off`, nothing changes. Only `keploy mock record` and `keploy mock replay` set that variable.

## Under keploy

- **Claim.** Each worker claims keploy's run before its first hook or test (`POST /agent/instance`). The claim uses the worker's Playwright project name as its key, or `.` for an unnamed project. A worker that Playwright starts again after a failed test claims again.
- **Begin.** Each test begins (`POST /agent/scope/begin`) before its `beforeEach` hooks. keploy builds the test's ID as `[project::]file::titles`. The file is relative to the project directory keploy names (`KEPLOY_MOCK_PROJECT`). The titles are the describe groups and the test's own title, joined with ` › `.
- **Skip.** When keploy has set a test aside, the test is skipped through Playwright's own skip, and the report shows keploy's reason (`keploy: ...`).
- **Settle.** Before the test's `context` (and its `page`) closes, the test waits until none of its pages' calls is in flight. Calls are counted by the browser's own request events, and no time window decides. A call that never ends keeps the test waiting until Playwright's test timeout fails it. The test shows a step that names each call still in flight. A call is not waited for once the browser has cancelled it, which happens when its page loads another document, its frame is removed, its page closes or the test closes its own context. A closing page cancels only its own calls: a popup that closes cancels nothing of its opener's. A route change inside the page (the history API) cancels nothing, even after an answered call. A service worker's calls are counted too. No page's change drops them, but the test closing its own context does, as it ends the worker.
- **End.** The test ends (`POST /agent/scope/end`) after its last teardown, with its own outcome. When keploy saw a call with no recorded answer, or a recorded call the test no longer made, the test fails with keploy's text.
- **Given values.** `await given(name, value)` tells keploy a value the test made itself. Examples are a session token it signs and a name it stamps with the time. keploy then follows the value wherever the test's calls carry it, as it follows an incoming request's values in a server test. Give a distinctive value, whole, as the calls carry it. `given` must be called inside a test, or in its `beforeEach` or `afterEach` hooks. It returns the value. It needs a keploy whose agent serves `POST /agent/scope/given` (see the limits below).

Any answer from the agent that this package cannot trust fails the test. That covers an agent it cannot reach, an error answer, an unreadable answer, and an answer that does not speak scope protocol v1. Nothing is retried.

## Limits of this version

- Settle covers the `context` and `page` fixtures. A context the test makes with `browser.newContext()` is the test's own to close.
- Settle's rows run in Chromium. Which calls the browser itself ends (a removed frame's) is proven there only; Firefox and WebKit are not rowed.
- Calls that `beforeAll`/`afterAll` hooks and worker fixtures make outside every test are answered from the worker's claim, one slice per Playwright project.
- The `beforeAll` and `afterAll` hooks of a describe group whose tests keploy has all set aside still run, and their calls are answered from the claim. Each test is skipped at its begin, after the group's hooks have started; skipping the whole group before its hooks run needs a way to ask keploy for the set-aside tests first, which this version does not have.
- `given` needs the agent route `POST /agent/scope/given` (keploy/keploy#5016). keploy v3.6.192 and later serve it; with the enterprise keploy, that is an enterprise release on keploy v3.6.192 or later. An older keploy answers that route with a plain 404, and `given` fails the test with `keploy: this keploy agent answered 404 to POST /agent/scope/given, so it cannot take the given value <name>: given needs keploy v3.6.192 or later (keploy/keploy#5016), or an enterprise release on keploy v3.6.192 or later; an older keploy answers 404 there`. Claim, begin, settle and end do not need that route.

## Releasing

1. In a pull request, set `"version": "X.Y.Z"` in `playwright/package.json`. The pull request's publish dry run checks the package.
2. Once it is merged, push the tag `playwright-vX.Y.Z` on that commit of `main`: `git tag playwright-vX.Y.Z <sha> && git push origin playwright-vX.Y.Z`. The tag runs the rows, then publishes to npm (`.github/workflows/playwright-publish.yml`).

Push the tag as a plain git tag. Never create a GitHub release for it: a published GitHub release also starts any workflow of this repository that runs on releases, such as the root `@keploy/sdk` release workflow (keploy/typescript-sdk#117), which would try to publish that package under the tag's name and fail.
118 changes: 118 additions & 0 deletions playwright/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading