Skip to content

fix: ship @keploy/sdk without typescript, and check licenses on publish - #123

Merged
slayerjain merged 9 commits into
mainfrom
fix/lic-ts-sdk-next
Nov 4, 2026
Merged

slayerjain merged 9 commits into
mainfrom
fix/lic-ts-sdk-next

Conversation

@slayerjain

Copy link
Copy Markdown
Member

Problem

@keploy/sdk lists typescript under dependencies, so every npm install @keploy/sdk also installs typescript (5.9.3 today). Its ThirdPartyNoticeText.txt carries the Unicode, Inc. license agreement and Creative Commons Attribution text, which license scanners flag in the SDK's dependency tree. Nothing imports typescript at run time: it only builds dist/.

More problems in what the package declares and ships:

  • package.json says "license": "MIT", while LICENSE, the README badge and the jest and playwright packages say Apache-2.0.
  • package.json has no files list, so a pack of main also ships jest/ and playwright/: their sources, rows, e2e lane and lockfiles. Those lockfiles name a CC-BY-4.0 package.
  • dist/v2/dedup/middleware.js requires js-yaml and dist/v2/dedup/register.js requires body-parser, but neither is declared, so both entry points fail with MODULE_NOT_FOUND unless the app happens to have them.

There is no publish workflow for this package, and nothing checks the licenses of what an install of @keploy/sdk, @keploy/playwright or @keploy/jest brings.

Change

  • typescript moves to devDependencies.

  • The license field becomes Apache-2.0.

  • files lists dist, keployCli.ts and v2/. 2.0.10 also shipped 17 repository files that the package no longer carries: .github/, .husky/, tsconfig.json, .eslintrc, .editorconfig, .commitlintrc.json, CODE_OF_CONDUCT.md, CONTRIBUTING.md and SECURITY.md.

  • The version becomes 2.0.11 (npm has 2.0.10, and package.json said 2.0.2).

  • body-parser (^1.20.9, express 4's line) becomes a dependency: register.js needs it.

  • The dedup middleware needs no YAML package now. 2.0.10's required whichever js-yaml the app happened to have (nyc's js-yaml 3, most often), and on every request read, parsed, re-serialised and rewrote the whole dedupData.yaml, which gains an entry a request and is never cut: each request cost time and memory in proportion to the file, with the event loop blocked. It now appends the new entry's text to the file, written by our own code (v2/dedup/entry.ts, standard library only). js-yaml 4 is not used because it needs argparse 2, licensed Python-2.0, which is outside the allow-list; js-yaml 3 brings in esprima, last released in 2018.

    • Nothing reads the old entries, so nothing is loaded at start either. A row sets the middleware up on a file of 1000 entries, 26 MiB (27,558,890 bytes), and runs one request: from the setup to the process's exit, the only calls into fs (fs.promises and a function the module binds when it loads included) are the setup's existsSync and one appendFileSync of the request's own entry's bytes.
    • The entry is the text js-yaml 4's dump gave it, with single quotes and YAML 1.1's quoting of names such as no or 2026-11-04. A row runs the installed middleware on 2.0.10's files, as js-yaml 3 and as js-yaml 4 wrote them: the file left is the old file, then the new entry, and js-yaml 3 and 4 each read back the three entries.
    • A row compares the entry with js-yaml 4's dump over 344 cases: names, ids and lines, YAML's words, indicators, escapes and control characters among them, and keys over 1024 characters. The text differs only where entry.ts's limits say, and there js-yaml 3 and 4 still read the same values: a name that starts like a number (1_000, +.5) or with ._ is quoted where js-yaml 4 leaves it plain (js-yaml 3 reads js-yaml 4's plain 1_000 as 1000, ._1 as 0.1 and ._e0 as NaN); an id with a line break is double-quoted, and one over 76 characters stays on one line, where js-yaml 4 writes a block; a line that is not a number is written as JSON.
    • What the file holds is left as it is and never parsed, so a !!js/function tag in it loads nothing (a row). One limit: a file that is not such a list (edited by hand into a mapping, say) was reset by 2.0.10 to the one new entry, dropping what it held, with an error logged; the entry is now appended after it. A file removed after the setup is created again by the append, without 2.0.10's "Error reading existing file" line.
    • The setup creates a missing dedupData.yaml at once, by appending nothing to it, so it never empties the file. 2.0.10 checked for the file asynchronously and, when the check found it missing, wrote it empty in the check's callback, so requests that finished before that callback lost their entries: 40 requests in the setup's tick left 0 bytes. A row runs those 40 requests and reads back 40 entries; on 2.0.10's setup it reads none. As in 2.0.10, the setup writes nothing to a file that exists, so an app whose dedupData.yaml it may not write still starts (a row). One change: when the file is missing and the folder may not be written, middleware() now throws (EACCES) at setup; 2.0.10 threw the same error a moment later, uncaught, in the check's callback.
  • @types/express and @types/node stay in dependencies. With them moved, a strict TypeScript user of the dedup middleware fails with middleware.d.ts(1,49): error TS2307: Cannot find module 'express'.

  • The lockfile stays at version 1. It adds body-parser and body-parser's tree, changes its version, and marks typescript as dev. yaml (ISC, no dependencies) is a devDependency, not shipped: the workflow rows parse the workflows with it.

  • New .github/workflows/sdk-publish.yml publishes from an sdk-vX.Y.Z tag:

    • the tag must name package.json's version;
    • the rows run;
    • the package is packed once, and the tarball must carry its build;
    • scripts/license-gate.mjs checks that tarball;
    • that same tarball is published with provenance.

    On a pull request, the same pack and checks run with no token.

  • playwright-publish.yml and jest-publish.yml work the same way now. They build, pack once, check the tarball's build files and licenses, and publish that tarball. Their pull request dry run checks the same tarball. playwright-publish.yml now pins its actions by commit, to the commits the other two use; it named actions/checkout@v4 and actions/setup-node@v4 in the job that holds the npm token.

  • test/workflows.test.mjs holds the three workflows to this. Every publish and dry-run job runs it from the repository's root before its license check: the sdk jobs as part of all the rows, the jest and playwright jobs as their own step (npm ci --ignore-scripts at the root, only for the rows' YAML parser). sdk-publish.yml also runs on a pull request that changes jest-publish.yml or playwright-publish.yml. So no tag or pull request that runs a publish workflow skips the rows. They fail when:

    • a job publishes anything but the tarball it packed and license-checked;
    • a Licenses step holds anything but its name, TARBALL and its command (a shell of its own, or another variable such as NODE_OPTIONS);
    • the workflow or any job sets env; a pack, check or publish job has a key beyond name, if, needs, runs-on, timeout-minutes, permissions, defaults and steps (a container, say), runs on anything but ubuntu-latest, has a defaults.run other than shell: bash and its folder, or runs an action other than the pinned actions/checkout and actions/setup-node with inputs beyond node-version, registry-url and scope (the registry must be https://registry.npmjs.org, which the license check installs from, and the scope @keploy); a step's env names anything but TARBALL, NPM_TOKEN and NODE_AUTH_TOKEN; a rows step has a key beyond its name, command and folder; or a step's command names GITHUB_ENV, GITHUB_PATH, ::set-env or ::add-path (in any letter case). Each of these could give the license check's node a variable such as NODE_OPTIONS or another shell, or change what the check installs. One limit: the rows read the workflow, not what its commands do, so a command or script that writes the file GITHUB_ENV points to without naming the variable is not caught.
    • a pack, check or publish job, or any of its steps, has an if beyond the job's event, or a continue-on-error;
    • a job loses its rows step;
    • a workflow's tag trigger changes, or its pull request trigger gains a filter or stops covering the workflow, the rows or the license script;
    • a job, those of a called workflow included, has no timeout of 60 minutes or less;
    • a workflow that runs npm publish is not one of the three.
  • Every publish and check job has a 30 minute timeout, and so do the rows jobs of jest-sdk.yml and playwright-sdk.yml.

  • New scripts/license-gate.mjs TARBALL uses only Node's standard library. It installs the tarball into an empty project the way a user's npm install does, with install scripts off. Optional and peer dependencies are installed whatever an npmrc or an npm_config_ variable says (--include=optional --include=peer --legacy-peer-deps=false). It does this once for each of linux x64 and arm64, each with glibc and with musl, darwin arm64 and x64, and win32 x64 (npm's --os, --cpu and, on linux, --libc), so a dependency installed only on one of those platforms or C libraries is checked too, whatever npm_config_libc says. sharp's LGPL libvips for musl is one such dependency. It fails when:

    • any installed package's license field names an id outside the allow-list (MIT, BSD-2-Clause, BSD-3-Clause, 0BSD, Apache-2.0, ISC, Zlib, BSL-1.0, CC0-1.0, Unlicense, PostgreSQL, OFL-1.1), or a package has no license field;
    • any installed file names a flagged license by id, name or URL: GPL, LGPL, AGPL, MPL, Unicode terms, SSPL, BUSL, EPL, CDDL, EUPL, OSL, CPL, CPAL, the Creative Commons Attribution licenses (CC BY and its NC, ND and SA variants, by id, URL, English name, or the heading of the 3.0 Unported legal code), ODbL, Elastic, Commons Clause, APSL, NPL, RPL, MS-RL or Sleepycat. Every file is read whole, whatever its name or size, a block at a time with a bounded window. Gzip data is decompressed, and UTF-16 text is read by its byte order mark;
    • a license page URL on opensource.org (/licenses/ or /license/), choosealicense.com or spdx.org names a license that is not allowed, or an SPDX-License-Identifier line names an id that is not allowed;
    • an install fails, or puts nothing on disk.

    Text is matched normalised: lower case, with any run of spaces, hyphens, underscores and dots as one separator. So "CC BY 4.0", "CC-BY-4.0" and "cc_by_4.0" are one spelling, and each license is written once. There is no exception list, so any finding stops the publish.

    A package's own peerDependencies are the user's install, not ours: the user's own jest or playwright tree is already in their project whatever we ship. The gate names them, and does not judge them. Peers of our dependencies are still judged. Only jest, jest-environment-node and @playwright/test are left to the user this way: any other peerDependency of a package fails the check until someone adds it to that list. Jest's own tree carries a CC-BY-4.0 package and BlueOak-1.0.0 packages, and every jest user already has those.

    The script's header states its limits:

    • a license the list does not name passes when only a file's text names it (a license field naming it fails). Such text is caught only once a fuller license scanner replaces this script.
    • Unicode-3.0 is matched only in an SPDX-License-Identifier line, because "Unicode 3.0" in code most often names a version of the standard.
    • a name split by anything but those separators is not matched, and neither is one spelt in non-ASCII letters.
    • a license text in a language other than English passes unless it also names a flagged id, English name or URL. The Creative Commons ports to German, French, Dutch, Austrian or Japanese law are such texts.
    • only gzip is decompressed, one level deep.
    • only the seven platforms above are installed for: an optional dependency only for another (win32 arm64, linux arm, ia32, the BSDs or AIX, say) is not installed and not checked.

Releasing 2.0.11: push the tag sdk-v2.0.11 on the merge commit, as a plain git tag and not as a GitHub release. #117 proposes a workflow that publishes on GitHub releases, and it has no license check; this workflow covers the same release.

Verified

Each package was packed, installed into an empty project with npm install (Node 22, npm 10.9) for the seven platforms, and checked with the script:

package result
@keploy/sdk@2.0.10 from npm fails: node_modules/typescript/ThirdPartyNoticeText.txt, Unicode terms (twice) and Creative Commons Attribution (53 packages)
main, packed fails: the same, plus CC-BY in the two jest lockfiles shipped inside it
@keploy/sdk from this branch passes: 83 packages, 2058 files on each platform, every license allowed
@keploy/playwright 0.1.1 from this branch passes: 1 package, 10 files; 3 packages of its peer @playwright/test not checked
@keploy/jest 0.1.0 from this branch passes: 1 package, 8 files; 284 packages of its peers jest and jest-environment-node not checked
  • The SDK tarball holds 15 files: dist/, keployCli.ts, v2/, package.json, README.MD and LICENSE. dist/keployCli.js, dist/v2/dedup/register.js and their .d.ts files are byte for byte those of 2.0.10. dist/v2/dedup/middleware.js differs in the setup (it creates a missing file with an append) and in the request's write (it appends the entry), and dist/v2/dedup/entry.js is new.
  • The installed tree is 2.0.10's without typescript, plus body-parser and body-parser's tree: no YAML package. Every package in it is MIT (82), ISC (3), BSD-3-Clause (1) or Apache-2.0 (the SDK itself), and a plain npm install of the tarball (scripts on) loads all three entry points.
  • npm ci, the build (tsc 5.3.3 from the lockfile) and the 51 rows (node --test test/*.test.mjs) pass. One row requires each entry point (@keploy/sdk and both dedup entry points) from a clean install of the packed tarball.
  • Rows also pin the gate's behaviour:
    • each pattern of the gate has a sample;
    • text fed one character at a time, or in blocks, gives the findings of the whole text;
    • files over 3 MiB, gzip and UTF-16 files are read;
    • files of every kind are read in the walk: .js, .cjs, .mjs, .d.ts, .js.map, README.md, and a 300 KiB minified file;
    • an optional dependency, a dependency for darwin only and one for win32 only, and the peer of a dependency are each installed and flagged, also with npm_config_legacy_peer_deps=true and npm_config_omit set to optional or to peer;
    • an optional dependency for musl Linux only and one for glibc Linux only are each installed and flagged on their own legs, also with npm_config_libc set to musl or to glibc;
    • a peerDependency of the package other than the three above fails;
    • the opening lines of the CC BY 3.0 and CC BY-SA 3.0 Unported legal code are flagged, and CC0's legal code is not.

@keploy/sdk listed typescript under dependencies, so every install of the
SDK pulled in typescript, whose ThirdPartyNoticeText.txt carries the
Unicode, Inc. license agreement that license scanners flag. Nothing
imports typescript at run time; it only builds dist/. It moves to
devDependencies, so an install of the SDK no longer brings it.

- package.json said "license": "MIT", while LICENSE, the README badge and
  the jest and playwright packages say Apache-2.0. It now says Apache-2.0.
- package.json lists the files the package ships: dist/ and the sources
  that 2.0.10 shipped. Without that list, a pack of main also carried
  jest/ and playwright/ (sources, rows, the e2e lane and their lockfiles,
  which name a CC-BY-4.0 package).
- The version is 2.0.11 (npm has 2.0.10; package.json said 2.0.2).
- @types/express and @types/node stay in dependencies:
  dist/v2/dedup/middleware.d.ts imports express's types, which import
  node's, so a TypeScript user of the middleware needs them. Both trees
  are MIT.

sdk-publish.yml publishes @keploy/sdk from an sdk-vX.Y.Z tag, the way the
jest and playwright packages publish: the tag must name package.json's
version, the rows run, the package is packed once, the tarball must
carry its build, scripts/license-gate.mjs checks it, and that same
tarball is published with provenance. On a pull request the same pack
and checks run, with no token.

scripts/license-gate.mjs installs the tarball into an empty project as
a user's npm install does (no devDependencies), then fails when any
installed package's license field names an id outside the allow-list
(MIT, BSD-2-Clause, BSD-3-Clause, 0BSD, Apache-2.0, ISC, Zlib, BSL-1.0,
CC0-1.0, Unlicense, PostgreSQL, OFL-1.1), when a field is missing, and
when any installed file names a flagged license by name, SPDX id or URL
(GPL, LGPL, AGPL, MPL, Unicode terms, SSPL, BUSL, EPL, CDDL, EUPL, OSL,
Creative Commons, Elastic, Commons Clause). It has no exception list.

Checked by installing each package into an empty project:
- @keploy/sdk@2.0.10 from npm: the check fails on
  node_modules/typescript/ThirdPartyNoticeText.txt (53 packages).
- main, packed: the same, plus CC-BY in the two jest lockfiles it ships.
- this commit, packed: 52 packages, every license allowed (51 MIT, the
  SDK Apache-2.0). The installed tree is 2.0.10's without typescript, and
  dist/*.js and dist/*.d.ts are byte for byte those of 2.0.10.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The license check missed spellings a scanner flags ("CC BY 4.0",
"MPL 2.0", an lgpl-3.0 page on choosealicense.com, opensource.org's
/license/ URLs), and nothing pinned that it reads every file whole, or
installs optional and peer dependencies. It now:
- matches text normalised: lower case, any run of spaces, hyphens,
  underscores and dots as one separator, each license written once as
  its ids, names and URLs. License pages on opensource.org (/licenses/
  and /license/), choosealicense.com and spdx.org must name an allowed
  license, and so must every SPDX-License-Identifier line.
- reads every installed file whole, a block at a time with a bounded
  window, whatever its name or size; gzip data decompressed, UTF-16 by
  its byte order mark.
- installs the tarball for linux x64 and arm64, darwin arm64 and x64 and
  win32 x64 (npm's --os and --cpu), so a dependency of another platform
  is checked too.
- leaves out what only the package's own peerDependencies install: the
  user's jest or @playwright/test, which the package does not ship. The
  peers of its dependencies are checked.
- states its limits in its header: a license it does not name passes
  when only file text names it; musl-only optional dependencies are not
  installed, since npm checks the runner's real C library.

playwright-publish.yml and jest-publish.yml now build, pack once, check
that tarball and publish it, as sdk-publish.yml does. Every publish job
has a 30 minute timeout, and test/workflows.test.mjs holds the three
workflows to that.

The dedup entry points require js-yaml and body-parser, which were not
declared, so they failed with MODULE_NOT_FOUND unless the app had them.
Both are dependencies now: js-yaml ^3.15.2 (js-yaml 4 needs argparse 2,
licensed Python-2.0, outside the allow-list) and body-parser ^1.20.9.
test/package.test.mjs requires each entry point from a clean install.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A license page URL of 92 characters that ends up to 63 characters before
the end of a block is found once the next block is read. A window kept
any shorter than the gate keeps loses it.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The walk row gains a flagged file of each kind a scan might skip: .js,
.cjs, .mjs, .d.ts, .js.map, README.md and a 300 KiB minified file.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The jest and playwright publish and dry-run jobs now run
test/workflows.test.mjs from the repository's root before their license
check, so no tag or pull request that runs one of them skips the rows that
hold it to the check. sdk-publish.yml's check also runs on a change to
either workflow.

The rows now also fail when a job that packs, checks or publishes, or one
of its steps, carries an if beyond the job's event or a continue-on-error;
when a job loses its rows step; when a workflow's tag trigger changes, or
its pull request trigger gains a filter or stops covering it, the rows or
the check; when a job of a workflow one calls has no timeout; and
when a workflow that runs npm publish is not one they hold. The rows jobs
of jest-sdk.yml and playwright-sdk.yml get a 30-minute timeout.

The license check now:
- flags the Creative Commons 3.0 Unported legal code by its opening words
  ("Creative Commons Legal Code / Attribution");
- installs with optional and peer dependencies in, whatever an npmrc or an
  npm_config_ variable says;
- leaves to the user only the peers jest, jest-environment-node and
  @playwright/test: any other peerDependency of the package fails.

The dedup middleware reads and writes dedupData.yaml with yaml (ISC, no
dependencies) in place of js-yaml 3, which brought in esprima, last
released in 2018. It writes the file byte for byte as js-yaml 4 did,
reads the files js-yaml 3 and 4 wrote, and loads no js/ tag as code.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The dedup middleware read, parsed, re-serialised and rewrote the whole
dedupData.yaml on every request, so each request cost time and memory in
proportion to a file that only grows, with the event loop blocked. It now
appends the new entry's text, written by our own code (v2/dedup/entry.ts)
as js-yaml 4's dump wrote it, and needs no YAML package: yaml leaves the
dependencies (it stays a devDependency for the workflow rows), and
nothing new joins the installed tree. Rows: one request on a file of
1000 entries makes one appendFileSync of its own entry's bytes and reads
nothing; 2.0.10's files, as js-yaml 3 and 4 wrote them, keep their bytes
and both readers read back every entry; over 335 names, ids and lines the
entry is js-yaml 4's text but where entry.ts's limits say, and both
readers read the same values.

The license gate installs for musl Linux too: linux x64 and arm64 each
with glibc and with musl, --libc passed on every linux leg, so a red
optional dependency for one C library only is found whatever
npm_config_libc says. A row covers both C libraries under three settings.
The workflow rows pin each Licenses step to its name, TARBALL and its
command.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
entry.ts double-quoted an id over 76 characters, as a stand-in for the
folded block js-yaml 4 writes there. No reader needs it: a plain value
of any length on one line is valid YAML, and only an implicit key has a
length limit (over 1024 characters a key is still written explicit).
The rule had no row that could fail on it, so it goes: a long id is
plain or quoted by the same rules as a short one. The row over 335
names, ids and lines still shows js-yaml 3 and 4 read every entry back
as the same values, with the same list of entries whose text differs
from js-yaml 4's.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The dedup middleware's setup checked for dedupData.yaml asynchronously
and, when the check found it missing, wrote it empty in its callback.
Requests that finished before that callback lost their entries: 40
requests in the setup's tick left 0 bytes. The setup now creates a
missing file at once with an append of nothing, which never empties
it, and as before writes nothing to a file that exists. Rows run those
40 requests and read back 40 entries, and start an app whose
dedupData.yaml is a folder.

A name that starts with "._" is now quoted: js-yaml 3 reads plain "._1"
as 0.1 and "._e0" as NaN. The differential row covers three such names.

The bytes row wraps fs and fs.promises (require('fs/promises') is the
same object) before the middleware loads and logs each call as it is
made, from the setup to the exit: only the setup's existsSync and the
request's append may show. A whole-file read at setup, through
promises, deferred, in an exit listener or through a function bound at
load fails it.

The workflow rows now fail on an env on a publish workflow or any of its
jobs; a pack, check or publish job key outside the allowed set (a
container, say), a runner other than ubuntu-latest, a defaults.run other
than bash and the job's folder, or an action other than the pinned
checkout and setup-node with their three inputs (the registry npmjs.org,
the scope @keploy); a step env beyond TARBALL, NPM_TOKEN and
NODE_AUTH_TOKEN; a rows step key beyond its name, command and folder;
and a command that names GITHUB_ENV, GITHUB_PATH, ::set-env or
::add-path, in any case. Each could give the license check's node a
NODE_OPTIONS that exits 0. playwright-publish.yml now pins its actions
by commit, as the other two publish workflows do.

The license gate's header names no unreleased tool.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain
slayerjain merged commit 5ef72e4 into main Nov 4, 2026
20 checks passed
@slayerjain
slayerjain deleted the fix/lic-ts-sdk-next branch November 4, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant