Repository navigation
fix: ship @keploy/sdk without typescript, and check licenses on publish - #123
Merged
Merged
Conversation
@keploy/sdk listed typescript under dependencies, so every install of the SDK pulled in typescript, whose ThirdPartyNoticeText.txt carries the Unicode, Inc. license agreement that license scanners flag. Nothing imports typescript at run time; it only builds dist/. It moves to devDependencies, so an install of the SDK no longer brings it. - package.json said "license": "MIT", while LICENSE, the README badge and the jest and playwright packages say Apache-2.0. It now says Apache-2.0. - package.json lists the files the package ships: dist/ and the sources that 2.0.10 shipped. Without that list, a pack of main also carried jest/ and playwright/ (sources, rows, the e2e lane and their lockfiles, which name a CC-BY-4.0 package). - The version is 2.0.11 (npm has 2.0.10; package.json said 2.0.2). - @types/express and @types/node stay in dependencies: dist/v2/dedup/middleware.d.ts imports express's types, which import node's, so a TypeScript user of the middleware needs them. Both trees are MIT. sdk-publish.yml publishes @keploy/sdk from an sdk-vX.Y.Z tag, the way the jest and playwright packages publish: the tag must name package.json's version, the rows run, the package is packed once, the tarball must carry its build, scripts/license-gate.mjs checks it, and that same tarball is published with provenance. On a pull request the same pack and checks run, with no token. scripts/license-gate.mjs installs the tarball into an empty project as a user's npm install does (no devDependencies), then fails when any installed package's license field names an id outside the allow-list (MIT, BSD-2-Clause, BSD-3-Clause, 0BSD, Apache-2.0, ISC, Zlib, BSL-1.0, CC0-1.0, Unlicense, PostgreSQL, OFL-1.1), when a field is missing, and when any installed file names a flagged license by name, SPDX id or URL (GPL, LGPL, AGPL, MPL, Unicode terms, SSPL, BUSL, EPL, CDDL, EUPL, OSL, Creative Commons, Elastic, Commons Clause). It has no exception list. Checked by installing each package into an empty project: - @keploy/sdk@2.0.10 from npm: the check fails on node_modules/typescript/ThirdPartyNoticeText.txt (53 packages). - main, packed: the same, plus CC-BY in the two jest lockfiles it ships. - this commit, packed: 52 packages, every license allowed (51 MIT, the SDK Apache-2.0). The installed tree is 2.0.10's without typescript, and dist/*.js and dist/*.d.ts are byte for byte those of 2.0.10. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The license check missed spellings a scanner flags ("CC BY 4.0",
"MPL 2.0", an lgpl-3.0 page on choosealicense.com, opensource.org's
/license/ URLs), and nothing pinned that it reads every file whole, or
installs optional and peer dependencies. It now:
- matches text normalised: lower case, any run of spaces, hyphens,
underscores and dots as one separator, each license written once as
its ids, names and URLs. License pages on opensource.org (/licenses/
and /license/), choosealicense.com and spdx.org must name an allowed
license, and so must every SPDX-License-Identifier line.
- reads every installed file whole, a block at a time with a bounded
window, whatever its name or size; gzip data decompressed, UTF-16 by
its byte order mark.
- installs the tarball for linux x64 and arm64, darwin arm64 and x64 and
win32 x64 (npm's --os and --cpu), so a dependency of another platform
is checked too.
- leaves out what only the package's own peerDependencies install: the
user's jest or @playwright/test, which the package does not ship. The
peers of its dependencies are checked.
- states its limits in its header: a license it does not name passes
when only file text names it; musl-only optional dependencies are not
installed, since npm checks the runner's real C library.
playwright-publish.yml and jest-publish.yml now build, pack once, check
that tarball and publish it, as sdk-publish.yml does. Every publish job
has a 30 minute timeout, and test/workflows.test.mjs holds the three
workflows to that.
The dedup entry points require js-yaml and body-parser, which were not
declared, so they failed with MODULE_NOT_FOUND unless the app had them.
Both are dependencies now: js-yaml ^3.15.2 (js-yaml 4 needs argparse 2,
licensed Python-2.0, outside the allow-list) and body-parser ^1.20.9.
test/package.test.mjs requires each entry point from a clean install.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A license page URL of 92 characters that ends up to 63 characters before the end of a block is found once the next block is read. A window kept any shorter than the gate keeps loses it. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The walk row gains a flagged file of each kind a scan might skip: .js, .cjs, .mjs, .d.ts, .js.map, README.md and a 300 KiB minified file. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The jest and playwright publish and dry-run jobs now run
test/workflows.test.mjs from the repository's root before their license
check, so no tag or pull request that runs one of them skips the rows that
hold it to the check. sdk-publish.yml's check also runs on a change to
either workflow.
The rows now also fail when a job that packs, checks or publishes, or one
of its steps, carries an if beyond the job's event or a continue-on-error;
when a job loses its rows step; when a workflow's tag trigger changes, or
its pull request trigger gains a filter or stops covering it, the rows or
the check; when a job of a workflow one calls has no timeout; and
when a workflow that runs npm publish is not one they hold. The rows jobs
of jest-sdk.yml and playwright-sdk.yml get a 30-minute timeout.
The license check now:
- flags the Creative Commons 3.0 Unported legal code by its opening words
("Creative Commons Legal Code / Attribution");
- installs with optional and peer dependencies in, whatever an npmrc or an
npm_config_ variable says;
- leaves to the user only the peers jest, jest-environment-node and
@playwright/test: any other peerDependency of the package fails.
The dedup middleware reads and writes dedupData.yaml with yaml (ISC, no
dependencies) in place of js-yaml 3, which brought in esprima, last
released in 2018. It writes the file byte for byte as js-yaml 4 did,
reads the files js-yaml 3 and 4 wrote, and loads no js/ tag as code.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The dedup middleware read, parsed, re-serialised and rewrote the whole dedupData.yaml on every request, so each request cost time and memory in proportion to a file that only grows, with the event loop blocked. It now appends the new entry's text, written by our own code (v2/dedup/entry.ts) as js-yaml 4's dump wrote it, and needs no YAML package: yaml leaves the dependencies (it stays a devDependency for the workflow rows), and nothing new joins the installed tree. Rows: one request on a file of 1000 entries makes one appendFileSync of its own entry's bytes and reads nothing; 2.0.10's files, as js-yaml 3 and 4 wrote them, keep their bytes and both readers read back every entry; over 335 names, ids and lines the entry is js-yaml 4's text but where entry.ts's limits say, and both readers read the same values. The license gate installs for musl Linux too: linux x64 and arm64 each with glibc and with musl, --libc passed on every linux leg, so a red optional dependency for one C library only is found whatever npm_config_libc says. A row covers both C libraries under three settings. The workflow rows pin each Licenses step to its name, TARBALL and its command. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
entry.ts double-quoted an id over 76 characters, as a stand-in for the folded block js-yaml 4 writes there. No reader needs it: a plain value of any length on one line is valid YAML, and only an implicit key has a length limit (over 1024 characters a key is still written explicit). The rule had no row that could fail on it, so it goes: a long id is plain or quoted by the same rules as a short one. The row over 335 names, ids and lines still shows js-yaml 3 and 4 read every entry back as the same values, with the same list of entries whose text differs from js-yaml 4's. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The dedup middleware's setup checked for dedupData.yaml asynchronously
and, when the check found it missing, wrote it empty in its callback.
Requests that finished before that callback lost their entries: 40
requests in the setup's tick left 0 bytes. The setup now creates a
missing file at once with an append of nothing, which never empties
it, and as before writes nothing to a file that exists. Rows run those
40 requests and read back 40 entries, and start an app whose
dedupData.yaml is a folder.
A name that starts with "._" is now quoted: js-yaml 3 reads plain "._1"
as 0.1 and "._e0" as NaN. The differential row covers three such names.
The bytes row wraps fs and fs.promises (require('fs/promises') is the
same object) before the middleware loads and logs each call as it is
made, from the setup to the exit: only the setup's existsSync and the
request's append may show. A whole-file read at setup, through
promises, deferred, in an exit listener or through a function bound at
load fails it.
The workflow rows now fail on an env on a publish workflow or any of its
jobs; a pack, check or publish job key outside the allowed set (a
container, say), a runner other than ubuntu-latest, a defaults.run other
than bash and the job's folder, or an action other than the pinned
checkout and setup-node with their three inputs (the registry npmjs.org,
the scope @keploy); a step env beyond TARBALL, NPM_TOKEN and
NODE_AUTH_TOKEN; a rows step key beyond its name, command and folder;
and a command that names GITHUB_ENV, GITHUB_PATH, ::set-env or
::add-path, in any case. Each could give the license check's node a
NODE_OPTIONS that exits 0. playwright-publish.yml now pins its actions
by commit, as the other two publish workflows do.
The license gate's header names no unreleased tool.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
This was referenced Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
@keploy/sdkliststypescriptunderdependencies, so everynpm install @keploy/sdkalso installs typescript (5.9.3 today). ItsThirdPartyNoticeText.txtcarries the Unicode, Inc. license agreement and Creative Commons Attribution text, which license scanners flag in the SDK's dependency tree. Nothing imports typescript at run time: it only buildsdist/.More problems in what the package declares and ships:
package.jsonsays"license": "MIT", whileLICENSE, the README badge and the jest and playwright packages say Apache-2.0.package.jsonhas nofileslist, so a pack of main also shipsjest/andplaywright/: their sources, rows, e2e lane and lockfiles. Those lockfiles name a CC-BY-4.0 package.dist/v2/dedup/middleware.jsrequiresjs-yamlanddist/v2/dedup/register.jsrequiresbody-parser, but neither is declared, so both entry points fail withMODULE_NOT_FOUNDunless the app happens to have them.There is no publish workflow for this package, and nothing checks the licenses of what an install of
@keploy/sdk,@keploy/playwrightor@keploy/jestbrings.Change
typescriptmoves todevDependencies.The license field becomes
Apache-2.0.fileslistsdist,keployCli.tsandv2/. 2.0.10 also shipped 17 repository files that the package no longer carries:.github/,.husky/,tsconfig.json,.eslintrc,.editorconfig,.commitlintrc.json,CODE_OF_CONDUCT.md,CONTRIBUTING.mdandSECURITY.md.The version becomes 2.0.11 (npm has 2.0.10, and
package.jsonsaid 2.0.2).body-parser(^1.20.9, express 4's line) becomes a dependency:register.jsneeds it.The dedup middleware needs no YAML package now. 2.0.10's required whichever js-yaml the app happened to have (nyc's js-yaml 3, most often), and on every request read, parsed, re-serialised and rewrote the whole
dedupData.yaml, which gains an entry a request and is never cut: each request cost time and memory in proportion to the file, with the event loop blocked. It now appends the new entry's text to the file, written by our own code (v2/dedup/entry.ts, standard library only). js-yaml 4 is not used because it needs argparse 2, licensed Python-2.0, which is outside the allow-list; js-yaml 3 brings in esprima, last released in 2018.fs.promisesand a function the module binds when it loads included) are the setup'sexistsSyncand oneappendFileSyncof the request's own entry's bytes.noor2026-11-04. A row runs the installed middleware on 2.0.10's files, as js-yaml 3 and as js-yaml 4 wrote them: the file left is the old file, then the new entry, and js-yaml 3 and 4 each read back the three entries.entry.ts's limits say, and there js-yaml 3 and 4 still read the same values: a name that starts like a number (1_000,+.5) or with._is quoted where js-yaml 4 leaves it plain (js-yaml 3 reads js-yaml 4's plain1_000as 1000,._1as 0.1 and._e0as NaN); an id with a line break is double-quoted, and one over 76 characters stays on one line, where js-yaml 4 writes a block; a line that is not a number is written as JSON.!!js/functiontag in it loads nothing (a row). One limit: a file that is not such a list (edited by hand into a mapping, say) was reset by 2.0.10 to the one new entry, dropping what it held, with an error logged; the entry is now appended after it. A file removed after the setup is created again by the append, without 2.0.10's "Error reading existing file" line.dedupData.yamlat once, by appending nothing to it, so it never empties the file. 2.0.10 checked for the file asynchronously and, when the check found it missing, wrote it empty in the check's callback, so requests that finished before that callback lost their entries: 40 requests in the setup's tick left 0 bytes. A row runs those 40 requests and reads back 40 entries; on 2.0.10's setup it reads none. As in 2.0.10, the setup writes nothing to a file that exists, so an app whosededupData.yamlit may not write still starts (a row). One change: when the file is missing and the folder may not be written,middleware()now throws (EACCES) at setup; 2.0.10 threw the same error a moment later, uncaught, in the check's callback.@types/expressand@types/nodestay independencies. With them moved, a strict TypeScript user of the dedup middleware fails withmiddleware.d.ts(1,49): error TS2307: Cannot find module 'express'.The lockfile stays at version 1. It adds body-parser and body-parser's tree, changes its version, and marks
typescriptas dev.yaml(ISC, no dependencies) is a devDependency, not shipped: the workflow rows parse the workflows with it.New
.github/workflows/sdk-publish.ymlpublishes from ansdk-vX.Y.Ztag:package.json's version;scripts/license-gate.mjschecks that tarball;On a pull request, the same pack and checks run with no token.
playwright-publish.ymlandjest-publish.ymlwork the same way now. They build, pack once, check the tarball's build files and licenses, and publish that tarball. Their pull request dry run checks the same tarball.playwright-publish.ymlnow pins its actions by commit, to the commits the other two use; it namedactions/checkout@v4andactions/setup-node@v4in the job that holds the npm token.test/workflows.test.mjsholds the three workflows to this. Every publish and dry-run job runs it from the repository's root before its license check: the sdk jobs as part of all the rows, the jest and playwright jobs as their own step (npm ci --ignore-scriptsat the root, only for the rows' YAML parser).sdk-publish.ymlalso runs on a pull request that changesjest-publish.ymlorplaywright-publish.yml. So no tag or pull request that runs a publish workflow skips the rows. They fail when:TARBALLand its command (ashellof its own, or another variable such asNODE_OPTIONS);env; a pack, check or publish job has a key beyondname,if,needs,runs-on,timeout-minutes,permissions,defaultsandsteps(acontainer, say), runs on anything butubuntu-latest, has adefaults.runother thanshell: bashand its folder, or runs an action other than the pinnedactions/checkoutandactions/setup-nodewith inputs beyondnode-version,registry-urlandscope(the registry must behttps://registry.npmjs.org, which the license check installs from, and the scope@keploy); a step'senvnames anything butTARBALL,NPM_TOKENandNODE_AUTH_TOKEN; a rows step has a key beyond its name, command and folder; or a step's command namesGITHUB_ENV,GITHUB_PATH,::set-envor::add-path(in any letter case). Each of these could give the license check's node a variable such asNODE_OPTIONSor another shell, or change what the check installs. One limit: the rows read the workflow, not what its commands do, so a command or script that writes the fileGITHUB_ENVpoints to without naming the variable is not caught.ifbeyond the job's event, or acontinue-on-error;npm publishis not one of the three.Every publish and check job has a 30 minute timeout, and so do the rows jobs of
jest-sdk.ymlandplaywright-sdk.yml.New
scripts/license-gate.mjs TARBALLuses only Node's standard library. It installs the tarball into an empty project the way a user'snpm installdoes, with install scripts off. Optional and peer dependencies are installed whatever an npmrc or annpm_config_variable says (--include=optional --include=peer --legacy-peer-deps=false). It does this once for each of linux x64 and arm64, each with glibc and with musl, darwin arm64 and x64, and win32 x64 (npm's--os,--cpuand, on linux,--libc), so a dependency installed only on one of those platforms or C libraries is checked too, whatevernpm_config_libcsays. sharp's LGPL libvips for musl is one such dependency. It fails when:/licenses/or/license/), choosealicense.com or spdx.org names a license that is not allowed, or anSPDX-License-Identifierline names an id that is not allowed;Text is matched normalised: lower case, with any run of spaces, hyphens, underscores and dots as one separator. So "CC BY 4.0", "CC-BY-4.0" and "cc_by_4.0" are one spelling, and each license is written once. There is no exception list, so any finding stops the publish.
A package's own peerDependencies are the user's install, not ours: the user's own jest or playwright tree is already in their project whatever we ship. The gate names them, and does not judge them. Peers of our dependencies are still judged. Only
jest,jest-environment-nodeand@playwright/testare left to the user this way: any other peerDependency of a package fails the check until someone adds it to that list. Jest's own tree carries a CC-BY-4.0 package and BlueOak-1.0.0 packages, and every jest user already has those.The script's header states its limits:
Unicode-3.0is matched only in anSPDX-License-Identifierline, because "Unicode 3.0" in code most often names a version of the standard.Releasing 2.0.11: push the tag
sdk-v2.0.11on the merge commit, as a plain git tag and not as a GitHub release. #117 proposes a workflow that publishes on GitHub releases, and it has no license check; this workflow covers the same release.Verified
Each package was packed, installed into an empty project with
npm install(Node 22, npm 10.9) for the seven platforms, and checked with the script:@keploy/sdk@2.0.10from npmnode_modules/typescript/ThirdPartyNoticeText.txt, Unicode terms (twice) and Creative Commons Attribution (53 packages)@keploy/sdkfrom this branch@keploy/playwright0.1.1 from this branch@playwright/testnot checked@keploy/jest0.1.0 from this branchjestandjest-environment-nodenot checkeddist/,keployCli.ts,v2/,package.json,README.MDandLICENSE.dist/keployCli.js,dist/v2/dedup/register.jsand their.d.tsfiles are byte for byte those of 2.0.10.dist/v2/dedup/middleware.jsdiffers in the setup (it creates a missing file with an append) and in the request's write (it appends the entry), anddist/v2/dedup/entry.jsis new.npm installof the tarball (scripts on) loads all three entry points.npm ci, the build (tsc 5.3.3 from the lockfile) and the 51 rows (node --test test/*.test.mjs) pass. One row requires each entry point (@keploy/sdkand both dedup entry points) from a clean install of the packed tarball.npm_config_legacy_peer_deps=trueandnpm_config_omitset tooptionalor topeer;npm_config_libcset tomuslor toglibc;