Skip to content

chore: remove shelljs from automation scripts - #2502

Merged
r0b1n merged 8 commits into
mainfrom
chore/automation-utils-drop-shelljs
Nov 16, 2026
Merged

r0b1n merged 8 commits into
mainfrom
chore/automation-utils-drop-shelljs

Conversation

@r0b1n

@r0b1n r0b1n commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Pull request type

Refactoring (e.g. file rename, variable rename, etc.)


Description

Removes shelljs from the repo's own scripts (follow-up to #2484), as eight commits that can be reviewed separately. Replacing exec (execa v5) with zx is the next step.

1. Trim shell exports and pass cwd instead of pushd (83f482d)

  • src/shell.ts exported 29 shelljs functions; only the ones in use are kept.
  • pushUpdateToTestProject (steps.ts) and updateChangelogsAndCreatePR (changelog.ts) run their git commands with execa's cwd option instead of pushd/popd.

This fixes two cases where the process was left in a different directory:

  • pushUpdateToTestProject called pushd and then returned early on "Nothing to commit" without popd.
  • zip() only called popd on success, so a failed zip left the process inside the zipped folder.

2. Replace shelljs file helpers with node:fs (b729ca3)

  • rm -rf, mkdir -p and cp call sites now use node:fs/promises, in automation/utils and in the charts-web and data-widgets scripts. shelljs is removed from automation/utils/package.json.
  • shell.ts keeps small helpers only where node:fs has no direct equivalent:
    • findFiles(...patterns): fs.globSync that throws when a pattern matches nothing. shelljs find with fatal failed the same way, so a missing mpk still stops the release instead of being skipped silently.
    • move(source, dest): rename with a copy and remove fallback when crossing file systems, like shelljs mv. This replaces the cp + rm workaround in moveModuleToDist for mv silently swallows errors shelljs/shelljs#878.
    • makeWritableRecursive(dir): the chmod -R a+rw used before re-zipping mpks.
  • ensureFileExists uses fs.access and is now async.
  • zip() switches directories with process.chdir; cross-zip has no cwd option.
  • rui-oss-clearance imports node:fs/promises statically instead of the inline await import(...) calls.

Copying a file into a directory now passes the full destination path, since fs.cp doesn't copy into an existing directory the way shelljs cp does.

pnpm-lock.yaml: removes shelljs 0.8.5.

3. Replace shelljs in run-e2e with node:fs (3961d22)

  • ls, rm, mkdir, mv, cp, test and cat in automation/run-e2e/lib now use node:fs; globs (tests/testProjects-*, *.mpr, dist/<version>/*.mpk) use fs.globSync.
  • gh auth token (dev mode) and the themesource chmod -R +w use spawnSync.
  • shelljs is removed from automation/run-e2e/package.json and from the lockfile.

Behaviour change: shelljs was not in fatal mode in run-e2e, so a failed cp/mv only printed an error and continued. These now throw. The Atlas theme releases (atlasui-theme-files-2024-01-25, -2025-10-08) contain web and native, and the Atlas core releases (3.17.0, 4.5.0) contain themesource, so the copies succeed. A missing results/runtime.log is still reported instead of thrown. A missing widget mpk after the release script throws, but updateTestProject already catches and logs errors from that step.

4. Drop the unused shx dependency of pluggable-widgets-tools (2555853)

@mendix/pluggable-widgets-tools depends on shx, which brings in shelljs 0.9.2 (and an old execa@1 chain), but neither the tools nor this repo call it. It is removed with a "@mendix/pluggable-widgets-tools>shx": "-" override, like the other unused tools dependencies in the root package.json.

The only shelljs left in the lockfile is 0.10.0, used by the tools' rollup config to copy XML, icons, LICENSE and the mpk. The upstream fix is merged but not yet released; the override and that dependency can go once we upgrade the tools.

5. Remove the stale shelljs hoist pattern from .npmrc (949e9f6)

No code in the repo imports shelljs anymore. The public-hoist-pattern[]=*shelljs* line also had no effect, because shamefully-hoist=true already hoists every package to the root; installs are unchanged.

6. Tests for the shell.ts file helpers (29779f4)

automation/utils had no tests. This adds src/__tests__/shell.spec.ts, run with Node's built-in test runner (node --test, no new dependencies) through a new test script, so CI runs it with the other unit tests. It covers ensureFileExists, findFiles, move, makeWritableRecursive (including not following symlinks) and zip/unzip: archive entries at the root, and the working directory restored after both a successful and a failed zip. Putting back the old "no restore on failure" behaviour makes the test fail.

7. Tests for the release steps that rebuild mpks (ac4a10c)

src/__tests__/mpk-archive.spec.ts runs the real release steps against a fake module mpk and then unzips the result to check it:

  • addTestProjectWidgetsToMpk (the same insertWidgetsIntoMpk path as addWidgetsToMpk): widget mpks end up under widgets/, including one with a space in its name, and are listed in package.xml. A nested widget mpk comes back byte-for-byte, and the temp folder is removed.
  • copyFilesToMpk: the files are added and listed in package.xml.
  • includeReadmeOssIntoMpk: the READMEOSS file is added to the existing mpk in place.

The fake mpk contains a read-only file, like mxbuild output (POSIX only). shell.spec.ts also gets zip round trips with spaces in the archive, folder and entry paths, and with binary content. When zip() is changed to zip the wrong folder, 6 tests fail.

8. Run these tests on the release build runners (98d822c)

Unit tests only ran in the ubuntu check job, but the release job runs on ubuntu and Windows, and cross-zip works differently on each: it calls zip/unzip on Linux and PowerShell on Windows. A new "Test release scripts" step runs pnpm --filter @mendix/automation-utils run test in the release job, before the release task. It runs on every PR, not only when automation/utils changes, so a zip break on either runner shows up before a release does. It passes on both runners; the mpk tests take a few seconds on Windows.

What should be covered while testing?

Verified locally:

  • automation/utils type-checks and lints; script lints pass for all modules and charts-web.

  • Helper smoke test: zip/unzip, zip restoring the directory on success and failure, ensureFileExists throwing ENOENT, findFiles matching and throwing, makeWritableRecursive (600 → 666), move.

  • pnpm turbo build --filter=@mendix/charts-web: Charts.mpk contains all 10 widgets merged under com/ plus package.xml and License.txt.

  • pnpm turbo build --filter=@mendix/badge-web --force and its unit tests pass without shx installed.

  • run-e2e: setupTestProject (twice, covering the cleanup branch) and updateTestProject for badge-web: the mpr, Atlas theme/web and theme/native, themesource/atlas_core and Badge.mpk are in place, and the themesource files are writable.

  • build:module for data-widgets (all 9 widget mpks copied, themesource, .version, LICENSE) and web-actions (themesource identical to source).

  • Module release pipeline for data-widgets, run locally with CI unset: the steps from scripts/release.ts, with only the clone swapped for an anonymous one because cloneTestProject needs GH_PAT. That covers createModuleMpk in Docker (mxbuild 10.24.0.73019), addWidgetsToMpk (9 widgets added) and moveModuleToDist. The resulting dist/3.11.8/DataWidgets.mpk has the same file list and the same package.xml entries as the published data-widgets-v3.11.8 asset. The only difference is the READMEOSS HTML, which OSS clearance adds after release. No temp folders were left behind.

Not run locally or in CI, worth watching on the next module release:

  • Release-only paths: the GitHub parts of the OSS clearance scripts (downloading and uploading release assets; includeReadmeOssIntoMpk is now covered by tests) and rui-create-gh-release.
  • pushUpdateToTestProject and the changelog flow, which push to remotes.
  • The EXDEV fallback in move.
  • The Docker parts of run-e2e (ci.mjs, docker-utils.mjs); the E2E jobs exercise them.

@r0b1n
r0b1n requested a review from a team as a code owner October 29, 2026 13:04
@github-actions

This comment has been minimized.

@r0b1n r0b1n changed the title chore(automation-utils): trim shell exports and pass cwd instead of pushd chore(automation-utils): remove shelljs from automation utils Oct 29, 2026
@github-actions

This comment has been minimized.

@r0b1n r0b1n changed the title chore(automation-utils): remove shelljs from automation utils chore: remove shelljs from automation scripts Oct 29, 2026
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

gjulivan
gjulivan previously approved these changes Nov 2, 2026
@r0b1n
r0b1n force-pushed the chore/automation-utils-drop-shelljs branch from c7e0f34 to 0cbc705 Compare November 9, 2026 11:51
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@r0b1n
r0b1n force-pushed the chore/automation-utils-drop-shelljs branch from 58f7782 to ba5f620 Compare November 10, 2026 11:05
@github-actions

This comment has been minimized.

leonardomendix
leonardomendix previously approved these changes Nov 10, 2026
@github-actions

This comment has been minimized.

…ushd

First step towards dropping shelljs. Export only the shelljs helpers that are
used, and run git commands with execa's cwd option instead of pushd/popd.

This also fixes two cases where the process directory was left changed:
pushUpdateToTestProject returned early on "Nothing to commit" without popd,
and zip() skipped popd when zipping failed.
Use node:fs/promises for rm, mkdir and cp in automation/utils and the
charts-web and data-widgets scripts, and remove shelljs from automation/utils.

shell.ts keeps small helpers where node:fs has no direct equivalent:
- findFiles: glob that fails when a pattern matches nothing, like shelljs find
- move: rename with a copy and remove fallback across file systems, like mv
- makeWritableRecursive: chmod -R a+rw
zip() now switches directories with process.chdir instead of pushd/popd.
Use node:fs for ls, rm, mkdir, mv, cp, test and cat in run-e2e, spawnSync for
`gh auth token` and the themesource chmod, and remove shelljs from run-e2e.

shelljs was not in fatal mode here, so a failed cp or mv only printed an
error. They now throw; the Atlas theme and core releases in use contain the
copied folders. A missing runtime.log is still reported instead of thrown.
pluggable-widgets-tools depends on shx, which pulls in shelljs 0.9.2, but
neither the tools nor this repo call it. Remove it with a pnpm override, like
the other unused tools dependencies. The tools still use shelljs 0.10.0 in
their rollup config until a release without it is available.
No code in the repo imports shelljs anymore. The pattern also had no effect,
because shamefully-hoist=true already hoists every package to the root.
Add node:test specs for ensureFileExists, findFiles, move,
makeWritableRecursive and zip/unzip, run by a new test script so CI runs
them with the other unit tests. No new dependencies.
@r0b1n
r0b1n force-pushed the chore/automation-utils-drop-shelljs branch from 98d822c to 3028ca3 Compare November 16, 2026 08:44
@github-actions

Copy link
Copy Markdown
Contributor

AI Code Review

✅ Approved — no issues found


What was reviewed

Area Change
automation/utils (shell.ts, steps.ts, oss-clearance.ts, monorepo.ts, mpk.ts, changelog.ts, bin/*) shelljs replaced with node:fs/promises and execa cwd. New helpers: findFiles, move, makeWritableRecursive. zip() now restores the working directory on failure.
automation/utils/src/__tests__ New shell.spec.ts and mpk-archive.spec.ts, run with node --test.
automation/run-e2e/lib shelljs replaced with node:fs and spawnSync.
packages/pluggableWidgets/charts-web/scripts/build.mts, packages/modules/data-widgets/scripts/start.ts Switched to node:fs/promises.
.github/workflows/BuildJobs.yml New "Test release scripts" step in the release job.
Root package.json, .npmrc, pnpm-lock.yaml shx override added, stale shelljs hoist pattern removed. Lockfile changes only follow from the dependency removals.

Skipped (out of scope): pnpm-lock.yaml details.

CI: pending: Run code quality check, Plan E2E matrix, sigridci · unrelated failures: security/snyk (external scanner; does not point to a code issue in this diff)


Positives

  • Tested the cwd-leaking cases the PR fixes: zip() now restores the working directory on failure, and the git steps use cwd instead of pushd/popd. Tests cover both.
  • findFiles keeps the old fail-on-no-match behaviour of shelljs find under fatal. A missing mpk still stops the release.
  • The mpk round-trip tests run in the release job on both ubuntu and Windows. That is the environment where cross-zip behaves differently.

I checked these as well and found nothing to report:

  • setLocalGitUserInfo(cwd) accepts a working directory.
  • No other package consumes the removed shell exports. Only charts-web imports @mendix/automation-utils/shell, and it was updated.
  • Copy-into-directory call sites now pass the full destination path, as the PR describes.
  • No changelog entry is needed: the changes are in repo automation and build scripts, with no widget behaviour change.

@iobuhov iobuhov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@r0b1n
r0b1n merged commit a657732 into main Nov 16, 2026
20 of 21 checks passed
@r0b1n
r0b1n deleted the chore/automation-utils-drop-shelljs branch November 16, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants