Large diffs get slow and can crash the runner: O(n²) list building and unbounded output

[diogokiss]

Note

This issue was opened by an AI agent, which has been previously reviewed and green-lit by a human, @diogokiss.

What happens

On a large diff, changed-files slows down far more than the size of the diff would suggest, and the
job can fail with the runner running out of memory:

Error: write ENOBUFS
    at afterWriteDispatched (node:internal/stream_base_commons:159:15)
    at Socket._writev (node:net:1033:8)
##[error]Unable to process file command 'output' successfully.
##[error]Exception of type 'System.OutOfMemoryException' was thrown.

There are three separate causes. They are reported together because one run hits all three.

How to reproduce

Build a repository whose second commit changes N files under content/, then run the action on a
push event with one filter that matches all of them:

# .bench-filters.yaml
content:
  - 'content/**'

- uses: tj-actions/changed-files@v47
  with:
    json: true
    escape_json: false
    write_output_files: true
    files_yaml_from_source_file: .bench-filters.yaml

I measured it by running the built dist/index.js directly with the environment variables the runner
sets, so the numbers below do not include any runner overhead. This script takes a checkout of this
repository and a file count:

#!/usr/bin/env bash
# usage: bench.sh <path-to-changed-files-checkout> <n-files>
set -euo pipefail
checkout="$1"; n="$2"
work="$(mktemp -d)"; repo="${work}/repo"; mkdir -p "${repo}" "${work}/home"

git -C "${repo}" init -q -b main
git -C "${repo}" config user.email bench@example.com
git -C "${repo}" config user.name bench

seed() {
  python3 -c "
import os, sys
repo, n, body = sys.argv[1], int(sys.argv[2]), sys.argv[3]
for i in range(n):
    d = os.path.join(repo, 'content', 'ns%d' % (i % 40), 'pages', 'g%d' % (i // 500))
    os.makedirs(d, exist_ok=True)
    open(os.path.join(d, 'page-%d.mdx' % i), 'w').write(body)
" "${repo}" "${n}" "$1"
}

seed base;    git -C "${repo}" add -A; git -C "${repo}" commit -qm base
base="$(git -C "${repo}" rev-parse HEAD)"
seed changed; git -C "${repo}" add -A; git -C "${repo}" commit -qm changed
head="$(git -C "${repo}" rev-parse HEAD)"

printf "content:\n  - 'content/**'\n" > "${repo}/.bench-filters.yaml"
printf '{"before":"%s","after":"%s"}\n' "${base}" "${head}" > "${work}/event.json"
: > "${work}/github_output"

# core.getInput reads INPUT_* only. The runner is what applies the defaults declared
# in action.yml, so a standalone run has to set them itself.
eval "$(python3 -c "
import re, shlex, sys
name = None
for line in open(sys.argv[1]):
    m = re.match(r'^  ([a-z0-9_]+):\s*$', line)
    if m:
        name = m.group(1); continue
    d = re.match(r'^    default:\s*(.*)$', line)
    if d and name:
        v = d.group(1).strip()
        if v[:1] in ('\"', \"'\") and v[-1:] == v[:1]:
            v = v[1:-1]
        print('export INPUT_%s=%s' % (name.upper(), shlex.quote(v.replace('\\\\n', '\n'))))
        name = None
" "${checkout}/action.yml")"

# The action writes core.quotepath and diff.relative to the global git config.
export HOME="${work}/home"
export GITHUB_WORKSPACE="${repo}" GITHUB_EVENT_NAME=push
export GITHUB_EVENT_PATH="${work}/event.json" GITHUB_OUTPUT="${work}/github_output"
export GITHUB_REPOSITORY=bench/bench GITHUB_REF=refs/heads/main GITHUB_REF_NAME=main
export INPUT_TOKEN='' INPUT_USE_REST_API=false INPUT_JSON=true INPUT_ESCAPE_JSON=false
export INPUT_WRITE_OUTPUT_FILES=true INPUT_OUTPUT_DIR="${work}/outputs"
export INPUT_FILES_YAML_FROM_SOURCE_FILE=.bench-filters.yaml INPUT_SKIP_INITIAL_FETCH=true
export INPUT_API_URL=https://api.github.com

start=$SECONDS
node "${checkout}/dist/index.js" > "${work}/stdout.log"
echo "n=${n} time=$((SECONDS - start))s github_output=$(wc -c < "${work}/github_output") stdout=$(wc -c < "${work}/stdout.log")"

Measurements

RUNNER_DEBUG was not set for any of these runs.

Files changed	Time	$GITHUB_OUTPUT	stdout
1,000	1.21 s	0.14 MiB	0.25 MiB
5,000	1.74 s	0.72 MiB	1.26 MiB
10,000	2.48 s	1.46 MiB	2.55 MiB
20,000	5.38 s	2.98 MiB	5.20 MiB
40,000	22.64 s	6.01 MiB	10.51 MiB
80,000	91.70 s	12.19 MiB	21.33 MiB

Bytes grow in line with the number of files, but time does not. Every doubling of the diff makes the
run about four times slower: 5.38 s → 22.64 s → 91.70 s for 20,000 → 40,000 → 80,000 files. Note
also that stdout is consistently larger than $GITHUB_OUTPUT, even though debug logging is off.

Cause 1 — debug lines are always written to stdout

@actions/core.debug() always writes to stdout. isDebug() exists but debug() never calls it, so
RUNNER_DEBUG only decides whether the runner shows the line, not whether the action sends it:

function debug(message) {
    issueCommand('debug', {}, message);   // -> process.stdout.write(...)
}

The action calls it with whole file lists already converted to strings. Because these are template
strings, JSON.stringify runs whether or not anyone will read the result. Twelve of these run for
every filter key in src/changedFilesOutput.ts
(lines 39, 68, 91, 114, 137, 160, 183, 205, 237, 273, 326 and 410), and a thirteenth at
L430
when dir_names_deleted_files_include_only_deleted_dirs is on. There are two more at
src/utils.ts#L1463
and src/main.ts#L204.
In that same dir_names mode, L420 and L422 additionally emit one debug line per deleted file.

On a large diff this pushes hundreds of MiB into the stdout pipe that the runner then throws away.
That is where write ENOBUFS comes from.

Suggested fix: wrap these in if (core.isDebug()), with the JSON.stringify inside the check.

Cause 2 — the other_*_files lists are O(n²)

src/changedFilesOutput.ts#L284-L286:

const otherChangedFiles = allOtherChangedFilesPaths.filter(
  (filePath: string) => !allChangedFilesPaths.includes(filePath)
)

Array.includes inside Array.filter scans the whole array for every element. The same pattern
appears at L374-L376
and L474-L476,
and all three run once per filter key. This is what bends the time curve above.

Suggested fix: build a Set once and test against it.

Cause 3 — write_output_files does not replace $GITHUB_OUTPUT

src/utils.ts#L1504:

core.setOutput(key, cleanedValue)

if (writeOutputFiles) {
  ...
  await fs.writeFile(outputFilePath, cleanedValue.replace(/\\"/g, '"'))
}

core.setOutput runs first and always. Setting write_output_files: true writes the files as well
as $GITHUB_OUTPUT, and no input turns the $GITHUB_OUTPUT write off. Each filter key produces 35
outputs, 14 of which are full file lists.

The runner reads that file with File.ReadAllText and applies no size limit to it (unlike step
summaries, which are capped at 1 MB), then copies every value again into its own
Debug($"Set output {pair.Key} = {pair.Value}"). That is where the OutOfMemoryException comes from.

For workflows that already consume the files in output_dir, the $GITHUB_OUTPUT copy is pure cost.

Suggested fix: an input to skip the $GITHUB_OUTPUT write. It would be most useful if it applied
only to the file-list outputs and left the small ones (*_count, any_*, only_*) in place, so that
if: steps.changed.outputs.x_any_changed == 'true' keeps working. Happy to follow whatever name and
shape you prefer.

A real example

A documentation repository (~293,000 tracked files) runs this action on push. A routine
"merge main into branch" commit meant the push range covered three weeks of the default branch:
477,742 files changed. The job ran for 37 minutes and then died with the errors at the top of this
report. With one filter key matching nearly every changed path, the file lists alone come to about
198 MiB written to $GITHUB_OUTPUT.

That particular range was larger than it should have been, for a reason unrelated to this report. But
large diffs are routine in the same repository with no such problem. Five recent commits on the
default branch, each a single squashed commit with one parent, produced by a content sync:

Commit	Files changed
docs: update engine 6000.7	103,526
docs: update engine 6000.6	103,317
docs: update engine 6000.5	96,071
docs: update engine 6000.3	82,651
docs: update engine 6000.0	75,216

Those are correct, ordinary diffs. They reach this action twice: through the Pull Request that creates
them, and again through the deployment workflow on the default branch. At 80,000 files the table above
is about 92 seconds and 21 MiB of stdout per run, so that cost is being paid on every one of them.

Related issues

I searched open and closed issues for ENOBUFS, OutOfMemoryException, "out of memory",
GITHUB_OUTPUT, write_output_files, RUNNER_DEBUG, "quadratic", "slow", "performance", "timeout",
"large repository" and "many files", and could not find this reported. Two that are nearby:

• [BUG] Maximum call stack size exceeded #1167 (closed) reported "Maximum call stack size exceeded" on a 210,000 file repository and was
  fixed in v36. Different symptom and different cause, but the same area, and the discussion there
  ended with more performance work being planned.
• [Feature] pin-two-dot-endpoints #2932 is about which two commits get compared and about avoiding fetch-depth: 0. That is a
  separate question from what processing a diff costs once the two endpoints are chosen.

Environment

• tj-actions/changed-files v47 (24d32ffd492484c1d75e0c0b894501ddb9d30d62); all three still present
  on main at 934b2d2c7e653bb8c968afed5a0428617f09aa24, including with @actions/core 2.0.2
• Ubuntu runner, Node 24, git 2.54.0
• use_rest_api: false

---

What fixing causes 1 and 2 gives

I have both working locally. Same measurement, same repositories:

Files changed	Time before	Time after	stdout before	stdout after
1,000	1.21 s	1.65 s	0.25 MiB	0.00 MiB
5,000	1.74 s	1.50 s	1.26 MiB	0.00 MiB
10,000	2.48 s	1.43 s	2.55 MiB	0.00 MiB
20,000	5.38 s	1.75 s	5.20 MiB	0.00 MiB
40,000	22.64 s	1.78 s	10.51 MiB	0.00 MiB
80,000	91.70 s	2.17 s	21.33 MiB	0.00 MiB

Time is close to flat instead of growing four times per doubling. Nothing about the results changes:
on a 20,000 file run every file in output_dir and the whole of $GITHUB_OUTPUT are byte for byte
identical before and after, and with RUNNER_DEBUG=1 both builds emit the same 48 debug lines.

$GITHUB_OUTPUT is untouched by these two fixes, which is what cause 3 is about.

I am happy to open a Pull Request for causes 1 and 2 straight away. Cause 3 adds an input, so I would
rather agree the name and behavior with you first.

[diogokiss]

Note for anyone reading later: I edited the description shortly after opening this, before there were
any replies. The only change is extra evidence in the "A real example" section.

The original example came from a diff range that was larger than it should have been, for a reason
unrelated to this report, which made it weak on its own. I added five ordinary commits from the same
repository's default branch, each a single squashed commit with one parent, changing 75,000 to 104,000
files. Those reach the action with entirely correct endpoints, so they show the cost is paid on normal
work and not only in an unusual case.

Nothing else in the report changed.