From 92720b24f167a3b766b0c0c5333a28cee4b3311e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bern=C3=A1t=20G=C3=A1bor?= Date: Fri, 6 Nov 2026 19:05:52 -0800 Subject: [PATCH] =?UTF-8?q?=F0=9F=90=9B=20fix(cooled-prek):=20admit=20pinn?= =?UTF-8?q?ed=20hook=20versions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hook environments build under the seven-day cutoff, so a repository that bumps a hook to a release younger than that fails cooled-hooks.yaml even though the bump is deliberate. An exact pin in a hook's additional_dependencies, or in its repository's install requirements at rev (install_requires in setup.py or setup.cfg, [project].dependencies in pyproject.toml), now writes the release's upload time plus one second to uv's exclude-newer-package while that release is younger than the cutoff. The pinned release installs, and a newer release of the package uploaded after it stays blocked. Once the release is a week old nothing is written. Its dependencies, ranges, and dev or extra requirements stay cooled. A pin and a [tool.nab] exception for the same package keep the later date. The snippet reads the config with PyYAML from uv under the same cutoff, because the runner's python3 may not have it. It runs under -I, which drops the runner's PYTHONPATH and keeps the repository off sys.path. --- .github/workflows/cooled-hooks.yaml | 2 +- changelog.d/+cooled-prek-pins.bugfix.md | 2 + docs/locked-tools.md | 13 +- docs/pr-checks.md | 14 +- docs/troubleshooting.md | 11 +- tests/cooled_prek/test_cooled_prek.py | 300 +++++++++++++++++++++++- tools/cooled_prek.sh | 158 ++++++++++++- 7 files changed, 474 insertions(+), 26 deletions(-) create mode 100644 changelog.d/+cooled-prek-pins.bugfix.md diff --git a/.github/workflows/cooled-hooks.yaml b/.github/workflows/cooled-hooks.yaml index 833bacc..242293e 100644 --- a/.github/workflows/cooled-hooks.yaml +++ b/.github/workflows/cooled-hooks.yaml @@ -7,7 +7,7 @@ on: type: string default: 7 days exclude-newer-package: - description: Newline-separated name=date cutoffs that override exclude-newer for single packages + description: Newline-separated name=date cutoffs for single locked tools; hook environments ignore them type: string default: "" permissions: {} diff --git a/changelog.d/+cooled-prek-pins.bugfix.md b/changelog.d/+cooled-prek-pins.bugfix.md new file mode 100644 index 0000000..3e6340b --- /dev/null +++ b/changelog.d/+cooled-prek-pins.bugfix.md @@ -0,0 +1,2 @@ +`cooled-prek` admits a hook's exact pinned release up to its upload time while that release is younger than the +seven-day cutoff, so a hook bumped to a release younger than that installs instead of failing `cooled-hooks.yaml`. diff --git a/docs/locked-tools.md b/docs/locked-tools.md index e8f2382..8a74244 100644 --- a/docs/locked-tools.md +++ b/docs/locked-tools.md @@ -93,10 +93,15 @@ configurations, which no guard checks. Input `args` **(required)**: whitespace-separated prek arguments, such as `install-hooks`. prek must be on `PATH`. -A package with a `packages..uploaded-prior-to` exception in `pyproject.toml`'s `[tool.nab]` is admitted up to that -date in hook environments too. So a hook dependency pinned to a release newer than the cutoff, such as `sphinx` for -`docstrfmt`, needs the same temporary exception the release lock uses; delete it once the release is a week old. -Pitfall: hook environments read exceptions only from `pyproject.toml`, not from a standalone `nab.toml`. +A Python hook's exact pin, `name==version`, admits that release up to its upload time while it is younger than the +cutoff, so a hook bumped to a release younger than seven days installs. The pin comes from the hook's +`additional_dependencies` or from its repository's install requirements at the hook's `rev`: `install_requires` in +`setup.py` or `setup.cfg`, or `[project].dependencies` in `pyproject.toml`. Dev, test, and extra requirements are not +read. Once the release is a week old, nothing is written and the normal cutoff applies. A newer release of the pinned +package, uploaded after the pinned one, stays blocked in every hook environment. The pinned release's dependencies and +any ranges stay cooled. A pin and a `packages..uploaded-prior-to` exception for the same package: the later date +wins. `cooled-prek` fetches the hook repository and looks the pinned release up on PyPI; a failed fetch or lookup fails +the run. Pitfall: hook environments read exceptions only from `pyproject.toml`, not from a standalone `nab.toml`. ## Install locked tools: `install-locked-tools` diff --git a/docs/pr-checks.md b/docs/pr-checks.md index f859a69..a05b692 100644 --- a/docs/pr-checks.md +++ b/docs/pr-checks.md @@ -96,14 +96,16 @@ fails instead. needs: [test, hooks] ``` -| Input | Default | Meaning | -| ----------------------- | -------- | -------------------------------------------------------------------------------------- | -| `exclude-newer` | `7 days` | Upload cutoff for the lock reader that installs the caller's locked `release-prepare`. | -| `exclude-newer-package` | empty | Newline-separated `name=date` cutoffs that override `exclude-newer` for one package. | +| Input | Default | Meaning | +| ----------------------- | -------- | ----------------------------------------------------------------------------------------- | +| `exclude-newer` | `7 days` | Upload cutoff for the lock reader that installs the caller's locked `release-prepare`. | +| `exclude-newer-package` | empty | Newline-separated `name=date` cutoffs for one locked tool; hook environments ignore them. | prek comes from the caller's `release-prepare` lock group. Hook environments read cutoff exceptions from `[tool.nab]` in -`pyproject.toml`, as [`cooled-prek`](locked-tools.md#run-prek-under-the-cutoff-cooled-prek) describes. The prek cache is -keyed on `.pre-commit-config.yaml` and `pyproject.toml`, so a changed hook or exception rebuilds its environment. +`pyproject.toml`, and an exact hook pin admits its release while younger than the cutoff, as +[`cooled-prek`](locked-tools.md#run-prek-under-the-cutoff-cooled-prek) describes. The `exclude-newer-package` input only +reaches the locked release tools, never the hook environments. The prek cache is keyed on `.pre-commit-config.yaml` and +`pyproject.toml`, so a changed hook or exception rebuilds its environment. ## Release dry run diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index dc250c4..0231793 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -60,10 +60,13 @@ Run `nab lock --all-groups --all-extras` with the Nab version the lock records, The lock's `[tool.nab] created-at` is missing, has no time zone, or lies ahead of the runner's clock. Regenerate the lock. -**A hook fails to install during a release or maintenance run, with uv finding no matching version.** The hook pins a -release younger than seven days, and [`cooled-prek`](locked-tools.md#run-prek-under-the-cutoff-cooled-prek) builds hook -environments under that cutoff. Wait for the release to age, or add a dated `[tool.nab.packages.]` exception in -`pyproject.toml`; hook environments ignore exceptions in `nab.toml`. +**A hook fails to install during a release or maintenance run, with uv finding no matching version.** A package the hook +resolves to a release younger than seven days is not exactly pinned, and +[`cooled-prek`](locked-tools.md#run-prek-under-the-cutoff-cooled-prek) builds hook environments under that cutoff. An +exact pin (`name==version` in `additional_dependencies` or the hook repository's install requirements) admits its +release while that release is younger than the cutoff, so this is a range, a transitive dependency, or a release newer +than the pin. Wait for the release to age, or add a dated `[tool.nab.packages.]` exception in `pyproject.toml`; +hook environments ignore exceptions in `nab.toml`. ## Release preparation diff --git a/tests/cooled_prek/test_cooled_prek.py b/tests/cooled_prek/test_cooled_prek.py index ae2bb30..df732c8 100644 --- a/tests/cooled_prek/test_cooled_prek.py +++ b/tests/cooled_prek/test_cooled_prek.py @@ -2,12 +2,50 @@ import os import re +import shlex import subprocess +import sys +from datetime import UTC, datetime, timedelta from pathlib import Path +from textwrap import dedent import pytest import yaml from conftest import action +from pytest_httpserver import HTTPServer + +SCRIPT = Path(__file__).resolve().parents[2] / "tools/cooled_prek.sh" + + +def _fake_uv(directory: Path, httpserver: HTTPServer) -> None: + # The snippet runs as `uv run --with pyyaml python -I -`. The fake drops everything but the interpreter and + # imports a sitecustomize from a test-only directory, since -I ignores PYTHONPATH; it sends pypi.org to the server. + site = directory / "pypi-site" + site.mkdir() + base = httpserver.url_for("").rstrip("/") + (site / "sitecustomize.py").write_text( + dedent( + f""" + import urllib.request + from urllib.parse import urlsplit + + base = {base!r} + original_open = urllib.request.urlopen + + def route_open(request, *args, **kwargs): + parsed = urlsplit(request.full_url) + if parsed.hostname == "pypi.org": + request = urllib.request.Request(base + parsed.path, headers=dict(request.header_items())) + return original_open(request, *args, **kwargs) + + urllib.request.urlopen = route_open + """ + ) + ) + code = f"import sys; sys.path.insert(0, {str(site)!r}); import sitecustomize; exec(sys.stdin.read())" + fake = directory / "uv" + fake.write_text(f"#!/bin/sh\nexec {sys.executable!r} -I -c {shlex.quote(code)}\n") + fake.chmod(0o755) def test_no_hook_run_bypasses_cooled_prek() -> None: @@ -41,11 +79,12 @@ def test_prepare_installs_no_hook_it_does_not_run(name: str) -> None: assert "install-hooks" not in str(workflow["jobs"]["assemble"]["steps"]) -def test_cooled_prek_action_passes_each_argument(tmp_path: Path, action_scripts: Path) -> None: +def test_cooled_prek_action_passes_each_argument(tmp_path: Path, action_scripts: Path, httpserver: HTTPServer) -> None: fake = tmp_path / "bin/prek" fake.parent.mkdir() fake.write_text('#!/bin/sh\nprintf "%s\\n" "$@" >"$RUNNER_TEMP/args"\n') fake.chmod(0o755) + _fake_uv(fake.parent, httpserver) result = action( tmp_path, @@ -76,13 +115,14 @@ def test_cooled_prek_action_passes_each_argument(tmp_path: Path, action_scripts: ], ) def test_cooled_prek_builds_hooks_cooled_and_runs_them_with_the_caller_environment( - tmp_path: Path, arguments: list[str], install: list[str] + tmp_path: Path, arguments: list[str], install: list[str], httpserver: HTTPServer ) -> None: fake = tmp_path / "prek" fake.write_text( '#!/bin/sh\nn=$(ls calls 2>/dev/null | wc -l)\nmkdir -p calls/$n\nprintf "%s\\n" "$@" >calls/$n/args\nenv >calls/$n/env\n' ) fake.chmod(0o755) + _fake_uv(tmp_path, httpserver) leaked = dict.fromkeys(("UV_INDEX_URL", "UV_EXCLUDE_NEWER_PACKAGE", "UV_SYSTEM_CERTS", "SSL_CERT_FILE"), "leak") subprocess.run( @@ -113,10 +153,11 @@ def test_cooled_prek_builds_hooks_cooled_and_runs_them_with_the_caller_environme assert leaked.items() <= run_env.items() -def test_cooled_prek_admits_nab_cutoff_exceptions(tmp_path: Path) -> None: +def test_cooled_prek_admits_nab_cutoff_exceptions(tmp_path: Path, httpserver: HTTPServer) -> None: fake = tmp_path / "prek" fake.write_text('#!/bin/sh\ncat "$UV_CONFIG_FILE" >config\n') fake.chmod(0o755) + _fake_uv(tmp_path, httpserver) (tmp_path / "pyproject.toml").write_text( '[tool.nab]\nuploaded-prior-to = "P7D"\npackages.sphinx.uploaded-prior-to = 2026-10-21T00:05:13Z\n' 'packages.release-creator.uploaded-prior-to = "2026-10-26T18:24:46Z"\npackages.uv.decision-order = "stable"\n' @@ -136,10 +177,11 @@ def test_cooled_prek_admits_nab_cutoff_exceptions(tmp_path: Path) -> None: ] -def test_cooled_prek_without_exceptions_sets_no_config_file(tmp_path: Path) -> None: +def test_cooled_prek_without_exceptions_sets_no_config_file(tmp_path: Path, httpserver: HTTPServer) -> None: fake = tmp_path / "prek" fake.write_text('#!/bin/sh\nprintf "%s" "${UV_CONFIG_FILE-unset}" >config\n') fake.chmod(0o755) + _fake_uv(tmp_path, httpserver) (tmp_path / "pyproject.toml").write_text('[tool.nab]\nuploaded-prior-to = "P7D"\n') subprocess.run( @@ -150,3 +192,253 @@ def test_cooled_prek_without_exceptions_sets_no_config_file(tmp_path: Path) -> N ) assert (tmp_path / "config").read_text() == "unset" + + +def _hook_repo(root: Path, files: dict[str, str]) -> Path: + repo = root / "hook" + repo.mkdir() + for name, text in {"README.md": "hook\n", **files}.items(): + (repo / name).write_text(text) + git = ["git", "-C", str(repo), "-c", "user.name=test", "-c", "user.email=test@example.com"] + subprocess.run(["git", "init", "-q", str(repo)], check=True) + subprocess.run([*git, "add", "."], check=True) + subprocess.run([*git, "-c", "commit.gpgsign=false", "commit", "-q", "-m", "release"], check=True) + subprocess.run([*git, "tag", "v1.0.0"], check=True) + return repo + + +def _config(root: Path, repo: Path, hook: dict[str, object]) -> None: + (root / ".pre-commit-config.yaml").write_text( + yaml.safe_dump({"repos": [{"repo": str(repo), "rev": "v1.0.0", "hooks": [{"id": "simplint", **hook}]}]}) + ) + + +def _cooled_prek(root: Path, httpserver: HTTPServer) -> subprocess.CompletedProcess[str]: + fake = root / "prek" + fake.write_text('#!/bin/sh\ncat "${UV_CONFIG_FILE:-/dev/null}" >config\n') + fake.chmod(0o755) + _fake_uv(root, httpserver) + return subprocess.run( + [str(SCRIPT), "install-hooks"], + cwd=root, + env=os.environ | {"PATH": f"{root}{os.pathsep}{os.environ['PATH']}", "NO_PROXY": "localhost,127.0.0.1"}, + capture_output=True, + text=True, + check=False, + ) + + +def _uploaded(days: int) -> datetime: + return datetime.now(UTC).replace(microsecond=0) - timedelta(days=days) + + +def _stamp(value: datetime) -> str: + return value.strftime("%Y-%m-%dT%H:%M:%SZ") + + +def _simple(httpserver: HTTPServer, releases: dict[str, datetime]) -> None: + # One wheel per release on the Simple JSON page of simplint, uploaded at the given time. + files = [ + {"filename": f"simplint-{version}-py3-none-any.whl", "upload-time": uploaded.isoformat()} + for version, uploaded in releases.items() + ] + httpserver.expect_request("/simple/simplint/").respond_with_json({"meta": {"api-version": "1.0"}, "files": files}) + + +def _admitted(uploaded: datetime) -> str: + return f'"simplint" = "{_stamp(uploaded + timedelta(seconds=1))}"' + + +PIN_SOURCES = [ + pytest.param({"setup.py": 'setup(install_requires=["simplint==0.7.1"])\n'}, {"language": "python"}, id="setup-py"), + pytest.param( + {"setup.cfg": "[options]\ninstall_requires =\n simplint==0.7.1\n"}, {"language": "python"}, id="setup-cfg" + ), + pytest.param( + {"pyproject.toml": '[project]\nname = "hook"\ndependencies = ["simplint==0.7.1"]\n'}, + {"language": "python"}, + id="pyproject", + ), + pytest.param( + { + "setup.py": 'setup(install_requires=["simplint==0.7.1"])\n', + ".pre-commit-hooks.yaml": "- id: simplint\n language: python\n entry: simplint\n", + }, + {}, + id="language-from-manifest", + ), + pytest.param( + {}, {"language": "python", "additional_dependencies": ["simplint==0.7.1"]}, id="additional-dependencies" + ), + pytest.param( + {"setup.py": 'setup(install_requires=["simplint[socks]==0.7.1; python_version>=3.10"])\n'}, + {"language": "python"}, + id="extras-and-marker", + ), +] + + +@pytest.mark.parametrize(("files", "hook"), PIN_SOURCES) +def test_cooled_prek_admits_pin_up_to_its_upload( + tmp_path: Path, httpserver: HTTPServer, files: dict[str, str], hook: dict[str, object] +) -> None: + uploaded = _uploaded(1) + _simple(httpserver, {"0.7.1": uploaded}) + _config(tmp_path, _hook_repo(tmp_path, files), hook) + + result = _cooled_prek(tmp_path, httpserver) + + assert result.returncode == 0, result.stderr + assert (tmp_path / "config").read_text().splitlines() == ["[exclude-newer-package]", _admitted(uploaded)] + assert _admitted(uploaded) in result.stdout + + +@pytest.mark.parametrize( + ("files", "hook"), + [ + pytest.param({}, {"language": "python", "additional_dependencies": ["simplint>=0.7"]}, id="range"), + pytest.param( + {"setup.py": 'setup(install_requires=["simplint>=0.7"])\n'}, {"language": "python"}, id="metadata-range" + ), + pytest.param({}, {"language": "system", "additional_dependencies": ["simplint==0.7.1"]}, id="non-python-hook"), + pytest.param( + {"setup.py": 'setup(install_requires=["simplint==0.7.1"])\n'}, + {"language": "system"}, + id="non-python-metadata", + ), + pytest.param( + {"setup.py": 'setup(extras_require={"test": ["simplint==0.7.1"]}, tests_require=["simplint==0.7.1"])\n'}, + {"language": "python"}, + id="dev-extras", + ), + pytest.param( + {"setup.cfg": "[options.extras_require]\ntest =\n simplint==0.7.1\n"}, + {"language": "python"}, + id="dev-setup-cfg", + ), + pytest.param( + { + "pyproject.toml": ( + '[project.optional-dependencies]\ntest = ["simplint==0.7.1"]\n' + '[dependency-groups]\ndev = ["simplint==0.7.1"]\n' + ) + }, + {"language": "python"}, + id="dev-pyproject", + ), + ], +) +def test_cooled_prek_leaves_ranges_dev_pins_and_non_python_hooks_cooled( + tmp_path: Path, httpserver: HTTPServer, files: dict[str, str], hook: dict[str, object] +) -> None: + _config(tmp_path, _hook_repo(tmp_path, files), hook) + + result = _cooled_prek(tmp_path, httpserver) + + assert result.returncode == 0, result.stderr + assert (tmp_path / "config").read_text() == "" + assert httpserver.log == [] + + +def test_cooled_prek_does_not_admit_newer_release(tmp_path: Path, httpserver: HTTPServer) -> None: + pinned = _uploaded(2) + _simple(httpserver, {"0.7.1": pinned, "0.8.0": _uploaded(1)}) + _config( + tmp_path, + _hook_repo(tmp_path, {"setup.py": 'setup(install_requires=["simplint==0.7.1"])\n'}), + {"language": "python"}, + ) + + result = _cooled_prek(tmp_path, httpserver) + + assert result.returncode == 0, result.stderr + assert (tmp_path / "config").read_text().splitlines() == ["[exclude-newer-package]", _admitted(pinned)] + + +def test_cooled_prek_does_not_write_pin_older_than_window(tmp_path: Path, httpserver: HTTPServer) -> None: + _simple(httpserver, {"0.7.1": _uploaded(8)}) + _config( + tmp_path, + _hook_repo(tmp_path, {"setup.py": 'setup(install_requires=["simplint==0.7.1"])\n'}), + {"language": "python"}, + ) + + result = _cooled_prek(tmp_path, httpserver) + + assert result.returncode == 0, result.stderr + assert (tmp_path / "config").read_text() == "" + + +@pytest.mark.parametrize( + ("nab", "expected"), + [ + pytest.param("{later}", "{later}", id="toml-datetime-later-wins"), + pytest.param("{earlier}", "pin", id="toml-datetime-earlier-loses"), + pytest.param('"{later}"', "{later}", id="string-later-wins"), + pytest.param('"{earlier}"', "pin", id="string-earlier-loses"), + pytest.param('"P3D"', "pin", id="duration-loses-to-pin"), + ], +) +def test_cooled_prek_compares_nab_exception_with_pin( + tmp_path: Path, httpserver: HTTPServer, nab: str, expected: str +) -> None: + pinned = _uploaded(2) + later, earlier = _stamp(pinned + timedelta(days=1)), _stamp(pinned - timedelta(days=1)) + _simple(httpserver, {"0.7.1": pinned}) + (tmp_path / "pyproject.toml").write_text( + f"[tool.nab]\npackages.simplint.uploaded-prior-to = {nab.format(later=later, earlier=earlier)}\n" + ) + _config( + tmp_path, + _hook_repo(tmp_path, {"setup.py": 'setup(install_requires=["simplint==0.7.1"])\n'}), + {"language": "python"}, + ) + + result = _cooled_prek(tmp_path, httpserver) + + assert result.returncode == 0, result.stderr + value = _admitted(pinned) if expected == "pin" else f'"simplint" = "{expected.format(later=later)}"' + assert (tmp_path / "config").read_text().splitlines() == ["[exclude-newer-package]", value] + + +def test_cooled_prek_keeps_nab_duration_without_pin(tmp_path: Path, httpserver: HTTPServer) -> None: + (tmp_path / "pyproject.toml").write_text('[tool.nab]\npackages.sphinx.uploaded-prior-to = "P3D"\n') + _config(tmp_path, _hook_repo(tmp_path, {}), {"language": "python"}) + + result = _cooled_prek(tmp_path, httpserver) + + assert result.returncode == 0, result.stderr + assert (tmp_path / "config").read_text().splitlines() == ["[exclude-newer-package]", '"sphinx" = "P3D"'] + + +@pytest.mark.parametrize( + ("status", "error"), + [ + pytest.param(404, "cannot read simplint from PyPI", id="unknown-project"), + pytest.param(200, "PyPI has no files for simplint==0.7.1", id="version-missing"), + ], +) +def test_cooled_prek_fails_when_pin_cannot_be_looked_up( + tmp_path: Path, httpserver: HTTPServer, status: int, error: str +) -> None: + if status == 404: + httpserver.expect_request("/simple/simplint/").respond_with_data("", status=404) + else: + _simple(httpserver, {"0.6.0": _uploaded(1)}) + repo = _hook_repo(tmp_path, {"setup.py": 'setup(install_requires=["simplint==0.7.1"])\n'}) + _config(tmp_path, repo, {"language": "python"}) + + result = _cooled_prek(tmp_path, httpserver) + + assert result.returncode != 0 + assert error in result.stderr + assert str(repo) in result.stderr + + +def test_cooled_prek_fails_when_hook_repo_cannot_be_fetched(tmp_path: Path, httpserver: HTTPServer) -> None: + _config(tmp_path, tmp_path / "missing", {"language": "python"}) + + result = _cooled_prek(tmp_path, httpserver) + + assert result.returncode != 0 + assert "cannot fetch v1.0.0 of" in result.stderr diff --git a/tools/cooled_prek.sh b/tools/cooled_prek.sh index f2bac0b..ee350f8 100755 --- a/tools/cooled_prek.sh +++ b/tools/cooled_prek.sh @@ -8,18 +8,162 @@ cooled() ( unset UV_INSECURE_HOST UV_SYSTEM_CERTS SSL_CERT_FILE SSL_CERT_DIR export UV_EXCLUDE_NEWER="7 days" UV_NO_CONFIG=1 UV_NO_SOURCES=1 # uv pip, which prek calls, reads per-package cutoffs only from a config file, so the [tool.nab] exceptions the - # release lock already admits are written to one that holds nothing else. + # release lock already admits and the exact hook pins are written to one that holds nothing else. A pinned release + # stays admitted while it is younger than the cutoff, up to its upload time, so the pin installs and newer releases + # of the package stay blocked; its dependencies stay cooled. PyYAML comes from uv under the same cutoff because the + # runner's python3 may not have it and the hook manifests are YAML. The snippet runs under -I, which drops the + # runner's PYTHONPATH and keeps the repository directory off sys.path. exceptions=$( - python3 -I - <<'PY' + uv run --no-project --with pyyaml python -I - <<'PY' +import configparser +import datetime +import json import pathlib +import re +import subprocess +import sys +import tempfile import tomllib +import urllib.request + +import yaml + +HOOK_LANGUAGES = {"python", "python_venv"} +NAME = r"[A-Za-z0-9][A-Za-z0-9._-]*" +VERSION = r"[A-Za-z0-9][A-Za-z0-9.+!_-]*" +REQUIREMENT = re.compile(rf"\s*({NAME})(?:\[[^\]]*\])?==({VERSION})\s*") +SETUP_PY_INSTALL_REQUIRES = re.compile(r"install_requires\s*=\s*\[((?:[^\[\]]|\[[^\]]*\])*)\]") +QUOTED = re.compile(r"[\"']([^\"']+)[\"']") +SEVEN_DAYS_AGO = datetime.datetime.now(datetime.UTC) - datetime.timedelta(days=7) + + +def normalize(name): + return re.sub(r"[-_.]+", "-", name).lower() + + +def exact_pins(requirements): + pins = set() + for requirement in requirements: + if match := REQUIREMENT.fullmatch(requirement.split(";")[0]): + pins.add((normalize(match[1]), match[2])) + return pins + + +def setup_py_pins(text): + match = SETUP_PY_INSTALL_REQUIRES.search(text) + return exact_pins(QUOTED.findall(match[1])) if match else set() + + +def setup_cfg_pins(text): + parser = configparser.ConfigParser(interpolation=None) + parser.read_string(text) + return exact_pins(parser.get("options", "install_requires", fallback="").splitlines()) + + +def pyproject_pins(text): + return exact_pins(tomllib.loads(text).get("project", {}).get("dependencies", [])) + + +PACKAGING_READERS = {"setup.py": setup_py_pins, "setup.cfg": setup_cfg_pins, "pyproject.toml": pyproject_pins} + + +def repo_files(url, rev): + with tempfile.TemporaryDirectory() as directory: + git = ["git", "-C", directory] + try: + subprocess.run(["git", "init", "-q", directory], capture_output=True, text=True, check=True) + subprocess.run( + [*git, "fetch", "-q", "--depth", "1", "--filter=blob:none", url, rev], + capture_output=True, + text=True, + check=True, + ) + root = subprocess.run( + [*git, "ls-tree", "--name-only", "FETCH_HEAD"], capture_output=True, text=True, check=True + ).stdout.split() + wanted = {".pre-commit-hooks.yaml", *PACKAGING_READERS} + return { + name: subprocess.run( + [*git, "show", f"FETCH_HEAD:{name}"], capture_output=True, text=True, check=True + ).stdout + for name in root + if name in wanted + } + except subprocess.CalledProcessError as error: + sys.exit(f"cooled-prek: cannot fetch {rev} of {url}: {error.stderr.strip()}") + + +def hook_pins(url, rev, hooks): + files = repo_files(url, rev) + manifest = { + hook["id"]: hook.get("language") for hook in yaml.safe_load(files.get(".pre-commit-hooks.yaml", "")) or [] + } + python_hooks = [hook for hook in hooks if (hook.get("language") or manifest.get(hook["id"])) in HOOK_LANGUAGES] + if not python_hooks: + return set() + pins = exact_pins(requirement for hook in python_hooks for requirement in hook.get("additional_dependencies", [])) + for name, reader in PACKAGING_READERS.items(): + if name in files: + pins |= reader(files[name]) + return pins + + +def upload_time(url, name, version): + # Fixed PyPI origin, read with the same request as tools/verify_pypi_files.py. + request = urllib.request.Request( + f"https://pypi.org/simple/{name}/", + headers={"Accept": "application/vnd.pypi.simple.v1+json", "Cache-Control": "no-cache"}, + ) + parts = "[-_.]+".join(re.escape(part) for part in name.split("-")) + pattern = re.compile(rf"{parts}-{re.escape(version)}(?:-|\.tar\.gz$|\.zip$)", re.IGNORECASE) + try: + with urllib.request.urlopen( # ruff: ignore[suspicious-url-open-usage] # Fixed PyPI origin. + request, timeout=30 + ) as response: + files = json.load(response)["files"] + times = [ + datetime.datetime.fromisoformat(file["upload-time"]) for file in files if pattern.match(file["filename"]) + ] + except (OSError, ValueError, KeyError) as error: + sys.exit(f"cooled-prek: cannot read {name} from PyPI for the hook repository {url}: {error}") + if not times: + sys.exit(f"cooled-prek: PyPI has no files for {name}=={version}, pinned by the hook repository {url}") + return max(times) + + +def as_datetime(value): + if isinstance(value, str): + try: + value = datetime.datetime.fromisoformat(value) + except ValueError: + return None + return value if isinstance(value, datetime.datetime) and value.tzinfo is not None else None + path = pathlib.Path("pyproject.toml") packages = tomllib.loads(path.read_text()).get("tool", {}).get("nab", {}).get("packages", {}) if path.is_file() else {} -for name, settings in sorted(packages.items()): - if (cutoff := settings.get("uploaded-prior-to")) is not None: - value = cutoff.isoformat().replace("+00:00", "Z") if hasattr(cutoff, "isoformat") else str(cutoff) - print(f'"{name}" = "{value}"') +cutoffs = { + normalize(name): settings["uploaded-prior-to"] + for name, settings in packages.items() + if settings.get("uploaded-prior-to") is not None +} +path = pathlib.Path(".pre-commit-config.yaml") +config = (yaml.safe_load(path.read_text()) if path.is_file() else None) or {} +pins = {} +for repo in config.get("repos", []): + if repo["repo"] in {"local", "meta"}: + continue + for name, version in hook_pins(repo["repo"], repo["rev"], repo.get("hooks", [])): + cutoff = upload_time(repo["repo"], name, version).replace(microsecond=0) + datetime.timedelta(seconds=1) + if cutoff > SEVEN_DAYS_AGO: + pins[name] = max(pins.get(name, cutoff), cutoff) +for name, cutoff in pins.items(): + current = as_datetime(cutoffs.get(name)) + if current is None or current < cutoff: + cutoffs[name] = cutoff +for name, value in sorted(cutoffs.items()): + value = value.isoformat().replace("+00:00", "Z") if hasattr(value, "isoformat") else str(value) + print(f'"{name}" = "{value}"') PY ) if [[ -n "$exceptions" ]]; then @@ -27,7 +171,7 @@ PY printf '[exclude-newer-package]\n%s\n' "$exceptions" >"$config" export UV_CONFIG_FILE="$config" echo "Running prek $* with hook environments built under the 7 days cutoff, repository uv settings ignored," \ - "[tool.nab] cutoff exceptions admitted: $(tr '\n' ' ' <<<"$exceptions")" + "[tool.nab] exceptions and exact hook pins admitted up to their dates: $(tr '\n' ' ' <<<"$exceptions")" else echo "Running prek $* with hook environments built under the 7 days cutoff, repository uv settings ignored" fi