diff --git a/docs/SECURITY-ASSURANCE.md b/docs/SECURITY-ASSURANCE.md index 27bf602..86433ee 100644 --- a/docs/SECURITY-ASSURANCE.md +++ b/docs/SECURITY-ASSURANCE.md @@ -29,7 +29,7 @@ The repository ships no server component and no container image. It stores nothi ## Actors and trust boundaries - **Skill user and agent.** The agent reads the skill and runs `gh` and `git` with the user's authentication. What it runs is decided by the agent and the user. `allowed-tools` in `SKILL.md` only pre-approves `gh`, `git`, `Read`, `Write`, `Edit`, `Glob` and `Grep`; it does not take any tool away from the agent. -- **Hook payload.** Claude Code passes each Bash tool call to the guards as JSON on stdin. The command text is untrusted input: the guards parse it (`parse_command` in both guards, `split_invocations` in `_invocations.py`) and only decide an exit code. Exit code 2 blocks the call; 0 allows it. Any other outcome — another exit code, a guard that cannot start (no `uv` on `PATH`), a timeout — blocks the call too, because `hooks/hooks.json` sets `"onFailure": "block"` (Claude Code v2.1.295 or later; older versions let the call through). The guards start through `uv run`, which installs their two pinned dependencies, `tree-sitter` 0.26.0 and `tree-sitter-bash` 0.25.1, on first use; with no network and an empty uv cache that install fails, `uv` exits 1, and the call is blocked. +- **Hook payload.** Claude Code passes each Bash tool call to the guards as JSON on stdin. The command text is untrusted input: the guards parse it (`parse_command` in both guards, `split_invocations` in `_invocations.py`) and only decide an exit code. Exit code 2 blocks the call; 0 allows it. Any other outcome — another exit code, a guard that crashes, a launcher that cannot start (no `sh`), a timeout — blocks the call too, because `hooks/hooks.json` sets `"onFailure": "block"` (Claude Code v2.1.295 or later; older versions let the call through). Both hooks match every Bash call, not only release commands. They start through `run-guard.sh`, which first asks uv whether the guard's two pinned dependencies, `tree-sitter` 0.26.0 and `tree-sitter-bash` 0.25.1, can be loaded from the uv cache (`uv sync --script --offline`). If not, it tries one install with a 3-second HTTP timeout and no retries, at most once in 5 minutes. With the dependencies in place the guard runs offline and its exit code decides. Without them a fallback decides that needs no dependencies: it blocks every call that names `gh` or `git` as a word, after removing quotes and backslashes, and every call that uses ANSI-C quoting (`$'...'`), which it cannot decode; it says how to install the dependencies and lets every other call through. A plain `uv run` blocked every Bash call on a machine with no network and an empty uv cache, and waited about 49 seconds, past the hook timeout, for an unreachable package index. The launcher needs uv 0.5.31 or later for `uv sync --script`; with no `uv` on `PATH` or an older uv the fallback decides every call, and its message names that cause. A failing `grep` or `tr` in the fallback blocks the call. - **GitHub and registries.** The scripts read from GitHub through `gh api` and from `raw.githubusercontent.com`, `repo.packagist.org` and `extensions.typo3.org` through `curl`. Responses are treated as data and compared, not executed. - **Assessed project.** The checkpoints run in the working directory of the project an assessment tool checks, with the privileges of whoever starts that tool. - **Contributors and CI.** Changes reach `main` through pull requests checked by `.github/workflows/`. Workflows start from `permissions: {}` (except `validate.yml`, which grants `contents: read` at the top) and each job is granted only the scopes its reusable workflow needs. The two `pull_request_target` workflows (`auto-merge-deps.yml`, `labeler.yml`) call reusables that merge or label and do not check out pull request code; `auto-merge-deps.yml` passes two named secrets instead of `secrets: inherit`. @@ -41,7 +41,8 @@ The repository ships no server component and no container image. It stores nothi | An agent runs `gh release create` without an existing tag, which creates an unsigned lightweight tag and, under immutable releases, burns the tag name | Pattern check on the command text for a `--verify-tag` that is on; the blocked and allowed forms are pinned in the test file | `guard-gh-release.py` (`_verify_tag_is_on`, `_strip_quoted`); `tests/guard-gh-release-invocations.test.sh` | | An agent deletes or rewrites a published release outside CI | Pattern checks on `gh release` subcommands and flags and on `gh api` calls to release endpoints; the blocked and allowed forms are pinned in the test file | `guard-gh-release.py` (`_check_invocation`, `_judge_gh_api`); `tests/guard-gh-release-invocations.test.sh` | | An agent creates an unsigned version tag, or deletes or force-pushes one | `git tag` arguments are split into words with `shlex` and read as options and tag names, so a flag inside a quoted value or after a shell comment is not taken for an option; `git push` gets pattern checks; the blocked and allowed forms, including the exception for bare major pointers such as `v4`, are pinned in the test file | `guard-lightweight-tag.py` (`_judge_tag_args`); `tests/guard-tag-invocations.test.sh` | -| A guarded command hides inside a larger Bash call — on its own line, in a loop, a subshell, a command substitution, behind `sudo` or an env assignment, or in a script handed to a shell (`bash -c`, `eval`, a heredoc or here-string a shell reads, text piped into a shell) | Each call is parsed with tree-sitter-bash and every simple command is judged on its own, including the commands of a script handed to a shell; a heredoc body is data unless a shell reads it or bash expands a `$( )` in it. The command name is recognised escaped, quoted or with a directory (`\gh`, `"git"`, `/usr/bin/gh`), and `git`'s and `gh`'s options before the subcommand (`git -C tag`, `gh -R release`) are skipped. A part the parser cannot read is parsed again from the token after the error | `_invocations.py` (`split_invocations`, `command_word`, `GIT_GLOBAL_OPTIONS`, `GH_GLOBAL_OPTIONS`, `INVOCATION_PREFIX`); both invocation test files | +| A guarded command hides inside a larger Bash call — on its own line, in a loop, a subshell, a command substitution, behind `sudo` or an env assignment, or in a script handed to a shell (`bash -c`, `eval`, a heredoc or here-string a shell reads, text piped into a shell) | Each call is parsed with tree-sitter-bash and every simple command is judged on its own, including the commands of a script handed to a shell; a heredoc body is data unless a shell reads it or bash expands a `$( )` in it. The command name is read as the shell reads it after quote removal, so it is recognised escaped, quoted, split by quotes or ANSI-C quoting, or with a directory (`\gh`, `"git"`, `g""h`, `g\h`, `g$'h'`, `/usr/bin/gh`), and no text pre-check skips a call before it is parsed, and `git`'s and `gh`'s options before the subcommand (`git -C tag`, `gh -R release`) are skipped. A part the parser cannot read is parsed again from the token after the error | `_invocations.py` (`split_invocations`, `command_word`, `GIT_GLOBAL_OPTIONS`, `GH_GLOBAL_OPTIONS`, `INVOCATION_PREFIX`); both invocation test files | +| The guards' dependencies cannot be installed (no network, an unreachable package index, an empty uv cache) and either every Bash call is blocked or a release command passes unchecked | `run-guard.sh` decides before the guard runs whether its environment is ready, so uv's own exit codes, which overlap with the guard's, never count as a verdict; the install attempt is bounded to stay inside the hook timeout; the fallback blocks every call naming `gh` or `git`, which covers every blocking case of both guards' suites | `run-guard.sh`; `tests/guard-launcher.test.sh` (runs the `hooks.json` command strings offline, against a blackholed index, and both invocation suites through the launcher) | | The payload shape changes and the guards silently stop working | The command is read from `tool_input.command`, with a top-level `command` as fallback | `parse_command` in both guards; `tests/guard-payload-shape.test.sh` | | Crafted input makes a guard hang past its 10-second hook timeout (CWE-1333) | `gh api` and `git tag` arguments are split with `shlex` instead of backtracking regexes. `shlex` slows down faster than linearly on one very long word: the tag guard took about 0.2 s for a 128 KiB message, the largest single argument Linux passes to a program (`MAX_ARG_STRLEN`), and 3.4 to 3.8 s for 600 KB (16 runs), which git can no longer receive as one argument. tree-sitter-bash parses a command it accepts in linear time, but its error recovery is not linear: 3,000 parse errors in 27 KB take 0.9 s for one parse, 6,000 in 54 KB take 3.5 s. The parses that follow an error stop after 2 s and fall back to a coarse cut, so with those 27 KB the release guard finishes in 2.7 s; the first parse is not bounded. A guard that does time out blocks the call (`onFailure: block`) | `guard-gh-release.py` (`_judge_gh_api`), `guard-lightweight-tag.py` (`_judge_tag_args`), `_invocations.py` (`_REPARSE_BUDGET_SECONDS`); `hooks/hooks.json` (`timeout: 10`, `onFailure: block`) | | The command under judgement is executed by the guard (CWE-78) | Besides `_invocations.py`, the guards import only `json`, `re`, `shlex` and `sys`; `_invocations.py` adds `itertools`, `time` and the tree-sitter parser, which builds a syntax tree from the text. They parse the text and exit, and nothing in them starts a process | `guard-gh-release.py`, `guard-lightweight-tag.py`, `_invocations.py` | @@ -60,7 +61,7 @@ Which of these checks must pass before a pull request can merge is set in the br - **Least privilege:** the helper scripts and checkpoints only read; workflows start from explicit `permissions`. - **Fail-safe defaults inside a guard:** an unknown `gh release` subcommand and a `gh api` call on a release endpoint whose method the guard cannot read block (`guard-gh-release.py`). - **Complete mediation:** each invocation of a Bash call is judged, not only the first match (`check_command` in both guards). -- **Economy of mechanism:** the guards need the Python standard library plus two pinned packages, `tree-sitter` and `tree-sitter-bash`, for reading the shell grammar; the scripts need bash, `git`, `gh`, `jq` and `curl`. +- **Economy of mechanism:** the guards need the Python standard library plus two pinned packages, `tree-sitter` and `tree-sitter-bash`, for reading the shell grammar; their launcher `run-guard.sh` needs a POSIX shell with `cat`, `date`, `id`, `tr` and `grep`, `uv` and, to read the command out of the payload, `python3` (without it the fallback scans the whole payload); the scripts need bash, `git`, `gh`, `jq` and `curl`. ## What a user cannot expect diff --git a/hooks/hooks.json b/hooks/hooks.json index a15fcfd..0472494 100644 --- a/hooks/hooks.json +++ b/hooks/hooks.json @@ -7,13 +7,13 @@ "hooks": [ { "type": "command", - "command": "uv run \"${CLAUDE_PLUGIN_ROOT}/skills/github-release/scripts/guard-gh-release.py\"", + "command": "sh \"${CLAUDE_PLUGIN_ROOT}/skills/github-release/scripts/run-guard.sh\" guard-gh-release.py", "timeout": 10, "onFailure": "block" }, { "type": "command", - "command": "uv run \"${CLAUDE_PLUGIN_ROOT}/skills/github-release/scripts/guard-lightweight-tag.py\"", + "command": "sh \"${CLAUDE_PLUGIN_ROOT}/skills/github-release/scripts/run-guard.sh\" guard-lightweight-tag.py", "timeout": 10, "onFailure": "block" } diff --git a/skills/github-release/SKILL.md b/skills/github-release/SKILL.md index 12f5e6d..abcbaab 100644 --- a/skills/github-release/SKILL.md +++ b/skills/github-release/SKILL.md @@ -4,7 +4,7 @@ name: github-release description: "Use when creating releases, version bumps, tagging, release health checks, or when user says 'release', 'tag', 'version bump'. A version lives in every file that states it: a TYPO3 extension carries it in ext_emconf.php, composer.json, Documentation/guides.xml, the changelog page under Documentation/ and CHANGELOG.md, and updating some of them ships metadata that disagrees with itself. Also use when writing, reviewing or fixing a workflow or CI step that publishes a TYPO3 extension to TER or calls tailor ter:publish or tailor set-version, including in someone else's repository, not only when releasing our own. Also activates on gh release commands to BLOCK them and redirect to safe process." license: "(MIT AND CC-BY-SA-4.0). See LICENSE-MIT and LICENSE-CC-BY-SA-4.0" -compatibility: "Requires uv (runs the scripts on Python 3.11 or later), gh CLI, git with GPG/SSH signing configured. The plugin hooks fail closed from Claude Code v2.1.295 (onFailure)." +compatibility: "Requires uv (runs the scripts on Python 3.11 or later), gh CLI, git with GPG/SSH signing configured. The plugin hooks fail closed from Claude Code v2.1.295 (onFailure); with uv before 0.5.31 their launcher blocks every gh and git call." metadata: author: Netresearch DTT GmbH version: "1.5.1" diff --git a/skills/github-release/scripts/_invocations.py b/skills/github-release/scripts/_invocations.py index e53daff..b7d00f7 100644 --- a/skills/github-release/scripts/_invocations.py +++ b/skills/github-release/scripts/_invocations.py @@ -472,7 +472,20 @@ def _collect(command: str, out: list, depth: int, deadline: float) -> None: # ("2>/dev/null gh release delete") would defeat the prefix match. name = node.child_by_field_name("name") start = name.start_byte if name is not None else None - out.append(_render(node, data, start).strip()) + text = _render(node, data, start) + if name is not None: + # The shell runs gh for g""h and g\h too: the checks match the + # name as the shell reads it, after quote removal. + raw = _render(name, data) + value = _word_value(name, data) + if ( + value + and value != raw + and not re.search(r"\s", value) + and text.startswith(raw) + ): + text = shlex.quote(value) + text[len(raw) :] + out.append(text.strip()) shell = _shell_script(node, data) if shell is not None and shell[0] is not None: _collect(shell[0], out, depth + 1, deadline) diff --git a/skills/github-release/scripts/guard-gh-release.py b/skills/github-release/scripts/guard-gh-release.py index 2706e54..df1110c 100755 --- a/skills/github-release/scripts/guard-gh-release.py +++ b/skills/github-release/scripts/guard-gh-release.py @@ -437,10 +437,8 @@ def main() -> None: if not command: sys.exit(0) - # Quick pre-check: skip if command does not mention gh at all. - if "gh" not in command.lower(): - sys.exit(0) - + # No text pre-check: the shell runs gh for g$'h', g""h and g\h, none of + # which contains "gh", and parsing every call costs well under a millisecond. check_command(command) # If we get here, the command is allowed. sys.exit(0) diff --git a/skills/github-release/scripts/guard-lightweight-tag.py b/skills/github-release/scripts/guard-lightweight-tag.py index bf8d24c..77125e3 100755 --- a/skills/github-release/scripts/guard-lightweight-tag.py +++ b/skills/github-release/scripts/guard-lightweight-tag.py @@ -426,9 +426,8 @@ def main() -> None: if not command: sys.exit(0) - # Quick pre-check: skip if command does not mention git at all. - if "git" not in command.lower(): - sys.exit(0) + # No text pre-check: the shell runs git for gi$'t', g""it and g\it, none of + # which contains "git", and parsing every call costs well under a millisecond. check_command(command) # If we get here, the command is allowed. diff --git a/skills/github-release/scripts/run-guard.sh b/skills/github-release/scripts/run-guard.sh new file mode 100755 index 0000000..da0ba4c --- /dev/null +++ b/skills/github-release/scripts/run-guard.sh @@ -0,0 +1,109 @@ +#!/bin/sh +# SPDX-License-Identifier: MIT +# SPDX-FileCopyrightText: Netresearch DTT GmbH +# +# Starts one guard hook (hooks/hooks.json) and keeps the machine usable when +# the guard's dependencies cannot be installed. +# +# Usage: run-guard.sh (the hook payload arrives on stdin) +# +# The guards declare tree-sitter and tree-sitter-bash as inline dependencies, +# which uv installs on first use. Both hooks match EVERY Bash call and block on +# any failure, so with no network and an empty uv cache a plain `uv run` exits +# non-zero and every Bash call is blocked, `ls` included. An unreachable index +# is worse: uv waits about 49 s, past the hook timeout. Its exit codes cannot +# tell such failures from the guard's own verdict either, because uv exits 2 +# on a network error and 2 is the guard's "block". +# +# So the decision happens before the guard runs: +# 1. `uv sync --script --offline` reports whether the environment can be +# built from the cache alone (about 10 ms when it can). +# 2. If not, one bounded online install is tried (3 s HTTP timeout, no +# retries), at most once per 5 minutes per user. +# 3. If the environment is ready, the guard runs offline and its exit code +# is passed through unchanged. +# 4. Otherwise a fallback without dependencies decides: it blocks every call +# that names `gh` or `git` and lets every other call through. +# +# Requires uv 0.5.31 or later (`uv sync --script`). With an older uv the +# fallback decides every call. +set -u + +here=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) || exit 1 +name=${1:?usage: run-guard.sh } +guard="$here/$name" +[ -f "$guard" ] || { + echo "run-guard.sh: no guard script at $guard" >&2 + exit 1 +} + +# The payload is read once and handed to whichever path decides. +payload=$(cat) + +env_ready() { + uv sync --script "$guard" --offline >/dev/null 2>&1 +} + +if ! env_ready; then + # The stamp holds the epoch second of the last attempt. `date +%s` works + # with GNU, BSD, busybox and Git Bash; `find -mmin` may resolve to Windows' + # find.exe there. + stamp="${TMPDIR:-/tmp}/github-release-guard-install-$(id -u).stamp" + now=$(date +%s) + last=$(cat "$stamp" 2>/dev/null) + case $last in + '' | *[!0-9]*) last=0 ;; + *) ;; + esac + if [ $((now - last)) -ge 300 ]; then + printf '%s\n' "$now" >"$stamp" 2>/dev/null + UV_HTTP_TIMEOUT=3 UV_HTTP_RETRIES=0 uv sync --script "$guard" >/dev/null 2>&1 + fi +fi + +if env_ready; then + printf '%s' "$payload" | uv run --offline "$guard" + exit $? +fi + +# --- fallback: the guard cannot run ----------------------------------------- +# Read the command out of the payload the way the guards' parse_command does +# when a Python is at hand; otherwise scan the whole payload, which can only +# block more, never less. +command=$(printf '%s' "$payload" | python3 -I -c ' +import json, sys +d = json.load(sys.stdin) +ti = d.get("tool_input") if isinstance(d.get("tool_input"), dict) else {} +c = ti.get("command") or d.get("command") or "" +if not isinstance(c, str): + sys.exit(1) +sys.stdout.write(c) +' 2>/dev/null) || command=$payload + +# The guards recognise `\gh`, `"git"`, `g""h` and `/usr/bin/gh`: drop quotes +# and backslashes, then look for gh or git as a word of its own. Only grep's +# "no match" (1) allows; a match (0) or an error (2, or a failed tr) blocks. +# ANSI-C quoting ($'\x67h' is gh) cannot be decoded here, so it blocks too. +case $command in + *"\$'"*) normalised=gh ;; + *) normalised=$(printf '%s\n' "$command" | tr -d "\\\\\"'") || normalised=gh ;; +esac +printf '%s\n' "$normalised" | grep -Eq '(^|[^A-Za-z0-9_-])(gh|git)([^A-Za-z0-9_-]|$)' +[ $? -eq 1 ] && exit 0 + +if ! command -v uv >/dev/null 2>&1; then + cause="uv is not on PATH" + fix="Install uv 0.5.31 or later." +elif ! uv sync --help 2>/dev/null | grep -q -- '--script'; then + cause="this uv is older than 0.5.31 and cannot prepare a script environment" + fix="Upgrade uv to 0.5.31 or later (uv self update)." +else + cause="its dependencies (tree-sitter, tree-sitter-bash) are not in the uv cache and could not be installed" + fix="With network access, run once: uv sync --script \"$guard\"" +fi +cat >&2 < sends each case through the launcher, +# the path hooks/hooks.json takes, instead of `uv run` on the guard. +# BLOCK_ONLY=1 checks only the cases that must block; guard-launcher.test.sh +# uses it for the launcher's fallback, which blocks more than the guard does. +run_guard() { + if [[ -n "${GUARD_LAUNCHER:-}" ]]; then + sh "$GUARD_LAUNCHER" "$(basename "$GH_GUARD")" + else + uv run "$GH_GUARD" + fi +} + +skip_case() { # + local want="$1" desc="$2" + [[ "${BLOCK_ONLY:-}" == 1 && "$want" != 2 ]] || return 1 + printf 'skip - %s\n' "$desc" +} + check() { # local want="$1" desc="$2" command="$3" payload got + skip_case "$want" "$desc" && return payload=$(CMD="$command" python3 -c \ 'import json,os;print(json.dumps({"tool_name":"Bash","tool_input":{"command":os.environ["CMD"]}}))') - printf '%s' "$payload" | uv run "$GH_GUARD" >/dev/null 2>&1 + printf '%s' "$payload" | run_guard >/dev/null 2>&1 got=$? if [[ "$got" == "$want" ]]; then printf 'ok - %s\n' "$desc" @@ -58,6 +77,14 @@ gh release create v1.2.3' # --- an invocation may carry a prefix -------------------------------------- # Regression: "gh" had to sit directly after the separator, so any prefix hid it. check 2 'prefixed with sudo' 'sudo gh release create v1.2.3' + +# --- the program name spelled so that the text never holds "gh" ------------- +# Regression: a text pre-check skipped every call without the substring "gh", +# so these ran unjudged although the shell runs gh for each of them. +check 2 'name split by ANSI-C quoting' "g\$'h' release delete v1.2.3" +check 2 'name fully ANSI-C escaped' "\$'\\x67h' release delete v1.2.3" +check 2 'name split by empty quotes' 'g""h release create v1.2.3' +check 2 'name split by a backslash' 'g\h release delete v1.2.3' check 2 'prefixed with an env assignment' 'GH_TOKEN=x gh release delete v1.2.3' # shellcheck disable=SC2016 # the guard must see "$v" literally, unexpanded check 2 'inside a loop body' 'for v in v1.2.3; do gh release create "$v"; done' diff --git a/skills/github-release/scripts/tests/guard-launcher.test.sh b/skills/github-release/scripts/tests/guard-launcher.test.sh new file mode 100755 index 0000000..b69b129 --- /dev/null +++ b/skills/github-release/scripts/tests/guard-launcher.test.sh @@ -0,0 +1,192 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MIT +# SPDX-FileCopyrightText: Netresearch DTT GmbH +# Self-test for run-guard.sh, the launcher both guard hooks start through. +# +# The guards need tree-sitter and tree-sitter-bash, which uv installs on first +# use. Started as a plain `uv run` from hooks/hooks.json, a machine without +# network and with an empty uv cache blocked EVERY Bash call, `ls` included, +# because both hooks match all Bash calls and block on any failure. These cases +# run the command strings from hooks/hooks.json, so a hook that bypasses the +# launcher fails here. +# +# "Offline" below is UV_OFFLINE=1 with an empty UV_CACHE_DIR; the unreachable +# index case points uv at a blackholed address. Neither needs network. The +# warm cases need the two packages once, like the other guard tests. +set -uo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$HERE/../../../.." && pwd)" +LAUNCHER="$HERE/../run-guard.sh" +HOOKS="$ROOT/hooks/hooks.json" + +fail=0 +SCRATCH=$(mktemp -d) +trap 'rm -rf "$SCRATCH"' EXIT + +# The hooks.json command for a guard, exactly as Claude Code runs it. +hook_cmd() { # + local guard="$1" + jq -r --arg g "$guard" '.hooks.PreToolUse[].hooks[].command | select(contains($g))' "$HOOKS" +} +REL_CMD=$(hook_cmd guard-gh-release.py) +TAG_CMD=$(hook_cmd guard-lightweight-tag.py) +for c in "$REL_CMD" "$TAG_CMD"; do + if [[ "$c" != *run-guard.sh* ]]; then + printf 'FAIL - hooks.json starts a guard without run-guard.sh: %s\n' "$c" + fail=1 + fi +done + +# The warm cases need the dependencies installed, not left to whichever suite +# happened to run first, and a stamp of their own: a fresh stamp in the real +# TMPDIR from a failed install would send them to the fallback. +for g in guard-gh-release.py guard-lightweight-tag.py; do + if ! uv sync --script "$HERE/../$g" >/dev/null 2>&1; then + printf 'FAIL - cannot install the dependencies of %s for the warm cases\n' "$g" + fail=1 + fi +done +WARM_TMP=$(mktemp -d "$SCRATCH/warm.XXXX") + +payload_for() { # + CMD="$1" python3 -c \ + 'import json,os;print(json.dumps({"tool_name":"Bash","tool_input":{"command":os.environ["CMD"]}}))' +} + +# [expected stderr text] +# mode: warm = the developer's own uv cache; offline = empty cache, no network. +check() { + local mode="$1" want="$2" hook="$3" desc="$4" command="$5" text="${6:-}" + local payload got stderr + payload=$(payload_for "$command") + if [[ "$mode" == offline ]]; then + local d + d=$(mktemp -d "$SCRATCH/offline.XXXX") + stderr=$(printf '%s' "$payload" | + CLAUDE_PLUGIN_ROOT="$ROOT" UV_OFFLINE=1 UV_CACHE_DIR="$d/cache" TMPDIR="$d" \ + sh -c "$hook" 2>&1 >/dev/null) + got=$? + else + stderr=$(printf '%s' "$payload" | CLAUDE_PLUGIN_ROOT="$ROOT" TMPDIR="$WARM_TMP" sh -c "$hook" 2>&1 >/dev/null) + got=$? + fi + if [[ "$got" == "$want" && "$stderr" == *"$text"* ]]; then + printf 'ok - [%s] %s\n' "$mode" "$desc" + else + printf 'FAIL - [%s] %s\n expected rc: %s, stderr containing: %s\n actual rc: %s\n actual stderr: %s\n' \ + "$mode" "$desc" "$want" "$text" "$got" "$stderr" + fail=1 + fi +} + +# --- warm cache: the guard decides, and the payload reaches it --------------- +# A launcher that dropped stdin would hand the guard an empty command, which +# the guard allows, so the blocking cases are what pins the forwarding. +check warm 2 "$REL_CMD" 'release guard blocks delete' 'gh release delete v1.2.3' +check warm 2 "$REL_CMD" 'release guard blocks create on its own line' $'cd /tmp\ngh release create v1.2.3' +check warm 0 "$REL_CMD" 'release guard allows view' 'gh release view v1.2.3' +check warm 0 "$REL_CMD" 'release guard allows ls' 'ls -la' +check warm 2 "$TAG_CMD" 'tag guard blocks lightweight tag' 'git tag v1.2.3' +check warm 0 "$TAG_CMD" 'tag guard allows signed tag' 'git tag -s v1.2.3 -m release' +check warm 0 "$TAG_CMD" 'tag guard allows git status' 'git status' + +# --- offline, empty cache: the fallback decides ----------------------------- +# Regression: every one of these calls exited 1 under `uv run` and was blocked. +check offline 0 "$REL_CMD" 'release guard lets ls through' 'ls -la' +check offline 0 "$TAG_CMD" 'tag guard lets ls through' 'ls -la' +check offline 0 "$REL_CMD" 'words that merely contain gh or git pass' 'echo highlight digit github-cli legit' +check offline 2 "$REL_CMD" 'release guard blocks gh release delete' 'gh release delete v1.2.3' 'uv sync --script' +check offline 2 "$TAG_CMD" 'tag guard blocks git tag' 'git tag v1.2.3' 'uv sync --script' +check offline 2 "$REL_CMD" 'any gh call is blocked' 'gh pr list' +check offline 2 "$TAG_CMD" 'any git call is blocked' 'git status' +check offline 2 "$REL_CMD" 'gh with a directory' '/usr/bin/gh release create v1.2.3' +check offline 2 "$REL_CMD" 'gh escaped' '\gh release create v1.2.3' +check offline 2 "$REL_CMD" 'gh split by empty quotes' 'g""h release create v1.2.3' +check offline 2 "$TAG_CMD" 'git quoted' "'git' tag v1.2.3" +check offline 2 "$TAG_CMD" 'git on a later line' $'echo start\nsudo git tag v1.2.3' + +# A payload whose tool_input.command is empty is read through to the top-level +# command, as the guards' parse_command does. +d=$(mktemp -d "$SCRATCH/shape.XXXX") +printf '%s' '{"tool_input":{"command":""},"command":"gh release delete v1"}' | + CLAUDE_PLUGIN_ROOT="$ROOT" UV_OFFLINE=1 UV_CACHE_DIR="$d/cache" TMPDIR="$d" \ + sh -c "$REL_CMD" >/dev/null 2>&1 +got=$? +if [[ "$got" == 2 ]]; then + printf 'ok - [offline] empty tool_input.command falls through to command\n' +else + printf 'FAIL - [offline] empty tool_input.command falls through to command: rc %s\n' "$got" + fail=1 +fi + +# --- no uv on PATH: the fallback decides and names the real cause ----------- +uv_dir=$(dirname "$(command -v uv)") +if [[ -x "$uv_dir/sh" || -x "$uv_dir/python3" ]]; then + printf 'skip - [no-uv] uv shares %s with sh or python3\n' "$uv_dir" +else + NOUV_PATH=$(tr ':' '\n' <<<"$PATH" | grep -vxF "$uv_dir" | paste -sd: -) + for c in 'ls -la|0|' 'gh pr list|2|uv is not on PATH'; do + IFS='|' read -r command want text <<<"$c" + d=$(mktemp -d "$SCRATCH/nouv.XXXX") + stderr=$(printf '%s' "$(payload_for "$command")" | + PATH="$NOUV_PATH" CLAUDE_PLUGIN_ROOT="$ROOT" TMPDIR="$d" sh -c "$REL_CMD" 2>&1 >/dev/null) + got=$? + if [[ "$got" == "$want" && "$stderr" == *"$text"* ]]; then + printf 'ok - [no-uv] %s\n' "$command" + else + printf 'FAIL - [no-uv] %s: rc %s (want %s), stderr: %s\n' "$command" "$got" "$want" "$stderr" + fail=1 + fi + done +fi + +# --- unreachable index: the call is decided inside the hook timeout --------- +# uv waits about 49 s for a blackholed index on its own; hooks.json allows 10. +d=$(mktemp -d "$SCRATCH/blackhole.XXXX") +for round in first second; do + start=$(date +%s) + printf '%s' "$(payload_for 'ls -la')" | + CLAUDE_PLUGIN_ROOT="$ROOT" UV_CACHE_DIR="$d/cache" TMPDIR="$d" \ + UV_DEFAULT_INDEX=http://10.255.255.1/simple sh -c "$REL_CMD" >/dev/null 2>&1 + got=$? + secs=$(($(date +%s) - start)) + limit=8 + [[ "$round" == second ]] && limit=2 # the install is not retried for 5 minutes + if [[ "$got" == 0 && "$secs" -le "$limit" ]]; then + printf 'ok - [blackhole] %s call allowed in %ss (limit %ss)\n' "$round" "$secs" "$limit" + else + printf 'FAIL - [blackhole] %s call: rc %s after %ss (want 0 within %ss)\n' "$round" "$got" "$secs" "$limit" + fail=1 + fi +done + +# --- every blocking case of the guards' own suites, through the launcher ---- +# warm: the full suites, verdicts unchanged. offline: the fallback must block +# every case the guards block (it may block more, so allowed cases are skipped). +run_suite() { # + local mode="$1" suite="$2" out rc d + if [[ "$mode" == offline ]]; then + d=$(mktemp -d "$SCRATCH/suite.XXXX") + out=$(GUARD_LAUNCHER="$LAUNCHER" BLOCK_ONLY=1 UV_OFFLINE=1 UV_CACHE_DIR="$d/cache" TMPDIR="$d" \ + bash "$HERE/$suite" 2>&1) + else + out=$(GUARD_LAUNCHER="$LAUNCHER" bash "$HERE/$suite" 2>&1) + fi + rc=$? + local oks + oks=$(grep -c '^ok ' <<<"$out") + if [[ "$rc" == 0 && "$oks" -gt 0 ]]; then + printf 'ok - [%s] %s through the launcher (%s cases)\n' "$mode" "$suite" "$oks" + else + printf 'FAIL - [%s] %s through the launcher (rc %s, %s ok)\n' "$mode" "$suite" "$rc" "$oks" + grep '^FAIL' <<<"$out" | head -20 + fail=1 + fi +} +for suite in guard-gh-release-invocations.test.sh guard-tag-invocations.test.sh; do + run_suite warm "$suite" + run_suite offline "$suite" +done + +exit "$fail" diff --git a/skills/github-release/scripts/tests/guard-tag-invocations.test.sh b/skills/github-release/scripts/tests/guard-tag-invocations.test.sh index 5c09501..392034d 100755 --- a/skills/github-release/scripts/tests/guard-tag-invocations.test.sh +++ b/skills/github-release/scripts/tests/guard-tag-invocations.test.sh @@ -19,11 +19,30 @@ TAG_GUARD="${1:-$HERE/../guard-lightweight-tag.py}" fail=0 +# GUARD_LAUNCHER= sends each case through the launcher, +# the path hooks/hooks.json takes, instead of `uv run` on the guard. +# BLOCK_ONLY=1 checks only the cases that must block; guard-launcher.test.sh +# uses it for the launcher's fallback, which blocks more than the guard does. +run_guard() { + if [[ -n "${GUARD_LAUNCHER:-}" ]]; then + sh "$GUARD_LAUNCHER" "$(basename "$TAG_GUARD")" + else + uv run "$TAG_GUARD" + fi +} + +skip_case() { # + local want="$1" desc="$2" + [[ "${BLOCK_ONLY:-}" == 1 && "$want" != 2 ]] || return 1 + printf 'skip - %s\n' "$desc" +} + check() { # local want="$1" desc="$2" command="$3" payload got + skip_case "$want" "$desc" && return payload=$(CMD="$command" python3 -c \ 'import json,os;print(json.dumps({"tool_name":"Bash","tool_input":{"command":os.environ["CMD"]}}))') - printf '%s' "$payload" | uv run "$TAG_GUARD" >/dev/null 2>&1 + printf '%s' "$payload" | run_guard >/dev/null 2>&1 got=$? if [[ "$got" == "$want" ]]; then printf 'ok - %s\n' "$desc" @@ -37,9 +56,10 @@ check_reason() { # local want="$1" desc="$2" command="$3" payload stderr got payload=$(CMD="$command" python3 -c \ 'import json,os;print(json.dumps({"tool_name":"Bash","tool_input":{"command":os.environ["CMD"]}}))') - stderr=$(printf '%s' "$payload" | uv run "$TAG_GUARD" 2>&1 >/dev/null) + stderr=$(printf '%s' "$payload" | run_guard 2>&1 >/dev/null) got=$? - if [[ "$got" == 2 && "$stderr" == *"$want"* ]]; then + # The fallback blocks with its own message, so BLOCK_ONLY checks the code only. + if [[ "$got" == 2 && ( "${BLOCK_ONLY:-}" == 1 || "$stderr" == *"$want"* ) ]]; then printf 'ok - %s\n' "$desc" else printf 'FAIL - %s\n expected rc 2 and reason containing: %s\n actual rc: %s\n actual stderr: %s\n' \ @@ -87,6 +107,11 @@ check 0 'unquoted heredoc delimiter' \ "cat > note.md < note.md <<'EOF'