From 3e5edad0156a8f0e1ff4b0d13235610771cc085c Mon Sep 17 00:00:00 2001 From: Johnson Lyu Date: Mon, 26 Oct 2026 01:22:24 -0400 Subject: [PATCH 1/3] fix: clarify no-workflows audit warnings (#584) Assisted-by: Cursor:Claude-Opus-5.5 Signed-off-by: Johnson Lyu --- CHANGELOG.md | 17 + docs/architecture/framework-design.md | 1 + .../src/darnit_baseline/github_workflows.py | 126 +++++ .../src/darnit_baseline/implementation.py | 12 + .../src/darnit_baseline/openssf-baseline.toml | 22 + .../sieve/test_control_status_baseline.py | 31 +- tests/darnit/sieve/test_expr_failure.py | 12 +- .../controls/test_no_workflows.py | 461 ++++++++++++++++++ .../controls/test_secure_branch_names.py | 14 +- .../darnit_baseline/test_github_workflows.py | 293 +++++++++++ 10 files changed, 976 insertions(+), 13 deletions(-) create mode 100644 packages/darnit-baseline/src/darnit_baseline/github_workflows.py create mode 100644 tests/darnit_baseline/controls/test_no_workflows.py create mode 100644 tests/darnit_baseline/test_github_workflows.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 2628d7f48..dcc129cf1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -650,6 +650,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- On a repository with no GitHub Actions workflow file, OpenSSF Baseline + OSPS-BR-01.01, OSPS-BR-01.02, and OSPS-AC-04.02 report WARN saying no + workflow files were found and there is nothing to audit, instead of + `WARN [unexpected_exit]` from zizmor exiting 3 or `grep` exiting 2, and + zizmor and `grep` are not run, so their `unexpected_exit` log lines are + gone. A new `darnit-baseline` step type, `github_workflows_present` + (ceiling `{fail}`), is the first step of the three controls and concludes + that WARN; it never reports PASS or FAIL, and with workflow files present + it is evidence only (later steps' indexes move up by one). The WARN is now + concluded by that step (`concluded_by = "github_workflows_present"`, + authority `dispositive`) rather than left undetermined, and it replaces an + ERROR when zizmor or `grep` is not installed. For OSPS-BR-01.01 an + `action.yml` or `action.yaml` anywhere in the repository also counts as + something to audit, since zizmor's `template-injection` audit reads action + definitions too, so zizmor still runs and reports findings in a standalone + action. A confirmed non-GitHub `ci_provider` still makes the controls N/A + (#584). - Configure `fail_exit_codes = [0]` for OpenSSF Baseline OSPS-BR-01.02 (`grep -rln github.head_ref .github/workflows/`) so exit 0 on a match records as non-conclusive evidence without an `unexpected_exit` error class (fixes #647). diff --git a/docs/architecture/framework-design.md b/docs/architecture/framework-design.md index 2b61fcf8c..b93effbf4 100644 --- a/docs/architecture/framework-design.md +++ b/docs/architecture/framework-design.md @@ -2032,6 +2032,7 @@ Implementation-registered sieve handlers (non-exhaustive): | Step type | Registered by | Ceiling | |-----------|---------------|---------| | `github_branch_protection` | `darnit-baseline`: the classic-branch-protection + repository-rulesets two-surface check for `OSPS-AC-03.01`, `OSPS-AC-03.02`, `OSPS-QA-03.01`, `OSPS-QA-07.01`; see `specs/032-ruleset-branch-protection/contracts/github-branch-protection-handler.md` | `{pass, fail}` (reads platform settings) | +| `github_workflows_present` | `darnit-baseline`: the first step of `OSPS-BR-01.01`, `OSPS-BR-01.02`, `OSPS-AC-04.02`; WARN saying there is nothing to audit when `.github/workflows/` holds no `.yml` or `.yaml` file inside the repository and, with `actions = true` (`OSPS-BR-01.01`, whose zizmor audit also reads action definitions), the repository holds no `action.yml` or `action.yaml` outside `.git`; otherwise INCONCLUSIVE (#584) | `{fail}`: it concludes only that WARN, never PASS or FAIL | | `generate_threat_model` | `darnit-baseline` (remediation, `supports_plan=True`; returns its files as `FileChange`s, section 4.2) | `{}` (a remediation handler, section 3.0.1) | | `gittuf_verify_policy`, `gittuf_commits_signed` | `darnit-gittuf` | `{pass, fail}` (cryptographic verification) | | `repro_deps_pinned`, `repro_build_env_declared`, `repro_hermetic_build`, `repro_provenance_exists`, `repro_bit_for_bit` | `darnit-reproducibility` | `{fail}`: they decide from text and file-presence signals, so PASS needs a corpus-backed promotion (section 3.0.1) | diff --git a/packages/darnit-baseline/src/darnit_baseline/github_workflows.py b/packages/darnit-baseline/src/darnit_baseline/github_workflows.py new file mode 100644 index 000000000..6f99f3545 --- /dev/null +++ b/packages/darnit-baseline/src/darnit_baseline/github_workflows.py @@ -0,0 +1,126 @@ +"""The ``github_workflows_present`` sieve handler (#584). + +The first step of the controls that audit GitHub Actions workflow files +(``OSPS-BR-01.01``, ``OSPS-BR-01.02``, ``OSPS-AC-04.02``). Without workflow +files their commands have nothing to read: zizmor exits 3 and ``grep`` +exits 2, which an ``exec`` step reports as ``unexpected_exit``. This step +says what happened instead, before either runs. + +* WARN when ``.github/workflows/`` holds no ``.yml`` or ``.yaml`` file inside + the repository. The step type's ceiling is ``{fail}``, so the WARN + concludes the control with this message and no error class, and no later + step runs. +* With ``actions = true``, an action definition (``action.yml`` or + ``action.yaml``) anywhere in the repository outside ``.git`` also counts as + something to audit, for a control whose zizmor audit reads actions as well + as workflows (``template-injection``). zizmor collects action definitions + from the whole tree, so without the setting this step would end such a + control before zizmor could report a finding in one. +* INCONCLUSIVE otherwise: the files are evidence and evaluation continues + with the control's other steps. + +It never returns PASS or FAIL. +""" + +from __future__ import annotations + +import os +from typing import Any + +from darnit.core.repo_files import escaping_entry, within_repository +from darnit.sieve.handler_registry import HandlerContext, HandlerResult, HandlerResultStatus + +WORKFLOWS_DIR = ".github/workflows" +WORKFLOW_SUFFIXES = (".yml", ".yaml") +ACTION_FILE_NAMES = ("action.yml", "action.yaml") +_EVIDENCE_LIMIT = 20 + +NO_WORKFLOWS_MESSAGE = ( + "No GitHub Actions workflow files (.github/workflows/*.yml or *.yaml) were found, " + "so there is nothing to audit. If this project does not use GitHub Actions, " + "confirm ci_provider to mark this control not applicable." +) +NO_WORKFLOWS_OR_ACTIONS_MESSAGE = ( + "No GitHub Actions workflow files (.github/workflows/*.yml or *.yaml) or action " + "definitions (action.yml or action.yaml) were found, so there is nothing to audit. " + "If this project does not use GitHub Actions, confirm ci_provider to mark this " + "control not applicable." +) + + +def _first_action_file(root: str, skipped: list[str]) -> str | None: + """The first action definition inside the repository, as a repository-relative path.""" + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = sorted(d for d in dirnames if d != ".git") + for name in sorted(filenames): + if name.lower() not in ACTION_FILE_NAMES: + continue + path = os.path.join(dirpath, name) + if not within_repository(root, path): + entry = escaping_entry(root, path) + if entry not in skipped: + skipped.append(entry) + continue + if os.path.isfile(path): + return os.path.relpath(path, root).replace(os.sep, "/") + return None + + +def github_workflows_present_handler(config: dict[str, Any], context: HandlerContext) -> HandlerResult: + """WARN when the repository has nothing for the control to audit, else INCONCLUSIVE.""" + directory = os.path.join(context.local_path, *WORKFLOWS_DIR.split("/")) + try: + names = sorted(entry.name for entry in os.scandir(directory)) + except (FileNotFoundError, NotADirectoryError): + names = [] + except OSError as e: + return HandlerResult( + status=HandlerResultStatus.INCONCLUSIVE, + message=f"Could not list {WORKFLOWS_DIR}/: {e}", + evidence={"workflows_dir": WORKFLOWS_DIR}, + ) + + found: list[str] = [] + skipped: list[str] = [] + for name in names: + if not name.lower().endswith(WORKFLOW_SUFFIXES): + continue + path = os.path.join(directory, name) + if not within_repository(context.local_path, path): + entry = escaping_entry(context.local_path, path) + if entry not in skipped: + skipped.append(entry) + continue + if os.path.isfile(path): + found.append(f"{WORKFLOWS_DIR}/{name}") + + evidence: dict[str, Any] = {"workflows_dir": WORKFLOWS_DIR, "workflow_files": found[:_EVIDENCE_LIMIT]} + if found: + if skipped: + evidence["skipped_outside_repository"] = skipped + return HandlerResult( + status=HandlerResultStatus.INCONCLUSIVE, + message=f"GitHub Actions workflow files found: {len(found)}", + evidence=evidence, + ) + + include_actions = config.get("actions") is True + action_file = _first_action_file(context.local_path, skipped) if include_actions else None + if skipped: + evidence["skipped_outside_repository"] = skipped + if action_file is not None: + evidence["action_files"] = [action_file] + return HandlerResult( + status=HandlerResultStatus.INCONCLUSIVE, + message=f"No GitHub Actions workflow files; action definition found: {action_file}", + evidence=evidence, + ) + return HandlerResult( + status=HandlerResultStatus.WARN, + message=NO_WORKFLOWS_OR_ACTIONS_MESSAGE if include_actions else NO_WORKFLOWS_MESSAGE, + confidence=1.0, + evidence=evidence, + ) + + +__all__ = ["NO_WORKFLOWS_MESSAGE", "NO_WORKFLOWS_OR_ACTIONS_MESSAGE", "github_workflows_present_handler"] diff --git a/packages/darnit-baseline/src/darnit_baseline/implementation.py b/packages/darnit-baseline/src/darnit_baseline/implementation.py index 3ea541125..b059865a7 100644 --- a/packages/darnit-baseline/src/darnit_baseline/implementation.py +++ b/packages/darnit-baseline/src/darnit_baseline/implementation.py @@ -126,6 +126,7 @@ def register_handlers(self) -> None: from darnit.sieve.handler_registry import get_sieve_handler_registry from .branch_protection import github_branch_protection_handler + from .github_workflows import github_workflows_present_handler from .threat_model.remediation import generate_threat_model_handler sieve_registry = get_sieve_handler_registry() @@ -155,6 +156,17 @@ def register_handlers(self) -> None: ceiling={"pass", "fail"}, settings={"requirement", "required_approvals_minimum", "owner", "repo", "branch"}, ) + # #584: WARN, concluding under {fail}, when there is no workflow file + # (or, with `actions = true`, no action definition either) to audit; + # otherwise evidence only. Never PASS or FAIL. + sieve_registry.register( + "github_workflows_present", + phase="deterministic", + handler_fn=github_workflows_present_handler, + description="WARN when .github/workflows/ holds no workflow file", + ceiling={"fail"}, + settings={"actions"}, + ) sieve_registry.set_plugin_context(None) diff --git a/packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml b/packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml index c59056e3d..ba689555c 100644 --- a/packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml +++ b/packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml @@ -670,6 +670,15 @@ run: | docs_url = "https://baseline.openssf.org/versions/2025-10-10#OSPS-BR-01.01" location_hint = ".github/workflows" +# No workflow file means nothing to audit (#584): WARN with that reason, and +# zizmor (exit 3, unexpected_exit) never runs. With workflow files present it +# is evidence only. zizmor's template-injection audit also reads action +# definitions, which it collects from the whole tree, so an action.yml or +# action.yaml anywhere also lets zizmor run. +[[controls."OSPS-BR-01.01".passes]] +handler = "github_workflows_present" +actions = true + [[controls."OSPS-BR-01.01".passes]] handler = "exec" command = ["zizmor", "--format", "json", "--offline", "$PATH"] @@ -756,6 +765,12 @@ run: | ``` """ +# No workflow file means nothing to audit (#584): WARN with that reason, and +# grep (exit 2, unexpected_exit) never runs. With workflow files present it +# is evidence only. +[[controls."OSPS-BR-01.02".passes]] +handler = "github_workflows_present" + # Keyword absence, evidence only (feature 041): not finding github.head_ref # does not show branch names are handled safely (github.ref_name and event # payload fields carry them too). @@ -2972,6 +2987,13 @@ help_md = """Minimize CI/CD job permissions. 4. Avoid using `write-all` or `admin` permissions """ +# No workflow file means nothing to audit (#584): WARN with that reason, and +# zizmor (exit 3, unexpected_exit) never runs. With workflow files present it +# is evidence only. No `actions = true`: zizmor's excessive-permissions audit +# reads workflows only. +[[controls."OSPS-AC-04.02".passes]] +handler = "github_workflows_present" + # Primary: zizmor's `excessive-permissions` deep audit. zizmor models per-step # action requirements and flags workflow- or job-level grants broader than the # steps actually exercise — the semantic least-privilege check a regex can't do. diff --git a/tests/darnit/sieve/test_control_status_baseline.py b/tests/darnit/sieve/test_control_status_baseline.py index 5869a1df1..fcba1113a 100644 --- a/tests/darnit/sieve/test_control_status_baseline.py +++ b/tests/darnit/sieve/test_control_status_baseline.py @@ -22,16 +22,36 @@ import pytest -from .baseline_capture import capture +from .baseline_capture import PARITY_FIXTURES, capture + +# #584: `github_workflows_present`, the first step of these controls, returns +# WARN by design on a repository with no workflow file. Neutralized, the +# controls run on to zizmor and grep, which end them ERROR where the tool is +# not installed (else the same WARN, which is no drift). That ERROR -> WARN on +# a fixture without workflow files is the only drift tolerated. +WARN_BY_DESIGN = {"OSPS-BR-01.01", "OSPS-BR-01.02", "OSPS-AC-04.02"} + + +def _warn_by_design(pair: str, control_id: str, before: str | None, after: str | None) -> bool: + framework, _, repo = pair.partition("::") + workflows = PARITY_FIXTURES / repo / ".github" / "workflows" + has_workflow_file = any(p.suffix.lower() in (".yml", ".yaml") for p in workflows.glob("*")) + return ( + framework == "openssf-baseline" + and control_id in WARN_BY_DESIGN + and (before, after) == ("ERROR", "WARN") + and not has_workflow_file + ) @pytest.mark.integration def test_warn_outcome_changes_no_existing_control() -> None: """SC-008 / FR-017. - No existing handler returns WARN, so neutralizing the outcome must be - invisible. A difference here means either a handler started returning WARN - or the disposition refactor changed how PASS/FAIL/INCONCLUSIVE are routed. + Only `github_workflows_present` returns WARN (#584), so neutralizing the + outcome must be invisible everywhere else. A difference here means either + another handler started returning WARN or the disposition refactor changed + how PASS/FAIL/INCONCLUSIVE are routed. """ with_warn = capture() without_warn = capture(neutralize=True) @@ -44,6 +64,9 @@ def test_warn_outcome_changes_no_existing_control() -> None: control_id: (without_warn[pair].get(control_id), with_warn[pair].get(control_id)) for control_id in set(with_warn[pair]) | set(without_warn[pair]) if with_warn[pair].get(control_id) != without_warn[pair].get(control_id) + and not _warn_by_design( + pair, control_id, without_warn[pair].get(control_id), with_warn[pair].get(control_id) + ) } if changed: drift[pair] = changed diff --git a/tests/darnit/sieve/test_expr_failure.py b/tests/darnit/sieve/test_expr_failure.py index 2e2997616..9d90ee2cb 100644 --- a/tests/darnit/sieve/test_expr_failure.py +++ b/tests/darnit/sieve/test_expr_failure.py @@ -47,6 +47,8 @@ "OSPS-BR-01.01": "template-injection", "OSPS-AC-04.02": "excessive-permissions", } +# The zizmor step follows github_workflows_present (#584). +ZIZMOR_STEP = 1 def _context(repo: Path, control_id: str, project: dict[str, Any] | None = None) -> CheckContext: @@ -114,8 +116,8 @@ def test_no_findings_document_is_error(self, control_id: str, repo: Path, zizmor assert result.status == "ERROR" assert result.error["class"] == "evaluation" - assert result.resolving_pass_index == 0 - first = result.pass_history[0].result + assert result.resolving_pass_index == ZIZMOR_STEP + first = result.pass_history[ZIZMOR_STEP].result assert first.outcome.value == "error" assert first.evidence["expr"].startswith("!output.json.exists") assert first.evidence["expr_error"] @@ -130,8 +132,8 @@ def test_matching_finding_is_fail(self, control_id: str, repo: Path, zizmor) -> result = _verify(_shipped(control_id), repo) assert result.status == "FAIL" - assert result.resolving_pass_index == 0 - assert result.pass_history[0].result.evidence["json"] == [{"ident": ZIZMOR_CONTROLS[control_id]}] + assert result.resolving_pass_index == ZIZMOR_STEP + assert result.pass_history[ZIZMOR_STEP].result.evidence["json"] == [{"ident": ZIZMOR_CONTROLS[control_id]}] @pytest.mark.parametrize("control_id", sorted(ZIZMOR_CONTROLS)) def test_other_findings_are_pass(self, control_id: str, repo: Path, zizmor) -> None: @@ -315,7 +317,7 @@ def test_invalid_use_fails_loading(self, step: dict[str, Any], message: str) -> def test_shipped_zizmor_steps_declare_it(self) -> None: for control_id in ZIZMOR_CONTROLS: - assert _shipped(control_id).metadata["handler_invocations"][0].expr_decides is True + assert _shipped(control_id).metadata["handler_invocations"][ZIZMOR_STEP].expr_decides is True def _write_project(repo: Path, darnit: dict[str, Any]) -> None: diff --git a/tests/darnit_baseline/controls/test_no_workflows.py b/tests/darnit_baseline/controls/test_no_workflows.py new file mode 100644 index 000000000..36b50b727 --- /dev/null +++ b/tests/darnit_baseline/controls/test_no_workflows.py @@ -0,0 +1,461 @@ +"""#584: controls that audit GitHub Actions workflows, on a repository without any. + +OSPS-BR-01.01, OSPS-BR-01.02, and OSPS-AC-04.02 run while ``ci_provider`` is +unconfirmed. Without workflow files zizmor exits 3 and ``grep`` exits 2, and +the controls used to end ``WARN [unexpected_exit]`` with a generic message and +log an ``unexpected_exit`` WARNING per command. ``github_workflows_present`` +now runs first: it ends them WARN with its reason, and neither command runs. +With workflow files present every decision is the one the controls give +without that step, one step index later; so is BR-01.01's with only a +standalone action definition, which zizmor's template-injection audit reads. + +Commands are answered by a stand-in for ``subprocess.run`` with the exit +codes and stderr the real tools give (#584), so the tests do not depend on +zizmor, ``grep``, or ``sh`` being installed. +""" + +from __future__ import annotations + +import dataclasses +import json +import logging +import subprocess +from pathlib import Path +from typing import Any + +import pytest + +from darnit.cli import format_result_text +from darnit.config import load_framework_config +from darnit.config.control_loader import control_from_framework +from darnit.sieve import builtin_handlers +from darnit.sieve.builtin_handlers import _classify_exec_failure +from darnit.sieve.handler_registry import reset_sieve_handler_registry +from darnit.sieve.models import CheckContext, ControlSpec, SieveResult +from darnit.sieve.orchestrator import SieveOrchestrator +from darnit_baseline.github_workflows import NO_WORKFLOWS_MESSAGE, NO_WORKFLOWS_OR_ACTIONS_MESSAGE + +REPO_ROOT = Path(__file__).resolve().parents[3] +BASELINE_TOML = REPO_ROOT / "packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml" +CONTROLS = ("OSPS-BR-01.01", "OSPS-BR-01.02", "OSPS-AC-04.02") +# BR-01.01's zizmor audit (template-injection) also reads action definitions. +NOTHING_TO_AUDIT = { + "OSPS-BR-01.01": NO_WORKFLOWS_OR_ACTIONS_MESSAGE, + "OSPS-BR-01.02": NO_WORKFLOWS_MESSAGE, + "OSPS-AC-04.02": NO_WORKFLOWS_MESSAGE, +} +GUARD = "github_workflows_present" +EXEC_LOGGER = builtin_handlers.logger.name + +ZIZMOR_NO_INPUTS = (3, "", "fatal: no audit was performed\nerror: no inputs collected\n") +GREP_NO_DIRECTORY = (2, "", "grep: .github/workflows/: No such file or directory\n") +# `! grep -rEq ... .github/workflows 2>/dev/null` exits 0 when grep exits 2. +SH_NEGATED_GREP = (0, "", "") +NO_WORKFLOW_TOOLS = {"zizmor": ZIZMOR_NO_INPUTS, "grep": GREP_NO_DIRECTORY, "sh": SH_NEGATED_GREP} + +WORKFLOW = "name: ci\non: push\npermissions: read-all\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - run: echo hi\n" + + +@pytest.fixture(autouse=True) +def _baseline_handlers(): + reset_sieve_handler_registry() + from darnit.core.discovery import get_implementation + + impl = get_implementation("openssf-baseline") + assert impl is not None + impl.register_handlers() + + +class Tools: + """``outcomes[executable] = (exit_code, stdout, stderr)``, or None for not installed; ``calls`` records each run.""" + + def __init__(self) -> None: + self.outcomes: dict[str, tuple[int, str, str] | None] = {} + self.calls: list[str] = [] + + +@pytest.fixture +def tools(monkeypatch: pytest.MonkeyPatch) -> Tools: + """Stand-in for ``subprocess.run`` for the stubbed executables; other commands run for real.""" + stub = Tools() + real_run = subprocess.run + + def run(cmd: Any, *args: Any, **kwargs: Any) -> subprocess.CompletedProcess: + if isinstance(cmd, (list, tuple)) and cmd and cmd[0] in stub.outcomes: + stub.calls.append(cmd[0]) + outcome = stub.outcomes[cmd[0]] + if outcome is None: + raise FileNotFoundError(cmd[0]) + code, stdout, stderr = outcome + return subprocess.CompletedProcess(cmd, code, stdout, stderr) + return real_run(cmd, *args, **kwargs) + + monkeypatch.setattr(builtin_handlers.subprocess, "run", run) + return stub + + +def _shipped(control_id: str) -> ControlSpec: + framework = load_framework_config(BASELINE_TOML) + return control_from_framework( + control_id, framework.controls[control_id].model_copy(deep=True), framework=framework.metadata.name + ) + + +def _without_guard(spec: ControlSpec) -> ControlSpec: + steps = [s for s in spec.metadata["handler_invocations"] if s.handler != GUARD] + return dataclasses.replace(spec, metadata={**spec.metadata, "handler_invocations": steps}) + + +def _verify(spec: ControlSpec, repo: Path, project: dict[str, Any] | None = None) -> SieveResult: + context = CheckContext( + owner="example-org", + repo="example", + local_path=str(repo), + default_branch="main", + control_id=spec.control_id, + project_context=dict(project or {}), + ) + return SieveOrchestrator(stop_on_llm=False).verify(spec, context) + + +def _decision(result: SieveResult) -> tuple: + return (result.status, result.message, result.error_class, result.error, result.concluded_by, result.authority) + + +def _steps(result: SieveResult) -> list[tuple]: + return [(a.checks_performed, a.result.outcome, a.result.message) for a in result.pass_history] + + +def _exec_warnings(caplog: pytest.LogCaptureFixture) -> list[str]: + return [ + r.getMessage() + for r in caplog.records + if r.name == EXEC_LOGGER and r.levelno >= logging.WARNING and "could not complete" in r.getMessage() + ] + + +def _with_workflow(tmp_path: Path, name: str) -> Path: + repo = tmp_path / "repo" + (repo / ".github" / "workflows").mkdir(parents=True) + (repo / ".github" / "workflows" / name).write_text(WORKFLOW, encoding="utf-8") + return repo + + +def _no_github(tmp_path: Path) -> Path: + repo = tmp_path / "repo" + repo.mkdir() + (repo / "README.md").write_text("# example\n", encoding="utf-8") + return repo + + +def _empty_workflows(tmp_path: Path) -> Path: + repo = _no_github(tmp_path) + (repo / ".github" / "workflows").mkdir(parents=True) + return repo + + +def _no_yaml_workflows(tmp_path: Path) -> Path: + repo = _empty_workflows(tmp_path) + (repo / ".github" / "workflows" / "README.md").write_text("CI lives elsewhere\n", encoding="utf-8") + return repo + + +NO_WORKFLOW_REPOS = {"no-github-dir": _no_github, "empty-workflows": _empty_workflows, "no-yaml": _no_yaml_workflows} + + +@pytest.mark.parametrize("layout", sorted(NO_WORKFLOW_REPOS)) +@pytest.mark.parametrize("control_id", CONTROLS) +class TestNoWorkflowFiles: + def test_warn_says_there_is_nothing_to_audit(self, control_id, layout, tmp_path, tools) -> None: + tools.outcomes.update(NO_WORKFLOW_TOOLS) + + result = _verify(_shipped(control_id), NO_WORKFLOW_REPOS[layout](tmp_path)) + + assert result.status == "WARN" + assert result.message == NOTHING_TO_AUDIT[control_id] + assert result.error_class is None + assert result.error is None + assert (result.concluded_by, result.resolving_pass_index) == (GUARD, 0) + assert [a.checks_performed for a in result.pass_history] == [[f"handler:{GUARD}"]] + + def test_no_command_runs_and_nothing_is_logged(self, control_id, layout, tmp_path, tools, caplog) -> None: + tools.outcomes.update(NO_WORKFLOW_TOOLS) + caplog.set_level(logging.WARNING) + + _verify(_shipped(control_id), NO_WORKFLOW_REPOS[layout](tmp_path)) + + assert tools.calls == [] + assert _exec_warnings(caplog) == [] + assert "unexpected_exit" not in caplog.text + + def test_without_the_step_it_was_unexpected_exit(self, control_id, layout, tmp_path, tools, caplog) -> None: + """The #584 report, reproduced with the step removed, so the assertions above are not vacuous.""" + tools.outcomes.update(NO_WORKFLOW_TOOLS) + caplog.set_level(logging.WARNING) + + before = _verify(_without_guard(_shipped(control_id)), NO_WORKFLOW_REPOS[layout](tmp_path)) + + assert (before.status, before.error_class, before.concluded_by) == ("WARN", "unexpected_exit", "none") + # #692 reports the last step's own message on this fallthrough, so the + # wording is no longer the generic manual-verification sentence. It is + # still the misleading result: an unexpected_exit, not the no-workflows reason. + assert before.message != NOTHING_TO_AUDIT[control_id] + assert "nothing to audit" not in before.message + assert tools.calls + assert any("error_class=unexpected_exit" in m for m in _exec_warnings(caplog)) + + def test_text_line_names_the_reason_without_an_error_class(self, control_id, layout, tmp_path, tools) -> None: + tools.outcomes.update(NO_WORKFLOW_TOOLS) + + line = format_result_text(_verify(_shipped(control_id), NO_WORKFLOW_REPOS[layout](tmp_path)).to_legacy_dict()) + + assert f"{control_id}: WARN - No GitHub Actions workflow files" in line + assert "[" not in line.split(" - ", 1)[0] + + def test_tools_not_installed(self, control_id, layout, tmp_path, tools) -> None: + tools.outcomes.update({"zizmor": None, "grep": None, "sh": None}) + + result = _verify(_shipped(control_id), NO_WORKFLOW_REPOS[layout](tmp_path)) + + assert (result.status, result.message, result.error_class) == ("WARN", NOTHING_TO_AUDIT[control_id], None) + assert tools.calls == [] + + +@pytest.mark.parametrize("control_id", CONTROLS) +class TestApplicability: + def test_unconfirmed_ci_provider_is_not_pass_fail_or_na(self, control_id, tmp_path, tools) -> None: + tools.outcomes.update(NO_WORKFLOW_TOOLS) + + result = _verify(_shipped(control_id), _no_github(tmp_path), project={}) + + assert result.status not in ("PASS", "FAIL", "N/A") + assert result.status == "WARN" + + @pytest.mark.parametrize("provider", ["gitlab", "none"]) + def test_confirmed_non_github_provider_is_na(self, control_id, provider, tmp_path, tools) -> None: + tools.outcomes.update(NO_WORKFLOW_TOOLS) + + result = _verify(_shipped(control_id), _no_github(tmp_path), project={"ci_provider": provider}) + + assert result.status == "N/A" + assert result.pass_history == [] + assert tools.calls == [] + + def test_confirmed_github_without_workflows_warns(self, control_id, tmp_path, tools) -> None: + tools.outcomes.update(NO_WORKFLOW_TOOLS) + + result = _verify(_shipped(control_id), _no_github(tmp_path), project={"ci_provider": "github"}) + + assert (result.status, result.message) == ("WARN", NOTHING_TO_AUDIT[control_id]) + + +# Tool answers on a repository with a workflow file. +WORKFLOW_SCENARIOS = { + "zizmor-clean": {"zizmor": (0, "[]", ""), "grep": (1, "", ""), "sh": (0, "", "")}, + "zizmor-template-injection": { + "zizmor": (14, json.dumps([{"ident": "template-injection"}]), ""), + "grep": (1, "", ""), + "sh": (1, "", ""), + }, + "zizmor-excessive-permissions": { + "zizmor": (14, json.dumps([{"ident": "excessive-permissions"}]), ""), + "grep": (1, "", ""), + "sh": (0, "", ""), + }, + "zizmor-no-findings-document": {"zizmor": (0, "", ""), "grep": (1, "", ""), "sh": (0, "", "")}, + "zizmor-unexpected-exit": {"zizmor": (2, "", "error: unrecognized input"), "grep": (1, "", ""), "sh": (0, "", "")}, + "tools-missing": {"zizmor": None, "grep": None, "sh": None}, + "grep-match-647": {"zizmor": (0, "[]", ""), "grep": (0, ".github/workflows/ci.yml\n", ""), "sh": (0, "", "")}, +} + + +@pytest.mark.parametrize("scenario", sorted(WORKFLOW_SCENARIOS)) +@pytest.mark.parametrize("name", ["ci.yml", "ci.yaml"]) +@pytest.mark.parametrize("control_id", CONTROLS) +class TestWorkflowFilesPresent: + def test_decision_is_unchanged_one_step_later(self, control_id, name, scenario, tmp_path, tools, caplog) -> None: + tools.outcomes.update(WORKFLOW_SCENARIOS[scenario]) + spec = _shipped(control_id) + repo = _with_workflow(tmp_path, name) + caplog.set_level(logging.WARNING) + + before = _verify(_without_guard(spec), repo) + before_calls, before_logs = list(tools.calls), _exec_warnings(caplog) + tools.calls.clear() + caplog.clear() + after = _verify(spec, repo) + + assert _decision(after) == _decision(before) + shift = None if before.resolving_pass_index is None else before.resolving_pass_index + 1 + assert after.resolving_pass_index == shift + assert after.pass_history[0].checks_performed == [f"handler:{GUARD}"] + assert after.pass_history[0].result.outcome.value == "inconclusive" + assert _steps(after)[1:] == _steps(before) + assert tools.calls == before_calls + assert _exec_warnings(caplog) == before_logs + + +class TestKnownResultsWithWorkflows: + """Spot checks of the pairs above, so an unchanged-but-wrong pair cannot pass.""" + + @pytest.mark.parametrize("control_id", ["OSPS-BR-01.01", "OSPS-AC-04.02"]) + def test_clean_zizmor_passes_at_the_zizmor_step(self, control_id, tmp_path, tools) -> None: + tools.outcomes.update(WORKFLOW_SCENARIOS["zizmor-clean"]) + + result = _verify(_shipped(control_id), _with_workflow(tmp_path, "ci.yml")) + + assert (result.status, result.concluded_by, result.resolving_pass_index) == ("PASS", "exec", 1) + + @pytest.mark.parametrize("control_id", ["OSPS-BR-01.01", "OSPS-AC-04.02"]) + def test_zizmor_unexpected_exit_keeps_its_class(self, control_id, tmp_path, tools) -> None: + tools.outcomes.update(WORKFLOW_SCENARIOS["zizmor-unexpected-exit"]) + + result = _verify(_shipped(control_id), _with_workflow(tmp_path, "ci.yml")) + + assert (result.status, result.error_class) == ("WARN", "unexpected_exit") + + def test_grep_match_is_evidence_without_unexpected_exit(self, tmp_path, tools) -> None: + """#656: grep exit 0 is a declared failure, and concludes = [] leaves it non-conclusive.""" + tools.outcomes.update(WORKFLOW_SCENARIOS["grep-match-647"]) + + result = _verify(_shipped("OSPS-BR-01.02"), _with_workflow(tmp_path, "ci.yml")) + + assert (result.status, result.error_class, result.concluded_by) == ("WARN", None, "none") + grep = result.pass_history[1] + assert grep.checks_performed == ["handler:exec"] + assert grep.result.outcome.value == "fail" + assert grep.result.evidence.get("exit_code") == 0 + + def test_grep_no_match_is_undetermined_warn(self, tmp_path, tools) -> None: + tools.outcomes.update(WORKFLOW_SCENARIOS["zizmor-clean"]) + + result = _verify(_shipped("OSPS-BR-01.02"), _with_workflow(tmp_path, "ci.yaml")) + + assert (result.status, result.error_class, result.concluded_by) == ("WARN", None, "none") + + @pytest.mark.parametrize("control_id", CONTROLS) + def test_missing_tools_stay_error(self, control_id, tmp_path, tools) -> None: + tools.outcomes.update(WORKFLOW_SCENARIOS["tools-missing"]) + + result = _verify(_shipped(control_id), _with_workflow(tmp_path, "ci.yml")) + + assert (result.status, result.error_class) == ("ERROR", "missing_tool") + + +# A composite action with the injection zizmor's template-injection audit reports. +ACTION = ( + "name: greet\nruns:\n using: composite\n steps:\n" + ' - run: echo "${{ github.event.issue.title }}"\n shell: bash\n' +) + + +def _with_action(tmp_path: Path, path: str, empty_workflows: bool = False) -> Path: + repo = _no_github(tmp_path) + if empty_workflows: + (repo / ".github" / "workflows").mkdir(parents=True) + (repo / path).parent.mkdir(parents=True, exist_ok=True) + (repo / path).write_text(ACTION, encoding="utf-8") + return repo + + +ACTION_REPOS = { + "root-action.yml": lambda tmp: _with_action(tmp, "action.yml"), + "nested-action.yaml": lambda tmp: _with_action(tmp, ".github/actions/setup/action.yaml"), + "action-and-empty-workflows": lambda tmp: _with_action(tmp, "action.yml", empty_workflows=True), +} +# Tool answers on a repository whose only GitHub Actions input is an action +# definition; `sh -c '! grep ... .github/workflows 2>/dev/null'` exits 0. +ACTION_SCENARIOS = { + "zizmor-clean": {"zizmor": (0, "[]", ""), "sh": SH_NEGATED_GREP}, + "zizmor-template-injection": { + "zizmor": (13, json.dumps([{"ident": "template-injection"}]), ""), + "sh": SH_NEGATED_GREP, + }, + "zizmor-other-finding": {"zizmor": (12, json.dumps([{"ident": "unpinned-uses"}]), ""), "sh": SH_NEGATED_GREP}, + "zizmor-no-findings-document": {"zizmor": (0, "", ""), "sh": SH_NEGATED_GREP}, + "zizmor-missing": {"zizmor": None, "sh": SH_NEGATED_GREP}, +} + + +@pytest.mark.parametrize("layout", sorted(ACTION_REPOS)) +class TestStandaloneActionDefinitions: + """zizmor collects action.yml/action.yaml from the whole tree and its template-injection audit reads them.""" + + @pytest.mark.parametrize("scenario", sorted(ACTION_SCENARIOS)) + def test_br_01_01_decision_is_unchanged_one_step_later(self, layout, scenario, tmp_path, tools, caplog) -> None: + tools.outcomes.update(ACTION_SCENARIOS[scenario]) + spec = _shipped("OSPS-BR-01.01") + repo = ACTION_REPOS[layout](tmp_path) + caplog.set_level(logging.WARNING) + + before = _verify(_without_guard(spec), repo) + before_calls, before_logs = list(tools.calls), _exec_warnings(caplog) + tools.calls.clear() + caplog.clear() + after = _verify(spec, repo) + + assert _decision(after) == _decision(before) + shift = None if before.resolving_pass_index is None else before.resolving_pass_index + 1 + assert after.resolving_pass_index == shift + assert after.pass_history[0].checks_performed == [f"handler:{GUARD}"] + assert after.pass_history[0].result.outcome.value == "inconclusive" + assert _steps(after)[1:] == _steps(before) + assert tools.calls == before_calls + assert "zizmor" in tools.calls + assert _exec_warnings(caplog) == before_logs + + def test_br_01_01_template_injection_in_an_action_is_fail(self, layout, tmp_path, tools) -> None: + tools.outcomes.update(ACTION_SCENARIOS["zizmor-template-injection"]) + + result = _verify(_shipped("OSPS-BR-01.01"), ACTION_REPOS[layout](tmp_path)) + + assert (result.status, result.concluded_by, result.resolving_pass_index) == ("FAIL", "exec", 1) + assert result.error_class is None + + def test_br_01_01_clean_action_passes_at_the_zizmor_step(self, layout, tmp_path, tools) -> None: + tools.outcomes.update(ACTION_SCENARIOS["zizmor-clean"]) + + result = _verify(_shipped("OSPS-BR-01.01"), ACTION_REPOS[layout](tmp_path)) + + assert (result.status, result.concluded_by, result.resolving_pass_index) == ("PASS", "exec", 1) + + def test_br_01_01_zizmor_missing_is_still_error(self, layout, tmp_path, tools) -> None: + tools.outcomes.update(ACTION_SCENARIOS["zizmor-missing"]) + + result = _verify(_shipped("OSPS-BR-01.01"), ACTION_REPOS[layout](tmp_path)) + + assert (result.status, result.error_class) == ("ERROR", "missing_tool") + + @pytest.mark.parametrize("control_id", ["OSPS-BR-01.02", "OSPS-AC-04.02"]) + def test_workflow_only_audits_still_warn_without_running(self, layout, control_id, tmp_path, tools, caplog) -> None: + """excessive-permissions reads workflows only and grep reads .github/workflows/ only.""" + tools.outcomes.update(NO_WORKFLOW_TOOLS) + caplog.set_level(logging.WARNING) + + result = _verify(_shipped(control_id), ACTION_REPOS[layout](tmp_path)) + + assert (result.status, result.message, result.error_class) == ("WARN", NO_WORKFLOWS_MESSAGE, None) + assert (result.concluded_by, result.resolving_pass_index) == (GUARD, 0) + assert tools.calls == [] + assert _exec_warnings(caplog) == [] + + +@pytest.mark.parametrize("path", [".github/dependabot.yml", ".pre-commit-config.yaml", ".git/x/action.yml"]) +def test_br_01_01_inputs_template_injection_does_not_read_still_warn(path, tmp_path, tools, caplog) -> None: + """zizmor may collect these, but its template-injection audit reads only workflows and actions.""" + tools.outcomes.update(NO_WORKFLOW_TOOLS) + caplog.set_level(logging.WARNING) + repo = _no_github(tmp_path) + (repo / path).parent.mkdir(parents=True, exist_ok=True) + (repo / path).write_text("version: 2\n", encoding="utf-8") + + result = _verify(_shipped("OSPS-BR-01.01"), repo) + + assert (result.status, result.message, result.concluded_by) == ("WARN", NO_WORKFLOWS_OR_ACTIONS_MESSAGE, GUARD) + assert tools.calls == [] + assert _exec_warnings(caplog) == [] + + +def test_562_classification_is_unchanged() -> None: + assert _classify_exec_failure(GREP_NO_DIRECTORY[2], 2, "grep") == "unexpected_exit" + assert _classify_exec_failure(ZIZMOR_NO_INPUTS[2], 3, "zizmor") == "unexpected_exit" diff --git a/tests/darnit_baseline/controls/test_secure_branch_names.py b/tests/darnit_baseline/controls/test_secure_branch_names.py index 7c62c683b..85a79ac34 100644 --- a/tests/darnit_baseline/controls/test_secure_branch_names.py +++ b/tests/darnit_baseline/controls/test_secure_branch_names.py @@ -63,7 +63,9 @@ def test_toml_declares_fail_exit_codes(self) -> None: config = load_framework_by_name("openssf-baseline") control = config.controls["OSPS-BR-01.02"] assert control.passes, "OSPS-BR-01.02 must have passes" - exec_step = control.passes[0] + # github_workflows_present is the first step (#584); the grep step follows it. + assert control.passes[0].handler == "github_workflows_present" + exec_step = control.passes[1] assert exec_step.handler == "exec" assert exec_step.model_extra.get("pass_exit_codes") == [1] assert exec_step.model_extra.get("fail_exit_codes") == [0] @@ -78,8 +80,10 @@ def test_grep_match_records_evidence_without_unexpected_exit(self, tmp_path: Pat result = SieveOrchestrator().verify(ctrl, _ctx(tmp_path)) assert result.status == "WARN" assert result.error_class is None - # The exec step was recorded in pass_history with exit 0 evidence without concluding - exec_attempt = result.pass_history[0] + # The guard is evidence only when a workflow file exists; the grep step follows it. + assert result.pass_history[0].checks_performed == ["handler:github_workflows_present"] + exec_attempt = result.pass_history[1] + assert exec_attempt.checks_performed == ["handler:exec"] assert exec_attempt.result.outcome == PassOutcome.FAIL assert exec_attempt.result.evidence.get("exit_code") == 0 assert "ci.yml" in exec_attempt.result.evidence.get("stdout", "") @@ -93,6 +97,8 @@ def test_grep_no_match_passes_cleanly(self, tmp_path: Path) -> None: result = SieveOrchestrator().verify(ctrl, _ctx(tmp_path)) assert result.status == "WARN" assert result.error_class is None - exec_attempt = result.pass_history[0] + assert result.pass_history[0].checks_performed == ["handler:github_workflows_present"] + exec_attempt = result.pass_history[1] + assert exec_attempt.checks_performed == ["handler:exec"] assert exec_attempt.result.outcome == PassOutcome.PASS assert exec_attempt.result.evidence.get("exit_code") == 1 diff --git a/tests/darnit_baseline/test_github_workflows.py b/tests/darnit_baseline/test_github_workflows.py new file mode 100644 index 000000000..688168cf8 --- /dev/null +++ b/tests/darnit_baseline/test_github_workflows.py @@ -0,0 +1,293 @@ +"""The ``github_workflows_present`` step type (#584). + +WARN with a reason when ``.github/workflows/`` holds no ``.yml``/``.yaml`` +file inside the repository (and, with ``actions = true``, the repository +holds no ``action.yml``/``action.yaml``), INCONCLUSIVE otherwise; never PASS +or FAIL. +""" + +from __future__ import annotations + +import os +from pathlib import Path + +import pytest + +from darnit.config import load_framework_config +from darnit.sieve.handler_registry import ( + HandlerContext, + HandlerResultStatus, + get_sieve_handler_registry, + reset_sieve_handler_registry, +) +from darnit_baseline.github_workflows import ( + NO_WORKFLOWS_MESSAGE, + NO_WORKFLOWS_OR_ACTIONS_MESSAGE, + github_workflows_present_handler, +) + +REPO_ROOT = Path(__file__).resolve().parents[2] +BASELINE_TOML = REPO_ROOT / "packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml" +WORKFLOW_CONTROLS = ("OSPS-BR-01.01", "OSPS-BR-01.02", "OSPS-AC-04.02") +ACTIONS = {"actions": True} + + +def _run(repo: Path, config: dict | None = None): + ctx = HandlerContext(local_path=str(repo), owner="o", repo="r", default_branch="main", control_id="TEST-01") + return github_workflows_present_handler(dict(config or {}), ctx) + + +def _write(path: Path, text: str = "runs:\n using: composite\n steps: []\n") -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + return path + + +def _workflows(repo: Path) -> Path: + path = repo / ".github" / "workflows" + path.mkdir(parents=True) + return path + + +class TestNoWorkflowFiles: + def test_no_github_directory(self, tmp_path: Path) -> None: + result = _run(tmp_path) + + assert result.status == HandlerResultStatus.WARN + assert result.message == NO_WORKFLOWS_MESSAGE + assert result.error_class is None + assert result.evidence["workflow_files"] == [] + + def test_empty_workflows_directory(self, tmp_path: Path) -> None: + _workflows(tmp_path) + + assert _run(tmp_path).status == HandlerResultStatus.WARN + + def test_workflows_directory_without_yaml_files(self, tmp_path: Path) -> None: + workflows = _workflows(tmp_path) + (workflows / "README.md").write_text("no workflows\n", encoding="utf-8") + (workflows / "ci.yml.disabled").write_text("on: push\n", encoding="utf-8") + (workflows / "nested.yml").mkdir() + + result = _run(tmp_path) + + assert result.status == HandlerResultStatus.WARN + assert result.evidence["workflow_files"] == [] + + def test_workflows_path_is_a_file(self, tmp_path: Path) -> None: + (tmp_path / ".github").mkdir() + (tmp_path / ".github" / "workflows").write_text("not a directory\n", encoding="utf-8") + + assert _run(tmp_path).status == HandlerResultStatus.WARN + + def test_yaml_only_in_a_subdirectory_does_not_count(self, tmp_path: Path) -> None: + nested = _workflows(tmp_path) / "archive" + nested.mkdir() + (nested / "old.yml").write_text("on: push\n", encoding="utf-8") + + assert _run(tmp_path).status == HandlerResultStatus.WARN + + def test_message_says_nothing_to_audit(self) -> None: + assert "No GitHub Actions workflow files" in NO_WORKFLOWS_MESSAGE + assert "nothing to audit" in NO_WORKFLOWS_MESSAGE + + +class TestWorkflowFilesPresent: + @pytest.mark.parametrize("name", ["ci.yml", "ci.yaml", "Release.YML"]) + def test_workflow_file_is_evidence_only(self, tmp_path: Path, name: str) -> None: + (_workflows(tmp_path) / name).write_text("on: push\n", encoding="utf-8") + + result = _run(tmp_path) + + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert result.error_class is None + assert result.evidence["workflow_files"] == [f".github/workflows/{name}"] + + def test_lists_every_workflow_file(self, tmp_path: Path) -> None: + workflows = _workflows(tmp_path) + for name in ("b.yaml", "a.yml", "notes.txt"): + (workflows / name).write_text("on: push\n", encoding="utf-8") + + result = _run(tmp_path) + + assert result.evidence["workflow_files"] == [".github/workflows/a.yml", ".github/workflows/b.yaml"] + + +class TestRepositoryBoundary: + def test_workflow_linked_outside_the_repository_does_not_count(self, tmp_path: Path) -> None: + outside = tmp_path / "outside.yml" + outside.write_text("on: push\n", encoding="utf-8") + repo = tmp_path / "repo" + workflows = _workflows(repo) + try: + os.symlink(outside, workflows / "ci.yml") + except (OSError, NotImplementedError): + pytest.skip("symbolic links are not available") + + result = _run(repo) + + assert result.status == HandlerResultStatus.WARN + assert result.evidence["skipped_outside_repository"] == [".github/workflows/ci.yml"] + + def test_action_linked_outside_the_repository_does_not_count(self, tmp_path: Path) -> None: + outside = _write(tmp_path / "outside" / "action.yml") + repo = tmp_path / "repo" + repo.mkdir() + try: + os.symlink(outside, repo / "action.yml") + except (OSError, NotImplementedError): + pytest.skip("symbolic links are not available") + + result = _run(repo, ACTIONS) + + assert result.status == HandlerResultStatus.WARN + assert result.evidence["skipped_outside_repository"] == ["action.yml"] + + +class TestActionDefinitions: + """``actions = true``: an action definition is something zizmor's template-injection audit reads.""" + + @pytest.mark.parametrize( + "path", ["action.yml", "action.yaml", ".github/actions/setup/action.yml", "tools/greet/action.yaml"] + ) + def test_action_definition_is_evidence_only(self, tmp_path: Path, path: str) -> None: + _write(tmp_path / path) + + result = _run(tmp_path, ACTIONS) + + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert result.error_class is None + assert result.evidence["workflow_files"] == [] + assert result.evidence["action_files"] == [path] + + def test_with_empty_workflows_directory(self, tmp_path: Path) -> None: + _workflows(tmp_path) + _write(tmp_path / "action.yml") + + assert _run(tmp_path, ACTIONS).status == HandlerResultStatus.INCONCLUSIVE + + def test_without_the_setting_action_definitions_do_not_count(self, tmp_path: Path) -> None: + _write(tmp_path / "action.yml") + + result = _run(tmp_path) + + assert (result.status, result.message) == (HandlerResultStatus.WARN, NO_WORKFLOWS_MESSAGE) + assert "action_files" not in result.evidence + + @pytest.mark.parametrize("value", [False, "true", 1]) + def test_only_true_enables_the_setting(self, tmp_path: Path, value) -> None: + _write(tmp_path / "action.yml") + + assert _run(tmp_path, {"actions": value}).status == HandlerResultStatus.WARN + + def test_nothing_to_audit_names_both_inputs(self, tmp_path: Path) -> None: + (tmp_path / "README.md").write_text("# example\n", encoding="utf-8") + + result = _run(tmp_path, ACTIONS) + + assert (result.status, result.message) == (HandlerResultStatus.WARN, NO_WORKFLOWS_OR_ACTIONS_MESSAGE) + assert result.error_class is None + assert "nothing to audit" in NO_WORKFLOWS_OR_ACTIONS_MESSAGE + + @pytest.mark.parametrize( + "path", + [".github/dependabot.yml", ".pre-commit-config.yaml", ".pre-commit-hooks.yaml", "actions.yml", "my-action.yml"], + ) + def test_other_files_do_not_count(self, tmp_path: Path, path: str) -> None: + _write(tmp_path / path) + + assert _run(tmp_path, ACTIONS).status == HandlerResultStatus.WARN + + def test_git_directory_is_not_searched(self, tmp_path: Path) -> None: + _write(tmp_path / ".git" / "modules" / "vendored" / "action.yml") + + assert _run(tmp_path, ACTIONS).status == HandlerResultStatus.WARN + + def test_directory_named_action_yml_does_not_count(self, tmp_path: Path) -> None: + (tmp_path / "action.yml").mkdir() + + assert _run(tmp_path, ACTIONS).status == HandlerResultStatus.WARN + + def test_workflows_are_reported_without_searching_for_actions(self, tmp_path: Path) -> None: + (_workflows(tmp_path) / "ci.yml").write_text("on: push\n", encoding="utf-8") + _write(tmp_path / "action.yml") + + result = _run(tmp_path, ACTIONS) + + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert result.evidence["workflow_files"] == [".github/workflows/ci.yml"] + assert "action_files" not in result.evidence + + +class TestNeverPassOrFail: + @pytest.mark.parametrize("config", [{}, ACTIONS]) + @pytest.mark.parametrize("files", [[], ["README.md"], ["ci.yml"], ["ci.yaml", "x.txt"], ["action.yml"]]) + def test_outcome_is_warn_or_inconclusive(self, tmp_path: Path, files: list[str], config: dict) -> None: + workflows = _workflows(tmp_path) + for name in files: + (workflows / name).write_text("on: push\n", encoding="utf-8") + + assert _run(tmp_path, config).status in (HandlerResultStatus.WARN, HandlerResultStatus.INCONCLUSIVE) + + +class TestRegistration: + @pytest.fixture(autouse=True) + def _registered(self): + reset_sieve_handler_registry() + from darnit.core.discovery import get_implementation + + impl = get_implementation("openssf-baseline") + assert impl is not None + impl.register_handlers() + + def test_ceiling_is_fail_only(self) -> None: + info = get_sieve_handler_registry().get("github_workflows_present") + + assert info is not None + assert set(info.ceiling) == {"fail"} + assert info.existence_ceiling is None + assert set(info.settings) == {"actions"} + + +class TestShippedSteps: + @pytest.fixture(scope="class") + def controls(self): + return load_framework_config(BASELINE_TOML).controls + + @pytest.mark.parametrize("control_id", WORKFLOW_CONTROLS) + def test_step_runs_first(self, controls, control_id: str) -> None: + handlers = [step.handler for step in controls[control_id].passes] + + assert handlers[:2] == ["github_workflows_present", "exec"] + assert handlers.count("github_workflows_present") == 1 + assert handlers[-1] == "manual" + + @pytest.mark.parametrize( + ("control_id", "settings"), + [("OSPS-BR-01.01", ACTIONS), ("OSPS-BR-01.02", {}), ("OSPS-AC-04.02", {})], + ) + def test_only_br_01_01_counts_action_definitions(self, controls, control_id: str, settings: dict) -> None: + """BR-01.01's zizmor audit (template-injection) reads actions; AC-04.02's (excessive-permissions) + and BR-01.02's grep of .github/workflows/ do not.""" + step = controls[control_id].passes[0] + + assert step.model_extra == settings + assert (step.concludes, step.existence, step.promotion, step.when) == (None, False, None, None) + + def test_no_other_control_uses_it(self, controls) -> None: + using = sorted( + cid + for cid, control in controls.items() + if any(s.handler == "github_workflows_present" for s in control.passes) + ) + + assert using == sorted(WORKFLOW_CONTROLS) + + def test_br_01_02_grep_step_keeps_the_merged_exit_codes(self, controls) -> None: + """#656: a match (exit 0) is a declared failure, and concludes = [] keeps it from concluding.""" + (grep,) = [s for s in controls["OSPS-BR-01.02"].passes if s.handler == "exec"] + + assert grep.model_extra["command"] == ["grep", "-rln", "github.head_ref", ".github/workflows/"] + assert grep.model_extra["pass_exit_codes"] == [1] + assert grep.model_extra["fail_exit_codes"] == [0] + assert grep.concludes == [] From b5937c4b5e5b0ff42614adf43828a9f39bd10904 Mon Sep 17 00:00:00 2001 From: Johnson Lyu Date: Mon, 26 Oct 2026 01:41:56 -0400 Subject: [PATCH 2/3] test: account for no-workflows WARN in baseline regression tests Assisted-by: Cursor:Claude-Opus-5.5 Signed-off-by: Johnson Lyu --- tests/darnit/sieve/baseline_capture.py | 24 ++++- .../sieve/test_control_status_baseline.py | 99 ++++++++++++++++--- 2 files changed, 102 insertions(+), 21 deletions(-) diff --git a/tests/darnit/sieve/baseline_capture.py b/tests/darnit/sieve/baseline_capture.py index 1209014ea..db50a21d9 100644 --- a/tests/darnit/sieve/baseline_capture.py +++ b/tests/darnit/sieve/baseline_capture.py @@ -161,7 +161,12 @@ def patched(handler_status, allowed, is_last_step=False): orchestrator.SieveOrchestrator.__module__ # touch, keeps linters honest -def _capture_one(framework: str, repo_path: str, neutralize: bool = False) -> dict[str, str]: +def _details(result: Any) -> str: + raw = result.get("details") if isinstance(result, dict) else getattr(result, "message", None) + return "" if raw is None else str(raw) + + +def _capture_one(framework: str, repo_path: str, neutralize: bool = False) -> dict[str, dict[str, str]]: """Audit one (framework, repo) pair in THIS process.""" if neutralize: _neutralize_warn() @@ -177,11 +182,19 @@ def _capture_one(framework: str, repo_path: str, neutralize: bool = False) -> di evaluate_claims=False, framework_name=framework, ) - return {_control_id(r): _status(r) for r in results} + return {_control_id(r): {"status": _status(r), "details": _details(r)} for r in results} def capture(neutralize: bool = False) -> dict[str, dict[str, str]]: - """Per-control status for every (framework, repo) pair. + """Per-control status for every (framework, repo) pair.""" + return { + pair: {control_id: result["status"] for control_id, result in controls.items()} + for pair, controls in capture_details(neutralize).items() + } + + +def capture_details(neutralize: bool = False) -> dict[str, dict[str, dict[str, str]]]: + """Per-control ``{"status", "details"}`` for every (framework, repo) pair. Each pair runs in a FRESH SUBPROCESS. This is not defensive style, it is required for the capture to mean anything: control registration is @@ -195,7 +208,7 @@ def capture(neutralize: bool = False) -> dict[str, dict[str, str]]: tracked separately. Subprocess isolation is how this baseline avoids measuring it instead of measuring SC-008. """ - out: dict[str, dict[str, str]] = {} + out: dict[str, dict[str, dict[str, str]]] = {} with tempfile.TemporaryDirectory() as scratch: # Copy once, not once per framework: repo_inputs() is not idempotent # against the same scratch directory. @@ -214,7 +227,8 @@ def capture(neutralize: bool = False) -> dict[str, dict[str, str]]: env=_neutral_env(Path(scratch)), ) if proc.returncode != 0: - out[key] = {"__capture_error__": proc.stderr.strip()[-400:]} + error = proc.stderr.strip()[-400:] + out[key] = {"__capture_error__": {"status": error, "details": error}} continue out[key] = json.loads(proc.stdout) return out diff --git a/tests/darnit/sieve/test_control_status_baseline.py b/tests/darnit/sieve/test_control_status_baseline.py index fcba1113a..c4936d701 100644 --- a/tests/darnit/sieve/test_control_status_baseline.py +++ b/tests/darnit/sieve/test_control_status_baseline.py @@ -22,28 +22,91 @@ import pytest -from .baseline_capture import PARITY_FIXTURES, capture +from .baseline_capture import PARITY_FIXTURES, capture, capture_details # #584: `github_workflows_present`, the first step of these controls, returns -# WARN by design on a repository with no workflow file. Neutralized, the -# controls run on to zizmor and grep, which end them ERROR where the tool is -# not installed (else the same WARN, which is no drift). That ERROR -> WARN on -# a fixture without workflow files is the only drift tolerated. -WARN_BY_DESIGN = {"OSPS-BR-01.01", "OSPS-BR-01.02", "OSPS-AC-04.02"} - - -def _warn_by_design(pair: str, control_id: str, before: str | None, after: str | None) -> bool: +# WARN by design on a repository with nothing for them to audit. Neutralized, +# the controls run on to zizmor and grep instead: ERROR where the tool is not +# installed; PASS for the zizmor controls where the suite's zizmor stand-in +# (tests/conftest.py), which answers "no findings" whatever it is given, is on +# PATH; else the same WARN, which is no drift. BR-01.02's grep step concludes +# nothing, so it never PASSes. Those statuses before the WARN are the only +# drift tolerated. +WARN_BY_DESIGN = { + "OSPS-BR-01.01": {"ERROR", "PASS"}, + "OSPS-BR-01.02": {"ERROR"}, + "OSPS-AC-04.02": {"ERROR", "PASS"}, +} + + +def _nothing_to_audit(repo: str) -> bool: + """No workflow file, and no action definition (which BR-01.01's zizmor audit reads).""" + root = PARITY_FIXTURES / repo + workflows = root / ".github" / "workflows" + if any(p.suffix.lower() in (".yml", ".yaml") for p in workflows.glob("*")): + return False + return not any(p.name.lower() in ("action.yml", "action.yaml") for p in root.rglob("*")) + + +def _warn_by_design(pair: str, control_id: str, before: str | None, after: dict[str, str] | None) -> bool: framework, _, repo = pair.partition("::") - workflows = PARITY_FIXTURES / repo / ".github" / "workflows" - has_workflow_file = any(p.suffix.lower() in (".yml", ".yaml") for p in workflows.glob("*")) return ( framework == "openssf-baseline" - and control_id in WARN_BY_DESIGN - and (before, after) == ("ERROR", "WARN") - and not has_workflow_file + and before in WARN_BY_DESIGN.get(control_id, set()) + and after is not None + and after["status"] == "WARN" + and after["details"].startswith("No GitHub Actions workflow files") + and "nothing to audit" in after["details"] + and _nothing_to_audit(repo) ) +NO_WORKFLOWS_WARN = { + "status": "WARN", + "details": "No GitHub Actions workflow files (.github/workflows/*.yml or *.yaml) were found, " + "so there is nothing to audit.", +} + + +@pytest.mark.parametrize( + ("control_id", "before"), + [("OSPS-BR-01.01", "ERROR"), ("OSPS-BR-01.01", "PASS"), ("OSPS-AC-04.02", "PASS"), ("OSPS-BR-01.02", "ERROR")], +) +def test_tolerated_drift(control_id: str, before: str) -> None: + assert _warn_by_design("openssf-baseline::empty_repo", control_id, before, NO_WORKFLOWS_WARN) + + +@pytest.mark.parametrize( + ("pair", "control_id", "before", "after"), + [ + ("openssf-baseline::empty_repo", "OSPS-BR-01.02", "PASS", NO_WORKFLOWS_WARN), + ("openssf-baseline::empty_repo", "OSPS-BR-01.01", "FAIL", NO_WORKFLOWS_WARN), + ("openssf-baseline::empty_repo", "OSPS-BR-01.01", "WARN", NO_WORKFLOWS_WARN), + ("openssf-baseline::empty_repo", "OSPS-BR-01.01", None, NO_WORKFLOWS_WARN), + ("openssf-baseline::empty_repo", "OSPS-AC-04.01", "PASS", NO_WORKFLOWS_WARN), + ("reproducibility::empty_repo", "OSPS-BR-01.01", "PASS", NO_WORKFLOWS_WARN), + ("openssf-baseline::empty_repo", "OSPS-BR-01.01", "PASS", {**NO_WORKFLOWS_WARN, "status": "FAIL"}), + ("openssf-baseline::empty_repo", "OSPS-BR-01.01", "PASS", {"status": "WARN", "details": "Could not verify"}), + ("openssf-baseline::empty_repo", "OSPS-BR-01.01", "PASS", None), + ], +) +def test_other_drift_is_not_tolerated(pair: str, control_id: str, before: str | None, after) -> None: + assert not _warn_by_design(pair, control_id, before, after) + + +@pytest.mark.parametrize( + "path", [".github/workflows/ci.yml", ".github/workflows/ci.YAML", "action.yml", "a/action.yaml"] +) +def test_no_drift_is_tolerated_on_a_fixture_with_inputs(path: str, tmp_path, monkeypatch) -> None: + import sys + + monkeypatch.setattr(sys.modules[__name__], "PARITY_FIXTURES", tmp_path) + (tmp_path / "repo" / path).parent.mkdir(parents=True) + (tmp_path / "repo" / path).write_text("on: push\n", encoding="utf-8") + + assert not _warn_by_design("openssf-baseline::repo", "OSPS-BR-01.01", "PASS", NO_WORKFLOWS_WARN) + + @pytest.mark.integration def test_warn_outcome_changes_no_existing_control() -> None: """SC-008 / FR-017. @@ -53,7 +116,11 @@ def test_warn_outcome_changes_no_existing_control() -> None: another handler started returning WARN or the disposition refactor changed how PASS/FAIL/INCONCLUSIVE are routed. """ - with_warn = capture() + with_details = capture_details() + with_warn = { + pair: {control_id: result["status"] for control_id, result in controls.items()} + for pair, controls in with_details.items() + } without_warn = capture(neutralize=True) assert set(with_warn) == set(without_warn), "corpus pairs differ between runs" @@ -65,7 +132,7 @@ def test_warn_outcome_changes_no_existing_control() -> None: for control_id in set(with_warn[pair]) | set(without_warn[pair]) if with_warn[pair].get(control_id) != without_warn[pair].get(control_id) and not _warn_by_design( - pair, control_id, without_warn[pair].get(control_id), with_warn[pair].get(control_id) + pair, control_id, without_warn[pair].get(control_id), with_details[pair].get(control_id) ) } if changed: From d268738d6125f96475b8dfc6ce5b750c0767c25c Mon Sep 17 00:00:00 2001 From: Johnson Lyu Date: Sat, 31 Oct 2026 18:17:47 -0400 Subject: [PATCH 3/3] docs: clarify concluding WARN semantics for #584 Assisted-by: Cursor:Claude-Opus-5.5 Signed-off-by: Johnson Lyu --- CHANGELOG.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dcc129cf1..15e24653d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -656,17 +656,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `WARN [unexpected_exit]` from zizmor exiting 3 or `grep` exiting 2, and zizmor and `grep` are not run, so their `unexpected_exit` log lines are gone. A new `darnit-baseline` step type, `github_workflows_present` - (ceiling `{fail}`), is the first step of the three controls and concludes - that WARN; it never reports PASS or FAIL, and with workflow files present - it is evidence only (later steps' indexes move up by one). The WARN is now - concluded by that step (`concluded_by = "github_workflows_present"`, - authority `dispositive`) rather than left undetermined, and it replaces an - ERROR when zizmor or `grep` is not installed. For OSPS-BR-01.01 an + (ceiling `{fail}`), is the first step of the three controls and the first + in-tree step type to conclude WARN under a `{fail}` ceiling. Because that + WARN is conclusive, the manual step no longer runs when there is nothing + to audit; the descriptive WARN is the operator guidance. The result stays + non-compliant and is not PASS, FAIL, or N/A. It is concluded by that step + (`concluded_by = "github_workflows_present"`, authority `dispositive`) + rather than left undetermined, and it replaces an ERROR when zizmor or + `grep` is not installed. With workflow files present the step is evidence + only (later steps' indexes move up by one). For OSPS-BR-01.01 an `action.yml` or `action.yaml` anywhere in the repository also counts as something to audit, since zizmor's `template-injection` audit reads action definitions too, so zizmor still runs and reports findings in a standalone - action. A confirmed non-GitHub `ci_provider` still makes the controls N/A - (#584). + action. A confirmed non-GitHub `ci_provider` still makes the controls N/A. + The SC-008 / FR-017 regression guard is narrowed only for these three + controls, and only for fixtures with no relevant auditable inputs (#584). - Configure `fail_exit_codes = [0]` for OpenSSF Baseline OSPS-BR-01.02 (`grep -rln github.head_ref .github/workflows/`) so exit 0 on a match records as non-conclusive evidence without an `unexpected_exit` error class (fixes #647).