Skip to content

fix(baseline): use declared fail_exit_codes = [0] for OSPS-BR-01.02 grep step (fixes #647) - #656

Merged
Marc-cn merged 1 commit into
darnitdevorg:mainfrom
i-am-paradox:fix/647-exec-grep-inconclusive-exit-code
Oct 29, 2026
Merged

Marc-cn merged 1 commit into
darnitdevorg:mainfrom
i-am-paradox:fix/647-exec-grep-inconclusive-exit-code

Conversation

@i-am-paradox

Copy link
Copy Markdown
Contributor

Summary

Resolves #647 where OpenSSF Baseline control OSPS-BR-01.02 reported a grep match as an environmental failure (error_class="unexpected_exit").

In packages/darnit/src/darnit/sieve/builtin_handlers.py, exec_handler treats exit codes outside pass_exit_codes and fail_exit_codes as undeclared, assigning an error class (defaulting to "unexpected_exit"). In OSPS-BR-01.02, grep -rln github.head_ref .github/workflows/ declares pass_exit_codes = [1]. When github.head_ref is present, grep exits 0. Because 0 was undeclared, it was classified as an unexpected exit rather than recording the found match as evidence.

Changes:

  • Added inconclusive_exit_codes: list[int] | None setting to exec_handler. When proc.returncode in inconclusive_exit_codes, returns HandlerResultStatus.INCONCLUSIVE with error_class=None and preserves stdout/stderr in evidence.
  • Registered "inconclusive_exit_codes" in register_builtin_handlers() settings schema.
  • Added inconclusive_exit_codes = [0] to OSPS-BR-01.02 in packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml.
  • Documented inconclusive_exit_codes and updated undeclared exit code rules in docs/architecture/framework-design.md.
  • Added unit tests in tests/darnit/sieve/test_builtin_handlers.py and integration tests for OSPS-BR-01.02 in tests/darnit_baseline/controls/test_secure_branch_names.py.
  • Added entry to CHANGELOG.md under ### Fixed.

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Framework Changes Checklist

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Control/TOML Changes Checklist

  • Control metadata defined in TOML (not Python code)
  • SARIF fields (description, severity, help_url) included where appropriate
  • Ran validation to confirm TOML schema compliance

Testing

  • Tests pass locally (uv run pytest tests/darnit/sieve/ tests/darnit_baseline/controls/ -v)
  • Added tests for new functionality (if applicable)
  • Linting passes (uv run ruff check .)

AI assistance

  • No AI assistance was used
  • AI assistance was used

Assisted-by: Antigravity:eni-gemini-2.5-pro
Used for diagnosing handler exit-code routing, implementing handler settings and test fixtures, and updating framework design documentation.

Additional Notes

Fixes #647.

@Marc-cn Marc-cn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clear write-up and the tests (91 pass here). I compared three runs on a repository whose workflow uses github.head_ref:

  • main: WARN, error_class: unexpected_exit (the bug in #647);
  • main with one line added to the step, fail_exit_codes = [0]: WARN, no error class, the match recorded as evidence;
  • this branch: WARN, no error class.

The step already has concludes = [], so a declared exit 0 is recorded as evidence and concludes nothing, which is what #647 asks for. That makes inconclusive_exit_codes a framework-wide setting plus a spec change for a control that upstream has retired. I'd rather keep the TOML fix and leave a new handler setting to a separate design discussion with @mlieberman85, if there is a second case that needs it.

#647 also asks that other exec grep steps be checked for the same mapping; a sweep of those with the same one-line fix would make this PR complete.

@i-am-paradox

Copy link
Copy Markdown
Contributor Author

Thanks @Marc-cn! Kept the TOML fix as recommended in 7a8bd3c:

  • Reverted the engine/spec inconclusive_exit_codes handler setting in builtin_handlers.py and framework-design.md.
  • Configured fail_exit_codes = [0] in openssf-baseline.toml for OSPS-BR-01.02. Because the step declares concludes = [], exit 0 records the match as evidence and concludes nothing, without triggering an unexpected_exit error class.
  • Checked other exec grep steps across baseline controls: the only other one is in OSPS-BR-01.01, which already declares pass_exit_codes = [0], fail_exit_codes = [1], and concludes = [] for ! grep.
  • Updated tests in test_secure_branch_names.py; all 3 tests pass cleanly.

Marc-cn
Marc-cn previously approved these changes Oct 26, 2026

@Marc-cn Marc-cn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the fix: one TOML line, and the match case is tested end to end through the orchestrator. One small thing, no need to block on it: the no-match test asserts the same WARN and no error class as the match test, so it does not show that exit 1 was taken as a pass (its docstring says PASS). Asserting pass_history[0].result.outcome == PassOutcome.PASS there would make the two tests distinguish the two cases

@Marc-cn

Marc-cn commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Approved, but it now conflicts with main (likely CHANGELOG). Could you rebase? Then it can merge.

…rep step (fixes darnitdevorg#647)

- Declare fail_exit_codes = [0] in openssf-baseline.toml for OSPS-BR-01.02; since concludes = [] is declared, exit 0 records matching evidence without concluding and prevents unexpected_exit error_class.
- Add integration tests in test_secure_branch_names.py verifying fail_exit_codes and distinguishing FAIL (exit 0 match) vs PASS (exit 1 clean).
- Document in CHANGELOG.md under Fixed.

Fixes darnitdevorg#647

Signed-off-by: Paradox <159635503+i-am-paradox@users.noreply.github.com>
@i-am-paradox
i-am-paradox force-pushed the fix/647-exec-grep-inconclusive-exit-code branch from 7a8bd3c to 98ca0bf Compare October 26, 2026 22:45
@i-am-paradox i-am-paradox changed the title fix(sieve): support declared inconclusive exit codes in exec handler (fixes #647) fix(baseline): use declared fail_exit_codes = [0] for OSPS-BR-01.02 grep step (fixes #647) Oct 26, 2026
@i-am-paradox

Copy link
Copy Markdown
Contributor Author

Rebased cleanly onto latest main (3822f28). The CHANGELOG conflict is resolved, and the branch is squashed into a single clean commit on top of main. The integration tests in test_secure_branch_names.py verify the TOML declaration and distinguish the FAIL (exit 0 match evidence) vs PASS (exit 1 clean) outcomes. All 1,127 baseline tests and linters pass cleanly.

@Marc-cn Marc-cn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the fix I asked for: fail_exit_codes = [0] makes a grep match a non-concluding FAIL with the file list as evidence, no unexpected_exit; the two integration tests cover match and no-match. One thing: the PR description still describes the earlier inconclusive_exit_codes design, which is gone from the diff. Please update it to match. Needs a rebase, then good to merge. Note #685 adds a step before this one in the same control.

@Marc-cn
Marc-cn merged commit e12e086 into darnitdevorg:main Oct 29, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OSPS-BR-01.02 reports a grep match as unexpected_exit

2 participants