[i-am-paradox]

Summary

Fixes #568 and fixes #589.

Framework controls frequently declare maturity levels, domains, and security severity within their tags dictionary (for example tags = { level = 1, domain = "AC" }), leaving the optional top-level attributes as None. In openssf-baseline.toml, 65 of 66 controls declared tags.level, while STAGE1-REF-SECURITY-01 set top-level level = 1. This mixed int and None keys in the level grouping dictionaries, causing sorted(by_level.items()) in darnit validate and sorted(by_level.keys()) in darnit plan to crash with TypeError: '<' not supported between instances of 'int' and 'NoneType'.

Changes:

1. In ControlConfig (packages/darnit/src/darnit/config/framework_schema.py), added an after-model validator synchronizing level, domain, and security_severity between the top-level attributes and tags. If top-level fields are omitted but declared in tags, the attributes are populated, giving maturity levels and domains a single source of truth across filtering, planning, validation, and compliance calculations.
2. In EffectiveControl (packages/darnit/src/darnit/config/merger.py), added fallback to tags for level, domain, and security_severity when building effective controls.
3. In cmd_plan and cmd_validate (packages/darnit/src/darnit/cli.py), sorted level breakdowns using key=lambda k: (k is None, k if k is not None else 0) so that controls without a maturity level sort last (No level / none=...) rather than raising a TypeError.
4. Documented the level and domain synchronization requirement in docs/architecture/framework-design.md section 2.3.
5. Added unit tests in tests/darnit/test_cli.py ensuring validate and plan run successfully on all shipped framework TOML files, and verifying sorting behavior on frameworks with mixed and unleveled controls. Added tests in tests/darnit/config/test_merger.py verifying tag synchronization and get_controls_by_level.

Type of Change

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update
• Refactoring (no functional changes)

Framework Changes Checklist

If this PR modifies the darnit framework (packages/darnit/):

• Updated framework spec (docs/architecture/framework-design.md) if behavior changed
• Ran uv run python scripts/validate_sync.py --verbose and it passes

Control/TOML Changes Checklist

If this PR modifies controls or TOML configuration:

• Control metadata defined in TOML (not Python code)
• SARIF fields (description, severity, help_url) included where appropriate
• Ran validation to confirm TOML schema compliance

Testing

• Tests pass locally (uv run pytest tests/ -v)
• Added tests for new functionality (if applicable)
• Linting passes (uv run ruff check .)

Ran uv run pytest tests/darnit/test_cli.py tests/darnit/config/test_merger.py -v (all passed). Ran darnit validate on all 8 shipped framework TOMLs in the repo (all passed). Ran scripts/validate_sync.py --verbose and ruff check . (all clean).

AI assistance

• No AI assistance was used
• AI assistance was used

Assisted by Antigravity using model eni-gemini-2.5-pro. Developed the validator logic, CLI sorting keys, and test cases; verified all execution and validations locally.

Additional Notes

[mlieberman85]

Thanks, this fixes both crashes, and having the validator feed get_controls_by_level and composition include_levels is a nice side effect. A few things before it's one source of truth:

1. Conflicts pass silently: level = 1 with tags = { level = 2 } loads with both values, and ControlSpec later keeps the top-level one. Could the validator reject a conflict (and the same for domain/security_severity)?
2. A non-integer tags.level ("L1", 1.5) becomes None with no error, and true becomes level True and passes the 1–3 check. Could those fail loading instead, so a typo can't drop a control out of level filters?
3. The spec text says "synchronize" but not which field is canonical. Could it name one (the issue suggested either) and say a conflict is an error?
4. With the validator in place, the fallbacks in merger.py and the two in control_loader.py are redundant. Could they go, so the rule lives in one place?
5. Please add a CHANGELOG entry (it also changes what composition include_levels selects for openssf-baseline).

Note for the spec line: plan now shows "No level", but audit/harness compliance count a level-less result as L1 (r.get("level") or 1); worth one sentence saying which is intended.

Drafted with Claude Code; reviewed and posted by me.

[i-am-paradox]

Thanks for the review. I have addressed all points in commit 4d4734b:

1. Reject conflicts: ControlConfig._sync_level_and_domain_with_tags now validates matching values when both the top-level attribute and tags declare level, domain, or security_severity, raising a ValueError on any conflict.
2. Strict tags.level typing: Enforced that tags["level"] must be an integer, explicitly rejecting boolean True/False as well as non-integer types like "L1" or 1.5. Added corresponding validation for tags.domain (must be string) and tags.security_severity (must be float or int, not bool).
3. Spec clarification: Updated docs/architecture/framework-design.md section 2.3 to state that top-level attributes are canonical, declaring conflicting values between top-level attributes and tags errors on load, non-integer tags.level errors on load, and noted the difference between display grouping (plan displays "No level") and compliance counting (audit and harness count an unleveled result as L1).
4. Removed redundant fallbacks: Removed the fallback ternaries in merger.py (merge_control) and the fallback checks in control_loader.py (load_control_from_effective and load_control_from_framework) so the schema validator is the single source of truth.
5. CHANGELOG entry: Added an entry under ### Fixed in CHANGELOG.md covering the sorting crash fix, canonical attribute synchronization, conflict rejection, and strict type validation.
6. Unit tests: Added unit tests in tests/darnit/config/test_merger.py asserting that conflicting declarations and invalid tag types raise ValidationError.

[mlieberman85]

Thanks, all five points from the last round are in. One gap: composite overrides go through model_copy in _apply_override (core/composition.py:500), so the new validator never runs on them. With a base control at level 1 and [overrides.X] tags = { level = 3 }, the control loads at level 1: merge_control writes the top-level value back over the tag. On main the override took effect. tags = { level = "L3" } also loads without an error. Could _apply_override revalidate the result (model_validate)? Also, please mention in the CHANGELOG that composition include_levels and get_controls_by_level now select tag-leveled controls, and rebase (the CHANGELOG conflict with #609 only).

Drafted with Claude Code; reviewed and posted by me.

[i-am-paradox]

Addressed review feedback in ff8f124:

1. Revalidate composite overrides: In _apply_override (core/composition.py), the updated model now runs through ControlConfig.model_validate(ctrl.model_copy(update=update)).
2. Synchronize tag overrides with top-level attributes: When an override sets tags = { level = 3 }, _apply_override synchronizes level into the update so the control's maturity level is updated rather than overwritten by the base control's previous level in merge_control. Overriding tags = { level = "L3" } now raises a ValidationError.
3. Rebase & CHANGELOG: Rebased cleanly onto latest main (resolving the CHANGELOG conflict with fix(sieve): one when evaluator for controls and steps #609) and noted that composition include_levels and get_controls_by_level now select tag-leveled controls.
4. Unit test: Added test_overrides_sync_tags_level_and_validate in tests/darnit/test_composition.py.

[mlieberman85]

Thanks, the points from both rounds are in: conflicting and non-integer levels are rejected, the spec names the canonical field, the fallbacks are gone, and overrides revalidate (an override of tags = { level = 3 } now loads at level 3; main loaded it at 1). On jonschlinkert/is-number, audits with openssf-baseline, gittuf, reproducibility and testchecks give the same level and status for every control on main and this branch, and darnit validate and darnit plan now exit 0 on the baseline instead of raising the TypeError.

It conflicts with main again, in CHANGELOG.md only. Once that's rebased I'll approve.

Two small things:

• include_tags = { level = N } now also selects controls that set a top-level level, because the validator writes tags.level (framework_schema.py:718-719). On the mock-source-a fixture it selects 2 controls where main selects 0. That looks right, but it's worth a line in the CHANGELOG entry.
• The formatting changes in merger.py, control_loader.py:182,264, framework_schema.py:417 and test_merger.py are unrelated to the fix. The project isn't formatted with ruff format, so please drop them.

Separate from this PR: ControlConfig.level has no range check (0, 7 and -1 all load, and a level above 3 is in no level verdict). That was already true on main, so it's for its own issue.

---

Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.

[i-am-paradox]

Rebased cleanly onto latest main (resolving the CHANGELOG.md conflict) and addressed both review notes in 566a668:

1. CHANGELOG entry: Noted in the CHANGELOG.md entry that include_tags = { level = N } now also selects controls that declare a top-level level.
2. Formatting cleanup: Dropped the unrelated formatting changes in merger.py, control_loader.py, framework_schema.py, and test_merger.py, leaving only the necessary functional changes and new tests.

All 93 tests in test_merger.py, test_cli.py, and test_composition.py pass cleanly.

[mlieberman85]

Approving 0ccef6c. The points from the earlier rounds are in: the formatting churn is gone (merger.py is untouched, control_loader.py only loses the fallbacks), the CHANGELOG has the include_tags = { level = N } sentence, and a bad or conflicting level, domain or severity in an override is now rejected.

command	main (075556e)	this PR (0ccef6c)
darnit validate openssf-baseline.toml	exit 1, TypeError: '<' not supported between instances of 'int' and 'NoneType'	exit 0, L1=26, L2=19, L3=21
darnit plan -f openssf-baseline <repo>	exit 1, same TypeError	exit 0
one control, level = 1, tags.level = 2	exit 0, loads at L1	exit 1, Conflicting level declarations
one control, tags.level = "L1" / true	exit 0, loads with no level	exit 1, tags.level must be an integer

On main, the last two rows also hit the TypeError when the file has a sibling control with a different kind of level. For example, the conflict plus a control with no level, or the string plus a control with a top-level level.

On 075556e merged with this PR, keeping both sides of the CHANGELOG conflict: every shipped framework TOML validates, the full suite gives 5141 passed, ruff and validate_sync pass, and a real-repo audit (wtf-frontend) gives the same per-control statuses as 075556e.

One thing before this can merge: it now conflicts with main in CHANGELOG.md, since #623 landed. Please rebase. The push will dismiss this approval, so I'll re-approve on the new head.

Not for this PR: the missing range check from my last review now has one more route. [overrides.X] tags = { level = 4 } on a control with a top-level level loads at that control's level on main, so its FAIL is reported. Here the control is dropped from the default audit with exit 0, though darnit validate flags it. A plain tags.level = 4 already behaves that way on main, and no shipped framework does this, so it belongs in the same separate issue.

Nits, non-blocking:

• The validator also writes domain and security_severity into self.tags, so include_tags on those now selects controls that only set the top-level field. No shipped framework changes, but the CHANGELOG only names level.
• An invalid override fails with 1 validation error for ControlConfig, without the control id the non-composite path prints.
• The PR description still lists a fallback in merger.py (item 2), which is no longer in the diff.

---

Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.

[i-am-paradox]

Rebased cleanly onto latest main (075556e) with CHANGELOG.md conflict resolved (keeping both entries). All CI checks are passing green.

[mlieberman85]

Re-approving on 89f4b18. The rebase changed only the CHANGELOG entry; the rest of the diff is identical to 0ccef6c, which I approved.

On 075556e merged with this branch and #576: full suite 5144 passed, 18 skipped; ruff and validate_sync pass. darnit plan and darnit validate on openssf-baseline.toml, which crash on main with the TypeError from #568, now work, and a full audit of jonschlinkert/is-odd reports the same status as main for all 66 controls.

---

Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.