[anth-volk]

Fixes #3914

Also fixes #3909: the country validator's print("Validating country") is removed.

Summary

Phase 1 step 1.8, group 6: security, logging and debugging leftovers. The bottom of its stack since #3913 merged.

• Rules: select gains S, T20, LOG, G, DTZ, ERA, TD, FIX and EXE. Every finding at the base (262, plus 12 in Stage 12A: Sign-in identity and v1-anchored user associations #3891's code after the restack) is fixed, kept with an inline reason, or, in tests, CI/CD scripts and command-line scripts, covered by a file-level exception. One commit per family.
• File-level exceptions (new):
  • Where they're allowed. A ruff per-file-ignores entry is now allowed only for:
    • tests (a tests or authenticated_tests directory);
    • CI/CD scripts (.github);
    • command-line scripts (a scripts directory, or a module with a top-level if __name__ == "__main__":).
  • How it's checked. The rules guard expands each entry's pattern the way ruff does: * and ? also match /, and a pattern also matches a file by its name. It fails when the pattern covers any other file, is negated, or matches nothing. scripts/rules/file_exceptions.py holds the definitions, docs/engineering/skills/rules-registry.md#file-level-exceptions documents them, and six rules-tool tests prove them.
  • The entries. Each lists exactly the group 6 codes those files had suppressed inline, not whole families, and each has a registered per-file-ignore rule:
    • tests (**/tests/**, extending api.quality.tests-ruff-exemptions): S105, S106, S603, S607, S608, DTZ001, DTZ011, T201;
    • CI/CD scripts (.github/**): S310, T201;
    • command-line scripts (scripts/**): T201, S603, S607, S310, S311, DTZ011;
    • the five modules that also run as programs (catalog initialization, the three migration qualifications, worker_spm_release): T201.
  • What it removed. 120 inline suppressions: 74 in tests, 28 in scripts, 10 in the program modules and 8 in .github.
  • Alembic revisions. Alembic revisions aren't any of those kinds of file, so their "migrations/*/versions/*.py" = ["ALL"] entry moves to [tool.ruff.lint] exclude. ruff check . still skips them, and ruff format still checks them. api.quality.alembic-revisions-unused-imports now describes the exclusion.
• Behaviour changes (three kinds, all deliberate):
  • Debug prints removed (T201): the country validator (validate_country prints "Validating country" on every request #3909), plus the create, get and update prints in the report output and simulation routes. The two update prints become logger.debug calls on a module logger, because their ids come from the request body.
  • Logger name (LOG015): the household calculation routes log their three errors on a module logger instead of the root logger. The message text is unchanged, and production records still reach the root observability handler.
  • Nothing else: error responses, message text and published schemas are unchanged.
• Asserts outside tests (S101, 11): each type-narrowing assert in the v2 household, policy, user-household and user-policy services, and two in the rules tool, becomes an explicit check that raises TypeError. None runs inside a Pydantic validator, and nothing catches AssertionError. Every check was already guaranteed by an earlier one, so none is expected to fire; the v2 routes and the mirror paths treat either type the same way. No production command runs Python with -O, so these asserts were live.
• Prints in commands (T201): command-line scripts, CI/CD scripts and the modules that run as programs (including the migration qualification commands' JSON evidence) keep their output under the file-level exceptions. capture_migration_baseline.py, smoke_test_local_boot.py and the rules tool's CLI route output through one _report helper. Application code keeps the api.py startup timings and the gateway-auth warning on stderr, each with an inline reason.
• Datetimes (DTZ, 21): no value switches between naive and aware. The catalog extraction builds its aware UTC value directly; the other 20 stay naive. Those in tests and scripts, such as the rules tool's calendar-day review dates and test values, are covered by the file-level exceptions. The 4 in application code (naive database columns, the local calendar year) keep an inline reason.
• Security (S): no vulnerability found.
  • Fixed: hashlib.sha1(..., usedforsecurity=False) (same digest).
  • Covered by file-level exceptions:
    • in tests: fake secrets, runs of this repository's scripts and git, and table names from a fixed tuple;
    • in scripts and .github: fixed https and loopback URLs, and a seeded load-test mix.
  • Kept inline with a reason:
    • environment-variable names, including the Auth0 and gateway client-secret names Stage 12A: Sign-in identity and v1-anchored user associations #3891 added;
    • the WWW-Authenticate challenge.
  • S110: the five try/except Exception: pass blocks group 5 left (two in the ASGI factory, and the user routes' failure logging, the GCP log handler and request-context binding) are now with contextlib.suppress(Exception): blocks with their reason in a comment above, like the eighteen Enable ruff exception rules (TRY, EM, BLE) #3913 converted. They need neither BLE001 nor S110, and behaviour is unchanged.
• Leftovers: 32 lines of commented-out code are deleted (29 in tests/to_refactor/python/test_policy.py). The two TODOs become plain notes; the user-profile one keeps its link to Utilize a technology like Marshmallow to validate inbound requests #2054 and says it was closed. Seven Python scripts with a shebang and a __main__ entry point are now executable.
• Also in this PR:
  • The GPT-4 dashboard experiment is removed. dashboard/experiments/gpt4_api_user.py and its requirements file were a Streamlit experiment nothing imports or runs. Removing it also removes its eval and its three S113 suppressions.
  • A doc-only rule, api.process.agents-suppress-as-last-resort: agents add an inline ruff suppression only when neither a code change nor a ruff setting removes the finding, and a PR lists every inline suppression it adds, with the reason. docs/engineering/skills/rules-registry.md#inline-suppressions gives the order to try, and the GitHub PRs skill cites the rule.
• New inline suppressions (12), as that rule asks:
  • S105 (6): environment-variable names that contain SECRET, in data/v2/settings.py, libs/auth0_management.py (2) and libs/gateway_auth.py (2), and the WWW-Authenticate challenge in fastapi_routes/v2/errors.py. Ruff's hard-coded-password check matches on the names; none is a secret.
  • T201 (2): the startup timings in api.py and the unauthenticated-client warning in libs/simulation_entrypoint.py, both written straight to stderr. Routing them through logging would change their output.
  • DTZ005 (2), DTZ011 (1), DTZ901 (1): the local calendar year in constants.py and services/household_calculation_service.py, and naive start_time values in services/economy_service.py and runtime_cache/reform_impacts.py that must match a naive database column. Making them aware would change stored values.
  • It also removes five BLE001 suppressions from master (the contextlib.suppress blocks above).
• Registry:
  • api.quality.ruff-lint now lists the nine families.
  • Four per-file-ignore rules describe the file-level exceptions.
  • Rules tool. The new check is ported by hand into Enable ruff security, logging and debugging rules (S, T20, LOG, G, DTZ, ERA, TD, FIX, EXE) policyengine-sim-api#785. Each repository owns its copy of the rules tool.

Testing

At this branch's tip (3f7deea, on master at 43e6c7f):

• ruff check and ruff format --check (0.15.12) are clean repo-wide, and adding RUF100 finds no unused suppression other than six E402 ones in a test file that master also has.
• The quality guards pass: migration contracts, rules registry and CI scripts.
• The 16 unit test files that cover the four rewritten modules: 351 passed, on Python 3.12 with the uv.lock dependencies.
• CI: this PR is now the bottom of its stack, so it runs the full PR workflow.

Before this update, on the earlier tip:

• tests/unit (2,685 passed) and the to_refactor, contract and dedupe set (97 passed).
• Every Pydantic model schema (230 models, 100 adapters), the FastAPI OpenAPI with every native router enabled, the /v2/openapi.json preview and the legacy spec were byte-identical to the base.
• mypy was clean (167 files).
• CI/CD: no workflow changed. Four .github scripts changed only in comments, output helpers or file mode, and the environment variables they read are unchanged.

🤖 Generated with Claude Code