[areej123-wq]

What does this PR do?

Adds an offline-first evaluation benchmark for the existing AI assistant: 24 synthetic scenarios, an isolated BM25 index, a versioned JSON baseline, and optional bounded live evaluation using the production ask prompt and provider abstraction.

Initial mean rule-source recall@5 is 0.6591 across 22 retrieval cases. Seven cases expect abstention. Offline mode measures retrieval only; live phrase/reference diagnostics require human semantic review.

Type of change

• AI evaluation tooling and regression tests
• Documentation

Implementation

• Validate fixtures, record dataset/index hashes, and add a conservative 0.60 retrieval regression gate.
• Add optional index paths to builder/retriever, preserving defaults and avoiding production index overwrites.
• Live mode requires explicit provider/model, an environment-only key, and a request cap; failures consume the cap, without retries. Errors are redacted.
• Document review rubric and limitations: no semantic correctness claim or live CI gate, unavailable token/cost metadata, and no guaranteed monetary cap.

Testing

• 100 tests passed: pytest tests/test_ai_evaluation.py tests/test_rag_pipeline.py tests/test_rag_dependencies.py tests/test_ai_prompt_guard.py -q
• Repository-wide ruff check . and ruff format --check . passed.
• Standard-library offline CLI generated the baseline without Azure/provider calls.
• Tested index isolation, invalid fixtures, request/prompt limits, production prompt fencing, provider-error redaction, and offline no-network behavior.
• Full GitHub CI pending.

Local validation used macOS/Python 3.13 with focused test dependencies. Linux/Python 3.11 locked dependencies and full PostgreSQL/security checks remain for CI.

Related issue

Closes #385. Provides the measured baseline for #386.

Checklist

• DCO Signed-off-by included.
• Synthetic data only; no credentials/customer data committed.
• feat/ branch convention, tests, and documentation.
• Maintainer review of expected facts and initial regression threshold.

Scanner-rule, playbook, and compliance-mapping checklist items are not applicable. Draft pending CI and dataset review. No paid live model run performed.

[areej123-wq]

Investigated the failed SCA (pip-audit) job.

Both vulnerable pins are pre-existing on the PR base:

Dependency	Base dev	PR head	CI advisory	Reported fix
multidict	6.8.0	6.8.0	CVE-2026-104874	6.9.1
werkzeug	3.1.8	3.1.8	CVE-2026-102598	3.1.9

Compared base ea5044f with PR head edb5045. Freshly fetched origin/dev still points at that base. All six dependency input/lock files have identical Git blob hashes across these commits: requirements.in, requirements.txt, requirements-dev.in, requirements-dev.txt, requirements-lock.in, requirements-lock.txt. Both packages are pinned in the runtime and development locks; this PR introduces no dependency changes. The audit workflow is also unchanged.

The CI log reports exactly these two vulnerabilities and exits 1. In the same run, Backend Tests, Rule & Compliance Validation (including lock consistency), lint, Bandit, Semgrep, and the container scan passed. CI Summary fails because the audit fails; other passing checks do not invalidate these findings.

Recommendation: fix these inherited pins in a focused dependency-security change on dev. Regenerate the runtime/development locks with reviewed targeted upgrades using the documented Linux/CPython 3.11 lock toolchain, then run hash-checked installation, pip check, pip-audit, and regression tests. This PR can then incorporate the patched base and rerun CI.

No dependency files, workflow settings, audit exceptions, or PR commits were changed during this investigation. No new local audit or compatibility run is claimed: this evidence comes from actual CI logs and identical committed dependency files. Keep the security check enforced; this explains provenance and does not request a bypass.

[Vishnu2707]

@areej123-wq - Thanks for your contribution, the CI seems to be failing, please address that when possible, post which this can be reviewed.

[TFT444]

Almost there, one fix needed. In runner.py, latency_seconds is only set in the try block, so a provider error leaves the key missing and callers get a KeyError on timing. Move it into a finally block and this is good to go.

[parthrohit22]

Reviewed edb5045. The design is careful. Offline mode never touches the network, the index is built in a temp directory, live mode has a request cap and redacts errors, and the README is clear that this isn’t a correctness score. Approving once it’s rebased.

About the requested latency_seconds change: in runner.py, that line runs after the try/except, so every attempted call records it, including failed ones. test_provider_errors_are_counted_and_redacted covers that path. I don’t think a change is needed.

What’s needed before merge:

Rebase onto dev to clear the conflict. pip-audit should go green once OWASP/openshield#401 is in.

Non-blocking:

The mean_recall_at_5 >= 0.60 test depends on the whole corpus. A new rule or doc could fail an unrelated PR. Consider documenting how to refresh the baseline.
Nothing checks that baseline.json’s index hash is still current, so the baseline can go stale silently.