Skip to content

feat(models): sync Enkrypt safety scores daily - #5622

Merged
intentionally-left-nil merged 5 commits into
mainfrom
feat/enkrypt-model-scores
Sep 4, 2026
Merged

feat(models): sync Enkrypt safety scores daily#5622
intentionally-left-nil merged 5 commits into
mainfrom
feat/enkrypt-model-scores

Conversation

@emilieschario

@emilieschario emilieschario commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Add daily, change-aware Enkrypt ingestion as optional model metadata—not runtime safety enforcement or a UI change.

  • Match only reviewed model identities and write only changed scores in one batch.
  • Use one shared statistics cache, with eligibility and freshness checked at final publication.
  • Keep health checks read-only; an external monitor owns notifications and recovery.
  • Use one consolidated migration and keep ingestion/publication independently disabled by default.

Verification

  • 1,272 focused tests passed, including database integration and review-regression coverage.
  • Migration consistency checks, lint, and typechecks passed.
  • Authenticated local checks confirmed publication and repeated unchanged runs without score-row rewrites.
  • Re-review found no remaining actionable P1/P2 issues in the implementation.

The broader schema suite has a pre-existing missing-fixture failure; validation above refers to the targeted checks. No credentials, live score values, or raw provider responses are included.

Visual Changes

None.

Follow-up actions

  • Merge current main and regenerate the single Enkrypt migration as 0235_enkrypt_sync_state.
  • Review CI results for the merged revision.
  • Provision isolated staging configuration through the shared environment workflow; migrate and bootstrap the catalog.
  • Run authenticated staging syncs and verify unchanged-row preservation, freshness, failure behavior, and both disable controls.
  • Configure one external monitor for the authenticated read-only health endpoint; test alert delivery, recovery, and scheduler-wide failure detection.
  • Confirm redistribution, attribution, and retention requirements before enabling publication.
  • Deploy the approved revision with one cron owner; enable ingestion and publication separately after their gates are satisfied.

Ready for code review. Production deployment and enablement have not been performed.

@emilieschario emilieschario changed the title feat(models): add external safety benchmark metadata feat(models): sync Enkrypt safety scores daily Sep 1, 2026
@emilieschario
emilieschario marked this pull request as ready for review September 1, 2026 21:35
@emilieschario emilieschario self-assigned this Sep 1, 2026
@kilo-code-bot

kilo-code-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

Re-reviewed the rebased Enkrypt daily ingestion stack (migration 0237, identity matching, publication gating, shared stats cache, health probe, cron/API routes, and the post-rebase build fix) at 9e0e3d67; no high-confidence defects on changed lines.

Files Reviewed (45 files)
  • ENVIRONMENT.md
  • apps/web/src/app/api/cron/check-enkrypt-health/route.test.ts
  • apps/web/src/app/api/cron/check-enkrypt-health/route.ts
  • apps/web/src/app/api/cron/sync-enkrypt/route.test.ts
  • apps/web/src/app/api/cron/sync-enkrypt/route.ts
  • apps/web/src/app/api/cron/sync-model-stats/route.test.ts
  • apps/web/src/app/api/cron/sync-model-stats/route.ts
  • apps/web/src/app/api/models/stats/[slug]/route.test.ts
  • apps/web/src/app/api/models/stats/[slug]/route.ts
  • apps/web/src/app/api/models/stats/route.test.ts
  • apps/web/src/app/api/models/stats/route.ts
  • apps/web/src/app/api/openrouter/models/route.ts
  • apps/web/src/lib/ai-gateway/providers/openrouter/index.ts
  • apps/web/src/lib/config.server.ts
  • apps/web/src/lib/model-stats/enkrypt-errors.ts
  • apps/web/src/lib/model-stats/enkrypt-fingerprint.test.ts
  • apps/web/src/lib/model-stats/enkrypt-fingerprint.ts
  • apps/web/src/lib/model-stats/enkrypt-identity.test.ts
  • apps/web/src/lib/model-stats/enkrypt-identity.ts
  • apps/web/src/lib/model-stats/enkrypt-publication.test.ts
  • apps/web/src/lib/model-stats/enkrypt-publication.ts
  • apps/web/src/lib/model-stats/enkrypt-status.test.ts
  • apps/web/src/lib/model-stats/enkrypt-status.ts
  • apps/web/src/lib/model-stats/enkrypt.test.ts
  • apps/web/src/lib/model-stats/enkrypt.ts
  • apps/web/src/lib/model-stats/model-stats-cache.test.ts
  • apps/web/src/lib/model-stats/model-stats-cache.ts
  • apps/web/src/lib/model-stats/sync-enkrypt.test.ts
  • apps/web/src/lib/model-stats/sync-enkrypt.ts
  • apps/web/src/lib/model-stats/sync-openrouter.test.ts
  • apps/web/src/lib/model-stats/sync-openrouter.ts
  • apps/web/src/lib/organizations/organization-models.test.ts
  • apps/web/src/lib/organizations/organization-models.ts
  • apps/web/src/lib/organizations/organization-types.ts
  • apps/web/src/routers/admin-router.ts
  • apps/web/src/scripts/enkrypt-coverage.ts
  • apps/web/src/tests/fixtures/enkrypt-scores.ts
  • apps/web/src/tests/openrouter-models.test.ts
  • apps/web/vercel.json
  • docs/enkrypt-sync-operations.md
  • packages/db/src/migrations/0237_enkrypt_sync_state.sql
  • packages/db/src/migrations/meta/0237_snapshot.json
  • packages/db/src/migrations/meta/_journal.json
  • packages/db/src/schema-types.ts
  • packages/db/src/schema.ts
Previous Review Summary (commit e7d17a4)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit e7d17a4)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Reviewed Enkrypt daily ingestion, identity matching, publication gating, shared stats cache, health probe, cron/API routes, and schema/migration changes; no high-confidence defects on changed lines.

Files Reviewed (45 files)
  • ENVIRONMENT.md
  • apps/web/src/app/api/cron/check-enkrypt-health/route.test.ts
  • apps/web/src/app/api/cron/check-enkrypt-health/route.ts
  • apps/web/src/app/api/cron/sync-enkrypt/route.test.ts
  • apps/web/src/app/api/cron/sync-enkrypt/route.ts
  • apps/web/src/app/api/cron/sync-model-stats/route.test.ts
  • apps/web/src/app/api/cron/sync-model-stats/route.ts
  • apps/web/src/app/api/models/stats/[slug]/route.test.ts
  • apps/web/src/app/api/models/stats/[slug]/route.ts
  • apps/web/src/app/api/models/stats/route.test.ts
  • apps/web/src/app/api/models/stats/route.ts
  • apps/web/src/app/api/openrouter/models/route.ts
  • apps/web/src/lib/ai-gateway/providers/openrouter/index.ts
  • apps/web/src/lib/config.server.ts
  • apps/web/src/lib/model-stats/enkrypt-errors.ts
  • apps/web/src/lib/model-stats/enkrypt-fingerprint.test.ts
  • apps/web/src/lib/model-stats/enkrypt-fingerprint.ts
  • apps/web/src/lib/model-stats/enkrypt-identity.test.ts
  • apps/web/src/lib/model-stats/enkrypt-identity.ts
  • apps/web/src/lib/model-stats/enkrypt-publication.test.ts
  • apps/web/src/lib/model-stats/enkrypt-publication.ts
  • apps/web/src/lib/model-stats/enkrypt-status.test.ts
  • apps/web/src/lib/model-stats/enkrypt-status.ts
  • apps/web/src/lib/model-stats/enkrypt.test.ts
  • apps/web/src/lib/model-stats/enkrypt.ts
  • apps/web/src/lib/model-stats/model-stats-cache.test.ts
  • apps/web/src/lib/model-stats/model-stats-cache.ts
  • apps/web/src/lib/model-stats/sync-enkrypt.test.ts
  • apps/web/src/lib/model-stats/sync-enkrypt.ts
  • apps/web/src/lib/model-stats/sync-openrouter.test.ts
  • apps/web/src/lib/model-stats/sync-openrouter.ts
  • apps/web/src/lib/organizations/organization-models.test.ts
  • apps/web/src/lib/organizations/organization-models.ts
  • apps/web/src/lib/organizations/organization-types.ts
  • apps/web/src/routers/admin-router.ts
  • apps/web/src/scripts/enkrypt-coverage.ts
  • apps/web/src/tests/fixtures/enkrypt-scores.ts
  • apps/web/src/tests/openrouter-models.test.ts
  • apps/web/vercel.json
  • docs/enkrypt-sync-operations.md
  • packages/db/src/migrations/0221_enkrypt_sync_state.sql
  • packages/db/src/migrations/meta/0221_snapshot.json
  • packages/db/src/migrations/meta/_journal.json
  • packages/db/src/schema-types.ts
  • packages/db/src/schema.ts

Reviewed by grok-4.6 · Input: 132.5K · Output: 18K · Cached: 738.4K

Review guidance: REVIEW.md from base branch main

@intentionally-left-nil

Copy link
Copy Markdown
Contributor

Fixed the rebase conflicts on feat/enkrypt-model-scores (rebasing onto updated main at a999002):

  1. Migration numbering collision: The branch's enkrypt_sync_state table was originally split across two branch-local migrations numbered 0221 and 0222, which collided with unrelated migrations main had since added at those same numbers (0221_melodic_hairball, 0222_military_war_machine, up through 0236). Per the git-rebase/database-migrations skills, I never hand-edited the generated snapshot/journal JSON — instead I took main's metadata at each conflict and removed the branch's colliding migration artifacts, then after the rebase completed, ran pnpm drizzle generate to regenerate the table's DDL fresh as 0237_abnormal_echo.sql (plus its snapshot/journal entry).
  2. Real content conflicts (not generated, resolved by hand):
  • ENVIRONMENT.md — merged the FAKE_LLM_URL doc update from main with the new ENKRYPT_API_KEY entry from the branch.
  • apps/web/src/app/api/openrouter/models/route.ts — composed main's appendLocalFakeDeterministicCatalogModels with the branch's modelResponse/publishEnkryptModels wrapper across all three response paths.
  • apps/web/src/lib/organizations/organization-models.ts — same composition (publishEnkryptModels(appendLocalFakeDeterministicCatalogModels(...), snapshot)).
  1. Ran pnpm format:changed (wrapped one long line) and amended the final commit with the regenerated 0237 migration.
    Verification: started the local Postgres test DB (pnpm test:db, applies all migrations including 0237 cleanly), then ran packages/db/src/schema.test.ts — 153/154 tests pass; the one failure (migration 0204 against duplicates) is pre-existing on main (unrelated file-naming mismatch, confirmed by checking main directly), not caused by this rebase. No leftover conflict markers or whitespace issues (git diff --check clean).

@intentionally-left-nil
intentionally-left-nil merged commit 714ccfa into main Sep 4, 2026
54 checks passed
@intentionally-left-nil
intentionally-left-nil deleted the feat/enkrypt-model-scores branch September 4, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants