Skip to content

feat: GSM Secrets - Backfill API to move plain-text secrets into the store - #477

Open
yuechao-qin wants to merge 1 commit into
gsm-t2-read-at-launchfrom
gsm-t3-backfill
Open

yuechao-qin wants to merge 1 commit into
gsm-t2-read-at-launchfrom
gsm-t3-backfill

Conversation

@yuechao-qin

@yuechao-qin yuechao-qin commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #475; merge after it.

What

An admin route moves existing plain-text secrets into the configured store, a batch at a time:

POST /api/admin/secrets/backfill_storage?limit=2      # three plain rows
→ {"moved_to_store": 2, "already_stored": 0, "still_plaintext": 1}

POST /api/admin/secrets/backfill_storage?user_id=alice&secret_name=TOKEN   # just that one secret

user_id and secret_name are optional filters: one secret, one owner's secrets, or one name across owners. The counts always cover the whole table, so still_plaintext shows how far the whole move has got.

Each call takes up to limit rows (1–1000, default 50) that have no storage entry. For each one, under a row lock, it checks the row is still plain, saves the value through the store's save, and commits that row before touching the next. Call it again until still_plaintext is 0.

Case What happens
plain row value saved in the store; the row keeps what save returned, plus its storage entry
row already in a store, of any type not touched; counted in already_stored
re-run moves only what is still plain, so it is safe to repeat
save fails part-way rows already moved stay committed; the failing row and the rest stay plain; the error is returned
no store configured 422, nothing changes

Why

#474 saves new and updated values through the store, and #475 reads them back at launch; rows written before a store was configured still hold plain text. This moves them. The value lands in the store outside the database, so each row commits on its own: a failure part-way never rolls back a row whose value is already saved. A crash between save and the commit leaves one extra stored value and the row still plain, so the next call moves it again.

Verification

  • tests/test_secret_value_store.py: moves plain rows; a re-run moves nothing new; limit is honoured across calls; a failing save keeps the earlier rows moved and the rest plain; rows in another store are left alone; no store is refused; the route answers admins and refuses other users (403); user_id, secret_name and both together move only the matching rows, through the service and the route.
  • Mutation-checked: committing once at the end, dropping the plain-row filter, dropping the no-store check, ignoring limit, dropping either filter, and the route not passing them each turn tests red.
  • Full suite: 673 passed, 22 failed — the same 22 in tests/instrumentation/test_execution_tracing.py fail on master.

@yuechao-qin
yuechao-qin requested a review from Ark-kun as a code owner November 18, 2026 05:37
@yuechao-qin
yuechao-qin added this pull request to stack #476 November 18, 2026 05:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant