Skip to content

Fix rejected OAuth refresh retries and expected-error telemetry - #225

Closed
jgpruitt wants to merge 5 commits into
mainfrom
fix-oauth-refresh-rejections
Closed

jgpruitt wants to merge 5 commits into
mainfrom
fix-oauth-refresh-rejections

Conversation

@jgpruitt

Copy link
Copy Markdown
Collaborator

Summary

  • Stop replaying definitively rejected refresh tokens across proactive/reactive calls and subsequent processes using a persisted token-specific rejection tombstone. Preserve newer peer-token recovery, refresh locking, transient retries, and injected credential overrides.
  • Surface a clear re-login-required auth error. A bounded hash-only local guard also prevents in-process replay when credential persistence fails; cross-process persistence remains best effort if storage is unwritable.
  • Add safe diagnostics to existing token-request spans: grant type, known first-party client ID or hashed unknown-client attribution, and allowlisted rejection reasons. Bound optional body-clone reads to 8 KiB/250 ms; preserve original request/response bodies and never export credentials, raw bodies, or arbitrary descriptions.
  • Classify narrowly selected Better Auth OAuth 400/401/403 rejections as info/UNSET without exception events, while preserving wire responses and genuine server errors.
  • Update authentication, observability, and troubleshooting documentation.

Telemetry boundaries

The span processor applies only to actual Better Auth APIError instances on its exact token route/handler scope with known OAuth client-error codes. Unknown errors, other routes/scopes, 5xx, database errors, and later unrelated failures retain their original error behavior. The adapter uses public live-span methods rather than rewriting already-exported spans and is covered against the pinned Better Auth implementation.

No extra spans, log events, or metric series are added. Token-endpoint query parameters are excluded from root URL attributes; successful token-response bodies are not parsed.

Validation

  • ./bun run check:full passed against local PostgreSQL, including all 33 CLI end-to-end tests.
  • Regression coverage includes persisted rejection in a separate process, real reactive transport behavior, newer peer/login recovery, transient failures, storage-write failures, metadata privacy/limits/body preservation, and real SDK/Better Auth error classification.
  • git diff --check passed.

Rollout

Not deployed. Server telemetry changes and CLI retry changes ship independently: clients need an updated CLI to gain terminal rejection handling. Production rollout and version tagging remain separate steps.

Copilot AI balanced review requested due to automatic review settings October 16, 2026 22:11
@jgpruitt jgpruitt self-assigned this Oct 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Credential-recovery gaps can replay rejected tokens or discard fresh logins, and tombstone writes can overwrite concurrent credential changes.

3 open issues
What changed in this PR

This PR prevents repeated use of rejected OAuth refresh tokens and adds privacy-bounded diagnostics for expected authentication failures.

Changes:

  • Persist token-specific rejection state and surface actionable re-login errors.
  • Classify expected OAuth errors without suppressing genuine server failures.
  • Add regression coverage and update authentication documentation.
File Description
packages/​server/​telemetry.ts Registers the OAuth span processor.
packages/​server/​start.ts Collects bounded token-request metadata.
packages/​server/​request-telemetry.ts Sanitizes URLs and classifies OAuth responses.
packages/​server/​request-telemetry.test.ts Tests response classification and privacy.
packages/​server/​auth/​oauth-span-processor.ts Normalizes expected OAuth error spans.
packages/​server/​auth/​oauth-span-processor.test.ts Tests SDK and Better Auth integration.
packages/​server/​auth/​oauth-request-metadata.ts Extracts bounded, safe request diagnostics.
packages/​server/​auth/​oauth-request-metadata.test.ts Tests metadata limits and body preservation.
packages/​server/​auth/​oauth-errors.ts Defines allowlisted outcomes and reasons.
packages/​cli/​util.ts Handles terminal re-login errors.
packages/​cli/​util.test.ts Tests re-login error presentation.
packages/​cli/​session.ts Adds rejection guards and credential recovery.
packages/​cli/​session.test.ts Tests rejection persistence and recovery.
packages/​cli/​credentials.ts Adds the rejection tombstone field.
docs/​troubleshooting.md Documents re-login recovery.
design/​runtime-observability.md Documents OAuth telemetry boundaries.
design/​authentication.md Documents terminal refresh rejection behavior.

Review effort: Balanced

Comment thread packages/cli/session.ts Outdated
Comment thread packages/cli/session.ts Outdated
Comment thread packages/cli/session.ts Outdated
Copilot AI balanced review requested due to automatic review settings October 16, 2026 22:50
@jgpruitt

Copy link
Copy Markdown
Collaborator Author

Review findings addressed in 50e87f9

All seven distinct Copilot/Opus findings are addressed:

  • Shared login/logout/refresh/rejection mutation locking, with stable keychain lock identity and conservative stale-401 cleanup.
  • Fresh access-only/device login recovery after a lock wait.
  • Local rejection memory before potentially empty keychain rereads.
  • me serve preserves terminal auth errors as 401/UNAUTHORIZED rather than retryable 502s.
  • Genuine Better Auth and pinned better-call APIError classes are both covered by the narrow telemetry classifier; real endpoint cases verify unchanged HTTP errors.
  • Background credential writes no longer switch default servers.
  • Client-ID telemetry hashes are domain separated from token lookup hashes.

Validation

  • ./bun install --frozen-lockfile passed.
  • ./bun run check:full passed, including all 33 CLI end-to-end tests.
  • Regression coverage includes real-process login/logout races, reactive access-only recovery, keychain read/write failure paths, proxy auth behavior with no transport retries, and concrete provider error classes.
  • A fresh independent Opus 5.5 re-review approves: all prior Opus findings and all three Copilot comments were addressed, with no medium-or-higher findings. It left low-severity operational follow-ups for read-only HOME with keychain locking and a long/hung refresh delaying final login persistence; these limitations are documented.

No deployment or release tag was created.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A temporary keychain lookup failure can discard a successful token rotation, causing revoked-token replay and unnecessary re-login.

2 open issues
2 resolved since last review

Review effort: Balanced

Comment thread packages/cli/session.ts Outdated
Copilot AI balanced review requested due to automatic review settings October 16, 2026 23:34
@jgpruitt

Copy link
Copy Markdown
Collaborator Author

Latest Copilot feedback addressed

Pushed 66faa0b to distinguish confirmed credential absence from unavailable storage and retain an already-successful OAuth rotation through transient keychain lookup failures. New hermetic platform tests and rotation/concurrent mutation regressions are included. Full ./bun run check:full passed, including all 33 CLI e2e tests; focused independent review found no medium/high issues.

Replied to and resolved the new rotation thread. Also clarified and resolved the original, outdated login/logout race thread, which was already fixed by the shared mutation boundary and now has real-process tests for both rotation and rejection writes.

Production is unchanged; PR remains open.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved cross-store credential coordination issues and live-span exception filtering require final human review.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Fallback tombstone masks newer shared-keychain login

packages/​cli/​credentials.ts:453

A fallback-file tombstone can hide a newer shared-keychain login across XDG directories. If process A writes a rejection to its fallback file while the keychain is unavailable, then process B using a different XDG_CONFIG_HOME logs in after recovery, B removes only its own fallback copy. A still returns its old tombstone here without checking the keychain, so restarting A does not restore access to the new login. Share fallback state for keychain-backed stores or reconcile token-specific tombstones with keychain credentials, while preserving rejection of the old token. Add a regression covering these two directories.

2 resolved since last review

Review effort: Balanced

Comment thread packages/cli/credentials.ts
Copilot AI balanced review requested due to automatic review settings October 19, 2026 14:36
@jgpruitt

Copy link
Copy Markdown
Collaborator Author

Mixed-mode/cross-XDG credential coordination addressed

Pushed 3aad3ae plus documentation commit 24de2c3. This is a coherent storage-source refactor rather than reversing file/keychain priority:

  • Keychain-enabled clients share token-free per-origin authority metadata selecting keychain, an actual canonical fallback path, or durable logout. Other XDG directories follow that source, so an old private-file tombstone cannot hide a fresh shared login.
  • Canonical origin lock precedes sorted/deduplicated whole-file locks. Mixed keychain/file-only processes sharing a physical fallback serialize, and different origins cannot lose each other’s entries.
  • Background updates remain on the selected fallback even after keychain recovery. Explicit login/logout retires readable active records before publication. Missing/corrupt selected sources never revive an alternate credential.
  • Legacy file-first precedence is preserved. Explicit recovery can replace damaged metadata; a readable file that cannot be locked is not silently abandoned. Distinct private file-only credentials are not overwritten by keychain failover. Healthy keychain operation makes no extra plaintext token copy.
  • File-only mode remains usable with writable XDG and read-only HOME. Unreachable private stores require cleanup in their owning environment; old clients outside this protocol remain uncoordinated.

Validation

  • ./bun run check: passed (lint, root/web typecheck, 1,555 tests passed; 5 existing skips).
  • ./bun run test --only-failures: passed (1,958 package tests, including database integration; 5 existing skips).
  • Independent black-box cross-store suite: 25 passed, including real processes, failure injection and metadata privacy.
  • Final Opus 5.5 verification: approve, no blocking findings; 126 focused tests passed.
  • Full live e2e is not green locally: 32/33 passed; semantic-search setup waits for embeddings and times out. A separate minimal provider probe returned HTTP 429 credit_balance_exhausted / insufficient_quota. Tests were not skipped or weakened. Live e2e needs rerun when credits are available (or with the CI project credential).

No deployment, merge, or release tag was created.

@jgpruitt

Copy link
Copy Markdown
Collaborator Author

CI status: blocked by exhausted embedding-provider credits

Latest CI run: https://github.com/timescale/memory-engine/actions/runs/39270573482

  • Lint/typecheck, Docker build, npm package builds, and the database integration step passed.
  • Four live OpenAI embedding tests failed/time out. Logs explicitly report: You have no credits remaining. Add credits to continue using the API.
  • CLI e2e: 32/33 passed; the semantic-search case times out waiting for embeddings for the same provider dependency.
  • No credential/auth regression failures were reported. Local hermetic/unit/package/integration suites and the focused Opus 5.5 verification passed as documented above.

A separate local probe returned HTTP 429 credit_balance_exhausted / insufficient_quota, consistent with CI. Tests were not disabled or weakened. Restore embedding-provider credits (or configure an authorized funded CI credential), then rerun the failed CI jobs before merging. No billing or secret configuration was changed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-process credential coordination and live-span adaptation warrant human review, and legacy-setting loss remains unresolved.

1 open issue
1 resolved since last review

Review effort: Balanced

keychainDelete(origin);
/** Refuse destructive read/modify/write of an unreadable or corrupt file. */
function writableSecrets(path: string): CredentialsFile {
const stored = readSecretsState(path);
@jgpruitt

Copy link
Copy Markdown
Collaborator Author

Superseded by #226, a minimal version of this fix (6 files, ~120 lines). The credential-authority/lock refactor here is left out; this branch remains as a reference.

@jgpruitt jgpruitt closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants