[shreyanshjain7174]

Plan

Phase 0, item 6 of the AgentGate scale-out prompt: real rate limiting.

cmd/agentgw called ratelimit.New(nil), so Allow() passed everything through. The limiter also ran after delegation verification, its map was unbounded, and it had no tenant or concurrency limits.

What changed

• Default-on token buckets.

  • Per API key: 20 rps, burst 40.
  • Per tenant: 100 rps, burst 200.
  • Per (key, service): optional, from rate_limit: in registry YAML.

• Per-tenant in-flight cap (default 32). A slot is held until the request ends, including SSE streams.

• Order. Key bucket, then the scope check, then tenant and service buckets, then delegation verification, vault, URL building and upstream. A key that is out of scope for a user can never spend that user's tenant tokens. Such requests keep their scope_denied receipt.

• Tenant identity.

  • Wildcard keys use key:<id>.
  • Scoped keys use human:<principal>, capped at 64 principals per key; extra principals share a key-scoped fallback.

  Rotating on_behalf_of cannot create new entries.

• Responses. Over the limit, the agent gets 429 rate_limited or concurrency_limited, a Retry-After of at least 1s, and a receipt, as before.

• Refunds. A concurrency_limited rejection refunds the tokens it took, so a tenant waiting at its cap does not drain its own buckets.

• Memory bound. Idle entries are evicted after 10 minutes, and the map is LRU-capped. When it is full, eviction scans past pinned entries rather than failing closed.

• Panic safety. The slot release is registered before dispatch.

• Config. AGENTGATE_RATE_KEY_RPS, AGENTGATE_RATE_KEY_BURST, AGENTGATE_RATE_TENANT_RPS, AGENTGATE_RATE_TENANT_BURST, AGENTGATE_TENANT_MAX_INFLIGHT. 0 disables a limit. Startup refuses a disabled key limit while tenant limits are on.

Validation

• go build ./...: passed
• go vet ./...: passed
• go test -race ./... -count=2: passed
• semgrep --config auto --error --quiet internal/ratelimit internal/gateway cmd/agentgw: passed, no findings

Tests use a fake clock. They cover:

• key, tenant and service limits;
• the concurrency cap, with SSE holding its slot;
• the limiter running before delegation;
• out-of-scope hammering causing zero 429s for the real tenant;
• a pinned LRU tail not blocking new entries;
• the per-key tenant cap;
• panic-safe release;
• refunds;
• env validation.

Review

Two rounds by a second model (Claude Opus 5.5). Round 1 found that the new ordering let out-of-scope keys drain other tenants' buckets, and that a pinned LRU tail failed closed. Both are fixed with tests.

Known limits

Quotas are per process. Replicas do not share buckets.