Skip to content

fix: generate AuthBridge routes in separate ConfigMap with correct format - #517

Open
Alan-Cha wants to merge 9 commits into
mainfrom
feat/agentruntime-auth-config
Open

fix: generate AuthBridge routes in separate ConfigMap with correct format#517
Alan-Cha wants to merge 9 commits into
mainfrom
feat/agentruntime-auth-config

Conversation

@Alan-Cha

Copy link
Copy Markdown
Member

Summary

Fixes two bugs in AuthBridge route generation from AgentRuntime spec.auth.outbound:

  1. Routes architecture bug: Generate routes in separate ConfigMap with file reference instead of inline in config.yaml
  2. Routes format bug: Generate routes in AuthBridge's routing.Route format instead of AgentRuntime CRD format

Bug 1: Routes Architecture

Problem: AuthBridge crashed with "cannot unmarshal array into Go struct field" when routes were inline in config.yaml

Root Cause: AuthBridge's token-exchange plugin in proxy-sidecar mode expects routes loaded from an external file, not inline in the config

Solution:

  • Generate routes in separate authbridge-routes-<crName> ConfigMap
  • Reference via routes: {file: "/etc/authproxy/routes.yaml"} in config.yaml
  • Mount the routes ConfigMap as a volume

Bug 2: Routes Format

Problem: Routes didn't match even after Bug 1 was fixed - AuthBridge logged "no matching route"

Root Cause: Operator generated routes in AgentRuntime CRD format, but AuthBridge expects routing.Route format

Wrong format (AgentRuntime CRD):

- destination:
    host: "hostname"
  audiences:
    - "audience"

Correct format (AuthBridge routing.Route):

- host: "hostname"
  target_audience: "audience"

Solution: Changed route generation to use flat AuthBridge format with:

  • Flat host field (not nested under destination)
  • Single target_audience string (not audiences array)

Testing

Complete E2E test on fresh Kind cluster verified:

  • ✅ Routes ConfigMap created with correct format
  • ✅ Config references file correctly
  • ✅ Routes match successfully (no "no matching route" errors)
  • ✅ Token exchange plugin invoked correctly

Files Modified

  • operator/internal/webhook/injector/pod_mutator.go: Route generation and ConfigMap creation
  • operator/internal/webhook/injector/volume_builder.go: Added overrideRoutesConfigMapInVolumes function

Related Issues

Fixes rossoctl/rossoctl#2334

Assisted-By: Claude Code

AuthBridge's token-exchange plugin in proxy-sidecar mode expects routes
to be loaded from an external file, not inlined in config.yaml. Inline
routes cause unmarshal errors at runtime.

Changes:
- Generate routes.yaml in separate authbridge-routes-<crName> ConfigMap
- Set config.routes to {file: "/etc/authproxy/routes.yaml"} instead of inline array
- Add overrideRoutesConfigMapInVolumes() to mount per-agent routes ConfigMap
- Update ensurePerAgentConfigMap() to return both config and routes CM names

Fixes: rossoctl/rossoctl#2334
Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
Routes must use AuthBridge's flat YAML structure:
- host: "hostname"  (flat field, not nested under destination)
- target_audience: "audience"  (single string, not audiences array)

Previous bug generated AgentRuntime CRD format:
- destination:
    host: "hostname"
  audiences:
    - "audience"

This caused AuthBridge to fail loading routes, resulting in
"no matching route" errors even though routes.yaml existed.

The router strips ports before matching, so routes should not include
ports in the host field.

Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
@Alan-Cha
Alan-Cha requested a review from a team as a code owner August 26, 2026 22:34
The function now returns (configCMName, routesCMName, error) instead of
(configCMName, error). Update all test callers to handle the new signature.

Assisted-By: Claude Code
Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
…ormat

The test was checking for inline routes in config.yaml, but the implementation
now generates routes in a separate ConfigMap and references it via file path.

Updated test to:
- Verify config.yaml contains routes file reference
- Fetch the separate routes ConfigMap
- Parse routes.yaml from that ConfigMap
- Check for flat route format (host + target_audience) instead of nested format

Assisted-By: Claude Code
Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
@Alan-Cha Alan-Cha added Ready for Review! ready-for-ai-review Request automated AI code review from clawgenti labels Aug 27, 2026

@clawgenti clawgenti left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes two real bugs (inline routes crash + wrong route format) and the approach is clean. A few items worth addressing before merge.

  • Silent audience truncation: when , only is used with no warning; operators who specify multiple audiences will silently lose the extras.
  • Shadowed err variable (line 1353): var err error inside the if agentRuntime != nil block shadows the named return err, so a marshal failure there won't set the named return — the error is still returned via the explicit return "", "", err, but this is fragile and should use the named return directly.
  • overrideRoutesConfigMapInVolumes has no unit test: the new volume_builder.go function follows the same pattern as overrideAuthBridgeConfigMapInVolumes but lacks a corresponding test case.

Reviewed by clawgenti using the github-pr-review skill

Comment thread operator/internal/webhook/injector/pod_mutator.go
Comment thread operator/internal/webhook/injector/pod_mutator.go Outdated
…warning

**Path constants:**
- Define AuthProxyMountPath and AuthProxyRoutesFile in namespace_config.go
- Replace all hardcoded "/etc/authproxy" and "/etc/authproxy/routes.yaml" strings
- Improves maintainability by centralizing path definitions

**Multiple audiences validation:**
- Add warning log when AgentRuntime.spec.auth.outbound route has >1 audience
- Only first audience is used (AuthBridge limitation)
- Addresses issue #518

Assisted-By: Claude Code
Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
Alan-Cha added a commit that referenced this pull request Aug 27, 2026
…ride test

**Shadowed variable fix:**
- Remove `var err error` inside agentRuntime block (line 1363)
- Use the function's named return parameter directly
- Prevents fragile error handling where marshal failure might not propagate

**Test coverage:**
- Add TestOverrideRoutesConfigMapInVolumes following the pattern of
  TestOverrideEnvoyConfigMapInVolumes
- Verifies volume swap without mutation
- Checks that Optional is set to false for per-agent routes

Addresses review comments from #517

Assisted-By: Claude Code
Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
@Alan-Cha

Copy link
Copy Markdown
Member Author

All review comments addressed in commit 9eb2aaf:

Silent audience truncation - Already fixed in d4a121a. Added warning log when multiple audiences specified:

if len(outboundRoute.Audiences) > 1 {
    mutatorLog.Info("multiple audiences specified but only first will be used",
        "namespace", namespace, "crName", crName,
        "route", outboundRoute.Destination.Host,
        "audiences", outboundRoute.Audiences,
        "using", outboundRoute.Audiences[0])
}

Also filed issue #518 for tracking.

Shadowed err variable - Fixed by removing var err error at line 1363. Now uses the function's named return parameter directly.

Missing unit test - Added TestOverrideRoutesConfigMapInVolumes in volume_builder_test.go following the same pattern as TestOverrideEnvoyConfigMapInVolumes. Verifies:

  • Volume swap without mutation
  • Optional is set to false for per-agent routes
  • Handles case where authproxy-routes volume doesn't exist

Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
…ride test

**Shadowed variable fix:**
- Remove `var err error` inside agentRuntime block (line 1363)
- Use the function's named return parameter directly
- Prevents fragile error handling where marshal failure might not propagate

**Test coverage:**
- Add TestOverrideRoutesConfigMapInVolumes following the pattern of
  TestOverrideEnvoyConfigMapInVolumes
- Verifies volume swap without mutation
- Checks that Optional is set to false for per-agent routes

Addresses review comments from #517

Assisted-By: Claude Code
Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
@Alan-Cha
Alan-Cha force-pushed the feat/agentruntime-auth-config branch from 9eb2aaf to 2bf20d8 Compare August 27, 2026 04:14

@clawgenti clawgenti left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactors AuthBridge routes from inline config.yaml injection to a separate ConfigMap, replacing hardcoded paths with named constants — clean and well-structured change overall.

Findings:

  • Silent truncation of multiple audiences (only Audiences[0] used) with a warn log but no error return — callers with multiple audiences will silently lose entries.
  • HostRegex is mapped to the host field with a comment noting the semantic mismatch; this is a behavioral change with no fallback or error.

Reviewed by clawgenti using the github-pr-review skill

// target_audience is a single string, not array
// Take first audience if multiple specified
if len(outboundRoute.Audiences) > 0 {
route["target_audience"] = outboundRoute.Audiences[0]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the first audience is used and the rest are silently dropped — the warning log is good but callers relying on multiple audiences (possible today or via future API extensions) will get no error. Consider returning an error or at minimum documenting this limitation in the field's godoc/API type. Tracking issue #518 is referenced, but that issue should probably be linked in the AgentRuntime CRD validation too.

}
if outboundRoute.Destination.HostRegex != "" {
// AuthBridge router doesn't support hostRegex - use glob pattern in host field
route["host"] = outboundRoute.Destination.HostRegex

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mapping HostRegex directly to the host field is a silent semantic change: a regex pattern (e.g. .*\.team1\.svc\.cluster\.local) will be passed where AuthBridge expects a glob. If AuthBridge doesn't support regex syntax at all, this will silently fail to match at runtime with no observable error at admission time. Consider validating the regex-to-glob conversion or adding a more explicit warning/event.

…yRoutesConfigMapName constant

Addresses goconst linter error - use existing constant instead of repeating
the string literal 3+ times.

Assisted-By: Claude Code
Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

@clawgenti clawgenti left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three items from the previous review have been addressed cleanly: shadowed err is gone, the audience-truncation warning is in place, and overrideRoutesConfigMapInVolumes has a proper unit test. One residual suggestion below.

  • HostRegex is silently coerced to host (line 1341): a regex pattern is put in a field that AuthBridge expects as a glob. These are semantically different — a regex like .*\.team1\.svc\.cluster\.local will not match as a glob. The comment acknowledges the mismatch but takes no action. Worth either (a) rejecting HostRegex routes with a clear error/warning, or (b) documenting the expected glob syntax in the CRD field itself so callers know they must supply a glob, not a regex.

Reviewed by clawgenti using the github-pr-review skill

}
if outboundRoute.Destination.HostRegex != "" {
// AuthBridge router doesn't support hostRegex - use glob pattern in host field
route["host"] = outboundRoute.Destination.HostRegex

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: HostRegex is being stored verbatim in the host field, but AuthBridge's router matches host as a glob, not a regex. A regex like .*\.team1\.svc\.cluster\.local won't match as a glob (*.team1.svc.cluster.local would). Consider logging a more actionable warning here (e.g. "HostRegex is not supported; treating value as a glob — ensure it uses glob syntax") or returning an error to surface the mismatch to the operator early.

…ntax)

The CRD field is named "hostRegex" but AuthBridge expects glob patterns
(github.com/gobwas/glob), not regex. Users who follow the CRD documentation
will provide patterns that don't match.

Added warning log to help users discover the mismatch:
- Log when hostRegex is used
- Explain that glob syntax is expected (*.example.com)
- Reference that regex syntax won't work (.*\.example\.com)

Filed issue #520 to track fixing the CRD documentation/field name.

Addresses review comment from #517

Assisted-By: Claude Code
Signed-off-by: Alan Cha <Alan.cha1@ibm.com>
@Alan-Cha

Copy link
Copy Markdown
Member Author

Review comment addressed in commit c9f1708:

HostRegex glob vs regex mismatch - Added warning log when hostRegex is used to alert users about the semantic mismatch:

mutatorLog.Info("hostRegex field is mapped to AuthBridge glob pattern (not regex)",
    "namespace", namespace, "crName", crName,
    "hostRegex", outboundRoute.Destination.HostRegex,
    "note", "use glob syntax like '*.team1.svc.cluster.local', not regex '.*\\.team1\\.svc\\.cluster\\.local'")

Also filed issue #520 to track fixing the CRD field documentation/name. The field name is misleading - it should be hostGlob or the documentation should clarify it expects glob syntax, not regex.

Why not error/reject? No examples currently use hostRegex (all use exact host matches), so the warning provides helpful guidance without blocking users. Issue #520 tracks a more comprehensive fix (field rename + deprecation or doc fix).

@clawgenti clawgenti left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All previous review findings have been addressed: the hostRegex→glob semantic mismatch now has a clear warning log and issue #520 filed for the CRD documentation fix. CI is green across all completed checks. All checks pass. Ready for human review.


Reviewed by clawgenti using the github-pr-review skill

@Alan-Cha

Copy link
Copy Markdown
Member Author

Test Coverage Update

Filed issue #521 to track adding E2E tests for runtime token exchange verification.

Current coverage (this PR):

  • ✅ Unit tests for routes ConfigMap generation
  • ✅ Unit tests for config.yaml file reference
  • ✅ Unit tests for volume mounting/override
  • ✅ Manual E2E testing (Kind cluster with weather agent demo)

Follow-up (issue #521):

  • ⏳ E2E test that verifies AuthBridge reads routes from file
  • ⏳ E2E test that confirms token exchange occurs at runtime
  • ⏳ E2E test that validates exchanged token has correct audience
  • ⏳ E2E test coverage for glob patterns and multiple routes

The manual E2E test from this PR's development confirmed everything works (routes match, token exchange happens, agent successfully calls tool), but we need automated E2E tests in the test suite to catch regressions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Review! ready-for-ai-review Request automated AI code review from clawgenti

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Operator SPIFFE auth: ghcr.io/rossoctl/cortex/spiffe-helper:latest sidecar image unpublished/inaccessible (403)

2 participants