Skip to content

feat(policy): lint authored policy files offline - #5358

Merged
johntmyers merged 4 commits into
NVIDIA:mainfrom
shiju-nv:feat/4035-offline-policy-lint/shiju-nv
Nov 17, 2026
Merged

johntmyers merged 4 commits into
NVIDIA:mainfrom
shiju-nv:feat/4035-offline-policy-lint/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add openshell policy lint <file> to validate authored policies without a gateway. After parsing, it reports file-only semantic violations together and identifies deferred live checks.

Related Issue

Closes #4035

Changes

  • Reject oversized encoded policies before expanding ambiguity or middleware diagnostics. Retain the canonical-size recheck and aggregate semantic errors for admitted files.
  • Reuse shared safety, authoring, ambiguity and built-in middleware validation.
  • Dispatch lint before gateway selection and authentication.
  • Cover offline operation, multiple violations and gateway rejection parity.

Release Notes

openshell policy lint policy.yaml validates authored files offline. Success does not establish that a sandbox can start.

Testing

  • Two focused authored-policy unit tests passed, including canonical encoded-size validation.
  • Offline CLI integration and two in-process gateway tests passed. The oversized case includes 256 conflicting endpoints and checks that ambiguity diagnostics are absent.
  • Scoped production and affected CLI integration Clippy with -D warnings, Rust formatting, whitespace and SPDX checks passed on macOS with Rust 1.95.0.
  • Five affected standalone test lockfiles pass cargo metadata --locked --offline after synchronizing the policy dependency.
  • Full sandbox E2E not run locally; hosted CI owns broad verification.

Checklist

  • Follows Conventional Commits.
  • Commits are signed off (DCO).
  • Architecture docs updated (not applicable; command guide and policy/CLI skills updated).

@shiju-nv shiju-nv added area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. rust Pull requests that update rust code labels Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

CreateSandbox size rejections after MCP defaulting previously included BadRequest(spec.policy) and ErrorInfo(INVALID_ARGUMENT), but now return bare INVALID_ARGUMENT.

  • Impact: SDK consumers handling policies that cross 262,144 bytes during MCP defaulting lose the structured field and reason needed for automated error handling.
  • Action: For v0.1.8, preserve BadRequest(spec.policy) and ErrorInfo(INVALID_ARGUMENT) by retaining the existing gateway size-check path or returning the equivalent structured error.
  • PR: feat(policy): lint authored policy files offline #5358
Review details
  • Execution: complete
  • Candidate: PR #5358 (4afdab630a7d3dfc4c2176bf892815e453513935)
  • Baseline: af8182f7b63da6dd8bbea8b3c313227764ee0dea (af8182f7b63da6dd8bbea8b3c313227764ee0dea)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shiju-nv
shiju-nv force-pushed the feat/4035-offline-policy-lint/shiju-nv branch from 74dcaf5 to b49c040 Compare November 7, 2026 09:18
@shiju-nv shiju-nv removed the area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. label Nov 7, 2026

@shvma shvma left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran 11 authored policies through openshell policy lint and through CreateSandbox on the in-process gateway from openshell_server::test_support. For the four built-in middleware cases I compared against MiddlewareRegistry::validate_policy_configs with the built-in services instead (see the note at the end). Lint agreed everywhere except the external middleware case, which lint skips by design, as the docs say. Two small things, both with patches below.

  1. The pair order in endpoint ambiguity messages isn't stable. With two conflicting rules a and b, openshell policy lint printed 'a' … and 'b' in 10 of 20 runs and 'b' … and 'a' in the other 10. find_endpoint_ambiguities walks the network_policies map without sorting. The code predates this PR, and the gateway message has the same problem. Lint makes it more visible because its output goes to CI logs and scripts. The crate already states the rule in lib.rs: "Sort rule keys so callers receive stable validation errors for equivalent policy inputs." Patch 1 sorts the keys the same way and adds a unit test that fails without the change (one run got [("mid", "alpha"), ("alpha", "zeta")]) and passes with it. cargo test -p openshell-policy --lib is 300/300 and clippy with -D warnings is clean. Fine as a follow-up if you'd rather keep this PR to lint.

  2. authored_policy_lint_matches_gateway_file_validation covers identity, reserved keys and credential bindings. policy_lint_integration.rs covers ambiguity on the lint side only, and I didn't find a test that CreateSandbox rejects the same authored file. The gateway rejects an ambiguous file on CreateSandbox even without providers, with FailedPrecondition (policy.rs), so it doesn't fit the existing loop that expects InvalidArgument. Patch 2 adds a separate test on the same harness. It uses one rule with two endpoints, so it doesn't depend on the ordering in point 1 and works with or without Patch 1. It passes, and clippy is clean.

Not for this PR, just context for why the parity test can't cover built-in middleware: gateway_service_with_driver doesn't register the built-ins that the real gateway does (openshell-server lib.rs), so openshell/regex is always "not registered" there.

Patch 1: sort ambiguity pairs by rule key
diff --git a/crates/openshell-policy/src/ambiguity.rs b/crates/openshell-policy/src/ambiguity.rs
index efb846a..0eb9739 100644
--- a/crates/openshell-policy/src/ambiguity.rs
+++ b/crates/openshell-policy/src/ambiguity.rs
@@ -58,9 +58,12 @@ struct EndpointRef<'a> {
 /// request-specific metadata) path selectors can match the same request.
 #[must_use]
 pub fn find_endpoint_ambiguities(policy: &SandboxPolicy) -> Vec<EndpointAmbiguity> {
-    let endpoints = policy
-        .network_policies
-        .iter()
+    // Protobuf maps do not preserve iteration order. Sort rule keys so callers
+    // receive the same ambiguity pairs, in the same order, for equivalent input.
+    let mut network_policies: Vec<_> = policy.network_policies.iter().collect();
+    network_policies.sort_by_key(|(key, _)| *key);
+    let endpoints = network_policies
+        .into_iter()
         .flat_map(|(key, rule)| {
             let policy_name = if rule.name.is_empty() {
                 key.as_str()
@@ -775,6 +778,43 @@ mod tests {
         assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty());
     }
 
+    #[test]
+    fn ambiguities_are_reported_in_rule_key_order() {
+        let pairs = |policy: &SandboxPolicy| {
+            find_endpoint_ambiguities(policy)
+                .into_iter()
+                .map(|ambiguity| (ambiguity.left_policy, ambiguity.right_policy))
+                .collect::<Vec<_>>()
+        };
+        let build = || {
+            let mut policy = SandboxPolicy::default();
+            for (key, tls) in [
+                ("zeta", NetworkTlsMode::Skip),
+                ("alpha", NetworkTlsMode::Unspecified),
+                ("mid", NetworkTlsMode::Skip),
+            ] {
+                let mut endpoint = endpoint("api.example.com", 443);
+                endpoint.tls = tls as i32;
+                policy.network_policies.insert(
+                    key.to_string(),
+                    NetworkPolicyRule {
+                        endpoints: vec![endpoint],
+                        ..Default::default()
+                    },
+                );
+            }
+            policy
+        };
+        let expected = vec![
+            ("alpha".to_string(), "mid".to_string()),
+            ("alpha".to_string(), "zeta".to_string()),
+        ];
+        // Each map gets its own hash seed, so repeat to catch unordered iteration.
+        for _ in 0..32 {
+            assert_eq!(pairs(&build()), expected);
+        }
+    }
+
     #[test]
     fn compatible_request_rules_may_overlap() {
         let mut left = endpoint("api.example.com", 443);
Patch 2: parity test for endpoint ambiguity
diff --git a/crates/openshell-cli/tests/policy_only_process_identity_integration.rs b/crates/openshell-cli/tests/policy_only_process_identity_integration.rs
index 559534e..90b77ca 100644
--- a/crates/openshell-cli/tests/policy_only_process_identity_integration.rs
+++ b/crates/openshell-cli/tests/policy_only_process_identity_integration.rs
@@ -183,3 +183,35 @@ async fn authored_policy_lint_matches_gateway_file_validation() {
         }
     }
 }
+
+#[tokio::test]
+async fn authored_policy_lint_matches_gateway_endpoint_ambiguity() {
+    let server = run_server().await;
+    let policy = openshell_policy::parse_sandbox_policy(
+        "version: 1\nnetwork_policies:\n  conflict:\n    endpoints:\n      - {host: example.com, port: 443}\n      - {host: example.com, port: 443, tls: skip}\n",
+    )
+    .unwrap();
+    let ambiguities = openshell_policy::find_endpoint_ambiguities(&policy);
+    assert_eq!(ambiguities.len(), 1, "{ambiguities:?}");
+    let mut client = grpc_client(&server.endpoint, &server.tls).await.unwrap();
+    let error = client
+        .create_sandbox(CreateSandboxRequest {
+            name: "ambiguity".to_string(),
+            spec: Some(SandboxSpec {
+                policy: Some(policy),
+                ..Default::default()
+            }),
+            workspace_scope: Some(openshell_core::proto::workspace_selector(
+                "default".to_string(),
+            )),
+            ..Default::default()
+        })
+        .await
+        .expect_err("an ambiguous authored file must be rejected by the gateway too");
+    assert_eq!(error.code(), tonic::Code::FailedPrecondition);
+    assert!(
+        error.message().contains(&ambiguities[0].to_string()),
+        "{error} did not report {}",
+        ambiguities[0]
+    );
+}

@purp purp added topic:policy Policy authoring, validation, proving, distribution, enforcement, ergonomics, and hardening. topic:ux CLI commands/flags, help, errors/exit codes, human-readable output, TUI, and first-run experience. meta:labeler-checkpoint Temporary label used by /label-issue to record a verified classification checkpoint. and removed meta:labeler-checkpoint Temporary label used by /label-issue to record a verified classification checkpoint. labels Nov 13, 2026
@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Nov 16, 2026

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The offline lint command fits validated issue #4035, and the command guide and related skills are updated. One blocking Warning remains: the new lint path expands ambiguity diagnostics before rejecting oversized policies, which can exhaust the lint process's memory.

Action required: @shiju-nv, reject oversized encoded policies before collecting ambiguity diagnostics, retaining the canonical-size recheck and aggregation for admitted files.

Blocking findings:

  • GATOR-41daeae1-01: enforce the size admission bound before expanding endpoint pairs; see the inline finding.

Carried findings: None.

Non-blocking suggestions: None.

Gator metadata
  • Validation: implements validated offline policy-lint issue #4035; author has repository write access.
  • Docs: Fern policy command guide and related public skills updated.
  • Checks: current-head Branch Checks, Helm Lint, Trivy Changes and DCO passed; separate Review Readiness failures remain visible.
  • E2E: standard runtime coverage requested with test:e2e; dispatch and results are tracked by supervised reconciliation.
  • Head SHA: 41daeae1f755b5f7e0ef199e2097c9ee70c1a3f9
  • Base SHA: af8182f7b63da6dd8bbea8b3c313227764ee0dea
  • Merge base SHA: 53bf840cd22c47c7002d47af7e9f7292b19174a4
  • Patch ID: 4d2192425fe91f234f37d0f7f12661a584a2f1eb
  • Gator payload: 13
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:in-review

Comment thread crates/openshell-cli/src/run.rs Resolved
@johntmyers johntmyers added the gator:in-review Gator is reviewing or awaiting PR review feedback label Nov 16, 2026
Validate authored files before gateway selection or authentication. Share creation safety checks and report file-only violations together while deferring checks that require live gateway state.

Closes NVIDIA#4035

Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv
shiju-nv force-pushed the feat/4035-offline-policy-lint/shiju-nv branch from 41daeae to 6791e9f Compare November 17, 2026 12:54
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@johntmyers

Copy link
Copy Markdown
Collaborator

/ok to test 6791e9f

@johntmyers

johntmyers commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

gator-agent

PR Review Status

@shiju-nv, the latest fix checks encoded policy size before expanding ambiguity or middleware diagnostics. I checked that the canonical-size recheck and error aggregation remain for admitted files, and that the regression covers oversized input with conflicting endpoints. GATOR-41daeae1-01 is fixed, and its review thread is resolved. The independent follow-up review found no new blockers in the author-only delta.

Blocking findings: None.

Carried findings: None; GATOR-41daeae1-01 is resolved.

The test mirror now matches the current head after /ok to test. Branch Checks, Helm Lint, and standard E2E are queued; Trivy Changes and DCO passed. Gator will monitor their results.

Gator metadata
  • Validation: implements previously validated issue feat: add a command to check a policy file before creating a sandbox #4035; author has repository write access.
  • Docs: policy guide and related skills remain updated; this fix adds no new UX surface.
  • Checks: Branch Checks and Helm Lint queued; Trivy Changes and DCO passed.
  • E2E: test:e2e is applied; current-head Branch E2E Checks run 44089329733 is queued.
  • Head SHA: 6791e9fd05c75dfd803103ec9201326440f9924d
  • Base SHA: ce58cf8e9b953281352d98a637e41616295b1318
  • Merge base SHA: 0be63c5b0de27692df03ba0e4b3bc212312d3b3c
  • Patch ID: 69ba4e54c88938dba89b3fe641c81712548251f3
  • Gator payload: 13
  • Review mode: follow_up
  • Previous reviewed SHA: 41daeae1f755b5f7e0ef199e2097c9ee70c1a3f9
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:in-review Gator is reviewing or awaiting PR review feedback labels Nov 17, 2026
Signed-off-by: Shiju <shiju@nvidia.com>
@johntmyers

Copy link
Copy Markdown
Collaborator

/ok to test 3d3edf7

@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

PR Review Status

@shiju-nv, I checked the standalone lockfile synchronization: all five changes add the policy crate's existing prost dependency without changing resolved package versions. The independent follow-up review found no new blockers in this delta. The earlier size-admission finding remains resolved.

Blocking findings: None.

Carried findings: None; GATOR-41daeae1-01 remains resolved.

The test mirror matches the current head after /ok to test. Branch Checks and Helm Lint are queued; the current-head E2E workflow is dispatched and pending scheduling. Trivy Changes and DCO passed. Gator will inspect the remaining results next cycle.

Gator metadata
  • Validation: continues previously validated issue feat: add a command to check a policy file before creating a sandbox #4035; author has verified repository write access.
  • Docs: existing policy guide and related skills remain updated; this lockfile delta adds no UX surface.
  • Checks: Branch Checks run 44092269209 and Helm Lint run 44092269051 queued; Trivy Changes and DCO passed.
  • E2E: test:e2e applied; current-head Branch E2E Checks run 44092269564 dispatched, pending scheduling.
  • Head SHA: 3d3edf7f28d13056a7ec0a729186212a517d509b
  • Base SHA: ce58cf8e9b953281352d98a637e41616295b1318
  • Merge base SHA: 0be63c5b0de27692df03ba0e4b3bc212312d3b3c
  • Patch ID: 8fa9099f32e01d787df6a654e9f78a53074ca202
  • Gator payload: 13
  • Review mode: follow_up
  • Previous reviewed SHA: 6791e9fd05c75dfd803103ec9201326440f9924d
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:approval-needed Gator completed review; maintainer approval needed and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Nov 17, 2026
@johntmyers
johntmyers enabled auto-merge November 17, 2026 23:08
@johntmyers
johntmyers added this pull request to the merge queue Nov 17, 2026
@johntmyers johntmyers added gator:merge-ready and removed gator:approval-needed Gator completed review; maintainer approval needed labels Nov 17, 2026
Merged via the queue into NVIDIA:main with commit ecce595 Nov 17, 2026
169 of 171 checks passed
@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Monitoring Complete

PR #5358 has merged. Gator monitoring is complete.

Gator metadata
  • Head SHA: 3d3edf7f28d13056a7ec0a729186212a517d509b
  • Gator payload: 13
  • Terminal state: merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. rust Pull requests that update rust code test:e2e Requires end-to-end coverage topic:policy Policy authoring, validation, proving, distribution, enforcement, ergonomics, and hardening. topic:ux CLI commands/flags, help, errors/exit codes, human-readable output, TUI, and first-run experience.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add a command to check a policy file before creating a sandbox

4 participants