Skip to content

Feat/multi report cli output - #1065

Open
andreasconrad-ruv wants to merge 7 commits into
NVIDIA:mainfrom
andreasconrad-ruv:feat/multi-report-cli-output
Open

andreasconrad-ruv wants to merge 7 commits into
NVIDIA:mainfrom
andreasconrad-ruv:feat/multi-report-cli-output

Conversation

@andreasconrad-ruv

@andreasconrad-ruv andreasconrad-ruv commented Nov 6, 2026 •

Copy link
Copy Markdown

Summary

  • Add repeatable --report FORMAT=PATH outputs alongside --format/--output.
  • Reuse existing scan results; retain recursive and batch scanning.
  • Remove unnecessary bounded-analysis and prepared-report infrastructure.

Validation

  • 20,096 unit and 249 integration tests passed.
  • Lint and formatting checks passed.

closes #1040

@andreasconrad-ruv
andreasconrad-ruv force-pushed the feat/multi-report-cli-output branch from f43a947 to cc8908e Compare November 6, 2026 09:59
Add repeatable --report FORMAT=PATH destinations alongside --format/--output. Reuse existing single, recursive, and batch scan results without new analysis infrastructure.

Validation: make test, make lint, make format, and make format-check passed.

Co-authored-by: codex <codex@ruv.de>
Signed-off-by: Andreas Conrad <andreas.conrad@ruv.de>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Changes Requested.

Value and readiness

The repeatable reports are useful: they reuse one analysis for JSON, SARIF and human-readable outputs. The new input-preservation check has a reachable data-loss bypass, so this is not ready for approval.

Material findings

Blocker — src/skillspector/cli.py:405-412 and contrib/batch_scan/batch_scan.py:405-408: validate the destination the writer will actually use. With a normal skill/SKILL.md and no skill/missing directory, --report json=skill/missing/../SKILL.md passes _validate_additional_report: output.exists() is false, and _validate_report_destination also permits its FileNotFoundError. parse_reports resolves paths only for duplicate comparisons and retains the original Path. After scanning, _write_reports calls write_text_no_follow (cli.py:424), whose open_output_parent invokes _normalize_root_owned_alias (file_output.py:53); that helper uses os.path.abspath (input_handler.py:252-255), collapses missing/.., and the writer replaces the real skill/SKILL.md. The new batch guard has the same existence-before-normalization gap. Thus an additional report can destroy a scanned input despite the advertised guard. Normalize once with the writer's rules before all alias/input checks, or reject unresolved forms consistently, and add byte-preservation tests for this alias in single and batch scans. The added input test covers an existing direct path only.

PIC tradeoffs

None identified. The supported report formats and scan-once behavior fit the request; correcting destination validation preserves that design.

Verification and gaps

Read the complete five-file diff, pinned source for the CLI, report node, writer, normalization helper, parser and added tests, plus linked #1040 and the complete empty review/discussion history. All six attached CI checks report success in run 42200940946; those results do not cover the missing-component alias above. The finding is a source trace, not a claimed runtime reproduction.
Static source review only. No contributor code, tests, snippets, imports, builds, setup, hooks, scanner invocations or model calls were executed.

Reviewed head: e6ed7b2df522b61f6bd5f2c50923d03ee1ffa6ce.

Match the no-follow writer's path normalization before duplicate and input-preservation checks in single and batch scans. Retain normalized report destinations through validation and writing.

Add byte-preservation regressions for missing/../ aliases targeting manifests and helper files, plus duplicate and symlink normalization coverage.

Co-authored-by: codex <codex@ruv.de>
Signed-off-by: Andreas Conrad <andreas.conrad@ruv.de>
@andreasconrad-ruv

Copy link
Copy Markdown
Author

@rng1995 All requested changes done with the last commit. Please review again.

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The destination-normalization change addresses the earlier missing/../ input-overwrite concern in source. Independent source review and a completed Greptile comparison found two other issues in the additional-report paths below.

Runtime tests, fresh-wheel verification, sample scans and live-provider checks were not performed; those checks remain pending.

Comment thread src/skillspector/cli.py Resolved
Comment thread src/skillspector/cli.py Resolved
Retain canonical scan state at both transitive return paths so additional formats preserve components, executable risk, suppression, and runtime telemetry without rescanning. Initialize zero-depth traversal metadata before primary rendering for consistent output.

Normalize primary output with the no-follow writer's rules and reject primary and additional destinations that alias explicit or effective shipped baselines before scanning. Add format-parity, state-preservation, baseline-alias, cleanup, and offline end-to-end regressions.

Signed-off-by: Andreas Conrad <andreas.conrad@ruv.de>
Signed-off-by: Andreas Conrad <andreas.conrad@ruv.de>
@andreasconrad-ruv
andreasconrad-ruv force-pushed the feat/multi-report-cli-output branch from 1a0253e to 480fce8 Compare November 12, 2026 10:42
@andreasconrad-ruv

Copy link
Copy Markdown
Author

@yashrajp22 all done, request for review.

And if you have appetite for output stuff still, is this something which would be appreciated? #1099

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Approved

Value and readiness

Repeatable --report FORMAT=PATH outputs let callers produce machine and human reports from one scan. The current revision is ready for source approval: all three previously raised technical issues are repaired, and I found no new material defect in the changed paths.

Material findings

None requiring changes. parse_reports now retains destinations normalized with the secure writer's rules before duplicate and input checks, closing the missing/../SKILL.md overwrite path. Explicit baseline checks run before local/URL input branching, and effective shipped baselines are checked before scanning. Both transitive return paths retain canonical scan state before adding report projections, preserving findings, suppression, executable risk, components, completeness and LLM telemetry when another format is rendered. Additional formats call the report node rather than the scan graph. Recursive scanning keeps its shared traversal and aggregate budgets; each output starts from the same aggregate state, keeps its output bound, and cannot alter the primary report's completeness exit decision. Batch outputs reuse the same sorted results. Single and recursive cleanup covers additional-output failures, and findings/risk exits occur after all requested reports are written.

PIC tradeoffs

Each requested format adds rendering and file-writing cost while reusing the completed analysis. Existing format support, registry JSON-only behavior, recursive limits and primary exit semantics remain consistent with the request. No new product-policy or public-schema decision is needed.

Verification and gaps

Read the complete five-file diff, all 797 added test lines, exact-head CLI/report/parser/writer/normalization source, linked request and all review/comment/thread bodies. The test source covers scan-once behavior, transitive depth and format order, canonical state equality, suppression, components, independent executable scores, telemetry, input/baseline alias byte preservation, structured stdout, output failures, cleanup and recursive per-format fallbacks. All six hosted checks in the inventory succeeded. Both technical review threads are resolved, but the active external changes-requested review remains a merge gate, and the fresh merge state is DIRTY because of conflicts. Pre-review and post-review heads match the frozen inventory. No contributor code, tests, imports, builds, hooks, scanner, regex probes or model calls were executed; this is static source review plus hosted-check evidence.

Reviewed head: 480fce80c346ef6840ded9160231416d1179b17b.

Preserve main's lazy CLI imports and shipped ZIP-baseline support alongside multi-report rendering and destination protections. Add combined-behavior regression tests.

Co-authored-by: codex <codex@ruv.de>
Signed-off-by: Andreas Conrad <andreas.conrad@ruv.de>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Changes Requested

Value and readiness

Repeatable reports share completed analyzer results, and the destination, baseline and transitive-state fixes from the prior review remain in place after conflict resolution. The remaining issue is that rendering again can change the report's account of the same scan.

Material findings

[P1] src/skillspector/cli.py:470 reruns provider availability when producing a different format. _render_report calls report(), which consults the remaining workflow time and invokes is_llm_available() again at src/skillspector/nodes/report.py:2063–2079. A Markdown primary can record available LLM coverage while additional JSON rendered after a workflow deadline or a transient credential failure reports different availability or degradation for the same completed analysis. The Gemini path can refresh credentials during this check. Recursive reporting repeats the probe for every child and requested alternative format. Capture the primary report's provider and runtime assessment once and reuse it when rendering additional formats. Add a regression where a second availability check would fail, asserting one report-time probe and consistent metadata across formats.

PIC tradeoffs

Rendering and file-writing cost scale with the number of outputs. Provider authentication and availability should reflect the completed scan. The ZIP baseline and lazy registry paths are covered by the new conflict-resolution tests.

Verification and gaps

Re-read the complete seven-file diff and compared the previous approved source with the current merge resolution. Checked original and current review threads, full transitive and recursive result paths, report metadata, credential resolution and regression source. All six exact-head hosted checks succeeded. The existing transitive parity fixture replaces availability with a constant successful result, so it does not exercise a changing probe. I did not run contributor code, tests, authentication or models.

Reviewed head: f97b943aa2b0ec40bd23f451987432acdb14e521.

Capture provider availability, runtime metadata, suppression, risk, coverage and scan time in an immutable presentation snapshot. Render alternate formats from isolated copies without repeating credential probes, deadline checks or scan finalization; reject unprepared results instead of reassessing them.

Reuse a completed root snapshot when no transitive work is added, but prepare a fresh snapshot for merged scans. Retain the finalized report completeness and existing destination, baseline and no-follow protections.

Add regressions for changing availability, all format orders, nested mutations, recursive probe counts, transitive snapshot lifecycle and non-presentation workflow objects.

Signed-off-by: Andreas Conrad <andreas.conrad@ruv.de>
Merge origin/main at 48e0f94 and resolve report formatter conflicts by retaining immutable assessment snapshots and the captured scan timestamp. Sanitize upstream input-origin metadata once during report preparation and freeze it for all additional formats without repeating provider checks, finalization or scoring.

Keep acquisition origin separate from materialized input and finding identities. Add single/transitive regressions across all primary formats for redaction, one-time assessment and sanitization, consistent origin metadata and mutation-isolated rerendering.

Retain upstream parser, fixture and dependency updates unchanged, including the public upstream lockfile. Validate with lint, formatting, 27,162 unit tests, 250 integration tests, and fresh-wheel ZIP/baseline preservation smoke tests; live-provider tests remain excluded.

Signed-off-by: Andreas Conrad <andreas.conrad@ruv.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generate multiple report formats from a single scan

3 participants