Skip to content

feat: add test-bundle and test-compare evidence workflows - #33

Merged
HaoZeke merged 1 commit into
mainfrom
feat/test-bundle
Nov 17, 2026
Merged

HaoZeke merged 1 commit into
mainfrom
feat/test-bundle

Conversation

@HaoZeke

@HaoZeke HaoZeke commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

What

  • Added test-bundle to eb-keep-logs (Zig build) to emit schema-versioned flaky-test evidence bundles under a stable --test-id.
  • Added test-compare. It lists fields that differ across bundles and does not decide pass or fail.
  • Two JSON schemas are new:
    • docs/orgmode/reference/json/test-bundle.schema.json
    • docs/orgmode/reference/json/test-compare.schema.json
  • Added tests for new commands and schema validation:
    • tests/test_test_bundle.sh
    • extensions in tests/test_json_schemas.sh
  • Added a committed output-stability regression test for existing command behavior:
    • tests/test_output_stability.sh
  • Updated command help/docs (Diataxis):
    • reference CLI and JSON schema docs
    • new how-to: compare-flaky-test-bundles.org

Why

Gap 9 requires comparable cross-site flaky-test evidence bundles with:

  • normalized stdout/stderr
  • input and seed hashes
  • full module list and module paths
  • European Environment for Scientific Software Installations (EESSI) target
  • scheduler allocation and binding
  • CPU and GPU facts
  • test-suite revision

It also requires comparison output that reports differences only, without pass/fail judgment.

How to try

# Build and test
zig build test --summary all

# Create bundles
eb-keep-logs test-bundle \
  --test-id qe.pw.scf-si \
  --suite-revision 5f6a9cc \
  --eessi-target x86_64/intel/sapphirerapids \
  --stdout run-a.stdout \
  --stderr run-a.stderr \
  --input qe.in \
  --seed 42 \
  --module QuantumESPRESSO/7.4=/cvmfs/eessi/software/QuantumESPRESSO/7.4.lua \
  --allocation "nodes=1 ntasks=32" \
  --binding "rank0->core0, rank1->core1" \
  --cpu "AMD EPYC 7B13 (32)" \
  --gpu "none" \
  --out run-a.bundle.json

eb-keep-logs test-bundle ... --out run-b.bundle.json

# Compare bundles (text)
eb-keep-logs test-compare run-a.bundle.json run-b.bundle.json

# Compare bundles (JSON)
eb-keep-logs test-compare --json run-a.bundle.json run-b.bundle.json > compare.json

@HaoZeke
HaoZeke merged commit 41087d3 into main Nov 17, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant