This repository is the frozen, reviewer-oriented artifact for the paper Evidence-Calibrated Runtime Reconstruction for Agent Skills Across Heterogeneous Coding Agents. It contains the runnable methods, de-identified inputs, privacy-safe result archive, analysis code, paper source, and integrity manifests needed to reproduce or audit the reported evidence.
The artifact deliberately preserves failed gates, incomplete model responses, pilot protocols, and inference boundaries. It does not treat model agreement as human ground truth, one environment as deployment-wide reliability, or a single SkillRun as causal evidence of Skill effectiveness.
Requirements: macOS or Linux and Python 3.9 or newer. The deterministic core has no runtime dependency beyond Python.
bash scripts/reproduce.shThis command:
- verifies all 556 published evidence artifacts against SHA-256;
- runs the curated portable dependency-free test suite for the frozen reviewer artifact;
- executes the 13-gate local reproducibility suite;
- writes a fresh report outside the frozen evidence directory.
Expected deterministic outcome: 12/12 correctness gates pass. The thirteenth transport gate is environment-sensitive and reports the observed mechanism.
| Path | Purpose |
|---|---|
experiments/ |
Deterministic, live-Agent, model-backed, transport, migration, and external-validity protocols |
reproducibility/results/ |
Privacy-safe copies of all retained JSON/JSONL reports |
reproducibility/manifest.json |
Source and published digest, byte count, and path-redaction count for every result |
reproducibility/CHECKSUMS.sha256 |
Flat integrity ledger |
reproducibility/environments/ |
PAI-DSW software, hardware, and model-serving fingerprint |
paper/ |
LNCS manuscript source, references, analysis, and claims–evidence matrix |
src/ |
Runtime reconstruction and deterministic diagnosis implementation exercised by the experiments |
tests/ |
Unit and contract tests for the artifact surface |
The authoritative experiment inventory is
experiments/confirmatory_manifest_20260801.json. Cross-repository evaluation
is frozen in experiments/external_validity/confirmatory_manifest_20260801.json.
Both manifests record planned/completed counts, SHA-256, failed gates, retained
pilots, and the meaning of each gate.
Important retained limitations include:
- the cross-repository fixtures are controlled, de-identified three-file profiles rather than a prevalence sample of production failures;
- the Qwen matrix completed 378/378 requested calls, while the retained DeepSeek comparison completed 228/378, with timeout and invalid-output counts reported rather than discarded;
- blinded model annotation is not a human-gold diagnostic-utility study;
- transport and native-sender measurements characterize the recorded hosts and mechanisms, not every deployment environment;
- model weights and proprietary Agent executables are not redistributed.
- two original-repository history checks (historical commit migration and the v0.1.6 tag rebuild identity) are represented by their frozen reports and manifests rather than copied Git history; the working repository retains the executable checks.
Frozen model outputs can be audited without a GPU. Re-execution requires an
OpenAI-compatible endpoint and the named model; see
MODEL-REPRODUCTION.md. Exact numerical identity is
not promised for nondeterministic model generation. Deterministic scripts,
inputs, protocol digests, response counts, and frozen outputs are provided so
new runs can be compared explicitly.
No raw Agent session, user prompt, credential, private source repository, host
address, or model weight is included. Known machine-local path prefixes are
replaced only in published result copies. Metrics, labels, predictions,
timings, and model responses are unchanged. See DATA.md.
Code is provided under Apache-2.0. Cite the software metadata in CITATION.cff;
the paper citation should be updated after publication.