Fix/canonical subscription endpoints - #117
Merged
Merged
Conversation
…schemas
The repository ships 24 JSON Schemas under `schemas/v0.1/` and validated
against none of them where it mattered. `tests/` exercised them properly
via Ajv, but the *published CLI* — the tool a third party points at a
live deployment to certify it — had no JSON Schema validator at all:
`grep -rln ajv packages/` returned nothing. The manifest probe checked
five field names against a schema constraining 24 properties, and the
CloudEvents probe hand-listed five required attributes.
A deployment could therefore pass Full conformance while emitting
manifests and envelopes that violate the schemas this project publishes.
Separately, `--fixtures` — promised in ROADMAP.md as the way to run the
released vectors "without booting a live publisher" — did not exist.
The vectors were complete, indexed and packaged by `npm run
fixtures:pack`, but only this repo's own vitest and pytest suites could
execute them. A downstream implementor could not.
Changes:
- `src/schemas.ts`: Ajv registry loading every schema by `$id` so
cross-file `$ref`s resolve. Lifted from the wiring already proven in
`tests/conformance-fixtures.test.ts`. Returns null rather than throwing
when schemas are absent, so probes SKIP with a stated reason — an
unvalidated run must never look like a clean one.
- `scripts/bundle-schemas.mjs`: copy `schemas/v0.1/*.json` into
`dist/schemas/` at build time. The package ships only `dist`, so
without this the CLI degrades to "schemas not found" for everyone who
installs it from npm. Copying at build keeps `schemas/v0.1/` the single
source of truth — the bundle cannot drift.
- `src/fixtures.ts` + `--fixtures [dir]`: replay the published vectors
offline. No network, no API key, no port bound. Same reports and exit
codes as the live path.
- Probes for normative MUSTs that were previously unchecked:
- §4.3 `Last-Event-ID` replay — reconnects with a real event id and
asserts events strictly after it. Replay is EEP's headline
reliability claim over plain webhooks and conformance verified only
that the endpoint returned the right Content-Type.
- §4.4 15-second heartbeat.
- §5.3 timestamp freshness — the publisher-side half of the replay
window, which is what makes the receiver-side MUST enforceable.
- Whole-document validation of the live manifest and delivered envelope,
alongside the existing field-level probes (kept because naming the
exact missing field reads better in a report than a schema error path).
- CI runs the built CLI in `--fixtures` mode, so a break in the shipped
entry point fails even when the unit tests pass.
`ajv` and `ajv-formats` become the package's first runtime dependencies.
Refs: EEP audit 2026-08 findings A5, A7, O5
Signed-off-by: Ugur Cekmez <ucekmez@gmail.com>
feat(compliance-cli): validate live publishers against the normative schemas
There was a problem hiding this comment.
🟡 Changes recommended
The CLI docs/help for --fixtures don’t match the actual parsing behavior, and the live runner’s process.exit(...) prevents the intended shutdown path from running.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR strengthens the compliance CLI by making its conformance checks align with the repo’s normative JSON Schemas and by adding a fully-offline fixture replay mode so downstream implementors can validate against released vectors without running a live publisher.
Changes:
- Add an Ajv-backed schema registry and use it to validate live manifests (
eep-manifest.json) and event envelopes (event.envelope.json) instead of spot-checking a few fields. - Add
--fixturesoffline mode to replaytests/conformance-fixtures-style vector bundles (including signature vectors) and emit the same reports/exit codes as live runs. - Expand live probes with additional normative checks (webhook timestamp freshness, SSE heartbeat, SSE
Last-Event-IDreplay) and exercise the built CLI in CI.
File summaries
| File | Description |
|---|---|
| packages/@eep-dev/compliance-cli/src/schemas.ts | New Ajv schema registry to load and validate against bundled normative schemas. |
| packages/@eep-dev/compliance-cli/src/schemas.test.ts | Tests for schema discovery/loading and schema-based manifest/envelope validation behavior. |
| packages/@eep-dev/compliance-cli/src/index.ts | Adds schema validation in live mode, adds --fixtures offline mode, and expands webhook/SSE probes. |
| packages/@eep-dev/compliance-cli/src/fixtures.ts | New offline fixture runner that replays released conformance vectors and validates/signature-checks them. |
| packages/@eep-dev/compliance-cli/src/fixtures.test.ts | Tests that fixture replay runs cleanly and covers expected categories/vectors. |
| packages/@eep-dev/compliance-cli/scripts/bundle-schemas.mjs | Build-time bundling of schemas/v0.1 into the published dist/schemas/. |
| packages/@eep-dev/compliance-cli/README.md | Documents offline mode and schema validation; updates CLI options and conformance-level descriptions. |
| packages/@eep-dev/compliance-cli/package.json | Bundles schemas in build and adds Ajv dependencies. |
| packages/@eep-dev/compliance-cli/package-lock.json | Lockfile updates for Ajv dependencies. |
| .github/workflows/test.yml | CI step to run the built CLI in offline --fixtures mode. |
Review details
Files not reviewed (1)
- packages/@eep-dev/compliance-cli/package-lock.json: Generated file
- Files reviewed: 9/10 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+63
to
+66
| --fixtures [dir] Run the offline conformance vectors instead of | ||
| probing a live target. No network, no API key. | ||
| Defaults to ./tests/conformance-fixtures. | ||
| --schemas <dir> Override the schemas/v0.1 directory used for |
Comment on lines
+1049
to
+1051
| await writeReports(); | ||
|
|
||
| process.exit(failed > 0 ? 1 : 0); |
| | `--report-json` | — | `string` | — | Write machine-readable audit report JSON | | ||
| | `--report-md` | — | `string` | — | Write human-readable audit report markdown | | ||
| | `--report-html` | — | `string` | — | Write self-contained HTML audit report | | ||
| | `--fixtures` | — | `string` | `./tests/conformance-fixtures` | Replay the offline conformance vectors instead of probing a live target. No network, no API key. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scope
Checklist
npm test/pytestin affected packages).Notes for reviewers