Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,13 @@ jobs:
- run: npm ci
- run: npm run build
- run: npx vitest run --coverage
# Exercise the published entry point in offline mode. This is what a
# downstream implementor runs against the released vector tarball, so
# a break here means the shipped CLI is broken even though the unit
# tests pass.
- name: Replay conformance vectors through the built CLI (--fixtures)
working-directory: .
run: node packages/@eep-dev/compliance-cli/dist/index.js --fixtures ./tests/conformance-fixtures

test-discovery:
runs-on: ubuntu-latest
Expand Down
33 changes: 30 additions & 3 deletions packages/@eep-dev/compliance-cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,35 @@ The CLI runs tests across **three conformance levels**:

| Level | Tests | What's Verified |
|-------|-------|----------------|
| 🥉 **Core** | Platform reachability, EEP discovery (Link headers), subscription creation, WebSub intent verification, webhook delivery, Standard Webhooks headers, HMAC-SHA256 signature, CloudEvents envelope | Signal stream basics |
| 🥈 **Standard** | All Core tests + SSE stream endpoint, rate limit headers | SSE + rate limiting |
| 🥉 **Core** | Platform reachability, EEP discovery (Link headers), subscription creation, WebSub intent verification, webhook delivery, Standard Webhooks headers, HMAC-SHA256 signature, **timestamp freshness (§5.3)**, CloudEvents envelope **schema-validated** | Signal stream basics |
| 🥈 **Standard** | All Core tests + SSE stream endpoint, **SSE heartbeat (§4.4)**, **`Last-Event-ID` replay (§4.3)**, rate limit headers | SSE + rate limiting |
| 🏆 **Full** | All Standard tests + manifest/policy probes + extended probes (Layer 1 content negotiation, 402 payment gate, WebSocket pulse, CloudEvents/EEP helper validation) | Advanced baseline checks (partial full-tier automation) |

### Offline mode (`--fixtures`)

Every EEP release ships `eep-conformance-vectors-vX.Y.Z.tar.gz`: bytes-on-the-wire
test vectors for discovery documents, event envelopes, signatures, gate responses
and subscription requests. `--fixtures` replays them with **no publisher, no
network and no API key**, so you can check a new implementation before you have
anything deployed:

```bash
tar xzf eep-conformance-vectors-v0.1.0.tar.gz
npx @eep-dev/compliance-cli --fixtures ./conformance-fixtures
```

Both modes write the same `--report-json` / `--report-md` / `--report-html`
artifacts and use the same exit codes.

### JSON Schema validation

Both modes validate documents against the normative schemas in `schemas/v0.1/`,
which are bundled into the published package at build time. Live manifests are
checked against `eep-manifest.json` in full and delivered events against
`event.envelope.json`, rather than spot-checking a handful of attribute names.
If the schemas cannot be located, the affected probes **skip with a stated
reason** — an unvalidated run never reports as a clean one.

> `--level full` currently performs **partial** full-tier automation. WebSocket commerce state machine, PoI cryptographic verification, and some sector-specific checks still require manual/stack-specific validation.

---
Expand Down Expand Up @@ -76,14 +101,16 @@ node --experimental-strip-types src/index.ts \

| Flag | Short | Type | Default | Description |
|------|-------|------|---------|-------------|
| `--target` | `-t` | `string` | — | **Required.** Platform base URL |
| `--target` | `-t` | `string` | — | Platform base URL. Required unless `--fixtures` is used. |
| `--api-key` | `-k` | `string` | — | API key for authenticated requests |
| `--entity` | `-e` | `string` | — | Entity DID or `{prefix}/{username}` to subscribe to |
| `--level` | `-l` | `string` | `standard` | Conformance level: `core`, `standard`, `full` |
| `--port` | `-p` | `string` | `9876` | Local port for the test webhook receiver |
| `--report-json` | — | `string` | — | Write machine-readable audit report JSON |
| `--report-md` | — | `string` | — | Write human-readable audit report markdown |
| `--report-html` | — | `string` | — | Write self-contained HTML audit report |
| `--fixtures` | — | `string` | `./tests/conformance-fixtures` | Replay the offline conformance vectors instead of probing a live target. No network, no API key. |
| `--schemas` | — | `string` | bundled | Override the `schemas/v0.1` directory used for JSON Schema validation |
| `--help` | `-h` | `boolean` | — | Show help message |

---
Expand Down
74 changes: 74 additions & 0 deletions packages/@eep-dev/compliance-cli/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion packages/@eep-dev/compliance-cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"CHANGELOG.md"
],
"scripts": {
"build": "tsc",
"build": "tsc && node scripts/bundle-schemas.mjs",
"prepublishOnly": "npm run build",
"start": "node dist/index.js",
"test": "vitest run"
Expand All @@ -50,5 +50,9 @@
"publishConfig": {
"access": "public",
"provenance": true
},
"dependencies": {
"ajv": "^8.17.1",
"ajv-formats": "^3.0.1"
}
}
47 changes: 47 additions & 0 deletions packages/@eep-dev/compliance-cli/scripts/bundle-schemas.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#!/usr/bin/env node
/**
* Copy `schemas/v0.1/*.json` into `dist/schemas/` at build time.
*
* The published npm package ships only `dist`, so without this step the
* CLI has no schemas to validate against once it is installed from the
* registry — it would silently degrade to "schemas not found" for every
* user who is not running from a repo checkout.
*
* Copying at build time (rather than committing a second copy) keeps
* `schemas/v0.1/` the single source of truth: the bundle cannot drift,
* because it is regenerated from the originals on every build and publish.
*/
import { readdirSync, readFileSync, writeFileSync, mkdirSync, statSync, existsSync } from 'node:fs';
import { join, resolve, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';

const HERE = dirname(fileURLToPath(import.meta.url));
const PACKAGE_ROOT = resolve(HERE, '..');
const SOURCE = resolve(PACKAGE_ROOT, '../../../schemas/v0.1');
const DEST = join(PACKAGE_ROOT, 'dist', 'schemas');

if (!existsSync(SOURCE)) {
console.error(`[bundle-schemas] source not found: ${SOURCE}`);
process.exit(1);
}

mkdirSync(DEST, { recursive: true });

let copied = 0;
for (const filename of readdirSync(SOURCE).sort()) {
if (!filename.endsWith('.json')) continue;
const from = join(SOURCE, filename);
if (!statSync(from).isFile()) continue;
const raw = readFileSync(from, 'utf8');
// Parse to fail loudly on a corrupt schema rather than shipping it.
JSON.parse(raw);
writeFileSync(join(DEST, filename), raw);
copied += 1;
}

if (copied === 0) {
console.error(`[bundle-schemas] no schemas found in ${SOURCE}`);
process.exit(1);
}

console.error(`[bundle-schemas] bundled ${copied} schemas into dist/schemas/`);
65 changes: 65 additions & 0 deletions packages/@eep-dev/compliance-cli/src/fixtures.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
import { describe, expect, it } from 'vitest';
import { resolve } from 'node:path';
import { findFixturesDir, runFixtures, type FixtureReporter } from './fixtures.js';

const REPO_FIXTURES = resolve(import.meta.dirname, '../../../../tests/conformance-fixtures');
const REPO_SCHEMAS = resolve(import.meta.dirname, '../../../../schemas/v0.1');

function collector() {
const passed: string[] = [];
const failed: Array<{ name: string; detail: string }> = [];
const skipped: Array<{ name: string; reason: string }> = [];
const reporter: FixtureReporter = {
pass: (name) => passed.push(name),
fail: (name, detail) => failed.push({ name, detail }),
skip: (name, reason) => skipped.push({ name, reason }),
};
return { reporter, passed, failed, skipped };
}

describe('offline fixture runner', () => {
it('resolves an explicit fixture directory', () => {
expect(findFixturesDir(REPO_FIXTURES)).toBe(REPO_FIXTURES);
});

it('returns null for a directory with no manifest', () => {
expect(findFixturesDir('/nonexistent/fixtures/path')).toBeNull();
});

// The point of `--fixtures`: a downstream implementor unpacks the
// released vector tarball and gets a verdict with no publisher running.
it('replays every published vector with no failures', () => {
const { reporter, passed, failed, skipped } = collector();
const { specVersion, total } = runFixtures(REPO_FIXTURES, reporter, REPO_SCHEMAS);

expect(specVersion).toBe('0.1');
expect(total).toBeGreaterThanOrEqual(17);
expect(failed).toEqual([]);
expect(skipped).toEqual([]);
expect(passed.length).toBe(total);
});

it('covers each fixture category', () => {
const { reporter, passed } = collector();
runFixtures(REPO_FIXTURES, reporter, REPO_SCHEMAS);
const joined = passed.join(' ');
for (const category of ['discovery', 'envelope', 'signature', 'gates', 'subscription']) {
expect(joined).toContain(`fixture:${category}`);
}
});

it('exercises the truncated-signature vector added for §5.3', () => {
const { reporter, passed } = collector();
runFixtures(REPO_FIXTURES, reporter, REPO_SCHEMAS);
expect(passed).toContain('fixture:signature-truncated-signature');
});

// Every reported outcome must carry a reason string; a bare pass/fail
// with no detail is what made the old runner hard to act on.
it('attaches a detail or reason to every reported outcome', () => {
const { reporter, failed, skipped } = collector();
runFixtures(REPO_FIXTURES, reporter, REPO_SCHEMAS);
expect(failed).toEqual([]);
expect(skipped.every((s) => typeof s.reason === 'string' && s.reason.length > 0)).toBe(true);
});
});
Loading
Loading