diff --git a/.claude/skills/build-plugin/SKILL.md b/.claude/skills/build-plugin/SKILL.md index d9e34a22..6a8f0050 100644 --- a/.claude/skills/build-plugin/SKILL.md +++ b/.claude/skills/build-plugin/SKILL.md @@ -21,7 +21,9 @@ This skill **tests every data stream against a live, authenticated plugin** in y 1. Run `squaredup status --json`. If the command isn't found, install the CLI first: `npm i -g @squaredup/cli`. If it exits non-zero, the user is not logged in — ask them to run `! squaredup login` in this session, then re-run `squaredup status --json`. Capture the JSON output (`{ tenantName, region }`) — you'll need `region` in Checkpoint A. Login/region mechanics live in the `deploy-plugin` skill. 2. Confirm the user has a SquaredUp tenant where they can add and authenticate the plugin. -Checkpoint B drives the import with `squaredup index` / `index-status`, so a current `squaredup` CLI is assumed. +Checkpoint B drives the import with `squaredup index` / `index-status`, and confirms relationships with `squaredup correlate-status` / `edges`, so a current `squaredup` CLI is assumed — the correlation commands need **1.1.0 or later**. If `squaredup edges --help` reports an unknown command, the CLI is too old: `npm i -g @squaredup/cli`. + +One of them needs more than a login: `squaredup correlate` re-runs rules on demand and requires a **tenant admin**, returning "requires a tenant admin" otherwise. `correlate-status` and `edges` are reads and need no such permission, so a non-admin can still confirm everything — they just can't re-run rules without a fresh import. If login or a tenant is unavailable, **stop** — this skill cannot build a plugin it cannot test. @@ -288,7 +290,7 @@ Author them inline in the main agent (they're small, and they depend on the impo 3. **Write the rule** — source is the side holding the foreign key; always write both `forward` and `reverse` labels; `"operator": "equals"` on every condition. Never write `pluginId`, `ruleType`, `schemaVersion`, or condition `id`s — those are stamped for you. 4. **Validate** — run `squaredup validate --json` from the plugin dir and confirm the summary's `Correlation Rules` count equals the number of files you wrote. An invalid rule fails the whole plugin validation, so fix any error before Checkpoint B. -Edges can't be confirmed until Checkpoint B's import has run — verification is a step there. +`validate` only proves the files are well-formed — it says nothing about whether they installed or matched anything. Both are confirmed after Checkpoint B's import, with `squaredup correlate-status` (did the rules install and run?) and `squaredup edges` (did they relate anything?). Verification is a step there. --- @@ -297,10 +299,16 @@ Edges can't be confirmed until Checkpoint B's import has run — verification is Scoped data streams can't be tested until objects exist, which means the import steps must be live and an import must have run. The CLI triggers and tracks the import for you, so **drive it yourself — don't ask the user to run it in the UI.** 1. **Redeploy** — invoke `deploy-plugin` again so the new import steps ship. The import definitions only take effect once this redeploy lands, so the import must run _after_ it. -2. **Trigger** — `squaredup index --datasource-id --no-wait --json`. `--no-wait` returns immediately with a `since` anchor (capture it) instead of blocking until the import finishes — you poll for completion in the next step. (Plain `squaredup index` now waits and prints progress itself, which can outlast an agent command timeout on a long import; `--no-wait` is the orchestration path.) If an import was already running it reports `alreadyRunning: true` and adopts that run — poll with the `since` it returns either way. +2. **Trigger** — `squaredup index --datasource-id --no-wait --json`. `--no-wait` returns immediately with a `since` anchor (capture it) instead of blocking until the import finishes — you poll for completion in the next step. (Plain `squaredup index` now waits and prints progress itself, which can outlast an agent command timeout on a long import; `--no-wait` is the orchestration path.) If an import was already running it reports `alreadyRunning: true` and adopts that run — poll with the `since` it returns either way. If Phase 5b shipped correlation rules, capture the **correlation** anchor before you trigger — see step 5. 3. **Wait** — poll `squaredup index-status --datasource-id --since --json` until `done` is `true`, passing the `since` from step 2. `succeeded: true` means objects are indexed; `succeeded: false` means the import failed — read the run-level `message` and the per-step `steps[]` (which step has `status: "failed"` and its `errorReason`) to pinpoint the break, fix that import stream, and re-trigger before continuing. Imports can take several minutes; use a generous timeout. See [checkpoints.md](references/checkpoints.md). 4. **Confirm** — check objects landed with an **inline scope**: `squaredup objects --matches '{"sourceType":{"type":"equals","value":""}}' --plugin-id --datasource-id --json` should return a non-empty list. `` is a `sourceType` from the `objectTypes` you defined in `metadata.json` / `indexDefinitions/default.json`. Use `--matches` here, **not** `objects `: that form resolves a data stream file's `matches`, but no scoped data stream exists yet (those come in Phase 6) and the import streams written so far have no `matches` to resolve. For the same reason, pass **inline** JSON — `--matches @.json` won't work, as an import stream's `matches` is `none`/absent. -5. **Confirm relationships** — only if Phase 5b shipped correlation rules. Correlation is triggered automatically once the import succeeds, but fire-and-forget: it is _not_ part of the import status, so edges appear shortly **after** `index-status` reports done. The CLI has no edge-query command, so confirm in the tenant — ask the user to open an object of the source type and check its relationships — or use the SquaredUp MCP server's `graph_query` if one is connected. Zero edges from a rule that validated almost always means an unmapped join key or a `types` string that doesn't match `objectTypes`; see the common-mistakes table in [correlation-rules.md](references/correlation-rules.md). +5. **Confirm relationships** — only if Phase 5b shipped correlation rules. Correlation is triggered automatically once the import succeeds, but fire-and-forget: it is _not_ part of the import status, so edges appear shortly **after** `index-status` reports done. Drive the confirmation yourself: + - **Anchor first, before step 2's trigger.** `squaredup correlate-status --datasource-id --json` and keep the highest `rules[].lastEvaluated` (`0` if the rules have never run). `--since ` is what makes `done` mean *this* run — without it, `done: true` can be the previous import's correlation, so a rule that matched last time reads as a pass before this run has even started. Don't reuse the `since` from step 2: that is the previous import's start, which the previous correlation already cleared. + - `squaredup correlate-status --datasource-id --since --json` — lists every rule this data source has installed, with its last-run outcome and `edgesCreated`. **The rule list is itself the check that your rules installed**: a rule you wrote that isn't listed never deployed. Poll until `done` is `true`, every ~5s, and **stop after ~5 minutes**. + - **On failure, stop here.** If the poll times out, or `succeeded` is `false`, or any rule reports `status: "failed"`, don't read `edges` — report the run-level outcome plus that rule's `ruleName` and `errorCount`, and fix it first. A timeout isn't a failure (the run continues server-side) but it isn't a confirmation either: say which rules were still in flight. + - `squaredup edges --datasource-id --plugin-id --json` — the edges those rules actually produced. `--rule ` narrows to one rule. + + A rule reporting `edgesCreated: 0` names which rule to investigate — usually an unmapped join key or a `types` string that doesn't match `objectTypes`. Zero is the signal to look, not proof of a bug: a correct rule reports zero against data that holds nothing to relate, so check the join-key values actually pair up on both sides (`squaredup objects`) before rewriting the rule. See the common-mistakes table in [correlation-rules.md](references/correlation-rules.md), which also covers the fix loop. ### Re-indexing rule diff --git a/.claude/skills/build-plugin/references/checkpoints.md b/.claude/skills/build-plugin/references/checkpoints.md index af67d342..bbc5de5b 100644 --- a/.claude/skills/build-plugin/references/checkpoints.md +++ b/.claude/skills/build-plugin/references/checkpoints.md @@ -60,9 +60,44 @@ Notes: - **`status` is the run's lifecycle, not the outcome.** While running it's `ready`/`inProgress`; once `done` it's one of `succeeded`, `failed`, `warning`, or `cancelled`. `succeeded` and `warning` both report `succeeded: true` (a `warning` run finished but a step emitted warnings — check the `steps[]`); `failed` and `cancelled` report `succeeded: false`. A datasource that has never imported reports `status: "notRun"`, `done: false`. - `--since` is **exclusive** (`scheduledStart > since`): always pass the `since` from `index` so `done` can't latch on a stale previous run. - If `index` reports `alreadyRunning: true`, it adopted the in-flight run — poll with the `since` it returned. Imports can take several minutes (object import allows up to ~10 min); use a generous overall timeout. -- **Correlation runs after the import, not as part of it.** If the plugin ships `correlationRules/*.json`, a successful import triggers correlation fire-and-forget — `done: true, succeeded: true` says nothing about whether edges were written, and they land shortly afterwards. There is no CLI command for edges; confirm relationships in the tenant UI. See [correlation-rules.md](correlation-rules.md). +- **Correlation runs after the import, not as part of it.** If the plugin ships `correlationRules/*.json`, a successful import triggers correlation fire-and-forget — `done: true, succeeded: true` says nothing about whether edges were written, and they land shortly afterwards. Confirm them with the two commands below rather than in the UI. - The `--matches` confirm in step 3 must be **inline JSON** — `--matches @.json` only resolves a real scope, and an import stream's `matches` is `none`/absent. Likewise `objects ` needs a scoped stream, which doesn't exist until Phase 6. +## Checkpoint B, step 5: confirm relationships + +Only if the plugin ships correlation rules. Correlation lands after the import, so poll for it — with an anchor, the same way you poll `index-status`: + +```bash +# 0. BEFORE triggering the import (step 2): read the anchor — the newest lastEvaluated across the rules. +squaredup correlate-status --datasource-id --json --silent +# → { "rules": [ { "ruleName": "relate-pod-to-node", "lastEvaluated": 1755500000000, ... } ] } +# anchor = max(rules[].lastEvaluated), or 0 if the rules have never run. + +# 1. After index-status reports done: poll until THIS run's correlation is done. +squaredup correlate-status --datasource-id --since --json --silent +# → { "done": true, "succeeded": true, +# "rules": [ { "ruleName": "relate-pod-to-node", "status": "succeeded", +# "done": true, "succeeded": true, "edgesCreated": 42, +# "verticesProcessed": 50, "errorCount": 0, "limitExceeded": false, +# "lastEvaluated": 1755500123456 } ] } + +# 2. The edges themselves. --rule narrows to one rule. +squaredup edges --datasource-id --plugin-id --json --silent +# → { "edges": [ { "source": {...}, "target": {...}, "label": "runs on", "origin": "plugin" } ], "truncated": false } +``` + +Notes: + +- **`--since` is what makes `done` mean *this* run.** It's an exclusive anchor on `lastEvaluated`, exactly like `index-status --since` on `scheduledStart`. Omit it and `done: true` can be the *previous* import's correlation — a rule that ran and matched last time reads as a pass before this run has even started. Don't reuse the `since` from `index`: that is the previous import's start time, which the previous correlation run already cleared. Take the anchor from a pre-import `correlate-status`, or from `squaredup correlate --no-wait --json`, which prints one. +- **Bound the poll.** Poll every ~5s and give up after ~5 minutes (the CLI's own `correlate` polls at 5s and waits 600s by default). A timeout is **not** a failure — the run continues server-side — but it is also not a confirmation: report the relationships as unconfirmed and say which rules were still in flight, rather than reading `edges` and treating whatever is there as the result. +- **Stop before `edges` on a failure.** If `succeeded` is `false`, or any rule reports `status: "failed"`, report the run-level outcome plus the offending rule's `ruleName`, `status` and `errorCount`, and fix that before reading edges. `succeeded` is `null` while any rule is still in flight; a rule that has never been evaluated reports `status: "notRun"`, `done: false`. `status: "warnings"` counts as succeeded — the rule created edges but logged errors, so check `errorCount`. +- **The `rules` list is how you confirm the rules installed.** A `correlationRules/*.json` you wrote that doesn't appear never deployed. `validate` only proves the file parses. +- **Read per rule, not just the top-level flags.** `succeeded: true` means every rule ran, not that any matched — a rule with `edgesCreated: 0` and a healthy `verticesProcessed` ran fine and matched nothing. That is usually a rule bug, but a correct rule reports zero on data that holds nothing to relate, so check the join-key values actually pair up on both sides (`squaredup objects`) before rewriting it. See the common-mistakes table in [correlation-rules.md](correlation-rules.md). +- `limitExceeded: true` means the rule stopped at the tenant edge limit, so `edgesCreated` is a floor rather than the full match count. +- `edges` needs `--plugin-id` as well as `--datasource-id`; `correlate-status` needs only the datasource. +- **An empty `edges` result is exit 0, not an error** — poll it the same way you poll `index-status`, rather than treating the first empty read as failure. +- To re-run rules without a fresh import (the authoring loop after fixing a rule), use `squaredup correlate` — **tenant admin only**. Without admin, trigger another import instead. See [correlation-rules.md](correlation-rules.md). + ## Imported objects are frozen at import time — re-index to refresh them The objects this import created carry the shape defined by the `indexDefinitions/*.json` and import streams **as they were when the import ran**. Editing either afterwards does **not** retroactively change the objects already in the graph — every existing object stays **stale**, and a newly mapped property is absent on it until the datasource is re-imported. This is the trap behind the shipped `undefined === undefined` scope bug: a property was added to `objectMapping.properties` *after* the import, sub-agents were told it existed, and they filtered on a field that was `undefined` on every object. diff --git a/.claude/skills/build-plugin/references/correlation-rules.md b/.claude/skills/build-plugin/references/correlation-rules.md index e0f83f28..9c715658 100644 --- a/.claude/skills/build-plugin/references/correlation-rules.md +++ b/.claude/skills/build-plugin/references/correlation-rules.md @@ -244,7 +244,63 @@ Matching semantics: So the sequence to see edges is always: deploy the rules → run an import → wait a moment. A rule added after the last import produces nothing until the next import completes. -**Verifying:** the CLI has no edge-query command. Confirm rules deployed via the `Correlation Rules: N` count from `squaredup validate`, and confirm edges via the tenant UI — open an object and check its relationships/graph view — or with the SquaredUp MCP server's `graph_query` tool if one is connected. +A rule added after the last import can also be evaluated on demand — see the iteration loop below — which is what makes authoring rules bearable: you do not need a fresh import for every fix. + +--- + +## Verifying and iterating with the CLI + +Three commands cover the whole loop. All take `--json`, and `--datasource-id` makes them folder-independent. + +| Command | Answers | +| --- | --- | +| `squaredup correlate-status --datasource-id --json` | Which rules are installed for this data source, and what did each last run do? `--since ` scopes `done` to a run newer than that anchor. | +| `squaredup edges --datasource-id --plugin-id --json` | Which edges exist on this data source's objects? `--rule ` narrows to one rule; `--object ` to one object. | +| `squaredup correlate --datasource-id --json` | Re-run the rules now and wait (5s polls, `--timeout` seconds, default 600). `--rule ` runs one; `--no-wait` returns immediately with the `since` anchor to poll on. **Tenant admin only.** | + +**Confirming the rules installed.** `correlate-status` lists one entry per rule the data source has, keyed by `ruleName` — the filename you chose. Compare that list against your `correlationRules/` directory: a file that isn't listed never installed, which usually means the deploy that shipped it hasn't landed. This is a stronger check than `validate`, which only proves the file parses. + +```jsonc +{ + "done": true, + "succeeded": true, + "rules": [ + { "ruleName": "relate-pod-to-node", "status": "succeeded", "done": true, "succeeded": true, + "edgesCreated": 42, "verticesProcessed": 50, "errorCount": 0, "limitExceeded": false, + "lastEvaluated": 1755500123456 }, + { "ruleName": "relate-pod-to-namespace", "status": "succeeded", "done": true, "succeeded": true, + "edgesCreated": 0, "verticesProcessed": 50, "errorCount": 0, "limitExceeded": false, + "lastEvaluated": 1755500123456 } + ] +} +``` + +Read it per rule, not just the top-level flags. `succeeded: true` means every rule *ran*, not that any of them matched: `relate-pod-to-namespace` above ran cleanly and produced nothing. `edgesCreated: 0` alongside a healthy `verticesProcessed` is the classic signature of a join key that doesn't match — take that rule to the common-mistakes table below. It is not *proof* of a bug, though: a correct rule reports zero against data that holds nothing to relate. Before rewriting the rule, confirm the tenant actually has a matching pair — read the join-key property on both sides with `squaredup objects` and check the values pair up. Only rewrite once you've seen a pair the rule should have joined and didn't. + +**Anchor the poll, and branch on failure.** `done` without `--since` means "every rule has a finished status", which a *previous* run satisfies — so a rule that matched last import reads as a pass before this run has even started. `--since` is exclusive on `lastEvaluated`, exactly like `index-status --since` on `scheduledStart`: `correlate --no-wait --json` prints the anchor, or read the highest `rules[].lastEvaluated` yourself before triggering. Poll every ~5s and cap the wait; a timeout is not a failure (the run continues server-side) but it is not a confirmation either. `succeeded: false`, or any rule with `status: "failed"`, means stop and read that rule's `errorCount` rather than going on to `edges`. `status: "warnings"` counts as succeeded — edges were created but errors were logged. `limitExceeded: true` means the rule stopped at the tenant edge limit, so `edgesCreated` is a floor. + +**The iteration loop.** Correlation normally only runs after an import, but `correlate` re-runs it on demand, so fixing a rule costs a redeploy rather than a full re-index: + +1. Fix the rule file. +2. Redeploy (invoke `deploy-plugin`) — rules ship with the plugin, so the fix isn't live until it lands. +3. `squaredup correlate --datasource-id --rule --json` — re-runs just that rule and waits for it, reporting `edgesCreated`. +4. `squaredup edges --datasource-id --plugin-id --rule --json` — look at the edges themselves, not just the count. + +Only step 2 is slow. **This loop needs no re-index as long as the imported objects themselves are unchanged**: they are already in the graph, and correlation re-reads them. The moment the fix touches the *shape* of those objects — any edit to `indexDefinitions/*.json` or an import stream, whether that's adding a join key to `objectMapping.properties`, changing an existing mapping, or changing `id`/`name`/`type` — the objects in the graph are stale and correlation is reading the old values. Then the [re-indexing rule](../SKILL.md#re-indexing-rule) applies and you owe the full Checkpoint B cycle before trusting any correlation result. + +**Reading `edges`.** Each edge names both ends, so you can see whether the rule joined what you intended and in which direction: + +```jsonc +{ "source": { "id": "node-...", "name": "web-01", "sourceType": "Pod" }, + "target": { "id": "node-...", "name": "ip-10-0-1-4", "sourceType": "Node" }, + "label": "runs on", "reverseLabel": "hosts", "origin": "plugin", "configId": "config-..." } +``` + +- Endpoints reading backwards means source and target are swapped in the rule. +- An empty list while `correlate-status` reports `edgesCreated > 0` for that rule means the two are looking at different things — most often a `--datasource-id` that isn't the one the rule is scoped to, or a `truncated: true` result you read past. +- `origin: "plugin"` confirms the edge came from a plugin rule rather than a user-authored or platform one. + +**If you aren't a tenant admin**, `correlate` returns "requires a tenant admin". `correlate-status` and `edges` still work, so you can confirm everything — you just have to trigger a fresh import (Checkpoint B) to re-run rules instead of calling `correlate`. --- @@ -252,8 +308,8 @@ So the sequence to see edges is always: deploy the rules → run an import → w | Mistake | Symptom / fix | | ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | -| Join key not in `objectMapping.properties` | Rule validates, zero edges forever. Map the property, then **re-index** — existing objects are stale. | -| Rule written but no import run since | Zero edges. Trigger an import (Checkpoint B) after deploying rules. | +| Join key not in `objectMapping.properties` | `correlate-status` shows the rule ran with `edgesCreated: 0`. Map the property, then **re-index** — existing objects are stale. | +| Rule written but no import run since | The rule is absent from `correlate-status`, or present with `status: "notRun"`. Redeploy, then `squaredup correlate` (admin) or run an import. | | `types` doesn't match `objectTypes` exactly | Zero edges, no error. Copy the string from `metadata.json`. | | `operator` omitted | Validation failure — `equals` is required on every condition. | | `reverse` label omitted | The relationship reads identically in both directions. Always write both. | diff --git a/REVIEW.md b/REVIEW.md index 4780b8dd..b5a6e29a 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -107,6 +107,7 @@ When suggesting changes: - Multiple conditions are AND-ed by default. Only expect `conditionLogic` when the author genuinely needs OR/parentheses, and check every condition index it references exists. - Format expressions - `sourceExpression`/`targetExpression` run per candidate object on every correlation run. Where the upstream data allows, prefer mapping a clean join key in the import over transforming one here. - Scope - Never accept `pluginId`, `configs`, `ruleType`, `schemaVersion`, `origin`, or condition `id`s in the file; all are stamped at deploy time. A plugin rule can only relate its own object types. +- Evidence - A rule that validates can still match nothing, and nothing in CI catches that. Expect the author to say they confirmed edges against a real data source (`squaredup correlate-status` shows each rule's `edgesCreated`, `squaredup edges --rule ` shows the edges themselves). Treat a new rule with no such evidence as unverified. ### Documentation - (docs/README.md)