Fail the build when a schema that built on the last release stops building

.claude/skills/verify-tools/SKILL.md

@@ -28,8 +28,8 @@ and run another checkout's `probe.sh` from inside it.
   reads `~/.m2` but never writes it. The install runs `mvn clean install`, so it empties the checkout's `target/`. Don't
   run it while another build uses the same checkout. The work dir is keyed on the checkout's path, so clones and worktrees
   run in parallel, even when two share a folder name. If `doctor` shows the lock held by a live pid, wait for that run.
-- It builds the schema from the scenario's `parser()`, prints it, then executes every `*.graphql` in
-  `probe/src/main/resources/scenarios/<name>/` in name order, with `<op>.variables.json` as variables. It collects
+- It builds the schema from the scenario's files, resolvers and `configure`, prints it, then executes every `*.graphql` in
+  `src/test/resources/compat/scenarios/<name>/` in name order, with `<op>.variables.json` as variables. It collects
   subscription events until the stream completes, for at most 10 seconds.
 - Exit `0`: the schema built and every operation ran, though operations can still return GraphQL errors. `2`: the build
   threw. `1`: harness failure (install log or `maven.log`) or a held lock. `64`: bad usage, such as a missing or unknown
@@ -65,17 +65,21 @@ Proof standards:
 
 ## Adding a scenario
 
-1. Write the SDL as `probe/src/main/resources/scenarios/<name>/schema.graphqls` and load it with `.file(...)`, unless the
-   behavior is about `schemaString`.
-2. Add `object <Name> : Scenario` with `name` and `parser()` in `probe/src/main/kotlin/scenarios/<Name>.kt`. Being an
-   object in that package registers it. Classes share the package, so prefix resolver and data classes with the scenario.
-   Use only public API. If the probe needs `internal` or test classes, a user can't reach the behavior either, so report
-   that.
+Scenarios live in the library's tests, so `mvn verify` compiles them and `CompatTest` builds each one, plus invalid
+variants of it, against the last release's results.
+
+1. Write the SDL as `src/test/resources/compat/scenarios/<name>/schema.graphqls`.
+2. Add `object <Name> : Scenario` with `name`, `files` and `resolvers` in
+   `src/test/kotlin/graphql/kickstart/tools/compat/scenarios/<Name>.kt`, and override `configure(builder)` for scalars,
+   dictionary, directives or options. Being an object in that package registers it. Classes share the package, so prefix
+   resolver and data classes with the scenario. Use only public API and nothing from other test packages, since the probe
+   compiles this directory alone. If the probe needs `internal` or test classes, a user can't reach the behavior either,
+   so report that.
 3. Add operations as `NN-<what>.graphql`, plus `NN-<what>.variables.json` when they take variables. graphql-java rejects
    a second `__type` in one operation and still exits `0`, so read several types through `__schema { types { ... } }`.
 4. Override what the scenario needs. `operations` reuses another scenario's operation directory. `schema(parser)` takes
    another build path such as `toReadOnlySchema()`. `inspect(schema)` returns evidence the printer doesn't show, and
-   `probe.dumpDirectives(schema)` lists every element's applied and legacy directives. `graphQL(builder)` and
+   `dumpDirectives(schema)` lists every element's applied and legacy directives. `graphQL(builder)` and
    `input(builder)` add instrumentation, context or data loaders.
 5. A scenario that doesn't compile breaks every checkout's run, so finish it before running. Keep a scenario that proves
    a lasting feature and add its exact expected results to the matching file in `features/`. Delete a one-off

.claude/skills/verify-tools/features/README.md

@@ -32,12 +32,12 @@ that waste or invalidate a run, and how the result looked before the fix).
 - [Schema parser options](./schema-options.md): `missingResolverDataFetcherProvider`
   (`options`).
 - [Type extensions](./type-extensions.md): fields, interfaces, members, values and directives of every kind of extension,
-  the repeat and kind-mismatch warnings, repeated object and input fields, and sealed subclasses found through
+  the repeat errors and the kind-mismatch warning, repeated object and input fields, and sealed subclasses found through
   extensions (`type-extensions`, `type-extensions-repeated-fields`, `type-extensions-sealed`).
-- [Proxied resolvers](./proxies.md): Spring JDK dynamic and CGLIB proxies, and the warning for fields a JDK proxy can't
+- [Proxied resolvers](./proxies.md): Spring JDK dynamic and CGLIB proxies, and the build error for fields a JDK proxy can't
   call (`jdk-proxy`, `jdk-proxy-cglib`, `jdk-proxy-plain`).
 - [Context and environment parameters](./context-and-environment.md): `contextClass`, `GraphQLContext` and
-  `DataFetchingEnvironment` after the schema arguments, in any order, the misbound-argument warning, and resolver methods
+  `DataFetchingEnvironment` after the schema arguments, in any order, the misbound-argument error, and resolver methods
   without the source parameter (`context-and-environment`, `context-and-environment-invalid`, `context-misbound`,
   `sourceless`, `sourceless-shadowed`, `sourceless-unimplemented`, `sourceless-misbound`).
 

.claude/skills/verify-tools/features/arguments-and-inputs.md

@@ -44,10 +44,10 @@ from an explicit `null` (`Optional.empty()`).
   `Validation error (WrongType@[search])` and names `POETRY`.
 - **Unmapped input fields.** `run unmapped-inputs`. Exit `0`. `probe.log` has
   exactly one `UnmappedInputFieldChecker` WARN, starting `Input field 'ItemInput.name' has no matching property in
-  scenarios.ItemInput` and listing `Known properties: nmae`. There is no WARN for `InnerInput.legacy`.
-  `01-ignored.json` has `data.ignored = "a"`. `02-typo.json` has one error containing
-  `Unrecognized property "name" (class scenarios.ItemInput)` (`Unrecognized field` before Jackson 3, in 14.x), which shows
-  the reported field really fails.
+  graphql.kickstart.tools.compat.scenarios.ItemInput` and listing `Known properties: nmae`. There is no WARN for
+  `InnerInput.legacy`. `01-ignored.json` has `data.ignored = "a"`. `02-typo.json` has one error containing
+  `Unrecognized property "name" (class graphql.kickstart.tools.compat.scenarios.ItemInput)` (`Unrecognized field` before
+  Jackson 3, in 14.x), which shows the reported field really fails.
 - **Type variable parameters.** `run generic-input-resolver`. Exit `0`.
   `summary.txt` shows `errors=0` for both operations. `02-generic.json` has
   `data.generic = "generic got GreetingInput(name=a)"`, and `01-plain.json` has

.claude/skills/verify-tools/features/context-and-environment.md

@@ -11,7 +11,7 @@ Any other trailing parameter type fails the build with "No method found".
 - `ctx-env-type-resolver` passes `graphql.GraphQLContext` and the environment to a `GraphQLResolver<AppUser>` method.
 - `ctx-env-single` keeps a single environment or a single context parameter working.
 - `ctx-env-invalid` rejects a trailing `String` parameter at build time.
-- `ctx-env-misbound` warns at build when a schema argument lands on an environment or context parameter.
+- `ctx-env-misbound` fails the build when a schema argument would land on an environment or context parameter.
 - `sourceless` binds a field resolver method that leaves out the source and takes only arguments, the environment or the
   context, when no method with the source and no property matches.
 
@@ -24,14 +24,12 @@ Any other trailing parameter type fails the build with "No method found".
 - **Invalid.** `run context-and-environment-invalid`. Exit `2`.
   `build-error.txt` starts with `graphql.kickstart.tools.resolver.FieldResolverError: No method found ... (optionally
   followed by any of [interface graphql.schema.DataFetchingEnvironment, class graphql.GraphQLContext, class
-  scenarios.AppContext])`.
-- **Misbound.** `run context-misbound`. Exit `0`. `probe.log` has 3 `SchemaClassScanner` WARNs, for `Query.users`,
-  `Query.contextArgument` and `User.thumbnail`, none for `Query.greeting`. The first reads `Field 'Query.users' has no
-  parameter for its argument 'first: Int' in scenarios.ContextMisboundQuery.users, so the DataFetchingEnvironment
-  parameter gets the argument instead.` All four operations have `errors=1`. `01` fails with `Parameter specified as
-  non-null is null`, `02` and `04` with ``Cannot construct instance of `graphql.schema.DataFetchingEnvironment` ``, and
-  `03-context-and-correct.json` has `data = {"contextArgument": null, "greeting": "greeting: hi ada"}` with one error, for
-  `/contextArgument`. Before #890 the same run has the same four errors and an empty `probe.log`.
+  graphql.kickstart.tools.compat.scenarios.AppContext])`.
+- **Misbound.** `run context-misbound`. Exit `2`, no operations run, and `probe.log` is empty. `build-error.txt` starts
+  with `graphql.kickstart.tools.resolver.FieldResolverError: Field 'Query.users' has no parameter for its argument
+  'first: Int' in graphql.kickstart.tools.compat.scenarios.ContextMisboundQuery.users, so the DataFetchingEnvironment
+  parameter gets the argument instead. Add a parameter for each argument, in order, before the DataFetchingEnvironment
+  parameter.` The build stops at that first field, so `Query.contextArgument` and `User.thumbnail` aren't named.
 - **Without the source.**
   - `run sourceless`: exit `0`, `errors=0` for both operations. In `01-local-context.json` each image has
     `imageBlob = "blob for [png, jpg]"` (`imageBlob(env)` reads the `localContext` that `images` returned),
@@ -44,12 +42,18 @@ Any other trailing parameter type fails the build with "No method found".
     throws, so `01-helper.json` is `{"threw": "java.util.concurrent.CompletionException: kotlin.NotImplementedError: An
     operation is not implemented: Schema resolver not implemented"}`.
   - `run sourceless-misbound`: exit `2`. `thumbnail(GraphQLContext, env)` for `thumbnail(size: Int)` still fails the build.
-    The signatures listed for `scenarios/sourceless-misbound/schema.graphqls:11` end with
-    `scenarios.MisboundImageResolver.thumbnail(~size, <environment or context>)` and
-    `scenarios.MisboundImageResolver.getThumbnail(~size, <environment or context>)`.
+    The signatures listed for `compat/scenarios/sourceless-misbound/schema.graphqls:11` end with
+    `graphql.kickstart.tools.compat.scenarios.MisboundImageResolver.thumbnail(~size, <environment or context>)` and
+    `graphql.kickstart.tools.compat.scenarios.MisboundImageResolver.getThumbnail(~size, <environment or context>)`.
 
 ## Gotchas
 
+- Before #911 (commit 296ac4bf) `context-misbound` builds with exit `0` and all four operations have `errors=1`. `01`
+  fails with `Parameter specified as non-null is null`, `02` and `04` with
+  ``Cannot construct instance of `graphql.schema.DataFetchingEnvironment` ``, and `03-context-and-correct.json` has
+  `data = {"contextArgument": null, "greeting": "greeting: hi ada"}` with one error, for `/contextArgument`. From #890 on,
+  `probe.log` has 3 `SchemaClassScanner` WARNs, for `Query.users`, `Query.contextArgument` and `User.thumbnail`, none for
+  `Query.greeting`. Before it, `probe.log` is empty.
 - Before #888 only one trailing parameter was allowed, so `context-and-environment` fails to
   build on older versions with the same "No method found" error.
 - Before #904 `sourceless` fails the build (exit `2`) with `No method or field found ... imageBlob` and six listed

.claude/skills/verify-tools/features/directives-and-relay.md

@@ -5,8 +5,8 @@ can wrap the data fetcher of every field that carries the directive. `@deprecate
 field marked `@connection(for: "Member")` gets its `MemberConnection`, `MemberConnectionEdge` and `PageInfo` types
 generated, so the SDL never declares them, and a resolver returning `graphql.relay.Connection<Member>` fills them. A
 wiring also runs on interfaces, unions, enums and input objects, including ones that refer to themselves or each other.
-The build checks every applied directive against its definition. It logs a WARN with the source line for an invalid
-application, and fails for a directive with no definition unless `allowUndeclaredDirectives(true)` is set.
+The build checks every applied directive against its definition. It fails for an invalid application, listing each with
+its source line, and for a directive with no definition unless `allowUndeclaredDirectives(true)` is set.
 
 ## Sub-features
 
@@ -18,19 +18,19 @@ application, and fails for a directive with no definition unless `allowUndeclare
 - `dir-connection-paging` returns edges with cursors and `pageInfo` from `SimpleListConnection` for `first: 2`.
 - `dir-scalar` applies the directives of a scalar and its extension, runs `onScalar`, and reports `specifiedByURL` from
   the SDL `@specifiedBy` or else from the provided scalar (`scalar-directives`).
-- `dir-scalar-undeclared` logs a WARN for an undeclared directive on a scalar and builds the scalar without it
-  (`scalar-directives`).
+- `dir-scalar-undeclared` keeps an undeclared directive on a scalar in `appliedDirectives` when
+  `allowUndeclaredDirectives(true)` is set (`directive-drift`).
 - `dir-wiring-recursive` hands a wiring on a field of a self-referencing input object the built `GraphQLInputObjectType`,
   not a `GraphQLTypeReference` (`wiring-shapes`).
 - `dir-wiring-shapes` runs wirings on an interface, a union, an enum and a self-referencing object type, and lets a wiring
   make a self-referencing field non-null (`wiring-shapes`).
-- `dir-applied-legacy` lists every applied directive, repeats and undeclared ones included, in `appliedDirectives`, and
-  only the declared ones in the legacy `directives`, keeping the first of a non-repeatable directive
-  (`directive-drift`).
+- `dir-applied-legacy` lists every applied directive, undeclared ones included, in `appliedDirectives`, and only the
+  declared ones in the legacy `directives` (`directive-drift`).
 - `dir-undeclared-fails` fails the build for an applied directive with no definition unless
   `allowUndeclaredDirectives(true)` is set (`directive-drift-undeclared`).
-- `dir-invalid-warn` logs a WARN with the source line and column for a directive applied where its definition doesn't
-  allow it, without a required argument, or twice when it isn't repeatable, and still builds (`directives-invalid`).
+- `dir-invalid-fails` fails the build for a directive applied where its definition doesn't allow it, without a required
+  argument, or twice when it isn't repeatable, and lists every one with its source line and column
+  (`directives-invalid`).
 
 ## Expected results
 
@@ -51,37 +51,32 @@ application, and fails for a directive with no definition unless `allowUndeclare
   `scalar Date @since(date : "2020-01-01") @unit(name : "day") @specifiedBy(url : "https://example.com/date")`, and
   `scalar DateTime @specifiedBy(url : "https://scalars.graphql.org/andimarek/date-time")`. In `01-introspect-scalars.json`,
   `Date` has `specifiedByURL: "https://example.com/date"` and `DateTime` has
-  `"https://scalars.graphql.org/andimarek/date-time"`. `probe.log` has exactly `Scalar 'Money' is built without its
-  directives from the schema: Found applied directive currency without corresponding directive definition.`
+  `"https://scalars.graphql.org/andimarek/date-time"`. `probe.log` is empty.
 - **Wirings on cycles.** `run wiring-shapes`. Exit `0`, `errors=0` for both operations and an empty `probe.log`.
   `schema.graphqls` has `"sees GraphQLInputObjectType TreeFilter"` on `TreeFilter.children`, `friend: Person! @required`
   and `state: State! @required` where the SDL declares both nullable, and `"described"` above `Named`, `Someone`,
   `Person`, `State` and `TreeFilter`. `01-trees.json` has `data.trees = 4`. `02-person.json` has
   `data.person = {"name": "ada", "friend": {"name": "ada"}, "state": "ALIVE", "someone": {"name": "ada"}}` and
   `data.someone = {"name": "grace"}`.
 - **Applied and legacy lists.** `run directive-drift`. Exit `0` and `schema build: OK`, with no operations. In
-  `inspect.txt`, `input field Meta.owner` has two `applied @once` (`reason` `"x"`, then `"because"`) and one `legacy @once`
-  with `reason = "x"`. `field Query.search` has two `applied @undeclared`, the first with arguments typed from their
-  literals (`i: Int`, `f: Float`, `b: Boolean`, `l: [Int]`), and no legacy one. `enum value Color.BLUE` has
-  `applied @deprecated` and `applied @undeclared` but only `legacy @deprecated`.
-- **Drift warnings.** `probe.log` of `directive-drift` has 12 `SchemaParser` WARNs. Six are `isn't repeatable but is applied
-  more than once`, two are an extension of `Date` or `Query` applying `@once` again, two are `is missing its required
-  argument` (`@tag` and `@needs`), one is `Directive @fieldOnly at line 38, column 28 in
-  scenarios/directive-drift/schema.graphqls is applied to ARGUMENT_DEFINITION, but it's declared on FIELD_DEFINITION.`, and
-  one is `Scalar 'Money' is built without its directives from the schema: Found an unexpected directive argument tag#nam .`
-  None names `@undeclared`, because `directive-drift` sets `allowUndeclaredDirectives(true)`.
+  `inspect.txt`, `field Query.search` has two `applied @undeclared`, the first with arguments typed from their literals
+  (`i: Int`, `f: Float`, `b: Boolean`, `l: [Int]`), then two `applied @tag`, and two `legacy @tag` but no legacy
+  `@undeclared`. `enum value Color.BLUE` has `applied @deprecated` and `applied @undeclared` but only
+  `legacy @deprecated`. `type Money` has one `applied @undeclared` with `code: String = literal "USD"` and no legacy
+  directive. `type Query` has `applied @tag` (`"query"`), then the extension's `@once` (`"extension"`), `@tag` and
+  `@undeclared`, and the same list without `@undeclared` as legacy. `probe.log` is empty.
 - **Undeclared.** `run directive-drift-undeclared`. Exit `2`. `summary.txt` has `schema build: FAILED
   graphql.kickstart.tools.SchemaError: Found applied directive undeclared without corresponding directive definition.`,
   and `build-error.txt` starts with the same message.
-- **Invalid applications.** `run directives-invalid`. Exit `0` and `errors=0`. `01-query.json` has
-  `data.motto = "KEEP IT SIMPLE"`, so the misplaced `@uppercase` still wires. `probe.log` has exactly 5 `SchemaParser`
-  WARNs, each naming its line and column in `scenarios/directives-invalid/schema.graphqls`.
-  `Directive @uppercase at line 12, column 20 ... is applied to FIELD_DEFINITION, but it's declared on OBJECT.`
+- **Invalid applications.** `run directives-invalid`. Exit `2`, no operations run, and `probe.log` is empty. `summary.txt`
+  has `schema build: FAILED graphql.kickstart.tools.SchemaError:` followed by one line per invalid application, each
+  naming its line and column in `compat/scenarios/directives-invalid/schema.graphqls`, in this order.
   `Directive @deprecated at line 17, column 13 ... is applied to OBJECT, but it's declared on FIELD_DEFINITION |
   ARGUMENT_DEFINITION | ENUM_VALUE | INPUT_FIELD_DEFINITION.`
+  `Directive @uppercase at line 12, column 20 ... is applied to FIELD_DEFINITION, but it's declared on OBJECT.`
   `Directive @auth at line 13, column 20 ... is missing its required argument 'role'.` Two
   `Directive @cached at line ... isn't repeatable but is applied more than once`, at line 14, column 32 (the second
-  `@cached` on `members`) and line 9, column 15 (`extend schema`).
+  `@cached` on `members`) and line 9, column 15 (`extend schema`). `build-error.txt` starts with the same message.
 
 ## Gotchas
 
@@ -96,4 +91,6 @@ application, and fails for a directive with no definition unless `allowUndeclare
   graphql-java versions.
 - Before #895 (commit 21f3796e) `wiring-shapes` has the same `schema.graphqls`, except that `TreeFilter.children` reads
   `"sees GraphQLTypeReference TreeFilter"`.
-- Before #885 (commit 94a98a01) `directives-invalid` has an empty `probe.log` and the same `01-query.json`.
+- Before #914 (commit 2114b149) `directives-invalid` builds with exit `0`, and `01-query.json` has
+  `data.motto = "KEEP IT SIMPLE"`, so the misplaced `@uppercase` still wires. From #885 (commit 94a98a01) on, `probe.log`
+  has the five messages as `SchemaParser` WARNs. Before it, `probe.log` is empty.