[oryan-block]

Why

Agents working in graphql-java-tools keep repeating the same mistakes, and the repo should change so a change that looks right from one file is right for the whole repo. One of them is rejecting invalid input that used to build, without the WARN, issue and TODO step that the compatibility rule asks for. It happened 3 times: a kind-mismatched extension became an error (c4029496), the build threw for JDK proxied resolvers (8611c516), and scalar Date @specifiedBy(url: 5) started throwing (#894, which shipped). Each looked fine from the file it touched, and no test noticed because nothing remembered what the last release accepted.

Checklist

• Pull requests follows the contribution guide
• New or modified functionality is covered by tests

Description

• The 40 verify-tools scenarios moved into src/test/kotlin/graphql/kickstart/tools/compat/scenarios as data: SDL files, resolvers and a configure for the rest of the builder. The probe compiles the same directory, so there's one copy.
• CompatTest builds every scenario 3 ways: as it is, with its resolvers behind JDK proxies, and with one invalid SDL snippet appended per rule of the grammar it can break (an extension of another kind, an undeclared or misplaced directive, a wrong argument value, a repeated field, member, value or directive). That's 1,422 cases, 604 of which build on 15.0.0.
• It fails when a case that built on the release in src/test/resources/compat/<version>/ throws now. The message lists the cases with their full error and says what to do instead: log a WARN, open an issue for the next major and leave the TODO.
• A deliberate break goes into that release's accepted.txt as one line: the case exactly as the failure lists it, a tab, and why, naming the issue (#N) or the release notes (15.1.0 release notes). A regex only counts when the line starts with re:, so a pasted async | kind | ... accepts that one case and not every case its | would match. A re: line may accept at most 20 cases, all in one scenario, so re:.* fails instead of accepting everything. 20 fits one operator on one directive or type, and the widest deliberate break so far (14.3.0 applying directives on extensions) needs at most 15 per line that way. A line without a reference, a bad regex, or a line that matches no failing case fails the test.
• It can't pass against a capture of the branch itself. -Dcompat.capture fails at a -SNAPSHOT version and says captures come from the release tag. The test fails while src/test/resources/compat/ has a capture for a version the pom says isn't released yet (15.0.1/ at 15.0.1-SNAPSHOT), so making one by hand doesn't work either.
• It can't pass by having nothing to check. An empty scenario list fails, an empty cases.tsv fails, and so does a case that built on the release and isn't generated any more (a scenario renamed, removed or edited). That failure gives the commands to capture the release again with the current scenarios, or says to list the case in accepted.txt.
• When the pom's major is above the release's, a case may break only if a WARN on the release names it. For a kind case that's the type. For the other snippet cases it's the type and the item (the member or the directive). A directive WARN may point at the application (line 1, column 19) instead of the type, since that's how 14.x worded them. For a seed or proxy case every name its error quotes has to be in one of the WARNs. So the next major can drop what 15.x warned about, but not something it accepted silently, and not something an unrelated WARN in the same scenario happened to mention.
• Errors keep their whole message, so an InvalidSchemaException shows invalid schema: Invalid argument 'url' for applied directive of name 'specifiedBy' instead of invalid schema:. The 15.0.0 capture is retaken for that, with v15.0.0's pom.xml and src/main. Its cases, WARNs and built count didn't change.
• AGENTS.md gets the capture step for each release (one mvn command on the tag) and the accepted.txt format. Its rule table row now says what CompatTest checks: invalid input that built on the last release keeps building until the next major.
• The feature pages quote the scenario classes as the probe prints them since the move (graphql.kickstart.tools.compat.scenarios.ItemInput, not scenarios.ItemInput). type-extensions.md says a repeated directive's error names the directive and its line, not the type.
• The type-extensions scenario repeated a field, a member, a value and a directive in its extensions, which 15.0.0 rejects, so probe.sh run type-extensions exited 2 while its feature page said 0. I dropped the repeats and updated the page.
• 6 more scenarios exited 2 on 15.0.0 while their pages said 0. directive-drift and scalar-directives teach things that still build (applied versus legacy directive lists, scalar directives), so they drop the applications 15.0.0 rejects. directives-invalid, context-misbound and jdk-proxy only exist to show input that 15.0.0 now rejects, so their pages say exit 2 and quote the error, and the old WARN results moved to Gotchas. type-extensions-repeated-fields was already right.

Scope

• It only checks whether a schema builds. A silent behaviour change on input that still builds isn't caught. Changed WARN text isn't either, except that in major mode a WARN has to keep naming its case.
• It only knows the shapes in the scenarios. A rejection of a shape none of them has, plus the variants, gets through.
• jdk-proxy-cglib can't build inside the library's Java module in tests (CGLIB can't subclass a class there), so that scenario isn't protected. The probe still runs it.
• nested-wrappers-exact and type-extensions-input-order have no feature page entry, same as before this PR.

Blast Radius

Nobody using the library notices. Contributors get about 2 s more in mvn verify on JDK 17, and a PR that starts rejecting input fails CompatTest with the list of cases. The capture adds a 420 KB cases.tsv to the test resources, and to the test jar.

Verification

• mvn -B -ntp clean verify on JDK 17 at the head: 458 tests, build passes, CompatTest 1.6 s.
• Re-adding c4029496's throw on this branch fails with 512 cases, {kind=511, seed=1}. With the throw still in, -Dcompat.capture fails with pom.xml is at 15.0.1-SNAPSHOT, so a capture would record this branch instead of a release and writes nothing. A 15.0.1/ capture made by setting the pom to 15.0.1 for the capture run passes on its own, then once the pom is back at 15.0.1-SNAPSHOT the test fails with src/test/resources/compat/15.0.1 can't hold a release.
• re:.* in 15.0.0's accepted.txt fails with accepts 604 cases in 30 scenarios.
• accepted.txt, with the throw re-added: the pasted line async | kind | extend interface Query { compatZ: String } takes it to 511. re:async \| kind \| .* takes it to 498, the 14 async | kind cases. A line naming a type the scenario doesn't have fails as matching nothing. A line with no tab, one with no #N, and re: with an unclosed group each fail with their line number.
• Renaming the async scenario fails with the 15 cases that aren't generated any more. An empty cases.tsv fails, and so does an empty scenario list.
• Major mode, v15.0.0 against a 14.5.0 capture taken with these scenarios: 366 cases broke. A 14.5.0 WARN names 365 of them, and optional-results-nested (Unwrap a DataFetcherResult inside an Optional #940) is the only one left, which is the same result the any-WARN rule gives. It passes once 14.5.0's accepted.txt names it. Editing two 14.5.0 WARNs to name another type, or a different column, makes exactly those 2 cases fail too.
• The 14 class names I changed on the feature pages match the probe's output. 4 were checked at the head (context-and-environment-invalid, sourceless-misbound, unmapped-inputs, bookstore-missing-resolver) and the Gotchas ones on the commits they describe (a4736c9a for nested-wrappers and nested-wrappers-suspend, 3f022b91 for optional-results, f4800425^ for jdk-proxy).
• The test and scenarios at 1987124a ran unchanged on the old commits, with captures of v14.2.0 and v14.3.0 taken with these scenarios. c4029496 fails with 239 cases with an empty accepted.txt, and with 173, {kind=172, seed=1}, once 14.3.0's undeclared directive break is accepted. 8611c516 fails with 25, {proxy=24, seed=1}. 69e18bbd fails with 53, {argval=53}, including bookstore | argval | extend scalar Date @specifiedBy(url: 5). d47bac59 (master before Apply directives and @specifiedBy to scalars #894) passes. a17693b7 and 76b67394 carry 14.3.0's two deliberate breaks (undeclared directives and wrong argument values on extensions), so they pass only once 14.2.0's accepted.txt lists them.
• Those two breaks were accepted with one re: line each across every scenario. With this revision those lines fail (accepts 72 cases in 25 scenarios and accepts 27 cases in 1 scenarios). Written as 27 lines, one per scenario and operator and split by directive in type-extensions, a17693b7 and 76b67394 pass with this revision's test.
• probe.sh list shows 40 scenarios from the new directory. I ran all 40. The 38 with a page entry exit with the code their page gives, 29 with 0 and 9 with 2. The 2 without one exit 0.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud