Repository navigation
Carry the formatter inside the Gradle plugin and resolve per project - #198
Merged
Merged
Conversation
No test ran a build with the build cache on, so nothing showed that spotlessJava is taken from it only when it should be. BuildCacheTest does, with the configuration cache and without: spotlessJava is up to date when the source is written again unchanged, runs when the source or the formatter's jars change, and comes from the cache when the source changes back, after the build directory is deleted and in a copy of the project in another directory. Each build is spotlessApply, and each test checks the file it leaves. That shows the output spotlessJava ran for, kept or took from the cache is the source formatted; a step that changes nothing passes a spotlessCheck of a formatted file as well. GradleTestProject can now run its builds with the build cache, in a directory of the test's own: TestKit builds share one Gradle user home, and with it the default local cache. It can also delete a file or a directory of the project.
A Spotless test passed as long as its task did: three of them would have passed with a step that formats nothing, as they did while the configuration cache lost the formatter. spotlessJava_is_up_to_date_until_the_formatter_jars_change ran spotlessCheck on a formatted file, which a step that changes nothing passes. It now runs spotlessApply on the unformatted source each time and checks the file it leaves; line endings no longer matter, so the build no longer sets lineEndings to PRESERVE. format_ignores_excluded_directories passed when nothing was formatted at all. It now runs spotlessApply with a file outside the excluded directory too, and checks that the one is formatted and the other is left as it was. format_checks_non_generated_files passed on any failure of the check, a lint error from a failing formatter as well; it now looks for the format violation. With the step's formatter returning its input, every test that runs a Spotless task now fails.
SpotlessInterop picked NativePalantirJavaFormatStep when the native formatter was switched on and Gradle ran on a JVM older than 21. The plugin is compiled for 21 and does not load on an older JVM, so the branch never ran: Spotless formats on the Gradle JVM, as the docs say, whatever openjavaformat.native.formatter is. The step, the version check and the native configuration lookup go. So does PalantirJavaFormatStep.create(Configuration, Supplier), deprecated when the step stopped using the supplier; nothing calls it.
MultiProjectTest builds the layout the docs recommend: the plugins declared in the root project, applied to two subprojects, the root project without java. It fails on two counts. With --parallel, a subproject's spotlessJava resolves the root project's palantirJavaFormat without the root project's lock, and Gradle refuses: "Resolution of the configuration ':palantirJavaFormat' was attempted without an exclusive lock". A root project with java and Spotless of its own fails the same way. Without java on the root project, palantirJavaFormat asks for no attributes, and Gradle cannot pick a variant of a library that has one for the JRE and one for Android, as Guava does. The formatter depends on Guava, so the default formatter fails to resolve. The test uses a library of its own with two such variants, in a repository it writes under build/, so that it runs offline.
A subproject's Spotless step resolved the root project's palantirJavaFormat itself, which a parallel build does not allow: the subproject's task runs without the root project's lock. And palantirJavaFormat asked for no attributes, so in a root project without java Gradle could not pick between Guava's variants for the JRE and for Android, and the default formatter did not resolve. The layout the docs recommend has no java on the root project. The root project now offers palantirJavaFormat as a variant of its own, openJavaFormatFormatterElements, with a capability of its own so that a root project with java of its own still hands out this variant and not runtimeElements. Every project that applies the Spotless step resolves it through openJavaFormatFormatterClasspath, a dependency on the root project that Gradle resolves in the project that asks. palantirJavaFormat and both new configurations ask for the runtime jars of a JVM library. A formatter declared in palantirJavaFormat, files as well as modules, reaches every project as before. MultiProjectTest passes, with the configuration cache and without, and a new test there shows that a build that does not format still resolves nothing.
formatDiff fails there as the Spotless step did: with the Java-based formatter, a subproject's formatDiff resolves the root project's palantirJavaFormat as its classpath, without the root project's lock, and Gradle refuses. With the configuration cache the build fails as soon as the entry is stored.
With the Java-based formatter, formatDiff's classpath was the root project's palantirJavaFormat, which a subproject's task cannot resolve in a parallel build. It now takes openJavaFormatFormatterClasspath, as the Spotless step does, resolved in the project the task belongs to. The native image is left as it was: formatDiff already runs it in parallel builds, this repository's among them.
The plugin resolved the Java-based formatter from Maven Central, as dev.openjavaformat:open-java-format in the plugin's version, through the root project's palantirJavaFormat: a build needed a repository for it, and the formatter could be another version than the plugin (#20). bundleFormatter now puts the formatter's runtime classpath, the jars of open-java-format and its dependencies from the same commit, into the plugin's resources under open-java-format/formatter/, unchanged, next to jars.txt: a line per jar with its sha256 and its name, in the order of the classpath. The plugin jar grows from 64 KB to 6.3 MB; its POM and module metadata do not change, and the sources jar leaves the jars out. TestKit sees them through build/resources/main. Nothing reads them yet. BundledFormatterTest checks that the list is the classpath, and that every jar in it is in the plugin as listed.
A class loader cannot load a jar inside a jar, and the IDE plugin needs files it can still find after a clean. BundledFormatter.jars extracts the plugin's formatter jars into <Gradle user home>/open-java-format-caches/<hash>/, where the hash is of jars.txt, which holds every jar's sha256: another formatter gets another directory, and a directory, once there, never changes. A daemon that holds the jars open, as Windows does not let a file be replaced, is in nobody's way. Builds and daemons may extract at the same time. Each copies the jars into a directory of its own next to the target, checking every jar against its sha256, and renames it into place, which fails when the target is already there; the build that loses deletes its copy and uses the one that won. There is no file lock: FileChannel.lock fails in a daemon whose other build holds the same file through another class loader of the plugin. A directory that was deleted, or lost a jar, is extracted again. A jar that does not match the list fails with its name and leaves no directory behind. Nothing calls this yet.
WithoutRepositoriesTest applies the plugin to a build that declares no repository and no formatter, alone and as the docs lay out several projects, and runs spotlessApply and formatDiff, in parallel where there are subprojects. Every test fails: palantirJavaFormat depends on dev.openjavaformat:open-java-format by default, and "no repositories are defined" to resolve it. Under TestKit its version is even null, as the plugin's classes are not in a jar with a manifest; the other tests put the formatter in palantirJavaFormat themselves. GradleTestProject can leave out the repository it otherwise declares.
palantirJavaFormat no longer depends on dev.openjavaformat:open-java- format by default: it is empty unless a build declares a formatter there, and BundledFormatter.jarsUnlessOverridden then gives the jars inside the plugin, extracted on first use. The Spotless step, formatDiff and the palantirJavaFormat extension pick their jars that way, so a build formats where it applies the plugin, with no repository and nothing declared. An empty override resolves to nothing, which needs no repository either. The Spotless step keeps what the build declared and the Gradle user home, and signs the jars it formats with when Gradle fingerprints the task: by name, size and content, as before, so the build cache key does not depend on the directory they were extracted into. A build that reuses its configuration cache entry after that directory was deleted extracts the jars again. WithoutRepositoriesTest passes. The provider plugin's description now says what palantirJavaFormat is for.
With the formatter inside the plugin, the Spotless step signs the extracted jars, and Spotless stores that signature in the configuration cache entry: the jars are extracted, and checked for, while the entry is stored. Gradle takes those checks for inputs, so a build after the directory is deleted, or after the first extraction, cannot reuse the entry and stores another; on a new machine or a new plugin version that is one more configuration of every project. spotlessApply_reuses_the_entry_after_the_extracted_formatter_is_deleted fails on that, and is left out on Windows, which does not delete the jars a daemon holds open.
The Spotless step's signature, which the up-to-date check and the build cache key compare and which the configuration cache entry stores, now goes by the hash of jars.txt for the formatter inside the plugin. jars.txt holds every jar's name and sha256, so the signature changes with any of them as before, and still holds no path; but nothing on disk is read to make it. Storing a configuration cache entry no longer extracts the jars or checks for them, and a build after the extracted directory was deleted reuses the entry. The jars are extracted when the step makes its formatter, as the task runs. A formatter declared in palantirJavaFormat is still signed by its jars' names, sizes and contents.
Since palantirJavaFormat is empty by default, updateOpenJavaFormatXml writes an empty implementationClassPath into .idea/open-java-format.xml, and the IDE no longer runs the build's formatter. And once it lists the extracted jars, the task must not stay up to date after their directory is deleted, or the IDE keeps reading jars that are gone. Both new tests in WithoutRepositoriesTest fail on that.
updateOpenJavaFormatXml now writes the jars BundledFormatter extracts into implementationClassPath when palantirJavaFormat is empty, and the declared ones otherwise, so the IDE runs the build's formatter as it did when the formatter came from Maven Central. The extracted jars are where the IDE loads them from after a clean, too. The task takes the bundle's hash as an input, and is not up to date while the extracted jars are missing: after their directory is deleted the next build extracts them again and writes the XML, instead of leaving the IDE with paths to jars that are gone. The check is a class, which the configuration cache stores with the task.
The Spotless tests declared the jars this build made in palantirJavaFormat, which was the only way they could format: under TestKit the default dependency had no version. They now run with nothing declared, as a build that applies the plugin does, through the jars the plugin carries. BuildCacheTest keeps two tests of a declared formatter: declaring one makes spotlessJava run again, and so does a change of the declared jars. ConfigurationCacheTest keeps its test of the declared jars too, and MultiProjectTest stays on a declared formatter, as resolving that in the subprojects is what it tests. With the step's formatter returning its input, every test that runs a Spotless task still fails.
FormatDiffWithoutJavacExportsTest and the Java-based case of PalantirJavaFormatPluginTest no longer declare the jars this build made: formatDiff runs the formatter the plugin carries, and still names both settings when the Gradle JVM does not export javac. The native case still declares the binary the native jobs build. The tests that read impl.classpath are now those of a declared formatter, BundledFormatterTest, and FormatDiffTest, which loads the formatter itself.
Spotless turns whatever a step throws into a lint of the file, and spotlessJava succeeds with that lint as its output. Without the --add-exports flags the formatter's check fails inside the step, so spotlessJava succeeds, the lint goes into the build cache, and after the flags are added the same key hands the lint back: spotlessApply fails until the cache is cleared. SpotlessWithoutJavacExportsTest fails on both. Two more tests hold what is right today: a declared formatter that does not resolve fails the build, while storing the configuration cache entry or in spotlessJava, and a file the formatter cannot parse is still a lint of that file.
The formatter checks that javac's internals are exported to it when it loads, and inside the Spotless step that failure became a lint of every file: spotlessJava succeeded, the build cache kept the lint, and spotlessApply failed on it until the cache was cleared, even after the flags were added. spotlessJava now runs the same check before the step does, as an action of its own, without loading the formatter: the flags export the packages to every unnamed module, so the plugin's module answers for the formatter's. spotlessJava fails with the message formatDiff gives, nothing is stored in the build cache, and once the flags are there the next build formats. A file the formatter cannot parse is still a lint of that file.
NativeImageResolutionTest builds what #20 reports: the root project declares no repository, the subprojects declare theirs, and openjavaformat.native.formatter is on. formatDiff fails, alone and in parallel, with "Cannot resolve external dependency dev.openjavaformat:open-java-format-native:1.0 because no repositories are defined": the root project resolves palantirJavaFormatNative. The image is a shell script that answers formatDiff with the replacements that format the file, in a Maven repository the test writes under build/, so that the test runs in every build on Linux and macOS, offline; the build has no --add-exports, so only the image could have formatted the file. The test also expects the image to run from a copy in open-java-format-caches, made executable.
The root project resolved palantirJavaFormatNative through its own repositories, so a build that declares them in its subprojects only failed (#20), and the image ran from the output of ExecutableTransform. The root project now offers palantirJavaFormatNative as a variant, openJavaFormatNativeElements, as it offers palantirJavaFormat, and every project with formatDiff resolves it through openJavaFormatNativeImage, with its own repositories: a build behind a mirror of Maven Central declares the mirror where it declares its other repositories. The default dependency on the image of the plugin's version and platform stays, and so does declaring another. formatDiff copies the image it resolved into <Gradle user home>/open-java-format-caches/<hash>/, next to the jars of the formatter inside the plugin, makes the copy executable and runs it from there. NativeImageAtomicCopy is back for that, without its file lock: it writes through AtomicCacheDirectory, which BundledFormatter now uses too. The hash is of the image's contents, kept per daemon by path, size and time, as the image is some 60 MB. ExecutableTransform goes; the copy sets the executable bit. NativeImageResolutionTest passes.
Spotless checks for, and writes, the platform's line endings by default, CRLF on Windows, while the tests write their files with LF. On Windows a check then reports the line endings instead of the formatting: a_file_the_formatter_cannot_parse_is_still_a_lint_of_that_file failed there, as the up-to-date test did before d289fcd set lineEndings to PRESERVE in its own build. GradleTestProject now sets lineEndings to PRESERVE in every project of a generated build that applies Spotless, so no test has to remember it. Spotless then writes LF back, and the tests compare the files they leave exactly instead of ignoring line endings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #20. Closes #194.
Where the plugin is applied, it now formats with nothing else configured: no repository, no dependency and no
javaon the root project for the Java-based formatter, and the native image comes through the repositories of the project that formats.spotlessJavais taken from the build cache only for the same sources, formatter and Spotless settings, and only when formatting actually happened. All of it holds with the configuration cache and without, and in parallel builds.bundleFormatterputs the formatter's runtime classpath (16 jars, unchanged) into the plugin's resources withjars.txt, a sha256 per jar. On first useBundledFormatterextracts them into<Gradle user home>/open-java-format-caches/<hash of jars.txt>/, through a copy of its own renamed into place, so builds and daemons can race; no file lock, asFileChannel.lockfails across the plugin's class loaders in one daemon. The formatter still loads in a class loader of its own. The plugin's POM does not change.palantirJavaFormatis an optional override. It has no default dependency any more; empty, the Spotless step,formatDiff,updateOpenJavaFormatXmlandJavaFormatExtension.serviceLoad()use the bundled jars, and declared, the declared ones.palantirJavaFormatandpalantirJavaFormatNativeas variants (openJavaFormatFormatterElements,openJavaFormatNativeElements, each with a capability of its own), and every project that formats resolves them through its own repositories. The formatter's configurations ask for Java runtime jars, without which Gradle could not pick Guava's JRE variant in a root project withoutjava.open-java-format-caches/<hash of the image>/, executable, byNativeImageAtomicCopy, whichformatDiffruns.ExecutableTransformgoes.jars.txtfor the bundled formatter and the declared jars' names, sizes and contents otherwise: never a path, so the key holds across directories and machines, and nothing is read from disk while a configuration cache entry is stored.--add-exportsflags,spotlessJavanow fails before the step runs, withformatDiff's message, and nothing is cached. A file the formatter cannot parse is still a lint.updateOpenJavaFormatXmllists the extracted jars, and is not up to date while they are missing.NativePalantirJavaFormatStepand theJavaVersion < 21branch that picked it, which could never run (Remove the native Spotless step: it only runs on a JVM the plugin cannot load on #194);PalantirJavaFormatStep.create(Configuration, Supplier), deprecated in Run the Gradle plugin's tasks with the configuration cache #195 and unused.Tests
BuildCacheTest: up to date when a source is written again unchanged, runs when it changes, from the cache when it changes back, after the build directory is deleted and in another directory; runs again when a formatter is declared or the declared jars change.MultiProjectTest,WithoutRepositoriesTest: the docs' multi-project layout, with--parallel, with nojavaon the root project, and with no repository at all, for Spotless,formatDiffand the IntelliJ task. A library with a JRE and an Android variant, in a repository written underbuild/, stands in for Guava offline.NativeImageResolutionTest: Ship the formatter jar inside the Gradle plugin instead of resolving it from the root project #20's layout, repositories in the subprojects only, with a shell script for the image that answersformatDiffwith the replacements that format the file (Linux and macOS).SpotlessWithoutJavacExportsTest,BundledFormatterTest,NativeImageAtomicCopyTest, and a configuration cache test that deletes the extracted jars between two builds.formatDiffchecks the files it leaves: with the step's formatter returning its input, every Spotless test fails. Most tests now run with the bundled formatter instead of declaring this build's jars.On JDK 21, macOS aarch64:
./gradlew testpasses, 1626 tests;-PnativeImage=true :gradle-open-java-format:testpasses, 152 tests, with an image built on GraalVM 25. By hand, with the plugin and the image published to a local repository: the docs' layout without a single repository in the projects formats through Spotless andformatDiff, and #20's layout formats with the native image, with the configuration cache and without, in parallel and not.Possible downsides
palantirJavaFormat, and the native image, now resolve through the repositories of each project that formats, not the root project's: a build with repositories in the root project only has to declare them where it formats. The IntelliJ task still resolves a declared formatter in the root project.--add-exports,spotlessJavaand sospotlessApplyfail, wherespotlessCheckused to fail on a lint andspotlessApplydid nothing.open-java-format-cacheskeeps a directory per formatter (6.5 MB) and per native image (some 60 MB); nothing cleans old ones yet.PalantirJavaFormatStep.createand theJavaFormatExtensionconstructor take the Gradle user home; the classes above are gone.Not here