ci: shard Iceberg Spark tests across four runners

[sunchao]

Why are the changes needed?

The main Iceberg Spark test task is one of Comet CI's longest paths. In a recent successful job, test execution consumed about 67 minutes, including 18.7 minutes in TestStructuredStreamingRead3 and 10.7 minutes in TestRewriteDataFilesAction. The job finished after 78 minutes including setup and compilation.

Which issue does this PR close? None. This is a CI scheduling improvement that retains the existing supported versions and test coverage.

What changes were proposed in this PR?

Split the main Iceberg Spark test task across four isolated workers. Keep the long structured-streaming family together and assign all remaining classes automatically by a stable hash. New classes therefore receive a worker without maintaining a test allowlist. Nested classes stay with their enclosing class, and all classes remain available on the test classpath for inherited fixtures and framework discovery.

A workflow-side Gradle init script adds an exclusion predicate to the existing candidate selection. It does not replace Gradle's existing includes/excludes or JUnit's tag and parameterized-test handling. Each worker uploads its candidate inventory and JUnit reports, with distinct names for reruns.

Run a lightweight coverage-equivalence fixture once per Iceberg version in the existing short runtime job. Native compilation, extensions, runtime-test commands, and existing Comet/Gradle build flags are otherwise unchanged. This PR intentionally leaves repeated JVM compilation for separate work.

How was this PR tested?

• Ran the real Gradle/JUnit fixture with Gradle 8.14.4/JDK 17 and Gradle 8.12.1/JDK 11. Both runs verified that the four workers equal the unsharded set of 10 candidate classes and 12 executed cases exactly once.
• Covered nested, parameterized, inherited, dynamic, and newly named tests; existing include/exclude patterns, exclusion predicates, and JUnit tags; single-worker equivalence; and isolation from another Test task.
• Verified that a failing test still fails Gradle, and invalid worker indices or task names fail explicitly.
• Checked the streaming family in all four configured Iceberg releases, the change filters, and the unchanged native/extension jobs and existing test flags. Passed actionlint, Python syntax, documentation formatting, Apache RAT, and git diff --check.

Full Iceberg suites and hosted elapsed-time improvements have not been measured for this patch. Additional workers repeat the existing setup/compilation work; the expected improvement is in wall-clock test completion, not necessarily total runner-minutes. The main test workers have new shard-suffixed check names, so any external requirements that refer to the old individual check should be checked.