[SPARK-58855][ML] Optimize IsotonicRegressionModel transform closure

[zhengruifeng]

What changes were proposed in this pull request?

This PR optimizes IsotonicRegressionModel.transform by snapshotting the underlying
boundaries and predictions arrays before creating the transform UDF. The double and vector
feature transform paths now call an array-backed companion helper instead of closing over the
legacy mllib.regression.IsotonicRegressionModel wrapper.

The legacy mllib.regression.IsotonicRegressionModel prediction path also reuses the helper, and
its RDD prediction path snapshots the arrays before creating the map closure.

Why are the changes needed?

Avoiding the model capture reduces transform-closure retained memory, which is useful for Spark
Connect server workloads and follows the closure-size optimizations tracked under SPARK-58584.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Ran local static checks:

git diff --check HEAD~1 HEAD
grep -rn -P "[^\x00-\x7F]" mllib/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
awk 'length>100 && $0 !~ /^[[:space:]]*(import|package) / && $0 !~ /https?:\/\// {print FILENAME":"FNR": "length" chars"}' mllib/src/main/scala/org/apache/spark/ml/regression/IsotonicRegression.scala mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala

Compiled MLlib:

build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 mllib/compile

Measured serialized ScalaUDF.function closure size with a temporary local probe:

Case	Before	After	Delta
isotonic.double	3862 B	1354 B	-2508 B, -64.9%
isotonic.vector	3931 B	1547 B	-2384 B, -60.6%

No unit test was added because this is an internal refactor of existing prediction logic.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)