Skip to content

feat: support Managed Spark runtime 4.0 - #233

Open
dborowitz wants to merge 2 commits into
mainfrom
dborowitz-s8s-4
Open

dborowitz wants to merge 2 commits into
mainfrom
dborowitz-s8s-4

Conversation

@dborowitz

Copy link
Copy Markdown
Contributor

Add client, documentation, and CI support for Managed Spark runtime 4.0 (Apache Spark 4.2, Python 3.13), while keeping runtime 3.0 (Spark 4.0, Python 3.12) and the [client] / [spark] extra pins (~=4.0.0) as the default.

Runtime 4.0 compatibility & UX

  • Runtime version table (_runtimes.py): Register runtime 4.0 (Spark 4.2, Python 3.13) so _check_python_version_compatibility() and _check_pyspark_version_compatibility() validate client Python and PySpark versions against runtime 4.0 sessions. Because Spark Connect servers accept older client minor versions, the default [client] / [spark] PySpark 4.0 extras also work against runtime 4.0 (using Spark 4.0 APIs), while installing pyspark-client~=4.2.0 (or pyspark[connect]~=4.2.0) in Python 3.13 enables Spark 4.2 APIs and matching Python UDF execution.
  • Subnetwork error hints (session.py): Runtime 4.0 requires an IPv4-only subnetwork rather than defaulting to the default VPC network. When CreateSession fails with InvalidArgument due to a missing subnetwork or an unsupported IPv6 / dual-stack subnetwork, append actionable guidance pointing to .subnetwork('<subnet>'), MANAGED_SPARK_CONNECT_SUBNET, or .runtimeProfile('<profile>').
  • MANAGED_SPARK_CONNECT_DEFAULT_DATASOURCE=bigquery catalog precedence (session.py): Runtime 4.0 defaults spark.sql.defaultCatalog to the BigLake Iceberg default_catalog. When MANAGED_SPARK_CONNECT_DEFAULT_DATASOURCE=bigquery configures spark.sql.catalog.spark_catalog = BigQuerySparkSessionCatalog, also set spark.sql.defaultCatalog = spark_catalog (unless already set by the user) so BigQuery remains the active default catalog on runtime 4.0.
  • Documentation (README.md, .env.example): Add a Runtime versions section covering runtime 3.0 vs. 4.0 installation options, the IPv4-only subnetwork requirement, opting out of default_catalog via .config('spark.sql.defaultCatalog', 'spark_catalog'), and current remote PyPI package installation limitations on runtime 4.0.

Remote PyPI artifact installation (pypi_artifacts.py, magics.py, session.py)

Remote PyPI package installation via %dpip and ManagedSparkSession.addArtifacts(..., pypi=True) is not yet supported on Managed Spark runtime 4.0 (which manages its Python environment with Pixi rather than pip, and does not ship pip in the session environment). To ensure consistent behavior across both entry points and surface a clear failure on runtime 4.0:

  • Route addArtifacts(..., pypi=True) through DataprocCommandRunner: Share a single ExecuteExternalCommand (DataprocCommandRunner / PipInstallPackages) implementation in pypi_artifacts.py across both %dpip and ManagedSparkSession.addArtifacts(..., pypi=True), replacing the legacy add-artifacts-1729-*.json marker upload path.
  • Catch IOException and Pixi installer failures: Broaden server output error detection from the narrow "Pip install failed with non-zero exit code" check to r"(?:Pip|Pixi) install failed\b", so IOException / InterruptedException failures (such as when pip is absent on runtime 4.0) and Pixi installer failures raise an error instead of silently reporting "Finished installing packages."

CI & testing

  • Unit tests (.github/workflows/tests.yaml): Add a dedicated ubuntu-latest Python 3.13 + pyspark-client~=4.2.0 job alongside the primary Python 3.12 [client] / [spark] matrix, plus unit tests covering runtime 4.0 compatibility checks, subnetwork hints, catalog precedence, and install_pypi_packages().
  • Integration tests (.github/workflows/integration-tests.yaml, tests/integration/): Add MANAGED_SPARK_CONNECT_TEST_RUNTIME_VERSION support in conftest.py, add an ubuntu-latest, runtime 4.0 (Python 3.13, pyspark-client~=4.2.0) integration job, qualify single-part spark_catalog DDL in test_session.py, and mark %dpip integration tests xfail(strict=False) on runtime 4.0 until server-side Pixi package installation rolls out.

Runtime 4.0 runs Spark 4.2 and Python 3.13. Add it to the runtime table so
that the client warns about a newer PySpark and mismatched Python, append
actionable hints when session creation fails due to a missing or dual-stack
subnetwork, keep spark_catalog active on runtime 4.0 when
MANAGED_SPARK_CONNECT_DEFAULT_DATASOURCE=bigquery is set, and document how to
opt into runtime 4.0 and its behavioral differences. The [client]/[spark]
extras' PySpark 4.0 works against it too, since Spark Connect supports older
clients, so that warns about nothing but Python.

CI now runs the full unit suite and the integration tests on each runtime's
own Python and pyspark: runtime 3.0 (3.12, the pinned extras) as before, and
runtime 4.0 (3.13, pyspark-client 4.2, which replaces the unpinned newest
pyspark-client job) on Linux, selected for the integration tests via the new
MANAGED_SPARK_CONNECT_TEST_RUNTIME_VERSION.

The default runtime and the [client]/[spark] pins stay on 3.0.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for Managed Spark runtime version 4.0, including documentation updates, subnetwork validation error hints, and a new remote PyPI package installation mechanism via DataprocCommandRunner. Feedback is provided regarding robust error handling in install_pypi_packages to catch potential TypeError exceptions when the response properties are missing or null.

@dborowitz
dborowitz added this pull request to stack #235 November 11, 2026 17:51
@dborowitz
dborowitz requested review from medb and tim-u November 11, 2026 17:51
…installer IOExceptions

Route ManagedSparkSession.addArtifacts(..., pypi=True) through the same
ExecuteExternalCommand / DataprocCommandRunner PipInstallPackages path used by
%dpip, rather than uploading legacy add-artifacts-1729-*.json marker files.

Broaden failure detection from the narrow 'Pip install failed with non-zero
exit code' substring to r'(?:Pip|Pixi) install failed\b' so IOException errors
(such as when pip is absent on Managed Spark runtime 4.0), InterruptedException
errors, and Pixi installer failures raise instead of silently succeeding.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant