Repository navigation
Conversation
Runtime 4.0 runs Spark 4.2 and Python 3.13. Add it to the runtime table so that the client warns about a newer PySpark and mismatched Python, append actionable hints when session creation fails due to a missing or dual-stack subnetwork, keep spark_catalog active on runtime 4.0 when MANAGED_SPARK_CONNECT_DEFAULT_DATASOURCE=bigquery is set, and document how to opt into runtime 4.0 and its behavioral differences. The [client]/[spark] extras' PySpark 4.0 works against it too, since Spark Connect supports older clients, so that warns about nothing but Python. CI now runs the full unit suite and the integration tests on each runtime's own Python and pyspark: runtime 3.0 (3.12, the pinned extras) as before, and runtime 4.0 (3.13, pyspark-client 4.2, which replaces the unpinned newest pyspark-client job) on Linux, selected for the integration tests via the new MANAGED_SPARK_CONNECT_TEST_RUNTIME_VERSION. The default runtime and the [client]/[spark] pins stay on 3.0.
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for Managed Spark runtime version 4.0, including documentation updates, subnetwork validation error hints, and a new remote PyPI package installation mechanism via DataprocCommandRunner. Feedback is provided regarding robust error handling in install_pypi_packages to catch potential TypeError exceptions when the response properties are missing or null.
…installer IOExceptions Route ManagedSparkSession.addArtifacts(..., pypi=True) through the same ExecuteExternalCommand / DataprocCommandRunner PipInstallPackages path used by %dpip, rather than uploading legacy add-artifacts-1729-*.json marker files. Broaden failure detection from the narrow 'Pip install failed with non-zero exit code' substring to r'(?:Pip|Pixi) install failed\b' so IOException errors (such as when pip is absent on Managed Spark runtime 4.0), InterruptedException errors, and Pixi installer failures raise instead of silently succeeding.
dborowitz
force-pushed
the
dborowitz-s8s-4
branch
from
November 11, 2026 18:02
19c567a to
718cc8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add client, documentation, and CI support for Managed Spark runtime
4.0(Apache Spark4.2, Python3.13), while keeping runtime3.0(Spark4.0, Python3.12) and the[client]/[spark]extra pins (~=4.0.0) as the default.Runtime
4.0compatibility & UX_runtimes.py): Register runtime4.0(Spark 4.2,Python 3.13) so_check_python_version_compatibility()and_check_pyspark_version_compatibility()validate client Python and PySpark versions against runtime4.0sessions. Because Spark Connect servers accept older client minor versions, the default[client]/[spark]PySpark4.0extras also work against runtime4.0(using Spark4.0APIs), while installingpyspark-client~=4.2.0(orpyspark[connect]~=4.2.0) in Python3.13enables Spark4.2APIs and matching Python UDF execution.session.py): Runtime4.0requires an IPv4-only subnetwork rather than defaulting to thedefaultVPC network. WhenCreateSessionfails withInvalidArgumentdue to a missing subnetwork or an unsupported IPv6 / dual-stack subnetwork, append actionable guidance pointing to.subnetwork('<subnet>'),MANAGED_SPARK_CONNECT_SUBNET, or.runtimeProfile('<profile>').MANAGED_SPARK_CONNECT_DEFAULT_DATASOURCE=bigquerycatalog precedence (session.py): Runtime4.0defaultsspark.sql.defaultCatalogto the BigLake Icebergdefault_catalog. WhenMANAGED_SPARK_CONNECT_DEFAULT_DATASOURCE=bigqueryconfiguresspark.sql.catalog.spark_catalog = BigQuerySparkSessionCatalog, also setspark.sql.defaultCatalog = spark_catalog(unless already set by the user) so BigQuery remains the active default catalog on runtime4.0.README.md,.env.example): Add aRuntime versionssection covering runtime3.0vs.4.0installation options, the IPv4-only subnetwork requirement, opting out ofdefault_catalogvia.config('spark.sql.defaultCatalog', 'spark_catalog'), and current remote PyPI package installation limitations on runtime4.0.Remote PyPI artifact installation (
pypi_artifacts.py,magics.py,session.py)Remote PyPI package installation via
%dpipandManagedSparkSession.addArtifacts(..., pypi=True)is not yet supported on Managed Spark runtime4.0(which manages its Python environment with Pixi rather thanpip, and does not shippipin the session environment). To ensure consistent behavior across both entry points and surface a clear failure on runtime4.0:addArtifacts(..., pypi=True)throughDataprocCommandRunner: Share a singleExecuteExternalCommand(DataprocCommandRunner/PipInstallPackages) implementation inpypi_artifacts.pyacross both%dpipandManagedSparkSession.addArtifacts(..., pypi=True), replacing the legacyadd-artifacts-1729-*.jsonmarker upload path.IOExceptionand Pixi installer failures: Broaden server output error detection from the narrow"Pip install failed with non-zero exit code"check tor"(?:Pip|Pixi) install failed\b", soIOException/InterruptedExceptionfailures (such as whenpipis absent on runtime4.0) and Pixi installer failures raise an error instead of silently reporting"Finished installing packages."CI & testing
.github/workflows/tests.yaml): Add a dedicatedubuntu-latestPython3.13+pyspark-client~=4.2.0job alongside the primary Python3.12[client]/[spark]matrix, plus unit tests covering runtime4.0compatibility checks, subnetwork hints, catalog precedence, andinstall_pypi_packages()..github/workflows/integration-tests.yaml,tests/integration/): AddMANAGED_SPARK_CONNECT_TEST_RUNTIME_VERSIONsupport inconftest.py, add anubuntu-latest, runtime 4.0(Python3.13,pyspark-client~=4.2.0) integration job, qualify single-partspark_catalogDDL intest_session.py, and mark%dpipintegration testsxfail(strict=False)on runtime4.0until server-side Pixi package installation rolls out.