Skip to content

fix(session): preserve 3.0+ runtimeProfile versions and warn when overriding pre-3.0 - #234

Open
dborowitz wants to merge 1 commit into
dborowitz-s8s-4from
dborowitz-version-compat
Open

dborowitz wants to merge 1 commit into
dborowitz-s8s-4from
dborowitz-version-compat

Conversation

@dborowitz

@dborowitz dborowitz commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Previously, ManagedSparkSession.Builder._get_session_config() unconditionally populated session_config.runtime_config.version with ManagedSparkSession._DEFAULT_RUNTIME_VERSION ("3.0") whenever no runtime version was set on the builder or Session config. Because CreateSession merges the session request on top of the referenced Runtime Profile (SessionTemplate), that client default silently overwrote any runtime_config.version configured on the Runtime Profile:

  • A Runtime Profile configured for runtime 4.0 was silently downgraded back to "3.0" unless .runtimeVersion("4.0") was also duplicated in code.
  • A Runtime Profile configured for a pre-3.0 runtime (such as "2.3") was silently evaluated as "3.0", either starting on a different runtime than configured without notice or failing CreateSession validation with confusing "3.0 runtime version does not support ..." errors on 2.3-only fields such as staging_bucket.

Changes

Extract Runtime Profile (SessionTemplate) lookup, runtime-version resolution, and MANAGED_SPARK_CONNECT_* execution-default precedence out of session.py into google/cloud/managed_spark_connect/_session_templates.py (modeled around an immutable TemplateRuntimeResolution value object so Builder carries no cross-method mutable state and session.py shrinks net relative to base):

When .runtimeProfile() / .sessionTemplate() is set and no explicit runtime version is configured on the builder or Session proto, fetch the SessionTemplate best-effort (reusing the single GetSessionTemplate call already used for MANAGED_SPARK_CONNECT_* execution environment variable precedence):

  1. 3.0+ Runtime Profile version ("3.0", "4.0", ...): Leave session.runtime_config.version unset ("") on CreateSessionRequest so the profile's runtime version is preserved, and run _check_python_version_compatibility() and _check_pyspark_version_compatibility() against the profile's runtime version.
  2. Pre-3.0 Runtime Profile version (e.g., "2.3"): Emit a UserWarning explaining that google-cloud-spark-connect requires runtime 3.0 or higher (with 'dataproc-spark-connect~=0.9.0' guidance for Spark 3.5 / runtime 2.3) and override session.runtime_config.version with _DEFAULT_RUNTIME_VERSION. If CreateSession subsequently fails with InvalidArgument (for example, on a 2.3-only staging_bucket field), append the override context to the ManagedSparkConnectException message as well.
  3. Empty Runtime Profile version (""): Populate session.runtime_config.version with _DEFAULT_RUNTIME_VERSION silently so the server does not fall back to an older pre-3.0 default runtime.
  4. Unreadable Runtime Profile (e.g., caller lacks dataproc.sessionTemplates.get): Leave session.runtime_config.version unset ("") rather than risking clobbering a 3.0+ profile. Because the control plane accepts spark_connect_session on 2.2 / 2.3 during CreateSession, inspect GetSession immediately after create_session() returns: if the server resolved < 3.0 (e.g., '2.3.x' or unversioned server-default '2.2.x'), initiate best-effort session termination and raise ManagedSparkConnectException immediately without waiting for VM startup; if it resolved 3.0+, run the Python and PySpark compatibility checks against the resolved version.
  5. Explicit pre-3.0 .runtimeVersion(): Include 'dataproc-spark-connect~=0.9.0' (Python 3.11) guidance in _check_runtime_compatibility() when a pre-3.0 runtime version is requested directly in code.
  6. Testing: Add unit tests (tests/unit/test_session_templates.py and tests/unit/test_session.py) and live control-plane integration tests covering 3.0+ profile preservation (with bare profile IDs), 2.3 warn-and-override, 2.3 + staging_bucket error context, and unreadable < 3.0 / unversioned profile fail-fast handling.

Builder × Runtime Profile × PySpark Matrix

Installed pyspark never changes which runtime version is selected; it only determines whether _check_pyspark_version_compatibility() emits a warning after the effective runtime is resolved (pyspark 4.0 is compatible with both 3.0 and 4.0; pyspark 4.2 warns [5] whenever the resolved runtime is 3.0, since Spark Connect does not support clients newer than the server). Client Python major.minor independently emits a UDF mismatch warning [7] whenever it differs from the resolved runtime's Python (3.12 on 3.0, 3.13 on 4.0).

Current default (_DEFAULT_RUNTIME_VERSION = "3.0", [client] / [spark] = pyspark 4.0)

Rows resolving to runtime 4.0 require an IPv4-only subnetwork via .subnetwork(), MANAGED_SPARK_CONNECT_SUBNET, or the Runtime Profile (raises ManagedSparkConnectException [6] if missing or dual-stack).

# Builder runtimeVersion() Profile runtime_config.version Sent on Session Effective Runtime Outcome with pyspark 4.0 Outcome with pyspark 4.2
1 Unset (No profile) "3.0" 3.0 OK — no warning Warn (PySpark) [5]
2 Unset "" (unset on profile) "3.0" 3.0 OK — no warning Warn (PySpark) [5]
3 Unset "3.0" "" (preserved) 3.0 OK — no warning Warn (PySpark) [5]
4 Unset "4.0" "" (preserved) 4.0 OK — no warning (Spark 4.0 client APIs against 4.2 server) OK — no warning
5 Unset "2.3" (no 2.3-only fields) "3.0" (overridden) 3.0 Warn (Profile override) [1]; session starts on 3.0 Warn (Profile override) [1] + Warn (PySpark) [5]
6 Unset "2.3" (with staging_bucket) "3.0" (overridden) Fails on server Warn (Profile override) [1] + Error [2] Warn (Profile override) [1] + Warn (PySpark) [5] + Error [2]
7 Unset Unreadable (3.0 or 4.0 on server) "" 3.0 / 4.0 OK — starts with profile's version (checked via GetSession after CreateSession) OK on 4.0; on 3.0, Warn (PySpark) [5] (via GetSession)
8 Unset Unreadable (2.3 or "" on server) "" Fails fast after CreateSession Error [4] (aborts immediately via GetSession before VM startup) Error [4]
9 "3.0" (Any, or no profile) "3.0" 3.0 OK — no warning (explicit builder value wins silently; skips GetSessionTemplate when no exec env vars are set) Warn (PySpark) [5]
10 "4.0" (Any, or no profile) "4.0" 4.0 OK — no warning (unless profile sets a 4.0-incompatible field like staging_bucket) OK — no warning
11 "2.3" (or < 3.0) (Any, or no profile) Not sent Fails on client Error [3] (fails fast before CreateSession) Error [3]

When the default runtime changes to 4.0 (_DEFAULT_RUNTIME_VERSION = "4.0", [client] / [spark] = pyspark 4.2)

Rows 3, 4, 7, 8, 9, 10, 11 behave identically to the table above; only the rows that fall back to _DEFAULT_RUNTIME_VERSION (1, 2, 5, 6) shift from "3.0" to "4.0":

# Builder runtimeVersion() Profile runtime_config.version Sent on Session Effective Runtime Outcome with pyspark 4.2 Outcome with pyspark 4.0
1' Unset (No profile) "4.0" 4.0 OK — no warning (requires subnetwork [6]) OK — no warning (requires subnetwork [6])
2' Unset "" (unset on profile) "4.0" 4.0 OK — no warning (uses profile's subnetwork if set) OK — no warning
3 Unset "3.0" "" (preserved) 3.0 Warn (PySpark) [5] OK — no warning
5' Unset "2.3" (clean, with subnetwork) "4.0" (overridden) 4.0 Warn (Profile override) [1] (overridden with '4.0') Warn (Profile override) [1] (overridden with '4.0')
6' Unset "2.3" (with staging_bucket or no subnetwork) "4.0" (overridden) Fails on server Warn (Profile override) [1] + Error [2] Warn (Profile override) [1] + Error [2]

Exact Warning & Error Reference

Ref Kind Exact Message
[1] UserWarning (Pre-3.0 profile override) Runtime Profile '<profile>' specifies Managed Spark runtime version '2.3', which is not supported by google-cloud-spark-connect (minimum 3.0); overriding with runtime version '<default>'. Update the Runtime Profile to 3.0+, or install 'dataproc-spark-connect~=0.9.0' in a Python 3.11 environment to use runtime version '2.3'.
[2] ManagedSparkConnectException (Pre-3.0 profile override + CreateSession InvalidArgument) Error while creating Managed Spark Session: <server error, e.g. 3.0 runtime version does not support configuring Dataproc staging bucket via the staging_bucket field.> Note: Runtime Profile '<profile>' specifies runtime version '2.3', which was overridden because google-cloud-spark-connect requires runtime 3.0 or higher (install 'dataproc-spark-connect~=0.9.0' in a Python 3.11 environment to use runtime '2.3').
[3] ManagedSparkConnectException (Explicit < 3.0 runtimeVersion()) Specified 2.3 Managed Spark Runtime version is not supported by google-cloud-spark-connect, which requires runtime version 3.0 or higher. To use runtime 2.3 (Spark 3.5), install 'dataproc-spark-connect~=0.9.0' in a Python 3.11 environment.
[4] ManagedSparkConnectException (Unreadable < 3.0 or unversioned profile resolved by server) Runtime Profile '<profile>' resolved to Managed Spark runtime version '2.3.x' [or '2.2.x'], which is not supported by google-cloud-spark-connect (minimum 3.0). Pass .runtimeVersion('3.0') (or '4.0') to override the Runtime Profile's runtime version, or install 'dataproc-spark-connect~=0.9.0' in a Python 3.11 environment for runtime 2.3.
[5] UserWarning (PySpark newer than runtime) PySpark 4.2.1 is installed, but Managed Spark runtime 3.0 runs Spark 4.0. Spark Connect does not support clients newer than the server, so some operations will fail. Install a matching PySpark, e.g. pip install 'pyspark-client~=4.0.0', or choose a newer runtime version with runtimeVersion().
[6] ManagedSparkConnectException (Missing or dual-stack subnetwork on runtime 4.0) Error while creating Managed Spark Session: Subnetwork must be specified. Pass .subnetwork('<subnet>') (for example .subnetwork('default')), set MANAGED_SPARK_CONNECT_SUBNET, or configure a subnetwork in .runtimeProfile('<profile>'). (or for dual-stack: ... Managed Spark runtime 4.0 requires an IPv4-only subnetwork; pass one with .subnetwork('<subnet>') or MANAGED_SPARK_CONNECT_SUBNET.)
[7] UserWarning (Python minor mismatch for UDFs) Python version mismatch detected: Client is using Python <X.Y>, but Managed Spark runtime <R> uses Python <M.N>. This mismatch may cause issues with Python UDF (User Defined Function) compatibility. Consider using Python <M.N> for optimal UDF execution.

@dborowitz
dborowitz added this pull request to stack #235 November 11, 2026 17:51

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the google-cloud-spark-connect client to properly resolve and validate runtime versions specified in Dataproc Session Templates (Runtime Profiles). It adds logic to detect unsupported pre-3.0 runtime versions in templates, issue warnings, and override them with the default supported version. Additionally, it improves error messages with actionable suggestions, such as installing dataproc-spark-connect~=0.9.0 for runtime 2.3 compatibility, and includes comprehensive unit tests for these new behaviors. I have no additional feedback to provide as the changes are well-implemented and thoroughly tested.

@dborowitz
dborowitz force-pushed the dborowitz-version-compat branch 3 times, most recently from c7c7f37 to 5e9293e Compare November 11, 2026 19:05
@dborowitz
dborowitz requested review from medb and tim-u November 11, 2026 19:51

@medb medb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May we extract new functionality in its own py files where it makes sense logically, so session.py is more managed in size?

…rriding pre-3.0

Previously, Builder._get_session_config() unconditionally populated
session_config.runtime_config.version with ManagedSparkSession._DEFAULT_RUNTIME_VERSION
whenever it was unset on the builder, silently overriding any runtime version
configured on a Runtime Profile / SessionTemplate.

Extract SessionTemplate (Runtime Profile) resolution and execution-default
precedence into google/cloud/managed_spark_connect/_session_templates.py:
- Represent runtime version resolution with an immutable
  TemplateRuntimeResolution value object so Builder carries no cross-method
  mutable state.
- If the template specifies a supported 3.0+ runtime_config.version, leave
  session_config.runtime_config.version unset on CreateSessionRequest so the
  template's version is preserved, while running Python UDF and PySpark version
  compatibility checks against the template's version.
- If the template specifies a pre-3.0 runtime_config.version (e.g. 2.3), emit a
  UserWarning and override with the client's default runtime version (and
  append the override note if CreateSession subsequently fails with
  InvalidArgument, such as on a 2.3-only staging_bucket field).
- If the template is readable and its runtime_config.version is empty, populate
  the client's default runtime version silently so the server does not fall
  back to an unsupported pre-3.0 runtime.
- If GetSessionTemplate fails (e.g. missing dataproc.sessionTemplates.get),
  leave runtime_config.version unset rather than risking overriding a 3.0+
  template, and inspect GetSession immediately after create_session() returns
  (with post-wait fallback) so pre-3.0 or unversioned (server-default 2.2)
  resolved sessions fail fast before VM startup while 3.0+ sessions still run
  client compatibility checks.
- Include 'dataproc-spark-connect~=0.9.0' downgrade guidance in pre-3.0 runtime
  compatibility errors and warnings, and add unit tests in
  tests/unit/test_session_templates.py plus Builder wiring and live integration
  tests.
@dborowitz

Copy link
Copy Markdown
Contributor Author

May we extract new functionality in its own py files where it makes sense logically, so session.py is more managed in size?

Done, factored out _session_templaets.py for the template lookup and runtime version resolution.

@dborowitz
dborowitz force-pushed the dborowitz-version-compat branch from 5e9293e to 7edc81f Compare November 12, 2026 00:05
@dborowitz
dborowitz requested a review from medb November 13, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants