Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the google-cloud-spark-connect client to properly resolve and validate runtime versions specified in Dataproc Session Templates (Runtime Profiles). It adds logic to detect unsupported pre-3.0 runtime versions in templates, issue warnings, and override them with the default supported version. Additionally, it improves error messages with actionable suggestions, such as installing dataproc-spark-connect~=0.9.0 for runtime 2.3 compatibility, and includes comprehensive unit tests for these new behaviors. I have no additional feedback to provide as the changes are well-implemented and thoroughly tested.
c7c7f37 to
5e9293e
Compare
medb
left a comment
There was a problem hiding this comment.
May we extract new functionality in its own py files where it makes sense logically, so session.py is more managed in size?
…rriding pre-3.0 Previously, Builder._get_session_config() unconditionally populated session_config.runtime_config.version with ManagedSparkSession._DEFAULT_RUNTIME_VERSION whenever it was unset on the builder, silently overriding any runtime version configured on a Runtime Profile / SessionTemplate. Extract SessionTemplate (Runtime Profile) resolution and execution-default precedence into google/cloud/managed_spark_connect/_session_templates.py: - Represent runtime version resolution with an immutable TemplateRuntimeResolution value object so Builder carries no cross-method mutable state. - If the template specifies a supported 3.0+ runtime_config.version, leave session_config.runtime_config.version unset on CreateSessionRequest so the template's version is preserved, while running Python UDF and PySpark version compatibility checks against the template's version. - If the template specifies a pre-3.0 runtime_config.version (e.g. 2.3), emit a UserWarning and override with the client's default runtime version (and append the override note if CreateSession subsequently fails with InvalidArgument, such as on a 2.3-only staging_bucket field). - If the template is readable and its runtime_config.version is empty, populate the client's default runtime version silently so the server does not fall back to an unsupported pre-3.0 runtime. - If GetSessionTemplate fails (e.g. missing dataproc.sessionTemplates.get), leave runtime_config.version unset rather than risking overriding a 3.0+ template, and inspect GetSession immediately after create_session() returns (with post-wait fallback) so pre-3.0 or unversioned (server-default 2.2) resolved sessions fail fast before VM startup while 3.0+ sessions still run client compatibility checks. - Include 'dataproc-spark-connect~=0.9.0' downgrade guidance in pre-3.0 runtime compatibility errors and warnings, and add unit tests in tests/unit/test_session_templates.py plus Builder wiring and live integration tests.
Done, factored out _session_templaets.py for the template lookup and runtime version resolution. |
5e9293e to
7edc81f
Compare
Previously,
ManagedSparkSession.Builder._get_session_config()unconditionally populatedsession_config.runtime_config.versionwithManagedSparkSession._DEFAULT_RUNTIME_VERSION("3.0") whenever no runtime version was set on the builder orSessionconfig. BecauseCreateSessionmerges the session request on top of the referenced Runtime Profile (SessionTemplate), that client default silently overwrote anyruntime_config.versionconfigured on the Runtime Profile:4.0was silently downgraded back to"3.0"unless.runtimeVersion("4.0")was also duplicated in code.3.0runtime (such as"2.3") was silently evaluated as"3.0", either starting on a different runtime than configured without notice or failingCreateSessionvalidation with confusing"3.0 runtime version does not support ..."errors on2.3-only fields such asstaging_bucket.Changes
Extract Runtime Profile (
SessionTemplate) lookup, runtime-version resolution, andMANAGED_SPARK_CONNECT_*execution-default precedence out ofsession.pyintogoogle/cloud/managed_spark_connect/_session_templates.py(modeled around an immutableTemplateRuntimeResolutionvalue object soBuildercarries no cross-method mutable state andsession.pyshrinks net relative to base):When
.runtimeProfile()/.sessionTemplate()is set and no explicit runtime version is configured on the builder orSessionproto, fetch theSessionTemplatebest-effort (reusing the singleGetSessionTemplatecall already used forMANAGED_SPARK_CONNECT_*execution environment variable precedence):3.0+Runtime Profile version ("3.0","4.0", ...): Leavesession.runtime_config.versionunset ("") onCreateSessionRequestso the profile's runtime version is preserved, and run_check_python_version_compatibility()and_check_pyspark_version_compatibility()against the profile's runtime version.3.0Runtime Profile version (e.g.,"2.3"): Emit aUserWarningexplaining thatgoogle-cloud-spark-connectrequires runtime3.0or higher (with'dataproc-spark-connect~=0.9.0'guidance for Spark3.5/ runtime2.3) and overridesession.runtime_config.versionwith_DEFAULT_RUNTIME_VERSION. IfCreateSessionsubsequently fails withInvalidArgument(for example, on a2.3-onlystaging_bucketfield), append the override context to theManagedSparkConnectExceptionmessage as well.""): Populatesession.runtime_config.versionwith_DEFAULT_RUNTIME_VERSIONsilently so the server does not fall back to an older pre-3.0default runtime.dataproc.sessionTemplates.get): Leavesession.runtime_config.versionunset ("") rather than risking clobbering a3.0+profile. Because the control plane acceptsspark_connect_sessionon2.2/2.3duringCreateSession, inspectGetSessionimmediately aftercreate_session()returns: if the server resolved< 3.0(e.g.,'2.3.x'or unversioned server-default'2.2.x'), initiate best-effort session termination and raiseManagedSparkConnectExceptionimmediately without waiting for VM startup; if it resolved3.0+, run the Python and PySpark compatibility checks against the resolved version.3.0.runtimeVersion(): Include'dataproc-spark-connect~=0.9.0'(Python3.11) guidance in_check_runtime_compatibility()when a pre-3.0runtime version is requested directly in code.tests/unit/test_session_templates.pyandtests/unit/test_session.py) and live control-plane integration tests covering3.0+profile preservation (with bare profile IDs),2.3warn-and-override,2.3+staging_bucketerror context, and unreadable< 3.0/ unversioned profile fail-fast handling.Builder × Runtime Profile × PySpark Matrix
Installed
pysparknever changes which runtime version is selected; it only determines whether_check_pyspark_version_compatibility()emits a warning after the effective runtime is resolved (pyspark 4.0is compatible with both3.0and4.0;pyspark 4.2warns [5] whenever the resolved runtime is3.0, since Spark Connect does not support clients newer than the server). Client Pythonmajor.minorindependently emits a UDF mismatch warning [7] whenever it differs from the resolved runtime's Python (3.12on3.0,3.13on4.0).Current default (
_DEFAULT_RUNTIME_VERSION = "3.0",[client]/[spark]=pyspark 4.0)Rows resolving to runtime
4.0require an IPv4-only subnetwork via.subnetwork(),MANAGED_SPARK_CONNECT_SUBNET, or the Runtime Profile (raisesManagedSparkConnectException[6] if missing or dual-stack).runtimeVersion()runtime_config.versionSessionpyspark 4.0pyspark 4.2"3.0"3.0""(unset on profile)"3.0"3.0"3.0"""(preserved)3.0"4.0"""(preserved)4.0"2.3"(no2.3-only fields)"3.0"(overridden)3.03.0"2.3"(withstaging_bucket)"3.0"(overridden)3.0or4.0on server)""3.0/4.0GetSessionafterCreateSession)4.0; on3.0, Warn (PySpark) [5] (viaGetSession)2.3or""on server)""CreateSessionGetSessionbefore VM startup)"3.0""3.0"3.0GetSessionTemplatewhen no exec env vars are set)"4.0""4.0"4.04.0-incompatible field likestaging_bucket)"2.3"(or< 3.0)CreateSession)When the default runtime changes to
4.0(_DEFAULT_RUNTIME_VERSION = "4.0",[client]/[spark]=pyspark 4.2)Rows 3, 4, 7, 8, 9, 10, 11 behave identically to the table above; only the rows that fall back to
_DEFAULT_RUNTIME_VERSION(1, 2, 5, 6) shift from"3.0"to"4.0":runtimeVersion()runtime_config.versionSessionpyspark 4.2pyspark 4.0"4.0"4.0""(unset on profile)"4.0"4.0"3.0"""(preserved)3.0"2.3"(clean, with subnetwork)"4.0"(overridden)4.0'4.0')'4.0')"2.3"(withstaging_bucketor no subnetwork)"4.0"(overridden)Exact Warning & Error Reference
UserWarning(Pre-3.0profile override)Runtime Profile '<profile>' specifies Managed Spark runtime version '2.3', which is not supported by google-cloud-spark-connect (minimum 3.0); overriding with runtime version '<default>'. Update the Runtime Profile to 3.0+, or install 'dataproc-spark-connect~=0.9.0' in a Python 3.11 environment to use runtime version '2.3'.ManagedSparkConnectException(Pre-3.0profile override +CreateSessionInvalidArgument)Error while creating Managed Spark Session: <server error, e.g. 3.0 runtime version does not support configuring Dataproc staging bucket via the staging_bucket field.> Note: Runtime Profile '<profile>' specifies runtime version '2.3', which was overridden because google-cloud-spark-connect requires runtime 3.0 or higher (install 'dataproc-spark-connect~=0.9.0' in a Python 3.11 environment to use runtime '2.3').ManagedSparkConnectException(Explicit< 3.0runtimeVersion())Specified 2.3 Managed Spark Runtime version is not supported by google-cloud-spark-connect, which requires runtime version 3.0 or higher. To use runtime 2.3 (Spark 3.5), install 'dataproc-spark-connect~=0.9.0' in a Python 3.11 environment.ManagedSparkConnectException(Unreadable< 3.0or unversioned profile resolved by server)Runtime Profile '<profile>' resolved to Managed Spark runtime version '2.3.x' [or '2.2.x'], which is not supported by google-cloud-spark-connect (minimum 3.0). Pass .runtimeVersion('3.0') (or '4.0') to override the Runtime Profile's runtime version, or install 'dataproc-spark-connect~=0.9.0' in a Python 3.11 environment for runtime 2.3.UserWarning(PySpark newer than runtime)PySpark 4.2.1 is installed, but Managed Spark runtime 3.0 runs Spark 4.0. Spark Connect does not support clients newer than the server, so some operations will fail. Install a matching PySpark, e.g. pip install 'pyspark-client~=4.0.0', or choose a newer runtime version with runtimeVersion().ManagedSparkConnectException(Missing or dual-stack subnetwork on runtime4.0)Error while creating Managed Spark Session: Subnetwork must be specified. Pass .subnetwork('<subnet>') (for example .subnetwork('default')), set MANAGED_SPARK_CONNECT_SUBNET, or configure a subnetwork in .runtimeProfile('<profile>').(or for dual-stack:... Managed Spark runtime 4.0 requires an IPv4-only subnetwork; pass one with .subnetwork('<subnet>') or MANAGED_SPARK_CONNECT_SUBNET.)UserWarning(Python minor mismatch for UDFs)Python version mismatch detected: Client is using Python <X.Y>, but Managed Spark runtime <R> uses Python <M.N>. This mismatch may cause issues with Python UDF (User Defined Function) compatibility. Consider using Python <M.N> for optimal UDF execution.