Repository navigation
Conversation
Vendor minimal Spark Connect `GetStatus` (`spark_connect_status.proto`) and Dataproc Cell Monitor (`cell_monitor.proto`) protocol buffer definitions and generated Python stubs under `third_party/apache_spark/`: - Include Apache 2.0 `LICENSE`, upstream Apache Spark `NOTICE`, `METADATA`, and `README.google.md` in `third_party/apache_spark/`. - Remap `third_party/apache_spark` into the private package namespace `google.cloud.managed_spark_connect._third_party.apache_spark` in `setup.py` (packaging `LICENSE` and `NOTICE` in wheels and grafting `third_party/apache_spark` into sdists via `MANIFEST.in`). - Add `scripts/regen_protos.py` (supporting offline compilation, `--check`, and `--ref` upstream refreshes) with isolated `DescriptorPool` / `SymbolDatabase` registration on `spark_connect_status_pb2` so `spark.connect.*` descriptors never collide with `pyspark.sql.connect.proto.base_pb2` across PySpark 4.0-4.2+. - Add unit and packaging tests covering `Any.Pack`/`Unpack` roundtrips, wire compatibility with PySpark, `DescriptorPool` isolation across import orders and protobuf backends, and wheel/sdist layout.
There was a problem hiding this comment.
Code Review
This pull request vendors the Apache Spark Connect GetStatus and Dataproc Cell Monitor protocol buffer definitions and generated Python stubs, adding a regeneration script, setup configuration updates, and comprehensive unit tests. The review feedback suggests several improvements to the robustness of the proto parsing and path resolution logic, such as handling leading whitespace and new-line braces in the parser, ensuring pyink is present for formatting, and guarding against IndexError when resolving parent directories.
| depth = 0 | ||
| end_idx = None | ||
| for idx in range(start_idx, len(lines)): | ||
| # Strip // line comments before counting braces. | ||
| code_part = lines[idx].split("//", 1)[0] | ||
| depth += code_part.count("{") - code_part.count("}") | ||
| if depth == 0: | ||
| end_idx = idx | ||
| break |
There was a problem hiding this comment.
If the upstream proto file is reformatted such that the opening brace { of a message is on a new line, the brace counting loop will immediately exit on the first line because depth remains 0. To prevent this, we should only check if depth == 0 after we have actually seen the opening brace of the message block.
depth = 0
end_idx = None
started = False
for idx in range(start_idx, len(lines)):
# Strip // line comments before counting braces.
code_part = lines[idx].split("//", 1)[0]
if "{" in code_part:
started = True
depth += code_part.count("{") - code_part.count("}")
if started and depth == 0:
end_idx = idx
break| _HERE = Path(__file__).resolve().parent | ||
| _REPO_THIRD_PARTY = _HERE.parents[3] / "third_party" | ||
| if ( | ||
| not (_HERE / "apache_spark").is_dir() | ||
| and (_REPO_THIRD_PARTY / "apache_spark").is_dir() | ||
| ): | ||
| __path__.append(str(_REPO_THIRD_PARTY)) |
There was a problem hiding this comment.
Accessing _HERE.parents[3] directly can raise an IndexError if the package is loaded in certain environments (such as frozen packages, custom importers, or shallow directory structures) where the resolved path has fewer than 4 parents. It is safer to check the length of _HERE.parents before accessing it.
| _HERE = Path(__file__).resolve().parent | |
| _REPO_THIRD_PARTY = _HERE.parents[3] / "third_party" | |
| if ( | |
| not (_HERE / "apache_spark").is_dir() | |
| and (_REPO_THIRD_PARTY / "apache_spark").is_dir() | |
| ): | |
| __path__.append(str(_REPO_THIRD_PARTY)) | |
| _HERE = Path(__file__).resolve().parent | |
| if len(_HERE.parents) > 3: | |
| _REPO_THIRD_PARTY = _HERE.parents[3] / "third_party" | |
| if ( | |
| not (_HERE / "apache_spark").is_dir() | |
| and (_REPO_THIRD_PARTY / "apache_spark").is_dir() | |
| ): | |
| __path__.append(str(_REPO_THIRD_PARTY)) |
| def _extract_proto_block(proto_text: str, message_name: str) -> str: | ||
| """Extracts a top-level message block with its leading comment lines.""" | ||
| lines = proto_text.splitlines() | ||
| decl_re = re.compile(rf"^message\s+{re.escape(message_name)}\s*\{{") |
There was a problem hiding this comment.
The regular expression ^message\s+... assumes that the message declaration is at the very beginning of the line with no leading whitespace. If the upstream proto file is reformatted with leading spaces or tabs, this regex will fail to match. Allowing optional leading whitespace makes the parser more robust.
| decl_re = re.compile(rf"^message\s+{re.escape(message_name)}\s*\{{") | |
| decl_re = re.compile(rf"^\s*message\s+{re.escape(message_name)}\s*\{{") |
| if cmd is not None: | ||
| subprocess.run( | ||
| [*cmd, *path_list], | ||
| check=True, | ||
| capture_output=True, | ||
| text=True, | ||
| ) |
There was a problem hiding this comment.
If pyink is not installed, the script silently skips formatting. This can lead to confusing mismatches or test failures when running regen_protos.py --check (comparing unformatted generated code with formatted checked-in code). It is better to raise a clear error message explaining that pyink is required.
| if cmd is not None: | |
| subprocess.run( | |
| [*cmd, *path_list], | |
| check=True, | |
| capture_output=True, | |
| text=True, | |
| ) | |
| if cmd is None: | |
| raise SystemExit( | |
| "pyink is required to format the generated proto stubs. " | |
| "Install it with: pip install pyink" | |
| ) | |
| subprocess.run( | |
| [*cmd, *path_list], | |
| check=True, | |
| capture_output=True, | |
| text=True, | |
| ) |
578a771 to
2f461be
Compare
|
Superseded by GoogleCloudDataproc/managed-spark-cell-monitoring#54 (moving vendored protos + GetStatus poller into managed-spark-cell-monitoring, updated to CL 196087 PS 8). |
Summary
Vendors minimal Apache Spark Connect
GetStatus(spark_connect_status.proto) and Dataproc Cell Monitor (cell_monitor.proto) protocol buffer definitions and generated Python stubs underthird_party/apache_spark/in preparation for Spark Connect inline cell monitoring:third_party/apache_spark/): AddsLICENSE(Apache 2.0), upstream Apache SparkNOTICE,METADATA, andREADME.google.md.DescriptorPoolcollisions:UserContext,GetStatusRequest,GetStatusResponse, andSparkConnectService.GetStatusintospark_connect_status.proto(avoiding ~15K lines of Spark Connect plan/expression protos already provided bypyspark-client/pyspark).spark_connect_status_pb2in an isolatedDescriptorPoolandSymbolDatabasesospark.connect.*descriptors never collide withpyspark.sql.connect.proto.base_pb2across PySpark 4.0, 4.1, and 4.2+ (under bothupband pure-Python protobuf backends).dataproc.cellmonitor.v1(cell_monitor_pb2) in the default pool sogoogle.protobuf.AnyPack/Unpacktype URLs remain wire-identical.setup.py,MANIFEST.in): Remapsthird_party/apache_sparkintogoogle.cloud.managed_spark_connect._third_party.apache_sparkin built wheels (includingLICENSEandNOTICEinpackage_dataand graftingthird_party/apache_sparkinto sdists), ensuring no top-levelthird_party/directory is ever installed intosite-packages/.scripts/regen_protos.py): Compiles and formats*_pb2.pystubs, supports--checkfreshness verification, and supports--refrefreshes.Testing
pyink . --check&vermin --target=3.10-python -m build --no-isolation&twine check --stricttests/unit/test_vendored_protos.py&tests/unit/test_pypi_artifacts.py) verified against bothpyspark-client~=4.0.0andpyspark-client>=4.2.