Skip to content

chore: vendor Spark Connect GetStatus and Cell Monitor proto stubs - #236

Closed
ptch314 wants to merge 1 commit into
mainfrom
feat/vendor-cell-monitor-protos
Closed

ptch314 wants to merge 1 commit into
mainfrom
feat/vendor-cell-monitor-protos

Conversation

@ptch314

@ptch314 ptch314 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Vendors minimal Apache Spark Connect GetStatus (spark_connect_status.proto) and Dataproc Cell Monitor (cell_monitor.proto) protocol buffer definitions and generated Python stubs under third_party/apache_spark/ in preparation for Spark Connect inline cell monitoring:

  • Compliance & provenance (third_party/apache_spark/): Adds LICENSE (Apache 2.0), upstream Apache Spark NOTICE, METADATA, and README.google.md.
  • Minimal proto footprint & zero DescriptorPool collisions:
    • Extracts only UserContext, GetStatusRequest, GetStatusResponse, and SparkConnectService.GetStatus into spark_connect_status.proto (avoiding ~15K lines of Spark Connect plan/expression protos already provided by pyspark-client / pyspark).
    • Registers spark_connect_status_pb2 in an isolated DescriptorPool and SymbolDatabase so spark.connect.* descriptors never collide with pyspark.sql.connect.proto.base_pb2 across PySpark 4.0, 4.1, and 4.2+ (under both upb and pure-Python protobuf backends).
    • Keeps dataproc.cellmonitor.v1 (cell_monitor_pb2) in the default pool so google.protobuf.Any Pack/Unpack type URLs remain wire-identical.
  • Private wheel namespace remapping (setup.py, MANIFEST.in): Remaps third_party/apache_spark into google.cloud.managed_spark_connect._third_party.apache_spark in built wheels (including LICENSE and NOTICE in package_data and grafting third_party/apache_spark into sdists), ensuring no top-level third_party/ directory is ever installed into site-packages/.
  • Regeneration script (scripts/regen_protos.py): Compiles and formats *_pb2.py stubs, supports --check freshness verification, and supports --ref refreshes.

Testing

  • pyink . --check & vermin --target=3.10-
  • python -m build --no-isolation & twine check --strict
  • Unit tests (tests/unit/test_vendored_protos.py & tests/unit/test_pypi_artifacts.py) verified against both pyspark-client~=4.0.0 and pyspark-client>=4.2.

Vendor minimal Spark Connect `GetStatus` (`spark_connect_status.proto`)
and Dataproc Cell Monitor (`cell_monitor.proto`) protocol buffer
definitions and generated Python stubs under `third_party/apache_spark/`:

- Include Apache 2.0 `LICENSE`, upstream Apache Spark `NOTICE`,
  `METADATA`, and `README.google.md` in `third_party/apache_spark/`.
- Remap `third_party/apache_spark` into the private package namespace
  `google.cloud.managed_spark_connect._third_party.apache_spark` in
  `setup.py` (packaging `LICENSE` and `NOTICE` in wheels and grafting
  `third_party/apache_spark` into sdists via `MANIFEST.in`).
- Add `scripts/regen_protos.py` (supporting offline compilation,
  `--check`, and `--ref` upstream refreshes) with isolated
  `DescriptorPool` / `SymbolDatabase` registration on
  `spark_connect_status_pb2` so `spark.connect.*` descriptors never
  collide with `pyspark.sql.connect.proto.base_pb2` across PySpark
  4.0-4.2+.
- Add unit and packaging tests covering `Any.Pack`/`Unpack` roundtrips,
  wire compatibility with PySpark, `DescriptorPool` isolation across
  import orders and protobuf backends, and wheel/sdist layout.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request vendors the Apache Spark Connect GetStatus and Dataproc Cell Monitor protocol buffer definitions and generated Python stubs, adding a regeneration script, setup configuration updates, and comprehensive unit tests. The review feedback suggests several improvements to the robustness of the proto parsing and path resolution logic, such as handling leading whitespace and new-line braces in the parser, ensuring pyink is present for formatting, and guarding against IndexError when resolving parent directories.

Comment thread scripts/regen_protos.py
Comment on lines +137 to +145
depth = 0
end_idx = None
for idx in range(start_idx, len(lines)):
# Strip // line comments before counting braces.
code_part = lines[idx].split("//", 1)[0]
depth += code_part.count("{") - code_part.count("}")
if depth == 0:
end_idx = idx
break

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If the upstream proto file is reformatted such that the opening brace { of a message is on a new line, the brace counting loop will immediately exit on the first line because depth remains 0. To prevent this, we should only check if depth == 0 after we have actually seen the opening brace of the message block.

    depth = 0
    end_idx = None
    started = False
    for idx in range(start_idx, len(lines)):
        # Strip // line comments before counting braces.
        code_part = lines[idx].split("//", 1)[0]
        if "{" in code_part:
            started = True
        depth += code_part.count("{") - code_part.count("}")
        if started and depth == 0:
            end_idx = idx
            break

Comment on lines +22 to +28
_HERE = Path(__file__).resolve().parent
_REPO_THIRD_PARTY = _HERE.parents[3] / "third_party"
if (
not (_HERE / "apache_spark").is_dir()
and (_REPO_THIRD_PARTY / "apache_spark").is_dir()
):
__path__.append(str(_REPO_THIRD_PARTY))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Accessing _HERE.parents[3] directly can raise an IndexError if the package is loaded in certain environments (such as frozen packages, custom importers, or shallow directory structures) where the resolved path has fewer than 4 parents. It is safer to check the length of _HERE.parents before accessing it.

Suggested change
_HERE = Path(__file__).resolve().parent
_REPO_THIRD_PARTY = _HERE.parents[3] / "third_party"
if (
not (_HERE / "apache_spark").is_dir()
and (_REPO_THIRD_PARTY / "apache_spark").is_dir()
):
__path__.append(str(_REPO_THIRD_PARTY))
_HERE = Path(__file__).resolve().parent
if len(_HERE.parents) > 3:
_REPO_THIRD_PARTY = _HERE.parents[3] / "third_party"
if (
not (_HERE / "apache_spark").is_dir()
and (_REPO_THIRD_PARTY / "apache_spark").is_dir()
):
__path__.append(str(_REPO_THIRD_PARTY))

Comment thread scripts/regen_protos.py
def _extract_proto_block(proto_text: str, message_name: str) -> str:
"""Extracts a top-level message block with its leading comment lines."""
lines = proto_text.splitlines()
decl_re = re.compile(rf"^message\s+{re.escape(message_name)}\s*\{{")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The regular expression ^message\s+... assumes that the message declaration is at the very beginning of the line with no leading whitespace. If the upstream proto file is reformatted with leading spaces or tabs, this regex will fail to match. Allowing optional leading whitespace makes the parser more robust.

Suggested change
decl_re = re.compile(rf"^message\s+{re.escape(message_name)}\s*\{{")
decl_re = re.compile(rf"^\s*message\s+{re.escape(message_name)}\s*\{{")

Comment thread scripts/regen_protos.py
Comment on lines +385 to +391
if cmd is not None:
subprocess.run(
[*cmd, *path_list],
check=True,
capture_output=True,
text=True,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If pyink is not installed, the script silently skips formatting. This can lead to confusing mismatches or test failures when running regen_protos.py --check (comparing unformatted generated code with formatted checked-in code). It is better to raise a clear error message explaining that pyink is required.

Suggested change
if cmd is not None:
subprocess.run(
[*cmd, *path_list],
check=True,
capture_output=True,
text=True,
)
if cmd is None:
raise SystemExit(
"pyink is required to format the generated proto stubs. "
"Install it with: pip install pyink"
)
subprocess.run(
[*cmd, *path_list],
check=True,
capture_output=True,
text=True,
)

@ptch314

ptch314 commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by GoogleCloudDataproc/managed-spark-cell-monitoring#54 (moving vendored protos + GetStatus poller into managed-spark-cell-monitoring, updated to CL 196087 PS 8).

@ptch314 ptch314 closed this Nov 16, 2026
@ptch314
ptch314 deleted the feat/vendor-cell-monitor-protos branch November 16, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant