Repository navigation
Conversation
Add WindowSpec, DocumentWindowFrame, RangeWindowFrame, and WindowBound types in pipeline_types, the AddWindowFields stage and add_window_fields() pipeline builder method, and the Rank() window function and Expression.over() frame override method. Include unit tests and pipeline_e2e acceptance tests in window.yaml.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces window functions and window framing capabilities to the Google Cloud Firestore Python client library, enabling advanced analytical operations such as running totals and rankings. It adds support for the add_window_fields pipeline stage, along with new classes like DocumentWindowFrame, RangeWindowFrame, WindowSpec, and Rank. Feedback on the changes highlights a potential bug in WindowSpec initialization where passing a string to the sort parameter would silently convert it into a list of characters, leading to serialization errors. A validation check has been suggested to prevent this issue.
| if sort is None: | ||
| self.sort: list[Ordering] = [] | ||
| else: | ||
| self.sort = [sort] if isinstance(sort, Ordering) else list(sort) |
There was a problem hiding this comment.
If sort is a string (e.g., "rating"), isinstance(sort, Ordering) is False, so list(sort) is called. Since strings are iterable in Python, this silently converts "rating" into a list of its characters: ['r', 'a', 't', 'i', 'n', 'g']. This will later cause an AttributeError during serialization because str objects do not have a _to_pb() method.
To prevent this silent bug and provide a clear error message, we should validate that all elements in the sort sequence are instances of Ordering.
| if sort is None: | |
| self.sort: list[Ordering] = [] | |
| else: | |
| self.sort = [sort] if isinstance(sort, Ordering) else list(sort) | |
| if sort is None: | |
| self.sort: list[Ordering] = [] | |
| else: | |
| sort_list = [sort] if isinstance(sort, Ordering) else list(sort) | |
| for s in sort_list: | |
| if not isinstance(s, Ordering): | |
| raise TypeError(f"Expected Ordering, got {type(s).__name__}") | |
| self.sort = sort_list |
ohmayr
changed the title
…rser Look up pipeline option and window frame classes (such as DocumentWindowFrame and RangeWindowFrame) directly in pipeline_types within _parse_expressions and _is_stage_string.
Define over() on WindowFunction instead of Expression and make AggregateFunction inherit from WindowFunction. This restricts over() to window and aggregate functions and simplifies AddWindowFields type annotations. Also sort imports in test_pipeline_acceptance.py.
Convert DocumentWindowFrame and RangeWindowFrame to dataclasses inheriting from a shared _WindowFrame base class to eliminate duplicated __init__, __repr__, __eq__, and _to_pb boilerplate.
Introduce _OverExpression as a WindowFunction subclass to bind a WindowFunction to a WindowSpec without passing WindowSpec into FunctionExpression.params or suppressing mypy list-item checks. Make AliasedExpression type variable T covariant and preserve T in Expression.as_() so AliasedExpression[AggregateFunction] is assignable to AliasedExpression[WindowFunction].
…mantics Split test_over_and_rank into test_over and test_rank, remove redundant dataclass boilerplate assertions, and apply .over() frame overrides to aggregate functions rather than Rank() to match Firestore backend rules.
Split multi-scenario window expression, type, and stage tests into focused single-purpose test methods. Remove dataclass field-echo and private attribute assertions, cover TimeGranularity in RangeWindowFrame, and import WindowSpec from pipeline_types.
Separate rank with ties (sorted by rating ASC to exercise tie ranking) from accumulator-level over() frames (sorted by unique published ASC to avoid positional tie ambiguity and year-sum math) in tests/system/pipeline_e2e/window.yaml. Remove the redundant expression-partition window system test whose permutations are already covered by unit tests, and update test_over_repr_and_equality to pass a valid frame-only WindowSpec.
Context & Problem
Firestore Pipelines supports evaluating window functions and aggregations over partitioned, sorted, and framed subsets of documents without collapsing rows (
add_window_fields). This PR adds support for pipeline window functions to the Python Firestore SDK.Summary of Changes
WindowBound,DocumentWindowFrame,RangeWindowFrame, andWindowSpecingoogle/cloud/firestore_v1/pipeline_types.pyand re-export them fromgoogle.cloud.firestoreandgoogle.cloud.firestore_v1(including.librarian/generator-input/client-post-processing/firestore-integration.yaml).AddWindowFieldsstage ingoogle/cloud/firestore_v1/pipeline_stages.pyandadd_window_fields()builder method on_BasePipelineingoogle/cloud/firestore_v1/base_pipeline.py.WindowFunction,Rank(), andExpression.over()ingoogle/cloud/firestore_v1/pipeline_expressions.py.test_pipeline_stages.py,test_pipeline_expressions.py,test_pipeline.py, andtest_async_pipeline.py, along with end-to-end acceptance tests intests/system/pipeline_e2e/window.yaml.Verification
pytest tests/unit/v1/test_pipeline_stages.py tests/unit/v1/test_pipeline_expressions.py tests/unit/v1/test_pipeline.py tests/unit/v1/test_async_pipeline.py) with 100% line and branch coverage on modified lines.tests/system/test_pipeline_acceptance.py -k window.yaml(sync and async) against the Firestore nightly sandbox backend (test-firestore.sandbox.googleapis.com:443).