Repository navigation
Write Parquet files for the mocked test datasets - #929
Merged
Merged
Conversation
The dataset mocker now writes each node and edge entity as Parquet next to its TFRecords, with the row-count manifest, and sets parquet_uri_prefix in the preprocessed metadata. The Spark components keep reading the TFRecords. The GLT loaders prefer Parquet, so tests that load mocked datasets go through ParquetDataLoader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user-defined-edges toy dataset hardcodes positive/negative edges over node ids 0-26, but #68 repointed the shared toy graph config at the 14-node notebook graph, so regeneration failed with KeyError: 24. Point that dataset back at the 27-node toy_graph_data.yaml, which reproduces the previously published asset's nodes and edges. Update the mocked dataset artifact metadata to the regenerated version. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-assets # Conflicts: # CHANGELOG.md
yliu2-sc
approved these changes
Nov 20, 2026
mkolodner-sc
approved these changes
Nov 20, 2026
kmontemayor2-sc
marked this pull request as ready for review
November 20, 2026 23:13
kmontemayor2-sc
requested review from
svij-sc,
xgao4-sc and
zfan3-sc
as code owners
November 20, 2026 23:13
kmontemayor2-sc
enabled auto-merge
November 20, 2026 23:14
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 20, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 21, 2026
The end-of-file-fixer lint hook failed in the merge queue because mocked_dataset_artifact_metadata.json had no final newline. The metadata writer now adds the newline, so regeneration keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GLT loaders now prefer Parquet, but the mocked test datasets only held TFRecords, so tests that load them could not
exercise
ParquetDataLoader. The dataset mocker now also writes each node and edge entity as Parquet and setsparquet_uri_prefixin the preprocessed metadata. The Spark components keep reading the TFRecords.All 10 mocked datasets are regenerated under one new version,
2026-11-20--04-55-51-UTC.Approach
parquet/directory plus the row-count manifest, using the samelayout as the data preprocessor's Parquet output.
generate_preprocessed_tfrecord_datais renamedgenerate_preprocessed_data, since it now writes both formats.toy_graph_homogeneous_node_anchor_lp_user_defined_edges. It hardcodespositive/negative edges over node ids 0-26, but the shared toy graph config had been repointed at the 14-node
notebook graph, so the run failed with
KeyError: 24. That dataset now reads the 27-nodemocking_assets/toy_graph_data.yaml. The published 2024 asset has the same 27 nodes and 21 main edges and thesame 11 positive and 14 negative edges, so the content is unchanged.
User-facing changes
mocked_dataset_artifact_metadata.jsonpoints all 10 datasets at the new version, so every test that loads a mockeddataset now reads the regenerated assets.
ParquetDataLoader. Spark tests still read TFRecords.Verification
make mock_assetsgenerated 9 datasets before failing on the toy user-defined-edges dataset. After the graphsource fix, that dataset was regenerated alone to the same version folder and completed.