Skip to content

Write Parquet files for the mocked test datasets - #929

Merged
kmontemayor2-sc merged 7 commits into
mainfrom
kmonte/parquet-mocked-assets
Nov 21, 2026
Merged

kmontemayor2-sc merged 7 commits into
mainfrom
kmonte/parquet-mocked-assets

Conversation

@kmontemayor2-sc

@kmontemayor2-sc kmontemayor2-sc commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

GLT loaders now prefer Parquet, but the mocked test datasets only held TFRecords, so tests that load them could not
exercise ParquetDataLoader. The dataset mocker now also writes each node and edge entity as Parquet and sets
parquet_uri_prefix in the preprocessed metadata. The Spark components keep reading the TFRecords.

All 10 mocked datasets are regenerated under one new version, 2026-11-20--04-55-51-UTC.

Approach

  • Next to each TFRecord directory, the mocker writes a parquet/ directory plus the row-count manifest, using the same
    layout as the data preprocessor's Parquet output.
  • generate_preprocessed_tfrecord_data is renamed generate_preprocessed_data, since it now writes both formats.
  • Regeneration exposed a latent bug in toy_graph_homogeneous_node_anchor_lp_user_defined_edges. It hardcodes
    positive/negative edges over node ids 0-26, but the shared toy graph config had been repointed at the 14-node
    notebook graph, so the run failed with KeyError: 24. That dataset now reads the 27-node
    mocking_assets/toy_graph_data.yaml. The published 2024 asset has the same 27 nodes and 21 main edges and the
    same 11 positive and 14 negative edges, so the content is unchanged.

User-facing changes

  • mocked_dataset_artifact_metadata.json points all 10 datasets at the new version, so every test that loads a mocked
    dataset now reads the regenerated assets.
  • GLT tests load mocked datasets through ParquetDataLoader. Spark tests still read TFRecords.

Verification

  • make mock_assets generated 9 datasets before failing on the toy user-defined-edges dataset. After the graph
    source fix, that dataset was regenerated alone to the same version folder and completed.

kmonte and others added 2 commits November 19, 2026 17:27
The dataset mocker now writes each node and edge entity as Parquet next to
its TFRecords, with the row-count manifest, and sets parquet_uri_prefix in
the preprocessed metadata. The Spark components keep reading the TFRecords.
The GLT loaders prefer Parquet, so tests that load mocked datasets go
through ParquetDataLoader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user-defined-edges toy dataset hardcodes positive/negative edges over
node ids 0-26, but #68 repointed the shared toy graph config at the
14-node notebook graph, so regeneration failed with KeyError: 24. Point
that dataset back at the 27-node toy_graph_data.yaml, which reproduces
the previously published asset's nodes and edges.

Update the mocked dataset artifact metadata to the regenerated version.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kmontemayor2-sc kmontemayor2-sc changed the title Kmonte/parquet mocked assets Write Parquet files for the mocked test datasets Nov 20, 2026
@kmontemayor2-sc
kmontemayor2-sc marked this pull request as ready for review November 20, 2026 23:13
@kmontemayor2-sc
kmontemayor2-sc added this pull request to the merge queue Nov 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 20, 2026
@kmontemayor2-sc
kmontemayor2-sc added this pull request to the merge queue Nov 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 21, 2026
The end-of-file-fixer lint hook failed in the merge queue because
mocked_dataset_artifact_metadata.json had no final newline.
The metadata writer now adds the newline, so regeneration keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kmontemayor2-sc
kmontemayor2-sc added this pull request to the merge queue Nov 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 21, 2026
@kmontemayor2-sc
kmontemayor2-sc added this pull request to the merge queue Nov 21, 2026
Merged via the queue into main with commit be6fea3 Nov 21, 2026
8 checks passed
@kmontemayor2-sc
kmontemayor2-sc deleted the kmonte/parquet-mocked-assets branch November 21, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants