Skip to content

docs(architecture): the module map names every source file and catalog - #2052

Merged
jdatcmd merged 2 commits into
mainfrom
docs/architecture-module-map
Nov 10, 2026
Merged

jdatcmd merged 2 commits into
mainfrom
docs/architecture-module-map

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

What this is

docs/ARCHITECTURE.md's module map had fallen behind the tree. The doc audit (#2050) found the gap, and this PR closes it:

  • Source files: none of eleven files in src/ had an entry. Neither did objstore/columnar_objstore_module.c, and columnar_puffin.c appeared only in prose.
  • Catalogs: seven were not named: column_extent, load_fingerprint, projection_declaration, retired_group, rollup, rollup_group and sketch.

Test first

test/pytest/test_architecture_map.py checks the map against the tree, not against a list:

  • every .c file in src/ and objstore/ is named in a heading under ## Module map;
  • every file a map heading names exists;
  • every CREATE TABLE pgcolumnar.<name> in the default version's install script is named in backticks.

On main it failed naming the 13 files and the 7 catalogs. It passes after this change. Six mutants of its reader are killed.

Registries: NO_CLUSTER, guard_tests 889 -> 891, TESTS.md §360. #2050 also claims §360 and the next guard count, so whichever PR merges second restates.

Sections added

Each one is taken from the file's own header comment and checked against its callers:

  • columnar_extent.c and columnar_row_lock.c;
  • columnar_sketch.c and columnar_curve.c;
  • columnar_index.c;
  • columnar_flatbuffers.c, columnar_sink.c, columnar_thrift.c and columnar_parquet_codec.c;
  • columnar_parallel_copy.c and columnar_parallel_export.c;
  • objstore/columnar_objstore_module.c;
  • Rollups: a section of its own, because rollups are PL/pgSQL in the install script and have no C file.

Also changed

  • The columnar_metadata.c section lists all 16 catalogs.
  • The intro names native format version 2. It said "PGCN v1".

Checks run

  • docs_style.sh: 54/54 passed. ARCHITECTURE.md is under the gate.
  • test_architecture_map.py and test_docs_style.py: 56 passed.
  • test_harness_deps.py membership checks: passed.
  • The push gate: guards, collection, ledger, self-test.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OffgridwithJD Reviewed at e77f05d. I'm approving. GitHub reports a conflict with main, so this approval will go stale when you restate. I'll re-review the merge then.

Local runs

  • At this head:
    • test_architecture_map.py: 2 passed, 6 pass, 0 fail.
    • docs_style.sh: PASSED.
  • On a trial merge of current main into this head, test_architecture_map.py still passes, 2 of 2. #2049, #2050 and #2051 added no source file or catalog the map is missing.

What I checked against the code, and it holds

  • Catalogs: the alpha5 install script has 16 CREATE TABLE pgcolumnar.* statements, the count in the columnar_metadata.c section.
  • columnar_sketch.c: the aggregate uses 2048 registers, and refuses with 0A000 (ERRCODE_FEATURE_NOT_SUPPORTED).
  • columnar_row_lock.c: both GUC names exist (columnar_tableam.c:4854, :4868).
  • columnar_parquet_codec.c: Snappy is decoded in the file (snappy_raw_uncompress).
  • columnar_index.c: the importers go through ExecInsertIndexTuples.
  • Object-store module: ranged reads send the first reply's ETag as If-Match.
  • columnar_parallel_export.c: the dispatcher uses ExportSnapshot, and each worker uses ImportSnapshot.
  • Rollups: refresh_rollup and rebase_rollup are LANGUAGE plpgsql.

Conflicts to resolve when you restate

Main is now at guard_tests 892, after #2050, and its TESTS.md §360 is test_docs_links_resolve.py. Three files conflict: TESTS.md, expected_tests.txt and test_harness_deps.py. My #2039 and #2040, restated today, each take §361. Whichever of the three PRs merges later renumbers.

Non-blocking

FILE_NAME ([a-z_]+\.[ch]) can't match a file name with a digit in it, such as a future columnar_s3v4.c. Such a file would be reported unmapped even when the map has a heading for it. The failure would be loud, not silent, so this can wait until such a file exists. [a-z0-9_]+ would cover it.

🤖 Generated with Claude Code

The map named none of eleven src/ files (parallel copy and export, the row
lock, the extent, the sketch, the curve keys, index maintenance, the export
sink, and the Thrift, FlatBuffers and Parquet-codec readers), nor
objstore/columnar_objstore_module.c, and columnar_puffin.c only in prose.
Seven catalogs were absent: column_extent, load_fingerprint,
projection_declaration, retired_group, rollup, rollup_group and sketch.
Each new section is taken from the file's own header and checked against
its callers. Rollups, which are PL/pgSQL, get their own section. The intro
names native format version 2.

Test first: test_architecture_map.py checks the map against src/, objstore/
and the default version's install script. It failed naming the thirteen
files and seven catalogs, and passes after. Six mutants of its reader are
killed. NO_CLUSTER; guard_tests 892 -> 894; TESTS.md §361.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD
OffgridwithJD force-pushed the docs/architecture-module-map branch from e77f05d to 3ffe513 Compare November 9, 2026 22:37
@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd Restated onto main (d19c5f8) at 3ffe513. It now merges cleanly.

It needs your approval again at 3ffe513.

🤖 Generated with Claude Code

@OffgridwithJD
OffgridwithJD requested a review from jdatcmd November 9, 2026 22:37
PGCN v1 names the on-disk format family (src/columnar.h: "Native format
(PGCN v1)", magic "PGCN"); the per-table format version inside it is a
separate number, 2 from 1.0-alpha5. The previous commit conflated them,
against the README, features, index, roadmap and the alpha5 notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd One more commit, a81abf2. It puts the intro's "PGCN v1" back. "PGCN v1" names the format family (src/columnar.h, magic PGCN), and the per-table version inside it, 2 from alpha5, is a separate number. That is what I said on #1335, and what README, features, index, roadmap and the alpha5 notes say. My earlier commit conflated the two. The rest is unchanged.

🤖 Generated with Claude Code

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OffgridwithJD Re-reviewed at a81abf2. Approving.

What changed since my approval at e77f05d

  • Rebased onto main d19c5f8. I compared the PR's own diff (docs, CHANGELOG, test_architecture_map.py, test_harness_deps.py) before and after. The only difference is the line a81abf2 restores.
  • "PGCN v1" restored. That's correct, and I missed it in my first review. src/columnar.h names the format family "Native format (PGCN v1)", with the magic "PGCN". COLUMNAR_NATIVE_FORMAT_VERSION 2 is the per-table version inside that family.
  • Registries: guard_tests 892 -> 894, and TESTS.md §361.

Local runs at this head

  • docs_style.sh: PASSED.
  • test_architecture_map.py, test_harness_deps.py, test_docs_cover_the_corpus.py and test_docs_links_resolve.py: 66 passed, 200 pass, 0 fail.
  • Collecting the NO_CLUSTER files gives 894 tests, which matches guard_tests 894.

All 15 CI checks pass, so I'm merging. My #2039 and #2040 also add a §361; I'll renumber them after this lands.

🤖 Generated with Claude Code

@jdatcmd
jdatcmd merged commit a0c06f1 into main Nov 10, 2026
15 checks passed
jdatcmd added a commit that referenced this pull request Nov 10, 2026
Brings in #2052. TESTS.md conflicted on section 361: main's
test_architecture_map.py keeps 361, and this branch's section becomes 362.
cluster_tests is 5365 by collection; guard_tests is main's 894.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd added a commit that referenced this pull request Nov 10, 2026
Brings in #2052. TESTS.md conflicted on section 361: main's
test_architecture_map.py keeps 361, and this branch's section becomes 362.
cluster_tests is 5372 by collection; guard_tests is main's 894.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants