Skip to content

Classify CZDS malformed record structure without disclosing zone data - #8

Closed
tulik wants to merge 4 commits into
mainfrom
fix/czds-record-structure-codes
Closed

tulik wants to merge 4 commits into
mainfrom
fix/czds-record-structure-codes

Conversation

@tulik

@tulik tulik commented Oct 22, 2026

Copy link
Copy Markdown
Member

Context

The protected October 22 NSDB preflight of gov.txt.gz failed with reason=record_fields. PR #7 (already merged) kept sensitive exception messages out of public logs but did not distinguish record shapes.

Changes

  • Shared, deterministic classify_record_fields() for malformed records in build_nsdb.py and the preflight header check.
  • Fixed public reason codes: record_blank, record_fields_comment, record_fields_directive, record_fields_space_delimited, record_fields_missing_tabs, record_fields_partial_tabs.
  • New synthetic compressed-zone subprocess fixtures for each category, including malformed first records.
  • Preflight/full-build CLI parity checks and assertions that source names and RDATA do not appear in logs.
  • README documents that categories are hints, not evidence that whitespace master-file parsing is safe.

Safety

  • No acceptance rule changed: input still fails whenever it has fewer than five tab-separated fields.
  • No SOA consistency, canonical ordering, delegation deduplication, or incomplete-input rejection changes.
  • process_input() and the private aggregation interface are unchanged for issue Resume CZDS aggregation from verified per-zone checkpoints #6.
  • Daily-release workflow unchanged; failing input still blocks publication.
  • The licensed gov.txt.gz record is unavailable to public CI; the next production preflight must reveal the new structural reason before a semantic parser correction is considered.

Follow-up to #7; related to #6.

tulik commented Oct 22, 2026

Copy link
Copy Markdown
Member Author

Conflict resolution: superseded by merged PR #9 (commit 78669f1757086155d2922ac69840c2868d98769b).

I compared the four modified paths in this PR against current main. The shared first-record/streaming classify_record_fields() implementation, public fixed diagnostic codes, privacy assertions, and compressed synthetic-fixture parity checks are already present on main. The scripts/preflight_czds.py blob is identical. main additionally covers record_fields_mixed_delimiters; its tests and documentation reflect that case.

GitHub reports this PR as mergeable_state=dirty because it overlaps the merged changes. Re-merging an older competing classifier would add no necessary behavior and risks regressing the newer test coverage. Closing as superseded, with the branch left intact for historical reference. This does not change production validation or publish private CZDS data.

Verification: merged PR #9 CI passed 49 tests, and CI on main passed. The protected runner must still rerun preflight to identify the actual .gov structural subcategory.

@tulik tulik closed this Oct 22, 2026
@tulik
tulik deleted the fix/czds-record-structure-codes branch October 22, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant