Temporarily exclude .gov with explicit partial-coverage release markers

.github/workflows/daily-release.yml

@@ -26,6 +26,7 @@ jobs:
       ICANN_USERNAME: ${{ secrets.ICANN_USERNAME }}
       ICANN_PASSWORD: ${{ secrets.ICANN_PASSWORD }}
       NSDB_STATE_DIR: ${{ vars.NSDB_STATE_DIR || '/var/lib/nsdb' }}
+      NSDB_TEMP_EXCLUDE_ZONE: gov
 
     steps:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
@@ -69,7 +70,8 @@ jobs:
         run: |
           python scripts/preflight_czds.py \
             --zones-dir "$NSDB_STATE_DIR/zones/$DAY" \
-            --special-zones config/locally-served-dns-zones.txt
+            --special-zones config/locally-served-dns-zones.txt \
+            --exclude-zone "$NSDB_TEMP_EXCLUDE_ZONE"
 
       - name: Build private exact aggregate
         env:
@@ -78,6 +80,7 @@ jobs:
           python scripts/build_nsdb.py \
             --zones-dir "$NSDB_STATE_DIR/zones/$DAY" \
             --special-zones config/locally-served-dns-zones.txt \
+            --exclude-zone "$NSDB_TEMP_EXCLUDE_ZONE" \
             --private-counts "$NSDB_STATE_DIR/private/$DAY/nameserver-counts.txt" \
             --coverage "$NSDB_STATE_DIR/private/$DAY/coverage.txt" \
             --counts "$NSDB_STATE_DIR/private/$DAY/counts.txt"
@@ -194,10 +197,29 @@ jobs:
           DAY: ${{ steps.date.outputs.day }}
           PREVIOUS_TAG: ${{ steps.previous.outputs.tag }}
         run: |
+          COVERAGE_STATUS="$(sed -n 's/^coverage_complete=//p' dist/counts.txt)"
+          EXCLUDED_ZONES="$(sed -n 's/^excluded_zones=//p' dist/counts.txt)"
+          case "$COVERAGE_STATUS:$EXCLUDED_ZONES" in
+            true:none)
+              RELEASE_TITLE="NSDB snapshot $DAY"
+              COVERAGE_LABEL="Complete approved CZDS manifest"
+              ;;
+            false:gov)
+              RELEASE_TITLE="NSDB snapshot $DAY (PARTIAL: .gov excluded)"
+              COVERAGE_LABEL="PARTIAL - .gov excluded"
+              ;;
+            *)
+              echo "Invalid release coverage metadata" >&2
+              exit 1
+              ;;
+          esac
+
           NOTES="$(cat <<EOF
           Daily ICANN CZDS nameserver delegation aggregate.
 
           UTC snapshot date: $DAY
+          Coverage: $COVERAGE_LABEL
+          Excluded zones: $EXCLUDED_ZONES
           Compared with: $PREVIOUS_TAG
 
           Snapshot counts:
@@ -209,11 +231,11 @@ jobs:
           )"
 
           if gh release view "$TAG" >/dev/null 2>&1; then
-            gh release edit "$TAG" --notes "$NOTES"
+            gh release edit "$TAG" --title "$RELEASE_TITLE" --notes "$NOTES"
             gh release upload "$TAG" dist/* --clobber
           else
             gh release create "$TAG" dist/* \
-              --title "NSDB snapshot $DAY" \
+              --title "$RELEASE_TITLE" \
               --notes "$NOTES"
           fi
 

README.md

@@ -184,6 +184,11 @@ coverage_changed=true
 comparison_break=true
 ```
 
+When a zone is intentionally excluded, each public release artifact includes
+the `coverage_complete=false` and `excluded_zones=gov` status, regardless
+of whether a prior release exists. Such comparisons are conservatively marked
+`comparison_break=true` even if both releases omit the same zone.
+
 This distinguishes nameserver movement from changes caused by CZDS access or source coverage.
 
 ## Aggregate counts
@@ -345,6 +350,41 @@ source digest in cache invalidation.
 
 Normal first-SOA archives are not fully re-parsed during this preflight. A passing preflight therefore does **not** certify the entire CZDS dataset, and the full build remains mandatory. The current aggregation uses an ephemeral global SQLite database, so a failed run reprocesses earlier zones; per-zone private checkpoints are a separate future reliability improvement.
 
+## Temporary .gov omission (partial-coverage releases)
+
+**As of October 22, 2026**, protected `gov.txt.gz` fails the strict CZDS record
+structure check. Until the defect is understood, the daily-release workflow
+passes `--exclude-zone gov` to **both** preflight and private aggregation
+through the pinned `NSDB_TEMP_EXCLUDE_ZONE=gov` setting. This is an
+explicit, temporary exception tracked by [issue #11](https://github.com/soft-cloud-dev/nsdb/issues/11).
+No other zone may be excluded by this option.
+
+The approved download manifest and raw, licensed zone files are **not**
+modified, deleted or republished. The selector requires that `gov` be
+present in that manifest and that at least one other zone remain; it still
+checks the existence of every manifest file, including the excluded one.
+All included zones undergo their normal SOA, DNS ordering and deduplication
+validation. Failing included zones continue to block publication.
+
+Private `counts.txt` and `coverage.txt`, and the public snapshot,
+coverage, diffs and summary counts are explicitly labeled with
+`coverage_complete=false` and `excluded_zones=gov`. The term
+`coverage_complete` refers only to intentional exclusions from the
+**approved CZDS download manifest**, not to all Internet TLDs. Partial
+releases display `PARTIAL: .gov excluded` in the release title and notes;
+`comparison_break=true` remains set while either compared snapshot has
+partial coverage. A restored `.gov` release changes the coverage set and
+is explicitly flagged with `coverage_changed=true`.
+
+The CLI defaults to **no exclusions**. For a protected, manual partial run,
+pass `--exclude-zone gov` to both `scripts/preflight_czds.py` and
+`scripts/build_nsdb.py`. Passing any other zone, requesting `gov` when
+it is missing, or excluding the sole manifest entry fails closed. Remove
+both workflow arguments and the pinned environment setting after the
+protected .gov input is validated; [issue #11](https://github.com/soft-cloud-dev/nsdb/issues/11)
+tracks the restoration and its regression tests. This mechanism does **not**
+fix or silently accept the malformed .gov source.
+
 ## Same-day reruns
 
 `scripts/fetch_czds.py` writes a persistent per-day manifest. If that day's download set is complete, a rerun reuses it without contacting the zone-download endpoints again.

scripts/build_nsdb.py

@@ -284,6 +284,44 @@ def infer_apex_from_filename(filename: str) -> str | None:
     return None
 
 
+def select_zone_inputs(
+    zones_dir: Path, excluded_zones: tuple[str, ...] = ()
+) -> tuple[list[Path], tuple[str, ...]]:
+    """Apply exact, explicit exclusions to the unchanged CZDS download manifest.
+
+    All manifest entries must still exist; only a known temporary exception
+    can be omitted from parsing. Preflight and aggregation call this same
+    selector to prevent disagreement about coverage.
+    """
+    requested = set(excluded_zones)
+    if requested - {"gov"}:
+        raise ValueError("unsupported CZDS zone exclusion")
+    manifest = json.loads((zones_dir / "manifest.json").read_text(encoding="utf-8"))
+    entries = manifest.get("files", [])
+    if not isinstance(entries, list) or not entries:
+        raise ValueError("CZDS manifest contains no files")
+    selected = []
+    found = set()
+    for entry in entries:
+        if not isinstance(entry, dict):
+            raise ValueError("malformed CZDS manifest entry")
+        filename = entry.get("filename")
+        if not isinstance(filename, str) or not filename or Path(filename).name != filename:
+            raise ValueError("CZDS manifest filename must be a basename")
+        path = zones_dir / filename
+        if path.is_symlink() or not path.is_file():
+            raise ValueError("CZDS manifest references missing or nonregular file")
+        if infer_apex_from_filename(filename) in requested:
+            found.add(infer_apex_from_filename(filename))
+        else:
+            selected.append(path)
+    if requested != found:
+        raise ValueError("requested CZDS exclusion was not found in manifest")
+    if not selected:
+        raise ValueError("no CZDS zones remain after exclusion")
+    return sorted(selected), tuple(sorted(found))
+
+
 def process_zone_stream(
     stream,
     label: str,
@@ -493,19 +531,12 @@ def main():
     p.add_argument("--private-counts", required=True, type=Path)
     p.add_argument("--coverage", required=True, type=Path)
     p.add_argument("--counts", required=True, type=Path)
+    p.add_argument("--exclude-zone", choices=("gov",), action="append", default=[])
     args = p.parse_args()
 
-    manifest_path = args.zones_dir / "manifest.json"
-    if not manifest_path.exists():
-        raise SystemExit(f"missing {manifest_path}")
-    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
-    files = [args.zones_dir / item["filename"] for item in manifest.get("files", [])]
-    if not files:
-        raise SystemExit("zone manifest is empty")
-
-    for path in files:
-        if not path.is_file():
-            raise SystemExit(f"zone file missing: {path}")
+    files, excluded = select_zone_inputs(args.zones_dir, tuple(args.exclude_zone))
+    coverage_complete = "false" if excluded else "true"
+    excluded_text = ",".join(excluded) if excluded else "none"
 
     args.private_counts.parent.mkdir(parents=True, exist_ok=True)
     args.coverage.parent.mkdir(parents=True, exist_ok=True)
@@ -541,13 +572,17 @@ def main():
 
     with open(args.coverage, "w", encoding="utf-8", newline="\n") as fh:
         fh.write("# format=zone|soa_serial|records|delegated_names|delegation_edges\n")
+        fh.write(f"# coverage_complete={coverage_complete}\n")
+        fh.write(f"# excluded_zones={excluded_text}\n")
         for row in coverage:
             fh.write(
                 f"{row['zone']}|{row['soa_serial']}|{row['records']}|"
                 f"{row['delegated_names']}|{row['delegation_edges']}\n"
             )
 
     with open(args.counts, "w", encoding="utf-8", newline="\n") as fh:
+        fh.write(f"coverage_complete={coverage_complete}\n")
+        fh.write(f"excluded_zones={excluded_text}\n")
         fh.write(f"zones={len(coverage)}\n")
         fh.write(f"delegated_names={delegated_names}\n")
         fh.write(f"delegation_edges={delegation_edges}\n")

scripts/preflight_czds.py

@@ -17,6 +17,7 @@
     load_special_zones,
     open_text_stream,
     process_input,
+    select_zone_inputs,
 )
 
 
@@ -40,21 +41,17 @@ def has_nonleading_soa(path: Path) -> bool:
     return fields[3].lower() != "soa" or fields[2].lower() != "in"
 
 
-def preflight(zones_dir: Path, special_zones_file: Path) -> int:
-    manifest = json.loads((zones_dir / "manifest.json").read_text(encoding="utf-8"))
-    entries = manifest.get("files", [])
-    if not entries:
-        raise ValueError("CZDS manifest contains no files")
+def preflight(
+    zones_dir: Path, special_zones_file: Path, excluded_zones: tuple[str, ...] = ()
+) -> int:
+    files, excluded = select_zone_inputs(zones_dir, excluded_zones)
+    if excluded:
+        print("Temporary CZDS exclusion: gov (partial coverage)", flush=True)
 
     special_zones = load_special_zones(special_zones_file)
     completed = 0
-    for entry in entries:
-        filename = entry["filename"]
-        if Path(filename).name != filename:
-            raise ValueError("manifest contains a non-basename path")
-        path = zones_dir / filename
-        if not path.is_file():
-            raise ValueError(f"missing zone file: {filename}")
+    for path in files:
+        filename = path.name
         if not has_nonleading_soa(path):
             continue
         if infer_apex_from_filename(filename) is None:
@@ -81,10 +78,11 @@ def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("--zones-dir", type=Path, required=True)
     parser.add_argument("--special-zones", type=Path, required=True)
+    parser.add_argument("--exclude-zone", choices=("gov",), action="append", default=[])
     args = parser.parse_args()
 
     try:
-        preflight(args.zones_dir, args.special_zones)
+        preflight(args.zones_dir, args.special_zones, tuple(args.exclude_zone))
     except (OSError, ValueError, KeyError, json.JSONDecodeError) as exc:
         raise SystemExit(
             f"NSDB preflight failed (reason={public_validation_code(exc)})"

scripts/publish_nsdb.py

@@ -78,10 +78,12 @@ def write_gzip_copy(path: Path):
             shutil.copyfileobj(src, dst)
 
 
-def write_public_snapshot(path: Path, counts: dict[str, int], k: int, current_tag: str):
+def write_public_snapshot(path: Path, counts: dict[str, int], k: int, current_tag: str, coverage_complete: str = "true", excluded_zones: str = "none"):
     with open(path, "w", encoding="utf-8", newline="\n") as fh:
         fh.write(f"# k={k}\n")
         fh.write(f"# snapshot={current_tag}\n")
+        fh.write(f"# coverage_complete={coverage_complete}\n")
+        fh.write(f"# excluded_zones={excluded_zones}\n")
         fh.write("# format=nameserver|delegations\n")
         for name in sorted(counts):
             fh.write(f"{name}|{counts[name]}\n")
@@ -115,6 +117,33 @@ def load_coverage(path: Path | None) -> set[str]:
     return zones
 
 
+def coverage_status(path: Path, *, required: bool = True) -> tuple[str, str]:
+    """Read explicit, public coverage metadata from the coverage artifact."""
+    if not path.exists():
+        if required:
+            raise ValueError("coverage file is missing")
+        return "true", "none"
+    metadata = {}
+    with open(path, "r", encoding="utf-8") as source:
+        for line in source:
+            if not line.startswith("# "):
+                continue
+            item = line[2:].strip()
+            if "=" in item:
+                key, value = item.split("=", 1)
+                if key in ("coverage_complete", "excluded_zones"):
+                    if key in metadata:
+                        raise ValueError("duplicate coverage status field")
+                    metadata[key] = value
+    if not metadata and not required:
+        return "true", "none"
+    complete = metadata.get("coverage_complete")
+    excluded = metadata.get("excluded_zones")
+    if (complete, excluded) not in (("true", "none"), ("false", "gov")):
+        raise ValueError("invalid or inconsistent coverage status metadata")
+    return complete, excluded
+
+
 def bool_text(value: bool) -> str:
     return "true" if value else "false"
 
@@ -137,6 +166,17 @@ def main():
     if args.k < 1:
         raise SystemExit("k must be >= 1")
 
+    current_coverage_status, current_excluded = coverage_status(args.current_coverage)
+    current_coverage = load_coverage(args.current_coverage)
+    current_full = load_key_values(args.current_full_counts)
+    if (
+        current_full.get("coverage_complete") != current_coverage_status
+        or current_full.get("excluded_zones") != current_excluded
+        or current_full.get("zones") != str(len(current_coverage))
+        or (current_excluded != "none" and current_excluded in current_coverage)
+    ):
+        raise SystemExit("current private counts and CZDS coverage status disagree")
+
     args.out.mkdir(parents=True, exist_ok=True)
     current_visible = load_visible(args.current_private, args.k)
 
@@ -175,7 +215,7 @@ def main():
         raise SystemExit("previous tag supplied without previous public snapshot")
 
     current_snapshot = args.out / "nameservers.txt"
-    write_public_snapshot(current_snapshot, current_visible, args.k, args.current_tag)
+    write_public_snapshot(current_snapshot, current_visible, args.k, args.current_tag, current_coverage_status, current_excluded)
 
     has_previous = args.previous_public is not None and args.previous_public.exists()
     if has_previous:
@@ -192,16 +232,29 @@ def main():
     k_changed = previous_k is not None and previous_k != args.k
 
     previous_coverage = load_coverage(args.previous_coverage)
-    current_coverage = load_coverage(args.current_coverage)
-    coverage_changed = bool(previous_coverage) and previous_coverage != current_coverage
-    comparison_break = coverage_changed or comparison_mode == "k-lowered"
+    previous_status, previous_excluded = (
+        coverage_status(args.previous_coverage, required=False)
+        if args.previous_coverage is not None else ("true", "none")
+    )
+    coverage_changed = bool(previous_coverage) and (
+        previous_coverage != current_coverage
+        or (previous_status, previous_excluded) != (current_coverage_status, current_excluded)
+    )
+    # Partial releases are explicitly not comparable to a complete dataset.
+    comparison_break = (
+        coverage_changed or comparison_mode == "k-lowered"
+        or current_coverage_status == "false"
+        or (has_previous and previous_status == "false")
+    )
 
     diff_path = args.out / "delegation-changes.txt"
     with open(diff_path, "w", encoding="utf-8", newline="\n") as fh:
         fh.write(f"# k={args.k}\n")
         fh.write(f"# previous_k={previous_k_text}\n")
         fh.write(f"# previous_visibility_k={previous_visibility_k if has_previous else 'none'}\n")
         fh.write(f"# current_visibility_k={args.k}\n")
+        fh.write(f"# coverage_complete={current_coverage_status}\n")
+        fh.write(f"# excluded_zones={current_excluded}\n")
         fh.write(f"# comparison_mode={comparison_mode}\n")
         fh.write(f"# k_changed={bool_text(k_changed)}\n")
         fh.write(f"# coverage_changed={bool_text(coverage_changed)}\n")
@@ -220,6 +273,8 @@ def main():
         fh.write(f"# previous_k={previous_k_text}\n")
         fh.write(f"# previous_visibility_k={previous_visibility_k if has_previous else 'none'}\n")
         fh.write(f"# current_visibility_k={args.k}\n")
+        fh.write(f"# coverage_complete={current_coverage_status}\n")
+        fh.write(f"# excluded_zones={current_excluded}\n")
         fh.write(f"# comparison_mode={comparison_mode}\n")
         fh.write(f"# comparison_break={bool_text(comparison_break)}\n")
         fh.write(f"# previous={args.previous_tag}\n")
@@ -231,7 +286,6 @@ def main():
 
     shutil.copyfile(args.current_coverage, args.out / "coverage.txt")
 
-    current_full = load_key_values(args.current_full_counts)
     previous_full = load_key_values(args.previous_full_counts)
     counts_path = args.out / "counts.txt"
     with open(counts_path, "w", encoding="utf-8", newline="\n") as fh:
@@ -261,6 +315,8 @@ def int_or_none(values, key):
         fh.write(f"k={args.k}\n")
         fh.write(f"previous_k={previous_k_text}\n")
         fh.write(f"comparison_mode={comparison_mode}\n")
+        fh.write(f"coverage_complete={current_coverage_status}\n")
+        fh.write(f"excluded_zones={current_excluded}\n")
         fh.write(f"k_changed={bool_text(k_changed)}\n")
         fh.write(f"coverage_changed={bool_text(coverage_changed)}\n")
         fh.write(f"comparison_break={bool_text(comparison_break)}\n")