Use bundle-certified dataset materialization

changelog.d/502.changed.md

@@ -0,0 +1,5 @@
+PolicyEngine.py now exposes one `materialize_dataset` entry point for
+bundle-managed datasets, explicit Hugging Face URIs, and explicit local paths.
+Managed datasets are reused or downloaded from the exact repository type,
+immutable revision, and SHA-256 recorded in the release bundle; the other two
+inputs remain opt-in.

changelog.d/502.removed.md

@@ -0,0 +1,6 @@
+Removed the legacy dataset-specific GCS downloader and its direct dependencies,
+the public `resolve_local_managed_dataset_source` export, and the former
+`policyengine.provenance.dataset_sources.materialize_dataset_source` function.
+Use `policyengine.provenance.materialize_dataset` for bundle-managed datasets,
+explicit Hugging Face references, and explicit local files. The separate UK
+geography asset implementation is unchanged.

docs/bundles.md

@@ -29,17 +29,45 @@ When run from `uvx` or `pipx`, the installer creates or reuses `./.venv`.
 Inside an existing virtualenv or conda environment, it installs into that active
 environment. The installer then installs the
 exact bundled package scaffold with pip, downloads certified default US and UK
-datasets into `./data`, moves replaced dataset files into
-`./data/.policyengine-bundle-backups/<timestamp>/`, and writes a
-`./data/.policyengine-bundle-receipt.json` receipt that records the target
-Python.
+datasets into `./data`, and writes a `./data/.policyengine-bundle-receipt.json`
+receipt that records the target Python.
+
+Dataset pre-download and US and UK calculations share the same verified-download
+implementation. For every managed artifact, PolicyEngine.py reads the source
+data package name, Hugging Face repository type, immutable revision, and SHA-256
+from the bundle. It reuses an existing file only when its hash matches, downloads
+and verifies a replacement before atomically replacing an invalid local file,
+and records the verified result in the receipt.
 
 The bundle manifest can certify additional regional datasets, such as US state
 datasets. Those artifacts are part of the citable bundle manifest, but
 `policyengine bundle install` does not eagerly download every regional file.
 Runtime callers should use the manifest's regional dataset URI when a regional
 simulation needs one.
 
+To materialize a default or named artifact without installing the complete
+package scaffold:
+
+```python
+from policyengine.provenance import materialize_dataset
+
+result = materialize_dataset("us", "populace_us_2024")
+print(result.path)
+print(result.bundle_dataset.sha256)
+```
+
+`materialize_dataset` returns the selected source URI and local path. For a
+bundle-managed input, `bundle_dataset` also contains the selected source
+package, repository type, revision, verified SHA-256, and optional metadata
+path.
+`policyengine-*-data` and `populace-data` artifacts use the repository type
+recorded in the bundle. Callers do not infer repository type from the repository
+name.
+
+Managed datasets are downloaded from the Hugging Face artifact specified in the
+bundle. GCS dataset URIs are unsupported. The separate UK geography lookup files
+retain their existing storage implementation.
+
 Country-specific and package-only installs are supported:
 
 ```bash

docs/data-publishing-design.md

@@ -162,17 +162,22 @@ concrete `artifact_sha256` pin in the country release manifest.
 After that, the release manifest is what papers cite; the storage
 channel is just the cache.
 
-## Consumer resolver (what `pe.py` changes)
+## Consumer resolver (historical proposal)
 
-Minimal. The existing `pe.us.ensure_datasets` takes a URI today:
+PolicyEngine.py now resolves managed datasets by logical name through its
+certified release bundle:
 
 ```python
 pe.us.ensure_datasets(
-    datasets=["hf://policyengine/populace-us/populace_us_2024.h5@<release>"],
+    datasets=["populace_us_2024"],
     years=[2026],
 )
 ```
 
+Direct Hugging Face references require `allow_unmanaged=True`; GCS dataset
+references are unsupported. The `pe-data://` examples below remain an
+unimplemented design proposal rather than a description of current behavior.
+
 Under the substrate, the URI scheme gains a new prefix:
 
 ```python

docs/microsim.md

@@ -40,7 +40,15 @@ datasets = pe.us.ensure_datasets(
 dataset = datasets["populace_us_2024_2026"]
 ```
 
-The default US dataset is **Populace US 2024** — a Populace-built dataset calibrated to IRS, CMS, SNAP, Census, and other administrative totals. The UK default is **Populace UK 2023** — a Populace-built Family Resources Survey dataset calibrated to UK administrative targets.
+The default US dataset is **Populace US 2024** — a Populace-built dataset
+calibrated to IRS, CMS, SNAP, Census, and other administrative totals. The
+current UK certified default is **Enhanced FRS 2024–25**, supplied by
+`policyengine-uk-data`. **Populace UK 2023** remains available as a named,
+non-default bundle dataset.
+
+PolicyEngine.py obtains the repository type, immutable revision, and SHA-256
+from the installed release bundle. An existing file in the configured data
+directory is reused only after hash verification.
 
 List datasets already known to the country:
 
@@ -97,7 +105,7 @@ ca = Simulation(
 
 UK population data uses licensed Family Resources Survey inputs. The default
 UK release bundle points to the private
-`policyengine/populace-uk-private` Hugging Face dataset repository. Set
+`policyengine/policyengine-uk-data-private` Hugging Face repository. Set
 `HUGGING_FACE_TOKEN` to a token from a Hugging Face account with access:
 
 ```bash
@@ -113,11 +121,11 @@ import policyengine as pe
 from policyengine.core import Simulation
 
 datasets = pe.uk.ensure_datasets(
-    datasets=["populace_uk_2023"],
+    datasets=["enhanced_frs_2024_25"],
     years=[2026],
     data_folder="./data",
 )
-dataset = datasets["populace_uk_2023_2026"]
+dataset = datasets["enhanced_frs_2024_25_2026"]
 
 simulation = Simulation(
     dataset=dataset,
@@ -126,28 +134,25 @@ simulation = Simulation(
 simulation.run()
 ```
 
-To download the raw h5 artifact directly from Hugging Face, use
-`huggingface_hub` and specify `repo_type="dataset"`:
+To materialize the raw certified artifact without creating uprated yearly
+datasets, use PolicyEngine.py's bundle API:
 
 ```python
-import os
-from huggingface_hub import hf_hub_download
-
-path = hf_hub_download(
-    repo_id="policyengine/populace-uk-private",
-    filename="populace_uk_2023.h5",
-    repo_type="dataset",
-    token=os.environ["HUGGING_FACE_TOKEN"],
+from policyengine.provenance import materialize_dataset
+
+result = materialize_dataset(
+    "uk",
+    "enhanced_frs_2024_25",
 )
 
-print(path)
+print(result.path)
+print(result.bundle_dataset.sha256)
 ```
 
-The repository URL is
-<https://huggingface.co/datasets/policyengine/populace-uk-private>. A 404 from
-the website or `RepositoryNotFoundError` from the Hub API usually means the
-browser or token is not authenticated as an account with access, or that the
-Hub call omitted `repo_type="dataset"`.
+The bundle API uses the repository type recorded in the bundle, so callers do
+not need repository-specific download logic. Authentication or authorization
+failures are reported directly and do not cause a retry against another
+repository type.
 
 ## Simulations
 
@@ -219,6 +224,7 @@ Smaller custom H5 datasets can be passed explicitly for testing:
 datasets = pe.us.ensure_datasets(
     datasets=["/path/to/smoke_test_populace_us_2024.h5"],
     years=[2026],
+    allow_unmanaged=True,
 )
 ```
 
@@ -235,7 +241,12 @@ sim = managed_microsimulation()
 # `sim` is a policyengine_us.Microsimulation — use its API directly
 ```
 
-Pass `allow_unmanaged=True` with a custom `dataset=` to opt out of the release bundle.
+Pass `allow_unmanaged=True` with a custom `dataset=` to opt out of the release
+bundle. Explicit local paths and Hugging Face URIs remain supported in this
+mode. GCS dataset URIs are not supported.
+
+For managed simulations, `sim.policyengine_bundle` records the actual source
+package, repository type, revision, verified SHA-256, and local path.
 
 ## Pinned model versions
 

docs/release-bundles.md

@@ -41,8 +41,13 @@ environment. It installs the bundled Python packages with pip, downloads the
 certified default US and UK datasets into `./data`, and writes a
 `./data/.policyengine-bundle-receipt.json` receipt that records the target
 Python.
-Existing dataset files with the same filename are moved to
-`./data/.policyengine-bundle-backups/<timestamp>/`.
+An existing file is reused when its SHA-256 matches the manifest. Otherwise, a
+verified download atomically replaces it.
+
+The command invokes the same bundle materializer used by calculations. The
+materializer uses the manifest's exact Hugging Face repository type, immutable
+revision, and certified SHA-256. The data package name is retained only as
+provenance metadata; it does not select a download implementation.
 
 Regional datasets may also be certified in the bundle manifest. They are not
 eagerly downloaded by `policyengine bundle install`; callers should materialize
@@ -153,8 +158,10 @@ sibling `dataset_overlays.{country}` map:
 "dataset_overlays": {
   "us": {
     "populace_us_2024_acs_local": {
+      "data_package_name": "populace-data",
       "path": "populace_us_2024_acs_local.h5",
       "repo_id": "policyengine/populace-us",
+      "repo_type": "dataset",
       "revision": "populace-us-2024-buildo-acs-local-...",
       "sha256": "..."
     }
@@ -170,6 +177,11 @@ default resolution is untouched. Because certification only rewrites
 `data_releases`, overlays survive re-certification without any manual
 re-add step.
 
+Cross-package overlays must declare `data_package_name` and `repo_type`
+explicitly. Ordinary artifacts inherit these values from the country release's
+primary `data_package`. This prevents runtime code from guessing how a
+repository should be addressed.
+
 Earlier releases (policyengine 4.15.x–4.16.x) were certified through the
 `PolicyEngine/policyengine-bundles` archive flow; those bundles remain the
 historical record of their certifications.
@@ -601,6 +613,9 @@ The target implementation in `policyengine.py` should add:
 - explicit runtime bundle metadata on simulations, APIs, and app responses
 - checksum-backed dataset resolution from the certified bundle manifest
 
+The checksum-backed runtime resolution described above is now implemented.
+Managed materialization is Hugging Face-only; GCS is not a dataset source.
+
 ## Why not let `policyengine.py` build all country data directly?
 
 Because that would centralise the wrong concerns:

pyproject.toml

@@ -29,8 +29,6 @@ dependencies = [
     "requests>=2.31.0",
     "psutil>=5.9.0",
     "packaging>=23.0",
-    "google-cloud-storage>=3.1.0,<4.0.0",
-    "diskcache>=5.6.3,<6.0.0",
 ]
 
 [project.scripts]
@@ -85,6 +83,9 @@ where = ["src"]
 [tool.setuptools.package-data]
 "policyengine" = ["**/*"]
 
+[tool.setuptools.exclude-package-data]
+"policyengine" = ["**/__pycache__/*", "**/*.pyc", "**/*.pyo"]
+
 [tool.pytest.ini_options]
 addopts = "-v"
 testpaths = [