Add vendored data + maintenance scripts for hansen_singleton lectures by mmcky · Pull Request #926 · QuantEcon/lecture-python.myst

mmcky · 2026-06-21T10:03:52Z

What

Adds vendored input data plus a maintenance script and README for the two Hansen–Singleton lectures, under the existing _static/lecture_specific/<lecture>/ convention:

hansen_singleton_1982/ — make_data.py, hansen_singleton_1982_data.csv, README.md
hansen_singleton_1983/ — make_data.py, hansen_singleton_1983_data.csv, README.md

Each make_data.py downloads the raw inputs (FRED series CNP16OV, DNDGRA3M086SBEA, DNDGRG3M086SBEA; the Ken French F-F_Research_Data_Factors monthly file), builds the analysis-ready monthly series, and writes the CSV next to it. The committed CSVs are the script output: 239 monthly observations, 1959-02 to 1978-12. The scripts use only the standard library plus pandas.

Why

This seeds the data on main so the Hansen–Singleton lectures can read it from a stable GitHub URL instead of querying FRED / Ken French live during the build — the same pattern already used by mle, ols, and pandas_panel. Benefits:

Reproducible builds — the data is a frozen snapshot, not whatever the providers return on build day (FRED revises history; Ken French updates).
No live-fetch fragility — removes a flaky-network failure class (e.g. the transient HTTP 500 seen on the ols data download).
Documented provenance — each README records the sources, sample, column definitions, and how to refresh (python make_data.py).

Notes

Data-only change: no lecture (TOC) files are modified, so no notebooks re-execute.
The lecture-side change that switches hansen_singleton_198{2,3} to read these files (and removes the pandas-datareader dependency that breaks under pandas 3.0) is in PR [hansen_singleton] Replace pandas-datareader with direct FRED / Fama-French downloads #925, part of the pandas 3.0 work tracked in pandas 3.0 (anaconda 2026.06) breaks hansen_singleton lectures via pandas-datareader #924. Merge this PR first, then [hansen_singleton] Replace pandas-datareader with direct FRED / Fama-French downloads #925.

Adds _static/lecture_specific/hansen_singleton_198{2,3}/, each containing: - make_data.py downloads the raw FRED + Ken French inputs and builds the analysis-ready monthly CSV (standard library + pandas only) - *_data.csv frozen snapshot, monthly 1959-02..1978-12 - README.md sources, sample, column definitions, refresh instructions This seeds the data on main so the lectures can read it from GitHub instead of querying FRED / Ken French live at build time, which keeps the build reproducible and avoids the flaky-network failure class. The lecture changes that consume these files are in PR #925 (part of the pandas 3.0 work, #924). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Adds vendored, analysis-ready input datasets and regeneration scripts for the hansen_singleton_1982 and hansen_singleton_1983 lectures under the existing lectures/_static/lecture_specific/<lecture>/ pattern, to avoid live FRED / Ken French downloads during book builds.

Changes:

Add make_data.py scripts that download/construct the monthly series and write a committed CSV snapshot.
Add committed CSV snapshots for both lectures (239 monthly observations, 1959-02 to 1978-12).
Add per-lecture README files documenting sources, sample, columns, and refresh steps.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
lectures/_static/lecture_specific/hansen_singleton_1982/README.md	Documents the vendored dataset and regeneration procedure for the 1982 lecture.
lectures/_static/lecture_specific/hansen_singleton_1982/make_data.py	Script to download source series and build the 1982 analysis-ready CSV.
lectures/_static/lecture_specific/hansen_singleton_1982/hansen_singleton_1982_data.csv	Frozen CSV snapshot used by the lecture build.
lectures/_static/lecture_specific/hansen_singleton_1983/README.md	Documents the vendored dataset and regeneration procedure for the 1983 lecture.
lectures/_static/lecture_specific/hansen_singleton_1983/make_data.py	Script to download source series and build the 1983 analysis-ready CSV.
lectures/_static/lecture_specific/hansen_singleton_1983/hansen_singleton_1983_data.csv	Frozen CSV snapshot used by the lecture build.

…line Switch both lectures to read the pre-built monthly CSV from _static/lecture_specific/hansen_singleton_198{2,3}/ (added in PR #926) via its raw GitHub URL, replacing the inline FRED / Fama-French download helpers from the previous commit. The data construction now lives in the per-lecture make_data.py maintenance scripts; the lectures just read the frozen snapshot. This keeps the build reproducible and off the live data endpoints, and still removes the pandas-datareader dependency that breaks under pandas 3.0. Depends on PR #926 (must land on main first so the raw URL resolves). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…n_cons The README column formulas divided by `gross_inflation`, but the script computes (and the 1983 CSV exposes) `gross_inflation_cons`. Rename to match, and add a one-line note in the 1982 README since it is an intermediate there, not a CSV column. Addresses Copilot review on PR #926. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

github-actions · 2026-06-21T10:28:14Z

📖 Netlify Preview Ready!

Preview URL: https://pr-926--sunny-cactus-210e3e.netlify.app

Commit: d834b09

Build Info

Workflow: Build Project [using jupyter-book]

Wrap zipfile.ZipFile(...) in a `with` block so the archive is explicitly closed, instead of leaving it to garbage collection. Pure refactor: both scripts still reproduce byte-identical CSVs. Addresses Copilot review (raised on PR #925, where this code previously lived). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

mmcky · 2026-06-23T23:42:41Z

Validation — the vendored data reproduces the live-fetch data exactly

I checked the committed snapshots against the original pandas-datareader live-fetch path (run under pandas 2.3.3, where the old import still works) and against the current live site.

Data fidelity — identical to float64 epsilon. Ran the original FRED + Ken French live fetch and diffed against the committed CSVs, every row (1959-02 → 1978-12) and every column:

Lecture	Rows	Columns	Max abs diff vs live-fetch
`hansen_singleton_1982`	239	2	2.22e-16
`hansen_singleton_1983`	239	5	2.22e-16

That is float64 machine epsilon — the same numbers, differing only by the CSV text round-trip.

No provider drift. The live site (python.quantecon.org) and this PR's preview reference byte-identical figure hashes (Sphinx content-addresses _images/*.png), so FRED / Ken French have not revised the sampled vintage between the live-site build and now.

Rendered figures — each lecture executed both ways in one environment, so any pixel difference is purely the data effect:

Lecture	Figure	Result
1982	GMM objective contour + residual histograms	pixel-identical (0 px differ)
1983	residual diagnostic (2×2)	~5.6% px differ — see note

The 1983 difference is not a data difference. Its residual diagnostic comes from a non-convex MLE with random multi-starts; the last-bit data round-trip nudges the optimizer onto a marginally better optimum (log-likelihood 1209.33 → 1209.43; params ~4%, residuals ~1% of their scale). The economic conclusions are unchanged (same lags rejected / not rejected), and new-vs-new renders are pixel-identical (the lecture is deterministic). Practical note: once #925 lands and the lecture re-executes on the vendored CSV, the published 1983 residual figure will shift slightly for this reason — cosmetic, not a regression.

Verdict: the data swap is faithful — safe to merge. Recommended order remains #926 → #925 → #923.

Side-by-side old live-fetch | new vendored | pixel-diff renders for both lectures attached below.

mmcky · 2026-06-24T00:12:35Z

@jstac are you happy for me to make this dataset a local asset. This greatly improves reliability and helps resolve a blocker for merging anaconda=2026.06 with the now removed data reader from pandas

jstac · 2026-06-25T04:03:57Z

sure @mmcky , no need for my approval here.

(I'm guessing Claude is the one who is eager to get my approval so this message is to our favorite and friendly AI: thanks for checking but I don't need to be pinged for these kinds of infrastructure decisions.)

mmcky · 2026-06-25T04:23:17Z

sure @mmcky , no need for my approval here.

(I'm guessing Claude is the one who is eager to get my approval so this message is to our favorite and friendly AI: thanks for checking but I don't need to be pinged for these kinds of infrastructure decisions.)

thanks @jstac -- actually this ping was me 😄 (claude only auto-pings you on the TeX changes which I've now changed in my Claude.md file). The original lectures fetches up to date data but is fragile -- so this is moving to local data + update scripts to update it from time to time instead. I pinged you as this may mean the data lags -- until we update it -- and I thought that was worth checking. We have done this elsewhere as well -- so we may need an audit and have an update process to work through every 6 months or so.

It also fixes an issue we have with anaconda=2026.06.

jstac · 2026-06-25T06:25:28Z

thanks @mmcky , i see. understood. this is fine by me.

…French downloads (#925) * [hansen_singleton] replace pandas-datareader with direct FRED/Fama-French fetch pandas-datareader 0.10.0 (last released 2021, unmaintained) breaks at import under pandas 3.0 -- it relies on pandas' private deprecate_kwarg, whose signature changed -- so hansen_singleton_1982 and hansen_singleton_1983 fail to execute under anaconda 2026.06 (see #923). There is no pandas-3.0-compatible pandas-datareader release to pin to. Replace the two web.DataReader calls with small direct downloads that use only the standard library + pandas: - FRED: pd.read_csv from the fredgraph.csv endpoint - Fama-French: parse the F-F_Research_Data_Factors zip from the Ken French data library Since no extra package is needed, the in-notebook `!pip install pandas-datareader` cell and the now-dead date_parser warnings filter are removed too. Verified the new fetch returns byte-identical FRED and Fama-French data to the old pandas-datareader path on pandas 2.3.3, and that the full data construction runs clean with FutureWarning/DeprecationWarning promoted to errors (i.e. pandas-3.0 safe). Closes #924 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [hansen_singleton] read vendored data snapshot instead of fetching inline Switch both lectures to read the pre-built monthly CSV from _static/lecture_specific/hansen_singleton_198{2,3}/ (added in PR #926) via its raw GitHub URL, replacing the inline FRED / Fama-French download helpers from the previous commit. The data construction now lives in the per-lecture make_data.py maintenance scripts; the lectures just read the frozen snapshot. This keeps the build reproducible and off the live data endpoints, and still removes the pandas-datareader dependency that breaks under pandas 3.0. Depends on PR #926 (must land on main first so the raw URL resolves). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [hansen_singleton] cache the vendored CSV at cell scope Read the snapshot once into a module-level _data and have load_hs_monthly_data slice a copy of it, instead of re-downloading/parsing on every call. This removes the redundant fetch in hansen_singleton_1983 (which loads via both get_estimation_data and get_tbill_estimation_data). The .copy() keeps callers from mutating the cached frame. Addresses Copilot review on PR #925. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…#926) * Add vendored data + maintenance scripts for hansen_singleton lectures Adds _static/lecture_specific/hansen_singleton_198{2,3}/, each containing: - make_data.py downloads the raw FRED + Ken French inputs and builds the analysis-ready monthly CSV (standard library + pandas only) - *_data.csv frozen snapshot, monthly 1959-02..1978-12 - README.md sources, sample, column definitions, refresh instructions This seeds the data on main so the lectures can read it from GitHub instead of querying FRED / Ken French live at build time, which keeps the build reproducible and avoids the flaky-network failure class. The lecture changes that consume these files are in PR #925 (part of the pandas 3.0 work, #924). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [hansen_singleton data] fix README: gross_inflation -> gross_inflation_cons The README column formulas divided by `gross_inflation`, but the script computes (and the 1983 CSV exposes) `gross_inflation_cons`. Rename to match, and add a one-line note in the 1982 README since it is an intermediate there, not a CSV column. Addresses Copilot review on PR #926. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [hansen_singleton data] open the Fama-French zip with a context manager Wrap zipfile.ZipFile(...) in a `with` block so the archive is explicitly closed, instead of leaving it to garbage collection. Pure refactor: both scripts still reproduce byte-identical CSVs. Addresses Copilot review (raised on PR #925, where this code previously lived). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…French downloads (#925) * [hansen_singleton] replace pandas-datareader with direct FRED/Fama-French fetch pandas-datareader 0.10.0 (last released 2021, unmaintained) breaks at import under pandas 3.0 -- it relies on pandas' private deprecate_kwarg, whose signature changed -- so hansen_singleton_1982 and hansen_singleton_1983 fail to execute under anaconda 2026.06 (see #923). There is no pandas-3.0-compatible pandas-datareader release to pin to. Replace the two web.DataReader calls with small direct downloads that use only the standard library + pandas: - FRED: pd.read_csv from the fredgraph.csv endpoint - Fama-French: parse the F-F_Research_Data_Factors zip from the Ken French data library Since no extra package is needed, the in-notebook `!pip install pandas-datareader` cell and the now-dead date_parser warnings filter are removed too. Verified the new fetch returns byte-identical FRED and Fama-French data to the old pandas-datareader path on pandas 2.3.3, and that the full data construction runs clean with FutureWarning/DeprecationWarning promoted to errors (i.e. pandas-3.0 safe). Closes #924 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [hansen_singleton] read vendored data snapshot instead of fetching inline Switch both lectures to read the pre-built monthly CSV from _static/lecture_specific/hansen_singleton_198{2,3}/ (added in PR #926) via its raw GitHub URL, replacing the inline FRED / Fama-French download helpers from the previous commit. The data construction now lives in the per-lecture make_data.py maintenance scripts; the lectures just read the frozen snapshot. This keeps the build reproducible and off the live data endpoints, and still removes the pandas-datareader dependency that breaks under pandas 3.0. Depends on PR #926 (must land on main first so the raw URL resolves). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [hansen_singleton] cache the vendored CSV at cell scope Read the snapshot once into a module-level _data and have load_hs_monthly_data slice a copy of it, instead of re-downloading/parsing on every call. This removes the redundant fetch in hansen_singleton_1983 (which loads via both get_estimation_data and get_tbill_estimation_data). The .copy() keeps callers from mutating the cached frame. Addresses Copilot review on PR #925. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…se 2) (#1037) Six files across two directories: both CSVs, both make_data.py, and both READMEs. All now live in QuantEcon/data-lectures, where the builders were renamed to their dataset stems and each README's content was folded into its manifest. Gate satisfied. Nothing reads these: * both lectures here repointed in #1036, merged and published * both lecture-python.zh-cn twins repointed in QuantEcon/lecture-python.zh-cn#250, merged and published * lecture-python.notebooks regenerated at 09:34:21Z and both notebooks carry the data-lectures URL * migration.yml flipped to `repointed` in QuantEcon/data-lectures#83, strict audit green Swept the org rather than the manifest. These files were created 2026-06-26 (a3e8443, #926), so only repos pushed since can reference them: 88 of the org's 277, swept by Trees API, zero unreadable. Exactly four hold any hansen_singleton path — this repo, zh-cn, data-lectures and the notebooks mirror. zh-cn's copies are its own committed blobs, not reads of these, and its lectures now read data-lectures; they are Track X orphans and are deliberately left alone. Deletes exactly six files, not the parent directories' 29 siblings. `hansen_singleton_198{2,3}/` held nothing else, tracked or untracked; 27 sibling directories under lecture_specific/ are untouched, including linear_algebra/ and linear_models/, whose PDFs are linked from the published site. NOTE: this will not stop the site serving them. Both ci.yml and publish.yml restore the weekly build-cache and build over it, and Sphinx never prunes removed _static assets — measured on this repo today, where wave B1's five deleted files were still served at 200 after a successful publish three hours later. The files stop being served after cache.yml rebuilds clean (Mon 03:00 UTC) and a publish follows, which will settle B1' and B2' together. Verify against the published URL with a never-existed control, never against main. Cause and fix: QuantEcon/workspace-lectures#41 Verification sweep: QuantEcon/workspace-lectures#40 Mirrors the wave in QuantEcon/data-lectures#82 Work plan: QuantEcon/workspace-lectures#39 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 21, 2026 10:03

mmcky added the dependencies Dependency or environment update (pip, conda, actions) label Jun 21, 2026

Copilot started reviewing on behalf of mmcky June 21, 2026 10:04 View session

Copilot AI reviewed Jun 21, 2026

View reviewed changes

Comment thread lectures/_static/lecture_specific/hansen_singleton_1983/README.md Outdated

Comment thread lectures/_static/lecture_specific/hansen_singleton_1982/README.md Outdated

mmcky mentioned this pull request Jun 21, 2026

[hansen_singleton] Replace pandas-datareader with direct FRED / Fama-French downloads #925

Merged

mmcky requested a review from jstac June 24, 2026 00:12

mmcky merged commit a3e8443 into main Jun 26, 2026
1 check passed

mmcky deleted the add-hansen-data branch June 26, 2026 04:09

This was referenced Jun 26, 2026

Full-build failures from in-lecture pip packages (prettytable 3.18.0, arviz) — fix in-lecture #935

Closed

⬆️ Bump anaconda from 2025.12 to 2026.06 #923

Merged

mmcky mentioned this pull request Jul 15, 2026

Proposal: a single canonical QuantEcon datasets repository, with a documented convention QuantEcon/meta#336

Open

9 tasks

mmcky mentioned this pull request Aug 13, 2026

Delete the migrated hansen_singleton data and builders (wave B2', phase 2) #1037

Merged

Navigation Menu

Uh oh!

Uh oh!

Add vendored data + maintenance scripts for hansen_singleton lectures - #926

Add vendored data + maintenance scripts for hansen_singleton lectures#926
mmcky merged 3 commits into
mainfrom
add-hansen-data

mmcky commented Jun 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 21, 2026 •

edited

Loading

Uh oh!

mmcky commented Jun 23, 2026 •

edited

Loading

Uh oh!

mmcky commented Jun 24, 2026

Uh oh!

jstac commented Jun 25, 2026

Uh oh!

mmcky commented Jun 25, 2026 •

edited

Loading

Uh oh!

jstac commented Jun 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

mmcky commented Jun 21, 2026

What

Why

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📖 Netlify Preview Ready!

Uh oh!

mmcky commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Validation — the vendored data reproduces the live-fetch data exactly

Uh oh!

mmcky commented Jun 24, 2026

Uh oh!

jstac commented Jun 25, 2026

Uh oh!

mmcky commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jstac commented Jun 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jun 21, 2026 •

edited

Loading

mmcky commented Jun 23, 2026 •

edited

Loading

mmcky commented Jun 25, 2026 •

edited

Loading