Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/workflows/python-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@ jobs:
python-version: "3.14"
cache: "pip"
- name: Install ruff
# Keep this version aligned with the ruff-pre-commit revision.
run: pip install ruff==0.16.1
run: pip install -e .[lint]
- name: Lint with ruff
run: |
ruff check . --output-format=github
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ repos:
- id: debug-statements

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.1
rev: v0.16.5
hooks:
- id: ruff-check
args: [--fix]
Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@ can predict where a thing lives.
- Python >= 3.10; the CI test matrix is 3.10, 3.13, 3.14.

## Commands
- Lint: `ruff check .` and `ruff format --check .` (pinned to the version in
`.pre-commit-config.yaml` and the CI lint job — keep them aligned).
- Lint: `ruff check .` and `ruff format --check .` (`pip install -e .[lint]`
pins the version; keep the `.pre-commit-config.yaml` rev aligned with it).
- Tests: `coverage run -m pytest tests/ && coverage report`, or focused like
`pytest tests/waterdata_test.py::test_mock_get_samples`. `coverage report` is
a merge gate: branch coverage with a `fail_under` ratchet in
Expand Down
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
**08/30/2026:** **Bug fix:** `waterdata.get_ratings(..., file_path=...)` wrote each rating with no explicit encoding, so on a non-UTF-8 locale an unrepresentable character raised `UnicodeEncodeError` and the rating was silently dropped from the returned dict; ratings are now written as UTF-8, byte for byte with the response. **Bug fix:** that same write blocked the event loop, so on a multi-location request every other in-flight download stalled for its duration; it now runs in a worker thread.

**08/27/2026:** **Bug fix:** `nwis.get_discharge_peaks` and `nwis.get_record(service='peaks')` silently discarded every peak whose date is only partly known. NWIS zero-fills the unknown part of a historical peak's date -- `YYYY-MM-00` when the day is not known, `YYYY-00-00` when the month is not either (the `Bd` and `Bm` `peak_cd` qualifiers) -- and neither parses as a date, so `preformat_peaks_response` coerced both to `NaT` and then dropped the row along with its discharge value. These are real peaks, and disproportionately a site's largest: site 14105700 lost 20 of its 167 peaks, among them an 1859 flood of 847,000 ft3/s, and 06934500 lost its 1844 peak of 700,000 ft3/s. Such a peak is now kept, with `datetime` left as `NaT`. The date is not completed into one NWIS does not have: a `datetime64` column cannot hold a partial date, so any value there would assert a day the record does not claim. **Behavior change:** peaks queries return more rows than before, and `datetime` may now be `NaT` -- a caller selecting on the datetime index will not see those peaks and should filter on `peak_dt` instead. **Behavior change:** `peak_dt` is no longer removed from the returned frame. It is the only column carrying a censored peak's year, since the peaks response has no `water_yr`, and the only dependable way to tell an unknown day from a known one -- `peak_cd` does not always carry the qualifier (22 of 24 censored dates across six sites tested). For peaks with a resolved timestamp alongside explicit `year`/`month`/`day` and a `qualifier` field, use `waterdata.get_peaks()`.

**08/26/2026:** **Bug fix:** `nwis.format_response(df, service='peaks')` and `nwis.preformat_peaks_response` raised `KeyError('peak_dt')` on an empty peaks response instead of returning an empty frame. Both are public, and every other service already treated an empty result as a legitimate empty frame rather than an error (issue #171); the peaks arm slipped through because it reformats the datetime column before the empty-frame check. Callers can now check `df.empty` rather than catching an exception. A *non-empty* frame with no `peak_dt` column is malformed rather than empty, and still raises.
Expand Down
7 changes: 7 additions & 0 deletions dataretrieval/codes/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,9 @@
"""Facade over the ``states`` and ``timezones`` lookup tables.

Re-exports the state code maps and their normalizers (``to_state``,
``apply_state``) alongside the ``tz`` UTC-offset map, so one import
reaches every code lookup in the package.
"""

from .states import *
from .timezones import *
4 changes: 2 additions & 2 deletions dataretrieval/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -819,7 +819,7 @@ def _accepts(adapter: str, name: str) -> bool:
return name in _ALL_SETTINGS if accepted is None else name in accepted


def _display_api_key(adapter: str | None = None) -> str:
def _display_api_key(_adapter: str | None = None) -> str:
"""Render the key's presence, never its value."""
return "<set>" if api_key() else "<not set>"

Expand All @@ -829,7 +829,7 @@ def _display_concurrency(adapter: str | None = None) -> str:
return CONCURRENCY_UNBOUNDED if value is None else str(value)


def _display_progress(adapter: str | None = None) -> str:
def _display_progress(_adapter: str | None = None) -> str:
setting = progress()
return "auto" if setting is None else ("on" if setting else "off")

Expand Down
87 changes: 69 additions & 18 deletions dataretrieval/nwis.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,15 +146,15 @@ def _parse_json_or_raise(response: httpx.Response) -> pd.DataFrame:
raise


def _localize_datetime_index(df: pd.DataFrame) -> pd.DataFrame:
def _localized_datetime_index(index: pd.Index) -> pd.Index:
"""Localize a naive datetime index (or multi-index level) to UTC."""
if hasattr(df.index, "levels"):
if hasattr(index, "levels"):
# Multi-index: localize the datetime level (level 1)
if hasattr(df.index.levels[1], "tzinfo") and df.index.levels[1].tzinfo is None:
df = df.tz_localize("UTC", level=1)
elif hasattr(df.index, "tzinfo") and df.index.tzinfo is None:
df = df.tz_localize("UTC")
return df
if hasattr(index.levels[1], "tzinfo") and index.levels[1].tzinfo is None:
return index.set_levels(index.levels[1].tz_localize("UTC"), level=1)
elif hasattr(index, "tzinfo") and index.tzinfo is None:
return index.tz_localize("UTC")
return index


def format_response(
Expand Down Expand Up @@ -196,11 +196,19 @@ def format_response(
return df

if len(df["site_no"].unique()) > 1 and mi:
df.set_index(["site_no", "datetime"], inplace=True)
keys = ["site_no", "datetime"]
else:
df.set_index(["datetime"], inplace=True)
keys = ["datetime"]

# Index our own frame, never the caller's. The shallow copy shares the
# columns; ``set_index`` without ``inplace`` duplicates them, because
# pandas deep-copies the frame whenever copy-on-write is off.
df = df.copy(deep=False)
df.set_index(keys, inplace=True) # noqa: PD002 # our copy, not the caller's

df = _localize_datetime_index(df)
# Retag the index alone; ``DataFrame.tz_localize`` relabels the axis by
# duplicating every column whenever copy-on-write is off.
df.index = _localized_datetime_index(df.index)
return df.sort_index()


Expand Down Expand Up @@ -236,6 +244,8 @@ def preformat_peaks_response(df: pd.DataFrame) -> pd.DataFrame:
# still raise.
return df

# Derive the column on our own frame, never the caller's.
df = df.copy(deep=False)
df["datetime"] = pd.to_datetime(df["peak_dt"], errors="coerce")
return df

Expand Down Expand Up @@ -895,6 +905,41 @@ def what_sites(
return df, NWIS_Metadata(response, **kwargs)


# The value that means "not passed" -- which must stay in step with
# ``get_record``'s declared default -- and the replacement to name.
_DEFUNCT_RECORD_OPTIONS: dict[str, tuple[object, str]] = {
"wide_format": (True, "`waterdata.get_samples()`"),
"datetime_index": (
True,
"`waterdata.get_continuous()` or `waterdata.get_daily()`",
),
"state": (None, "`nwdc.get_wateruse(state=...)`"),
}


def _warn_defunct_record_options(**given: object) -> None:
"""Advise on each ``get_record`` option asked to do something it cannot.

Naming an option at its declared default is silent: the caller is asking
for what the dead default already gave them.
"""
for name, value in given.items():
unset, replacement = _DEFUNCT_RECORD_OPTIONS[name]
if value != unset:
warn_deprecated(
f"`nwis.get_record`'s `{name}` argument",
replacement=replacement,
removal=_NWIS_REMOVAL_DATE,
detail=(
"It is ignored, and has been since the service that read "
"it was retired."
),
# _warn_defunct_record_options -> get_record -> @_deprecated
# wrapper -> the caller's own line.
stacklevel=4,
)


@_deprecated
def get_record(
sites: list[str] | str | None = None,
Expand Down Expand Up @@ -927,12 +972,16 @@ def get_record(
If False, return a dataframe with a single-level index (datetime).
Default is True.
wide_format : bool, optional
If True, return data in wide format, with multiple samples per row and
one row per time. Default is True.
(defunct) Shaped the output of the retired 'qwdata' service. Ignored;
passing `False` warns. Use `waterdata.get_samples`, which returns one
row per result.
datetime_index : bool, optional
If True, create a datetime index. Default is True.
(defunct) Shaped the output of the retired 'qwdata' and 'gwlevels'
services. Ignored; passing `False` warns. Use `waterdata.get_continuous`
or `waterdata.get_daily`, which return `time` as a column.
state: string, optional, default is None
State full name, abbreviation, or id.
(defunct) Selected sites for the retired 'water_use' service. Ignored;
passing a state warns. Use `nwdc.get_wateruse`, which takes `state`.
service: string, default is 'iv'
- 'iv' : instantaneous data
- 'dv' : daily mean data
Expand Down Expand Up @@ -1011,6 +1060,10 @@ def get_record(
),
)

_warn_defunct_record_options(
wide_format=wide_format, datetime_index=datetime_index, state=state
)

if service == "iv":
df, _ = get_iv(
sites=sites,
Expand Down Expand Up @@ -1097,15 +1150,13 @@ def _parse_parameter_record(
record_df["qualifiers"] = (
record_df["qualifiers"].astype(str).str.strip("[]").str.replace("'", "")
)
record_df.rename(
return record_df.rename(
columns={
"value": col_name,
"dateTime": "datetime",
"qualifiers": col_name + "_cd",
},
inplace=True,
}
)
return record_df


def _parse_site_block(site_block: list[dict[str, Any]]) -> pd.DataFrame:
Expand Down
4 changes: 3 additions & 1 deletion dataretrieval/ogc/dates.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@ def _parse_datetime(value: str) -> datetime | None:
candidate = value[:-1] + "+00:00" if value.endswith("Z") else value
for fmt in _DATETIME_FORMATS:
try:
return datetime.strptime(candidate, fmt)
# DTZ007: naive is the documented outcome for a naive input --
# ``_DATETIME_FORMATS`` carries both the ``%z`` and the bare forms.
return datetime.strptime(candidate, fmt) # noqa: DTZ007
except ValueError:
continue
return None
Expand Down
4 changes: 2 additions & 2 deletions dataretrieval/transport/pagination.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ def report_page(page: httpx.Response, frame: pd.DataFrame) -> None:

try:
frame, cursor = parse_response(response)
except Exception as exc: # noqa: BLE001
except Exception as exc:
logger.warning("Initial response parse failed.")
raise DataRetrievalError(
paginated_failure_message(0, exc, response.url)
Expand All @@ -141,7 +141,7 @@ def report_page(page: httpx.Response, frame: pd.DataFrame) -> None:
nrows += len(frame)
total_elapsed += _safe_elapsed(response)
report_page(response, frame)
except Exception as exc: # noqa: BLE001
except Exception as exc:
logger.warning(
"Request failed at cursor %r. Data download interrupted.", cursor
)
Expand Down
4 changes: 2 additions & 2 deletions dataretrieval/transport/retry.py
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ async def attempt_once() -> _T:
while True:
try:
return await attempt_once()
except Exception as exc: # noqa: BLE001 - re-raised unless retryable
except Exception as exc:
attempt += 1
wait = _retry_delay(exc, attempt, policy)
if wait is None:
Expand All @@ -314,7 +314,7 @@ def retry_sync(fn: Callable[[], _T], policy: RetryPolicy | None = None) -> _T:
while True:
try:
return fn()
except Exception as exc: # noqa: BLE001 - re-raised unless retryable
except Exception as exc:
attempt += 1
wait = _retry_delay(exc, attempt, policy)
if wait is None:
Expand Down
4 changes: 2 additions & 2 deletions dataretrieval/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@

import dataretrieval._querying as _querying
import dataretrieval.transport.http as _transport_http
from dataretrieval._ambient import Ambient # noqa: F401 - compatibility re-export
from dataretrieval._ambient import Ambient # noqa: F401 # compatibility re-export
from dataretrieval._response_metadata import (
BaseMetadata, # noqa: F401 compatibility re-export; defined there now
BaseMetadata, # noqa: F401 # compatibility re-export; defined there now
)
from dataretrieval.codes import tz

Expand Down
16 changes: 14 additions & 2 deletions dataretrieval/waterdata/ratings.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from collections.abc import Iterable
from typing import Any, Literal, get_args

import anyio
import httpx
import pandas as pd

Expand Down Expand Up @@ -340,6 +341,16 @@ def _inert_response(
return httpx.Response(status, headers=headers, request=httpx.Request("GET", url))


def _write_rating(path: str, body: str) -> None:
"""Persist one rating to disk, off the event loop.

``_fetch_rating`` runs concurrently under a fan-out drive, so a blocking
write here would stall every other in-flight download for its duration.
"""
with open(path, "w", encoding="utf-8", newline="") as f:
f.write(body)


async def _fetch_rating(
feature: dict[str, Any], file_path: str | None
) -> tuple[pd.DataFrame, httpx.Response]:
Expand All @@ -366,8 +377,9 @@ async def _fetch_rating(
_raise_for_non_200(response)

if file_path is not None:
with open(os.path.join(file_path, fid), "w") as f:
f.write(response.text)
await anyio.to_thread.run_sync(
_write_rating, os.path.join(file_path, fid), response.text
)

df = read_rdb(response.text)
df.attrs["comment"] = extract_rdb_comment(response.text)
Expand Down
15 changes: 15 additions & 0 deletions dataretrieval/waterdata/types.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
"""The Water Data type vocabularies the getters validate their arguments against.

Each ``Literal`` alias here is one closed vocabulary: ``CODE_SERVICES`` for
the Samples code services, ``METADATA_COLLECTIONS`` for the reference-table
collections, ``SERVICES`` and ``PROFILES`` for the Samples resources and the
output profiles they offer, and ``WATERDATA_COLLECTIONS`` (permanent alias
``WATERDATA_SERVICES``) for the collections ``get_cql`` queries.
``PROFILE_LOOKUP`` pairs ``SERVICES`` with ``PROFILES``, mapping each Samples
resource to the profiles valid for it.

They live apart from the getters that check them so a caller may annotate an
argument with the same alias the getter validates it against, and so one
vocabulary cannot be spelled two ways in two modules.
"""

from typing import Literal, get_args

from dataretrieval._validation import require_one_of
Expand Down
2 changes: 1 addition & 1 deletion dataretrieval/wateruse.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@

from dataretrieval import nwdc as _nwdc
from dataretrieval._deprecation import REMOVALS, warn_deprecated
from dataretrieval.nwdc import * # noqa: F403 (re-export the public surface)
from dataretrieval.nwdc import * # noqa: F403 # re-export the public surface

#: When the alias may be deleted. Read from the shared horizon table rather
#: than spelled here, so it is audited and bumped with every other published
Expand Down
2 changes: 2 additions & 0 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
"""Sphinx build configuration."""

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
Expand Down
Loading