Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,14 @@
- **`CONTEXT.md` is the shared vocabulary** — getter, query, chunk, plan, fan-out,
page, adapter, facade, leaf, transport, collection, profile, effective
configuration, and the legacy names that are deliberately not renamed. Read it
before writing code, docstrings, or commit messages; when a term there
conflicts with a name in the code, the term wins.
before writing code, docstrings, or commit messages. Two kinds of term live
there and they bind differently (ADR 0013). A **core** term is ours and takes
one spelling everywhere *including identifiers*, so a name in the code that
conflicts with it is a defect. A **domain** term -- *monitoring location*,
*collection* -- is fixed for prose only: each adapter keeps its own service's
spelling in its parameters, so `nwis.get_record(service="dv")` and WQP's
`Station` are that service's language rather than drift. Each entry states
which kind it is.
- Architectural decisions and their rationale: `docs/source/architecture/decisions/`
(ADRs, referenced by number throughout the code and by `.importlinter`).
- Contributor workflow, style, and the quality gates in detail: `CONTRIBUTING.md`.
Expand Down
90 changes: 70 additions & 20 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,25 @@ specification: it fixes what words mean so that code, docstrings, ADRs, and
conversation use them the same way. Architectural decisions live in
`docs/source/architecture/decisions/`.

When a term here conflicts with a name in the code, the term wins and the name
is legacy. Legacy names are listed below.
Two kinds of term live here, and they carry different obligations (ADR 0013).

**Core terms** are ours. The package invented them and no service has a claim on
them — *chunk*, *page*, *fan-out*, *source*, *dialect*, *leaf*. One spelling,
everywhere it appears: prose, identifiers, tests. Where a core term conflicts
with a name in the code the term wins and the name is legacy, listed at the end.
A second spelling is a defect, not a variation.

**Domain terms** belong to the services, which name the same thing differently
and will not be reconciled. For these the glossary fixes one word for *prose*,
so that documents about the package read consistently. It does not fix the wire
or an adapter's public surface: each adapter keeps its own service's spelling in
its parameters. Such an entry names the per-service spellings itself; those are
not legacy names.

A word carrying a package-wide meaning has an entry here, and one entry may
name another; what no entry may do is lean on a word this document leaves
undefined. Naming a word only to say what an ADR calls it is a cross-reference,
not a use.

## Retrieval

Expand Down Expand Up @@ -102,13 +119,24 @@ collections. The distinction matters because the OGC machinery is shared: the
same code path retrieves a Water Data collection and an NGWMN one, and only the
service differs.

*Domain term.* Water Data and NGWMN say `collection`; legacy NWIS says
`service=` (`'dv'`, `'peaks'`); WQP names its collections as profiles
(`Result`, `Station`, `Activity`). Those parameters are frozen public surface
and keep their spelling. Prose says *collection*, including prose about the
adapters that spell it otherwise.

**Collection family** — A group of collections sharing a shape and therefore a
getter signature. Their getters deliberately resemble one another; the
resemblance is the public contract, not duplication to be removed.

**Monitoring location** — A place where measurements are recorded. The canonical
term. Legacy: the deprecated NWIS getters and the WQP profiles call this a
*site*, and their parameters keep that spelling.
**Monitoring location** — A place where measurements are recorded.

*Domain term.* The services disagree, and each adapter keeps its own service's
spelling in its parameters: NWIS `site_no` and `sites=`, WQP `Station` and
`siteid`, Water Data `monitoring_location_id`, NGWMN's `sites` collection. Where
a service names a thing `site-types` or `site_type_code`, that is its vocabulary
and is reproduced rather than translated. Prose that is not about one particular
adapter says *monitoring location*.

**Metadata** — The second half of every getter's return: the request URL, the
elapsed time, and the response headers. Describes the *retrieval*, not the data.
Expand Down Expand Up @@ -144,12 +172,18 @@ A public keyword is not automatically a setting. `ssl_check` is a getter
argument on four adapters and resolves through no chain at all; the settings are
the roster the configuration system knows.

**Scope** — How much of the package a setting's value applies to: the whole
package, or one adapter. Orthogonal to source: the scope says who a value is
for, the source says where it came from, and precedence orders sources first,
scopes within them. ADR 0010's word for a scope level is *tier* — the top-level
tier that survives, the host or gateway tier it defers.

**Package-wide setting** — A setting that applies to every adapter: the retry
count, the progress line, the stall timeout. Set once, honored everywhere.

**Adapter-scoped setting** — A setting named under one adapter, applying to
that adapter and no other. It overrides the package-wide value for that adapter
alone; it does not replace the package-wide tier. An adapter rejects a setting
alone, leaving that value standing for every other adapter. An adapter rejects a setting
it has no use for, rather than accepting and ignoring it.

The scope is the *adapter*, not the service and not the host, because the
Expand All @@ -162,18 +196,34 @@ Credentials are host-scoped; tunables are adapter-scoped.
categories: a `configure()` block, the environment, the file, the built-in
default. The order is resolved per setting rather than per source: a value
supplied for one setting does not displace another setting's value from a lower
source. ADR 0010 calls a source a *tier* and ADR 0011 a *rung*; both are this
term, and the accepted records keep their own wording.
source.

*Core term.* The accepted records already say it: ADR 0009 resolves settings
by source, and ADR 0010 keeps precedence *source-major*. ADR 0010's *tier* is a
different axis — the scope — and ADR 0011's *rungs* are positions of its merged
precedence ladder, where sources and scopes interleave. Neither is a second
spelling of this term, and the records keep their own wording.

An external system is a *service*, never a source. NLDI's `data_source`
parameters are that service's own vocabulary, reproduced like `site-types`.

**Origin label** — The exact thing a value came from, at finer grain than its
source: `$API_USGS_RETRIES`, a path to the config file, the profile a caller
selected. What `show_configuration()` prints beside each value, and what a
parser names when it rejects one. A source is the category; an origin label is
the instance within it.

The code carries both, and spells them the other way around: `_resolve` returns
its origin label under the name `source` and its source under the name `tier`.
Prose uses the terms above.
*Core terms.* The configuration chain is shared machinery, so one spelling binds
its identifiers as well as its prose. Where the chain's identifiers have said
`tier` for the source and `source` for the origin label, that is a defect to
correct — ADR 0010's scope word misapplied to a different axis — not a second
spelling to work around.

**Precedence ladder** — The one linear order resolution walks, highest first:
sources in their order and, within each source, scopes. Each position is a
**rung**; ADR 0011 states the ladder in seven. A rung is finer than a source —
one source can span several rungs — so prose that means a whole category says
*source*, reserving *rung* for when the position itself matters.

**Selection** — Naming which profile an adapter should use. Done in code; a
profile is never selected by the environment or implied by the file, so the
Expand Down Expand Up @@ -221,7 +271,7 @@ these and `.importlinter`.

**Facade** — A module that re-exports a subsystem's public surface and contains
no logic of its own, so callers depend on a stable name rather than on internal
layout.
layout. The name is the structural design pattern's, not a coinage.

**Leaf** — A module with no dependencies inside the package beyond other leaves,
holding one general mechanism so that anything may use it without acquiring the
Expand All @@ -234,7 +284,11 @@ and is not public API.

## Known legacy names

Recorded so they are not mistaken for the canonical term, and not re-litigated:
Core-term spellings recorded so they are not mistaken for drift, and not
re-litigated: frozen misnamings, permanent aliases, and names that agree with
this glossary by more than luck. A domain term at an adapter's surface is not a
legacy name and is not listed here; it belongs with that term's own entry
(ADR 0013).

- `completed_chunks` / `total_chunks` on interruptions, and `set_chunks()` /
`start_chunk()` on the progress reporter, count chunks as defined above and
Expand All @@ -243,17 +297,13 @@ Recorded so they are not mistaken for the canonical term, and not re-litigated:
- `ChunkInterrupted` is a permanent alias of `FanOutInterrupted` — the same
class object under the name it was first published as. Both spellings are
correct; neither is scheduled for removal.
- *No-progress budget* is ADR 0006's name for the **stall timeout**. Both
spellings are current; the setting is `stall_timeout`.
- *No-progress budget* is ADR 0006's name for the **stall timeout**. The
record keeps its wording; prose outside it says *stall timeout*, and the
setting is `stall_timeout`.
- `ChunkedCall` is a permanent alias of `FanOut`, published on the OGC
compatibility path. Like `ChunkInterrupted`, both spellings are correct.
- `utils.query` is one *request*, not a query as defined above. It is a frozen
public path (`dataretrieval.utils.query`) and predates this glossary.
- `site` appears in deprecated NWIS and WQP parameter names where *monitoring
location* is meant. These are frozen public surfaces and will not be renamed.
Where the Water Data API itself names a thing `site-types` or
`site_type_code`, that is the service's vocabulary and is reproduced
faithfully rather than translated.
- `service` named a collection throughout the OGC machinery. Resolved: the
OGC internals, the Water Data wrappers, and all eleven typed getters now say
`collection`; `waterdata.get_cql` takes `collection`; and the type alias is
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ link checking.
* Follow the [PEP8 style guidelines](https://peps.python.org/pep-0008/).
* The public interface should emphasize functions over classes; classes can and
should be used internally and in tests.
* Group public download functions by data portal. For example, modern Water
* Group public download functions by service. For example, modern Water
Data functions belong in `dataretrieval.waterdata`; legacy NWIS functions
remain quarantined in `dataretrieval.nwis` during deprecation.
* Treat a change to a service's documented return shape or metadata type as a
Expand Down
147 changes: 147 additions & 0 deletions docs/source/architecture/decisions/0013-core-and-domain-terms.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
ADR 0013: Distinguish the terms we own from the terms the services own
======================================================================

Status
------

Accepted

Context
-------

``CONTEXT.md`` is one flat glossary. Every term in it reads as equally binding,
and every place the code disagrees is filed under *Known legacy names* -- a
list whose framing is that the disagreement is debt, tolerated until someone
gets to it.

For most of the glossary that framing is right. But it is wrong for a small set
of terms, and being wrong about those has produced the same review argument
repeatedly: whether a docstring may say *site*, whether ``service=`` may name a
collection, whether prose about NWIS is bound by a word chosen from the Water
Data API.

The two sets behave differently because their authority differs.

Terms like *chunk*, *page*, *fan-out*, *plan*, *interruption*, *dialect* and
*leaf* appear nowhere in any USGS API's vocabulary. They were invented here to
describe machinery this package owns. Nothing external constrains them, so when
the package spells one of them two ways -- the resolution chain's code said
*tier* for what its founding records, ADRs 0009 and 0010, call a *source* --
that is simply an inconsistency, and one that can be removed by deciding.

Terms like *monitoring location* and *collection* are different. The services
name those things, and they do not agree with each other:

.. list-table::
:header-rows: 1

* - Concept
- NWIS
- WQP
- Water Data
- NGWMN
* - a place where measurements are recorded
- ``site_no``, ``sites=``
- ``Station``, ``siteid``
- ``monitoring_location_id``
- the ``sites`` collection
* - a named set of records
- ``service=``
- ``Result``, ``Station``
- ``collection``
- ``collection``

No decision here makes those agree. A caller who has read the WQP
documentation looks for ``Station``; one reading Water Data's looks for
``monitoring_location_id``. An adapter that renamed either would be harder to
use, not easier, and the parameter names are public surface besides.

Treating both sets under one rule forces a choice between two bad options:
abandon the glossary, and the shared modules lose the vocabulary that lets them
be shared; or enforce it everywhere, and every adapter's public surface drifts
from the API it wraps.

Decision
--------

The glossary holds two kinds of term, and they carry different obligations.

**Core terms are ours.** The package invented them and no service has a claim
on them: everything under *Retrieval*, *Failure and resumption*, *Configuration*
and *Boundaries*, plus *Collection family* and *Metadata*. One spelling,
enforced everywhere it appears -- prose, identifiers, tests. A second spelling
of a core term is a defect, not a variation, and is fixed rather than recorded.
This is what makes the lower-level modules shareable: transport, configuration
and the OGC engine can be written once because the words they are written in
answer to nothing outside this package.

**Domain terms belong to the services.** *Monitoring location* and *collection*
name things the services define and spell differently. For these the glossary
chooses one term for **prose**, so that documents about the package are
internally consistent. It does not choose for the wire, and it does not choose
for an adapter's public surface: each adapter keeps its own service's spelling
in its parameters, and reproduces that service's vocabulary faithfully where it
appears in returned data.

An adapter is where the two meet. Its public surface speaks its service's
language; what it hands to the shared modules speaks the core's. The
translation is the adapter's job, and a divergence at that boundary is the
design working rather than debt.

Two rules follow:

- **A term the glossary does not define is not used in the glossary.** A word
that earns a place in ``CONTEXT.md``'s prose earns an entry. Naming a term
only to say what an ADR calls it is a cross-reference, not a definition, and
does not license using the word elsewhere.
- **Only core misnamings are legacy.** *Known legacy names* records a core term
the code spells wrongly and cannot be renamed. A domain term at an adapter's
surface is not a legacy name; it is that adapter speaking its service's
language, and belongs with the term's own entry.

Consequences
------------

- The recurring question -- may this docstring say *site*? -- has an answer that
does not depend on who is reviewing. In ``nwis`` it may, because that is the
spelling its service and its parameters use. In ``transport`` it may not,
because nothing there is about NWIS.
- Enforcement splits. A core term can be checked mechanically, since one
spelling is correct everywhere. A domain term cannot: the correct spelling
depends on which adapter the prose is about, so it stays a review judgement.
- *Known legacy names* becomes shorter and means something narrower. The entries
it loses are not resolved; they move to the term they belong to, as part of
its definition rather than a list of exceptions.
- A glossary entry now carries an obligation to say which kind it is. That is a
small cost per term and the reason the distinction is usable at all.
- The package's own inconsistencies in core vocabulary become defects with a
deadline rather than curiosities. The resolution chain's ``tier``-for-*source*
identifiers are the standing example.

Compliance
----------

``CONTEXT.md`` marks each domain term as such and names the per-service
spellings in the entry itself, so a reader who needs to know whether a word is
negotiable can see it without asking.

The mechanical part is that the glossary must define what it uses:
``tests/architecture_test.py`` asserts every ``ADR NNNN`` citation resolves, and
the same file is where a check that ``CONTEXT.md`` defines its own vocabulary
would go. None is proposed yet -- a word-list check over prose has a poor
precision record in this repository, and the failure it would catch is one a
reader notices immediately.

Whether a given adapter docstring should say *site* or *monitoring location*
remains a review judgement, and is meant to.

Notes
-----

Prompted by review of :doc:`0000-documenting-decisions`, where a reviewer found
``CONTEXT.md`` using *tier* in its own prose without defining it. The word was
already there before that record was written; recording each explanation once
made the gap visible rather than creating it.

The per-service spellings in the table above were read from the adapters on
2026-09-01.
1 change: 1 addition & 0 deletions docs/source/architecture/decisions/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,5 @@ belong here rather than in a docstring, an inline comment, or a commit message.
0010-adapter-scoped-settings
0011-configuration-profiles
0012-deprecation-horizons
0013-core-and-domain-terms
template
3 changes: 1 addition & 2 deletions docs/source/architecture/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -202,8 +202,7 @@ than be forbidden, and full-graph cycle detection (see ADR 0003).
Interface view
--------------

The primary API is a collection of synchronous functions grouped by data
portal. Most tabular download functions return ``(DataFrame, metadata)``.
The primary API is a collection of synchronous functions grouped by service. Most tabular download functions return ``(DataFrame, metadata)``.
NLDI and StreamStats retain service-specific geospatial or response-object
contracts.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/userguide/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ So one file can hold an overnight bulk shape beside a polite daytime one, and
name as many of each as an adapter has uses for.

A named profile states only what differs: everything it does not name still
comes from the adapter's default profile, the package-wide keys, and the tiers
comes from the adapter's default profile, the package-wide keys, and the rungs
below — per setting.

``load`` reads the table and hands you a configuration object, so a name the
Expand Down