diff --git a/AGENTS.md b/AGENTS.md index 8a19b7b4..b0c9006f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,8 +4,14 @@ - **`CONTEXT.md` is the shared vocabulary** — getter, query, chunk, plan, fan-out, page, adapter, facade, leaf, transport, collection, profile, effective configuration, and the legacy names that are deliberately not renamed. Read it - before writing code, docstrings, or commit messages; when a term there - conflicts with a name in the code, the term wins. + before writing code, docstrings, or commit messages. Two kinds of term live + there and they bind differently (ADR 0013). A **core** term is ours and takes + one spelling everywhere *including identifiers*, so a name in the code that + conflicts with it is a defect. A **domain** term -- *monitoring location*, + *collection* -- is fixed for prose only: each adapter keeps its own service's + spelling in its parameters, so `nwis.get_record(service="dv")` and WQP's + `Station` are that service's language rather than drift. Each entry states + which kind it is. - Architectural decisions and their rationale: `docs/source/architecture/decisions/` (ADRs, referenced by number throughout the code and by `.importlinter`). - Contributor workflow, style, and the quality gates in detail: `CONTRIBUTING.md`. diff --git a/CONTEXT.md b/CONTEXT.md index eda9288d..f89fb2c4 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,8 +5,25 @@ specification: it fixes what words mean so that code, docstrings, ADRs, and conversation use them the same way. Architectural decisions live in `docs/source/architecture/decisions/`. -When a term here conflicts with a name in the code, the term wins and the name -is legacy. Legacy names are listed below. +Two kinds of term live here, and they carry different obligations (ADR 0013). + +**Core terms** are ours. The package invented them and no service has a claim on +them — *chunk*, *page*, *fan-out*, *source*, *dialect*, *leaf*. One spelling, +everywhere it appears: prose, identifiers, tests. Where a core term conflicts +with a name in the code the term wins and the name is legacy, listed at the end. +A second spelling is a defect, not a variation. + +**Domain terms** belong to the services, which name the same thing differently +and will not be reconciled. For these the glossary fixes one word for *prose*, +so that documents about the package read consistently. It does not fix the wire +or an adapter's public surface: each adapter keeps its own service's spelling in +its parameters. Such an entry names the per-service spellings itself; those are +not legacy names. + +A word carrying a package-wide meaning has an entry here, and one entry may +name another; what no entry may do is lean on a word this document leaves +undefined. Naming a word only to say what an ADR calls it is a cross-reference, +not a use. ## Retrieval @@ -102,13 +119,24 @@ collections. The distinction matters because the OGC machinery is shared: the same code path retrieves a Water Data collection and an NGWMN one, and only the service differs. +*Domain term.* Water Data and NGWMN say `collection`; legacy NWIS says +`service=` (`'dv'`, `'peaks'`); WQP names its collections as profiles +(`Result`, `Station`, `Activity`). Those parameters are frozen public surface +and keep their spelling. Prose says *collection*, including prose about the +adapters that spell it otherwise. + **Collection family** — A group of collections sharing a shape and therefore a getter signature. Their getters deliberately resemble one another; the resemblance is the public contract, not duplication to be removed. -**Monitoring location** — A place where measurements are recorded. The canonical -term. Legacy: the deprecated NWIS getters and the WQP profiles call this a -*site*, and their parameters keep that spelling. +**Monitoring location** — A place where measurements are recorded. + +*Domain term.* The services disagree, and each adapter keeps its own service's +spelling in its parameters: NWIS `site_no` and `sites=`, WQP `Station` and +`siteid`, Water Data `monitoring_location_id`, NGWMN's `sites` collection. Where +a service names a thing `site-types` or `site_type_code`, that is its vocabulary +and is reproduced rather than translated. Prose that is not about one particular +adapter says *monitoring location*. **Metadata** — The second half of every getter's return: the request URL, the elapsed time, and the response headers. Describes the *retrieval*, not the data. @@ -144,12 +172,18 @@ A public keyword is not automatically a setting. `ssl_check` is a getter argument on four adapters and resolves through no chain at all; the settings are the roster the configuration system knows. +**Scope** — How much of the package a setting's value applies to: the whole +package, or one adapter. Orthogonal to source: the scope says who a value is +for, the source says where it came from, and precedence orders sources first, +scopes within them. ADR 0010's word for a scope level is *tier* — the top-level +tier that survives, the host or gateway tier it defers. + **Package-wide setting** — A setting that applies to every adapter: the retry count, the progress line, the stall timeout. Set once, honored everywhere. **Adapter-scoped setting** — A setting named under one adapter, applying to that adapter and no other. It overrides the package-wide value for that adapter -alone; it does not replace the package-wide tier. An adapter rejects a setting +alone, leaving that value standing for every other adapter. An adapter rejects a setting it has no use for, rather than accepting and ignoring it. The scope is the *adapter*, not the service and not the host, because the @@ -162,8 +196,16 @@ Credentials are host-scoped; tunables are adapter-scoped. categories: a `configure()` block, the environment, the file, the built-in default. The order is resolved per setting rather than per source: a value supplied for one setting does not displace another setting's value from a lower -source. ADR 0010 calls a source a *tier* and ADR 0011 a *rung*; both are this -term, and the accepted records keep their own wording. +source. + +*Core term.* The accepted records already say it: ADR 0009 resolves settings +by source, and ADR 0010 keeps precedence *source-major*. ADR 0010's *tier* is a +different axis — the scope — and ADR 0011's *rungs* are positions of its merged +precedence ladder, where sources and scopes interleave. Neither is a second +spelling of this term, and the records keep their own wording. + +An external system is a *service*, never a source. NLDI's `data_source` +parameters are that service's own vocabulary, reproduced like `site-types`. **Origin label** — The exact thing a value came from, at finer grain than its source: `$API_USGS_RETRIES`, a path to the config file, the profile a caller @@ -171,9 +213,17 @@ selected. What `show_configuration()` prints beside each value, and what a parser names when it rejects one. A source is the category; an origin label is the instance within it. -The code carries both, and spells them the other way around: `_resolve` returns -its origin label under the name `source` and its source under the name `tier`. -Prose uses the terms above. +*Core terms.* The configuration chain is shared machinery, so one spelling binds +its identifiers as well as its prose. Where the chain's identifiers have said +`tier` for the source and `source` for the origin label, that is a defect to +correct — ADR 0010's scope word misapplied to a different axis — not a second +spelling to work around. + +**Precedence ladder** — The one linear order resolution walks, highest first: +sources in their order and, within each source, scopes. Each position is a +**rung**; ADR 0011 states the ladder in seven. A rung is finer than a source — +one source can span several rungs — so prose that means a whole category says +*source*, reserving *rung* for when the position itself matters. **Selection** — Naming which profile an adapter should use. Done in code; a profile is never selected by the environment or implied by the file, so the @@ -221,7 +271,7 @@ these and `.importlinter`. **Facade** — A module that re-exports a subsystem's public surface and contains no logic of its own, so callers depend on a stable name rather than on internal -layout. +layout. The name is the structural design pattern's, not a coinage. **Leaf** — A module with no dependencies inside the package beyond other leaves, holding one general mechanism so that anything may use it without acquiring the @@ -234,7 +284,11 @@ and is not public API. ## Known legacy names -Recorded so they are not mistaken for the canonical term, and not re-litigated: +Core-term spellings recorded so they are not mistaken for drift, and not +re-litigated: frozen misnamings, permanent aliases, and names that agree with +this glossary by more than luck. A domain term at an adapter's surface is not a +legacy name and is not listed here; it belongs with that term's own entry +(ADR 0013). - `completed_chunks` / `total_chunks` on interruptions, and `set_chunks()` / `start_chunk()` on the progress reporter, count chunks as defined above and @@ -243,17 +297,13 @@ Recorded so they are not mistaken for the canonical term, and not re-litigated: - `ChunkInterrupted` is a permanent alias of `FanOutInterrupted` — the same class object under the name it was first published as. Both spellings are correct; neither is scheduled for removal. -- *No-progress budget* is ADR 0006's name for the **stall timeout**. Both - spellings are current; the setting is `stall_timeout`. +- *No-progress budget* is ADR 0006's name for the **stall timeout**. The + record keeps its wording; prose outside it says *stall timeout*, and the + setting is `stall_timeout`. - `ChunkedCall` is a permanent alias of `FanOut`, published on the OGC compatibility path. Like `ChunkInterrupted`, both spellings are correct. - `utils.query` is one *request*, not a query as defined above. It is a frozen public path (`dataretrieval.utils.query`) and predates this glossary. -- `site` appears in deprecated NWIS and WQP parameter names where *monitoring - location* is meant. These are frozen public surfaces and will not be renamed. - Where the Water Data API itself names a thing `site-types` or - `site_type_code`, that is the service's vocabulary and is reproduced - faithfully rather than translated. - `service` named a collection throughout the OGC machinery. Resolved: the OGC internals, the Water Data wrappers, and all eleven typed getters now say `collection`; `waterdata.get_cql` takes `collection`; and the type alias is diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a16ef5d4..7c2b7cbf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -244,7 +244,7 @@ link checking. * Follow the [PEP8 style guidelines](https://peps.python.org/pep-0008/). * The public interface should emphasize functions over classes; classes can and should be used internally and in tests. -* Group public download functions by data portal. For example, modern Water +* Group public download functions by service. For example, modern Water Data functions belong in `dataretrieval.waterdata`; legacy NWIS functions remain quarantined in `dataretrieval.nwis` during deprecation. * Treat a change to a service's documented return shape or metadata type as a diff --git a/docs/source/architecture/decisions/0013-core-and-domain-terms.rst b/docs/source/architecture/decisions/0013-core-and-domain-terms.rst new file mode 100644 index 00000000..8e5d1158 --- /dev/null +++ b/docs/source/architecture/decisions/0013-core-and-domain-terms.rst @@ -0,0 +1,147 @@ +ADR 0013: Distinguish the terms we own from the terms the services own +====================================================================== + +Status +------ + +Accepted + +Context +------- + +``CONTEXT.md`` is one flat glossary. Every term in it reads as equally binding, +and every place the code disagrees is filed under *Known legacy names* -- a +list whose framing is that the disagreement is debt, tolerated until someone +gets to it. + +For most of the glossary that framing is right. But it is wrong for a small set +of terms, and being wrong about those has produced the same review argument +repeatedly: whether a docstring may say *site*, whether ``service=`` may name a +collection, whether prose about NWIS is bound by a word chosen from the Water +Data API. + +The two sets behave differently because their authority differs. + +Terms like *chunk*, *page*, *fan-out*, *plan*, *interruption*, *dialect* and +*leaf* appear nowhere in any USGS API's vocabulary. They were invented here to +describe machinery this package owns. Nothing external constrains them, so when +the package spells one of them two ways -- the resolution chain's code said +*tier* for what its founding records, ADRs 0009 and 0010, call a *source* -- +that is simply an inconsistency, and one that can be removed by deciding. + +Terms like *monitoring location* and *collection* are different. The services +name those things, and they do not agree with each other: + +.. list-table:: + :header-rows: 1 + + * - Concept + - NWIS + - WQP + - Water Data + - NGWMN + * - a place where measurements are recorded + - ``site_no``, ``sites=`` + - ``Station``, ``siteid`` + - ``monitoring_location_id`` + - the ``sites`` collection + * - a named set of records + - ``service=`` + - ``Result``, ``Station`` + - ``collection`` + - ``collection`` + +No decision here makes those agree. A caller who has read the WQP +documentation looks for ``Station``; one reading Water Data's looks for +``monitoring_location_id``. An adapter that renamed either would be harder to +use, not easier, and the parameter names are public surface besides. + +Treating both sets under one rule forces a choice between two bad options: +abandon the glossary, and the shared modules lose the vocabulary that lets them +be shared; or enforce it everywhere, and every adapter's public surface drifts +from the API it wraps. + +Decision +-------- + +The glossary holds two kinds of term, and they carry different obligations. + +**Core terms are ours.** The package invented them and no service has a claim +on them: everything under *Retrieval*, *Failure and resumption*, *Configuration* +and *Boundaries*, plus *Collection family* and *Metadata*. One spelling, +enforced everywhere it appears -- prose, identifiers, tests. A second spelling +of a core term is a defect, not a variation, and is fixed rather than recorded. +This is what makes the lower-level modules shareable: transport, configuration +and the OGC engine can be written once because the words they are written in +answer to nothing outside this package. + +**Domain terms belong to the services.** *Monitoring location* and *collection* +name things the services define and spell differently. For these the glossary +chooses one term for **prose**, so that documents about the package are +internally consistent. It does not choose for the wire, and it does not choose +for an adapter's public surface: each adapter keeps its own service's spelling +in its parameters, and reproduces that service's vocabulary faithfully where it +appears in returned data. + +An adapter is where the two meet. Its public surface speaks its service's +language; what it hands to the shared modules speaks the core's. The +translation is the adapter's job, and a divergence at that boundary is the +design working rather than debt. + +Two rules follow: + +- **A term the glossary does not define is not used in the glossary.** A word + that earns a place in ``CONTEXT.md``'s prose earns an entry. Naming a term + only to say what an ADR calls it is a cross-reference, not a definition, and + does not license using the word elsewhere. +- **Only core misnamings are legacy.** *Known legacy names* records a core term + the code spells wrongly and cannot be renamed. A domain term at an adapter's + surface is not a legacy name; it is that adapter speaking its service's + language, and belongs with the term's own entry. + +Consequences +------------ + +- The recurring question -- may this docstring say *site*? -- has an answer that + does not depend on who is reviewing. In ``nwis`` it may, because that is the + spelling its service and its parameters use. In ``transport`` it may not, + because nothing there is about NWIS. +- Enforcement splits. A core term can be checked mechanically, since one + spelling is correct everywhere. A domain term cannot: the correct spelling + depends on which adapter the prose is about, so it stays a review judgement. +- *Known legacy names* becomes shorter and means something narrower. The entries + it loses are not resolved; they move to the term they belong to, as part of + its definition rather than a list of exceptions. +- A glossary entry now carries an obligation to say which kind it is. That is a + small cost per term and the reason the distinction is usable at all. +- The package's own inconsistencies in core vocabulary become defects with a + deadline rather than curiosities. The resolution chain's ``tier``-for-*source* + identifiers are the standing example. + +Compliance +---------- + +``CONTEXT.md`` marks each domain term as such and names the per-service +spellings in the entry itself, so a reader who needs to know whether a word is +negotiable can see it without asking. + +The mechanical part is that the glossary must define what it uses: +``tests/architecture_test.py`` asserts every ``ADR NNNN`` citation resolves, and +the same file is where a check that ``CONTEXT.md`` defines its own vocabulary +would go. None is proposed yet -- a word-list check over prose has a poor +precision record in this repository, and the failure it would catch is one a +reader notices immediately. + +Whether a given adapter docstring should say *site* or *monitoring location* +remains a review judgement, and is meant to. + +Notes +----- + +Prompted by review of :doc:`0000-documenting-decisions`, where a reviewer found +``CONTEXT.md`` using *tier* in its own prose without defining it. The word was +already there before that record was written; recording each explanation once +made the gap visible rather than creating it. + +The per-service spellings in the table above were read from the adapters on +2026-09-01. diff --git a/docs/source/architecture/decisions/index.rst b/docs/source/architecture/decisions/index.rst index a5978707..29825505 100644 --- a/docs/source/architecture/decisions/index.rst +++ b/docs/source/architecture/decisions/index.rst @@ -31,4 +31,5 @@ belong here rather than in a docstring, an inline comment, or a commit message. 0010-adapter-scoped-settings 0011-configuration-profiles 0012-deprecation-horizons + 0013-core-and-domain-terms template diff --git a/docs/source/architecture/index.rst b/docs/source/architecture/index.rst index 85fa47e8..4e3983ff 100644 --- a/docs/source/architecture/index.rst +++ b/docs/source/architecture/index.rst @@ -202,8 +202,7 @@ than be forbidden, and full-graph cycle detection (see ADR 0003). Interface view -------------- -The primary API is a collection of synchronous functions grouped by data -portal. Most tabular download functions return ``(DataFrame, metadata)``. +The primary API is a collection of synchronous functions grouped by service. Most tabular download functions return ``(DataFrame, metadata)``. NLDI and StreamStats retain service-specific geospatial or response-object contracts. diff --git a/docs/source/userguide/configuration.rst b/docs/source/userguide/configuration.rst index 85472629..c2dcafa6 100644 --- a/docs/source/userguide/configuration.rst +++ b/docs/source/userguide/configuration.rst @@ -316,7 +316,7 @@ So one file can hold an overnight bulk shape beside a polite daytime one, and name as many of each as an adapter has uses for. A named profile states only what differs: everything it does not name still -comes from the adapter's default profile, the package-wide keys, and the tiers +comes from the adapter's default profile, the package-wide keys, and the rungs below — per setting. ``load`` reads the table and hands you a configuration object, so a name the