From ba6967254fb40fc04de44dee300ec806b2f3ae12 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 18:37:22 +0000 Subject: [PATCH 1/2] =?UTF-8?q?docs(adr):=20ADR-0104=20addendum=20?= =?UTF-8?q?=E2=80=94=20the=20file=20family's=20stored=20column=20holds=20t?= =?UTF-8?q?he=20bare=20sys=5Ffile=20id,=20switched=20per=20deployment=20on?= =?UTF-8?q?=20the=20adr-0104-file-references=20flag?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the maintainer's decision-batch #49 item 1 ruling (Option A) as a 2026-09-05 addendum to ADR-0104: the media family's (image / file / avatar / video / audio) single-value physical column is a string column holding the bare sys_file id — the generator's VARCHAR(2048) / table.string is the ruled end-state and the SQL driver moves to it; the encoding switch is per deployment, keyed on the existing adr-0104-file-references sys_migration row and never on a version, with the column move as a further step of `os migrate files-to-references --apply` after zero blocking findings; the dual-encoding window this implies, its invariant (column type and write encoding never disagree on one deployment), the three populations it must hold over, and its end (the first protocol major after the driver lands, with a loud boot refusal for un-moved deployments); the two confidence gaps carried from the measurement stated as gaps with what closes them; and the sequencing behind the driver card. Governed surface (docs/adr). No code, schema, generated artifact or changeset moves with this commit. Anchors are symbol / file anchors only. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01M59rPZZFzqhfMUPFqqZTkf --- ...0104-field-runtime-value-shape-contract.md | 288 ++++++++++++++++++ 1 file changed, 288 insertions(+) diff --git a/docs/adr/0104-field-runtime-value-shape-contract.md b/docs/adr/0104-field-runtime-value-shape-contract.md index 408d1c5872..39af18e7c8 100644 --- a/docs/adr/0104-field-runtime-value-shape-contract.md +++ b/docs/adr/0104-field-runtime-value-shape-contract.md @@ -1020,3 +1020,291 @@ carries the aliased vocabulary's *coverage* across with its names. `FileRef` scoped to `file`/`image`/`video`/`audio` would leave `avatar` on the legacy inline shape, which is precisely the per-type carve-out that addendum rejected. + +## Addendum (2026-09-05) — the media family's physical column holds the bare `sys_file` id, switched per deployment on the `adr-0104-file-references` flag + +**Provenance.** Maintainer ruling, decision batch #49 item 1, recorded on +#15041 (comment `5551135629`) by the director seat from the maintainer's live +reply, verbatim: 「15041 应该改为实际 id 保存。选A，其他同意」. The measurement it +ruled on is #15041's `os-dev-report` (comment `5550175673`) and its H1 table +(comment `5550175730`), taken on `origin/main` `8e500f23e` on 2026-09-05. This +addendum is step 1 of the ruling's execution order: it records the decisions +about the column and the switch; no code moves with it. + +### The fork it closes + +D1 handed the column to the driver in one clause — "DDL column choice remains +the driver's decision; the *classification* moves to the spec" — and D3 then +narrowed the media family's **stored value** to an opaque `sys_file` id. Every +declared D3 wave has landed (spec `17.0.0`); none narrowed the column, because +nothing had scheduled that. So two readings of "the value is an id" coexisted +on `main`, each internally consistent: + +- **The driver: a JSON-quoted id in a JSON column.** + `packages/drivers/driver-sql/src/sql-driver.ts#JSON_COLUMN_TYPES` seeds the + family by spreading `FILE_REFERENCE_TYPES`, so + `packages/drivers/driver-sql/src/sql-driver.ts#isJsonField` answers true for + all five, `packages/drivers/driver-sql/src/sql-driver.ts#formatInput` + JSON-stringifies the id on every dialect (the #12380 canonical encoding), and + the column is `packages/drivers/driver-sql/src/sql-driver.ts#jsonColumn` — + `json` on Postgres and MySQL, TEXT on SQLite. Measured on SQLite (two + in-memory cells, 2026-09-05): an id written through the driver lands on disk + as the quoted text `"file_01HXYZ"` and reads back as the bare string; an + inline object written through the driver reads back as an object; a raw + bare id inserted by hand makes the SQLite read arm's `JSON.parse` throw, the + catch keeps the raw string, and it reads back as `file_01HXYZ` — no fault. + The same four rows in a hand-created `VARCHAR(2048)` column read back + byte-identical, and `initObjects` over that column left it `VARCHAR(2048)`: + additive sync never alters a column's type. Postgres and MySQL were **not** + measured (no live cell in the container); reasoned from the source: the + write quotes identically, the read path has no parse arm because those + clients parse a native `json` column, so a varchar there returns the quoted + text verbatim, `isFileIdToken` refuses it, nothing expands it and nothing + claims ownership. The reachable harm of a hand-run generated migration on + those dialects is therefore write-side quoting into a varchar — silent — and + `packages/drivers/driver-sql/src/schema-drift.ts#declaresJsonColumn` keys + the json-vs-text finding on `multiple` alone, so the drift report never + names it (#15771). +- **The generator: a bare id in a string column.** + `packages/cli/src/commands/generate.ts#FIELD_TYPE_SQL_MAP` gives all five + members `VARCHAR(2048)`; the TypeScript form emits `table.string(...)` from + one switch arm — #14657's reading of the spec contract as DDL. The pin + `packages/cli/src/commands/generate-field-type-vocabulary.pin.test.ts` + records the disagreement under a block labelled "Recorded divergence, NOT + coverage", so it cannot change shape unnoticed and no reader mistakes it + for a ruling. + +The spec seat recommended that the generator copy the driver (B), on the +ground that the value contract and the physical column are different things +and the column the driver creates is the one every SQL deployment has. The +maintainer overruled that on exactly the point the seat had flagged as the +maintainer's to decide: **the physical column follows the contract.** The +ruling as the director seat recorded it: "Option A. The physical column for +`file` / `image` / `avatar` / `video` / `audio` holds the actual id — a bare +`sys_file` id string in a string column — not a JSON-quoted id in a JSON +column. The driver is the side that moves; the generator's `VARCHAR(2048)` +already states the ruled end-state and stands." B and C (status quo) are +rejected: the id is the value, and the column should say so. + +### The column: a string column holding the bare id + +For every member of `FILE_REFERENCE_TYPES` (`image`, `file`, `avatar`, +`video`, `audio`), the single-value physical column is a **string column +holding the bare `sys_file` id** — the value `FileReferenceIdValueSchema` +admits, stored as itself. The generator's `VARCHAR(2048)` / `table.string` +is the ruled end-state and stands; the driver moves to it: the family leaves +`JSON_COLUMN_TYPES`, and `isJsonField` / `formatInput` / +`packages/drivers/driver-sql/src/sql-driver.ts#formatOutput` stop treating it +as JSON. The driver's width mirror +(`packages/drivers/driver-sql/src/sql-driver.ts#varcharColumnChars`) answers +for the family what the generator answers, so the pin that sweeps the other +string classes can sweep this one — a width the two sides disagree on would be +this fork again, one level down. + +What this does *not* touch: `multiple: true` media fields keep the JSON array +column every multi-value field has (the `multiple` half of `isJsonField`); the +family loses only its own membership. `STRUCTURED_JSON_TYPES` and +`MULTI_OPTION_TYPES` stay JSON columns. The stored **value** contract does not +move — `valueSchemaFor(def, 'stored')` already returns +`FileReferenceIdValueSchema` for the family; this addendum aligns the column +with the contract, not the contract with the column. Everything above the +driver (the validator, the read resolver, the ownership hooks, `objectui`) +sees the same bare id it sees today: the wire shape is unchanged. + +This narrows D1's "DDL column choice remains the driver's decision" for one +class: the driver still owns DDL for every other class, and for this one it +implements a ruled column. It is the last step of D3's "same shape discipline +as `lookup`" — a reference column holds the target's id as a string, and now +the media family's column does too. + +### The switch: per deployment, on the existing flag, never per version + +The key is the deployment-level `sys_migration` row +`packages/spec/src/system/migration.zod.ts#FILE_REFERENCES_MIGRATION_ID` — +`adr-0104-file-references` — and nothing else. It is written by +`os migrate files-to-references --apply` +(`packages/cli/src/commands/migrate/files-to-references.ts#MigrateFilesToReferences` +running +`packages/services/service-storage/src/files-to-references-migration.ts#runFilesToReferencesMigration`) +only when backfill and reconciliation report zero blocking discrepancies — +`sys_migration { id: 'adr-0104-file-references', verified_at, blocking: 0 }` — +or at creation for a datastore born empty +(`packages/platform-objects/src/system/migration-flag.ts#attestFreshDatastore`). +The engine already reads it, memoized, to open strict media value-shape +enforcement and released-file collection +(`packages/objectql/src/engine.ts#isFileReferencesMigrationVerified`). The +column encoding now reads the same row. One flag, not a second gate that can +disagree (the 2026-07-27 principle) — and never a version number: installing a +release changes no deployment's column, exactly as it starts no deletion. + +The same flag is the right key because the fact it attests is precisely the +precondition under which the column move is lossless. A JSON column whose every +cell is a JSON *string* rewrites to a string column with nothing lost; a cell +still holding an inline object cannot be rewritten, and such a cell is exactly +a finding the reconciliation refuses to record the flag over. So the column +move is a further step of `--apply`, after the reconciliation and before the +row is recorded, and it aborts on any blocking finding — the R4 shape: + +``` +os migrate files-to-references --apply + 1. backfill (dry run by default; --apply writes) + 2. verifyFileReferences (reconcile the ledger against what records hold) + 3. zero blocking findings → move this datastore's media columns: + unquote every cell, retype the column to the + string column — per dialect, transactional + where the dialect allows it, aborting on the + first cell that is not a JSON string + 4. record sys_migration { id: 'adr-0104-file-references', verified_at, blocking: 0 } + 5. the driver's encoding, strict enforcement and collection read THAT ROW +``` + +The per-dialect sketch, carried from the measurement and **unrehearsed** (see +the gaps below): SQLite rewrites each cell in place with `json_extract` where +`json_type` is `'text'` (the column keeps TEXT affinity; only the encoding +changes); Postgres retypes the column to a varchar with a `USING` clause that +unquotes the JSON string; MySQL retypes with `MODIFY COLUMN` and unquotes with +`JSON_UNQUOTE` — the order of those two is one of the things the rehearsal +settles. A dry run prints the statements it would execute and writes nothing, +as #3617 already requires of every mode but `--apply`. + +### The window: two encodings, one invariant, one end + +Two encodings will exist across deployments at once, and the driver carries +both arms until the window closes: + +- A deployment **without** the flag keeps today's encoding — a JSON column + (`json` on Postgres and MySQL, TEXT on SQLite) holding the JSON-quoted id + and, on the default warn-first posture, possibly an inline object the + backfill has not yet converted. The driver writes and reads it exactly as + it does today. +- A deployment **with** the flag, after step 3 above, holds the bare id in a + string column; the driver writes the bare id and reads it with no JSON + codec. +- Throughout the window the driver **reads both encodings on every dialect**: + a JSON-quoted string in a media column decodes to the bare id, a bare id is + returned as it is, an inline object still decodes to the object (the + dual-read of §D3, unchanged). SQLite's parse-and-catch read arm already + behaves this way; Postgres and MySQL return a native `json` column parsed + and a varchar verbatim, so the decode there keys on what arrives, not on + the dialect. + +**The invariant the window rests on: on one deployment, the media column's +type and the driver's write encoding never disagree.** A quoted write into a +string column is #15771's silent corruption on Postgres and MySQL; a bare +write into a native `json` column is a loud `22P02` on Postgres. They are one +mistake — the key and the column drifting apart — and the driver card's pins, +per dialect and for both encodings, exist to make that mistake unreachable. +Three populations the invariant must hold over (the implementing card's +checklist; the mechanism is that card's decision and is stated in its PR): + +1. **Un-flagged deployments** upgrading to a driver that carries the change: + nothing moves until they run `--apply`. A media column that additive sync + creates on such a deployment during the window takes today's JSON form, + because that deployment's encoding is still JSON. +2. **Deployments flagged before step 3 existed** — every datastore + creation-attested since 17.0, the dogfood boots among them, and every + `--apply` run before the driver card lands. They hold the flag *and* + JSON-quoted ids in a JSON column. The flag row alone cannot tell such a + deployment from one whose columns have moved, so the driver may not open + the bare-id write arm on the flag alone: step 3 must run on them + (re-running `--apply` is idempotent and re-records the row), and until it + has, the deployment is on the JSON arm whatever the row says. Whether the + step's completion is recorded on the same row or observed from the column + itself is the driver card's to settle — it must be one answer, pinned, and + it must fail toward the JSON arm. +3. **Datastores born after the driver card lands** are attested at + `kernel:ready`, after schema sync has already created their columns. A + store the driver is itself creating from empty carries no legacy encoding, + so nothing stops its media columns taking the ruled form from birth — but + the seam that makes column and encoding agree across the gap between + creation and attestation is the driver card's to choose. The born-strict + dogfood boots are the standing canary for this population, as they are + for R2. + +**Where the window ends.** "Every deployment has moved" is not a fact anyone +can observe — the same reason the evidence gate went per deployment — so the +end is located where the code is and made safe where the data is. The JSON +arm leaves in the **first protocol major after the driver card lands**: +removing a storage encoding the driver reads is a breaking change and rides a +major under ADR-0087, with its disposition recorded. From that major the +driver has one encoding, and a deployment that reaches it without its own +flag-and-column step is **refused loudly at boot for its media fields**, +naming `os migrate files-to-references --apply` — never read through a codec +the driver no longer has, never silently corrupted. This is not the +per-version switch the previous section forbids: the switch of a deployment's +encoding stays that deployment's own `--apply`; the major only deletes the +fallback and makes the missing step loud, and installing it converts nothing +and deletes nothing ("installing a new version is not consent", 2026-07-27). +One major and no longer: a standing two-arm codec is the staged transition +the 2026-08-27 ruling refuses. + +### What does not change + +- **`syncSchema` / `initObjects`** stay additive and never alter an existing + column's type — measured above — so the column move is step 3's alone, + never an upgrade's side effect. During the window, sync creates a new media + column in the form of the arm its deployment is on (the invariant); it + does not consult the flag to retype anything. +- **The generator** changes nothing: `FIELD_TYPE_SQL_MAP`'s `VARCHAR(2048)` + and the TypeScript `table.string` for all five members are the end-state. + Its pin block labelled "Recorded divergence, NOT coverage" stays as written + until the driver card's PR lands and retires it to coverage in the same + change — the divergence is true on every deployment until the driver + moves, and the triage on #15041 asked for that retirement to be deliberate. +- **The evidence table** of the 2026-07-27 amendment gains no row: the media + line's evidence is still the `adr-0104-file-references` flag; the fact it + attests widens to include the column. +- **Consumers above the driver** and the `objectui` wire shape: unchanged, as + the column section says. + +### Confidence gaps — stated, not assumed + +1. **Postgres and MySQL are reasoned, not measured.** Only SQLite was measured + (two in-memory cells, four rows each, the driver's own TEXT column against + a hand-created `VARCHAR(2048)` column, byte-identical reads). The + Postgres/MySQL statements above — identical write-side quoting, no read + parse arm, a varchar returning quoted text verbatim, `22P02` on a bare + write into `json` — are read from the driver source. What closes it: the + driver card's per-dialect pins for both encodings run against the live + services of the `Temporal Conformance (live PG + MySQL)` job + (`postgres:16`, `mysql:8.0`), replacing this paragraph's reasoning with a + measurement before `--apply` gains step 3. +2. **The migration sketch is unrehearsed.** No datastore, copy or fixture has + had the unquote-and-retype step run against it; the per-dialect statement + order is unsettled (MySQL in particular: a `JSON` column refuses a bare + `file_x` as invalid JSON, so the retype must precede the unquote or the two + must be one statement). What closes it: a rehearsal on a copy of a real + datastore per dialect — the showcase dogfood store is the in-repo + candidate — with the dry run printing every statement it would execute and + the apply run proving every media cell reads back equal before and after. + +Until both close, the window section's Postgres/MySQL descriptions are the +ruling's intent, not a measurement, and the driver card carries that caveat +into its PR body. + +### Sequencing + +1. **This addendum** — #15041 is its carrier; governed `docs/adr/**`, draft + PR, human merge. +2. **The driver card, #15989** (`domain:engine`, `pm:blocked` on this + addendum) implements the three sections above: the family leaves + `JSON_COLUMN_TYPES`; `isJsonField` / `formatInput` / `formatOutput` stop + treating it as JSON, keyed on the flag through the window; step 3 in + `--apply`; the `varcharColumnChars` mirror; schema-drift's single-value + JSON-class finding (#15771 folded in or left adjacent — the driver card + decides); pins per dialect for both encodings across the window; a + changeset with its ADR-0087 disposition. +3. **The generator pin block** retires to coverage only in the driver card's + PR. Adjacent and unblocked: #15769 (stale class-doc prose over + `FILE_REFERENCE_TYPES`) and objectui#7699 (the legacy blob still submitted + when no `fileId` surfaced) proceed on their own. + +### Why this stays inside ADR-0104 + +For the reason the 2026-07-27 addenda gave: it settles *how* D3's accepted +"field values point into `sys_file`" reaches the physical column, and amends +one D1 clause for one class rather than revisiting a decision. It is recorded +here so that a grep for `JSON_COLUMN_TYPES`, `VARCHAR(2048)` or "bare id" +lands on the decision (Prime Directive #13), and so that the driver card +implements a ruling rather than a changeset that quietly does the opposite of +D1's clause. From 58f98996a0f5dd471d676b350573af981a851e1f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 18:49:56 +0000 Subject: [PATCH 2/2] docs(adr): ADR-0104 Status bullet indexes the 2026-09-05 addendum MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One sentence in the ADR's running index of addenda, in the spelling the existing entries use: the 2026-09-05 addendum rules the media family's physical column — a string column holding the bare sys_file id, switched per deployment on the adr-0104-file-references flag, never per version; driver card #15989 implements it. Seat ruling on the report's open question 1 (B). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01M59rPZZFzqhfMUPFqqZTkf --- docs/adr/0104-field-runtime-value-shape-contract.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/adr/0104-field-runtime-value-shape-contract.md b/docs/adr/0104-field-runtime-value-shape-contract.md index 39af18e7c8..cb52a624d9 100644 --- a/docs/adr/0104-field-runtime-value-shape-contract.md +++ b/docs/adr/0104-field-runtime-value-shape-contract.md @@ -9,7 +9,10 @@ **errata only** — it registers the outside vocabulary (`fieldRuntimeType`, `FileRef`, `RecordRef`, "read-time expansion") against this ADR's names so a grep for those words lands here, and corrects three membership facts; it - changes no decision. + changes no decision. The 2026-09-05 addendum rules the media family's + physical column — a string column holding the bare `sys_file` id, switched + per deployment on the `adr-0104-file-references` flag, never per version; + driver card #15989 implements it. - **Date**: 2026-07-22 - **Issue**: design follow-up generalizing #3405 / #3406 (inline lookup param silently stripped); relates #3407 (silently dropped writes), #1878 / #1891