diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 349790f..1ffa8ca 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -7,7 +7,7 @@ name: CI # moves through a pull request whose own run tested the merge, so main has no # push trigger. # -# The contract is the root README's conventions: pnpm with a committed +# The contract is the conventions in docs/development.md: pnpm with a committed # lockfile, and `lint`, `typecheck`, `test`, `build` scripts in package.json, # plus `lint:css` (stylelint). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..ba370a2 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,43 @@ +# Contributing + +Issues and pull requests are welcome. The ONE Data team maintains this repository. For a large change, open an issue first so we can agree on the approach before you write it. + +Participation is governed by the [Code of Conduct](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/CODE_OF_CONDUCT.md). Report a security problem through GitHub private vulnerability reporting (the "Report a vulnerability" button under the Security tab), as set out in the [security policy](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/SECURITY.md), and not in a public issue. For other questions see [SUPPORT.md](https://github.com/DevelopmentFinanceObservatory/.github/blob/main/SUPPORT.md). + +## Set up + +You need Node 22.18 or newer, pnpm, [just](https://github.com/casey/just), and Docker for the database tests. + +```bash +just setup # pnpm install and .env.local from .env.example +just doctor # checks node, pnpm and the pnpm version in package.json +just dev # http://localhost:3000 +``` + +The README says which parts of the site need a reachable `apis` or `ai` service. No contribution needs access to ONE's cloud project. + +## Before you open a pull request + +```bash +just db-up # local Postgres 18 on port 5433, needed by the tests +just check # lint, typecheck, test, build +``` + +`just check` runs what the `ci` workflow runs (ESLint, Stylelint, `tsc`, the tests and the build), apart from the migration bundle build. Run `just format` to apply Prettier and Stylelint fixes. + +A pull request needs: + +- a passing `ci` check (draft pull requests run no CI until marked ready) +- an approving review from a code owner +- tests for changed behaviour in the `test` directories under `db/` and `src/lib/` +- updated documentation when you change a command, an environment variable or an endpoint + +If you change `db/schema.ts`, generate the migration with `just db-generate`, review the SQL, and commit it with the schema change. Never edit a migration that has already been applied. Write a new one. `docs/development.md` has the full workflow. + +## Branches + +`dev` is the default branch. Branch from it and open the pull request against it. `main` is the production branch and moves only by a release pull request from `dev`. + +## Maintainer notes + +`docs/development.md` covers code conventions, releases, the deployed services and the deploy configuration. `deploy/README.md` covers the image, migrations on deploy and the environment contract. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..4b7a716 --- /dev/null +++ b/LICENSE @@ -0,0 +1,22 @@ + +MIT License + +Copyright (c) 2026 ONE Campaign + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 2c96018..2da195e 100644 --- a/README.md +++ b/README.md @@ -1,103 +1,54 @@ -# dfo-website - -The public site of the Development Finance Observatory (DFO). A single -Next.js application serves the frontend (chat and canvas workspace, -reference pages) and acts as the backend proxy for **all** API traffic to -the platform, including MCP. This is not the DFO brochure website, the small -live site that gets archived at launch. - -## Status: in progress - -The application code lands in slices as Ode builds it; the homepage, the -data search, the Studio and user accounts are in. -Around it: - -- `deploy/` holds the Dockerfile pattern and the division of responsibility - with `infra`. They define the Cloud Run service, and this repo's CI - only swaps images onto it. -- `db/` holds the schema migration runner: a Cloud Run Job connects to the - website databases as an IAM service account through the Cloud SQL - connector and applies forward-only migrations before each deploy moves - the service to a new image. -- `.github/workflows/` holds the CI and deploy workflows. CI runs `lint`, - `typecheck`, `test`, and `build` on every PR that is not a draft, and on - each push to `dev`. The deploy workflow runs on pushes to `dev` - that touch the image's inputs. -- `docs/development.md` — how to run the app locally and the conventions - the code follows (styles, components, animation system). -- `docs/feedback.md` — the feedback endpoint (`POST /api/feedback`): the - frontend contract, the storage/Slack/email sinks and their config, and - the local dev loop. -- `docs/analytics.md` — usage events (the `event` table and what each - surface records), Sentry, and the Studio prompt allowance. - -## Repo map - -| Repo | What it is | -| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [`dcp`](https://github.com/DevelopmentFinanceObservatory/dcp) | The knowledge graph engine, ONE's fork of Data Commons, serving the graph over the mixer and SDMX APIs. | -| [`apis`](https://github.com/DevelopmentFinanceObservatory/apis) | The Search, Datasets, Graph, and Presentation APIs. The Search, Datasets, and Graph APIs read from the `dcp` graph, and the Presentation API persists to PostgreSQL. | -| [`ai`](https://github.com/DevelopmentFinanceObservatory/ai) | The agent harness and the DFO's MCP servers, deployed as one Cloud Run service. | -| [`spec-kit`](https://github.com/DevelopmentFinanceObservatory/spec-kit) | Shared spec models (spec-id hashing, materialisation, manifest and component-spec schemas) that publish the TS types this app renders from [forthcoming]. | -| [`website`](https://github.com/DevelopmentFinanceObservatory/website) | This repo, the public Next.js site and the proxy that fronts the platform for external traffic. | -| [`skills`](https://github.com/DevelopmentFinanceObservatory/skills) | The open repository of domain-knowledge packages agents load dynamically at runtime [forthcoming]. | -| [`infra`](https://github.com/DevelopmentFinanceObservatory/infra) | Infrastructure as code: the Cloud Run service definitions, service accounts, scaling, and secrets. | - -## Deployed services - -All services run in the `dev-fin-observatory` GCP project, region -`northamerica-northeast1`. - -| Repo | Service | -| --------- | ---------------------------------------------------------------------------------------------------------------- | -| `website` | https://dfo-dev-website-staging-zms5lj4zhq-nn.a.run.app | -| `apis` | https://dfo-dev-apis-zms5lj4zhq-nn.a.run.app/docs (needs an invoker identity; reach it through the gcloud proxy) | -| `ai` | https://dfo-dev-ai-zms5lj4zhq-nn.a.run.app/docs | -| `dcp` | https://dfo-dev-dc-datacommons-service-zms5lj4zhq-nn.a.run.app | - -`website`'s service is the one `dev` deploys to. `apis`, `ai`, and `dcp` -run their own dev services. `apis` and `ai` return 404 at the root and -serve Swagger UI at `/docs`. The website service is reachable by anyone -with the URL. - -## Conventions - -Changing any of these is fine — change the workflows and `deploy/Dockerfile` -in the same PR. - -- **pnpm** as the package manager, with a committed `pnpm-lock.yaml` and a - `packageManager` field in `package.json`. `ci.yml`'s pnpm setup reads its - version from that field and fails without it. -- **TypeScript, strict**, Next.js App Router. -- `package.json` defines `lint`, `typecheck`, `test`, and `build` scripts. - That is the contract `ci.yml` runs. -- `next.config.*` sets `output: 'standalone'`, so `deploy/Dockerfile` can - copy the standalone server into the runtime image. - -## Deploys - -GitHub Actions builds the `dfo-website` image and updates the Cloud Run -dev service automatically on every push to `dev`. `main` is the -production release branch. A push to `main` builds and deploys production -through the `production` GitHub Environment, which needs approval from the -release reviewers. -`infra` owns the Cloud Run service _definition_ (service account, -scaling, secrets, env vars). This repo's CI only swaps images on the -service `infra` already defined. See `deploy/README.md` for the -vars/secrets and prerequisites. +# DFO website + +The Next.js application behind [developmentfinance.org](https://developmentfinance.org): the data search, the Studio (a chat and canvas workspace for building charts and tables), reference pages, and user accounts. It also acts as the proxy that carries all outside traffic, including MCP, to the platform APIs. + +It is part of the [Development Finance Observatory](https://github.com/DevelopmentFinanceObservatory), an initiative of [ONE Data](https://data.one.org) built on [Google Data Commons](https://datacommons.org). + +This repository is under active development. Interfaces, schemas, and data may change without notice. + +## What's inside + +- `src/` holds the application: the App Router pages and API routes in `src/app`, atomic components in `src/components`, and the proxy, search, Studio, auth and feedback code in `src/lib`. +- `db/` holds the Drizzle schema, the forward-only migrations and the runner that applies them. +- `deploy/` holds the Dockerfile and the deploy notes. +- `.github/workflows/` holds CI and the deploy workflows. +- `docs/` holds the maintainer and endpoint documentation. + +The site talks to three neighbouring repositories: [`apis`](https://github.com/DevelopmentFinanceObservatory/apis) for search, datasets and graph queries, [`ai`](https://github.com/DevelopmentFinanceObservatory/ai) for the Studio agent and MCP servers, and [`infra`](https://github.com/DevelopmentFinanceObservatory/infra) for the Cloud Run services that run it. The org profile has the [full repository map](https://github.com/DevelopmentFinanceObservatory). + +## Getting started + +You need Node 22.18 or newer and pnpm. `just doctor` checks both, and `just setup` installs dependencies and creates `.env.local` from `.env.example`. + +```bash +pnpm install +cp .env.example .env.local +pnpm dev +``` + +Open [http://localhost:3000](http://localhost:3000). + +Only the dev apis proxy below needs access to ONE's Google Cloud project (`dev-fin-observatory`). What runs depends on which back ends you can reach: + +- The static pages, `pnpm lint` and `pnpm typecheck` run with no configuration. +- The data search and graph pages call the DFO apis service at `SEARCH_API_BASE`. `.env.example` points it at `http://localhost:8083`, where ONE maintainers run a `gcloud run services proxy` to the dev deployment; that proxy needs an account with invoker access in `dev-fin-observatory`, as `docs/development.md` describes. Without that access, point it at your own `apis` instance. +- The Studio needs an `ai` harness at `STUDIO_API_BASE` (default `http://localhost:8080`) and a `STUDIO_TOKEN`, which ONE issues for its own deployments. Run the `ai` repository locally to use it. +- Accounts and saved conversations need local Postgres (`just db-up && just db-migrate`, which requires Docker) and your own OAuth client credentials for at least one sign-in provider. +- `pnpm test` needs the local Postgres but nothing from the cloud. + +Without `DATABASE_URL`, the dev server logs a warning and keeps serving the pages that do not use the database. `docs/development.md` lists every variable and the full command set. + +## Documentation + +- [`docs/development.md`](docs/development.md): running locally, the database, code conventions, branches, releases and the deployed services. +- [`docs/feedback.md`](docs/feedback.md): the feedback endpoint (`POST /api/feedback`), its sinks and the local development loop. +- [`docs/analytics.md`](docs/analytics.md): usage events, Sentry and the Studio prompt allowance. +- [`deploy/README.md`](deploy/README.md): the image build, migrations on deploy, GitHub variables and the runtime environment contract. ## Contributing -`dev` is the default branch. Branch off it and open a pull request -into it. The `ci` check has to pass before the PR merges, and force-pushes -are blocked. Each merge deploys to the dev service. +Issues and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md). -`main` is production, the same model `apis` uses. To release: +## Licence -1. Confirm the `dev` commit you want is deployed and validated on the - dev service. -1. Open a pull request from `dev` into `main`. `main` requires one approval - and the `ci` check, and blocks force-pushes and deletion. -1. Merge it. `deploy-prod.yml` builds and deploys that commit once the - release reviewers approve the `production` environment. -1. Tag it: `git tag prod-YYYY-MM-DD && git push origin prod-YYYY-MM-DD`. +MIT. See [LICENSE](LICENSE). diff --git a/deploy/Dockerfile b/deploy/Dockerfile index b481b4b..1b192b4 100644 --- a/deploy/Dockerfile +++ b/deploy/Dockerfile @@ -1,8 +1,8 @@ # syntax=docker/dockerfile:1 # # Cloud Run image for the DFO website: the Next.js app in standalone output -# mode (next.config.ts sets `output: 'standalone'`, per the root README's -# conventions). +# mode (next.config.ts sets `output: 'standalone'`, per the conventions in +# docs/development.md). # # Build from the REPO ROOT, not from deploy/: # diff --git a/deploy/README.md b/deploy/README.md index 754aead..259d84c 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -31,8 +31,7 @@ pushed image's digest rather than the tag. Every deploy also runs a Cloud Run Job that applies the website database's schema migrations before the service moves to the new image: `build-migrate-deploy.yml` points the Job at the image it just pushed, -executes it with `--wait`, and only updates the service once the Job exits -0. A failed migration fails the deploy and leaves the service on its current +executes it with `--wait`, and only updates the service once the Job exits 0. A failed migration fails the deploy and leaves the service on its current image. infra defines each environment's Job (named by `MIGRATE_JOB_WEBSITE`, `dfo-dev-website-staging-migrate` on dev) and the migrator identity it runs as. This repo owns the migrate entrypoint and the @@ -75,7 +74,7 @@ migration. `build-migrate-deploy.yml` reads these from a GitHub Environment matching the deploy target (`dev`, `production`). Configure the same variable and secret -*names* in each environment with per-environment values. The workflow YAML +_names_ in each environment with per-environment values. The workflow YAML doesn't change between environments, only the environment it targets does. These are the same names `ai` uses, except the service and migration Job vars, so the `infra` foundation outputs slot into both repos the same way. @@ -102,10 +101,10 @@ For dev, infra has set up: Workload Identity Federation pool and provider that already serves `ai` (foundation), with the per-repo restriction at the SA binding. - That deployer service account, with: - - `roles/run.developer` (update Cloud Run services) - - `roles/artifactregistry.writer` to push images - - `roles/iam.serviceAccountUser` (act as the Cloud Run runtime SA, if - that's a separate identity) + - `roles/run.developer` (update Cloud Run services) + - `roles/artifactregistry.writer` to push images + - `roles/iam.serviceAccountUser` (act as the Cloud Run runtime SA, if + that's a separate identity) - The Cloud Run service for the `dfo-website` image, including its runtime service account, scaling config, env vars, and secrets. This repo's workflows only ever update the `--image` on the service that already @@ -124,26 +123,26 @@ service, and migration Job with its migrator identity. The app defines the definitive contract once it exists. Known so far: -| Name (provisional) | What it is | -| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `STUDIO_API_BASE` | The `dfo-ai` harness the `/api/studio/*` proxy forwards to; plain env var set by infra | -| `STUDIO_TOKEN` | Bearer token the studio proxy sends to the harness; secret reference set by infra | -| `SEARCH_API_BASE` | The `dfo-apis` Search, Datasets and graph APIs; required, a production server without it refuses to start; plain env var set by infra | -| `MCP_API_TOKEN` | Bearer token the `/mcp` proxy sends to the harness; infra mounts it from Secret Manager (`dfo-dev-mcp-api-token`, the harness's own secret) | -| `CALLER_HMAC_KEY` | HMAC key for the client-IP digest `/mcp` sends as `x-dfo-caller`; unset, none is sent; secret set by infra | -| `RATE_LIMIT_WINDOW_S` | Rate-limit window in seconds; default 60 | -| `RATE_LIMIT_API` | Calls per window per address on `/api`; default 120 | -| `RATE_LIMIT_PIN` | Calls per window per address on `/api/download-pin`; default 10 | -| `RATE_LIMIT_MCP` | Requests per window per address on `/mcp`; default 300, above `/api` because a vendor's address can carry many users | -| `OPENAI_APPS_CHALLENGE` | OpenAI's domain-verification token; `/.well-known/openai-apps-challenge` serves it as plain text, 404 while unset | +| Name (provisional) | What it is | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `STUDIO_API_BASE` | The `dfo-ai` harness the `/api/studio/*` proxy forwards to; plain env var set by infra | +| `STUDIO_TOKEN` | Bearer token the studio proxy sends to the harness; secret reference set by infra | +| `SEARCH_API_BASE` | The `dfo-apis` Search, Datasets and graph APIs; required, a production server without it refuses to start; plain env var set by infra | +| `MCP_API_TOKEN` | Bearer token the `/mcp` proxy sends to the harness; infra mounts it from Secret Manager (`dfo-dev-mcp-api-token`, the harness's own secret) | +| `CALLER_HMAC_KEY` | HMAC key for the client-IP digest `/mcp` sends as `x-dfo-caller`; unset, none is sent; secret set by infra | +| `RATE_LIMIT_WINDOW_S` | Rate-limit window in seconds; default 60 | +| `RATE_LIMIT_API` | Calls per window per address on `/api`; default 120 | +| `RATE_LIMIT_PIN` | Calls per window per address on `/api/download-pin`; default 10 | +| `RATE_LIMIT_MCP` | Requests per window per address on `/mcp`; default 300, above `/api` because a vendor's address can carry many users | +| `OPENAI_APPS_CHALLENGE` | OpenAI's domain-verification token; `/.well-known/openai-apps-challenge` serves it as plain text, 404 while unset | | `TRUSTED_PROXY_HOPS` | Which `X-Forwarded-For` entry is the client, counted from the right; keys the `/api` and `/mcp` limits, the guest Studio allowance and the caller digest; default 0 (Cloud Run's); +1 per load balancer | -| `WEBSITE_DB_INSTANCE_CONNECTION_NAME` | Cloud SQL instance the connector dials over PSC; set by infra | -| `WEBSITE_DB_USER` | Runtime IAM database user; set by infra | -| `WEBSITE_DB_NAME` | Database name; set by infra | -| `BETTER_AUTH_SECRET` | Signs auth sessions; generated and stored in Secret Manager by infra | -| `BETTER_AUTH_URL` | This app's own public origin; plain env var set by infra | -| `GOOGLE_CLIENT_ID` | Google sign-in OAuth client ID; optional, from Secret Manager | -| `GOOGLE_CLIENT_SECRET` | Google sign-in OAuth client secret; optional, from Secret Manager | +| `WEBSITE_DB_INSTANCE_CONNECTION_NAME` | Cloud SQL instance the connector dials over PSC; set by infra | +| `WEBSITE_DB_USER` | Runtime IAM database user; set by infra | +| `WEBSITE_DB_NAME` | Database name; set by infra | +| `BETTER_AUTH_SECRET` | Signs auth sessions; generated and stored in Secret Manager by infra | +| `BETTER_AUTH_URL` | This app's own public origin; plain env var set by infra | +| `GOOGLE_CLIENT_ID` | Google sign-in OAuth client ID; optional, from Secret Manager | +| `GOOGLE_CLIENT_SECRET` | Google sign-in OAuth client secret; optional, from Secret Manager | The public MCP URL is `/mcp`. Next redirects `/mcp/` to `/mcp` with a 308, which keeps the method and body. The proxy forwards either form to the @@ -162,7 +161,7 @@ every client under one shared address, each with one error logged. ## Repo-side prerequisites -- The application itself, following the conventions in the root README +- The application itself, following the conventions in `docs/development.md` (pnpm, `output: 'standalone'`, the `lint`/`typecheck`/`test`/`build` script contract). - The migrate entrypoint and its migrations, both under `db/`. The diff --git a/docs/development.md b/docs/development.md index afcafbf..4ab817c 100644 --- a/docs/development.md +++ b/docs/development.md @@ -15,9 +15,11 @@ pnpm dev Open [http://localhost:3000](http://localhost:3000). -Copy `.env.example` to `.env.local`. The homepage and the data search run -without anything set; the Studio needs `STUDIO_TOKEN`, and accounts need -`DATABASE_URL` and `BETTER_AUTH_SECRET`. To open the dev server from a +Copy `.env.example` to `.env.local`. The static pages run without it. The +data search and graph pages need `SEARCH_API_BASE`, which the sample points +at the proxy in [Reach the dev APIs](#reach-the-dev-apis); the Studio needs a harness at +`STUDIO_API_BASE` and a `STUDIO_TOKEN`; accounts need `DATABASE_URL` and +`BETTER_AUTH_SECRET`. To open the dev server from a phone/tablet on the same network, put the machine's LAN IP in `DEV_ORIGINS`. @@ -50,8 +52,8 @@ The scripts below reach the APIs the same way. ## just -The [justfile](../justfile) is the entry point for the commands below — it -wraps the pnpm scripts (see the table) plus the container and deploy +The [justfile](../justfile) is the entry point for the commands below. It +wraps the pnpm scripts (listed under Scripts) plus the container and deploy workflows. `just` with no arguments lists everything. Install it with `brew install just` if you don't have it. @@ -71,8 +73,8 @@ one-line descriptions. ## Database Accounts and saved conversations live in Postgres, migrated by -`db/apply-migrations.ts` — `db/migrate.ts` runs it against Cloud SQL in -deployed environments, `db/migrate-local.ts` runs it locally. +`db/apply-migrations.ts`. `db/migrate.ts` runs it against Cloud SQL in +deployed environments, and `db/migrate-local.ts` runs it locally. First-time setup: @@ -85,9 +87,9 @@ just db-migrate # create the migrator/runtime roles and apply migrations points at the `website_runtime` role `db-migrate` creates, so nothing else to configure. Until `just db-up` and `just db-migrate` have run, database-backed requests (sign-in, saved conversations) fail with a -connection error; `pnpm dev`'s "logs a warning and keeps serving" fallback -only applies when neither `DATABASE_URL` nor `WEBSITE_DB_*` is set at all. -A production start fails outright instead of serving with no database. +connection error. The `pnpm dev` fallback that logs a warning and keeps +serving applies only when neither `DATABASE_URL` nor `WEBSITE_DB_*` is set. +A production start with no database fails outright. The container uses port 5433 so it can run next to a Postgres installed on the machine. If 5433 is taken too, set `PGPORT` to another port for @@ -99,23 +101,22 @@ off 5432, needs `just db-reset`. Schema change workflow: 1. Edit `db/schema.ts`. -2. `just db-generate` — generates a migration under `db/migrations` from +2. `just db-generate` generates a migration under `db/migrations` from the diff. 3. Review the generated SQL. -4. `just db-migrate` — applies it to the local database. +4. `just db-migrate` applies it to the local database. 5. Commit the migration alongside the schema change. Never run `drizzle-kit push`, `drizzle-kit migrate`, or `better-auth -migrate` directly — migrations only ever apply through +migrate` directly. Migrations apply only through `db/apply-migrations.ts` (via `db-migrate`), which renders the migrator/runtime role placeholders in migration `0000` and takes the -advisory lock that keeps concurrent runs from double-applying; the other -tools know about neither. A new migration is always a new file — -never edit one that has already applied, even locally, and write a -follow-up migration instead. +advisory lock that keeps concurrent runs from double-applying. The other +tools do neither. Never edit a migration that has already applied, even +locally. Write a follow-up migration. -This reaches the dev environment the same way. Every dev deploy runs a migration -Job against the same image before the service rolls out to it. +In the dev environment, every deploy runs a migration Job against the same +image before the service rolls out. ### Card reference backfill @@ -156,6 +157,32 @@ puts the wrong-match rate below 1% at 95% confidence. A write changes one row per transaction and only the matched cards in it. Running it again writes nothing. +## Conventions the pipeline relies on + +Changing any of these is fine if the workflows and `deploy/Dockerfile` change in the same pull request. + +- pnpm is the package manager, with a committed `pnpm-lock.yaml` and a `packageManager` field in `package.json`. The pnpm setup in `ci.yml` reads its version from that field and fails without it. +- TypeScript runs in strict mode on the Next.js App Router. +- `package.json` defines `lint`, `typecheck`, `test` and `build` scripts, which `ci.yml` runs. +- `next.config.ts` sets `output: 'standalone'`, so `deploy/Dockerfile` can copy the standalone server into the runtime image. + +## Branches and releases + +`dev` is the default branch. Contributors branch from it and open a pull request into it. The `ci` check has to pass before merge, and force-pushes are blocked. Each merge deploys to the dev service. + +`main` is production. To release: + +1. Confirm the `dev` commit you want is deployed and validated on the dev service. +2. Open a pull request from `dev` into `main`. `main` requires one approval and the `ci` check, and blocks force-pushes and deletion. +3. Merge it. `deploy-prod.yml` builds and deploys that commit once the release reviewers approve the `production` GitHub Environment. +4. Tag it: `git tag prod-YYYY-MM-DD && git push origin prod-YYYY-MM-DD`. + +## Deployed services + +The services run in the `dev-fin-observatory` GCP project, region `northamerica-northeast1`. GitHub Actions builds the `dfo-website` image and updates the Cloud Run dev service on every push to `dev` that touches the image's inputs. `infra` owns the Cloud Run service definition (service account, scaling, secrets, environment variables); this repository's CI only swaps images onto the service `infra` defined. `just deployed` prints the live service's URL, revision and image, and `just logs` tails its logs. + +The `apis`, `ai` and `dcp` repositories run their own dev services. `apis` and `ai` return 404 at the root and serve Swagger UI at `/docs`. `deploy/README.md` lists the GitHub variables, secrets and GCP prerequisites. + ## Scripts | Script | What it does | @@ -180,14 +207,14 @@ through the proxy in [Reach the dev APIs](#reach-the-dev-apis). ## Conventions -- **Styles** — global partials live in `src/styles` (`_variables`, `_mixins`, `_breakpoints`, …). They are on the Sass load path, so import them from anywhere as `@use "variables" as *;` (no relative paths). Fluid rem: `html { font-size: vw(16px) }` scales the whole layout with the viewport (separate bases for desktop / tablet / phone, see `$widths`); size things with `rem()` / `@include font-size()` — **always `rem(x)` for design px values, never raw px**. Breakpoints: `@include breakpoint(phone)` (< 660px) and `@include breakpoint(tablet)` (660–1023px). -- **Colors & fonts — always `var(--color-*)` / `var(--font-*)`**, never `$color-*`/`$font-*` directly in components. The custom properties are emitted once in `src/styles/_root.scss` (from the SCSS palette in `_variables.scss` — add new colors in both places). CSS vars are used so sections can re-theme parts of the UI (e.g. the header changes theme depending on what's underneath it) by overriding them locally. SCSS variables remain only for things CSS vars can't do: breakpoints, sizes, `rem()` math. -- **Fonts** — Outfit (300–800) and DM Mono (300–500), self-hosted via `@fontsource`, imported in `src/app/layout.tsx`. -- **Icons** — put the SVG in `src/assets/svg/`, register it in `src/components/atoms/Icon/Icon.tsx` (`icons` map), use as ``. SVGs are compiled to React components by SVGR (Turbopack rule in `next.config.ts`); use `stroke="currentColor"` so icons inherit text color. Static image imports are disabled (`images.disableStaticImages`) — reference bitmaps from `public/` by URL instead. -- **Components** — atomic design: `src/components/atoms|molecules|organisms/Name/Name.tsx` + `Name.module.scss`. -- **Buttons** — always use the `Button` atom (`src/components/atoms/Button`); add new variants there instead of styling buttons ad hoc. When a button has an icon, the icon renders in its own wrapper segment 1px next to the label (icon itself 20×20) — pass `icon="name"`; icon-only buttons get `aria-label`. Hover plays a one-shot swap (arrow flies out in its own direction, a copy rides in; label words roll up with a stagger) driven by the `.swapping` class from JS — timings live only in `Button.tsx` (`SWAP_MS`, `WORD_STAGGER_MS`, `HOVER_INTENT_MS`) and reach the SCSS via `--swap-duration`/`--word-stagger`. To make a whole card trigger its button's hover, put `data-button-zone` on the card — the button listens there instead of on itself. Hover styles are gated behind `@media (hover: hover)` so touch screens don't get stuck hover states. -- **Animations** — one system, don't hand-roll: - - _Reveals_ (`src/styles/_reveal.scss` + `useReveal` hook): attach `useReveal()` once per section (returns the section ref), mark elements with `data-reveal` (rise+fade+subtle scale), `data-reveal="text"` (no scale — always use for text), `data-reveal-group` (card-style unit: container + inner cascade) or `data-reveal-group="trigger"` (shared trigger, container static). Stagger via `style={revealIndex(n)}`; global pace in `$reveal-stagger`. Elements trigger individually, reset when leaving the viewport, and animate out quickly. The observer toggles the `data-revealed` **attribute** (never a class — React rewrites `className` wholesale on re-render and would wipe it). - - _Text fill on scroll_: `SectionHeading` (label + h2, one continuous progression, per-section colors via `--section-label-color`/`--section-heading-color`) or the `TextFill` atom for any other text (`--text-fill-color`). Both are driven by `useFillProgress`, which lerps the displayed progress toward the scroll target so the fill trails the scroll smoothly; the scrub window lives in `utils/scroll.ts#travelProgress`. - - _Parallax / scroll work_: `useScrollFrame(callback)` (rAF-throttled scroll+resize; pass `null` to disable), progress formulas in `utils/scroll.ts` (`centeredProgress`, `travelProgress`, `enterProgress`), `usePrefersReducedMotion()` for gating, `useInView(ref)` for visibility flags. The header hides on scroll down and returns on scroll up (direction tracking in `Header.tsx`, same scroll frame as its theme probe). -- **Safe areas** — the viewport is `viewport-fit: cover`; the page paints under the notch / home-indicator bands and the fixed pieces (header, footer, Studio, drawers) pad by `env(safe-area-inset-*)` themselves. +- Styles: global partials live in `src/styles` (`_variables`, `_mixins`, `_breakpoints`, …). They are on the Sass load path, so import them from anywhere as `@use "variables" as *;` with no relative paths. Fluid rem sizing, `html { font-size: vw(16px) }`, scales the whole layout with the viewport (separate bases for desktop, tablet and phone, see `$widths`). Size things with `rem()` and `@include font-size()`, and write every design px value as `rem(x)`. Breakpoints are `@include breakpoint(phone)` (< 660px) and `@include breakpoint(tablet)` (660–1023px). +- Colors and fonts: components use `var(--color-*)` and `var(--font-*)`, and do not use `$color-*` or `$font-*` directly. The custom properties are emitted once in `src/styles/_root.scss` from the SCSS palette in `_variables.scss`, so add new colors in both places. CSS vars let sections re-theme parts of the UI by overriding them locally (for example, the header changes theme depending on what is underneath it). SCSS variables are for what CSS vars cannot do: breakpoints, sizes, and `rem()` math. +- Fonts: Outfit (300–800) and DM Mono (300–500), self-hosted via `@fontsource` and imported in `src/app/layout.tsx`. +- Icons: put the SVG in `src/assets/svg/`, register it in the `icons` map in `src/components/atoms/Icon/Icon.tsx`, and use it as ``. SVGR compiles SVGs to React components (Turbopack rule in `next.config.ts`), so use `stroke="currentColor"` for icons to inherit text color. Static image imports are disabled (`images.disableStaticImages`), so reference bitmaps from `public/` by URL. +- Components: atomic design, `src/components/atoms|molecules|organisms/Name/Name.tsx` + `Name.module.scss`. +- Buttons: use the `Button` atom (`src/components/atoms/Button`) for every button, and add new variants there. When a button has an icon, the icon renders in its own wrapper segment 1px next to the label (icon itself 20×20). Pass `icon="name"`, and give icon-only buttons an `aria-label`. Hover plays a one-shot swap (the arrow flies out in its own direction, a copy rides in, and the label words roll up with a stagger) driven by the `.swapping` class from JS. Timings live only in `Button.tsx` (`SWAP_MS`, `WORD_STAGGER_MS`, `HOVER_INTENT_MS`) and reach the SCSS via `--swap-duration` and `--word-stagger`. To make a whole card trigger its button's hover, put `data-button-zone` on the card, and the button listens there. Hover styles are gated behind `@media (hover: hover)` so touch screens do not get stuck hover states. +- Animations use one system: + - _Reveals_ (`src/styles/_reveal.scss` + `useReveal` hook): attach `useReveal()` once per section (it returns the section ref), and mark elements with `data-reveal` (rise, fade and subtle scale), `data-reveal="text"` (no scale, for all text), `data-reveal-group` (card-style unit: container + inner cascade) or `data-reveal-group="trigger"` (shared trigger, container static). Stagger via `style={revealIndex(n)}`, with the global pace in `$reveal-stagger`. Elements trigger individually, reset when leaving the viewport, and animate out quickly. The observer toggles the `data-revealed` attribute, because React rewrites `className` wholesale on re-render and would wipe a class. + - _Text fill on scroll_: `SectionHeading` (label + h2, one continuous progression, per-section colors via `--section-label-color` and `--section-heading-color`) or the `TextFill` atom for any other text (`--text-fill-color`). Both are driven by `useFillProgress`, which lerps the displayed progress toward the scroll target so the fill trails the scroll smoothly. The scrub window lives in `utils/scroll.ts#travelProgress`. + - _Parallax and scroll work_: `useScrollFrame(callback)` (rAF-throttled scroll and resize, disabled by passing `null`), progress formulas in `utils/scroll.ts` (`centeredProgress`, `travelProgress`, `enterProgress`), `usePrefersReducedMotion()` for gating, and `useInView(ref)` for visibility flags. The header hides on scroll down and returns on scroll up (direction tracking in `Header.tsx`, same scroll frame as its theme probe). +- Safe areas: the viewport is `viewport-fit: cover`. The page paints under the notch and home-indicator bands, and the fixed pieces (header, footer, Studio, drawers) pad by `env(safe-area-inset-*)` themselves.