Skip to content

feat(cli): migrate the database before replacing app-api - #719

Merged
Medformatik merged 1 commit into
mainfrom
feat/migrate-before-api-update
Oct 31, 2026
Merged

Medformatik merged 1 commit into
mainfrom
feat/migrate-before-api-update

Conversation

@Medformatik

@Medformatik Medformatik commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on #715 (merged), whose migration runner the one-off container uses.

What changed and why

app-api applied its migrations at startup, before it answered its healthcheck. A long migration therefore looked like an unhealthy app-api:

  • Compose failed the update.
  • The admin-panel updater reinstalled the previous images, which stopped the migration. Such a migration could never be applied from there.
  • The new data-manager, replaced in the same step, could start against the schema before app-api had migrated it.

When services update replaces app-api, it now runs these steps after the pull and before up:

  1. It stops any migration an earlier update left running.
  2. It stops the old app-api, so old code never runs against the migrated schema.
  3. It runs the new image's dist/migrate.js in a one-off app-api container, using the runner from fix(api): make startup migrations visible, serialized and stoppable #715 (lock, progress lines, cancellation). Its output is the update's output.
  4. It starts the new containers against the migrated schema.

If the migration fails, or the update is stopped, the migration rolls back as a whole. The previous app-api container then starts again and nothing is replaced. In the admin-panel updater, the agent's rollback then runs as before.

Things that took care:

  • compose run containers outlive the Compose client. I checked this with Compose 5.6: after SIGTERM to the client, the container kept running. The agent also SIGKILLs the CLI 2 s after asking it to stop. So:
    • each step first stops this stack's leftover migration containers, found by Compose's project label plus an org.openmapx.migration=app-api marker;
    • a stop request starts a detached process that stops the migration and runs compose start app-api;
    • this matters when the admin update hits its 30-minute limit, because the agent's own rollback is then killed straight away and only finishes on the agent's next start.
  • Rollback to an image from before this change. An image without dist/migrate.js makes the container exit with status 78. The step then leaves migrating to app-api's startup, as earlier releases did.
  • app-api's healthcheck startPeriod goes back to 30s. That reverts fix(api): make startup migrations visible, serialized and stoppable #715's stopgap: on the update path app-api no longer migrates for long when it starts, so a hung app-api is detected promptly again.

The upgrade guide and CLI reference describe the step. They also say that an admin-panel update stops after 30 minutes, and that a release needing a longer migration should be applied from the host with pnpm openmapx services update, which has no limit.

Verification

  • The real app-api image (built from this branch with apps/api/Dockerfile), run with the step's exact compose run command against a PostGIS at 0038 holding 34,207 POI-ingest jobs and 136,828 stages:
    • Stop: it logged Applying 6 database migrations, then a progress line with 0039's DELETE. Running the stopMigrationArgs script stopped it in about 1 s: exit 143, container removed, applied count still 39, all jobs intact, no migration session left.
    • Success: it applied all 6 migrations in 465 s, logging progress every minute, and exited 0. The result was 45 applied migrations, the jobs deleted, job_stages_job_id_idx present, and the run container removed.
    • Image without the entry point: exit 78.
    • The leftover filter (Compose project label plus marker) found the running migration container, and compose config reported the same project name.
  • I didn't run the whole services update command against a full OpenMapX stack. Its orchestration is covered by unit tests:
    • api-migrations.test.ts covers the order of operations, leftovers, failure → start app-api, the skip status, an unreadable project name, and signal listeners only while the container runs. It also runs stopMigrationArgs through sh with a stub docker, with a Compose path that contains a space.
    • cli-services-update.test.ts checks that migration comes after the pull and before up, that a failure recreates nothing, and that there's no step without app-api.
  • npx vitest run --project node: 18,783 tests pass. Type checks for api and cli, and pnpm lint, pass. The api build emits dist/server.js and dist/migrate.js.
  • Not run: an admin-panel update end to end, and the ingestion and export-restore database groups.

Release and upgrade impact

Product impact: ordinary

No operator action. Both update paths run the step from the first update to this release: the admin-panel updater replaces the operations agent, which carries the CLI, before the other services, and the host procedure runs the CLI from the updated checkout.

@Medformatik Medformatik self-assigned this Oct 31, 2026
Comment thread packages/cli/src/lib/api-migrations.ts Dismissed
Comment thread packages/cli/src/lib/api-migrations.ts Dismissed
Base automatically changed from fix/api-startup-migration-hardening to main October 31, 2026 18:11
@Medformatik
Medformatik force-pushed the feat/migrate-before-api-update branch from 7ecd169 to 3dbf0fc Compare October 31, 2026 18:16
Medformatik added a commit that referenced this pull request Oct 31, 2026
)

## What changed and why

`test (web)` failed on #719, which touches no web code. All 5,664 tests
passed, but Vitest reported an unhandled error from
`apps/web/src/hooks/usePinMarker.test.tsx`:

```text
TypeError: map.getContainer(...).getBoundingClientRect is not a function
 ❯ fitPinLabel apps/web/src/hooks/usePinMarker.ts:34:38
```

The cause is the test "reattaches the pin after the map is replaced by a
retry":

- It swaps in a second fake map whose container has no
`getBoundingClientRect`. Only the first map's container is given one at
the top of the file.
- Because the label is visible, the hook fits it against the new map's
container on the next animation frame.
- The test usually unmounted first, which cancels that frame. On a slow
runner the frame ran and threw outside the test.

The hook itself does cancel its pending frame on cleanup, so this is a
test-only race.

The replacement container now gets the same size as the first. The test
also waits for one animation frame before unmounting, as the other label
tests in the file do, so it always runs the fit on the replacement map
instead of racing it.

## Verification

- With a forced wait before the unmount, the unmodified test reproduces
the CI error every time.
- With the fix, three runs of the file pass with no errors. With the
frame wait but without the container size, the error comes back.
- `pnpm exec vitest run --project web`: 601 files, 5,667 tests, no
errors. Biome passes.

## Release and upgrade impact

Product impact: ordinary
app-api applied its migrations at startup, before it answered its
healthcheck. A long migration therefore turned into an unhealthy
app-api: Compose failed the update, and the admin-panel updater
reinstalled the previous images, which stopped the migration, so such a
migration could never be applied from there. The new data-manager could
also start against the schema before app-api had migrated it.

When `services update` replaces app-api, it now stops the old app-api,
then applies the new image's migrations in a one-off app-api container
before it starts any new container. The migration's output is the
update's output, and app-api starts against a migrated schema. If the
migration fails, or the update is stopped, the migration rolls back
and the previous app-api starts again.

The one-off container runs the new `dist/migrate.js` entry point.
`docker compose run` containers outlive a killed Compose client, so
each step first stops any migration an earlier one left running, and
a stop request starts a detached process that stops the migration and
restarts the previous app-api. Images without the entry point, such as
a rollback to an earlier release, skip the step and migrate at startup
as before.

app-api's healthcheck start period returns to 30 seconds: on the
update path it no longer migrates for long when it starts.
@Medformatik
Medformatik force-pushed the feat/migrate-before-api-update branch from 3dbf0fc to 9ef9aa5 Compare October 31, 2026 18:59
@Medformatik
Medformatik enabled auto-merge (squash) October 31, 2026 19:04
@Medformatik
Medformatik disabled auto-merge October 31, 2026 19:20
@Medformatik
Medformatik enabled auto-merge (squash) October 31, 2026 19:23
@Medformatik
Medformatik merged commit 1f64c88 into main Oct 31, 2026
27 checks passed
@Medformatik
Medformatik deleted the feat/migrate-before-api-update branch October 31, 2026 20:20
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 31, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants