Skip to content

docs(create-app): issue-resolution spec for local Docker start failure and the resulting permanent UVE 403 (#37262) - #37263

Open
fmontes wants to merge 7 commits into
mainfrom
issue-37262-create-app-docker-uve-spec
Open

docs(create-app): issue-resolution spec for local Docker start failure and the resulting permanent UVE 403 (#37262)#37263
fmontes wants to merge 7 commits into
mainfrom
issue-37262-create-app-docker-uve-spec

Conversation

@fmontes

@fmontes fmontes commented Aug 28, 2026

Copy link
Copy Markdown
Member

Proposed Changes

Spec-only PR (PR 1 of 2) for #37262, per the repo's spec-kit flow. No code changes — this PR is the approval gate for the approach. Implementation lands in the stacked PR on top of this branch.

  • Adds specs/37262-create-app-docker-uve/spec.md — an issue-resolution spec for the @dotcms/create-app local-Docker failure
  • Points .specify/feature.json at the new feature

What's broken (three compounding defects):

  1. dotCMS never starts. single-node-demo-site/docker-compose.yml lets dotcms boot before Postgres accepts connections and gives it no restart policy, so it exits and stays exited. The CLI reports "containers started successfully" and burns its retry budget on a dead container. Users get past this only by pressing ▶ in Docker Desktop.
  2. A non-essential config call aborts the run. Token and default site both resolve fine; the UVE app-config POST then returns a transient 403 and the CLI calls process.exit(1). Because UVE setup runs before scaffolding, the user is left with an empty directory and the working token and site ID are discarded without ever being printed.
  3. Recovery is blocked by the CLI's own side effects. The port pre-check hard-fails on the now-running dotCMS's ports, and the directory-clearing prompt would delete the docker-compose.yml needed to tear it down.

Verified against the tree before writing — every claim in the issue holds:

  • compose: dotcms has depends_on with no condition, no restart:, no healthcheck, and does not publish 8090; opensearch has no healthcheck; db's healthcheck exists but nothing consumes it
  • src/index.ts:370 exits before the scaffolding at :377
  • src/index.ts:597 tests if (!result) against a truthy { ok: false, val }, making the npm-install failure branch unreachable
  • checkPortsAvailability() (src/utils/index.ts:479) hard-fails on exactly the ports a successful run holds
  • the package contains no spec files at all — this fix establishes the test harness

Deliberately deferred as explicit non-goals, so the P0 fix stays bounded:

  • user.isAdmin() swallowing exceptions into a false "not admin" — real defect, but legacy Liferay code on a hot permission path with wide blast radius, and the P0 fix doesn't depend on it
  • Pinning the image tag alongside CUSTOM_STARTER_URL — intersects binding ADR-0019 (date-lockstep SDK versioning), so it deserves its own decision rather than riding along in a bug fix

Reviewers — and two questions for you

.github/CODEOWNERS does not cover docker/, so the highest-blast-radius file in this fix gets no automatic reviewer. Reviewers below are derived from git blame on the exact hunks the implementation will change, weighting design ownership over line count (the two largest raw counts are a bulk restore, #27432, and a pgvector bump, #29915 — both mechanical).

Reviewer Why
@wezell OpenSearch 1.x + SSL in this file (#27754) and the Postgres 18 upgrade (#34236); owns the db/opensearch shape we're adding healthchecks to. Most senior still-active owner of docker/.
@erickgonzalez Most recent semantic change to the dotcms service (#36490); closest to the CUSTOM_STARTER_URL / starter-import behavior the readiness race depends on
@jcastro-dotcms Not from blame on this file — added because blame's strongest signal is unavailable (below). Second-most-active contributor to docker/ over the last 12 months.

Blame's two strongest signals can't be assigned. spbolton holds dominant blame on every hunk in scope and authored lgtm-observability (#32980) — the exact condition: service_healthy pattern this change copies — and dcolina holds part of the opensearch block (#29915). Both are no longer collaborators on this repo (last commits 2026-03-30 and 2026-04-07), so GitHub rejects the review request. Flagging it rather than quietly dropping them: the person who designed the pattern we're copying is not available to check that we're copying it faithfully. If anyone has context on the lgtm-observability healthcheck choices, that's the gap to fill.

⚠️ Updated — the diagnosis changed

Planning tested this spec's own root cause 2 and disproved it. The 403 is not a transient
startup race. It is permanent damage caused by root cause 1, and the two "independent bugs"
are one causal chain. Details in Root-Cause Hypothesis; summary below.

Experiment 1 — a clean boot has no settling window. Measured on an M5/64GB host with dotCMS constrained to 2 CPUs / 4G:

Signal First success
POST api-token 46s
GET apps/dotema-config-v2/{site} 46s
/dotmgt/livez, /dotmgt/readyz 48s
/api/v1/appconfiguration (current CLI probe) 49s

The UVE endpoint is usable two seconds before readyz goes green. The starter import (T+20s) and ES reindex (T+44s) both complete inside Tomcat startup, and the connector accepts no traffic until after them. The hypothesised race cannot happen.

Experiment 2 — the reporter's actual path reproduces it, permanently. Killing dotCMS mid starter-import and hand-starting it (reproduction step 4):

T+39s  appconfiguration 200 — CLI proceeds
T+41s  api-token 200 ・ defaultSite 200 ・ UVE GET 403 ・ UVE POST 403
   … 193 consecutive attempts over ~7 minutes, zero successes …
T+440s UVE GET 403 ・ UVE POST 403

The reported log, line for line. Server-side:

DotSecurityException: User 'Admin User [dotcms.org.1][admin@dotcms.com]'
  does not have READ permissions on Site 'demo.dotcms.com'

The interrupted import never wrote the site's permission rows; the restart re-runs Task00004LoadStarter, reports a clean startup, and the permissions never appear. The instance does not recover.

What changed in the spec as a result:

  • AC-005 no longer polls until 200 — it would never terminate. Retry is restricted to 5xx; 403 explicitly does not retry.
  • The failure guidance is rewritten. "Configure UVE manually at this URL" is wrong advice — manual setup fails identically. The CLI must say the instance is unrecoverable and to run docker compose down -v.
  • The P2 backend non-goal is re-pointed. Not user.isAdmin() swallowing an exception (there is nothing to swallow — the data is absent), but a larger defect: any interrupted first boot silently bricks the instance while reporting success. Filed separately as An interrupted first boot silently bricks the instance: starter import leaves site permissions unwritten and every Apps API call 403s forever #37268.
  • Title and Reproducibility corrected — the 403 is deterministic once the crash has happened, not timing-dependent.

This raises US1's value. Fixing the compose file doesn't just stop a crash — it removes the 403 entirely, because it removes the interrupted boot that causes it.

Evidence limits: one host, one starter, one image, kill point fixed at 25s. Which kill-points corrupt is unmapped.


Updated after planning. Verifying the spec's claims against the tree turned up four that didn't hold, and I've folded the corrections back in before sign-off so you're approving what will actually be built:

  • "Six siblings use condition: service_healthy" — there are three, and none of them gates dotcms on OpenSearch being healthy (all use service_started). So doing both is a deliberate deviation from every precedent, not the house pattern the spec implied. Kept, with the reasoning stated, using single-node-os-migration's proven probe.
  • The flapping risk I flagged can't happen. Compose restart policies react to container exit, not health status. Replaced with the real exposure: a wrong probe blocks docker compose up --wait until timeout.
  • 8090:8090127.0.0.1:8090:8090. InfrastructureManagementFilter authorizes purely by arrival port — no credential check, no IP allowlist — so a wildcard binding puts /dotmgt/health and /dotmgt/metrics on the local network. New AC-011 asserts it's refused on the LAN address.
  • The Jest harness already exists; only spec files are missing. The spec said this fix establishes it.

Also added AC-012 for a landmine found while reading: updateDockerComposeStarterUrl rewrites the compose file with a regex and throws on no match, so reformatting CUSTOM_STARTER_URL would break --starter for every already-installed CLI, with no release able to reach them.

What's left for you to settle (the 8090 question is now answered above):

  1. Gating dotcms on opensearch: service_healthy goes further than all four existing examples. Right call for a stack driven by an unattended CLI, or should it match the house pattern? (My recommendation: keep the stricter gate.)
  2. Is start_period: 180s enough for a cold demo-starter import on a slow machine? Above both precedents (lgtm 120s, metrics-monitoring 20s). Too short and --wait blocks to timeout.

Note on blast radius

The compose file is fetched from main at runtime by every installed @dotcms/create-app. A change ships instantly and unversioned to all existing CLI users — including older versions that won't know about port 8090. That cuts both ways: it's why the fix is P0 and reaches users without a release, and it's the single largest regression risk here. Worth a careful look.

Checklist

  • Tests — n/a for a spec-only PR; the spec defines the Jest specs and cold-start compose verification that PR 2 must land first (constitution Principle V: Red before Green)
  • Translations — n/a
  • Security Implications Contemplated — publishing management port 8090 on the host is called out in the spec's Regression Risk and raised as review question 1 above

Additional Info

  • Follows the spec-kit flow in CLAUDE.md: /speckit-specify-fixPR 1 approved/speckit-plan/speckit-tasks/speckit-implement → PR 2
  • ADR consultation ran as the before_plan hook. ADR-0019 (accepted, binding) is recorded in the spec; ADR-0016 matched on keywords but is an unfilled template and imposes nothing. No conflicts with an accepted ADR.
  • Related: e2e: Add E2E test suite for @dotcms/create-app CLI #35096 (E2E suite for @dotcms/create-app) — owns the fault-injection coverage, out of scope here

Refs #37262

This PR fixes: #37262

@fmontes fmontes changed the title docs(create-app): add issue-resolution spec for #37262 docs(create-app): issue-resolution spec for local Docker start failure and transient UVE 403 (#37262) Aug 28, 2026
@claude

claude Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Claude finished @fmontes's task in 1m 28s —— View job


Spec review — specs/37262-create-app-docker-uve/spec.md

Spec-only PR (approval gate). I verified the concrete tree claims and read for internal consistency. The code references hold up — src/index.ts:369-371 exits on UVE failure before :376-378 (compose move → scaffold → move back), :597 tests if (!result) against the truthy Result, and the compose file matches the described shape (dotcms depends_on: [db, opensearch] with no condition/restart/healthcheck, publishes only 8082/8443, opensearch has no healthcheck, only db has one). Good, accurate diagnosis.

Three internal inconsistencies remain after the rescope — worth fixing before sign-off since reviewers are approving these exact values:

New Issues

  • 🟠 High: spec.md:266 vs spec.md:401start_period contradicts itself: 120s vs 180s. Fix Scope specifies the dotcms healthcheck at start_period: 120s ("~2.5× the measured ~46s boot"), but review question 2 asks reviewers "Is start_period: 180s enough headroom". These are different values for the same setting, so the reviewer question is asking about a number the spec doesn't actually propose. Pick one and make both places agree (120s matches the lgtm precedent cited; 180s does not). Fix this →

  • 🟡 Medium: spec.md:463-467Compose verification targets the file the rescope no longer changes. The Verification method runs docker compose -f docker/docker-compose-examples/single-node-demo-site/docker-compose.yml up -d --wait and asserts dotcms reaches healthy without manual start. But the rescope (lines 247-252) says the shared example is not modified and the CLI ships its own file at core-web/libs/sdk/create-app/assets/docker-compose.yml; AC-010 even asserts docker/docker-compose-examples/* is unchanged. So this verification step would test an unmodified file that still lacks the healthchecks/gating — it can't pass. Point it at the bundled asset. Fix this →

  • 🟡 Medium: spec.md:106"Three siblings" but four are listed and referenced. The sentence says "Three siblings already use condition: service_healthy" then lists four (lgtm-observability, single-node-metrics-monitoring, experiments, plus single-node-os-migration), and lines 397/401 both say "all four existing examples". I confirmed against the tree: exactly four compose files use condition: service_healthy (os-migration, metrics-monitoring, lgtm-observability, experiments). Change "Three" to "Four". Fix this →

Nothing blocking the substance of the approach — the diagnosis, the rescope to a bundled compose file, the loopback-only 8090 binding, and the non-poll/no-retry-on-403 decision are all well-argued and consistent with the evidence. These three are numeric/wording mismatches that would otherwise get baked into PR 2's acceptance criteria.

· issue-37262-create-app-docker-uve-spec

@fmontes
fmontes requested a review from jcastro-dotcms August 28, 2026 11:38
@fmontes fmontes changed the title docs(create-app): issue-resolution spec for local Docker start failure and transient UVE 403 (#37262) docs(create-app): issue-resolution spec for local Docker start failure and the resulting permanent UVE 403 (#37262) Aug 28, 2026
fmontes and others added 7 commits August 28, 2026 14:37
Spec-only PR 1 for the create-app local-Docker failure. Covers the compose
ordering/restart defect, the transient UVE 403, and the CLI's discarding of
recoverable state.

Verified every claim in the issue against the tree before writing:
- single-node-demo-site compose: dotcms has no depends_on condition, no
  restart policy, no healthcheck, and does not publish 8090; opensearch has
  no healthcheck; db's healthcheck exists but nothing consumes it
- src/index.ts:370 exits before the scaffolding at :377
- src/index.ts:597 tests `if (!result)` against a truthy `{ ok: false, val }`
- checkPortsAvailability() hard-fails on the ports a successful run holds
- the package contains no spec files, so this establishes the harness

Defers two P2 items as explicit non-goals: the user.isAdmin() exception
swallowing (legacy Liferay, hot permission path, wide blast radius) and
image-tag pinning (intersects binding ADR-0019). Neither blocks the P0 fix.

Names compose reviewers from git blame, since .github/CODEOWNERS does not
cover docker/ and the runtime-fetched compose file is the highest-blast-radius
part of this change.

Refs #37262

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two strongest git-blame signals on the compose file — spbolton (dominant
blame on every hunk in scope, and author of the lgtm-observability stack whose
service_healthy pattern this change copies) and dcolina — are no longer
collaborators on dotCMS/core, so GitHub rejects review requests for them.

Replaces them with jcastro-dotcms (second-most-active docker/ contributor over
the last 12 months) and records the resulting coverage gap explicitly: nobody
currently assignable designed the pattern being copied, so the plan phase should
read lgtm-observability/docker-compose.yml as the specification rather than rely
on a reviewer to catch a faithful-copy error.

Refs #37262

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Planning verified the spec's claims against the tree and four did not hold.
Correcting them here, before sign-off, so reviewers approve what we will
actually build.

- "Six siblings use condition: service_healthy" — there are three, and NONE
  gates dotcms on opensearch being healthy; all use service_started. Gating on
  both is therefore a deliberate deviation from every precedent, not the house
  pattern the spec implied. Kept, with the rationale stated and using
  os-migration's proven probe.

- The stated risk that a bad healthcheck would make restart: unless-stopped
  flap the container cannot happen: Compose restart policies react to container
  exit, not health status. Replaced with the real exposure — a wrong probe
  blocks `docker compose up --wait` until timeout — and required an explicit
  --wait-timeout.

- Publishing 8090 changed from "8090:8090, acceptable for a local stack" to
  "127.0.0.1:8090:8090". InfrastructureManagementFilter authorizes purely by
  arrival port: no credential check, no IP allowlist, so a wildcard binding puts
  /dotmgt/health and /dotmgt/metrics on the local network. Added AC-011.

- The package has no spec files but the Jest harness already exists, so this
  fix adds specs rather than establishing a harness. Recorded pnpm install as a
  Red-gate prerequisite and warned that passWithNoTests makes an empty run
  green.

Also adds AC-012 for a compose/CLI compatibility landmine found while reading:
updateDockerComposeStarterUrl rewrites the file with a regex and throws on no
match, so reformatting CUSTOM_STARTER_URL would break --starter for every
installed CLI. Notes that .env is new behavior, not a restoration, and downgrades
the /dotmgt/livez-on-latest assumption to partly-verified with a gating check.

Two of the three open review questions are now settled or reframed.

Refs #37262

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… not a race

Planning tested the spec's own hypothesis and disproved it. Correcting before
sign-off so reviewers approve the real diagnosis.

The spec claimed a transient startup race in which the CLI writes while roles
and permissions are still settling, with user.isAdmin() swallowing an exception
via Try.of(...).getOrElse(false). Two experiments (M5/64GB host, dotcms
constrained to 2 CPUs / 4G) say otherwise.

Clean boot has no settling window. The UVE endpoint is usable at 46s — two
seconds BEFORE /dotmgt/readyz goes green — because the starter import (T+20s)
and ES reindex (T+44s) both finish inside Tomcat startup and the connector
accepts no traffic until after them. The hypothesised race cannot occur.

The reporter's actual path does reproduce it, and permanently. Killing dotcms
mid starter-import and hand-starting it (reproduction step 4) yields the
reported log exactly — token 200, defaultSite 200, UVE 403 — and then 403 on
193 consecutive attempts over ~7 minutes with zero successes. The server says
the admin user lacks READ permission on demo.dotcms.com: the interrupted import
never wrote the site's permission rows, and the restart does not repair them.

So cause 2 is a consequence of cause 1, not an independent defect, and fixing
the compose file removes it.

Design consequences, not just narrative:
- AC-005 no longer polls until 200. A poll would never terminate; retry is
  restricted to 5xx and 403 explicitly does not retry.
- On 403 the CLI must tell the user the instance is unrecoverable and to run
  `docker compose down -v` — offering manual UVE setup steps is wrong advice,
  since manual configuration fails identically.
- The P2 backend non-goal is re-pointed: not isAdmin() exception swallowing,
  but the larger defect that any interrupted first boot silently bricks the
  instance while reporting a clean startup. Filed separately.

Title and Reproducibility updated: the 403 is deterministic once the crash has
happened, not timing-dependent.

Refs #37262

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The interrupted-first-boot corruption behind the 403 is filed as #37268.

Refs #37262

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AC-005 was corrected when the diagnosis changed, but the matching Fix Scope
bullet was not — leaving the spec contradicting itself: one section said "poll
GET until 200, retry on 401/403/5xx" while the other forbade exactly that.
Caught by /speckit-analyze.

Fix Scope now specifies a single probe, retry on 5xx only, and no retry on 403.

Also splits the non-fatal UVE guidance in two, because the cases need opposite
advice. On 403 the instance's permissions were never written, so manual UVE
setup fails identically — pointing the user at the configuration guide would
send them down a path that cannot work. That case tells them to recreate the
instance and references #37268. Every other failure keeps the guide link plus
host, site ID and app key.

Refs #37262, #37268

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fix no longer modifies
docker/docker-compose-examples/single-node-demo-site/docker-compose.yml. The CLI
gets its own file, bundled in the npm package, and the shared example is left
exactly as it is for README readers and installed CLIs.

Reason: every hardening step this fix wanted was otherwise a behavior change
shipped unversioned to consumers who never asked for it, because that file is
fetched from main at runtime. Gating dotcms on opensearch health was the sharpest
case — it introduces a way for dotCMS to never start if the probe later breaks
(an opensearch:1 -> :2 bump invalidating admin:admin), where today it starts
regardless. Owning the file makes strictness free.

This removes what the spec itself called the single largest regression risk in
the work. Two smaller risks replace it, both recorded: the bundled asset must be
listed in package.json `files` AND project.json esbuild `assets` or it ships
missing and every local-Docker run fails at step one (new AC-013); and strict
gating means a future broken opensearch probe stops dotCMS starting, contained to
this CLI's own stack.

Accepted consequence: users on <=1.2.5 keep the old shared file and are not
repaired. This starts fresh local instances rather than serving CI, no known
users have it in CI, and npx resolves to the latest published version.

AC-009 now requires continuous feedback for the whole wait, not just visible pull
progress — ten minutes of frozen spinner is the failure this issue was reported
for. AC-010 inverts to asserting docker/docker-compose-examples/* is UNCHANGED,
verified by diff.

Reviewer rationale updated: the blame-derived reviewers were chosen for a shared
file this work no longer touches.

Refs #37262

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fmontes
fmontes force-pushed the issue-37262-create-app-docker-uve-spec branch from 09da5c7 to 5d4cf45 Compare August 28, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

create-app: local Docker run never starts dotCMS, then a transient UVE 403 aborts the CLI and discards the project

1 participant