Skip to content

fix(docker): stage nginx config from RO source instead of envsubst - #96

Open
timoa wants to merge 3 commits into
offendingcommit:mainfrom
timoa:agent/devops/01a020ea
Open

fix(docker): stage nginx config from RO source instead of envsubst#96
timoa wants to merge 3 commits into
offendingcommit:mainfrom
timoa:agent/devops/01a020ea

Conversation

@timoa

@timoa timoa commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Problem

Two layered bugs caused the same symptom:

  1. The Dockerfile copied docker/nginx.conf.template to /etc/nginx/templates/default.conf.template, so the base image's 20-envsubst-on-templates.sh entrypoint tried to render it to /etc/nginx/conf.d/default.conf on every container start. Under the chart's readOnlyRootFilesystem: true that write failed with Read-only file system:

    20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
    20-envsubst-on-templates.sh: line 53: can't create /etc/nginx/conf.d/default.conf: Read-only file system
    
  2. Even after removing the template, the emptyDir tmpfs that the chart mounts at /etc/nginx/conf.d was being silently rejected by the kubelet — the template rendered the volume name as etc-nginx-conf.d (with a literal trailing dot from /etc/nginx/conf.d), which is not a valid DNS-1123 label. With the mount rejected, the image's default.conf stayed on the RO layer; 10-listen-on-ipv6-by-default.sh reported can not modify ... read-only file system? and 40-openconcho-config.sh's install failed with File exists (busybox install refuses to overwrite).

The template has no $VAR placeholders, so envsubst adds nothing anyway — the entire template/envsubst mechanism was unnecessary.

Fix

  • Dockerfile — ship the config at /etc/openconcho/nginx.conf (outside /etc/nginx/templates/) so 20-envsubst-on-templates.sh finds nothing to render and exits 0.
  • docker/40-openconcho-config.shcp -f /etc/openconcho/nginx.conf /etc/nginx/conf.d/default.conf (then chmod + chown to match the unprivileged image) at the top of the script. cp -f overwrites any file 10-listen-on-ipv6-by-default.sh may have created earlier in the boot chain.
  • charts/openconcho/templates/deployment.yaml — append replace "." "-" to the volume-name pipeline so /etc/nginx/conf.d sanitizes to etc-nginx-conf-d (valid DNS-1123) and the emptyDir actually mounts.
  • charts/openconcho/README.md — list /etc/nginx/conf.d in the "Writable paths" row.

The chart's tmpfsMounts: [ /etc/nginx/conf.d, ... ] and fsGroup: 101 settings stay required — kubelet chowns the emptyDir to group 101, then our writes land in the RW tmpfs.

Verification

  • helm template now renders all four volumes with valid DNS-1123 names (etc-nginx-conf-d, var-cache-nginx, var-run, tmp); the previous render emitted etc-nginx-conf.d which is invalid.
  • make smoke-docker → 8/8 PASS (/api proxy, allowlist, missing-header, reject sentinel)
  • Built image openconcho-web:fix2, ran on a plain filesystem: full entrypoint chain (10-listen reports "differs from packaged version" and exits 0, 20-envsubst has no templates and exits 0, 40-openconcho-config.sh overwrites default.conf and writes the resolver + allowlist maps), nginx starts, /healthz returns ok, /api proxy forwards correctly.
Dockerfile                                  |  8 ++++++--
charts/openconcho/README.md                 |  2 +-
charts/openconcho/templates/deployment.yaml |  4 ++--
docker/40-openconcho-config.sh              |  14 ++++++++++----

DevOps added 3 commits August 20, 2026 22:52
The base image's 20-envsubst-on-templates.sh entrypoint tried to render
docker/nginx.conf.template into /etc/nginx/conf.d/default.conf on every
container start. On Kubernetes with readOnlyRootFilesystem=true that
write failed with 'Read-only file system', blocking the container. The
template has no $VAR placeholders, so envsubst adds nothing anyway.

Ship the config at /etc/openconcho/nginx.conf and have
40-openconcho-config.sh install it into the chart's writable
/etc/nginx/conf.d/ tmpfs before nginx starts. The base image's 10-listen
and 20-envsubst scripts exit 0 cleanly with the templates dir empty.

The chart's tmpfsMounts entry for /etc/nginx/conf.d stays required
(fsGroup chowns it to 101, then install -o 101 -g 101 writes the file
the same UID the entrypoint runs as).
The previous fix assumed the chart's emptyDir at /etc/nginx/conf.d was
being mounted, but on a fresh K8s cluster the kubelet rejected the mount
because the rendered volume name 'etc-nginx-conf.d' (note trailing dot,
from the source path '/etc/nginx/conf.d') is not a valid DNS-1123 label.
Without the mount, the image's default.conf remained visible on the RO
layer; the 10-listen script reported 'can not modify ... read-only file
system?' and 40-openconcho-config.sh's install command failed with
'File exists' (busybox install refuses to overwrite).

Add 'replace "." "-"' to the volume name pipeline in deployment.yaml
so 'etc/nginx/conf.d' sanitizes to 'etc-nginx-conf-d'. Also switch
40-openconcho-config.sh from 'install' to 'cp -f' + chmod + chown so
the file is forced-overwritten even if 10-listen touched it earlier in
the boot chain (the emptyDir is RW on a working mount).

Verified 'helm template' now renders all four volumes with valid
DNS-1123 names; smoke test still passes 8/8.
Adds .github/workflows/chart-publish.yml so the fork can publish
charts/openconcho to ghcr.io/<owner>/charts on demand, instead of
waiting for an upstream release tag.

The existing docker-publish.yml publish-chart job is gated on
startsWith(github.ref, 'refs/tags/') and depends on the image build
job, so it can't be triggered manually. This new workflow is
standalone and takes a version input (default 0.14.0-dev) so the
fork can publish a chart with the same fix as the image (e.g.
0.14.0-sha-<short>) and install from oci://ghcr.io/timoa/charts/
without depending on the upstream OCI registry.

Mirrors the image pattern: push the workflow, then 'gh workflow
run' to publish.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant