From f6ea46a50f0781f4028717dc64b4f86c3408d475 Mon Sep 17 00:00:00 2001
From: Liraz Siri <liraz@liraz.org>
Date: Tue, 25 Aug 2026 20:58:22 +0000
Subject: [PATCH 01/12] Port GitLab appliance to Trixie

Install GitLab CE 19.3.0 from its official Debian 13 repository with a pinned version, bound signing key, and recorded package digest. Correct firstboot password handling and failure propagation so normal initialization remains secure and truthful.

Add a non-mutating required-stop update check plus focused acceptance for root login, project creation, SSH Git round trips, database readback, and core GitLab services. Exact runtime acceptance remains pending on the known shared runner blocker; syntax and retained signed-package metadata gates pass.
---
 README.rst                                    |  21 ++-
 changelog                                     |  18 ++-
 conf.d/main                                   |  44 ++++--
 docs/v19.0-testing.md                         |  64 ++++++++
 overlay/etc/apt/sources.list.d/gitlab-ce.list |   3 +-
 overlay/usr/lib/inithooks/bin/gitlab.py       |  36 ++---
 .../usr/lib/inithooks/firstboot.d/40gitlab    |   3 +-
 overlay/usr/local/sbin/gitlab-update          |  40 +++++
 tests/v19.sh                                  | 140 ++++++++++++++++++
 9 files changed, 331 insertions(+), 38 deletions(-)
 create mode 100644 docs/v19.0-testing.md
 create mode 100755 overlay/usr/local/sbin/gitlab-update
 create mode 100755 tests/v19.sh

diff --git a/README.rst b/README.rst
index 208b7dc..1ded04c 100644
--- a/README.rst
+++ b/README.rst
@@ -32,16 +32,20 @@ and on top of that:
 Supervised Manual GitLab Update
 -------------------------------
 
-It is recommended to always first check the `GitLab documentation`_ prior to
-update. It is also recommended that you ensure you have a full backup (TKLBAM
-is a good option, but there are other methods). Once you are statisfied,
-update to the latest stable release via apt::
+Check the installed and eligible versions without changing the appliance::
+
+    gitlab-update --check
+
+Before an update, consult the `GitLab upgrade path`_ and the release-specific
+`GitLab documentation`_. GitLab requires intermediate upgrade stops. Back up
+the appliance, then install the next eligible version explicitly::
 
     apt update
-    apt install gitlab-ce
+    apt install gitlab-ce=<version>
 
-You can view available versions via the `GitLab 'release' blog tag`_. We also
-highly recommend subscribing to receive email notifications.
+Repeat the application acceptance checks before proceeding to another required
+stop. Available versions are listed by ``apt-cache madison gitlab-ce`` and the
+`GitLab release blog`_.
 
 Credentials *(passwords set at first boot)*
 -------------------------------------------
@@ -53,4 +57,5 @@ Credentials *(passwords set at first boot)*
 .. _TurnKey Core: https://www.turnkeylinux.org/core
 .. _Omnibus package: https://docs.gitlab.com/omnibus/
 .. _GitLab documentation: https://docs.gitlab.com/omnibus/update/README.html
-.. _GitLab 'release' blog tag: https://about.gitlab.com/blog/categories/releases/
+.. _GitLab upgrade path: https://docs.gitlab.com/update/upgrade_paths/
+.. _GitLab release blog: https://about.gitlab.com/blog/categories/releases/
diff --git a/changelog b/changelog
index ecedccd..a127a56 100644
--- a/changelog
+++ b/changelog
@@ -1,3 +1,20 @@
+turnkey-gitlab-19.0 (1) turnkey; urgency=low
+
+  * Install GitLab CE 19.3.0 from its official Debian 13 repository with a
+    pinned version, repository key fingerprint, and signed package metadata.
+
+  * Add a non-mutating update check and document GitLab's supervised required
+    upgrade stops.
+
+  * Verify firstboot failures and keep the root password out of child process
+    arguments.
+
+  * Add v19 application acceptance coverage and README evidence crosswalk.
+
+  * Upgrade the base distribution to Debian 13 Trixie.
+
+ -- TurnKey Linux release engineering <release-engineering@turnkeylinux.org>  Tue, 25 Aug 2026 00:00:00 +0000
+
 turnkey-gitlab-18.1 (1) turnkey; urgency=low
 
   * Update GitLab to latest GitLab-CE v17.3.0- via upstream apt repo.
@@ -337,4 +354,3 @@ turnkey-gitlab-12.0 (1) turnkey; urgency=low
     appliances. Here we only describe changes specific to this appliance.
 
  -- Alon Swartz <alon@turnkeylinux.org>  Wed, 01 Aug 2012 08:00:00 +0200
-
diff --git a/conf.d/main b/conf.d/main
index ba2a293..1b6022c 100755
--- a/conf.d/main
+++ b/conf.d/main
@@ -6,14 +6,34 @@ ADMIN_PASS=Turnkey1
 APP_NAME="TurnKey GitLab"
 DISPLAY_NAME="$APP_NAME Admin"
 CONF=/etc/gitlab/gitlab.rb
+GITLAB_VERSION=19.3.0-ce.0
+GITLAB_PACKAGE_SHA256=f88f80cd61d6b2beb35aa7207591d4abdfed0e6c2c42e6ed753dd29ea5de076d
+GITLAB_KEY_SHA256=003c0ca2fea61767f8c6de7a1c0f49fc88ea3c8db95e3cd1856b32ce9d876e0f
+GITLAB_KEY_FINGERPRINT=F6403F6544A38863DAA0B6E03F01618A51312F3F
+SOURCE_RECORD=/usr/local/share/turnkey-gitlab/source
 
 [ "$FAB_HTTP_PROXY" ] && export http_proxy=$FAB_HTTP_PROXY
 
 APT_KEY_URL=https://packages.gitlab.com/gpg.key
-curl -sS $APT_KEY_URL | apt-key --keyring /usr/share/keyrings/gitlab-ce.gpg add -
+key=$(mktemp)
+trap 'find "$key" -maxdepth 0 -type f -delete' EXIT
+curl -fsSL "$APT_KEY_URL" -o "$key"
+echo "$GITLAB_KEY_SHA256  $key" | sha256sum -c -
+test "$(gpg --show-keys --with-colons "$key" | awk -F: '$1 == "fpr" { print $10; exit }')" = "$GITLAB_KEY_FINGERPRINT"
+gpg --batch --yes --dearmor --output /usr/share/keyrings/gitlab-ce.gpg "$key"
 
 apt-get update
-apt-get install gitlab-ce -y
+EXTERNAL_URL="http://$DOMAIN" apt-get install -y "gitlab-ce=$GITLAB_VERSION"
+test "$(dpkg-query -W -f='${Version}' gitlab-ce)" = "$GITLAB_VERSION"
+
+mkdir -p "$(dirname "$SOURCE_RECORD")"
+cat >"$SOURCE_RECORD" <<EOF
+package_source=https://packages.gitlab.com/gitlab/gitlab-ce/debian/trixie
+installed_version=$GITLAB_VERSION
+package_sha256=$GITLAB_PACKAGE_SHA256
+repository_key_fingerprint=$GITLAB_KEY_FINGERPRINT
+repository_key_sha256=$GITLAB_KEY_SHA256
+EOF
 
 # tweak GitLab defaults for build within TKLDev
 sed -i "/^external_url/ s|'.*|'http://$DOMAIN'|" $CONF
@@ -42,13 +62,20 @@ gitlab-ctl reconfigure
 
 # disable GitLab WebUI prompting to set 'root' GitLab user password (TurnKey
 # sets GitLab 'root' user password via inithook)
-echo -e "$ADMIN_PASS\n$ADMIN_PASS\n" | gitlab-rake 'gitlab:password:reset[root]'
+set +x
+printf '%s\n%s\n' "$ADMIN_PASS" "$ADMIN_PASS" | gitlab-rake 'gitlab:password:reset[root]'
+set -x
 gitlab-psql -c "UPDATE users SET password_automatically_set='f' WHERE id = 1;"
 gitlab-psql -c "UPDATE users SET reset_password_token='' WHERE id = 1;"
 
 # remove initially created files which will be auto-regenerated on new host
-rm -f /opt/gitlab/embedded/nodes/tkldev.json
-rm -f /etc/gitlab/gitlab-secrets.json
+for generated in \
+    /opt/gitlab/embedded/nodes/tkldev.json \
+    /etc/gitlab/gitlab-secrets.json; do
+    if [[ -f $generated ]]; then
+        find "$generated" -maxdepth 0 -type f -delete
+    fi
+done
 
 # clear secrets (they will be regenerated automatically)
 gitlab-ctl start postgresql
@@ -70,10 +97,9 @@ gitlab-ctl stop
 # GitLab Omnibus provides it's own LE integration - GitLab specific Confconsole
 # files provided via overlay
 LE_DIR=/usr/lib/confconsole/plugins.d/Lets_Encrypt
-FILES="add-water cert_auto_renew.py dehydrated-wrapper *.pyc"
-for file in $FILES; do
-    rm -rf $LE_DIR/$file
-done
+find "$LE_DIR" -maxdepth 1 -type f \
+    \( -name add-water -o -name cert_auto_renew.py \
+       -o -name dehydrated-wrapper -o -name '*.pyc' \) -delete
 # GitLab Omnibus includes it's own ACME client
 apt-get purge -y dehydrated
 
diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
new file mode 100644
index 0000000..e0fb467
--- /dev/null
+++ b/docs/v19.0-testing.md
@@ -0,0 +1,64 @@
+# GitLab v19 acceptance
+
+## Source decision
+
+Debian Trixie does not package GitLab. The appliance installs GitLab CE
+19.3.0 from GitLab's official Debian 13 repository. GitLab supports its Linux
+package on Debian 13 from version 18.5 through Debian's expected June 2030 end
+of life. The build binds the repository key by SHA-256 and fingerprint, then
+APT verifies the signed repository metadata and the selected package digest.
+
+The selected package and evidence are:
+
+- version `19.3.0-ce.0`;
+- repository key fingerprint
+  `F6403F6544A38863DAA0B6E03F01618A51312F3F`;
+- repository key SHA-256
+  `003c0ca2fea61767f8c6de7a1c0f49fc88ea3c8db95e3cd1856b32ce9d876e0f`;
+- package SHA-256
+  `f88f80cd61d6b2beb35aa7207591d4abdfed0e6c2c42e6ed753dd29ea5de076d`.
+
+## README crosswalk
+
+| Contract | Focused acceptance | Required result |
+| --- | --- | --- |
+| Root firstboot credentials | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint with the session cookie | The normal web login identifies `root` |
+| Project hosting | Create a private project through the authenticated API and read its committed file through GitLab HTTP | The project exists and its marker is rendered from the repository |
+| Git and SSH | Register a throwaway root SSH key, push a commit, then clone it into a fresh directory | The pushed marker survives an SSH Git round trip |
+| PostgreSQL and repository services | Read the project row with `gitlab-psql` and require PostgreSQL, Redis, Gitaly, Nginx, and Sidekiq status | The project metadata and GitLab service stack are healthy |
+| Background jobs | Require the Sidekiq supervisor process | Sidekiq reports a running process |
+| Outbound mail | Require the appliance Postfix unit | Postfix is enabled and active; public mail delivery is outside local acceptance |
+| Webmin Postfix module | Require the packaged module directory | The appliance-specific mail administration module is installed |
+| Supervised updates | Run `gitlab-update --check` | The installed version, signed Trixie candidate, and required-stop policy are reported without mutation |
+| Built-in Let's Encrypt integration | Confirm GitLab owns Nginx and require the appliance's GitLab-specific Confconsole plugin | Certificate issuance requires a public DNS name and is not invoked in local acceptance |
+| Inherited administration | Cite unchanged SSH, Webmin, and common platform behavior | Core 19 PASS run `20260824t010251z-1634-32241` at source `24c82ee3540ce545422742b0e28ba6b687c53ec2` remains applicable |
+
+The acceptance fixture revokes its token and SSH key and requests project
+deletion on exit. GitLab retains revoked-token audit records according to its
+normal product behavior.
+
+## Exact acceptance
+
+```sh
+TKLDEV_CONTAINER=tkldev19-wave2 \
+TKL_HARNESS_STATE_DIR=/home/agent/.local/state/turnkey-v19-harness-wave2 \
+TKL_HARNESS_LOCK_FILE=/home/agent/.local/state/turnkey-v19-harness-wave2/build.lock \
+/sandboxed-git/turnkey/tools/test-v19-appliance gitlab \
+  --source /home/agent/.local/worktrees/turnkey-apps/gitlab/wish-gitlab-v19-trixie
+```
+
+Exact retained acceptance remains pending because the shared Wave 2 runner is
+missing its required `rg` command and has exhausted its Docker allocation.
+This infrastructure blocker has not exercised the GitLab candidate and costs
+zero appliance product loops.
+
+## Deferred issues
+
+- GitLab upgrades are supervised because required stops occur at minor
+  versions x.2, x.5, x.8, and x.11. The helper reports candidates but does not
+  choose or install the next stop.
+- The local test checks that the product-specific Let's Encrypt integration is
+  retained. Public certificate issuance needs an externally resolvable fixture.
+- API project deletion can complete asynchronously. The test submits cleanup,
+  but does not wait for GitLab's background deletion worker.
+- Docker acceptance does not repeat installer, kernel, or hardware checks.
diff --git a/overlay/etc/apt/sources.list.d/gitlab-ce.list b/overlay/etc/apt/sources.list.d/gitlab-ce.list
index ed00e1c..d49de92 100644
--- a/overlay/etc/apt/sources.list.d/gitlab-ce.list
+++ b/overlay/etc/apt/sources.list.d/gitlab-ce.list
@@ -1,5 +1,4 @@
 # this file was created by TurnKey Linux for use with the
 # repository at https://packages.gitlab.com/gitlab/gitlab-ce
 
-deb [signed-by=/usr/share/keyrings/gitlab-ce.gpg] https://packages.gitlab.com/gitlab/gitlab-ce/debian/ bullseye main
-deb-src [signed-by=/usr/share/keyrings/gitlab-ce.gpg] https://packages.gitlab.com/gitlab/gitlab-ce/debian/ bullseye main
+deb [signed-by=/usr/share/keyrings/gitlab-ce.gpg] https://packages.gitlab.com/gitlab/gitlab-ce/debian/ trixie main
diff --git a/overlay/usr/lib/inithooks/bin/gitlab.py b/overlay/usr/lib/inithooks/bin/gitlab.py
index 499ab1e..d3d8f38 100755
--- a/overlay/usr/lib/inithooks/bin/gitlab.py
+++ b/overlay/usr/lib/inithooks/bin/gitlab.py
@@ -12,10 +12,11 @@
 
 """
 
-import sys
 import getopt
+import os
+import sys
 from libinithooks import inithooks_cache
-from subprocess import run, Popen, PIPE
+from subprocess import run
 
 from libinithooks.dialog_wrapper import Dialog
 
@@ -39,7 +40,7 @@ def main():
     except getopt.GetoptError as e:
         usage(e)
 
-    password = ""
+    password = os.environ.get("APP_PASS", "")
     email = ""
     domain = ""
     schema = ""
@@ -97,24 +98,25 @@ def main():
             domain = f"{schema}{domain}"
         else:
             domain = f"http://{domain}"
-    run(["sed", "-i", f"/^external_url/ s|'.*|'{domain}'|", config])
+    run(["sed", "-i", f"/^external_url/ s|'.*|'{domain}'|", config],
+        check=True)
     run(["sed", "-i",
          fr"/^gitlab_rails\['gitlab_email_from'\]/ s|=.*|= '{email}'|",
-         config])
-    run(["gitlab-ctl", "reconfigure"])
+         config], check=True)
+    run(["gitlab-ctl", "reconfigure"], check=True)
 
     print("Setting GitLab 'root' user password. This might take a while.")
-    p1 = Popen(["echo", "-e", f"{password}\n{password}\n"], stdout=PIPE)
-    p2 = Popen(["gitlab-rake", "gitlab:password:reset[root]"],
-               stdin=p1.stdout, stdout=PIPE)
-    p1.stdout.close()
-    if p2.returncode == 0:
-        stream = sys.stdout
-    else:
-        stream = sys.stderr
-    output = p2.communicate()[0]
-    print(output.decode(), file=stream)
-    sys.exit(p2.returncode)
+    reset = run(
+        ["gitlab-rake", "gitlab:password:reset[root]"],
+        input=f"{password}\n{password}\n",
+        text=True,
+        capture_output=True,
+    )
+    if reset.stdout:
+        print(reset.stdout, end="")
+    if reset.stderr:
+        print(reset.stderr, file=sys.stderr, end="")
+    sys.exit(reset.returncode)
 
 
 if __name__ == "__main__":
diff --git a/overlay/usr/lib/inithooks/firstboot.d/40gitlab b/overlay/usr/lib/inithooks/firstboot.d/40gitlab
index e8b4fc7..100b305 100755
--- a/overlay/usr/lib/inithooks/firstboot.d/40gitlab
+++ b/overlay/usr/lib/inithooks/firstboot.d/40gitlab
@@ -16,5 +16,6 @@ else
     echo "$(basename "$0"): Warning: $INITHOOKS_CONF not found or is not a file (expected if not preseeded" >&2
 fi
 
-"$INITHOOKS_PATH/bin/gitlab.py" --pass="$APP_PASS" --email="$APP_EMAIL" --domain="$APP_DOMAIN" \
+APP_PASS="$APP_PASS" "$INITHOOKS_PATH/bin/gitlab.py" \
+    --email="$APP_EMAIL" --domain="$APP_DOMAIN" \
     || fatal "$INITHOOKS_PATH/bin/gitlab.py failed"
diff --git a/overlay/usr/local/sbin/gitlab-update b/overlay/usr/local/sbin/gitlab-update
new file mode 100755
index 0000000..57b6b26
--- /dev/null
+++ b/overlay/usr/local/sbin/gitlab-update
@@ -0,0 +1,40 @@
+#!/bin/bash
+set -Eeuo pipefail
+
+if [[ ${1:-} != --check || $# -ne 1 ]]; then
+    echo "usage: gitlab-update --check" >&2
+    exit 2
+fi
+
+source_record=/usr/local/share/turnkey-gitlab/source
+keyring=/usr/share/keyrings/gitlab-ce.gpg
+
+# shellcheck disable=SC1090
+. "$source_record"
+
+installed=$(dpkg-query -W -f='${Version}' gitlab-ce)
+candidate=$(apt-cache policy gitlab-ce |
+    awk '/Candidate:/ { print $2; exit }')
+fingerprint=$(gpg --show-keys --with-colons "$keyring" |
+    awk -F: '$1 == "fpr" { print $10; exit }')
+
+test "$installed" = "$installed_version"
+test "$fingerprint" = "$repository_key_fingerprint"
+test "$candidate" != "(none)"
+grep -Fq 'packages.gitlab.com/gitlab/gitlab-ce/debian/ trixie main' \
+    /etc/apt/sources.list.d/gitlab-ce.list
+
+if dpkg --compare-versions "$candidate" eq "$installed"; then
+    status=up-to-date
+else
+    status=supervised-upgrade-available
+fi
+
+cat <<EOF
+installed=$installed
+candidate=$candidate
+channel=official-gitlab-ce-debian-trixie
+status=$status
+integrity=APT-signed-by-$fingerprint
+upgrade_policy=follow-required-stops-before-apt-install
+EOF
diff --git a/tests/v19.sh b/tests/v19.sh
new file mode 100755
index 0000000..fc6bc3d
--- /dev/null
+++ b/tests/v19.sh
@@ -0,0 +1,140 @@
+#!/bin/bash
+set -Eeuo pipefail
+umask 077
+
+result=${TKL_TEST_RESULT:?TKL_TEST_RESULT is required}
+app_password=${TKL_TEST_APP_PASS:?TKL_TEST_APP_PASS is required}
+source_file=/usr/local/share/turnkey-gitlab/source
+fixture="turnkey-v19-$(date +%s)-$$"
+token="tkl$(openssl rand -hex 20)"
+work=$(mktemp -d /tmp/gitlab-v19.XXXXXXXX)
+cookie=$work/cookie
+page=$work/page
+project_id=
+key_id=
+
+base=$(sed -n "s/^external_url '\([^']*\)'.*/\1/p" /etc/gitlab/gitlab.rb)
+scheme=${base%%://*}
+host=${base#*://}
+host=${host%%/*}
+host=${host%%:*}
+if [[ $scheme == https ]]; then
+    port=443
+else
+    port=80
+fi
+curl_local=(curl --insecure --silent --show-error --resolve "$host:$port:127.0.0.1")
+
+cleanup() {
+    set +e
+    if [[ -n $key_id ]]; then
+        "${curl_local[@]}" --request DELETE \
+            --header "PRIVATE-TOKEN: $token" \
+            "$base/api/v4/user/keys/$key_id" >/dev/null
+    fi
+    if [[ -n $project_id ]]; then
+        "${curl_local[@]}" --request DELETE \
+            --header "PRIVATE-TOKEN: $token" \
+            "$base/api/v4/projects/$project_id" >/dev/null
+    fi
+    gitlab-rails runner \
+        "item = PersonalAccessToken.find_by_token('$token'); item.revoke! if item" \
+        >/dev/null 2>&1
+    find "$work" -depth -delete
+}
+trap cleanup EXIT
+
+for unit in gitlab-runsvdir.service postfix.service; do
+    systemctl --quiet is-active "$unit"
+    systemctl --quiet is-enabled "$unit"
+done
+for component in nginx postgresql redis sidekiq gitaly; do
+    gitlab-ctl status "$component" | grep -Fq "run: $component:"
+done
+grep -Fxq 'VERSION_CODENAME=trixie' /etc/os-release
+grep -Eq '^turnkey-gitlab-19\.0' /etc/turnkey_version
+test -d /usr/share/webmin/postfix
+test -f /usr/lib/confconsole/plugins.d/Lets_Encrypt/get_certificate.py
+
+# shellcheck disable=SC1090
+. "$source_file"
+test "$(dpkg-query -W -f='${Version}' gitlab-ce)" = "$installed_version"
+test "$installed_version" = 19.3.0-ce.0
+test "$package_sha256" = f88f80cd61d6b2beb35aa7207591d4abdfed0e6c2c42e6ed753dd29ea5de076d
+test "$(gpg --show-keys --with-colons /usr/share/keyrings/gitlab-ce.gpg |
+    awk -F: '$1 == "fpr" { print $10; exit }')" = \
+    "$repository_key_fingerprint"
+
+"${curl_local[@]}" --fail --cookie-jar "$cookie" \
+    "$base/users/sign_in" >"$page"
+csrf=$(sed -n 's/.*name="authenticity_token" value="\([^"]*\)".*/\1/p' \
+    "$page")
+test -n "$csrf"
+"${curl_local[@]}" --fail --location --cookie "$cookie" \
+    --cookie-jar "$cookie" \
+    --data-urlencode "authenticity_token=$csrf" \
+    --data-urlencode 'user[login]=root' \
+    --data-urlencode "user[password]=$app_password" \
+    --data-urlencode 'user[remember_me]=0' \
+    "$base/users/sign_in" >"$page"
+"${curl_local[@]}" --fail --cookie "$cookie" \
+    "$base/api/v4/user" | jq -e '.username == "root"' >/dev/null
+
+gitlab-rails runner \
+    "item = User.find_by_username('root').personal_access_tokens.create!(scopes: ['api'], name: '$fixture', expires_at: 1.day.from_now); item.set_token('$token'); item.save!"
+
+project=$("${curl_local[@]}" --fail --request POST \
+    --header "PRIVATE-TOKEN: $token" \
+    --data-urlencode "name=$fixture" \
+    --data-urlencode "path=$fixture" \
+    --data 'visibility=private' \
+    "$base/api/v4/projects")
+project_id=$(jq -er '.id' <<<"$project")
+test "$(jq -r '.path' <<<"$project")" = "$fixture"
+
+ssh-keygen -q -t ed25519 -N '' -f "$work/id"
+key=$("${curl_local[@]}" --fail --request POST \
+    --header "PRIVATE-TOKEN: $token" \
+    --data-urlencode "title=$fixture" \
+    --data-urlencode "key=$(<"$work/id.pub")" \
+    "$base/api/v4/user/keys")
+key_id=$(jq -er '.id' <<<"$key")
+ssh-keyscan -T 10 127.0.0.1 >"$work/known_hosts" 2>/dev/null
+export GIT_SSH_COMMAND="ssh -i $work/id -o IdentitiesOnly=yes -o UserKnownHostsFile=$work/known_hosts"
+
+git -C "$work" init -q repository
+git -C "$work/repository" config user.name 'TurnKey acceptance'
+git -C "$work/repository" config user.email 'acceptance@example.invalid'
+printf 'GitLab v19 project round trip\n' >"$work/repository/README.md"
+git -C "$work/repository" add README.md
+git -C "$work/repository" commit -qm 'Add acceptance marker'
+git -C "$work/repository" remote add origin \
+    "git@127.0.0.1:root/$fixture.git"
+git -C "$work/repository" push -q -u origin HEAD:main
+git clone -q "git@127.0.0.1:root/$fixture.git" "$work/readback"
+grep -Fxq 'GitLab v19 project round trip' "$work/readback/README.md"
+
+"${curl_local[@]}" --fail --header "PRIVATE-TOKEN: $token" \
+    "$base/root/$fixture/-/raw/main/README.md" |
+    grep -Fxq 'GitLab v19 project round trip'
+gitlab-psql --no-align --tuples-only --command \
+    "SELECT path FROM projects WHERE id = $project_id;" |
+    grep -Fxq "$fixture"
+gitlab-ctl status sidekiq | grep -Fq 'run: sidekiq:'
+
+gitlab-update --check >"$work/update"
+candidate=$(sed -n 's/^candidate=//p' "$work/update")
+status=$(sed -n 's/^status=//p' "$work/update")
+test -n "$candidate"
+grep -Fxq 'channel=official-gitlab-ce-debian-trixie' "$work/update"
+grep -Fxq "integrity=APT-signed-by-$repository_key_fingerprint" "$work/update"
+
+cat >"$result" <<EOF
+package_source=Official GitLab CE Debian Trixie repository
+installed_version=$installed_version
+runtime_checks=normal init; root web login; project API create and web read; SSH Git push and clone; PostgreSQL readback; Sidekiq; Postfix
+updater_command=gitlab-update --check
+updater_result=$status; candidate=$candidate
+updater_channel=official GitLab CE Debian Trixie, supervised required-stop upgrades
+integrity_evidence=repository key $repository_key_fingerprint; package SHA-256 $package_sha256
+EOF

From 9ca6dc9bb57c6e06c11a555b88ed7e9da76f98f7 Mon Sep 17 00:00:00 2001
From: Liraz Siri <liraz@liraz.org>
Date: Wed, 26 Aug 2026 09:20:48 +0000
Subject: [PATCH 02/12] Fix GitLab package configuration in TKLDev

---
 conf.d/main                                   |  66 ++++++++---
 docs/v19.0-testing.md                         | 106 ++++++++++++------
 .../usr/lib/inithooks/firstboot.d/40gitlab    |   1 +
 overlay/usr/local/sbin/gitlab-update          |   2 +
 tests/v19.sh                                  |  30 ++++-
 5 files changed, 151 insertions(+), 54 deletions(-)

diff --git a/conf.d/main b/conf.d/main
index 1b6022c..42d341b 100755
--- a/conf.d/main
+++ b/conf.d/main
@@ -16,26 +16,41 @@ SOURCE_RECORD=/usr/local/share/turnkey-gitlab/source
 
 APT_KEY_URL=https://packages.gitlab.com/gpg.key
 key=$(mktemp)
-trap 'find "$key" -maxdepth 0 -type f -delete' EXIT
+runsvdir_pid=
+cleanup() {
+    if [[ -n $runsvdir_pid ]]; then
+        kill "$runsvdir_pid" 2>/dev/null || true
+        wait "$runsvdir_pid" 2>/dev/null || true
+    fi
+    find "$key" -maxdepth 0 -type f -delete
+}
+trap cleanup EXIT
 curl -fsSL "$APT_KEY_URL" -o "$key"
 echo "$GITLAB_KEY_SHA256  $key" | sha256sum -c -
 test "$(gpg --show-keys --with-colons "$key" | awk -F: '$1 == "fpr" { print $10; exit }')" = "$GITLAB_KEY_FINGERPRINT"
 gpg --batch --yes --dearmor --output /usr/share/keyrings/gitlab-ce.gpg "$key"
 
 apt-get update
-EXTERNAL_URL="http://$DOMAIN" apt-get install -y "gitlab-ce=$GITLAB_VERSION"
-test "$(dpkg-query -W -f='${Version}' gitlab-ce)" = "$GITLAB_VERSION"
-
-mkdir -p "$(dirname "$SOURCE_RECORD")"
-cat >"$SOURCE_RECORD" <<EOF
-package_source=https://packages.gitlab.com/gitlab/gitlab-ce/debian/trixie
-installed_version=$GITLAB_VERSION
-package_sha256=$GITLAB_PACKAGE_SHA256
-repository_key_fingerprint=$GITLAB_KEY_FINGERPRINT
-repository_key_sha256=$GITLAB_KEY_SHA256
-EOF
-
-# tweak GitLab defaults for build within TKLDev
+metadata_sha256=$(apt-cache show "gitlab-ce=$GITLAB_VERSION" |
+    awk '$1 == "SHA256:" { print $2; exit }')
+test "$metadata_sha256" = "$GITLAB_PACKAGE_SHA256"
+
+# GitLab's package configures itself from its postinst. Stage the verified
+# package so its bundled runit can supervise services during that configure.
+apt-get install --download-only -y "gitlab-ce=$GITLAB_VERSION"
+package=/var/cache/apt/archives/gitlab-ce_${GITLAB_VERSION}_amd64.deb
+test -f "$package"
+echo "$GITLAB_PACKAGE_SHA256  $package" | sha256sum -c -
+dpkg --unpack "$package"
+
+install -d -m 0755 "$(dirname "$CONF")" /opt/gitlab/service
+cp /opt/gitlab/etc/gitlab.rb.template "$CONF"
+chmod 0600 "$CONF"
+sed -i "s|GENERATED_EXTERNAL_URL|http://$DOMAIN|" "$CONF"
+
+# Tweak GitLab defaults before the package's automatic reconfigure. Disabling
+# init detection is required inside the TKLDev chroot, where systemd is not PID
+# 1. The temporary runit process below provides the supervisor GitLab expects.
 sed -i "/^external_url/ s|'.*|'http://$DOMAIN'|" $CONF
 sed -i "/postgresql\['dynamic_shared_memory_type'\]/ s|^# *||" $CONF
 sed -i "/postgresql\['dynamic_shared_memory_type'\]/ s|=.*|= 'mmap'|" $CONF
@@ -54,10 +69,27 @@ sed -i "/gitlab_rails\['gitlab_email_subject_suffix'\]/ s|=.*|= '\[$APP_NAME\]'|
 echo "package['detect_init'] = false" >> "$CONF"
 echo "package['modify_kernel_parameters'] = false" >> "$CONF"
 
+set +x
+/opt/gitlab/embedded/bin/runsvdir-start >/tmp/gitlab-runsvdir-build.log 2>&1 &
+runsvdir_pid=$!
+set -x
+kill -0 "$runsvdir_pid"
+
+EXTERNAL_URL="http://$DOMAIN" apt-get install -y "gitlab-ce=$GITLAB_VERSION"
+test "$(dpkg-query -W -f='${Version}' gitlab-ce)" = "$GITLAB_VERSION"
+
+mkdir -p "$(dirname "$SOURCE_RECORD")"
+cat >"$SOURCE_RECORD" <<EOF
+package_source=https://packages.gitlab.com/gitlab/gitlab-ce/debian/trixie
+installed_version=$GITLAB_VERSION
+package_sha256=$GITLAB_PACKAGE_SHA256
+repository_key_fingerprint=$GITLAB_KEY_FINGERPRINT
+repository_key_sha256=$GITLAB_KEY_SHA256
+EOF
+
 # set up GitLab
 export LC_ALL=C.UTF-8 # explicitly set UTF8 to avoid GitLab bitchin...
 
-service gitlab-runsvdir start
 gitlab-ctl reconfigure
 
 # disable GitLab WebUI prompting to set 'root' GitLab user password (TurnKey
@@ -92,6 +124,9 @@ EOF
 # fix occasional postgres stop timeout
 while ! gitlab-ctl stop postgresql | grep -q 'ok: down'; do sleep 1; done
 gitlab-ctl stop
+kill "$runsvdir_pid"
+wait "$runsvdir_pid" || true
+runsvdir_pid=
 
 # remove unneeded default TurnKey Confconsole Let's Encrypt integration files
 # GitLab Omnibus provides it's own LE integration - GitLab specific Confconsole
@@ -109,7 +144,6 @@ sed -i "/postgresql\['dynamic_shared_memory_type'\]/ s|= 'mmap'|= nil|" $CONF
 sed -i "/postgresql\['shared_buffers'\]/ s|^|# |" $CONF
 sed -i "/postgresql\['shared_buffers'\]/ s|=.*|= \"256MB\"|" $CONF
 
-service gitlab-runsvdir stop
 # ensure that the overlay turnkey service is the one that is enabled
 systemctl disable gitlab-runsvdir.service
 systemctl enable gitlab-runsvdir.service
diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index e0fb467..3e15aad 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -1,56 +1,86 @@
 # GitLab v19 acceptance
 
-## Source decision
+## Dependency source decision
 
-Debian Trixie does not package GitLab. The appliance installs GitLab CE
-19.3.0 from GitLab's official Debian 13 repository. GitLab supports its Linux
-package on Debian 13 from version 18.5 through Debian's expected June 2030 end
-of life. The build binds the repository key by SHA-256 and fingerprint, then
-APT verifies the signed repository metadata and the selected package digest.
+The appliance plan inherits its operating system packages from Debian Trixie.
+Debian does not package the GitLab application, so the application exception is
+GitLab CE 19.3.0 from GitLab's official Debian 13 repository. GitLab documents
+Debian 13 support from GitLab 18.5 through Debian's expected June 2030 end of
+life in its [Linux package supported platforms][supported-platforms].
 
-The selected package and evidence are:
+The exception is bounded as follows:
 
-- version `19.3.0-ce.0`;
-- repository key fingerprint
+- version `19.3.0-ce.0` is selected explicitly;
+- repository key fingerprint is
   `F6403F6544A38863DAA0B6E03F01618A51312F3F`;
-- repository key SHA-256
+- repository key SHA-256 is
   `003c0ca2fea61767f8c6de7a1c0f49fc88ea3c8db95e3cd1856b32ce9d876e0f`;
-- package SHA-256
-  `f88f80cd61d6b2beb35aa7207591d4abdfed0e6c2c42e6ed753dd29ea5de076d`.
+- package SHA-256 is
+  `f88f80cd61d6b2beb35aa7207591d4abdfed0e6c2c42e6ed753dd29ea5de076d`;
+- the build verifies the key bytes and primary fingerprint, reads the package
+  digest from APT's signed metadata, downloads through APT, and verifies the
+  downloaded Debian package again before unpacking it;
+- `gitlab-update --check` verifies the installed version, repository key,
+  Trixie channel, and eligible candidate without changing the appliance.
+
+GitLab's Debian post-install script configures the application immediately.
+The TKLDev build therefore stages the verified package with `dpkg --unpack`,
+writes the non-init chroot settings, starts GitLab's bundled runit, and lets APT
+perform the normal configure step. Runtime upgrades use the normal APT path
+under the running appliance service manager.
 
 ## README crosswalk
 
-| Contract | Focused acceptance | Required result |
+The focused command in every application-specific row is `tests/v19.sh`, run
+inside the configured Docker root by the exact acceptance command below.
+
+| README contract | Focused command and expected result | Actual result and retained evidence |
 | --- | --- | --- |
-| Root firstboot credentials | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint with the session cookie | The normal web login identifies `root` |
-| Project hosting | Create a private project through the authenticated API and read its committed file through GitLab HTTP | The project exists and its marker is rendered from the repository |
-| Git and SSH | Register a throwaway root SSH key, push a commit, then clone it into a fresh directory | The pushed marker survives an SSH Git round trip |
-| PostgreSQL and repository services | Read the project row with `gitlab-psql` and require PostgreSQL, Redis, Gitaly, Nginx, and Sidekiq status | The project metadata and GitLab service stack are healthy |
-| Background jobs | Require the Sidekiq supervisor process | Sidekiq reports a running process |
-| Outbound mail | Require the appliance Postfix unit | Postfix is enabled and active; public mail delivery is outside local acceptance |
-| Webmin Postfix module | Require the packaged module directory | The appliance-specific mail administration module is installed |
-| Supervised updates | Run `gitlab-update --check` | The installed version, signed Trixie candidate, and required-stop policy are reported without mutation |
-| Built-in Let's Encrypt integration | Confirm GitLab owns Nginx and require the appliance's GitLab-specific Confconsole plugin | Certificate issuance requires a public DNS name and is not invoked in local acceptance |
-| Inherited administration | Cite unchanged SSH, Webmin, and common platform behavior | Core 19 PASS run `20260824t010251z-1634-32241` at source `24c82ee3540ce545422742b0e28ba6b687c53ec2` remains applicable |
-
-The acceptance fixture revokes its token and SSH key and requests project
-deletion on exit. GitLab retains revoked-token audit records according to its
-normal product behavior.
-
-## Exact acceptance
+| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` at the configured domain. | Pending the final clean exact run. Loop 1 stopped during package configuration and did not reach runtime. |
+| Project hosting and Git repository readback | Create a private project, push a commit, clone it to a fresh directory, and read the committed file through GitLab HTTP. The marker survives every read path. | Pending the final clean exact run. |
+| Git over SSH | Register a throwaway Ed25519 key and use `git@127.0.0.1` for push and clone. Both operations succeed through GitLab Shell. | Pending the final clean exact run. |
+| Omnibus Nginx, PostgreSQL, Redis, Gitaly, and Sidekiq | Require every supervisor process, query the created project through `gitlab-psql`, enqueue `ProjectCacheWorker`, and wait for its uniquely keyed scheduled follow-up. The database row is exact and the background job completes without retry. | Pending the final clean exact run. |
+| Postfix and Webmin Postfix module | Require `postfix.service` active and enabled and the packaged Webmin module directory present. Local appliance integration passes; public delivery is outside local acceptance. | Pending the final clean exact run. |
+| Supervised manual updates | Run `gitlab-update --check`. It reports installed `19.3.0-ce.0`, the signed Trixie candidate, and the required-stop policy without mutation. | The disposable Trixie fixture reports candidate `19.3.1-ce.0` and `status=supervised-upgrade-available`; final appliance evidence is pending. |
+| GitLab-owned Nginx and Confconsole Let's Encrypt integration | Require GitLab Nginx running and the GitLab-specific `get_certificate.py` plugin present. Public issuance is not attempted without a public DNS fixture. | Pending the final clean exact run. |
+| Standard Core administration | Inherited SSH, Webmin, cron, Trixie identity, ISO, and normal init behavior are not customized by this application. | Core 19 PASS run `20260824t010251z-1634-32241`, source `24c82ee3540ce545422742b0e28ba6b687c53ec2`, report SHA-256 `1a6c2d266b6a2898e98a073421e49212dd0237598861200a3cbd6e81d35a2936`. |
+
+The fixture revokes its personal access token and SSH key, removes the scheduled
+test follow-up, requests project deletion, and deletes its local Git trees on
+exit. GitLab retains revoked-token audit records according to normal product
+behavior.
+
+## Commands, results, and evidence
+
+Static boundary checks pass for Bash syntax, Python compilation, ShellCheck,
+`git diff --check`, active old-suite detection, result-protocol fields, and the
+documentation ban on em dashes. A disposable Debian Trixie repository probe
+also verifies the selected key hash, fingerprint, signed package-metadata hash,
+and current update candidate.
+
+Product loop 1 used source commit
+`f6ea46a50f0781f4028717dc64b4f86c3408d475` in run
+`20260826t084052z-323768-26314`. The build reached GitLab's package
+auto-reconfigure, then its Cinc log stopped at `2026-08-26T08:49:19Z` after
+creating the logrotate runit supervision directories. The process chain was
+`apt-get install` to `gitlab-ctl upgrade` to `cinc-client`, and the log remained
+unchanged through `2026-08-26T09:09:48Z`. The run was stopped and retained as a
+build-stage FAIL. The correction moves the existing non-init configuration
+ahead of package configure and supplies the required runit supervisor.
+
+The final clean command is:
 
 ```sh
-TKLDEV_CONTAINER=tkldev19-wave2 \
-TKL_HARNESS_STATE_DIR=/home/agent/.local/state/turnkey-v19-harness-wave2 \
-TKL_HARNESS_LOCK_FILE=/home/agent/.local/state/turnkey-v19-harness-wave2/build.lock \
-/sandboxed-git/turnkey/tools/test-v19-appliance gitlab \
+TKLDEV_CONTAINER=tkldev19-wave2-2 \
+TKL_HARNESS_STATE_DIR=/home/agent/.local/state/turnkey-v19-harness-wave2-2 \
+TKL_HARNESS_LOCK_FILE=/home/agent/.local/state/turnkey-v19-harness-wave2-2/build.lock \
+/home/agent/turnkey/tools/test-v19-appliance gitlab \
   --source /home/agent/.local/worktrees/turnkey-apps/gitlab/wish-gitlab-v19-trixie
 ```
 
-Exact retained acceptance remains pending because the shared Wave 2 runner is
-missing its required `rg` command and has exhausted its Docker allocation.
-This infrastructure blocker has not exercised the GitLab candidate and costs
-zero appliance product loops.
+Final exact acceptance is pending while earlier queued wave2-2 work holds the
+assigned builder lock. Replace this statement with the PASS run ID, report
+path, exact source commit, and compact result before review handoff.
 
 ## Deferred issues
 
@@ -62,3 +92,5 @@ zero appliance product loops.
 - API project deletion can complete asynchronously. The test submits cleanup,
   but does not wait for GitLab's background deletion worker.
 - Docker acceptance does not repeat installer, kernel, or hardware checks.
+
+[supported-platforms]: https://docs.gitlab.com/install/package/#supported-platforms
diff --git a/overlay/usr/lib/inithooks/firstboot.d/40gitlab b/overlay/usr/lib/inithooks/firstboot.d/40gitlab
index 100b305..9023eb3 100755
--- a/overlay/usr/lib/inithooks/firstboot.d/40gitlab
+++ b/overlay/usr/lib/inithooks/firstboot.d/40gitlab
@@ -11,6 +11,7 @@ if ! systemctl is-active --quiet gitlab-runsvdir.service; then
 fi
 
 if [[ -f "$INITHOOKS_CONF" ]]; then
+    # shellcheck disable=SC1090
     source "$INITHOOKS_CONF"
 else
     echo "$(basename "$0"): Warning: $INITHOOKS_CONF not found or is not a file (expected if not preseeded" >&2
diff --git a/overlay/usr/local/sbin/gitlab-update b/overlay/usr/local/sbin/gitlab-update
index 57b6b26..c2d9b85 100755
--- a/overlay/usr/local/sbin/gitlab-update
+++ b/overlay/usr/local/sbin/gitlab-update
@@ -11,6 +11,8 @@ keyring=/usr/share/keyrings/gitlab-ce.gpg
 
 # shellcheck disable=SC1090
 . "$source_record"
+: "${installed_version:?installed_version is missing from $source_record}"
+: "${repository_key_fingerprint:?repository_key_fingerprint is missing from $source_record}"
 
 installed=$(dpkg-query -W -f='${Version}' gitlab-ce)
 candidate=$(apt-cache policy gitlab-ce |
diff --git a/tests/v19.sh b/tests/v19.sh
index fc6bc3d..da87f4a 100755
--- a/tests/v19.sh
+++ b/tests/v19.sh
@@ -58,6 +58,9 @@ test -f /usr/lib/confconsole/plugins.d/Lets_Encrypt/get_certificate.py
 
 # shellcheck disable=SC1090
 . "$source_file"
+: "${installed_version:?installed_version is missing from $source_file}"
+: "${package_sha256:?package_sha256 is missing from $source_file}"
+: "${repository_key_fingerprint:?repository_key_fingerprint is missing from $source_file}"
 test "$(dpkg-query -W -f='${Version}' gitlab-ce)" = "$installed_version"
 test "$installed_version" = 19.3.0-ce.0
 test "$package_sha256" = f88f80cd61d6b2beb35aa7207591d4abdfed0e6c2c42e6ed753dd29ea5de076d
@@ -121,6 +124,31 @@ gitlab-psql --no-align --tuples-only --command \
     "SELECT path FROM projects WHERE id = $project_id;" |
     grep -Fxq "$fixture"
 gitlab-ctl status sidekiq | grep -Fq 'run: sidekiq:'
+ruby=$work/background-job.rb
+cat >"$ruby" <<'RUBY'
+require 'sidekiq/api'
+project_id = Integer(ENV.fetch('TKL_PROJECT_ID'), 10)
+statistics = ['repository_size']
+lease_key = ['project_cache_worker', project_id, *statistics].join(':')
+jid = ProjectCacheWorker.perform_async(project_id, [], statistics)
+deadline = 90.seconds.from_now
+loop do
+  followup = Sidekiq::ScheduledSet.new.find do |job|
+    job.klass == 'UpdateProjectStatisticsWorker' &&
+      job.args[0] == lease_key && job.args[1] == project_id
+  end
+  if followup
+    followup.delete
+    puts "Sidekiq project cache round trip: #{jid}"
+    break
+  end
+  retry_job = Sidekiq::RetrySet.new.find_job(jid)
+  raise "ProjectCacheWorker entered retry: #{retry_job.error_message}" if retry_job
+  raise 'ProjectCacheWorker timed out' if Time.current >= deadline
+  sleep 1
+end
+RUBY
+TKL_PROJECT_ID=$project_id gitlab-rails runner "$ruby"
 
 gitlab-update --check >"$work/update"
 candidate=$(sed -n 's/^candidate=//p' "$work/update")
@@ -132,7 +160,7 @@ grep -Fxq "integrity=APT-signed-by-$repository_key_fingerprint" "$work/update"
 cat >"$result" <<EOF
 package_source=Official GitLab CE Debian Trixie repository
 installed_version=$installed_version
-runtime_checks=normal init; root web login; project API create and web read; SSH Git push and clone; PostgreSQL readback; Sidekiq; Postfix
+runtime_checks=normal init; root web login; project API create and web read; SSH Git push and clone; PostgreSQL readback; Sidekiq project cache job; Postfix
 updater_command=gitlab-update --check
 updater_result=$status; candidate=$candidate
 updater_channel=official GitLab CE Debian Trixie, supervised required-stop upgrades

From 21ad50fcec350f2c62bd4d6b304895af80d3169c Mon Sep 17 00:00:00 2001
From: Liraz Siri <liraz@liraz.org>
Date: Wed, 26 Aug 2026 10:03:34 +0000
Subject: [PATCH 03/12] Use GitLab PostgreSQL wrapper for secret cleanup

---
 conf.d/main           | 2 +-
 docs/v19.0-testing.md | 9 +++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/conf.d/main b/conf.d/main
index 42d341b..2b5312a 100755
--- a/conf.d/main
+++ b/conf.d/main
@@ -112,7 +112,7 @@ done
 # clear secrets (they will be regenerated automatically)
 gitlab-ctl start postgresql
 echo -e 'ApplicationSetting.first.delete\n' | gitlab-rails console -e production
-cat <<EOF | gitlab-rails dbconsole --database main
+cat <<EOF | gitlab-psql
 UPDATE projects SET runners_token = null, runners_token_encrypted = null;
 UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
 UPDATE application_settings SET runners_registration_token_encrypted = null;
diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index 3e15aad..22b0815 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -68,6 +68,15 @@ unchanged through `2026-08-26T09:09:48Z`. The run was stopped and retained as a
 build-stage FAIL. The correction moves the existing non-init configuration
 ahead of package configure and supplies the required runit supervisor.
 
+Product loop 2 used source commit
+`9ca6dc9bb57c6e06c11a555b88ed7e9da76f98f7` in run
+`20260826t093948z-997460-8925`. It passed the former runit hang and completed
+GitLab package configuration. The build then stopped while clearing generated
+secrets because GitLab 19.3's Rails dbconsole passed an empty
+`statement_timeout` to PostgreSQL. Earlier `gitlab-psql` calls against the same
+main database succeeded. The correction sends the existing SQL block through
+that direct Omnibus PostgreSQL wrapper.
+
 The final clean command is:
 
 ```sh

From 9c6323064650909dc87d54975e5a4b99f1ad0656 Mon Sep 17 00:00:00 2001
From: Liraz Siri <liraz@liraz.org>
Date: Wed, 26 Aug 2026 10:30:55 +0000
Subject: [PATCH 04/12] Adapt GitLab secret cleanup to v19 schema

---
 conf.d/main           | 22 ++++++++++++++++------
 docs/v19.0-testing.md | 10 ++++++++++
 2 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/conf.d/main b/conf.d/main
index 2b5312a..5fbe805 100755
--- a/conf.d/main
+++ b/conf.d/main
@@ -112,18 +112,28 @@ done
 # clear secrets (they will be regenerated automatically)
 gitlab-ctl start postgresql
 echo -e 'ApplicationSetting.first.delete\n' | gitlab-rails console -e production
-cat <<EOF | gitlab-psql
+cat <<EOF | gitlab-psql --set=ON_ERROR_STOP=on
 UPDATE projects SET runners_token = null, runners_token_encrypted = null;
 UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
 UPDATE application_settings SET runners_registration_token_encrypted = null;
 UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
-UPDATE ci_runners SET token = null, token_encrypted = null;
-TRUNCATE integrations, chat_names, issue_tracker_data, jira_tracker_data, slack_integrations, web_hooks, zentao_tracker_data, web_hook_logs, dingtalk_tracker_data, slack_integrations_scopes;
+UPDATE ci_runners SET token_encrypted = null;
+TRUNCATE integrations, chat_names, issue_tracker_data, jira_tracker_data, slack_integrations, web_hooks, zentao_tracker_data, dingtalk_tracker_data, slack_integrations_scopes;
 EOF
 
-# fix occasional postgres stop timeout
-while ! gitlab-ctl stop postgresql | grep -q 'ok: down'; do sleep 1; done
-gitlab-ctl stop
+# GitLab services occasionally need a second stop wait while shutting down in
+# the build chroot. Keep retries bounded and preserve a terminal failure.
+stop_gitlab() {
+    for _ in 1 2 3; do
+        if gitlab-ctl stop "$@"; then
+            return 0
+        fi
+        sleep 2
+    done
+    return 1
+}
+stop_gitlab postgresql
+stop_gitlab
 kill "$runsvdir_pid"
 wait "$runsvdir_pid" || true
 runsvdir_pid=
diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index 22b0815..a0ab67d 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -77,6 +77,16 @@ secrets because GitLab 19.3's Rails dbconsole passed an empty
 main database succeeded. The correction sends the existing SQL block through
 that direct Omnibus PostgreSQL wrapper.
 
+Product loop 3 used source commit
+`21ad50fcec350f2c62bd4d6b304895af80d3169c` in run
+`20260826t100426z-1257552-1524`. It passed both earlier blockers and connected
+with `gitlab-psql`. The scrub then found that GitLab 19 no longer has the
+`ci_runners.token` column or `web_hook_logs` relation. The final service stop
+also returned a timeout while Puma was still exiting. GitLab's v19.3 schema
+confirms that `ci_runners.token_encrypted` and every retained truncate target
+exist. The correction removes the obsolete targets, makes the SQL fail fast,
+and retries service shutdown three times before returning a terminal failure.
+
 The final clean command is:
 
 ```sh

From e2286dbca78775ed877f2c1539ed8a5e1632d918 Mon Sep 17 00:00:00 2001
From: Liraz Siri <liraz@liraz.org>
Date: Wed, 26 Aug 2026 11:21:06 +0000
Subject: [PATCH 05/12] Wait for GitLab readiness before acceptance

---
 docs/v19.0-testing.md | 14 ++++++++++++++
 tests/v19.sh          | 13 +++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index a0ab67d..3ab6577 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -87,12 +87,25 @@ confirms that `ci_runners.token_encrypted` and every retained truncate target
 exist. The correction removes the obsolete targets, makes the SQL fail fast,
 and retries service shutdown three times before returning a terminal failure.
 
+Product loop 4 used source commit
+`9c6323064650909dc87d54975e5a4b99f1ad0656` in run
+`20260826t104919z-1723156-31927`. The build completed, including the fail-fast
+v19 scrub and the bounded service stop, and firstboot completed normally. The
+focused test's first UI request then received HTTP 502 while GitLab was still
+settling after firstboot restarted Puma and Workhorse. GitLab documents that
+its [readiness probe][health-check] holds traffic until Rails and dependent
+services are ready, including the brief Puma restart window. The correction
+polls that probe with a five-minute bound before attempting the normal sign-in
+flow. Extracted-function fixtures pass for immediate success, success on the
+third probe, and exhaustion after exactly 61 probes and 60 two-second sleeps.
+
 The final clean command is:
 
 ```sh
 TKLDEV_CONTAINER=tkldev19-wave2-2 \
 TKL_HARNESS_STATE_DIR=/home/agent/.local/state/turnkey-v19-harness-wave2-2 \
 TKL_HARNESS_LOCK_FILE=/home/agent/.local/state/turnkey-v19-harness-wave2-2/build.lock \
+TKL_HARNESS_DOCKER_LIMIT_BYTES=137438953472 \
 /home/agent/turnkey/tools/test-v19-appliance gitlab \
   --source /home/agent/.local/worktrees/turnkey-apps/gitlab/wish-gitlab-v19-trixie
 ```
@@ -113,3 +126,4 @@ path, exact source commit, and compact result before review handoff.
 - Docker acceptance does not repeat installer, kernel, or hardware checks.
 
 [supported-platforms]: https://docs.gitlab.com/install/package/#supported-platforms
+[health-check]: https://docs.gitlab.com/administration/monitoring/health_check/
diff --git a/tests/v19.sh b/tests/v19.sh
index da87f4a..47730a3 100755
--- a/tests/v19.sh
+++ b/tests/v19.sh
@@ -25,6 +25,18 @@ else
 fi
 curl_local=(curl --insecure --silent --show-error --resolve "$host:$port:127.0.0.1")
 
+wait_gitlab_ready() {
+    for _ in {1..60}; do
+        if "${curl_local[@]}" --fail --connect-timeout 1 --max-time 3 \
+                --output /dev/null "$base/-/readiness?all=1" 2>/dev/null; then
+            return 0
+        fi
+        sleep 2
+    done
+    "${curl_local[@]}" --fail --connect-timeout 1 --max-time 3 \
+        --output /dev/null "$base/-/readiness?all=1"
+}
+
 cleanup() {
     set +e
     if [[ -n $key_id ]]; then
@@ -68,6 +80,7 @@ test "$(gpg --show-keys --with-colons /usr/share/keyrings/gitlab-ce.gpg |
     awk -F: '$1 == "fpr" { print $10; exit }')" = \
     "$repository_key_fingerprint"
 
+wait_gitlab_ready
 "${curl_local[@]}" --fail --cookie-jar "$cookie" \
     "$base/users/sign_in" >"$page"
 csrf=$(sed -n 's/.*name="authenticity_token" value="\([^"]*\)".*/\1/p' \

From 40267755e0a037c5f25a2517b009ad1b0e26e667 Mon Sep 17 00:00:00 2001
From: Codex <codex@openai.com>
Date: Wed, 26 Aug 2026 13:22:28 +0000
Subject: [PATCH 06/12] Use inherited Python for GitLab JSON checks

Keep appliance packages unchanged by replacing the test-only jq assumption with typed JSON extraction through Debian Python inherited from Core. Record the retained runtime-inspect retry and fail-closed fixture evidence.
---
 docs/v19.0-testing.md | 36 +++++++++++++++++++++++++++++++++---
 tests/v19.sh          | 32 +++++++++++++++++++++++++++-----
 2 files changed, 60 insertions(+), 8 deletions(-)

diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index 3ab6577..5ecfcf6 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -58,6 +58,13 @@ documentation ban on em dashes. A disposable Debian Trixie repository probe
 also verifies the selected key hash, fingerprint, signed package-metadata hash,
 and current update candidate.
 
+For the JSON instrumentation correction, the retained loop-5 retry build log
+records `python3-minimal` and `python3` version `3.13.5-1` installed from
+Debian Trixie. A disposable Trixie probe confirms the same signed Debian
+candidate, runs ShellCheck, and passes the exact extracted JSON helper for
+valid string and integer values. Wrong types, a boolean in place of an integer,
+a missing key, and malformed JSON all fail closed.
+
 Product loop 1 used source commit
 `f6ea46a50f0781f4028717dc64b4f86c3408d475` in run
 `20260826t084052z-323768-26314`. The build reached GitLab's package
@@ -99,6 +106,28 @@ polls that probe with a five-minute bound before attempting the normal sign-in
 flow. Extracted-function fixtures pass for immediate success, success on the
 third probe, and exhaustion after exactly 61 probes and 60 two-second sleeps.
 
+Product loop 5 used source commit
+`e2286dbca78775ed877f2c1539ed8a5e1632d918` in run
+`20260826t113407z-2022901-16188`. The build and image import completed,
+including the fail-fast v19 scrub and bounded service stop. The harness then
+reported that the runtime container stopped before reaching health. Docker's
+event record shows the container remained alive through repeated multi-user
+health polls, with no preceding `die` event, until harness cleanup sent signal
+9 at nanosecond timestamp `1787746535441440994`; the resulting exit was 137.
+The harness treated a transient or empty `docker inspect` result as a stopped
+container. GitLab readiness and the focused test were not reached, so this run
+provides build and import evidence but no application acceptance evidence.
+
+The infrastructure retry of product loop 5 used the same source commit with
+reviewed harness commit `5746bdca17347598f92767d1a9c76b6406a160d4` in run
+`20260826t124900z-2678316-2241`. Build, import, runtime health, GitLab
+readiness, and root web login passed. The test then stopped at its first JSON
+assertion because `jq` was a test-only assumption and is not an appliance
+dependency. The correction uses Debian Python already inherited through Core,
+preserves string and integer type checks, and does not change appliance
+packages or runtime behavior. This is test instrumentation, not a product
+loop.
+
 The final clean command is:
 
 ```sh
@@ -110,9 +139,10 @@ TKL_HARNESS_DOCKER_LIMIT_BYTES=137438953472 \
   --source /home/agent/.local/worktrees/turnkey-apps/gitlab/wish-gitlab-v19-trixie
 ```
 
-Final exact acceptance is pending while earlier queued wave2-2 work holds the
-assigned builder lock. Replace this statement with the PASS run ID, report
-path, exact source commit, and compact result before review handoff.
+Final exact acceptance is pending a clean retry of the corrected test
+instrumentation behind queued wave2-2 work. Replace this statement with the
+PASS run ID, report path, exact source commit, and compact result before review
+handoff.
 
 ## Deferred issues
 
diff --git a/tests/v19.sh b/tests/v19.sh
index 47730a3..bdf4614 100755
--- a/tests/v19.sh
+++ b/tests/v19.sh
@@ -37,6 +37,28 @@ wait_gitlab_ready() {
         --output /dev/null "$base/-/readiness?all=1"
 }
 
+json_value() {
+    local expected_type=$1
+    local key=$2
+
+    python3 -c '
+import json
+import sys
+
+expected_type, key = sys.argv[1:]
+value = json.load(sys.stdin)[key]
+if expected_type == "string":
+    valid = isinstance(value, str)
+elif expected_type == "integer":
+    valid = isinstance(value, int) and not isinstance(value, bool)
+else:
+    raise SystemExit(2)
+if not valid:
+    raise SystemExit(1)
+sys.stdout.write(str(value))
+' "$expected_type" "$key"
+}
+
 cleanup() {
     set +e
     if [[ -n $key_id ]]; then
@@ -93,8 +115,8 @@ test -n "$csrf"
     --data-urlencode "user[password]=$app_password" \
     --data-urlencode 'user[remember_me]=0' \
     "$base/users/sign_in" >"$page"
-"${curl_local[@]}" --fail --cookie "$cookie" \
-    "$base/api/v4/user" | jq -e '.username == "root"' >/dev/null
+test "$("${curl_local[@]}" --fail --cookie "$cookie" \
+    "$base/api/v4/user" | json_value string username)" = root
 
 gitlab-rails runner \
     "item = User.find_by_username('root').personal_access_tokens.create!(scopes: ['api'], name: '$fixture', expires_at: 1.day.from_now); item.set_token('$token'); item.save!"
@@ -105,8 +127,8 @@ project=$("${curl_local[@]}" --fail --request POST \
     --data-urlencode "path=$fixture" \
     --data 'visibility=private' \
     "$base/api/v4/projects")
-project_id=$(jq -er '.id' <<<"$project")
-test "$(jq -r '.path' <<<"$project")" = "$fixture"
+project_id=$(json_value integer id <<<"$project")
+test "$(json_value string path <<<"$project")" = "$fixture"
 
 ssh-keygen -q -t ed25519 -N '' -f "$work/id"
 key=$("${curl_local[@]}" --fail --request POST \
@@ -114,7 +136,7 @@ key=$("${curl_local[@]}" --fail --request POST \
     --data-urlencode "title=$fixture" \
     --data-urlencode "key=$(<"$work/id.pub")" \
     "$base/api/v4/user/keys")
-key_id=$(jq -er '.id' <<<"$key")
+key_id=$(json_value integer id <<<"$key")
 ssh-keyscan -T 10 127.0.0.1 >"$work/known_hosts" 2>/dev/null
 export GIT_SSH_COMMAND="ssh -i $work/id -o IdentitiesOnly=yes -o UserKnownHostsFile=$work/known_hosts"
 

From dce631550cd32d6d45fd59571f654dc05408da97 Mon Sep 17 00:00:00 2001
From: Codex <codex@openai.com>
Date: Wed, 26 Aug 2026 18:49:39 +0000
Subject: [PATCH 07/12] Authenticate GitLab raw reads with the web session

Use the established login cookie for private web raw-file readback because GitLab does not accept PRIVATE-TOKEN on the blob web route. Add phase-aware failures to every focused assertion and record the official v19.3 source and live response A/B evidence.
---
 docs/v19.0-testing.md | 59 +++++++++++++++++++++++++++++++++++++-
 tests/v19.sh          | 66 ++++++++++++++++++++++++++++++++++++-------
 2 files changed, 114 insertions(+), 11 deletions(-)

diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index 5ecfcf6..8854209 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -128,6 +128,57 @@ preserves string and integer type checks, and does not change appliance
 packages or runtime behavior. This is test instrumentation, not a product
 loop.
 
+The scoped infrastructure retry used source commit
+`40267755e0a037c5f25a2517b009ad1b0e26e667`, pinned harness commit
+`152bc9b876557266b90ed4ded77b611d9b817aff`, and scoped label
+`org.turnkeylinux.v19-harness.wave2-2` in run
+`20260826t181108z-156555-30392`. The exact rootfs build, configured-root
+fallback, scoped image import, runtime health, GitLab readiness, root login,
+typed JSON checks, private project creation, and both SSH Git operations
+passed. The focused test then exited after the SSH operations without a child
+result or assertion message. The retained report is
+`/home/agent/.local/state/turnkey-v19-harness-wave2-2/failures/gitlab/current/report.txt`,
+with SHA-256
+`739f72a48e3f4b0388fb0ceb1ca43250f466ca12a2b2244ec6553312c7574ade`.
+
+The next assertion used a personal access token on the non-API web raw route.
+GitLab 19.3's [raw controller][raw-controller] requests `:blob` sessionless
+authentication. Its [request authenticator][request-authenticator] tries web
+personal access tokens and then static-object tokens, but the
+[authentication finder][auth-finders] does not accept `:blob` as a personal
+access token web format. The documented personal access token contract is the
+[repository files API raw route][repository-files-api]. GitLab also documents
+that normal web sign-in sets the `_gitlab_session` cookie used by its web
+frontend in the [REST authentication guide][rest-authentication].
+
+A disposable rootless probe used the official GitLab CE 19.3.0 container image
+with digest
+`sha256:f7e453ff51d1910235365085fe836e4589716d26b44d99a8aa3e2c41377f034f`.
+It created a private project and file, then recorded this response A/B:
+
+```text
+PRIVATE-TOKEN, /root/.../-/raw/main/README.md:
+HTTP/1.1 302 Found
+Location: http://gitlab-probe.invalid:18081/users/sign_in
+
+_gitlab_session cookie, /root/.../-/raw/main/README.md:
+HTTP/1.1 200 OK
+GitLab v19 private raw authentication probe
+
+PRIVATE-TOKEN, /api/v4/projects/1/repository/files/README.md/raw?ref=main:
+HTTP/1.1 200 OK
+GitLab v19 private raw authentication probe
+```
+
+This directly confirms a test-only authentication mismatch. The focused test
+now reuses its signed-in session cookie on the web route. Named phase markers,
+an error trap with the failing line, and explicit messages on every `grep`
+assertion prevent another assertion from exiting silently. The disposable
+probe container and image were removed after capture. An extracted marker
+matrix confirms that a passing assertion reports its phase, an unannotated
+failure reports phase, status, and line, and an explicit assertion failure
+reports its phase and message. The three cases return status 0, 1, and 1.
+
 The final clean command is:
 
 ```sh
@@ -135,7 +186,8 @@ TKLDEV_CONTAINER=tkldev19-wave2-2 \
 TKL_HARNESS_STATE_DIR=/home/agent/.local/state/turnkey-v19-harness-wave2-2 \
 TKL_HARNESS_LOCK_FILE=/home/agent/.local/state/turnkey-v19-harness-wave2-2/build.lock \
 TKL_HARNESS_DOCKER_LIMIT_BYTES=137438953472 \
-/home/agent/turnkey/tools/test-v19-appliance gitlab \
+TKL_HARNESS_DOCKER_OBJECT_LABEL=org.turnkeylinux.v19-harness.wave2-2 \
+/home/agent/.local/worktrees/turnkey/harness-wave2-2-152bc9b/tools/test-v19-appliance gitlab \
   --source /home/agent/.local/worktrees/turnkey-apps/gitlab/wish-gitlab-v19-trixie
 ```
 
@@ -157,3 +209,8 @@ handoff.
 
 [supported-platforms]: https://docs.gitlab.com/install/package/#supported-platforms
 [health-check]: https://docs.gitlab.com/administration/monitoring/health_check/
+[raw-controller]: https://gitlab.com/gitlab-org/gitlab/-/blob/v19.3.0-ee/app/controllers/projects/raw_controller.rb
+[request-authenticator]: https://gitlab.com/gitlab-org/gitlab/-/blob/v19.3.0-ee/lib/gitlab/auth/request_authenticator.rb
+[auth-finders]: https://gitlab.com/gitlab-org/gitlab/-/blob/v19.3.0-ee/lib/gitlab/auth/auth_finders.rb
+[repository-files-api]: https://docs.gitlab.com/api/repository_files/#retrieve-a-raw-file-from-a-repository
+[rest-authentication]: https://docs.gitlab.com/api/rest/authentication/#session-cookie
diff --git a/tests/v19.sh b/tests/v19.sh
index bdf4614..a8882de 100755
--- a/tests/v19.sh
+++ b/tests/v19.sh
@@ -12,6 +12,29 @@ cookie=$work/cookie
 page=$work/page
 project_id=
 key_id=
+phase=initialization
+
+mark_phase() {
+    phase=$1
+    printf 'phase=%s\n' "$phase"
+}
+
+fail() {
+    printf 'error phase=%s message=%s\n' "$phase" "$1" >&2
+    exit 1
+}
+
+report_error() {
+    local status=$1
+    local line=$2
+
+    trap - ERR
+    printf 'error phase=%s status=%s line=%s\n' \
+        "$phase" "$status" "$line" >&2
+    exit "$status"
+}
+
+trap 'report_error "$?" "$LINENO"' ERR
 
 base=$(sed -n "s/^external_url '\([^']*\)'.*/\1/p" /etc/gitlab/gitlab.rb)
 scheme=${base%%://*}
@@ -61,6 +84,7 @@ sys.stdout.write(str(value))
 
 cleanup() {
     set +e
+    trap - ERR
     if [[ -n $key_id ]]; then
         "${curl_local[@]}" --request DELETE \
             --header "PRIVATE-TOKEN: $token" \
@@ -78,15 +102,19 @@ cleanup() {
 }
 trap cleanup EXIT
 
+mark_phase appliance-contract
 for unit in gitlab-runsvdir.service postfix.service; do
     systemctl --quiet is-active "$unit"
     systemctl --quiet is-enabled "$unit"
 done
 for component in nginx postgresql redis sidekiq gitaly; do
-    gitlab-ctl status "$component" | grep -Fq "run: $component:"
+    gitlab-ctl status "$component" | grep -Fq "run: $component:" ||
+        fail "GitLab component is not running: $component"
 done
-grep -Fxq 'VERSION_CODENAME=trixie' /etc/os-release
-grep -Eq '^turnkey-gitlab-19\.0' /etc/turnkey_version
+grep -Fxq 'VERSION_CODENAME=trixie' /etc/os-release ||
+    fail 'operating system is not Debian Trixie'
+grep -Eq '^turnkey-gitlab-19\.0' /etc/turnkey_version ||
+    fail 'appliance version is not GitLab 19.0'
 test -d /usr/share/webmin/postfix
 test -f /usr/lib/confconsole/plugins.d/Lets_Encrypt/get_certificate.py
 
@@ -102,7 +130,9 @@ test "$(gpg --show-keys --with-colons /usr/share/keyrings/gitlab-ce.gpg |
     awk -F: '$1 == "fpr" { print $10; exit }')" = \
     "$repository_key_fingerprint"
 
+mark_phase readiness
 wait_gitlab_ready
+mark_phase web-login
 "${curl_local[@]}" --fail --cookie-jar "$cookie" \
     "$base/users/sign_in" >"$page"
 csrf=$(sed -n 's/.*name="authenticity_token" value="\([^"]*\)".*/\1/p' \
@@ -118,6 +148,7 @@ test -n "$csrf"
 test "$("${curl_local[@]}" --fail --cookie "$cookie" \
     "$base/api/v4/user" | json_value string username)" = root
 
+mark_phase project-api
 gitlab-rails runner \
     "item = User.find_by_username('root').personal_access_tokens.create!(scopes: ['api'], name: '$fixture', expires_at: 1.day.from_now); item.set_token('$token'); item.save!"
 
@@ -140,6 +171,7 @@ key_id=$(json_value integer id <<<"$key")
 ssh-keyscan -T 10 127.0.0.1 >"$work/known_hosts" 2>/dev/null
 export GIT_SSH_COMMAND="ssh -i $work/id -o IdentitiesOnly=yes -o UserKnownHostsFile=$work/known_hosts"
 
+mark_phase ssh-git-round-trip
 git -C "$work" init -q repository
 git -C "$work/repository" config user.name 'TurnKey acceptance'
 git -C "$work/repository" config user.email 'acceptance@example.invalid'
@@ -150,15 +182,24 @@ git -C "$work/repository" remote add origin \
     "git@127.0.0.1:root/$fixture.git"
 git -C "$work/repository" push -q -u origin HEAD:main
 git clone -q "git@127.0.0.1:root/$fixture.git" "$work/readback"
-grep -Fxq 'GitLab v19 project round trip' "$work/readback/README.md"
+grep -Fxq 'GitLab v19 project round trip' "$work/readback/README.md" ||
+    fail 'cloned repository does not contain the pushed marker'
 
-"${curl_local[@]}" --fail --header "PRIVATE-TOKEN: $token" \
+mark_phase authenticated-web-read
+"${curl_local[@]}" --fail --location --cookie "$cookie" \
     "$base/root/$fixture/-/raw/main/README.md" |
-    grep -Fxq 'GitLab v19 project round trip'
+    grep -Fxq 'GitLab v19 project round trip' ||
+    fail 'authenticated web raw read does not contain the pushed marker'
+
+mark_phase database-readback
 gitlab-psql --no-align --tuples-only --command \
     "SELECT path FROM projects WHERE id = $project_id;" |
-    grep -Fxq "$fixture"
-gitlab-ctl status sidekiq | grep -Fq 'run: sidekiq:'
+    grep -Fxq "$fixture" ||
+    fail 'PostgreSQL project readback does not match the fixture'
+
+mark_phase background-jobs
+gitlab-ctl status sidekiq | grep -Fq 'run: sidekiq:' ||
+    fail 'Sidekiq is not running before the job round trip'
 ruby=$work/background-job.rb
 cat >"$ruby" <<'RUBY'
 require 'sidekiq/api'
@@ -185,13 +226,17 @@ end
 RUBY
 TKL_PROJECT_ID=$project_id gitlab-rails runner "$ruby"
 
+mark_phase package-update-channel
 gitlab-update --check >"$work/update"
 candidate=$(sed -n 's/^candidate=//p' "$work/update")
 status=$(sed -n 's/^status=//p' "$work/update")
 test -n "$candidate"
-grep -Fxq 'channel=official-gitlab-ce-debian-trixie' "$work/update"
-grep -Fxq "integrity=APT-signed-by-$repository_key_fingerprint" "$work/update"
+grep -Fxq 'channel=official-gitlab-ce-debian-trixie' "$work/update" ||
+    fail 'updater did not report the official GitLab CE Debian Trixie channel'
+grep -Fxq "integrity=APT-signed-by-$repository_key_fingerprint" "$work/update" ||
+    fail 'updater did not report the pinned repository key fingerprint'
 
+mark_phase result
 cat >"$result" <<EOF
 package_source=Official GitLab CE Debian Trixie repository
 installed_version=$installed_version
@@ -201,3 +246,4 @@ updater_result=$status; candidate=$candidate
 updater_channel=official GitLab CE Debian Trixie, supervised required-stop upgrades
 integrity_evidence=repository key $repository_key_fingerprint; package SHA-256 $package_sha256
 EOF
+mark_phase complete

From adbe2afbf46d2bffc3205451a17468ea86060f5a Mon Sep 17 00:00:00 2001
From: Codex <codex@openai.com>
Date: Wed, 26 Aug 2026 19:45:33 +0000
Subject: [PATCH 08/12] Make the GitLab job probe service-readable

Keep credentials and Git fixtures under the root-only directory while placing only the generated Sidekiq probe in an exclusive root-owned, git-group-readable temporary file. Remove that file during cleanup and record the retained run and direct gitlab-rails privilege-drop matrix.
---
 docs/v19.0-testing.md | 50 +++++++++++++++++++++++++++++++++++++++++++
 tests/v19.sh          |  8 ++++++-
 2 files changed, 57 insertions(+), 1 deletion(-)

diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index 8854209..b85f7e8 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -179,6 +179,56 @@ matrix confirms that a passing assertion reports its phase, an unannotated
 failure reports phase, status, and line, and an explicit assertion failure
 reports its phase and message. The three cases return status 0, 1, and 1.
 
+The second scoped infrastructure retry used source commit
+`dce631550cd32d6d45fd59571f654dc05408da97` and the same pinned harness in
+run `20260826t190222z-347388-9218`. Build, configured-root fallback, image
+import, runtime health, readiness, login, typed JSON checks, the private
+project and SSH Git round trip, authenticated web read, and database readback
+passed. The background-job phase then failed at line 227 because the test
+wrote `background-job.rb` below its root-owned mode-700 private fixture
+directory. `gitlab-rails runner` could not traverse that directory after its
+normal privilege drop. Cleanup, identity, and disk checks passed. The retained
+source archive SHA-256 is
+`3c4172f3b0046d335a975232a1c873f68b1683545fc47047b34d9e05dcd0b91d`,
+the input-tree SHA-256 is
+`042c8fdd27ae27520ffe5216a9619f7b833382f5f3304eb42481d03d7661a19f`,
+and the build-tree SHA-256 is
+`932d9ae36483a530798f22110e84704e4e4ef4b26c2097fb46a8d57d7c44a95f`.
+The retained report at
+`/home/agent/.local/state/turnkey-v19-harness-wave2-2/failures/gitlab/current/report.txt`
+has SHA-256
+`3dbbd281a2b243550eb09a6a63143a62a8d8524386e0873c0d12320d8880fab5`
+and the retained run log has SHA-256
+`e5ed10fdaeb4936d970ed51118a37747e748f88252a0925b3a842b7e88dd76d1`.
+The retained `SHA256SUMS` and `RETAINED-SHA256SUMS` files have SHA-256
+`8d8c4b64b171197754dbc007cfc42271308f41ae239bd99c7266ad8a6acdcdf8`
+and `6a84f16f7ed12ee71a3e98d12a06090e2eb9f4b4d26a8323ff414cb4e794487b`.
+
+A disposable probe used the same official GitLab CE 19.3.0 image digest as
+the raw-authentication probe. Its `/opt/gitlab/bin/gitlab-rails` wrapper has
+SHA-256
+`fab022de950858249d0ef68c02b443ef00c026b0d4b3d93c4c65c5dac7814fd9`
+and drops to the configured `git:git` service identity. An exact
+`gitlab-rails runner` matrix recorded:
+
+```text
+root:root mode 0700 parent, root:root mode 0644 script:
+status=1
+The file .../background-job.rb could not be found, please check and try again.
+
+root:git mode 0640 script directly under /tmp:
+status=0
+gitlab-rails runner user: 998:998
+```
+
+The correction creates only the generated background-job script as an
+exclusive `mktemp` file directly under `/tmp`, then grants read access to the
+`git` group with mode 0640. The script contains no credential or secret, the
+project identifier remains an environment variable, and cleanup removes the
+script. Cookies, the personal access token, the SSH private key, and Git trees
+remain below the original mode-700 directory. This is another test-only
+permission correction, so the product ledger remains at 5 of 6 loops.
+
 The final clean command is:
 
 ```sh
diff --git a/tests/v19.sh b/tests/v19.sh
index a8882de..3abafd9 100755
--- a/tests/v19.sh
+++ b/tests/v19.sh
@@ -12,6 +12,7 @@ cookie=$work/cookie
 page=$work/page
 project_id=
 key_id=
+ruby=
 phase=initialization
 
 mark_phase() {
@@ -98,6 +99,9 @@ cleanup() {
     gitlab-rails runner \
         "item = PersonalAccessToken.find_by_token('$token'); item.revoke! if item" \
         >/dev/null 2>&1
+    if [[ $ruby == /tmp/gitlab-v19-background-job.*.rb ]]; then
+        rm -f -- "$ruby"
+    fi
     find "$work" -depth -delete
 }
 trap cleanup EXIT
@@ -200,7 +204,7 @@ gitlab-psql --no-align --tuples-only --command \
 mark_phase background-jobs
 gitlab-ctl status sidekiq | grep -Fq 'run: sidekiq:' ||
     fail 'Sidekiq is not running before the job round trip'
-ruby=$work/background-job.rb
+ruby=$(mktemp /tmp/gitlab-v19-background-job.XXXXXXXX.rb)
 cat >"$ruby" <<'RUBY'
 require 'sidekiq/api'
 project_id = Integer(ENV.fetch('TKL_PROJECT_ID'), 10)
@@ -224,6 +228,8 @@ loop do
   sleep 1
 end
 RUBY
+chgrp git "$ruby"
+chmod 0640 "$ruby"
 TKL_PROJECT_ID=$project_id gitlab-rails runner "$ruby"
 
 mark_phase package-update-channel

From 39a65b0535c2a4af5e720e8540cf677162e706e6 Mon Sep 17 00:00:00 2001
From: Codex <codex@openai.com>
Date: Wed, 26 Aug 2026 20:31:25 +0000
Subject: [PATCH 09/12] Record the passing GitLab v19 acceptance

Map README claims to the retained scoped run results and capture source, harness, stage, updater, Sidekiq, cleanup, and artifact hashes.
---
 docs/v19.0-testing.md | 63 ++++++++++++++++++++++++++++++++++---------
 1 file changed, 50 insertions(+), 13 deletions(-)

diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index b85f7e8..a3f6b7d 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -33,16 +33,19 @@ under the running appliance service manager.
 
 The focused command in every application-specific row is `tests/v19.sh`, run
 inside the configured Docker root by the exact acceptance command below.
+Unless noted otherwise, PASS results refer to scoped exact run
+`20260826t194621z-508720-29542` and its report with SHA-256
+`4414e9e1ba4f188e6f7614e8748a3f942c30450beebd518cee02ba2e53abc1b0`.
 
 | README contract | Focused command and expected result | Actual result and retained evidence |
 | --- | --- | --- |
-| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` at the configured domain. | Pending the final clean exact run. Loop 1 stopped during package configuration and did not reach runtime. |
-| Project hosting and Git repository readback | Create a private project, push a commit, clone it to a fresh directory, and read the committed file through GitLab HTTP. The marker survives every read path. | Pending the final clean exact run. |
-| Git over SSH | Register a throwaway Ed25519 key and use `git@127.0.0.1` for push and clone. Both operations succeed through GitLab Shell. | Pending the final clean exact run. |
-| Omnibus Nginx, PostgreSQL, Redis, Gitaly, and Sidekiq | Require every supervisor process, query the created project through `gitlab-psql`, enqueue `ProjectCacheWorker`, and wait for its uniquely keyed scheduled follow-up. The database row is exact and the background job completes without retry. | Pending the final clean exact run. |
-| Postfix and Webmin Postfix module | Require `postfix.service` active and enabled and the packaged Webmin module directory present. Local appliance integration passes; public delivery is outside local acceptance. | Pending the final clean exact run. |
-| Supervised manual updates | Run `gitlab-update --check`. It reports installed `19.3.0-ce.0`, the signed Trixie candidate, and the required-stop policy without mutation. | The disposable Trixie fixture reports candidate `19.3.1-ce.0` and `status=supervised-upgrade-available`; final appliance evidence is pending. |
-| GitLab-owned Nginx and Confconsole Let's Encrypt integration | Require GitLab Nginx running and the GitLab-specific `get_certificate.py` plugin present. Public issuance is not attempted without a public DNS fixture. | Pending the final clean exact run. |
+| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` at the configured domain. | PASS: `web-login` authenticated the configured root account and its API identity. |
+| Project hosting and Git repository readback | Create a private project, push a commit, clone it to a fresh directory, and read the committed file through GitLab HTTP. The marker survives every read path. | PASS: `project-api`, `ssh-git-round-trip`, and `authenticated-web-read` preserved the same private-project marker. |
+| Git over SSH | Register a throwaway Ed25519 key and use `git@127.0.0.1` for push and clone. Both operations succeed through GitLab Shell. | PASS: the throwaway key pushed and cloned through GitLab Shell. |
+| Omnibus Nginx, PostgreSQL, Redis, Gitaly, and Sidekiq | Require every supervisor process, query the created project through `gitlab-psql`, enqueue `ProjectCacheWorker`, and wait for its uniquely keyed scheduled follow-up. The database row is exact and the background job completes without retry. | PASS: service checks, `database-readback`, and `background-jobs` completed. Sidekiq returned marker `fccc077c7203d30d44ba9566`. |
+| Postfix and Webmin Postfix module | Require `postfix.service` active and enabled and the packaged Webmin module directory present. Local appliance integration passes; public delivery is outside local acceptance. | PASS: the appliance contract found active Postfix and the packaged Webmin module. |
+| Supervised manual updates | Run `gitlab-update --check`. It reports installed `19.3.0-ce.0`, the signed Trixie candidate, and the required-stop policy without mutation. | PASS: `up-to-date; candidate=19.3.0-ce.0` from the official GitLab CE Debian Trixie supervised-update channel. |
+| GitLab-owned Nginx and Confconsole Let's Encrypt integration | Require GitLab Nginx running and the GitLab-specific `get_certificate.py` plugin present. Public issuance is not attempted without a public DNS fixture. | PASS: the appliance contract found GitLab Nginx and the product-specific Confconsole plugin. |
 | Standard Core administration | Inherited SSH, Webmin, cron, Trixie identity, ISO, and normal init behavior are not customized by this application. | Core 19 PASS run `20260824t010251z-1634-32241`, source `24c82ee3540ce545422742b0e28ba6b687c53ec2`, report SHA-256 `1a6c2d266b6a2898e98a073421e49212dd0237598861200a3cbd6e81d35a2936`. |
 
 The fixture revokes its personal access token and SSH key, removes the scheduled
@@ -229,7 +232,46 @@ script. Cookies, the personal access token, the SSH private key, and Git trees
 remain below the original mode-700 directory. This is another test-only
 permission correction, so the product ledger remains at 5 of 6 loops.
 
-The final clean command is:
+The final scoped acceptance retry used source commit
+`adbe2afbf46d2bffc3205451a17468ea86060f5a`, pinned harness commit
+`152bc9b876557266b90ed4ded77b611d9b817aff`, and scoped label
+`org.turnkeylinux.v19-harness.wave2-2` in run
+`20260826t194621z-508720-29542`. The source commit and transport archives both
+have SHA-256
+`114a632a43584f130408b55cfbf9684e8b661d4d983b0899ed269f3c8917888a`.
+The input-tree SHA-256 is
+`f4dcd9c4936703c741c8af9f3a90b26973a9ba839f71196512ce60b0cb3715ae`
+and the build-tree SHA-256 is
+`a152d803893465195ebafef769a6a56d26efb23e52831c7dbdfc467d849c4064`.
+
+Preflight, the exact rootfs build, scoped image import, runtime health, GitLab
+readiness, and every focused test phase passed. The build crossed the pinned
+GitLab package configuration, v19 schema scrub, and bounded service-stop
+boundaries. The configured-root check recorded the expected overlayfs fallback
+with status 32 before import. Runtime reached multi-user state, completed
+inithooks, and reported the appliance service active. The test passed
+`appliance-contract`, `readiness`, `web-login`, `project-api`,
+`ssh-git-round-trip`, `authenticated-web-read`, `database-readback`,
+`background-jobs`, `package-update-channel`, `result`, and `complete`.
+Sidekiq returned marker `fccc077c7203d30d44ba9566`. The updater reported
+`up-to-date; candidate=19.3.0-ce.0` on the official GitLab CE Debian Trixie
+supervised-update channel. Product and runtime tests returned status 0, cleanup
+was verified, and the final verdict was PASS. These final authentication and
+service-readable-script corrections remain test instrumentation, so the
+product ledger remains at 5 of 6 loops.
+
+The retained report is
+`/home/agent/.local/state/turnkey-v19-harness-wave2-2/runs/gitlab/20260826t194621z-508720-29542/report.txt`
+with SHA-256
+`4414e9e1ba4f188e6f7614e8748a3f942c30450beebd518cee02ba2e53abc1b0`.
+The run-log SHA-256 is
+`cf303ace0f1d54a182ff3d58ecce842206946e1572e0180a815e689103bcdbdf`,
+and the child and runtime result files both have SHA-256
+`a9abda94ef91953914eefea03c422f1f3895f718e4fe2c020b23b77be2efafc0`.
+The retained `SHA256SUMS` file, whose entries verify successfully, has SHA-256
+`cbe2e2bc16c11566d9ea5f49eb7addf7c117eea2077737ab464468538a514990`.
+
+The exact acceptance command was:
 
 ```sh
 TKLDEV_CONTAINER=tkldev19-wave2-2 \
@@ -241,11 +283,6 @@ TKL_HARNESS_DOCKER_OBJECT_LABEL=org.turnkeylinux.v19-harness.wave2-2 \
   --source /home/agent/.local/worktrees/turnkey-apps/gitlab/wish-gitlab-v19-trixie
 ```
 
-Final exact acceptance is pending a clean retry of the corrected test
-instrumentation behind queued wave2-2 work. Replace this statement with the
-PASS run ID, report path, exact source commit, and compact result before review
-handoff.
-
 ## Deferred issues
 
 - GitLab upgrades are supervised because required stops occur at minor

From d9f4cd9323de76d4f81e93bf47e448c74cd4b7ae Mon Sep 17 00:00:00 2001
From: Codex <codex@openai.com>
Date: Wed, 26 Aug 2026 20:51:29 +0000
Subject: [PATCH 10/12] Honor GitLab firstboot and repository trust contracts

Persist the prompted root email through GitLab's confirmed primary-email path and assert it through the authenticated API. Replace apt-key recovery with fingerprint-verified signed-by rotation, tighten updater trust checks, and correct the product-loop ledger.
---
 README.rst                              |   5 ++
 docs/update-apt-repo-key.rst            | 104 +++++++++++++++++-------
 docs/v19.0-testing.md                   |  50 ++++++++++--
 overlay/usr/lib/inithooks/bin/gitlab.py |  20 +++++
 overlay/usr/local/sbin/gitlab-update    |   8 +-
 tests/v19.sh                            |  21 ++++-
 6 files changed, 166 insertions(+), 42 deletions(-)

diff --git a/README.rst b/README.rst
index 1ded04c..a0c723e 100644
--- a/README.rst
+++ b/README.rst
@@ -47,6 +47,10 @@ Repeat the application acceptance checks before proceeding to another required
 stop. Available versions are listed by ``apt-cache madison gitlab-ce`` and the
 `GitLab release blog`_.
 
+If APT reports an expired repository key or ``NO_PUBKEY``, follow the
+`repository-key rotation procedure`_. It preserves the per-repository
+``signed-by`` restriction and verifies GitLab's full published fingerprint.
+
 Credentials *(passwords set at first boot)*
 -------------------------------------------
 
@@ -59,3 +63,4 @@ Credentials *(passwords set at first boot)*
 .. _GitLab documentation: https://docs.gitlab.com/omnibus/update/README.html
 .. _GitLab upgrade path: https://docs.gitlab.com/update/upgrade_paths/
 .. _GitLab release blog: https://about.gitlab.com/blog/categories/releases/
+.. _repository-key rotation procedure: docs/update-apt-repo-key.rst
diff --git a/docs/update-apt-repo-key.rst b/docs/update-apt-repo-key.rst
index 4d168c2..17498a8 100644
--- a/docs/update-apt-repo-key.rst
+++ b/docs/update-apt-repo-key.rst
@@ -1,5 +1,5 @@
-TunrKey Linux GitLab - Update GitLab apt repo key
-=================================================
+TurnKey Linux GitLab - Rotate the GitLab APT repository key
+============================================================
 
 .. contents::
 
@@ -7,38 +7,86 @@ TunrKey Linux GitLab - Update GitLab apt repo key
 Context
 =======
 
-This doc details how to fix a `GitLab "NO_PUBKEY" error`_ message when using
-apt.
+This document explains how to recover from a GitLab repository ``NO_PUBKEY``
+or expired-key error. Debian Trixie does not provide ``apt-key``. The GitLab
+repository is instead restricted to
+``/usr/share/keyrings/gitlab-ce.gpg`` by the source's ``signed-by`` option.
 
-Background
-==========
+Trust boundary
+==============
 
-To ensure that the packages that you download are the ones provided by the
-packager, apt repositories are cryptographically signed with a GPG key. From
-time to time, these keys are "rotated" (i.e. new keys generated and this new
-key used instead of the old one). When this happens, you will need to update
-the GPG keyring that apt checks against when downloadng apt package lists.
+Obtain the full current repository-metadata signing-key fingerprint from the
+official `GitLab Linux package signatures`_ page through a trusted browser.
+The fingerprint documented for this appliance release is
+``F6403F6544A38863DAA0B6E03F01618A51312F3F``. If GitLab has published a
+replacement, substitute its complete 40-character uppercase fingerprint in
+the procedure below. Do not trust a short key ID or the downloaded key alone.
 
-GitLab upstream `provide instructions` on how to do that. However, TurnKey
-Linux follows the "best practice" convention of specifying which particular
-key any 3rd party repository should use. To ensure that this is honored, the
-key needs to be stored in a particular location (as defined in the relevant
-`sources.list entry`_) and added in a way slightly
-different to the upstream instructions.
+The procedure verifies the download before changing trust, preserves the
+per-repository ``signed-by`` restriction, and updates the appliance source
+record consumed by ``gitlab-update --check``. Run it as ``root``::
 
-How to update the GitLab GPG key
-================================
+   set -eu
+   expected_fingerprint=F6403F6544A38863DAA0B6E03F01618A51312F3F
+   key_url=https://packages.gitlab.com/gpg.key
+   keyring=/usr/share/keyrings/gitlab-ce.gpg
+   source_list=/etc/apt/sources.list.d/gitlab-ce.list
+   source_record=/usr/local/share/turnkey-gitlab/source
+   source_line="deb [signed-by=$keyring] https://packages.gitlab.com/gitlab/gitlab-ce/debian/ trixie main"
+   work=$(mktemp -d /tmp/gitlab-key-rotation.XXXXXXXX)
+   staged_keyring=
+   staged_record=
+   cleanup() {
+       rm -rf -- "$work"
+       test -z "$staged_keyring" || rm -f -- "$staged_keyring"
+       test -z "$staged_record" || rm -f -- "$staged_record"
+   }
+   trap cleanup EXIT
+   trap 'exit 1' HUP INT TERM
 
-Assuming that the new keyfile provided by GitLab is the same as it was when
-they rotated their keys (April 2020), then this will resolve the issue::
+   test "$(id -u)" -eq 0
+   grep -Fxq "$source_line" "$source_list"
+   test "$(grep -c '^repository_key_fingerprint=' "$source_record")" -eq 1
+   test "$(grep -c '^repository_key_sha256=' "$source_record")" -eq 1
 
-   curl -o /tmp/gitlab-ce.key https://packages.gitlab.com/gpg.key
-   apt-key --keyring /usr/share/keyrings/gitlab-ce.gpg add /tmp/gitlab-ce.key
+   curl --proto '=https' --tlsv1.2 --fail --silent --show-error --location \
+       "$key_url" --output "$work/gitlab.key"
+   fingerprint=$(gpg --show-keys --with-colons "$work/gitlab.key" | \
+       awk -F: '$1 == "fpr" { print $10; exit }')
+   test "$fingerprint" = "$expected_fingerprint"
+   key_sha256=$(sha256sum "$work/gitlab.key" | awk '{ print $1 }')
 
-Note that if you are not running as root, 'sudo' will be required for the
-second line.
+   staged_keyring=$(mktemp /usr/share/keyrings/gitlab-ce.gpg.XXXXXXXX)
+   gpg --batch --yes --dearmor --output "$staged_keyring" \
+       "$work/gitlab.key"
+   chmod 0644 "$staged_keyring"
+   test "$(gpg --show-keys --with-colons "$staged_keyring" | \
+       awk -F: '$1 == "fpr" { print $10; exit }')" = \
+       "$expected_fingerprint"
 
+   staged_record=$(mktemp /usr/local/share/turnkey-gitlab/source.XXXXXXXX)
+   sed \
+       -e "s/^repository_key_fingerprint=.*/repository_key_fingerprint=$expected_fingerprint/" \
+       -e "s/^repository_key_sha256=.*/repository_key_sha256=$key_sha256/" \
+       "$source_record" >"$staged_record"
+   chmod --reference="$source_record" "$staged_record"
 
-.. _provide instructions: https://docs.gitlab.com/omnibus/update/package_signatures.html#fetching-new-keys-after-2020-04-06
-.. _GitLab "NO_PUBKEY" error: https://github.com/turnkeylinux/tracker/issues/1441
-.. _sources.list entry: https://github.com/turnkeylinux-apps/gitlab/blob/master/overlay/etc/apt/sources.list.d/gitlab-ce.list#L4
+   mv -f -- "$staged_keyring" "$keyring"
+   staged_keyring=
+   mv -f -- "$staged_record" "$source_record"
+   staged_record=
+
+   apt-get update
+   gitlab-update --check | tee "$work/update-check"
+   grep -Fxq "integrity=APT-signed-by-$expected_fingerprint" \
+       "$work/update-check"
+   grep -Fxq "repository_key_download_sha256=$key_sha256" \
+       "$work/update-check"
+
+Every trust check occurs before APT refreshes repository metadata. If the
+command is interrupted between the two final moves, ``gitlab-update --check``
+fails because the keyring and source record disagree. Rerun the complete
+procedure rather than weakening the ``signed-by`` restriction.
+
+
+.. _GitLab Linux package signatures: https://docs.gitlab.com/omnibus/update/package_signatures/
diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index a3f6b7d..8ff7a63 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -39,12 +39,12 @@ Unless noted otherwise, PASS results refer to scoped exact run
 
 | README contract | Focused command and expected result | Actual result and retained evidence |
 | --- | --- | --- |
-| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` at the configured domain. | PASS: `web-login` authenticated the configured root account and its API identity. |
+| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` and reports the firstboot email at the configured domain. | FIX-FIRST: the pre-review run authenticated `root` but omitted the email assertion. The corrected firstboot and focused assertion require a new exact run. |
 | Project hosting and Git repository readback | Create a private project, push a commit, clone it to a fresh directory, and read the committed file through GitLab HTTP. The marker survives every read path. | PASS: `project-api`, `ssh-git-round-trip`, and `authenticated-web-read` preserved the same private-project marker. |
 | Git over SSH | Register a throwaway Ed25519 key and use `git@127.0.0.1` for push and clone. Both operations succeed through GitLab Shell. | PASS: the throwaway key pushed and cloned through GitLab Shell. |
 | Omnibus Nginx, PostgreSQL, Redis, Gitaly, and Sidekiq | Require every supervisor process, query the created project through `gitlab-psql`, enqueue `ProjectCacheWorker`, and wait for its uniquely keyed scheduled follow-up. The database row is exact and the background job completes without retry. | PASS: service checks, `database-readback`, and `background-jobs` completed. Sidekiq returned marker `fccc077c7203d30d44ba9566`. |
 | Postfix and Webmin Postfix module | Require `postfix.service` active and enabled and the packaged Webmin module directory present. Local appliance integration passes; public delivery is outside local acceptance. | PASS: the appliance contract found active Postfix and the packaged Webmin module. |
-| Supervised manual updates | Run `gitlab-update --check`. It reports installed `19.3.0-ce.0`, the signed Trixie candidate, and the required-stop policy without mutation. | PASS: `up-to-date; candidate=19.3.0-ce.0` from the official GitLab CE Debian Trixie supervised-update channel. |
+| Supervised manual updates | Run `gitlab-update --check`. It reports installed `19.3.0-ce.0`, the signed Trixie candidate, and the required-stop policy without mutation. | The pre-review run passed with `up-to-date; candidate=19.3.0-ce.0`. The corrected check also requires the exact `signed-by` source and recorded key-download hash; exact rerun pending. |
 | GitLab-owned Nginx and Confconsole Let's Encrypt integration | Require GitLab Nginx running and the GitLab-specific `get_certificate.py` plugin present. Public issuance is not attempted without a public DNS fixture. | PASS: the appliance contract found GitLab Nginx and the product-specific Confconsole plugin. |
 | Standard Core administration | Inherited SSH, Webmin, cron, Trixie identity, ISO, and normal init behavior are not customized by this application. | Core 19 PASS run `20260824t010251z-1634-32241`, source `24c82ee3540ce545422742b0e28ba6b687c53ec2`, report SHA-256 `1a6c2d266b6a2898e98a073421e49212dd0237598861200a3cbd6e81d35a2936`. |
 
@@ -109,7 +109,7 @@ polls that probe with a five-minute bound before attempting the normal sign-in
 flow. Extracted-function fixtures pass for immediate success, success on the
 third probe, and exhaustion after exactly 61 probes and 60 two-second sleeps.
 
-Product loop 5 used source commit
+An infrastructure-only run after product loop 4 used source commit
 `e2286dbca78775ed877f2c1539ed8a5e1632d918` in run
 `20260826t113407z-2022901-16188`. The build and image import completed,
 including the fail-fast v19 scrub and bounded service stop. The harness then
@@ -121,7 +121,7 @@ The harness treated a transient or empty `docker inspect` result as a stopped
 container. GitLab readiness and the focused test were not reached, so this run
 provides build and import evidence but no application acceptance evidence.
 
-The infrastructure retry of product loop 5 used the same source commit with
+The infrastructure retry of that run used the same source commit with
 reviewed harness commit `5746bdca17347598f92767d1a9c76b6406a160d4` in run
 `20260826t124900z-2678316-2241`. Build, import, runtime health, GitLab
 readiness, and root web login passed. The test then stopped at its first JSON
@@ -230,9 +230,9 @@ exclusive `mktemp` file directly under `/tmp`, then grants read access to the
 project identifier remains an environment variable, and cleanup removes the
 script. Cookies, the personal access token, the SSH private key, and Git trees
 remain below the original mode-700 directory. This is another test-only
-permission correction, so the product ledger remains at 5 of 6 loops.
+permission correction, so the product ledger remains at 4 of 6 loops.
 
-The final scoped acceptance retry used source commit
+The scoped pre-review run used source commit
 `adbe2afbf46d2bffc3205451a17468ea86060f5a`, pinned harness commit
 `152bc9b876557266b90ed4ded77b611d9b817aff`, and scoped label
 `org.turnkeylinux.v19-harness.wave2-2` in run
@@ -258,7 +258,7 @@ Sidekiq returned marker `fccc077c7203d30d44ba9566`. The updater reported
 supervised-update channel. Product and runtime tests returned status 0, cleanup
 was verified, and the final verdict was PASS. These final authentication and
 service-readable-script corrections remain test instrumentation, so the
-product ledger remains at 5 of 6 loops.
+product ledger remained at 4 of 6 loops.
 
 The retained report is
 `/home/agent/.local/state/turnkey-v19-harness-wave2-2/runs/gitlab/20260826t194621z-508720-29542/report.txt`
@@ -271,7 +271,40 @@ and the child and runtime result files both have SHA-256
 The retained `SHA256SUMS` file, whose entries verify successfully, has SHA-256
 `cbe2e2bc16c11566d9ea5f49eb7addf7c117eea2077737ab464468538a514990`.
 
-The exact acceptance command was:
+Independent review then found that the firstboot prompt promised to set the
+root user's email, while the implementation only changed
+`gitlab_email_from`. Run `20260826t194621z-508720-29542` used
+`APP_EMAIL=admin@example.invalid`, but its focused login assertion checked only
+the username. This is an application behavior defect and consumes product loop
+5. The correction passes the address through the environment to
+`gitlab-rails runner`, calls `skip_reconfirmation!`, and persists the primary
+email with `update!`. GitLab 19.3's [user model specification][root-email-spec]
+confirms that this sequence persists a confirmed primary email without an
+unconfirmed value. The focused login phase now reads the expected firstboot
+value and asserts the authenticated API's `email` field. The product ledger is
+now 5 of 6 loops.
+
+The same review found Trixie-incompatible `apt-key` recovery instructions. The
+replacement procedure uses only the repository's `signed-by` keyring, verifies
+the full fingerprint published by GitLab before changing trust, updates the
+recorded fingerprint and download hash, and reruns `gitlab-update --check`.
+The updater now fails closed unless the source line names the exact appliance
+keyring and the source record contains a valid download hash.
+
+A disposable Debian Trixie probe fetched GitLab's current key and reconfirmed
+download SHA-256
+`003c0ca2fea61767f8c6de7a1c0f49fc88ea3c8db95e3cd1856b32ce9d876e0f`
+and fingerprint `F6403F6544A38863DAA0B6E03F01618A51312F3F`. The exact updater returned
+status 0 with matching keyring, source line, fingerprint, and download hash.
+An incorrect `signed-by` path, an incorrect fingerprint, and a malformed
+download hash each returned status 1. A stubbed firstboot matrix confirmed
+that the email is passed only through `TKL_ROOT_EMAIL`, successful persistence
+continues to password reset, and a persistence status of 17 stops before the
+password reset. The extracted authenticated-user assertion accepts the
+firstboot address and rejects the former `admin@example.com` value.
+
+Exact acceptance of the product-loop-5 and trust-recovery corrections is
+pending authorization. The command remains:
 
 ```sh
 TKLDEV_CONTAINER=tkldev19-wave2-2 \
@@ -301,3 +334,4 @@ TKL_HARNESS_DOCKER_OBJECT_LABEL=org.turnkeylinux.v19-harness.wave2-2 \
 [auth-finders]: https://gitlab.com/gitlab-org/gitlab/-/blob/v19.3.0-ee/lib/gitlab/auth/auth_finders.rb
 [repository-files-api]: https://docs.gitlab.com/api/repository_files/#retrieve-a-raw-file-from-a-repository
 [rest-authentication]: https://docs.gitlab.com/api/rest/authentication/#session-cookie
+[root-email-spec]: https://gitlab.com/gitlab-org/gitlab/-/blob/v19.3.0-ee/spec/models/user_spec.rb#L2971-2984
diff --git a/overlay/usr/lib/inithooks/bin/gitlab.py b/overlay/usr/lib/inithooks/bin/gitlab.py
index d3d8f38..81ac6b0 100755
--- a/overlay/usr/lib/inithooks/bin/gitlab.py
+++ b/overlay/usr/lib/inithooks/bin/gitlab.py
@@ -105,6 +105,26 @@ def main():
          config], check=True)
     run(["gitlab-ctl", "reconfigure"], check=True)
 
+    print("Setting GitLab 'root' user email. This might take a while.")
+    email_env = os.environ.copy()
+    email_env.pop('APP_PASS', None)
+    email_env['TKL_ROOT_EMAIL'] = email
+    update_email = run(
+        ["gitlab-rails", "runner",
+         "root = User.find_by!(username: 'root'); "
+         "root.skip_reconfirmation!; "
+         "root.update!(email: ENV.fetch('TKL_ROOT_EMAIL'))"],
+        env=email_env,
+        text=True,
+        capture_output=True,
+    )
+    if update_email.stdout:
+        print(update_email.stdout, end="")
+    if update_email.stderr:
+        print(update_email.stderr, file=sys.stderr, end="")
+    if update_email.returncode:
+        sys.exit(update_email.returncode)
+
     print("Setting GitLab 'root' user password. This might take a while.")
     reset = run(
         ["gitlab-rake", "gitlab:password:reset[root]"],
diff --git a/overlay/usr/local/sbin/gitlab-update b/overlay/usr/local/sbin/gitlab-update
index c2d9b85..fe7563f 100755
--- a/overlay/usr/local/sbin/gitlab-update
+++ b/overlay/usr/local/sbin/gitlab-update
@@ -8,11 +8,14 @@ fi
 
 source_record=/usr/local/share/turnkey-gitlab/source
 keyring=/usr/share/keyrings/gitlab-ce.gpg
+source_list=/etc/apt/sources.list.d/gitlab-ce.list
+source_line="deb [signed-by=$keyring] https://packages.gitlab.com/gitlab/gitlab-ce/debian/ trixie main"
 
 # shellcheck disable=SC1090
 . "$source_record"
 : "${installed_version:?installed_version is missing from $source_record}"
 : "${repository_key_fingerprint:?repository_key_fingerprint is missing from $source_record}"
+: "${repository_key_sha256:?repository_key_sha256 is missing from $source_record}"
 
 installed=$(dpkg-query -W -f='${Version}' gitlab-ce)
 candidate=$(apt-cache policy gitlab-ce |
@@ -22,9 +25,9 @@ fingerprint=$(gpg --show-keys --with-colons "$keyring" |
 
 test "$installed" = "$installed_version"
 test "$fingerprint" = "$repository_key_fingerprint"
+[[ $repository_key_sha256 =~ ^[0-9a-f]{64}$ ]]
 test "$candidate" != "(none)"
-grep -Fq 'packages.gitlab.com/gitlab/gitlab-ce/debian/ trixie main' \
-    /etc/apt/sources.list.d/gitlab-ce.list
+grep -Fxq "$source_line" "$source_list"
 
 if dpkg --compare-versions "$candidate" eq "$installed"; then
     status=up-to-date
@@ -39,4 +42,5 @@ channel=official-gitlab-ce-debian-trixie
 status=$status
 integrity=APT-signed-by-$fingerprint
 upgrade_policy=follow-required-stops-before-apt-install
+repository_key_download_sha256=$repository_key_sha256
 EOF
diff --git a/tests/v19.sh b/tests/v19.sh
index 3abafd9..12ee4e3 100755
--- a/tests/v19.sh
+++ b/tests/v19.sh
@@ -127,9 +127,11 @@ test -f /usr/lib/confconsole/plugins.d/Lets_Encrypt/get_certificate.py
 : "${installed_version:?installed_version is missing from $source_file}"
 : "${package_sha256:?package_sha256 is missing from $source_file}"
 : "${repository_key_fingerprint:?repository_key_fingerprint is missing from $source_file}"
+: "${repository_key_sha256:?repository_key_sha256 is missing from $source_file}"
 test "$(dpkg-query -W -f='${Version}' gitlab-ce)" = "$installed_version"
 test "$installed_version" = 19.3.0-ce.0
 test "$package_sha256" = f88f80cd61d6b2beb35aa7207591d4abdfed0e6c2c42e6ed753dd29ea5de076d
+test "$repository_key_sha256" = 003c0ca2fea61767f8c6de7a1c0f49fc88ea3c8db95e3cd1856b32ce9d876e0f
 test "$(gpg --show-keys --with-colons /usr/share/keyrings/gitlab-ce.gpg |
     awk -F: '$1 == "fpr" { print $10; exit }')" = \
     "$repository_key_fingerprint"
@@ -137,6 +139,9 @@ test "$(gpg --show-keys --with-colons /usr/share/keyrings/gitlab-ce.gpg |
 mark_phase readiness
 wait_gitlab_ready
 mark_phase web-login
+expected_root_email=$(sed -n 's/^APP_EMAIL=//p' /etc/inithooks.conf)
+test -n "$expected_root_email" ||
+    fail 'firstboot preseed does not define the expected root email'
 "${curl_local[@]}" --fail --cookie-jar "$cookie" \
     "$base/users/sign_in" >"$page"
 csrf=$(sed -n 's/.*name="authenticity_token" value="\([^"]*\)".*/\1/p' \
@@ -149,8 +154,13 @@ test -n "$csrf"
     --data-urlencode "user[password]=$app_password" \
     --data-urlencode 'user[remember_me]=0' \
     "$base/users/sign_in" >"$page"
-test "$("${curl_local[@]}" --fail --cookie "$cookie" \
-    "$base/api/v4/user" | json_value string username)" = root
+root_account=$("${curl_local[@]}" --fail --cookie "$cookie" \
+    "$base/api/v4/user")
+test "$(json_value string username <<<"$root_account")" = root ||
+    fail 'authenticated GitLab account is not root'
+test "$(json_value string email <<<"$root_account")" = \
+        "$expected_root_email" ||
+    fail 'root account email does not match the firstboot value'
 
 mark_phase project-api
 gitlab-rails runner \
@@ -241,15 +251,18 @@ grep -Fxq 'channel=official-gitlab-ce-debian-trixie' "$work/update" ||
     fail 'updater did not report the official GitLab CE Debian Trixie channel'
 grep -Fxq "integrity=APT-signed-by-$repository_key_fingerprint" "$work/update" ||
     fail 'updater did not report the pinned repository key fingerprint'
+grep -Fxq "repository_key_download_sha256=$repository_key_sha256" \
+        "$work/update" ||
+    fail 'updater did not report the verified repository key download hash'
 
 mark_phase result
 cat >"$result" <<EOF
 package_source=Official GitLab CE Debian Trixie repository
 installed_version=$installed_version
-runtime_checks=normal init; root web login; project API create and web read; SSH Git push and clone; PostgreSQL readback; Sidekiq project cache job; Postfix
+runtime_checks=normal init; root web login and firstboot email; project API create and web read; SSH Git push and clone; PostgreSQL readback; Sidekiq project cache job; Postfix
 updater_command=gitlab-update --check
 updater_result=$status; candidate=$candidate
 updater_channel=official GitLab CE Debian Trixie, supervised required-stop upgrades
-integrity_evidence=repository key $repository_key_fingerprint; package SHA-256 $package_sha256
+integrity_evidence=repository key $repository_key_fingerprint; repository key download SHA-256 $repository_key_sha256; package SHA-256 $package_sha256
 EOF
 mark_phase complete

From 99e44733373078290fe7a45072a64764277a8309 Mon Sep 17 00:00:00 2001
From: Codex <codex@openai.com>
Date: Wed, 26 Aug 2026 21:43:42 +0000
Subject: [PATCH 11/12] Use the exact GitLab email fixture

Assert the root API email against the harness's nonsecret firstboot constant instead of its post-firstboot scrubbed preseed file. Record the retained product-loop-5 evidence without advancing the product ledger.
---
 docs/v19.0-testing.md | 43 ++++++++++++++++++++++++++++++++++++++-----
 tests/v19.sh          |  4 +---
 2 files changed, 39 insertions(+), 8 deletions(-)

diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index 8ff7a63..df9e31d 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -39,7 +39,7 @@ Unless noted otherwise, PASS results refer to scoped exact run
 
 | README contract | Focused command and expected result | Actual result and retained evidence |
 | --- | --- | --- |
-| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` and reports the firstboot email at the configured domain. | FIX-FIRST: the pre-review run authenticated `root` but omitted the email assertion. The corrected firstboot and focused assertion require a new exact run. |
+| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` and reports the firstboot email at the configured domain. | Product loop 5 completed the corrected firstboot update and addressed its password-change mail to `admin@example.invalid`. A test-only retry of the authenticated API assertion is pending. |
 | Project hosting and Git repository readback | Create a private project, push a commit, clone it to a fresh directory, and read the committed file through GitLab HTTP. The marker survives every read path. | PASS: `project-api`, `ssh-git-round-trip`, and `authenticated-web-read` preserved the same private-project marker. |
 | Git over SSH | Register a throwaway Ed25519 key and use `git@127.0.0.1` for push and clone. Both operations succeed through GitLab Shell. | PASS: the throwaway key pushed and cloned through GitLab Shell. |
 | Omnibus Nginx, PostgreSQL, Redis, Gitaly, and Sidekiq | Require every supervisor process, query the created project through `gitlab-psql`, enqueue `ProjectCacheWorker`, and wait for its uniquely keyed scheduled follow-up. The database row is exact and the background job completes without retry. | PASS: service checks, `database-readback`, and `background-jobs` completed. Sidekiq returned marker `fccc077c7203d30d44ba9566`. |
@@ -281,8 +281,8 @@ the username. This is an application behavior defect and consumes product loop
 email with `update!`. GitLab 19.3's [user model specification][root-email-spec]
 confirms that this sequence persists a confirmed primary email without an
 unconfirmed value. The focused login phase now reads the expected firstboot
-value and asserts the authenticated API's `email` field. The product ledger is
-now 5 of 6 loops.
+value from the exact harness's nonsecret fixture contract and asserts the
+authenticated API's `email` field. The product ledger is now 5 of 6 loops.
 
 The same review found Trixie-incompatible `apt-key` recovery instructions. The
 replacement procedure uses only the repository's `signed-by` keyring, verifies
@@ -303,8 +303,38 @@ continues to password reset, and a persistence status of 17 stops before the
 password reset. The extracted authenticated-user assertion accepts the
 firstboot address and rejects the former `admin@example.com` value.
 
-Exact acceptance of the product-loop-5 and trust-recovery corrections is
-pending authorization. The command remains:
+Product loop 5 exact run `20260826t205442z-743235-32572` used source commit
+`d9f4cd9323de76d4f81e93bf47e448c74cd4b7ae`, commit and transport archive
+SHA-256
+`57df01af0b2268979631647fdb28b26a8617eff2ccd6f4e4e491a7d01090e353`,
+input-tree SHA-256
+`b223e1612840b51c3de6589fda156fdcc712eb7b1096af7801d939b154a0423e`,
+and build-tree SHA-256
+`b2d8a225dc32f2cabf938022b1028de49f47f242660235870719936a9b843c16`.
+The rootfs build, configured-root fallback, scoped import, and runtime health
+passed. Inithooks completed and the retained journal recorded the root-email
+update, successful `40gitlab`, and a password-change message addressed to
+`admin@example.invalid`. This proves the corrected product behavior reached
+the prompted root address.
+
+The focused test then failed before submitting web login because Core had
+scrubbed `APP_EMAIL` from the mounted firstboot preseed after inithooks. The
+expected-value lookup was test instrumentation, not appliance behavior. The
+correction uses the exact harness's nonsecret `admin@example.invalid` fixture
+constant and retains the positive and negative API assertion. This failure
+does not consume another product loop, so the ledger remains 5 of 6. The
+retained report at
+`/home/agent/.local/state/turnkey-v19-harness-wave2-2/failures/gitlab/current/report.txt`
+has SHA-256
+`2b751293e90e4bfa5053feecfb4af4fb21119bbc21f9cd7e42ec9af3ec438970`.
+The run-log SHA-256 is
+`fab0bece3897765f7b4181d9b6512c2285d761ead38ebc449588b6fe9a99e35d`,
+the `SHA256SUMS` SHA-256 is
+`a5d264feb3cbcaaef556d116d964251f962c2cc34febb898213caf61c4522ac7`,
+and the `RETAINED-SHA256SUMS` SHA-256 is
+`bd40294884492d6eb1f413bd20d8637521cd1345137f753f720818d495e8e1a5`.
+
+Exact acceptance is pending the test-only retry. The command remains:
 
 ```sh
 TKLDEV_CONTAINER=tkldev19-wave2-2 \
@@ -326,6 +356,9 @@ TKL_HARNESS_DOCKER_OBJECT_LABEL=org.turnkeylinux.v19-harness.wave2-2 \
 - API project deletion can complete asynchronously. The test submits cleanup,
   but does not wait for GitLab's background deletion worker.
 - Docker acceptance does not repeat installer, kernel, or hardware checks.
+- The retained product-loop-5 failure bundle's original `SHA256SUMS` still
+  names the pruned `failure-status.txt`. Its `RETAINED-SHA256SUMS` is the
+  authoritative post-retention manifest and verifies every retained file.
 
 [supported-platforms]: https://docs.gitlab.com/install/package/#supported-platforms
 [health-check]: https://docs.gitlab.com/administration/monitoring/health_check/
diff --git a/tests/v19.sh b/tests/v19.sh
index 12ee4e3..fb59339 100755
--- a/tests/v19.sh
+++ b/tests/v19.sh
@@ -5,6 +5,7 @@ umask 077
 result=${TKL_TEST_RESULT:?TKL_TEST_RESULT is required}
 app_password=${TKL_TEST_APP_PASS:?TKL_TEST_APP_PASS is required}
 source_file=/usr/local/share/turnkey-gitlab/source
+expected_root_email=admin@example.invalid
 fixture="turnkey-v19-$(date +%s)-$$"
 token="tkl$(openssl rand -hex 20)"
 work=$(mktemp -d /tmp/gitlab-v19.XXXXXXXX)
@@ -139,9 +140,6 @@ test "$(gpg --show-keys --with-colons /usr/share/keyrings/gitlab-ce.gpg |
 mark_phase readiness
 wait_gitlab_ready
 mark_phase web-login
-expected_root_email=$(sed -n 's/^APP_EMAIL=//p' /etc/inithooks.conf)
-test -n "$expected_root_email" ||
-    fail 'firstboot preseed does not define the expected root email'
 "${curl_local[@]}" --fail --cookie-jar "$cookie" \
     "$base/users/sign_in" >"$page"
 csrf=$(sed -n 's/.*name="authenticity_token" value="\([^"]*\)".*/\1/p' \

From ea6d0eb5af787ba5e27c454fb069f2b2c2efbb32 Mon Sep 17 00:00:00 2001
From: Codex <codex@openai.com>
Date: Wed, 26 Aug 2026 22:52:11 +0000
Subject: [PATCH 12/12] Record the passing GitLab acceptance

Seal the exact scoped run identity, source and harness hashes, phase results, and retained evidence checksums. Update the README crosswalk with the verified root email and updater trust results.
---
 docs/v19.0-testing.md | 50 +++++++++++++++++++++++++++++++++++++------
 1 file changed, 44 insertions(+), 6 deletions(-)

diff --git a/docs/v19.0-testing.md b/docs/v19.0-testing.md
index df9e31d..ac4ed00 100644
--- a/docs/v19.0-testing.md
+++ b/docs/v19.0-testing.md
@@ -34,17 +34,17 @@ under the running appliance service manager.
 The focused command in every application-specific row is `tests/v19.sh`, run
 inside the configured Docker root by the exact acceptance command below.
 Unless noted otherwise, PASS results refer to scoped exact run
-`20260826t194621z-508720-29542` and its report with SHA-256
-`4414e9e1ba4f188e6f7614e8748a3f942c30450beebd518cee02ba2e53abc1b0`.
+`20260826t214418z-934198-11400` and its report with SHA-256
+`9deb2382b15c6b5bed63e9ebe3a628c8e2c4f50527cb6f651f14eaf51d38e0aa`.
 
 | README contract | Focused command and expected result | Actual result and retained evidence |
 | --- | --- | --- |
-| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` and reports the firstboot email at the configured domain. | Product loop 5 completed the corrected firstboot update and addressed its password-change mail to `admin@example.invalid`. A test-only retry of the authenticated API assertion is pending. |
+| Root firstboot password, email, and domain | Submit the rendered sign-in form with `TKL_TEST_APP_PASS`, then query the authenticated user endpoint. The session identifies `root` and reports the firstboot email at the configured domain. | PASS: product loop 5 persisted `admin@example.invalid`, the password-change mail used the same recipient, and the exact authenticated API assertion returned that address for `root`. |
 | Project hosting and Git repository readback | Create a private project, push a commit, clone it to a fresh directory, and read the committed file through GitLab HTTP. The marker survives every read path. | PASS: `project-api`, `ssh-git-round-trip`, and `authenticated-web-read` preserved the same private-project marker. |
 | Git over SSH | Register a throwaway Ed25519 key and use `git@127.0.0.1` for push and clone. Both operations succeed through GitLab Shell. | PASS: the throwaway key pushed and cloned through GitLab Shell. |
-| Omnibus Nginx, PostgreSQL, Redis, Gitaly, and Sidekiq | Require every supervisor process, query the created project through `gitlab-psql`, enqueue `ProjectCacheWorker`, and wait for its uniquely keyed scheduled follow-up. The database row is exact and the background job completes without retry. | PASS: service checks, `database-readback`, and `background-jobs` completed. Sidekiq returned marker `fccc077c7203d30d44ba9566`. |
+| Omnibus Nginx, PostgreSQL, Redis, Gitaly, and Sidekiq | Require every supervisor process, query the created project through `gitlab-psql`, enqueue `ProjectCacheWorker`, and wait for its uniquely keyed scheduled follow-up. The database row is exact and the background job completes without retry. | PASS: service checks, `database-readback`, and `background-jobs` completed. Sidekiq returned marker `37465613dcfb7fbbe65c2bd5`. |
 | Postfix and Webmin Postfix module | Require `postfix.service` active and enabled and the packaged Webmin module directory present. Local appliance integration passes; public delivery is outside local acceptance. | PASS: the appliance contract found active Postfix and the packaged Webmin module. |
-| Supervised manual updates | Run `gitlab-update --check`. It reports installed `19.3.0-ce.0`, the signed Trixie candidate, and the required-stop policy without mutation. | The pre-review run passed with `up-to-date; candidate=19.3.0-ce.0`. The corrected check also requires the exact `signed-by` source and recorded key-download hash; exact rerun pending. |
+| Supervised manual updates | Run `gitlab-update --check`. It reports installed `19.3.0-ce.0`, the signed Trixie candidate, and the required-stop policy without mutation. | PASS: the exact corrected check required the appliance `signed-by` keyring and recorded key-download hash, then reported `up-to-date; candidate=19.3.0-ce.0`. |
 | GitLab-owned Nginx and Confconsole Let's Encrypt integration | Require GitLab Nginx running and the GitLab-specific `get_certificate.py` plugin present. Public issuance is not attempted without a public DNS fixture. | PASS: the appliance contract found GitLab Nginx and the product-specific Confconsole plugin. |
 | Standard Core administration | Inherited SSH, Webmin, cron, Trixie identity, ISO, and normal init behavior are not customized by this application. | Core 19 PASS run `20260824t010251z-1634-32241`, source `24c82ee3540ce545422742b0e28ba6b687c53ec2`, report SHA-256 `1a6c2d266b6a2898e98a073421e49212dd0237598861200a3cbd6e81d35a2936`. |
 
@@ -334,7 +334,45 @@ the `SHA256SUMS` SHA-256 is
 and the `RETAINED-SHA256SUMS` SHA-256 is
 `bd40294884492d6eb1f413bd20d8637521cd1345137f753f720818d495e8e1a5`.
 
-Exact acceptance is pending the test-only retry. The command remains:
+The test-only retry used source commit
+`99e44733373078290fe7a45072a64764277a8309`, pinned harness commit
+`152bc9b876557266b90ed4ded77b611d9b817aff`, and scoped label
+`org.turnkeylinux.v19-harness.wave2-2` in run
+`20260826t214418z-934198-11400`. The commit and transport archives both have
+SHA-256
+`7a35cdb9ca0fbfbdee7af43cfb4a6aa684ed7b27963fe4a924fb17625203c36a`.
+The input-tree SHA-256 is
+`7b9865913cfb8db0352564d1e51a01d2d8f677dd7418bcd28936da0fff910518`
+and the build-tree SHA-256 is
+`74762474f70bc4b1b4d21de8cf3bc59a8c0e19d6cedea34d4195b78bb4e7da2b`.
+
+The HTTPS preflight, exact rootfs build, configured-root fallback, scoped
+image import, runtime health, and every focused phase passed. Runtime reached
+multi-user state, completed inithooks, and reported the appliance service
+active. The test passed `appliance-contract`, `readiness`, `web-login`,
+`project-api`, `ssh-git-round-trip`, `authenticated-web-read`,
+`database-readback`, `background-jobs`, `package-update-channel`, `result`,
+and `complete`. The authenticated user API returned the exact firstboot
+address, the private repository marker survived the API, SSH, clone, and web
+read paths, PostgreSQL returned the created project, Sidekiq returned marker
+`37465613dcfb7fbbe65c2bd5`, and the updater validated the synchronized trust
+record before reporting `up-to-date; candidate=19.3.0-ce.0`.
+
+Product and runtime tests returned status 0, scoped cleanup was verified, and
+the final verdict was PASS. The retained report at
+`/home/agent/.local/state/turnkey-v19-harness-wave2-2/runs/gitlab/20260826t214418z-934198-11400/report.txt`
+has SHA-256
+`9deb2382b15c6b5bed63e9ebe3a628c8e2c4f50527cb6f651f14eaf51d38e0aa`.
+The run-log SHA-256 is
+`7cf0e9d5dfd53b0b9c0b0831cdfdb4c06ac898b21c83faf1b5a05e6e09739bbe`,
+the child and runtime result files both have SHA-256
+`cb3019189142fade79cddcfabbf4b2b51c32d19c044030f63886f240a292cb4a`,
+and the retained `SHA256SUMS` file has SHA-256
+`5054869a7790d7abea30787cb399e3d08b1fea23a20bf95184550aa68ae73d6e`.
+Every manifest entry verifies. This retry corrected only test instrumentation,
+so the product ledger remains 5 of 6 loops.
+
+The exact passing command was:
 
 ```sh
 TKLDEV_CONTAINER=tkldev19-wave2-2 \