From a96256c2837e9069c4ba868965909c4d3515c74f Mon Sep 17 00:00:00 2001 From: Mumpe73 Date: Fri, 6 Nov 2026 00:34:47 +0300 Subject: [PATCH 1/8] docs: add self-hosted upgrade and rollback guide --- .../docs/deployment-self-hosting/upgrading.md | 197 ++++++++++++++++++ .../diagrams/upgrade-rollback-restore.svg | 42 ++++ 2 files changed, 239 insertions(+) create mode 100644 docs/docs/deployment-self-hosting/upgrading.md create mode 100644 docs/docs/diagrams/upgrade-rollback-restore.svg diff --git a/docs/docs/deployment-self-hosting/upgrading.md b/docs/docs/deployment-self-hosting/upgrading.md new file mode 100644 index 000000000000..c41bca1ebdd9 --- /dev/null +++ b/docs/docs/deployment-self-hosting/upgrading.md @@ -0,0 +1,197 @@ +--- +sidebar_label: Upgrading +title: Upgrading and rolling back +sidebar_position: 4 +--- + +This page explains how to upgrade a self-hosted Flagsmith deployment that was installed with the Flagsmith Helm chart, +and what to do if you need to go back. It is based on a test upgrade from chart 0.80.0 (Flagsmith 2.222.0) to chart +0.85.0 (Flagsmith 2.286.0) on a local [kind](https://kind.sigs.k8s.io/) cluster with default values. + +An upgrade changes two things: the application and the database schema. Helm manages the first, but not the second. That +difference matters most when you need to roll back. + +![The application and database at each stage of an upgrade, rollback and restore](../diagrams/upgrade-rollback-restore.svg) + +## Before you upgrade + +1. Read the release notes for every version between the one you run and the one you are moving to. +2. Note your current Helm revision, so you know what to roll back to: + + ```bash + helm history -n flagsmith flagsmith + ``` + +3. Back up the database, check the backup can be read, and store a copy outside the cluster. The commands below show how + to do this for the PostgreSQL instance deployed by the chart. + +### Back up the PostgreSQL instance deployed by the chart + +:::info + +These commands use the pod name `flagsmith-dev-postgresql-0`, the `postgres` user and the `POSTGRES_PASSWORD` and +`POSTGRES_DB` variables that were set in that pod during testing. Check them with +`kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- env`. If you use an external or managed database, use your +provider's snapshot or dump tools instead. + +::: + +```bash +kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD pg_dump -U postgres -Fc -f /tmp/backup.dump $POSTGRES_DB' +``` + +Check the backup can be read. A number greater than zero means the file contains your tables: + +```bash +kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'pg_restore -l /tmp/backup.dump | grep -c TABLE' +``` + +Copy the backup out of the cluster: + +```bash +kubectl cp flagsmith/flagsmith-dev-postgresql-0:/tmp/backup.dump ./backup.dump +``` + +## Upgrade with Helm + +```bash +helm upgrade -n flagsmith flagsmith flagsmith/flagsmith --version -f values.yaml --wait --timeout 20m +``` + +Pass the same values file you installed with. Without `-f` or `--reuse-values`, Helm applies the chart defaults and +discards values you set earlier. The test used default values only, so this behaviour was not tested here. + +`--wait` prints nothing until every pod is ready. Downloading images can take several minutes on a slow connection. In +the test, a first install with a 10 minute timeout failed because the PostgreSQL image took more than five minutes to +download, even though the pods were healthy shortly afterwards. + +## How migrations run + +With the chart's default values, there is no migration Job. Database migrations run in an init container named +`migrate-db` in the API pod, and the API starts only after it finishes. To read its output, find the new API pod and +request the init container's logs: + +```bash +kubectl -n flagsmith get pods +kubectl -n flagsmith logs -c migrate-db +``` + +A successful run lists each migration it applied. For example: + +```text +Running migrations: + Applying code_references.0003_introduce_per_feature_scanned_references... OK +``` + +If the database is already up to date, the log says `No migrations to apply.` + +The chart also has a `jobs.migrateDb` option that runs migrations as a Job. This page did not test it, so check the +chart values for your version. + +:::warning + +Init container logs exist only as long as the pod does. Once the API pod is replaced, the migration logs are gone. Read +them straight after the upgrade, or send pod logs to a log system. + +::: + +## Verify the upgrade + +- `helm history -n flagsmith flagsmith` shows the new revision with the status `deployed`. +- `kubectl -n flagsmith get pods` shows the API pod as `Running` and ready. +- The dashboard shows the new version at the bottom of the left sidebar, and your projects and flags are still there. + +## Rolling back + +:::warning + +`helm rollback` restores the previous chart revision, so the older application runs again. It does **not** undo database +migrations. If the upgrade changed the schema, the older application runs against a newer database and can fail. + +::: + +In the test, migration `code_references.0003` dropped the table `code_references_featureflagcodereferencesscan`. After +`helm rollback`, Flagsmith 2.222.0 failed on the features list with this error in the API log: + +```text +django.db.utils.ProgrammingError: relation "code_references_featureflagcodereferencesscan" does not exist +``` + +Treat a rollback as a restore. These steps worked in the test: + +1. Roll back the chart: + + ```bash + helm rollback flagsmith -n flagsmith --wait --timeout 20m + ``` + +2. Stop the API, so nothing is connected to the database, and wait until no `flagsmith-api` pod is listed: + + ```bash + kubectl -n flagsmith scale deployment flagsmith-api --replicas=0 + kubectl -n flagsmith get pods + ``` + +3. Delete the database and create an empty one. This destroys the current data, so only do it once you have a backup you + have checked: + + ```bash + kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD dropdb -U postgres $POSTGRES_DB' + kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD createdb -U postgres $POSTGRES_DB' + ``` + +4. Restore the backup: + + ```bash + kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres -d $POSTGRES_DB --no-owner /tmp/backup.dump' + ``` + +5. Start the API and wait for it to become ready: + + ```bash + kubectl -n flagsmith scale deployment flagsmith-api --replicas=1 + kubectl -n flagsmith rollout status deployment flagsmith-api + ``` + +Recreate the database rather than restoring over the existing one. Tables created by the newer version would otherwise +be left behind, and a later upgrade could fail when it tries to create them again. + +:::warning + +A restore returns the database to the moment of the backup. Anything saved since then is lost. In the test, a flag +created after the backup was gone after the restore. + +::: + +## Troubleshooting + +| Symptom | Likely cause | What to do | +| ---------------------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| Dashboard says you do not have permission after a rollback | The older application cannot read the newer schema | Check the API log for `ProgrammingError`. Restore the backup as described in **Rolling back**. | +| `helm install` or `helm upgrade` times out | Images are still downloading, or migrations are still running | Check `kubectl get pods` and `kubectl get events`. Retry with a longer `--timeout`. | +| No migration Job exists to read logs from | Migrations run in the `migrate-db` init container by default | Run `kubectl logs -c migrate-db` straight after the upgrade. | +| `kubectl logs -c migrate-db` returns `NotFound` | The API pod was replaced and its logs were removed | Upgrade again on a test copy and capture the logs immediately, or use a log system. | + +The permissions message in the first row is misleading. The dashboard shows it whenever a request fails, and in the test +the real cause was the missing table above. + +## Changes in chart 0.85.0 + +Chart 0.85.0 no longer deploys the bundled InfluxDB. The notes printed by `helm upgrade` list the alternatives: +configure an external InfluxDB with `influxdbExternal`, or store analytics in PostgreSQL with `UsePostgresForAnalytics` +(which requires `taskProcessor.enabled`). To move existing analytics data to PostgreSQL, enable +`jobs.migrateAnalyticsData`. + +These notes appear only in the terminal output, so check them if you upgrade through CI or a GitOps tool. This page did +not test what happens to existing analytics data. + +## What was tested + +- Upgrade, rollback and restore between chart 0.80.0 (Flagsmith 2.222.0) and chart 0.85.0 (Flagsmith 2.286.0). +- A local kind cluster, default values and the PostgreSQL instance deployed by the chart. +- A database containing one organisation, one project and a few flags. + +Not tested: custom values, an external or managed database, the `jobs.migrateDb` Job, the analytics migration, other +version pairs, and downtime during the upgrade. diff --git a/docs/docs/diagrams/upgrade-rollback-restore.svg b/docs/docs/diagrams/upgrade-rollback-restore.svg new file mode 100644 index 000000000000..85f2eacd10c4 --- /dev/null +++ b/docs/docs/diagrams/upgrade-rollback-restore.svg @@ -0,0 +1,42 @@ +AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTozOWFiN2Y5NC1hZWE0LTRmNmQtOTRhNS1lMDdkOTdlNWE3MGUAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaER9Mp2pOlbup0m+10KuMYMAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDozNzkyZTVmNi00NjBmLTRkMDgtODJmOS1kYTBhODA5NTVmMGJscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNoPpsm4OPNT0mjuBkUY//psAAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggRk5+tXXoLIEzVmS/VMRx907B4gdBvS/Pcje7wLuKo4CkZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaNcaKqaaifzmw5ciIu1yw60AAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRNAAAAAAAAAAAAAAAAAGRoYXNoWCCVm1mPPcsELdgRCuLa8Yx6NEflKhfYwfNfL2JbBLXC9mRuYW1lbmp1bWJmIG1hbmlmZXN0amV4Y2x1c2lvbnOBomVzdGFydBjJZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOjM5YWI3Zjk0LWFlYTQtNGY2ZC05NGE1LWUwN2Q5N2U1YTcwZS9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOjZkZDY5ZmRmLTNiYTItNDVhOC04ZmI4LTMyMTJhNTIxZjkxMXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCBGTn61degsgTNWZL9UxHH3TsHiB0G9L89yN7vAu4qjgKJjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFgg+m3CQoEtRbABoKKxSk4hUyEeIesbZDlhI/MhrtawqoGiY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFgg8AMXnztfq6wxmiAy6tQ6Gkluh1yxr+qRvcmrLXQUP0N0Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQBGTbPcezYzb7lue9DRr3J6D4mS18ulgBEBfOt25TUwg8oGH5ivnwv5LK/LrcK7LlTaTuAoFgPauyHImSvK0Suc= +Application and database state across an upgrade, rollback and restore +Four stages. Before: old app, old schema. After helm upgrade: new app, new schema. After helm rollback: old app, new schema, which is a mismatch. After restore: old app, old schema restored from backup. + + + +1. Beforebackup taken here + +2. helm upgrademigrations run + +3. helm rollbackchart only + +4. Restorefrom the backup + + +Application + +Database + + +Flagsmith 2.222.0 + +Flagsmith 2.286.0 + +Flagsmith 2.222.0 + +Flagsmith 2.222.0 + + +Old schema + +New schema91 migrationsapplied in the test + +New schemanot reverted by Helm + +Old schemarestored fromthe backup + + +Mismatchold app, new schema + +Works againnewer changes lost + From d24ee251e7fed0efa3eb9fbabbaa865213bff51e Mon Sep 17 00:00:00 2001 From: Mumpe73 Date: Mon, 16 Nov 2026 10:24:54 +0300 Subject: [PATCH 2/8] docs: address review feedback on upgrade guide --- .../docs/deployment-self-hosting/upgrading.md | 119 ++++++++++++++---- 1 file changed, 92 insertions(+), 27 deletions(-) diff --git a/docs/docs/deployment-self-hosting/upgrading.md b/docs/docs/deployment-self-hosting/upgrading.md index c41bca1ebdd9..fc60caaf5ebd 100644 --- a/docs/docs/deployment-self-hosting/upgrading.md +++ b/docs/docs/deployment-self-hosting/upgrading.md @@ -22,49 +22,99 @@ difference matters most when you need to roll back. helm history -n flagsmith flagsmith ``` -3. Back up the database, check the backup can be read, and store a copy outside the cluster. The commands below show how - to do this for the PostgreSQL instance deployed by the chart. +3. Note how many replicas the API runs. Look at the `READY` column, for example `2/2`, and write the number down: If + your API uses autoscaling, the autoscaler can change the replica count on its own. This page did not test + autoscaling. + + ```bash + kubectl -n flagsmith get deployment flagsmith-api + ``` + +4. Back up the database, verify the backup, and store a copy outside the cluster. The commands below show how to do this + for the PostgreSQL instance deployed by the chart. ### Back up the PostgreSQL instance deployed by the chart :::info These commands use the pod name `flagsmith-dev-postgresql-0`, the `postgres` user and the `POSTGRES_PASSWORD` and -`POSTGRES_DB` variables that were set in that pod during testing. Check them with -`kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- env`. If you use an external or managed database, use your -provider's snapshot or dump tools instead. +`POSTGRES_DB` variables that were set in that pod during testing. Find your PostgreSQL pod name with +`kubectl -n flagsmith get pods` , then check the variables with `kubectl -n flagsmith exec -- env`. If +you use an external or managed database, use your provider's snapshot or dump tools instead. ::: +The dump is written inside the PostgreSQL pod first, so check there is enough free space: + +```bash +kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- df -h /tmp +``` + +Create the backup: + ```bash kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ 'PGPASSWORD=$POSTGRES_PASSWORD pg_dump -U postgres -Fc -f /tmp/backup.dump $POSTGRES_DB' ``` -Check the backup can be read. A number greater than zero means the file contains your tables: +Copy it out of the cluster, so it survives if the pod is replaced: ```bash -kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'pg_restore -l /tmp/backup.dump | grep -c TABLE' +kubectl cp flagsmith/flagsmith-dev-postgresql-0:/tmp/backup.dump ./backup.dump ``` -Copy the backup out of the cluster: +Verify the backup by restoring it into a scratch database. If this prints no errors, the backup can be restored: ```bash -kubectl cp flagsmith/flagsmith-dev-postgresql-0:/tmp/backup.dump ./backup.dump +kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD createdb -U postgres flagsmith_verify' +kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres -d flagsmith_verify --single-transaction /tmp/backup.dump' +``` + +Compare the number of tables in the scratch database with the live one. The two numbers should match: + +```bash +kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d flagsmith_verify -c \\dt | grep -c table' +kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d $POSTGRES_DB -c \\dt | grep -c table' +``` + +Then remove the scratch database: + +```bash +kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD dropdb -U postgres flagsmith_verify' ``` ## Upgrade with Helm ```bash -helm upgrade -n flagsmith flagsmith flagsmith/flagsmith --version -f values.yaml --wait --timeout 20m +helm upgrade -n flagsmith flagsmith flagsmith/flagsmith --version --reset-then-reuse-values \ + --wait --timeout 20m ``` -Pass the same values file you installed with. Without `-f` or `--reuse-values`, Helm applies the chart defaults and -discards values you set earlier. The test used default values only, so this behaviour was not tested here. +`--reset-then-reuse-values` needs Helm 3.14 or later. It keeps the values you set when you installed and applies the new +chart's defaults for everything else. Avoid plain `--reuse-values`, which ignores defaults introduced by the new chart +version and can make an upgrade across versions fail. Values you pass with -f or --set are applied on top of the values +from your install, so only pass the ones you want to change. The test used default values only, so this behaviour was +not tested with custom values. + +`--wait` prints nothing until the rollout is ready. Helm treats a Deployment as ready when the minimum number of +replicas is available (the desired count minus `maxUnavailable`), so check `kubectl -n flagsmith get pods` to confirm +that every replica is ready. Downloading images can take several minutes on a slow connection. In the test, a first +install with a 10 minute timeout failed because the PostgreSQL image took more than five minutes to download, even +though the pods were healthy shortly afterwards. + +:::warning + +During a rolling upgrade, old API pods keep serving requests while the new pod runs its migrations. If a migration +removes something the old version needs, requests handled by an old pod can fail until it is replaced. In the test, +migration `code_references.0003` dropped a table that Flagsmith 2.222.0 reads. If you cannot accept errors during this +window, schedule the upgrade for a quiet period. This was not tested. -`--wait` prints nothing until every pod is ready. Downloading images can take several minutes on a slow connection. In -the test, a first install with a 10 minute timeout failed because the PostgreSQL image took more than five minutes to -download, even though the pods were healthy shortly afterwards. +::: ## How migrations run @@ -133,25 +183,39 @@ Treat a rollback as a restore. These steps worked in the test: kubectl -n flagsmith get pods ``` -3. Delete the database and create an empty one. This destroys the current data, so only do it once you have a backup you - have checked: +3. Copy the backup into the PostgreSQL pod and check it is there. The pod may have been replaced since you took the + backup, so do this even if you think the file is still in `/tmp`: ```bash - kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD dropdb -U postgres $POSTGRES_DB' - kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD createdb -U postgres $POSTGRES_DB' + kubectl cp ./backup.dump flagsmith/flagsmith-dev-postgresql-0:/tmp/backup.dump + kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- ls -l /tmp/backup.dump ``` -4. Restore the backup: +4. Delete the database and create an empty one. This destroys the current data, so only do it once you have a backup you + have verified: ```bash kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres -d $POSTGRES_DB --no-owner /tmp/backup.dump' + 'PGPASSWORD=$POSTGRES_PASSWORD dropdb -U postgres $POSTGRES_DB' + kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD createdb -U postgres $POSTGRES_DB' ``` -5. Start the API and wait for it to become ready: +5. Restore the backup. `--single-transaction` makes the restore all or nothing, so a failure cannot leave a half + restored database. If the command reports any error, do not start the API. Fix the cause and run it again: + + ```bash + kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ + 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres -d $POSTGRES_DB --single-transaction /tmp/backup.dump' + ``` + + `pg_restore` recreates the original table owners, so the database roles must already exist. They do when you restore + into the same database server. + +6. Start the API with the replica count you noted before the upgrade, and wait for it to become ready: ```bash - kubectl -n flagsmith scale deployment flagsmith-api --replicas=1 + kubectl -n flagsmith scale deployment flagsmith-api --replicas= kubectl -n flagsmith rollout status deployment flagsmith-api ``` @@ -181,11 +245,12 @@ the real cause was the missing table above. Chart 0.85.0 no longer deploys the bundled InfluxDB. The notes printed by `helm upgrade` list the alternatives: configure an external InfluxDB with `influxdbExternal`, or store analytics in PostgreSQL with `UsePostgresForAnalytics` -(which requires `taskProcessor.enabled`). To move existing analytics data to PostgreSQL, enable -`jobs.migrateAnalyticsData`. +(which requires `taskProcessor.enabled`). The notes also mention `jobs.migrateAnalyticsData` for moving existing +analytics data. These notes appear only in the terminal output, so check them if you upgrade through CI or a GitOps tool. This page did -not test what happens to existing analytics data. +not test the analytics migration or what happens to existing analytics data, so check the chart values for the settings +that job needs, including the connection to the source InfluxDB. ## What was tested @@ -194,4 +259,4 @@ not test what happens to existing analytics data. - A database containing one organisation, one project and a few flags. Not tested: custom values, an external or managed database, the `jobs.migrateDb` Job, the analytics migration, other -version pairs, and downtime during the upgrade. +version pairs, and the behaviour of the API and downtime during a rolling upgrade. From ed2292498364a2d70e4b399186e33e69dd492495 Mon Sep 17 00:00:00 2001 From: Mumpe73 Date: Mon, 16 Nov 2026 12:26:23 +0300 Subject: [PATCH 3/8] docs: stop all database clients before restore --- docs/docs/deployment-self-hosting/upgrading.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/docs/deployment-self-hosting/upgrading.md b/docs/docs/deployment-self-hosting/upgrading.md index fc60caaf5ebd..22894a5e4c9a 100644 --- a/docs/docs/deployment-self-hosting/upgrading.md +++ b/docs/docs/deployment-self-hosting/upgrading.md @@ -176,7 +176,10 @@ Treat a rollback as a restore. These steps worked in the test: helm rollback flagsmith -n flagsmith --wait --timeout 20m ``` -2. Stop the API, so nothing is connected to the database, and wait until no `flagsmith-api` pod is listed: +2. Stop the API, so nothing is connected to the database, and wait until no `flagsmith-api` pod is listed: Stop + everything that connects to the database, and wait until none of those pods is listed. In the default install only + the API connects. If you enabled the task processor, scale it to 0 as well, and wait for any migration Job to finish. + These setups were not tested. ```bash kubectl -n flagsmith scale deployment flagsmith-api --replicas=0 @@ -192,7 +195,9 @@ Treat a rollback as a restore. These steps worked in the test: ``` 4. Delete the database and create an empty one. This destroys the current data, so only do it once you have a backup you - have verified: + have verified: Run these commands one at a time, and check each one finishes without an error before you run the + next. If dropdb fails because the database is in use, stop. Do not run createdb or pg_restore until you have found + what is still connected. ```bash kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ From 225825fc1d28328788002d0a2a50068e4377abbd Mon Sep 17 00:00:00 2001 From: Mumpe73 Date: Mon, 16 Nov 2026 12:31:32 +0300 Subject: [PATCH 4/8] createdb update --- docs/docs/deployment-self-hosting/upgrading.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/deployment-self-hosting/upgrading.md b/docs/docs/deployment-self-hosting/upgrading.md index 22894a5e4c9a..a0097511b9ee 100644 --- a/docs/docs/deployment-self-hosting/upgrading.md +++ b/docs/docs/deployment-self-hosting/upgrading.md @@ -81,7 +81,7 @@ kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d $POSTGRES_DB -c \\dt | grep -c table' ``` -Then remove the scratch database: +Then remove the scratch database. Run this command only if createdb succeeded. If createdb failed, the database already existed, so do not drop it. ```bash kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ From d49da3d2ccdc5bf5e4ac8f0953c31f76df29aca8 Mon Sep 17 00:00:00 2001 From: Mumpe73 Date: Mon, 16 Nov 2026 12:35:36 +0300 Subject: [PATCH 5/8] PostgreSQL --- docs/docs/deployment-self-hosting/upgrading.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/docs/deployment-self-hosting/upgrading.md b/docs/docs/deployment-self-hosting/upgrading.md index a0097511b9ee..0f26a1b7a7d0 100644 --- a/docs/docs/deployment-self-hosting/upgrading.md +++ b/docs/docs/deployment-self-hosting/upgrading.md @@ -49,6 +49,7 @@ The dump is written inside the PostgreSQL pod first, so check there is enough fr ```bash kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- df -h /tmp ``` +This checks the space for the dump file. The scratch restore below also creates a full second copy of the database on the volume where PostgreSQL stores its data, so check that volume has more free space than the database uses. If it does not, skip the scratch restore or run it on a separate PostgreSQL server. This was not tested. Create the backup: @@ -81,7 +82,7 @@ kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d $POSTGRES_DB -c \\dt | grep -c table' ``` -Then remove the scratch database. Run this command only if createdb succeeded. If createdb failed, the database already existed, so do not drop it. +Then remove the scratch database. Run this command only if `createdb` succeeded. If `createdb` failed, the database already existed, so do not drop it. ```bash kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ From 679d188d08d8a7281c77c70f71d1a0a16c944523 Mon Sep 17 00:00:00 2001 From: Mumpe73 Date: Mon, 16 Nov 2026 12:45:43 +0300 Subject: [PATCH 6/8] docs: restart task processor after restore --- docs/docs/deployment-self-hosting/upgrading.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/docs/deployment-self-hosting/upgrading.md b/docs/docs/deployment-self-hosting/upgrading.md index 0f26a1b7a7d0..8770ee56f229 100644 --- a/docs/docs/deployment-self-hosting/upgrading.md +++ b/docs/docs/deployment-self-hosting/upgrading.md @@ -177,10 +177,8 @@ Treat a rollback as a restore. These steps worked in the test: helm rollback flagsmith -n flagsmith --wait --timeout 20m ``` -2. Stop the API, so nothing is connected to the database, and wait until no `flagsmith-api` pod is listed: Stop - everything that connects to the database, and wait until none of those pods is listed. In the default install only - the API connects. If you enabled the task processor, scale it to 0 as well, and wait for any migration Job to finish. - These setups were not tested. +2. Stop the API, so nothing is connected to the database, and wait until no `flagsmith-api` pod is listed: + Stop everything that connects to the database, and wait until none of those pods is listed. In the default install only the API connects. If you enabled the task processor, note its replica count, then scale it to 0 as well, and wait for any migration Job to finish. These setups were not tested. ```bash kubectl -n flagsmith scale deployment flagsmith-api --replicas=0 @@ -224,7 +222,7 @@ Treat a rollback as a restore. These steps worked in the test: kubectl -n flagsmith scale deployment flagsmith-api --replicas= kubectl -n flagsmith rollout status deployment flagsmith-api ``` - +If you stopped the task processor, scale it back to the replica count you noted, so background tasks run again. This was not tested. Recreate the database rather than restoring over the existing one. Tables created by the newer version would otherwise be left behind, and a later upgrade could fail when it tries to create them again. From 35369765c15dd592246e918b46aaf9453520f199 Mon Sep 17 00:00:00 2001 From: Mumpe73 Date: Tue, 17 Nov 2026 09:44:11 +0300 Subject: [PATCH 7/8] docs: do not skip backup verification --- .../docs/deployment-self-hosting/upgrading.md | 22 +++++++++++++------ 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/docs/deployment-self-hosting/upgrading.md b/docs/docs/deployment-self-hosting/upgrading.md index 8770ee56f229..ed8e54abffad 100644 --- a/docs/docs/deployment-self-hosting/upgrading.md +++ b/docs/docs/deployment-self-hosting/upgrading.md @@ -49,7 +49,11 @@ The dump is written inside the PostgreSQL pod first, so check there is enough fr ```bash kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- df -h /tmp ``` -This checks the space for the dump file. The scratch restore below also creates a full second copy of the database on the volume where PostgreSQL stores its data, so check that volume has more free space than the database uses. If it does not, skip the scratch restore or run it on a separate PostgreSQL server. This was not tested. + +This checks the space for the dump file. The scratch restore below also creates a full second copy of the database on +the volume where PostgreSQL stores its data, so check that volume has more free space than the database uses. If it does +not, run the scratch restore on a separate PostgreSQL server with enough free space. Do not skip the verification, +because the rollback steps need a verified backup. This was not tested. Create the backup: @@ -82,7 +86,8 @@ kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d $POSTGRES_DB -c \\dt | grep -c table' ``` -Then remove the scratch database. Run this command only if `createdb` succeeded. If `createdb` failed, the database already existed, so do not drop it. +Then remove the scratch database. Run this command only if `createdb` succeeded. If `createdb` failed, the database +already existed, so do not drop it. ```bash kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ @@ -177,8 +182,10 @@ Treat a rollback as a restore. These steps worked in the test: helm rollback flagsmith -n flagsmith --wait --timeout 20m ``` -2. Stop the API, so nothing is connected to the database, and wait until no `flagsmith-api` pod is listed: - Stop everything that connects to the database, and wait until none of those pods is listed. In the default install only the API connects. If you enabled the task processor, note its replica count, then scale it to 0 as well, and wait for any migration Job to finish. These setups were not tested. +2. Stop the API, so nothing is connected to the database, and wait until no `flagsmith-api` pod is listed: Stop + everything that connects to the database, and wait until none of those pods is listed. In the default install only + the API connects. If you enabled the task processor, note its replica count, then scale it to 0 as well, and wait for + any migration Job to finish. These setups were not tested. ```bash kubectl -n flagsmith scale deployment flagsmith-api --replicas=0 @@ -222,9 +229,10 @@ Treat a rollback as a restore. These steps worked in the test: kubectl -n flagsmith scale deployment flagsmith-api --replicas= kubectl -n flagsmith rollout status deployment flagsmith-api ``` -If you stopped the task processor, scale it back to the replica count you noted, so background tasks run again. This was not tested. -Recreate the database rather than restoring over the existing one. Tables created by the newer version would otherwise -be left behind, and a later upgrade could fail when it tries to create them again. + + If you stopped the task processor, scale it back to the replica count you noted, so background tasks run again. This + was not tested. Recreate the database rather than restoring over the existing one. Tables created by the newer + version would otherwise be left behind, and a later upgrade could fail when it tries to create them again. :::warning From ac778023091ae6a586f4c866404e9b31accf928f Mon Sep 17 00:00:00 2001 From: Mumpe73 Date: Thu, 19 Nov 2026 16:01:39 +0300 Subject: [PATCH 8/8] docs: merge Helm upgrade guide into upgrades-and-rollbacks --- .../upgrades-and-rollbacks.md | 141 ++++++++- .../docs/deployment-self-hosting/upgrading.md | 274 ------------------ .../diagrams/upgrade-rollback-restore.svg | 20 +- 3 files changed, 146 insertions(+), 289 deletions(-) delete mode 100644 docs/docs/deployment-self-hosting/upgrading.md diff --git a/docs/docs/deployment-self-hosting/administration-and-maintenance/upgrades-and-rollbacks.md b/docs/docs/deployment-self-hosting/administration-and-maintenance/upgrades-and-rollbacks.md index e8eeecefcd94..a164733015f9 100644 --- a/docs/docs/deployment-self-hosting/administration-and-maintenance/upgrades-and-rollbacks.md +++ b/docs/docs/deployment-self-hosting/administration-and-maintenance/upgrades-and-rollbacks.md @@ -8,11 +8,15 @@ sidebar_position: 2 :::warning -These steps may result in data loss in the scenario where new models or fields have been added to the database. We recommend taking a full backup of the database before completing the rollback. +These steps may result in data loss in the scenario where new models or fields have been added to the database. We +recommend taking a full backup of the database before completing the rollback. ::: -This page covers the process of rolling back to a previous version of Flagsmith. If you need to roll back to a previous version, you will need to ensure that the database is also rolled back to the correct state. In order to do this, you will need to unapply all the migrations that happened between the version that you want to roll back to and the one that you are rolling back from. The following steps explain how to do that. +This page covers the process of rolling back to a previous version of Flagsmith. If you need to roll back to a previous +version, you will need to ensure that the database is also rolled back to the correct state. In order to do this, you +will need to unapply all the migrations that happened between the version that you want to roll back to and the one that +you are rolling back from. The following steps explain how to do that. ## Steps for v2.151.0 and Later @@ -20,7 +24,9 @@ This page covers the process of rolling back to a previous version of Flagsmith. :::tip -If you are unsure about when you completed the previous deployment, you can use the `django_migrations` table as a guide. If you query the table using the following query, you should see the migrations that have been applied (in descending order), grouped in batches corresponding to each deployment. +If you are unsure about when you completed the previous deployment, you can use the `django_migrations` table as a +guide. If you query the table using the following query, you should see the migrations that have been applied (in +descending order), grouped in batches corresponding to each deployment. ```sql SELECT * @@ -40,11 +46,13 @@ python manage.py rollbackmigrationsafter "" ## Steps for Versions Earlier Than v2.151.0 -If you are rolling back from a version earlier than v2.151.0, you will need to replace step 2 above with the following two steps. +If you are rolling back from a version earlier than v2.151.0, you will need to replace step 2 above with the following +two steps. ### Step 1: Generate Rollback Commands -Replace the datetime in the query below with a datetime after the deployment of the version you want to roll back to, and before any subsequent deployments. Execute the subsequent query against the Flagsmith database. +Replace the datetime in the query below with a datetime after the deployment of the version you want to roll back to, +and before any subsequent deployments. Execute the subsequent query against the Flagsmith database. ```sql {14} showLineNumbers select @@ -80,3 +88,126 @@ Example output: ### Step 2: Execute the Rollback Commands Run the generated commands inside a Flagsmith API container running the _current_ version of Flagsmith. + +## Upgrading and Rolling Back with Helm + +This section covers a Flagsmith deployment that was installed with the Flagsmith Helm chart. An upgrade changes two +things: the application and the database schema. Helm manages the first, but not the second. That difference matters +most when you need to roll back. + +![The application and database at each stage of an upgrade and rollback](../../diagrams/upgrade-rollback-restore.svg) + +### Before you upgrade + +1. Read the release notes for every version between the one you run and the one you are moving to. +2. Note your current Helm revision, so you know what to roll back to: + + ```bash + helm history -n flagsmith flagsmith + ``` + +3. Back up the database and test that the backup restores. See the + [PostgreSQL documentation](https://www.postgresql.org/docs/current/backup.html). + +### Upgrade with Helm + +```bash +helm upgrade -n flagsmith flagsmith flagsmith/flagsmith --version --reset-then-reuse-values \ + --wait --timeout 20m +``` + +`--reset-then-reuse-values` needs Helm 3.14 or later. It keeps the values you set when you installed and applies the new +chart's defaults for everything else. Avoid plain `--reuse-values`, which ignores defaults introduced by the new chart +version and can make an upgrade across versions fail. Values you pass with `-f` or `--set` are applied on top of the +values from your install, so only pass the ones you want to change. + +`--wait` prints nothing until the rollout is ready. Helm treats a Deployment as ready when the minimum number of +replicas is available (the desired count minus `maxUnavailable`), so check `kubectl -n flagsmith get pods` to confirm +that every replica is ready. Downloading images can take several minutes on a slow connection, so allow a generous +`--timeout`. + +:::warning + +During a rolling upgrade, old API pods keep serving requests while the new pod runs its migrations. If a migration +removes something the old version needs, requests handled by an old pod can fail until it is replaced. If you cannot +accept errors during this window, schedule the upgrade for a quiet period. + +::: + +### How migrations run + +With the chart's default values, database migrations run in an init container named `migrate-db` in the API pod, and the +API starts only after it finishes. To read its output, find the new API pod and request the init container's logs: + +```bash +kubectl -n flagsmith get pods +kubectl -n flagsmith logs -c migrate-db +``` + +A successful run lists each migration it applied. If the database is already up to date, the log says +`No migrations to apply.` + +The chart also has a `jobs.migrateDb` option that runs migrations as a Job. Check the chart values for your version. + +:::warning + +Init container logs exist only as long as the pod does. Once the API pod is replaced, the migration logs are gone. Read +them straight after the upgrade, or send pod logs to a log system. + +::: + +### Verify the upgrade + +- `helm history -n flagsmith flagsmith` shows the new revision with the status `deployed`. +- `kubectl -n flagsmith get pods` shows the API pod as `Running` and ready. +- The dashboard shows the new version at the bottom of the left sidebar, and your projects and flags are still there. + +### Rolling back with Helm + +:::warning + +`helm rollback` restores the previous chart revision, so the older application runs again. It does **not** undo database +migrations. If the upgrade changed the schema, the older application runs against a newer database and can fail. + +::: + +If you run only `helm rollback`, the API log can show an error like this: + +```text +django.db.utils.ProgrammingError: relation "code_references_featureflagcodereferencesscan" does not exist +``` + +To roll back safely, unapply the database migrations first, then roll back the chart. You do not need to stop the API. + +1. Pick a time after the previous deployment finished and before you ran `helm upgrade`, as described in + [Steps for v2.151.0 and Later](#steps-for-v21510-and-later). Flagsmith records migration times in UTC, but + `helm history` shows your local time, so convert it to UTC. +2. Run the rollback command in an API pod that runs the version you are rolling back from. It unapplies every migration + applied on or after the time you give, which must be in ISO format: + + ```bash + kubectl -n flagsmith exec deploy/flagsmith-api -- python manage.py rollbackmigrationsappliedafter "2026-11-19T10:12:03+00:00" + ``` + +3. Roll back the chart: + + ```bash + helm rollback flagsmith -n flagsmith --wait --timeout 20m + ``` + +If the rollback command fails partway, some migrations stay applied. In that case, restore the backup you took before +the upgrade. See the [PostgreSQL documentation](https://www.postgresql.org/docs/current/backup.html). + +If you are rolling back from a version earlier than v2.151.0, replace step 2 with the steps in +[Steps for Versions Earlier Than v2.151.0](#steps-for-versions-earlier-than-v21510). + +### Troubleshooting + +| Symptom | Likely cause | What to do | +| ---------------------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | +| Dashboard says you do not have permission after a rollback | The older application cannot read the newer schema | Check the API log for `ProgrammingError`. Upgrade to the newer chart version again, then follow **Rolling back with Helm**. | +| `helm install` or `helm upgrade` times out | Images are still downloading, or migrations are still running | Check `kubectl get pods` and `kubectl get events`. Retry with a longer `--timeout`. | +| No migration Job exists to read logs from | Migrations run in the `migrate-db` init container by default | Run `kubectl logs -c migrate-db` straight after the upgrade. | +| `kubectl logs -c migrate-db` returns `NotFound` | The API pod was replaced and its logs were removed | Upgrade again on a test copy and capture the logs immediately, or use a log system. | + +The permissions message in the first row is misleading. The dashboard shows it whenever a request fails. diff --git a/docs/docs/deployment-self-hosting/upgrading.md b/docs/docs/deployment-self-hosting/upgrading.md deleted file mode 100644 index ed8e54abffad..000000000000 --- a/docs/docs/deployment-self-hosting/upgrading.md +++ /dev/null @@ -1,274 +0,0 @@ ---- -sidebar_label: Upgrading -title: Upgrading and rolling back -sidebar_position: 4 ---- - -This page explains how to upgrade a self-hosted Flagsmith deployment that was installed with the Flagsmith Helm chart, -and what to do if you need to go back. It is based on a test upgrade from chart 0.80.0 (Flagsmith 2.222.0) to chart -0.85.0 (Flagsmith 2.286.0) on a local [kind](https://kind.sigs.k8s.io/) cluster with default values. - -An upgrade changes two things: the application and the database schema. Helm manages the first, but not the second. That -difference matters most when you need to roll back. - -![The application and database at each stage of an upgrade, rollback and restore](../diagrams/upgrade-rollback-restore.svg) - -## Before you upgrade - -1. Read the release notes for every version between the one you run and the one you are moving to. -2. Note your current Helm revision, so you know what to roll back to: - - ```bash - helm history -n flagsmith flagsmith - ``` - -3. Note how many replicas the API runs. Look at the `READY` column, for example `2/2`, and write the number down: If - your API uses autoscaling, the autoscaler can change the replica count on its own. This page did not test - autoscaling. - - ```bash - kubectl -n flagsmith get deployment flagsmith-api - ``` - -4. Back up the database, verify the backup, and store a copy outside the cluster. The commands below show how to do this - for the PostgreSQL instance deployed by the chart. - -### Back up the PostgreSQL instance deployed by the chart - -:::info - -These commands use the pod name `flagsmith-dev-postgresql-0`, the `postgres` user and the `POSTGRES_PASSWORD` and -`POSTGRES_DB` variables that were set in that pod during testing. Find your PostgreSQL pod name with -`kubectl -n flagsmith get pods` , then check the variables with `kubectl -n flagsmith exec -- env`. If -you use an external or managed database, use your provider's snapshot or dump tools instead. - -::: - -The dump is written inside the PostgreSQL pod first, so check there is enough free space: - -```bash -kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- df -h /tmp -``` - -This checks the space for the dump file. The scratch restore below also creates a full second copy of the database on -the volume where PostgreSQL stores its data, so check that volume has more free space than the database uses. If it does -not, run the scratch restore on a separate PostgreSQL server with enough free space. Do not skip the verification, -because the rollback steps need a verified backup. This was not tested. - -Create the backup: - -```bash -kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD pg_dump -U postgres -Fc -f /tmp/backup.dump $POSTGRES_DB' -``` - -Copy it out of the cluster, so it survives if the pod is replaced: - -```bash -kubectl cp flagsmith/flagsmith-dev-postgresql-0:/tmp/backup.dump ./backup.dump -``` - -Verify the backup by restoring it into a scratch database. If this prints no errors, the backup can be restored: - -```bash -kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD createdb -U postgres flagsmith_verify' -kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres -d flagsmith_verify --single-transaction /tmp/backup.dump' -``` - -Compare the number of tables in the scratch database with the live one. The two numbers should match: - -```bash -kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d flagsmith_verify -c \\dt | grep -c table' -kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD psql -U postgres -d $POSTGRES_DB -c \\dt | grep -c table' -``` - -Then remove the scratch database. Run this command only if `createdb` succeeded. If `createdb` failed, the database -already existed, so do not drop it. - -```bash -kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD dropdb -U postgres flagsmith_verify' -``` - -## Upgrade with Helm - -```bash -helm upgrade -n flagsmith flagsmith flagsmith/flagsmith --version --reset-then-reuse-values \ - --wait --timeout 20m -``` - -`--reset-then-reuse-values` needs Helm 3.14 or later. It keeps the values you set when you installed and applies the new -chart's defaults for everything else. Avoid plain `--reuse-values`, which ignores defaults introduced by the new chart -version and can make an upgrade across versions fail. Values you pass with -f or --set are applied on top of the values -from your install, so only pass the ones you want to change. The test used default values only, so this behaviour was -not tested with custom values. - -`--wait` prints nothing until the rollout is ready. Helm treats a Deployment as ready when the minimum number of -replicas is available (the desired count minus `maxUnavailable`), so check `kubectl -n flagsmith get pods` to confirm -that every replica is ready. Downloading images can take several minutes on a slow connection. In the test, a first -install with a 10 minute timeout failed because the PostgreSQL image took more than five minutes to download, even -though the pods were healthy shortly afterwards. - -:::warning - -During a rolling upgrade, old API pods keep serving requests while the new pod runs its migrations. If a migration -removes something the old version needs, requests handled by an old pod can fail until it is replaced. In the test, -migration `code_references.0003` dropped a table that Flagsmith 2.222.0 reads. If you cannot accept errors during this -window, schedule the upgrade for a quiet period. This was not tested. - -::: - -## How migrations run - -With the chart's default values, there is no migration Job. Database migrations run in an init container named -`migrate-db` in the API pod, and the API starts only after it finishes. To read its output, find the new API pod and -request the init container's logs: - -```bash -kubectl -n flagsmith get pods -kubectl -n flagsmith logs -c migrate-db -``` - -A successful run lists each migration it applied. For example: - -```text -Running migrations: - Applying code_references.0003_introduce_per_feature_scanned_references... OK -``` - -If the database is already up to date, the log says `No migrations to apply.` - -The chart also has a `jobs.migrateDb` option that runs migrations as a Job. This page did not test it, so check the -chart values for your version. - -:::warning - -Init container logs exist only as long as the pod does. Once the API pod is replaced, the migration logs are gone. Read -them straight after the upgrade, or send pod logs to a log system. - -::: - -## Verify the upgrade - -- `helm history -n flagsmith flagsmith` shows the new revision with the status `deployed`. -- `kubectl -n flagsmith get pods` shows the API pod as `Running` and ready. -- The dashboard shows the new version at the bottom of the left sidebar, and your projects and flags are still there. - -## Rolling back - -:::warning - -`helm rollback` restores the previous chart revision, so the older application runs again. It does **not** undo database -migrations. If the upgrade changed the schema, the older application runs against a newer database and can fail. - -::: - -In the test, migration `code_references.0003` dropped the table `code_references_featureflagcodereferencesscan`. After -`helm rollback`, Flagsmith 2.222.0 failed on the features list with this error in the API log: - -```text -django.db.utils.ProgrammingError: relation "code_references_featureflagcodereferencesscan" does not exist -``` - -Treat a rollback as a restore. These steps worked in the test: - -1. Roll back the chart: - - ```bash - helm rollback flagsmith -n flagsmith --wait --timeout 20m - ``` - -2. Stop the API, so nothing is connected to the database, and wait until no `flagsmith-api` pod is listed: Stop - everything that connects to the database, and wait until none of those pods is listed. In the default install only - the API connects. If you enabled the task processor, note its replica count, then scale it to 0 as well, and wait for - any migration Job to finish. These setups were not tested. - - ```bash - kubectl -n flagsmith scale deployment flagsmith-api --replicas=0 - kubectl -n flagsmith get pods - ``` - -3. Copy the backup into the PostgreSQL pod and check it is there. The pod may have been replaced since you took the - backup, so do this even if you think the file is still in `/tmp`: - - ```bash - kubectl cp ./backup.dump flagsmith/flagsmith-dev-postgresql-0:/tmp/backup.dump - kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- ls -l /tmp/backup.dump - ``` - -4. Delete the database and create an empty one. This destroys the current data, so only do it once you have a backup you - have verified: Run these commands one at a time, and check each one finishes without an error before you run the - next. If dropdb fails because the database is in use, stop. Do not run createdb or pg_restore until you have found - what is still connected. - - ```bash - kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD dropdb -U postgres $POSTGRES_DB' - kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD createdb -U postgres $POSTGRES_DB' - ``` - -5. Restore the backup. `--single-transaction` makes the restore all or nothing, so a failure cannot leave a half - restored database. If the command reports any error, do not start the API. Fix the cause and run it again: - - ```bash - kubectl -n flagsmith exec flagsmith-dev-postgresql-0 -- sh -c \ - 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres -d $POSTGRES_DB --single-transaction /tmp/backup.dump' - ``` - - `pg_restore` recreates the original table owners, so the database roles must already exist. They do when you restore - into the same database server. - -6. Start the API with the replica count you noted before the upgrade, and wait for it to become ready: - - ```bash - kubectl -n flagsmith scale deployment flagsmith-api --replicas= - kubectl -n flagsmith rollout status deployment flagsmith-api - ``` - - If you stopped the task processor, scale it back to the replica count you noted, so background tasks run again. This - was not tested. Recreate the database rather than restoring over the existing one. Tables created by the newer - version would otherwise be left behind, and a later upgrade could fail when it tries to create them again. - -:::warning - -A restore returns the database to the moment of the backup. Anything saved since then is lost. In the test, a flag -created after the backup was gone after the restore. - -::: - -## Troubleshooting - -| Symptom | Likely cause | What to do | -| ---------------------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | -| Dashboard says you do not have permission after a rollback | The older application cannot read the newer schema | Check the API log for `ProgrammingError`. Restore the backup as described in **Rolling back**. | -| `helm install` or `helm upgrade` times out | Images are still downloading, or migrations are still running | Check `kubectl get pods` and `kubectl get events`. Retry with a longer `--timeout`. | -| No migration Job exists to read logs from | Migrations run in the `migrate-db` init container by default | Run `kubectl logs -c migrate-db` straight after the upgrade. | -| `kubectl logs -c migrate-db` returns `NotFound` | The API pod was replaced and its logs were removed | Upgrade again on a test copy and capture the logs immediately, or use a log system. | - -The permissions message in the first row is misleading. The dashboard shows it whenever a request fails, and in the test -the real cause was the missing table above. - -## Changes in chart 0.85.0 - -Chart 0.85.0 no longer deploys the bundled InfluxDB. The notes printed by `helm upgrade` list the alternatives: -configure an external InfluxDB with `influxdbExternal`, or store analytics in PostgreSQL with `UsePostgresForAnalytics` -(which requires `taskProcessor.enabled`). The notes also mention `jobs.migrateAnalyticsData` for moving existing -analytics data. - -These notes appear only in the terminal output, so check them if you upgrade through CI or a GitOps tool. This page did -not test the analytics migration or what happens to existing analytics data, so check the chart values for the settings -that job needs, including the connection to the source InfluxDB. - -## What was tested - -- Upgrade, rollback and restore between chart 0.80.0 (Flagsmith 2.222.0) and chart 0.85.0 (Flagsmith 2.286.0). -- A local kind cluster, default values and the PostgreSQL instance deployed by the chart. -- A database containing one organisation, one project and a few flags. - -Not tested: custom values, an external or managed database, the `jobs.migrateDb` Job, the analytics migration, other -version pairs, and the behaviour of the API and downtime during a rolling upgrade. diff --git a/docs/docs/diagrams/upgrade-rollback-restore.svg b/docs/docs/diagrams/upgrade-rollback-restore.svg index 85f2eacd10c4..3ed73722e164 100644 --- a/docs/docs/diagrams/upgrade-rollback-restore.svg +++ b/docs/docs/diagrams/upgrade-rollback-restore.svg @@ -1,6 +1,6 @@ -AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTozOWFiN2Y5NC1hZWE0LTRmNmQtOTRhNS1lMDdkOTdlNWE3MGUAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaER9Mp2pOlbup0m+10KuMYMAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDozNzkyZTVmNi00NjBmLTRkMDgtODJmOS1kYTBhODA5NTVmMGJscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNoPpsm4OPNT0mjuBkUY//psAAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggRk5+tXXoLIEzVmS/VMRx907B4gdBvS/Pcje7wLuKo4CkZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaNcaKqaaifzmw5ciIu1yw60AAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRNAAAAAAAAAAAAAAAAAGRoYXNoWCCVm1mPPcsELdgRCuLa8Yx6NEflKhfYwfNfL2JbBLXC9mRuYW1lbmp1bWJmIG1hbmlmZXN0amV4Y2x1c2lvbnOBomVzdGFydBjJZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOjM5YWI3Zjk0LWFlYTQtNGY2ZC05NGE1LWUwN2Q5N2U1YTcwZS9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOjZkZDY5ZmRmLTNiYTItNDVhOC04ZmI4LTMyMTJhNTIxZjkxMXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCBGTn61degsgTNWZL9UxHH3TsHiB0G9L89yN7vAu4qjgKJjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFgg+m3CQoEtRbABoKKxSk4hUyEeIesbZDlhI/MhrtawqoGiY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFgg8AMXnztfq6wxmiAy6tQ6Gkluh1yxr+qRvcmrLXQUP0N0Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQBGTbPcezYzb7lue9DRr3J6D4mS18ulgBEBfOt25TUwg8oGH5ivnwv5LK/LrcK7LlTaTuAoFgPauyHImSvK0Suc= + Application and database state across an upgrade, rollback and restore -Four stages. Before: old app, old schema. After helm upgrade: new app, new schema. After helm rollback: old app, new schema, which is a mismatch. After restore: old app, old schema restored from backup. +Four stages. Before: old app, old schema. After helm upgrade: new app, new schema. After helm rollback: old app, new schema, which is a mismatch. After rolling back the migrations and then the chart: old app, old schema. @@ -10,7 +10,7 @@ 3. helm rollbackchart only -4. Restorefrom the backup +4. Migrations firstthen helm rollback Application @@ -18,25 +18,25 @@ Database -Flagsmith 2.222.0 +Old version -Flagsmith 2.286.0 +New version -Flagsmith 2.222.0 +Old version -Flagsmith 2.222.0 +Old version Old schema -New schema91 migrationsapplied in the test +New schemamigrations applied New schemanot reverted by Helm -Old schemarestored fromthe backup +Old schemamigrationsunapplied Mismatchold app, new schema -Works againnewer changes lost +Works againapp and schema match