diff --git a/deploy/README.md b/deploy/README.md index afbdf023d7..e800390e35 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -91,6 +91,11 @@ shows how they fit together. - **No compression** is configured. Compress at the TLS proxy or CDN in front of web. - **nginx looks up `RAILS_UPSTREAM` once, when it starts.** If Rails comes back at a different address, restart web. +## Moving a CloudFront deployment over + +[staged-rollout.md](staged-rollout.md) describes switching convention domains from the Rails-rendered shell to the +static shell one at a time: the order, what to check, what to watch and how to roll back. + ## Releases and stale pages Everything in `assets/` is named by a hash of its contents and cached for 180 days. `index.html` is the one file whose diff --git a/deploy/staged-rollout.md b/deploy/staged-rollout.md new file mode 100644 index 0000000000..7ba611e8f0 --- /dev/null +++ b/deploy/staged-rollout.md @@ -0,0 +1,107 @@ +# Staged rollout of the framework-mode build + +This is the plan for moving a deployment that uses the [`cloudfront_intercode`](../terraform/modules/cloudfront_intercode) +module from the Rails-rendered shell (the data-mode build) to the static framework-mode shell served from S3, one +convention domain at a time. It is part of the migration tracked in +[#12219](https://github.com/neinteractiveliterature/intercode/issues/12219) and +[#12260](https://github.com/neinteractiveliterature/intercode/issues/12260). + +Nothing here removes the data-mode build. Every release still builds and publishes both, so each step can be undone by +changing one Terraform variable. The code cutover (v9.0.0), which makes the framework build the only one and gives the +Rails image its own copy of the shell, comes after this rollout is finished. Domains that Rails serves directly, with no +CloudFront in front, move with that release. + +## Before the first domain + +- The repository variables `BUILD_FRAMEWORK`, `FRAMEWORK_SHELL_BUCKET` and `FRAMEWORK_SHELL_UPLOAD_ROLE_ARN` are set, so + every release builds the framework image and uploads it to the shell bucket. Leave them set for the whole rollout: + unsetting `BUILD_FRAMEWORK` while the bucket is in use means the bucket stops getting new releases. +- The shell bucket comes from [`assets_bucket_intercode`](../terraform/modules/assets_bucket_intercode) (a separate + bucket from the one for `/packs/*`), and its `cloudfront_distribution_arns` includes the distribution you are about to + change. +- At least one release has uploaded to it. Check the bucket has `index.html` and an `assets/` directory. +- The distribution has access logs on (`access_logs_enabled`, from + [`cloudfront_intercode_logging`](../terraform/modules/cloudfront_intercode_logging)), and the logging module's 5xx + alarm has an `alarm_actions` topic someone reads. Both are what you will be watching. + +## The order + +1. **P** (for example `p.interconlarp.org`), the test bed. +2. **I**, once P has been quiet through at least one release. +3. The rest of the CloudFront domains, together or in small groups. +4. The cutover release (v9.0.0), which moves everything that Rails serves directly. + +Each step is the same change on one distribution: + +```hcl +static_shell_enabled = true +static_shell_bucket_domain_name = module.shell_bucket.bucket_regional_domain_name +static_shell_origin_access_control_id = module.shell_bucket.origin_access_control_id +``` + +After the apply, the distribution takes a few minutes to deploy. Let it finish before testing. + +## Checking a domain + +Do this right after the apply, and again after the first release that lands while the domain is on the static shell. + +- **The shell comes from the bucket.** `curl -sI https:///` returns the shell with `Cache-Control: max-age=60` + (the same value as `static_shell_ttl`), and a deep link such as `/events` returns the same document. A request for a + file under `/assets/` that doesn't exist is a 404, not the shell. +- **Crawler previews.** A link-preview user agent gets per-resource metadata from Rails, not the shell: + + ```sh + curl -s -A 'Slackbot-LinkExpanding 1.0' https:///events/ | grep -E 'og:(title|url)' + ``` + + `og:title` should be the event's title and `og:url` the page's own address. Paste a link into Slack as well, since + that is the real consumer. + +- **Sign-in.** In a private window: sign in, reload (still signed in), open another page, sign out, reload (still + signed out). Sign-in goes through the root site's sign-in page, so check that the round trip ends signed in on the + domain you started from. +- **A page that needs the API.** Open a page with a loader, such as an event, and an admin page if you have the access. + These fail loudly if `/graphql` or `/client_configuration` isn't reaching Rails. +- **A stale tab survives a release.** Leave a tab open on the site across a release, then click through to a page you + haven't opened yet. It should keep working, because the bucket keeps the earlier release's files, and if a file is + missing the app reloads once and then shows a message with a reload button. + +## What to watch + +- **The 5xx alarm** from `cloudfront_intercode_logging`. It counts responses, not a rate, because these sites see tens + to hundreds of requests a day. Requests to a raw `*.cloudfront.net` name are excluded. +- **Logs Insights** on the `/cloudfront/` log group, for anything that is failing but not 5xx: + + ``` + filter `sc-status` like /^5/ + | stats count(*) by `x-host-header`, `cs-uri-stem` + ``` + + Also look at 404s under `/assets/`, which mean a page is asking for a file the bucket doesn't have: + + ``` + filter `sc-status` like /^404$/ and `cs-uri-stem` like /^\/assets\// + | stats count(*) by `x-host-header`, `cs-uri-stem` + ``` + + 4xx responses overall are not alarmed on: anonymous visitors get an expected 401 from `/oauth_session/refresh` on + every page load. + +- **Sentry and Rollbar** for new frontend errors from the domain after the change. +- **Your release workflow.** The `framework-shell-release-*` jobs have to succeed, or the bucket is a release behind. + +Leave each domain on the static shell for at least one release and a few days of normal traffic before moving the next. + +## Rolling back + +Remove the three variables above (or set `static_shell_enabled = false`) and apply. The default behavior goes back to the +Rails origin, and Rails renders the shell and loads the data-mode build from `/packs/`. The distribution takes a few +minutes to deploy, and a browser that already has the static shell keeps it until it reloads the page. Nothing else +changes with it: the API, the sign-in cookies and the database are the same in both modes, so a tab open on either build +keeps talking to the same backend. + +Don't delete the bucket or unset the repository variables to roll back; leaving them in place means going forward again +is the same one-line change. + +This only works while releases still build the data-mode build. Once v9.0.0 is out it no longer does, and a rollback is +redeploying the last 8.x image to Fly and removing the static shell variables. diff --git a/terraform/modules/cloudfront_intercode/README.md b/terraform/modules/cloudfront_intercode/README.md index 5db5913ba3..eed8b8d3e9 100644 --- a/terraform/modules/cloudfront_intercode/README.md +++ b/terraform/modules/cloudfront_intercode/README.md @@ -139,7 +139,7 @@ Rails, and a missing `/assets/...` file should be a 404. Don't send a `Host` hea instead: CloudFront answers 403 for a host that isn't one of the distribution's aliases, and without an alias Rails is asked for a host it doesn't serve. -**Rolling back** is removing `static_shell_enabled` (and the two variables). The next apply puts the Rails origin and the Rails-rendered shell back. +**Rolling back** is removing `static_shell_enabled` (and the two variables). The next apply puts the Rails origin and the Rails-rendered shell back. See [deploy/staged-rollout.md](../../../deploy/staged-rollout.md) for the order to move domains in, what to check and what to watch. ## Other web servers