feat(jobber): migrate to temporal

artifacts/docs/README.md

@@ -8,7 +8,7 @@ Profiles group services so you can deploy subsets on demand. Activate with `darg
 
 ### analytics
 
-Services: cadvisor, grafana, node-exporter, postgres, prometheus, sqitch, traefik
+Services: cadvisor, grafana, node-exporter, postgres, prometheus, sqitch, temporal, temporal-ui, temporal-worker, traefik
 
 ### default
 
@@ -78,10 +78,6 @@ You cannot access the ip geolocator via a web interface.
 You can access the observation dashboard at [grafana.app.localhost](https://grafana.app.localhost/).
 See `docs/grafana.md` for how to configure datasources, alerting, and dashboards after first login.
 
-### jobber
-
-You cannot access the jobber via a web interface.
-
 ### memcached
 
 You cannot access the caching system via a web interface.
@@ -151,6 +147,18 @@ You can access the event streaming platform's ui at [redpanda.app.localhost](htt
 
 You cannot access the database migrations directly.
 
+### temporal
+
+You cannot access the workflow orchestrator directly, use `temporal-ui`.
+
+### temporal-ui
+
+You can access the workflow orchestrator's ui at [temporal.app.localhost](https://temporal.app.localhost/).
+
+### temporal-worker
+
+You cannot access the workflow worker directly, use `temporal-ui`.
+
 ### traefik
 
 You can access the reverse proxy's dashboard at [traefik.app.localhost](https://traefik.app.localhost/).

dargstack.yaml

@@ -14,7 +14,7 @@ environment:
     # tag: latest # optional, defaults to "latest"
 
 metadata:
-  compatibility: ">=4.13.0-0 <5.0.0"
+  compatibility: ">=4.17.0-0 <5.0.0"
   name: "vibetype"
   source:
     name: "maevsi"

docs/grafana.md

@@ -79,6 +79,21 @@ Add under **Alerting → Alert rules → New alert rule**, in the `Infrastructur
 - **Summary annotation**: "There are notifications which are not sent out, or at least not marked as acknowledged."
 - **Contact point**: `Discord`
 
+### Alert rule: workflow failures
+
+Add under **Alerting → Alert rules → New alert rule**, in the `Infrastructure` folder.
+
+- **Query (A)**, on the Prometheus datasource, code mode, instant query over the last 10 minutes: `sum(increase(workflow_failed[10m])) or vector(0)`
+- **Reduce (B)**: last of A
+- **Threshold (C)**: B is above 0, this is the alert condition
+- **Evaluation**: every `1m`, for `1m`, group `Temporal`, repeat every 7 days
+- **No data / error state**: `NoData` / `Error`
+- **Summary annotation**: "Temporal reported one or more failed workflow executions in the last 10 minutes."
+- **Contact point**: `Discord`
+
+The `or vector(0)` keeps the query from going stale while no workflow has ever failed.
+Without it `workflow_failed` does not exist yet, the rule evaluates to no data, and the alert fires on the `NoData` state instead of staying quiet.
+
 ## Dashboards
 
 Add under **Dashboards → New → Import**.
@@ -93,7 +108,8 @@ The following were provisioned from public community dashboards and can be reimp
 The `Redpanda Ops Dashboard`, `Grafana metrics`, and `Prometheus 2.0 Stats` dashboards were also provisioned from community sources, but without a recorded grafana.com ID.
 Search grafana.com's dashboard library by name, or recover the exact JSON that was previously provisioned from this repository's git history (`git log --diff-filter=D -- 'src/development/grafana/configurations/dashboards/**'`) and import it via **Upload dashboard JSON file**.
 
-Two dashboards were specific to this project rather than imported from the community, both querying the PostgreSQL datasource. Recreate their panels as needed:
+Two dashboards were specific to this project rather than imported from the community, both querying the PostgreSQL datasource.
+Recreate their panels as needed:
 
 **KPIs** (folder `Management`):
 

docs/multi-node.md

@@ -17,11 +17,11 @@ On any other node that path does not exist, and Docker silently creates an empty
 
 ## Which services are pinned
 
-Services holding a production volume: `debezium`, `elasticsearch`, `grafana`, `jobber`, `portainer`, `postgres`, `postgres-backup`, `prometheus`, `reccoom-postgres`, `redis`, `redpanda`, `traefik`, `traefik-certs-dumper` and all six `zammad` services.
+Services holding a production volume: `debezium`, `elasticsearch`, `grafana`, `portainer`, `postgres`, `postgres-backup`, `prometheus`, `reccoom-postgres`, `redis`, `redpanda`, `temporal-worker`, `traefik`, `traefik-certs-dumper` and all six `zammad` services.
 
-Services holding only a `configurations/` bind mount: `adminer`, `debezium-postgres-connector`, `redpanda-console`.
+Services holding only a `configurations/` bind mount: `adminer`, `debezium-postgres-connector`, `redpanda-console`, `temporal`.
 
-Some of these also share a volume and therefore have to land on the same node as each other, which the shared label already guarantees: `postgres` with `postgres-backup` with `jobber`, `traefik` with `traefik-certs-dumper`, and the six `zammad` services with one another.
+Some of these also share a volume and therefore have to land on the same node as each other, which the shared label already guarantees: `postgres` with `postgres-backup` with `temporal-worker`, `traefik` with `traefik-certs-dumper`, and the six `zammad` services with one another.
 
 ## Adding a service
 

src/development/.env.template

@@ -1,4 +1,7 @@
 RECCOOM_ADMIN_ACCOUNT_IDS=
+TEMPORAL_WORKER_S3_ENDPOINT=http://minio:9000
+TEMPORAL_WORKER_S3_REGION=eu-central-1
+TEMPORAL_WORKER_S3_USE_PATH_STYLE=true
 TUSD_BUCKET=vibetype-images
 TUSD_ENDPOINT=http://minio:9000
 TUSD_MAX_SIZE=10485760

src/development/postgres/compose.yaml

@@ -23,6 +23,20 @@ secrets:
   postgres-role-service-reccoom-username:
     # The username of the recommendation engine's database role.
     file: ~~/artifacts/secrets/postgres-role-service-reccoom-username.secret
+  postgres-role-service-temporal-password:
+    # The password of the workflow orchestrator's database role.
+    # The role owns the orchestrator's own two databases and holds none of the application's data.
+    file: ~~/artifacts/secrets/postgres-role-service-temporal-password.secret
+  postgres-role-service-temporal-username:
+    # The username of the workflow orchestrator's database role.
+    file: ~~/artifacts/secrets/postgres-role-service-temporal-username.secret
+  postgres-role-service-temporal-worker-password:
+    # The password of the workflow worker's database role.
+    # The role exists for the outbox purge workflow, which stays disabled until the outbox table lands in `sqitch`.
+    file: ~~/artifacts/secrets/postgres-role-service-temporal-worker-password.secret
+  postgres-role-service-temporal-worker-username:
+    # The username of the workflow worker's database role.
+    file: ~~/artifacts/secrets/postgres-role-service-temporal-worker-username.secret
   postgres-role-service-vibetype-password:
     # The `vibetype` database role's password.
     file: ~~/artifacts/secrets/postgres-role-service-vibetype-password.secret
@@ -100,6 +114,16 @@ x-dargstack:
       type: random_string
     postgres-role-service-reccoom-username:
       type: wordlist_word
+    postgres-role-service-temporal-password:
+      special_characters: false
+      type: random_string
+    postgres-role-service-temporal-username:
+      type: wordlist_word
+    postgres-role-service-temporal-worker-password:
+      special_characters: false
+      type: random_string
+    postgres-role-service-temporal-worker-username:
+      type: wordlist_word
     postgres-role-service-vibetype-password:
       special_characters: false
       type: random_string

src/development/prometheus/configurations/prometheus.yaml

@@ -33,3 +33,11 @@ scrape_configs:
       - targets:
           - redpanda:9644
     metrics_path: /public_metrics
+
+  - job_name: "temporal"
+    static_configs:
+      - targets: ["temporal:8000"]
+
+  - job_name: "temporal-worker"
+    static_configs:
+      - targets: ["temporal-worker:9090"]

src/development/sqitch/compose.yaml

@@ -20,6 +20,10 @@ services:
       - postgres-role-service-postgraphile-username
       - postgres-role-service-reccoom-password
       - postgres-role-service-reccoom-username
+      - postgres-role-service-temporal-password
+      - postgres-role-service-temporal-username
+      - postgres-role-service-temporal-worker-password
+      - postgres-role-service-temporal-worker-username
       - postgres-role-service-vibetype-password
       - postgres-role-service-vibetype-username
       - postgres-role-service-zammad-username