This pattern is based on the RHDPS demo catalog item defined in the private-maas-automation repo.
On regular AWS OpenShift clusters, the production variant automatically runs the
Validated Patterns rhvp.cluster_utils.create_machineset playbook to provision
one g6e.2xlarge GPU worker. It clones an existing worker MachineSet's AMI and
network placement, creates a separate nvidia-gpu MachineSet with one replica,
and sets this node taint:
key: nvidia.com/gpu
value: l40-gpu
effect: NoScheduleThe model and AI hardware profile already tolerate this taint. The existing
nvidia-gpu-enablement application configures GPU support. Provisioning uses the
cluster's Machine API AWS credentials and requires an existing AWS worker
MachineSet. The imperative job reconciles on the framework's ten-minute schedule.
Its settings are in variants/prod/overrides/values-AWS-gpu.yaml, selected
automatically when global.clusterPlatform is AWS.
The production variant provisions one complete tenant, user1, with premium
MaaS access, a DevSpaces workspace, a Rocket.Chat account, and a GenAI playground
backed by pgvector. It uses local charts adapted from the source tenant; see
the source revision and adaptations.
global.tenant in values-global.yaml controls the username, MaaS group, and
workspace namespace prefix. The default workspace namespace is wksp-user1.
The framework supplies global.localClusterDomain at installation. Chart values
also allow changing the exercise repository, workspace image, and playground
model. The model must match the single deployed MaaS model.
Merge the entries in values-secret.yaml.template into your local
~/values-secret-maas.yaml. For a new installation, copy the template there.
Keep that file outside Git. The loader generates the tenant login and database
passwords without prompting. Tenant setup waits for Rocket.Chat and discovers its
administrator credentials from the deployed container environment, including
referenced Secrets. No administrator password needs to be supplied before
installation. The OpenShift and Rocket.Chat tenant logins share the generated
password.
export KUBECONFIG=/path/to/deployed-cluster/auth/kubeconfig
./pattern.sh make load-secretsCommit and push the pattern changes to the branch tracked by the installed
Pattern. Its clustergroup will create user-config and tenant-workspace.
For a new installation, use the normal ./pattern.sh make install flow.
If the cluster already has identity providers, include them in the user-config
chart's additionalIdentityProviders values before syncing; this chart manages
the OAuth provider list.
Monitor setup without displaying credentials:
oc get externalsecret -n openshift-config
oc get externalsecret,jobs,cronjobs -n tenant-provisioning
oc get devworkspace,ogxserver,pods -n wksp-user1
oc logs -n tenant-provisioning -l job-name --all-containers --tail=30The setup Job runs as an Argo CD Sync hook alongside tenant resources. It waits for secrets and
platform readiness, adds the user to premium-users, reconciles the Rocket.Chat
account, and logs in as the tenant to mint a one-hour premium MaaS key. Initial
setup allows 90 minutes for each readiness stage, with a two-hour overall Job
deadline, to cover cold GPU provisioning, large model image downloads, and model
loading. These limits are configurable through
provisioner.startupReadinessTimeoutSeconds and provisioner.startupDeadlineSeconds.
Argo's sync retry policy recreates a failed hook; successful hooks are deleted.
The hook verifies the workspace and playground even if a scheduled run already
created the API key. It runs during a full application sync, not a selective
resource sync, and uses Sync rather than PostSync because consumers need its
key before they can become healthy.
A CronJob runs every ten minutes and renews keys with twenty minutes or less remaining. Scheduled runs retain the 20-minute readiness budget and 24-minute Job deadline. It validates a new key before publishing it, waits for consumers, and revokes the superseded key. Failed runs retain state for the next reconciliation. Expired keys do not recover until provisioning and its dependencies are healthy.
Administrative Rocket.Chat credentials stay with its deployment and referenced
Secrets; provisioning reads them only at runtime. Tenant login credentials and
key lifecycle state stay in tenant-provisioning. The tenant has edit in its
own workspace namespace.
The workspace and playground receive the tenant MaaS key and its issuing CA;
TLS verification stays enabled. The playground runs an internal HTTP service,
matching the source deployment.
Retrieve only the tenant password with an administrator kubeconfig:
oc get secret tenant-credentials -n tenant-provisioning \
-o jsonpath='{.data.password}' | base64 -dOpen the cluster's DevSpaces URL in a separate browser session. Select
htpasswd_provider and log in as user1 using that password. Open exercises
and verify the coding-exercises repository is available. In its terminal, run:
curl --fail-with-body --cacert "$MAAS_CA_FILE" \
-H "Authorization: Bearer $(cat "$MAAS_API_KEY_FILE")" \
-H 'Content-Type: application/json' \
"$MAAS_BASE_URL/completions" \
-d "{\"model\":\"$MAAS_MODEL\",\"prompt\":\"Say hello\",\"max_tokens\":16,\"stream\":false}"Open the AI dashboard's Workspace user1 project and use its GenAI playground.
Log into Rocket.Chat as user1 with the same password. These checks should work
without the kubeadmin browser session.
DevSpaces mounts maas-api-key at /etc/maas using its
documented Secret mounting mechanism.
Both the mount and watch labels are required for the workspace to see the Secret.
Secret updates can restart running workspaces; the scheduled renewal therefore
can briefly interrupt the IDE roughly every forty minutes. Stopped workspaces
remain stopped. The playground also rolls over to the new key.
HTPasswd uses an ExternalSecret with OnChange refresh because bcrypt generates
a different salted hash on every render. This avoids repeated OAuth rollouts
when the underlying password has not changed. After updating the tenant password
in Vault through make load-secrets, explicitly refresh the HTPasswd Secret:
oc annotate externalsecret htpasswd-secret-es -n openshift-config \
force-sync="$(date +%s)" --overwriteThe next tenant reconciliation updates the Rocket.Chat password. Changing the username also requires changing its field name in your local secrets file. The database uses a generated persistent password; changing it requires restarting the database and playground to reload their Secret references.
If setup fails, inspect the latest provisioning Job logs and ExternalSecret conditions first. Then check the authentication operator, DevSpaces, model, Rocket.Chat, and pgvector Pod readiness. The provisioner logs only safe error summaries. A failed replacement key remains pending until consumers are ready; the previous key is not revoked early. Both the initial Job and CronJob use a shared Lease to avoid simultaneous reconciliations. The provisioner creates this lock at runtime because Argo CD excludes Lease resources from synchronization.
When upgrading from the original standalone setup Job, push the chart changes,
refresh tenant-workspace, and perform a full Sync with Prune enabled. This
removes the old failed tenant-setup-* Job and runs the new setup hook. A refresh
alone does not retry a completed or failed standalone Kubernetes Job.
After deployment, run the cluster integration suite:
VP_HUBCONFIG="$KUBECONFIG" ./pattern.sh make run-ci-testsThe tenant tests require exercises to be running. They verify the actual login,
permissions, premium inference, inference inside DevSpaces, playground health,
and Rocket.Chat login. Existing free-tier and anonymous tests are unchanged.