deploying-to-kubernetes

Type	Skill
Plugin	awl-cicd · v0.0.22
Invoke	/awl-cicd:deploying-to-kubernetes
Tools	Read, Write, Edit, Bash, Glob, Grep
Source	plugins/awl-cicd/skills/deploying-to-kubernetes/SKILL.md

When Claude uses it

This skill should be used when the user asks to “deploy”, “add deployment to pipeline”, “set up staging/testing/production”, “deploy to natron”, “DEPLOYMENT_VALUES”, “gitlabSecrets”, “deployment not triggering”, “ExternalSecret UpdateFailed”, “argocd red”, “stop environment”, “deploy redis next to the app”, or mentions autodeploy. Configures the GitLab CI deploy job that ships an app to the AWL Natron Kubernetes cluster via ArgoCD (deploy-default-branches, deploy-service, deploy-autodeploy-app), the DEPLOYMENT_VALUES contract, secrets wiring, and a pre-merge checklist. For cluster-side resources (PostgreSQL, S3, persistent volumes, basic auth, Teleport, custom domains) use natron-deployment.

Trigger phrases: deploy · add deployment to pipeline · set up staging/testing/production · deploy to natron · DEPLOYMENT_VALUES · gitlabSecrets · deployment not triggering · ExternalSecret UpdateFailed · argocd red · stop environment · deploy redis next to the app

Definition

Quick start

include:
  - component: $CI_SERVER_FQDN/devops/ci-cd-templates/deploy-default-branches@main

deploy-main-branch:
  variables:
    DEPLOYMENT_DOMAIN: myapp.staging.appswithlove.net
    DEPLOYMENT_VALUES: |
      port: 3000
      replicas: 1
      healthz: /api/health
      configmap:
        NODE_ENV: "production"
      gitlabSecrets:
        DATABASE_URL: DATABASE_URL

Before writing this for a new project, read an existing .gitlab-ci.yml that deploys the same stack (bafu/becasuisse-cms for Payload, awl-ecosystem/my-awl for Node, ensemble/mono-repo for .NET). Conventions such as job names, environment names, and variable names come from there, not from memory.

Prerequisites (one-time per project)

1. Add the GitLab user image-pull as project member with role Maintainer. Kubernetes uses it to pull images and read CI variables. Missing or lower role shows up as InvalidProviderConfig or 403 on /projects/<id>/variables in ArgoCD.
2. Protect the branches main, testing, and production. Deploy jobs fail on unprotected branches.
3. Commit a Dockerfile (see building-docker-images).
4. Make sure the container registry is enabled for the project.

Environment mapping

Branch	Environment	ArgoCD instance	Default domain
main	staging	natron-staging	<project-slug>.staging.appswithlove.net
testing	testing	natron-staging	<project-slug>.testing.appswithlove.net
production	production	natron-prod	<project-slug>.prod.appswithlove.net

<project-slug> is $CI_PROJECT_PATH_SLUG (group/app becomes group-app). With app-suffix, environment names become staging<suffix> etc.

Pre-merge checklist

Run this before every merge that touches a deploy job. Each item caused a failed deployment in a real project.

1. Every gitlabSecrets reference exists. glab variable list must show each name on the left side of the mapping. A missing variable makes the ExternalSecret UpdateFailed (404) and ArgoCD red, and the pod never starts.
2. Variable scope matches the environment name of the job (staging, testing, production, or *). Since 1 March 2026 * (All environments) works for External Secrets, so use it unless values differ per environment.
3. Variables are masked, not hidden. Hidden variables can’t be read by External Secrets.
4. Mapping direction is right. gitlabSecrets is GITLAB_VARIABLE_NAME: ENV_NAME_IN_POD. Left is the GitLab CI variable, right is the environment variable the container sees. Swapping them leaves the pod without the value.
5. Configmap values are strings. PORT: "3000", ENABLED: "true".
6. No YAML comments inside DEPLOYMENT_VALUES. The generator rejected them until at least August 2026. Explain in the job, not in the values block.
7. DEPLOYMENT_NAME + namespace + cluster stays under 53 characters, otherwise Helm rejects the release name.
8. healthz points at a route that exists in the image being deployed. /healthz and /api/healthz are different routes.
9. Jobs that read scoped variables have an environment: block. Build jobs that need a scoped SENTRY_AUTH_TOKEN need environment: { name: staging, action: verify } or the variable is silently empty.
10. Passwords in connection URIs are URL-safe. Generate with openssl rand -hex 32. /, +, = break URI parsing.
11. The pipeline lints. glab ci lint on the project .gitlab-ci.yml.

Set missing variables with glab variable set NAME --scope '*' --masked after confirming the value with the user. Never paste secret values into chat or commits.

DEPLOYMENT_VALUES reference

DEPLOYMENT_VALUES is a YAML string merged on top of the chart defaults (replicas: 1, port: 80, healthz: /, 96Mi/256Mi memory, public nginx-external ingress with Let’s Encrypt).

Basic settings

DEPLOYMENT_VALUES: |
  port: 3000
  replicas: 2
  healthz: /api/health          # "" disables the probe
  initialDelaySeconds: 30

Set replicas: 0 to pause a deployment without deleting it.

ConfigMap (non-sensitive environment variables)

DEPLOYMENT_VALUES: |
  configmap:
    NODE_ENV: "production"
    API_URL: "https://api.example.com"
    DB_PORT: "5432"
    FEATURE_X: "true"

GitLab secrets (sensitive environment variables)

DEPLOYMENT_VALUES: |
  gitlabSecrets:
    DATABASE_URI: DATABASE_URI          # same name on both sides is the common case
    POWERSYNC_SOURCE_URI: PS_SOURCE_URI # GitLab variable -> pod env name

External Secrets refreshes about once an hour. After changing a variable, either wait, delete the ExternalSecret in ArgoCD (needs ArgoCD permission, usually DevOps), or change the manifest (for example rename the mapping) to force a new fetch.

Resources

DEPLOYMENT_VALUES: |
  resources:
    requests:
      memory: 128Mi
      cpu: 100m
    limits:
      memory: 512Mi
      cpu: 500m

requests drive scheduling, limits.memory triggers OOMKilled. Check “terminated: OOMKilled” in the pod summary in ArgoCD.

Ingress

DEPLOYMENT_VALUES: |
  ingress:
    enabled: true
    className: nginx-external      # nginx-external = public, nginx = internal only
    path: /
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: "50m"
      # Custom annotations REPLACE the defaults. Re-add these:
      nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
      nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
      cert-manager.io/cluster-issuer: letsencrypt-http01

Additional patterns:

• Second domain / www redirect: extraTlsDomains: [www.example.org] plus annotation nginx.ingress.kubernetes.io/from-to-www-redirect: "true".
• WebSockets: annotation nginx.org/websocket-services: "${DEPLOYMENT_NAME}".
• Sub path: set DEPLOYMENT_INGRESS_PATH: /admin on the job, not inside DEPLOYMENT_VALUES.
• Internal only via Teleport: teleport: { enabled: true } and ingress: { enabled: false }. See natron-deployment.
• Custom domain: set DEPLOYMENT_DOMAIN on the job. Anything outside *.staging|testing|prod.appswithlove.net needs a DNS record first (natron-deployment).

Labels, annotations, metrics

DEPLOYMENT_VALUES: |
  labels:
    app.kubernetes.io/component: backend
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "3000"
    prometheus.io/path: /metrics

Scraped metrics land in the Grafana datasource “Victoria Metrics”, not “Prometheus”.

Templates

Template	Use
deploy-default-branches	Default. Build + deploy main/testing/production, monorepo via app-dir/app-suffix/watch
deploy-service	Redis or Meilisearch next to the app (service-name, preset)
deploy-autodeploy-app	Base job for custom branches, other clusters, prebuilt images

Inputs, generated jobs, and full sources: gitlab-cicd/references/templates/deploy.md.

Behavior of deploy-default-branches that surprises people:

• A GitLab deploy freeze ($CI_DEPLOY_FREEZE) skips every deploy job silently.
• Each environment gets a manual stop-*-branch job that removes the ArgoCD application but keeps persistent volumes. Deploy jobs no longer declare on_stop (since 2026-09-11), so overriding environment.name is safe.
• Deploy jobs include secret_detection and track-dependencies on the three deploy branches.
• In monorepo mode nothing runs unless app-dir/**/*, .gitlab-ci.yml, or a watch path changed. Start a manual pipeline to force a deploy.

Deploying a prebuilt image

build-docker:
  rules:
    - when: never

deploy-main-branch:
  variables:
    DEPLOYMENT_IMAGE: docker.io/library/nginx:1.27

Workflow

1. Push to main, testing, or production.
2. build-docker builds and pushes $CI_REGISTRY_IMAGE:<branch><suffix>.
3. deploy-*-branch triggers the GitOps pipeline with a signed ID token. The job log prints the ArgoCD link and the deployment URL.
4. ArgoCD syncs the application. Allow a few minutes; click Refresh in ArgoCD to skip the poll interval.
5. Verify the pod is Running and the URL answers. A green deploy job only means the trigger request was accepted.

Deploy does not mean data is there. Apps with cron-driven imports or content syncs need a manual first run after deploy.

ArgoCD access

Environment	URL
Staging	https://argocd.teleport.awl-staging.k8s.natron.cloud/
Production	https://argocd.teleport.awl.k8s.natron.cloud/

Needs a Teleport account and an ArgoCD account. Applications are named <DEPLOYMENT_NAME>-<namespace>-<cluster>. If the app is missing, check the root app root-multi-cluster → natron-staging-application-apps; if it isn’t listed there, ask DevOps.

Troubleshooting

Error	Cause	Fix
Deployment not triggering	Wrong branch, deploy freeze, or no change under app-dir	Use main/testing/production; check freeze; run a manual pipeline
curl: (22) or 404 from the trigger	Missing GITOPS_TRIGGER_TOKEN or wrong project ID	Instance variable; ask DevOps
ExternalSecret UpdateFailed, 404 Not Found on a variable	gitlabSecrets references a CI variable that doesn’t exist	Create it (glab variable set) or remove the mapping
secret "<name>" not found in pod events	Same as above, or variable is hidden / wrong scope	Fix variable, then delete the ExternalSecret to re-fetch
Pod env is undefined although the variable exists	gitlabSecrets mapping direction swapped	Left GitLab name, right pod name
InvalidProviderConfig / 403 on /variables	image-pull user missing or not Maintainer	Add as Maintainer
Cannot convert int64 to string	Number or boolean in configmap	Quote the value
invalid release name ... longer than 53	Long project or group name	Set short DEPLOYMENT_NAME / DEPLOYMENT_NAMESPACE
Pod failing health checks	healthz path doesn’t exist	Fix the path or set healthz: ""
OOMKilled	limits.memory too low	Raise limits
Old secret value after rotation	ExternalSecret refresh interval	Delete the ExternalSecret in ArgoCD or change the mapping
Job doesn’t see a scoped variable	No environment: on the job	Add environment: { name: <env>, action: verify }
on_stop job ... different environment name	Old template revision with on_stop	Re-run with current @main; keep names identical if overriding
Deleted app still exists in ArgoCD after rename	Rename creates a new app	Ask DevOps in #tools-autodeploy to delete the old one

Reference

• Template sources and inputs
• Full DEPLOYMENT_VALUES reference and examples
• Complete working examples
• Natron Deployment guide in ClickUp (Apps with love Wiki → Software Development → Deployment Guidance)