Kubernetes

kubernetescontainergoogle cloudcloudgoogle

Kubernetes automates the deployment, scaling and operation of containerized applications. This page is the section hub and a cookbook of things I actually run.

Pages in this section

Rollout, rollback and restart a deployment

Ship a new image. set image is the short form and the one to reach for:

kubectl set image deployment/$DEPLOYMENT $CONTAINER=$IMAGE

The general form is a strategic merge patch, useful when you change more than the image. Note the double quotes: inside single quotes the shell would not expand $CONTAINER and $IMAGE, and you would patch the deployment with those two literal strings.

kubectl patch deployment $DEPLOYMENT \
  -p "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"$CONTAINER\",\"image\":\"$IMAGE\"}]}}}}"

Watch it land, and get a non-zero exit code if it does not (handy in CI):

kubectl rollout status deployment/$DEPLOYMENT

Go back. Without --to-revision it undoes the last rollout; kubectl rollout history lists the revisions.

kubectl rollout history deployment/$DEPLOYMENT
kubectl rollout undo deployment/$DEPLOYMENT --to-revision=42

Restart every pod without changing the spec. It stamps a kubectl.kubernetes.io/restartedAt annotation on the pod template, so it goes through the normal rolling update:

kubectl rollout restart deployment/$DEPLOYMENT

Longer write-up: How to rollout or rollback a deployment on a Kubernetes cluster?

Environment from a ConfigMap

envFrom injects every key of a ConfigMap as an environment variable. Mind the indentation: name belongs to configMapRef, not to the list item.

envFrom:
- configMapRef:
    name: eat-at-joe

At two spaces, name would become a sibling of configMapRef and the API server rejects the pod (envFrom entries only accept prefix, configMapRef and secretRef).

Probes

Three probes, three different jobs.

  • Startup: is the application started yet? While it runs, the two others are held back. If it fails, the container is killed and restarted. Only useful for slow starters.
  • Liveness: is the application still making progress? On failure the container is restarted. This is the one that bites: a liveness probe that fails under load restarts pods that were merely busy, and turns a slowdown into an outage.
  • Readiness: can the application serve traffic right now? On failure the pod’s IP is pulled from the EndpointSlices of every matching Service, so it stops receiving traffic, but the container keeps running.

A probe fails only after failureThreshold consecutive failures (default 3), each periodSeconds apart (default 10).

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: example-image
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10
      failureThreshold: 30    # 30 x 10s = 5 minutes to start
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 5
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /healthz/ready
        port: 8080
      periodSeconds: 5

The failureThreshold of the startup probe is the whole point: leave it at the default 3 and the container gets 30 seconds to boot, which is exactly what you were trying to avoid. Give the startup probe a generous budget, keep the liveness probe tight.

With a startup probe in place you rarely need initialDelaySeconds on the other two: their delays only start counting once the startup probe has succeeded.

Capacity planning

Poor man’s swiss army knife: one line per container, with its requests and whether the workload spreads itself across nodes.

for rt in deploy daemonset statefulset; do
  echo "$rt"
  kubectl get "$rt" --all-namespaces -o json | jq -r '
    .items[]
    | [.metadata.namespace, .metadata.name, (.spec.replicas // "-")]
      + (.spec.template.spec.containers[]
         | [.name, (.resources.requests.cpu // "-"), (.resources.requests.memory // "-")])
      + [(.spec.template.spec.affinity.podAntiAffinity != null)]
    | @tsv'
  echo
done

A - in the cpu or memory column is a container with no request: the scheduler has nothing to place it with, and it is the first thing to fix before sizing anything.

Google Kubernetes Engine (GKE)

When traffic does not reach the pods, walk the chain in order rather than guessing:

  • Kubernetes side: Ingress, Service, Deployment, ReplicaSet, Pod.
  • Google Cloud side: firewall rules match the NodePort exposed by the Service, which kubectl describe service gives you.
  • Google Cloud side: the L7 load balancer is wired correctly (backend, routes, frontend).
  • Google Cloud side: health checks are green. They are separate from the Kubernetes probes, and a readiness probe on a path the load balancer does not check will not save you.

Reading list

On networking, the two CNI plugins worth knowing are Cilium (eBPF based, CNCF graduated since October 2023) and Calico (iptables, with an eBPF mode).

On managed clusters, Scaleway Kubernetes Kapsule is the European option I keep an eye on.

Related