Canary Deployments in Kubernetes

Why rolling updates are not enough

Kubernetes gives you a rolling update for free. Change the image in a Deployment and the controller creates a new ReplicaSet, scales it up, scales the old one down, and honours maxSurge and maxUnavailable along the way. It is safe in a narrow sense: it will not take all your pods down at once, and it will pause if new pods never become ready.

The catch is what "ready" means. A readiness probe answers one question — can this container serve traffic? It does not know whether the new build has doubled p95 latency, started returning HTTP 500 on one endpoint, quietly broken a downstream contract, or introduced a memory leak that will bite in forty minutes. Those defects pass every probe you have, and a rolling update will happily march them to one hundred percent of your traffic.

Blue-green, rolling and canary

All three shapes are used on Kubernetes and they solve different problems. Picking the wrong one is a common source of "we have progressive delivery" that does not actually reduce risk.

Blue-green gives you a fast rollback but tells you nothing before the switch, because no real user touched the new version. Canary gives you evidence, at the cost of running two versions simultaneously — which is an application design constraint, not just an infrastructure one.

Canary with replica ratios

The simplest canary on plain Kubernetes needs no extra components. Run two Deployments whose pods share a label that a single Service selects. Traffic then splits roughly in proportion to the replica counts, because kube-proxy load-balances across all matching endpoints.

# Stable: 9 replicas of v1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-stable
spec:
  replicas: 9
  selector:
    matchLabels: { app: payments, track: stable }
  template:
    metadata:
      labels: { app: payments, track: stable, version: v1 }
    spec:
      containers:
        - name: api
          image: registry.internal/payments:1.8.3
---
# Canary: 1 replica of v2 — same `app` label, different `track`
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-canary
spec:
  replicas: 1
  selector:
    matchLabels: { app: payments, track: canary }
  template:
    metadata:
      labels: { app: payments, track: canary, version: v2 }
    spec:
      containers:
        - name: api
          image: registry.internal/payments:1.9.0
---
# One Service selects only `app` — so it fronts both tracks
apiVersion: v1
kind: Service
metadata:
  name: payments
spec:
  selector: { app: payments }
  ports:
    - port: 80
      targetPort: 8080

Ten pods, one of them canary, gives about ten percent of requests to v2. Ramping means scaling the canary up and the stable set down in step. It works, it is transparent, and it requires nothing you do not already run.

The limitations are worth stating plainly, because teams hit all of them:

• Granularity is tied to pod count. One percent of traffic needs at least one hundred pods. For most services, five to ten percent is the realistic floor.
• The split is statistical, not enforced. kube-proxy balances connections, not requests. With HTTP keep-alive, a client that opens one long-lived connection to a canary pod sends all its requests there. With gRPC — multiplexed over a single HTTP/2 connection — the skew is far worse.
• You cannot target a cohort. There is no way to say "internal users first" or "only requests carrying this header".
• Scaling fights you. A HorizontalPodAutoscaler on either Deployment will move your traffic ratio without telling you.

Weighted routing at the ingress

Moving the split up to the ingress controller decouples the traffic percentage from the replica count. With the NGINX ingress controller, you deploy a second Ingress object annotated as a canary for the same host and path.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payments-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"        # 5% of requests
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /payments
            pathType: Prefix
            backend:
              service:
                name: payments-canary
                port: { number: 80 }

Now the canary can run a single replica while taking exactly five percent, and ramping is a one-line patch. More usefully, NGINX evaluates canary rules in a fixed precedence — header, then cookie, then weight — which lets you build a genuine progression:

# Stage 1 — opt-in only. Nobody reaches v2 without asking for it.
nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "always"

# Stage 2 — sticky cohort. A user who lands on the canary stays on it.
nginx.ingress.kubernetes.io/canary-by-cookie: "canary_payments"

# Stage 3 — percentage ramp: 5 -> 20 -> 50 -> 100
nginx.ingress.kubernetes.io/canary-weight: "20"

Service mesh and Gateway API

Ingress-level canaries only cover north-south traffic. If the change is in a service that is called by other services inside the cluster, the ingress never sees those requests. That is where a mesh — Istio, Linkerd — or the Gateway API earns its keep, because the split happens at every sidecar or proxy, east-west included.

# Istio: subsets defined once, weights adjusted per rollout step
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payments
spec:
  host: payments
  subsets:
    - name: stable
      labels: { version: v1 }
    - name: canary
      labels: { version: v2 }
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payments
spec:
  hosts: [ payments ]
  http:
    - route:
        - destination: { host: payments, subset: stable }
          weight: 90
        - destination: { host: payments, subset: canary }
          weight: 10

The vendor-neutral equivalent is Gateway API's HTTPRoute, which expresses weighted backends natively and is now the direction most controllers are converging on:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments
spec:
  parentRefs: [ { name: public-gateway } ]
  rules:
    - backendRefs:
        - name: payments-stable
          port: 80
          weight: 90
        - name: payments-canary
          port: 80
          weight: 10

A mesh also hands you the metrics for free. Because every request passes through a proxy that emits consistent request-count, latency-histogram and response-code series labelled by destination version, you get an apples-to-apples comparison between canary and stable without instrumenting the application at all. That matters enormously for the next step.

Automated analysis and promotion

A canary that a human watches on a dashboard is better than nothing, but it degrades the moment someone is busy. The mature form is a controller that ramps traffic, queries a metrics provider at each step, and promotes or aborts on the result. Argo Rollouts and Flagger both do this; here is the Argo Rollouts shape.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: payments-canary
      stableService: payments-stable
      trafficRouting:
        nginx:
          stableIngress: payments
      analysis:
        templates: [ { templateName: success-rate } ]
        startingStep: 2                 # analyse from the 20% step onwards
        args:
          - name: service
            value: payments-canary
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - setWeight: 20
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
  selector:
    matchLabels: { app: payments }
  template:
    metadata:
      labels: { app: payments }
    spec:
      containers:
        - name: api
          image: registry.internal/payments:1.9.0

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args: [ { name: service } ]
  metrics:
    - name: success-rate
      interval: 1m
      count: 5
      successCondition: result[0] >= 0.99
      failureLimit: 1                   # one bad reading aborts the rollout
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{
              service="{{args.service}}", code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service}}"}[2m]))

When a measurement fails, the controller sets the canary weight back to zero and scales the canary ReplicaSet down. No page, no human, no partial outage that ran for twenty minutes because the on-call engineer was in a meeting.

Choosing the right signals

The analysis is only as good as the queries behind it. A few rules that hold up:

• Compare canary to stable, not to a fixed threshold. A five percent error rate may be normal for a service that proxies a flaky third party. What matters is whether the canary is worse than the stable version right now, under the same conditions.
• Use rate windows longer than your scrape interval. A two-minute rate over a thirty-second scrape gives stable readings; anything shorter will fire on noise.
• Watch percentiles, not averages. Mean latency hides the tail that users feel. p95 and p99 against the stable baseline are the useful comparison.
• Require a minimum request volume. At five percent weight on a quiet service, a single 500 can be a one hundred percent error rate. Gate the analysis on a request-count floor so it does not abort on a sample size of two.
• Include at least one business metric. Technically healthy releases that stop users completing a payment are exactly the class of failure infrastructure metrics miss.

Pitfalls that bite in production

Database schema changes

This is the constraint that decides whether you can canary at all. During the rollout, v1 and v2 are both live against the same database. Any migration must therefore be backwards compatible: add nullable columns, never rename or drop in the same release, and split destructive changes across deployments — expand, migrate, contract. If your change cannot satisfy that, you need blue-green with a maintenance window, and pretending otherwise just moves the outage.

Stateful sessions

If a user's session lives in a pod's memory, a request that lands on the canary and the next that lands on stable will look like a random logout. Externalise session state — Redis, a database, a signed token — or pin the cohort with the cookie-based canary so a user stays on one version for the whole rollout.

Autoscalers moving your weights

With the replica-ratio method, an HPA on the stable Deployment changes the traffic split as a side effect of load. Either move the split to the ingress or mesh, or let the rollout controller own the replica counts during the rollout window.

Connection-level stickiness

Keep-alive and HTTP/2 mean that connection-level balancing is not request-level balancing. For gRPC in particular, a Service-based canary can send a wildly different percentage than the replica ratio implies. Use a proxy that balances per request.

Asynchronous work

Traffic weights govern inbound HTTP. They do not govern a canary pod that is also consuming from a Kafka topic or running a scheduled job — that pod will process whatever it picks up, at full effect. Keep queue consumers and cron workloads out of the canary Deployment, or scale them to zero on the canary track.

Observability that cannot tell the versions apart

If your dashboards aggregate every pod behind one service name, you cannot compare canary to stable and the whole exercise collapses. Propagate a version label from pod metadata into every metric, log line and trace before you run your first canary, not after.

A practical checklist

• Readiness and liveness probes reflect real dependency health, not just process liveness.
• Metrics, logs and traces carry a version label that distinguishes canary from stable.
• Schema changes are backwards compatible across the two versions that will run concurrently.
• Session state lives outside the pod, or the cohort is pinned by cookie.
• An explicit abort path exists and has been tested — deliberately deploy a broken build and confirm the rollback fires.
• Analysis compares canary against stable, over a window long enough to be stable, with a request-count floor.
• Async consumers and scheduled jobs are excluded from the canary track.
• Every rollout step has a maximum duration, so a stuck canary does not sit at fifty percent overnight.

← Back to home