Catalog My learning Blog
Docker and Kubernetes Tutorial: Containers to Production · Phase 4 · Lesson 24 · 4 min read

Kubernetes Horizontal Pod Autoscaler (HPA)

On this page
  1. Why an HPA shows unknown and never scales
  2. Kubernetes resource requests and limits
  3. How the Horizontal Pod Autoscaler works
  4. Scaling on custom metrics and KEDA
  5. Cluster Autoscaler and Karpenter: adding nodes
  6. Check yourself

Why an HPA shows unknown and never scales

A team adds a Horizontal Pod Autoscaler at 70 percent CPU. Under load, the Pods are clearly busy, but the replica count never changes, and kubectl get hpa shows <unknown>/70%. The Pods have no CPU requests, and the HPA measures utilization as a percentage of the request. No request, no percentage, no scaling.

Kubernetes resource requests and limits

YAML
resources:
  requests: { cpu: 250m, memory: 256Mi }   # reserved for scheduling
  limits:   { memory: 512Mi }              # hard cap
  • Requests are what the scheduler reserves on a node. A Pod is only placed where its requests fit. They're also the baseline for autoscaling percentages.
  • Limits cap usage. Exceeding the memory limit gets the container OOM-killed; exceeding a CPU limit throttles it (Day 14). Many teams set memory limits but leave CPU limits off, to avoid throttling while still guaranteeing requests.
  • 250m means 250 millicores, a quarter of a CPU.

Set requests from measured usage (kubectl top pods, or a Vertical Pod Autoscaler in recommendation mode). Too low, and nodes are overpacked and Pods fight for resources; too high, and you pay for idle capacity.

Requests and limits also set a Pod's QoS class, which decides who is evicted first when a node runs short of memory. Pods with no requests (BestEffort) go first, then Pods using more than they requested (Burstable). Pods whose requests equal their limits for every container (Guaranteed) go last. Critical services shouldn't run without requests.

The Vertical Pod Autoscaler (VPA), an add-on, watches actual usage and recommends requests. Run it with updateMode: "Off" and it only reports, which is the safe way to right-size: read kubectl describe vpa orders-api and copy its target values into your manifests. In its automatic modes it changes requests itself by replacing Pods. Don't let a VPA and an HPA both act on CPU for the same Deployment: the HPA adds Pods because utilization is high, the VPA raises requests so utilization falls, and the two keep undoing each other.

How the Horizontal Pod Autoscaler works

The HPA is a control loop: it reads metrics every 15 seconds or so, compares them with the target and changes the Deployment's replica count. CPU and memory metrics come from metrics-server, which must be installed.

YAML
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300      # wait 5 minutes of low load before removing Pods

The math: desired replicas = current replicas × current metric ÷ target. Four Pods averaging 105 percent of their request against a 70 percent target gives 4 × 105 ÷ 70 = 6 replicas. The scale-down stabilization window prevents flapping when load briefly dips.

The HPA reads Pod metrics relative to their requests and adjusts replicas; the cluster autoscaler adds nodes when new Pods can't fit.
The HPA reads Pod metrics relative to their requests and adjusts replicas; the cluster autoscaler adds nodes when new Pods can't fit.

Scaling on custom metrics and KEDA

CPU is a poor signal for many workloads. A queue worker can be idle on CPU while 50,000 messages wait. Options:

  • Custom or external metrics in the HPA (requests per second, queue depth) through a metrics adapter such as Prometheus Adapter.
  • KEDA (Kubernetes Event-Driven Autoscaling) scales on dozens of event sources, such as Kafka lag, RabbitMQ or SQS queue length, and can scale workers to zero when there's no work.

Cluster Autoscaler and Karpenter: adding nodes

The HPA adds Pods; it doesn't add machines. If new Pods can't fit, they stay Pending. The Cluster Autoscaler or Karpenter watches for unschedulable Pods and adds nodes, and removes underused nodes later. You need both levels: Pod autoscaling for the app, node autoscaling for the capacity underneath.

Node startup takes a minute or more, so keep some headroom or a minimum replica count for sudden spikes.

Check yourself

The HPA can't compute CPU utilization. Usually the Pods have no CPU requests, so there's nothing to compare usage against, or metrics-server isn't installed, so there's no usage data. Add requests to the containers, check kubectl top pods, and look at kubectl describe hpa for the exact error.

Node capacity. The HPA only creates Pods; if the nodes can't fit them, the extra Pods stay Pending. A cluster autoscaler or Karpenter adds nodes when Pods can't be scheduled, so pod-level and node-level scaling work together.

CPU is the wrong signal for this workload, because the worker spends its time waiting on I/O. Scale on queue depth or consumer lag instead, with KEDA or an HPA using an external metric, so the number of workers follows the backlog.

Run it with updateMode: "Off", read its recommendations and copy them into the manifests, and don't combine it with an HPA scaling on CPU.