Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 8 · Lesson 71 · 4 min read

Stateless Services and Horizontal Scaling Patterns

On this page
  1. What makes a service stateless: where state hides
  2. Disposable instances: fast startup and graceful shutdown
  3. Sticky sessions: a trap, not a pattern
  4. Autoscaling stateless services
  5. Deployment strategies: rolling, blue-green, canary and feature flags
  6. The forgotten tier: the database
  7. Check yourself

What makes a service stateless: where state hides

Day 2 said "make the app stateless, then scale out". In practice, state hides in more places than sessions. An instance is stateless when no future request depends on anything stored only on that instance. Check each of these:

Hidden stateProblem when scaled outMove it to
In-memory sessionsUsers logged out when routed elsewhereRedis or the database (Day 37)
Uploaded files on local diskFile missing on other instancesObject storage (S3, Azure Blob)
In-memory caches that must agreeInstances serve different dataShared cache, or accept short divergence with a TTL
Scheduled jobs in every instanceJob runs N timesLeader election or a job scheduler (Day 63)
WebSocket connectionsMessages must reach the right instanceA shared message channel (Day 17)
Local counters and rate limitsLimits multiplied by instance countShared store (Day 72)
Generated ids from in-memory countersDuplicate ids across instancesDatabase sequences or UUIDs
A stateless fleet behind a load balancer: any instance can serve any request because shared state lives in Redis and the database.
A stateless fleet behind a load balancer: any instance can serve any request because shared state lives in Redis and the database.

Disposable instances: fast startup and graceful shutdown

A good test: can you kill any instance at any moment, and can a new one start serving within seconds, with no manual steps? If yes, autoscaling, rolling deploys and self-healing all work. Requirements:

  • Configuration from the environment, not files edited on the server.
  • Fast startup, and a readiness check that reports ready only when it can serve.
  • Graceful shutdown: on a stop signal, stop taking new requests, finish in-flight ones, then exit. Kubernetes sends SIGTERM and waits for a grace period before killing the process.
  • No local data worth keeping: logs go to a log system (Day 86), files to object storage.

These ideas are summarized well in the "twelve-factor app" guidelines.

Sticky sessions: a trap, not a pattern

Routing a user back to the same instance hides state instead of removing it. Deploys and scale-downs still lose sessions, load spreads unevenly, and a heavy user can pin one instance. Use it only as a temporary step while moving state out.

Autoscaling stateless services

Once instances are disposable, add and remove them automatically:

  • Scale on a signal that reflects load: CPU, requests per second per instance, or queue depth for workers.
  • Set a minimum above one, for availability, and a maximum that the database can survive.
  • Scale out quickly and in slowly, to avoid flapping. Kubernetes' Horizontal Pod Autoscaler waits 5 minutes by default before scaling down for this reason.

Autoscaling reacts after load rises, and a new instance takes time: start the VM or pull the image, start the process, warm caches and JIT-compiled code. If that takes three minutes and traffic doubles in one, users see the gap. For peaks you can predict, such as a 9 a.m. rush or a marketing email, scale up on a schedule beforehand. For .NET services, ReadyToRun or Native AOT builds cut startup time.

Deployment strategies: rolling, blue-green, canary and feature flags

Disposable instances make safe deployment strategies possible:

StrategyHowRollback
RollingReplace instances a few at a timeRoll forward or back the same way; old and new run together briefly
Blue-greenStart a full new fleet, switch traffic at the load balancerSwitch back instantly; costs double capacity during the switch
CanarySend a small share of traffic to the new version, widen if healthySend the share back to zero (Day 57 shows one)
Feature flagsDeploy code switched off, turn it on per user or percentageTurn the flag off, no deploy needed

Every strategy except blue-green runs two versions side by side for a while, so changes to APIs, events, caches and database schemas must work with both (Days 32 and 55). That's the real cost of deploying without downtime.

The forgotten tier: the database

Adding app instances multiplies database connections and queries. Before scaling the app tier tenfold, check what it does to the database (Day 28): connection limits, pool sizes, query load. A stateless app tier that scales freely in front of a database that can't is just a faster way to overload the database. Caching (Module 4) and read replicas usually come with app-tier scaling.

Check yourself

Files were saved to one instance's local disk. Other instances don't have them. Store uploads in object storage shared by all instances.

On a shutdown signal the instance stops accepting new requests, usually by failing its readiness check so the load balancer drains it, finishes the requests already in flight, then exits. Without it, every scale-down and deploy cuts off requests halfway, which shows up as errors exactly when the system is changing.

Old and new versions ran together, and the old one still used the old name. Expand first (add the new column, write both), deploy, then remove the old column in a later release.

Each instance adds connections and queries. Cap autoscaling to what the database can handle, size pools, add a pooler, and offload reads with caching or replicas.

Blue-green runs two full environments and switches all traffic from old to new at once, with instant rollback by switching back. Canary sends a small percentage of traffic to the new version first, watches its metrics, then increases gradually. Canary catches problems with less exposure; blue-green is simpler.