Catalog My learning Blog
Redis Tutorial: Caching, Persistence and Scaling · Phase 2 · Lesson 7 · 4 min read

Redis Sentinel: High Availability Tutorial

On this page
  1. What Redis Sentinel does
  2. Step 1: run three Sentinels with a quorum
  3. Step 2: what happens during a Sentinel failover
  4. Step 3: connect clients through Sentinel
  5. Common Sentinel problems and fixes
  6. Check yourself

What Redis Sentinel does

Yesterday's failover was manual: someone notices the primary is down, promotes a replica and repoints everything. Sentinel automates exactly that. By the end of this lesson you'll know how to run it, what it does second by second, and what your application has to do to follow a failover.

Step 1: run three Sentinels with a quorum

A Sentinel is a Redis process started in sentinel mode. Each one gets a tiny config naming the primary to watch:

Code
# sentinel.conf
port 26379
sentinel monitor app 10.0.0.10 6379 2      # name, primary address, quorum
sentinel down-after-milliseconds app 3000  # no reply for 3 s means "down" to this Sentinel
sentinel failover-timeout app 10000
Code
redis-server sentinel.conf --sentinel
redis-cli -p 26379 SENTINEL ckquorum app
# OK 3 usable Sentinels. Quorum and failover authorization can be reached

Run at least three, on different hosts from each other and ideally not all next to the primary. Sentinels discover the replicas and each other automatically from the primary.

Step 2: what happens during a Sentinel failover

I ran this locally with three Sentinels, one primary and one replica, then froze the primary. The log of one Sentinel, trimmed:

Code
01:02:55.225 +sdown master app 127.0.0.1 6390
01:02:55.297 +odown master app 127.0.0.1 6390 #quorum 3/2
01:02:56.372 +switch-master app 127.0.0.1 6390 127.0.0.1 6391

Reading it line by line:

  1. Subjectively down (+sdown). This Sentinel got no valid reply for 3 seconds. One Sentinel's view could be its own network problem, so nothing happens yet.
  2. Objectively down (+odown). Enough Sentinels agree, here 3 of a required quorum of 2.
  3. Leader election. The Sentinels elect one of themselves to run the failover. This step needs a majority of all Sentinels, not just the quorum, which is why one Sentinel alone can never fail over.
  4. Promotion (+switch-master). The leader picks the best replica, mainly the one with the most replicated data, promotes it, and repoints the other replicas. When the old primary comes back, it's turned into a replica.

From the primary freezing to a new primary took about 4 seconds here. In production, plan for several seconds of failed writes. Sentinel shortens the outage; it doesn't remove it.

Several Sentinels ping the primary. One that cannot reach it marks it subjectively down, and when a quorum agrees it is objectively down. They elect a leader Sentinel, which promotes the replica with the most complete replication offset and reconfigures the others. Clients ask Sentinel who the primary is.
Several Sentinels ping the primary. One that cannot reach it marks it subjectively down, and when a quorum agrees it is objectively down. They elect a leader Sentinel, which promotes the replica with the most complete replication offset and reconfigures the others. Clients ask Sentinel who the primary is.

Step 3: connect clients through Sentinel

After a failover the primary has a new address, so clients can't hardcode it. They ask Sentinel instead:

Shell
redis-cli -p 26379 SENTINEL get-master-addr-by-name app
# 1) "127.0.0.1"
# 2) "6391"

Mainstream client libraries do this for you. In Python with redis-py:

Code
from redis.sentinel import Sentinel

sentinel = Sentinel([("10.0.0.21", 26379), ("10.0.0.22", 26379), ("10.0.0.23", 26379)],
                    socket_timeout=0.5)
primary = sentinel.master_for("app", socket_timeout=0.5)   # follows failovers
replica = sentinel.slave_for("app", socket_timeout=0.5)    # for reads
primary.set("greeting", "hello")

In .NET, StackExchange.Redis does the same when the connection string names the service: ConnectionMultiplexer.Connect("10.0.0.21:26379,10.0.0.22:26379,10.0.0.23:26379,serviceName=app") asks Sentinel for the current primary and follows it when it changes (StackExchange.Redis docs).

Make sure your code retries writes that fail during the failover window, with a short backoff.

Common Sentinel problems and fixes

SymptomLikely causeFix
Failover never happensToo few Sentinels reachable for a majorityRun 3 or 5 Sentinels on separate hosts
Failovers on every network blipdown-after-milliseconds too lowRaise it above normal latency spikes, often 5 to 30 seconds
App keeps writing to the old primaryClient hardcodes the addressUse a Sentinel-aware client connection
Writes lost after failoverAsync replication gap (Day 6)min-replicas-to-write, WAIT for critical writes

On a managed service you don't run Sentinel yourself. The provider detects the failure, promotes a replica and repoints a DNS name or endpoint, so your client connects to that endpoint and only needs to reconnect and retry.

Sentinel gives you high availability for one primary. It doesn't add capacity: all writes still go to one node. When one node can't hold the data or the write load, you need Redis Cluster, tomorrow's lesson.

Check yourself

No. Electing a failover leader needs a majority of all Sentinels, and one of two isn't a majority. Run at least three.

No, that's expected. Sentinel has to notice the primary is gone, which depends on down-after-milliseconds, agree on it, elect a leader and promote a replica, which together usually take a few seconds. Make sure clients reconnect automatically and retry writes with backoff during that window.

Its view could be caused by its own network problem. A quorum must agree, and a majority must elect a leader, before anything changes.

They ask Sentinel, not a fixed address. Sentinel-aware clients connect to the Sentinels, ask for the current primary by its service name, and reconnect when Sentinel announces a switch. Hard-coding the primary's IP address means the app keeps talking to the dead node after a failover.

On three independent failure domains, such as separate hosts or availability zones, ideally not all on the same machines as the Redis nodes. If two Sentinels share a host that fails, the remaining one can't form a majority, and no failover happens.