MongoDB Replica Sets and Failover
On this page
Why production MongoDB runs as a replica set
A single mongod holds your data on one machine. Disk failure, a bad kernel update or a cloud host being retired, and the application is down, maybe with data loss. Production MongoDB runs as a replica set: several mongod processes with the same data, which elect a new primary automatically when the current one fails.
Primary, secondaries and the oplog
- The primary accepts all writes.
- It records each change in the oplog, a capped collection in the
localdatabase. - Secondaries copy oplog entries and apply them in order, so their data follows the primary's.
The oplog's size determines the oplog window: how far back a secondary (or a change stream, Day 19) can catch up from. A local test replica set defaulted to a 192 MB oplog; production sizes are much larger. Size it for your longest expected maintenance or outage.
Replica set elections and failover
Members send heartbeats every 2 seconds. If the primary isn't heard from for electionTimeoutMillis (10 seconds by default), a secondary calls an election. A candidate needs votes from a majority of voting members, and the member with the most recent data is preferred. Typical failover takes around 10 to 15 seconds, during which writes fail.
Drivers handle it: they discover the new primary, and retryable writes (on by default in modern drivers) retry a write once after a failover. Applications should still treat brief write errors as possible.
Replica set layouts: three members, five members, arbiters
| Layout | Survives | Notes |
|---|---|---|
| 3 data-bearing members | One member down | The standard: majority of 2 out of 3 |
| 5 members | Two down | Larger deployments or more regions |
| 2 data + 1 arbiter | One down, but weaker durability | An arbiter votes but holds no data; avoid with majority write concern |
| 2 members | Nothing | Losing one leaves no majority, so no primary |
Spread members across availability zones (or data centers), so a zone outage leaves a majority. An odd number of voters avoids ties.
Rollback after failover
With asynchronous replication, the primary can acknowledge a write that no secondary has yet. If it then fails and a secondary without that write becomes primary, the old primary must roll back the unreplicated writes when it rejoins; they're saved to rollback files, not applied. Writes acknowledged with w: "majority" (the default write concern since MongoDB 5.0) are on a majority and can't be rolled back. Day 21 covers write and read concern.
Read preferences: reading from secondaries
By default all reads go to the primary. A read preference can send them elsewhere:
| Read preference | Reads from |
|---|---|
primary (default) | Primary only |
primaryPreferred | Primary, secondaries if it's unavailable |
secondary, secondaryPreferred | Secondaries (for analytics and reports) |
nearest | Lowest network latency, for multi-region reads |
Secondaries can be behind, so reads from them may be stale. Use secondaries for reports and analytics; read the user's own recent writes from the primary.
mongodb://db1,db2,db3/shop?replicaSet=rs0&readPreference=secondaryPreferred&maxStalenessSeconds=90Check yourself
Electing a primary needs votes from a majority of members. With two members, losing one leaves one vote out of two, which isn't a majority, so no primary can be elected and writes stop. Three members tolerate one failure.
It was acknowledged before replicating to a majority, and the new primary didn't have it, so it was rolled back. Majority write concern prevents this.
A secondary, using a secondary or secondaryPreferred read preference, or a hidden or analytics-tagged member, so the report doesn't compete with production traffic on the primary. The data may lag slightly, which is fine for a nightly report.
Usually around 10 to 12 seconds by default: about 10 seconds for members to decide the primary is unreachable, then a quick election. Drivers with retryable writes hide most of it from applications.
A member that votes but holds no data. With a primary, a secondary and an arbiter, losing a data node leaves one copy and can block majority writes, so a third data-bearing member is better.