Catalog My learning Blog
MongoDB Tutorial: Schema Design, Indexing and Sharding · Phase 4 · Lesson 20 · 3 min read

MongoDB Replica Sets and Failover

On this page
  1. Why production MongoDB runs as a replica set
  2. Primary, secondaries and the oplog
  3. Replica set elections and failover
  4. Replica set layouts: three members, five members, arbiters
  5. Rollback after failover
  6. Read preferences: reading from secondaries
  7. Check yourself

Why production MongoDB runs as a replica set

A single mongod holds your data on one machine. Disk failure, a bad kernel update or a cloud host being retired, and the application is down, maybe with data loss. Production MongoDB runs as a replica set: several mongod processes with the same data, which elect a new primary automatically when the current one fails.

Primary, secondaries and the oplog

  • The primary accepts all writes.
  • It records each change in the oplog, a capped collection in the local database.
  • Secondaries copy oplog entries and apply them in order, so their data follows the primary's.

The oplog's size determines the oplog window: how far back a secondary (or a change stream, Day 19) can catch up from. A local test replica set defaulted to a 192 MB oplog; production sizes are much larger. Size it for your longest expected maintenance or outage.

The primary writes to its oplog, secondaries replicate it, and if the primary stops responding the remaining members elect a new primary by majority vote.
The primary writes to its oplog, secondaries replicate it, and if the primary stops responding the remaining members elect a new primary by majority vote.

Replica set elections and failover

Members send heartbeats every 2 seconds. If the primary isn't heard from for electionTimeoutMillis (10 seconds by default), a secondary calls an election. A candidate needs votes from a majority of voting members, and the member with the most recent data is preferred. Typical failover takes around 10 to 15 seconds, during which writes fail.

Drivers handle it: they discover the new primary, and retryable writes (on by default in modern drivers) retry a write once after a failover. Applications should still treat brief write errors as possible.

Replica set layouts: three members, five members, arbiters

LayoutSurvivesNotes
3 data-bearing membersOne member downThe standard: majority of 2 out of 3
5 membersTwo downLarger deployments or more regions
2 data + 1 arbiterOne down, but weaker durabilityAn arbiter votes but holds no data; avoid with majority write concern
2 membersNothingLosing one leaves no majority, so no primary

Spread members across availability zones (or data centers), so a zone outage leaves a majority. An odd number of voters avoids ties.

Rollback after failover

With asynchronous replication, the primary can acknowledge a write that no secondary has yet. If it then fails and a secondary without that write becomes primary, the old primary must roll back the unreplicated writes when it rejoins; they're saved to rollback files, not applied. Writes acknowledged with w: "majority" (the default write concern since MongoDB 5.0) are on a majority and can't be rolled back. Day 21 covers write and read concern.

Read preferences: reading from secondaries

By default all reads go to the primary. A read preference can send them elsewhere:

Read preferenceReads from
primary (default)Primary only
primaryPreferredPrimary, secondaries if it's unavailable
secondary, secondaryPreferredSecondaries (for analytics and reports)
nearestLowest network latency, for multi-region reads

Secondaries can be behind, so reads from them may be stale. Use secondaries for reports and analytics; read the user's own recent writes from the primary.

Code
mongodb://db1,db2,db3/shop?replicaSet=rs0&readPreference=secondaryPreferred&maxStalenessSeconds=90

Check yourself

Electing a primary needs votes from a majority of members. With two members, losing one leaves one vote out of two, which isn't a majority, so no primary can be elected and writes stop. Three members tolerate one failure.

It was acknowledged before replicating to a majority, and the new primary didn't have it, so it was rolled back. Majority write concern prevents this.

A secondary, using a secondary or secondaryPreferred read preference, or a hidden or analytics-tagged member, so the report doesn't compete with production traffic on the primary. The data may lag slightly, which is fine for a nightly report.

Usually around 10 to 12 seconds by default: about 10 seconds for members to decide the primary is unreachable, then a quick election. Drivers with retryable writes hide most of it from applications.

A member that votes but holds no data. With a primary, a secondary and an arbiter, losing a data node leaves one copy and can block majority writes, so a third data-bearing member is better.