Catalog My learning Blog
MongoDB Tutorial: Schema Design, Indexing and Sharding · Phase 3 · Lesson 19 · 4 min read

MongoDB Change Streams Tutorial

On this page
  1. Change streams vs polling
  2. How change streams work: the oplog
  3. Resume tokens: surviving restarts
  4. Change stream delivery: at least once
  5. Common uses for change streams
  6. Check yourself

Change streams vs polling

A search index must reflect product changes within seconds. The first version polls: every 10 seconds, query for documents with updatedAt greater than the last check. It misses deletes, misses updates that forgot to set updatedAt, loads the database with repeated queries, and still lags. Change streams push every change to you instead.

How change streams work: the oplog

Every write on a replica set is recorded, in order, in the oplog (the operations log that secondaries replicate from, Day 20). A change stream is a supported, filtered, resumable view of it. You can watch a collection, a database or the whole deployment. Because they're built on replication, change streams need a replica set or sharded cluster.

Python
# watch.py: react to new and updated orders
from pymongo import MongoClient

orders = MongoClient("mongodb://localhost:27017/?replicaSet=rs0").shop.orders
pipeline = [{"$match": {"operationType": {"$in": ["insert", "update"]}}}]

with orders.watch(pipeline, full_document="updateLookup") as stream:
    for change in stream:
        print(change["operationType"], change["fullDocument"])
        save_resume_token(stream.resume_token)       # persist progress (see below)

Inserting order 991 and then setting its status to paid printed, on MongoDB 9:

Code
insert {'_id': 991, 'status': 'placed'}
update {'_id': 991, 'status': 'paid'}

An update event normally contains only the changed fields (updateDescription). full_document="updateLookup" adds the current full document. MongoDB 6.0+ can also include the document before the change (fullDocumentBeforeChange) if pre-images are enabled on the collection: db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: { enabled: true } }). Pre-images are stored in an internal collection and expire with the oplog window, so enable them only where consumers need the old values.

Writes are recorded in the oplog; a change stream delivers matching events to the application, which stores the resume token to continue after a restart.
Writes are recorded in the oplog; a change stream delivers matching events to the application, which stores the resume token to continue after a restart.

Resume tokens: surviving restarts

Each event carries a resume token, its position in the stream. Store the token after processing each event (or batch), and on restart pass it back:

TypeScript
with orders.watch(resume_after=saved_token, full_document="updateLookup") as stream:
    ...

After restarting from a saved token and changing the order's status to shipped, the resumed stream delivered {'_id': 991, 'status': 'shipped'} and nothing that had already been processed.

The limit: resuming works only while the token's position is still in the oplog. The oplog is a fixed-size capped collection, so if a consumer is down longer than the oplog window (hours to days, depending on size and write volume), its token is gone and it must re-sync from scratch. Size the oplog for your longest expected outage, and monitor consumer lag.

Change stream delivery: at least once

Store the token after processing, and an event can be processed twice after a crash: at-least-once delivery, the same as with Kafka. Make handlers idempotent (upsert by _id, record processed event ids).

Common uses for change streams

  • Keep a search index (Atlas Search does this internally; Elasticsearch via a small consumer) or cache in sync.
  • Publish domain events to Kafka (the MongoDB Kafka connector uses change streams).
  • Real-time notifications and dashboards.
  • An outbox relay: watch an outbox collection written in the same transaction as the business change.

Check yourself

They read the oplog, so they capture every insert, update and delete in order, including changes made by code that forgot to set updatedAt and deletes that polling can never see, and they put no repeated query load on the collection.

Its resume token is no longer in the oplog, so it can't resume and must rebuild from a full copy of the data.

After a crash, the stream resumes from the last stored resume token, so events processed after that token but before the crash are delivered again. Handlers must cope with seeing the same change twice, for example by upserting or checking an event id.

After the change has been processed, ideally in the same write as its effect. Storing it before processing risks skipping an event if the handler crashes.

For update events it adds the current version of the whole document, not just the changed fields. That version may include later changes, so it's not a snapshot of the exact update.