Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 5 · Lesson 50 · 5 min read

System Design: Order Processing Pipeline

On this page
  1. The problem: design an order processing pipeline
  2. What the customer waits for: the synchronous checkout
  3. Publishing events reliably with a transactional outbox
  4. Fan-out to side effects with pub/sub
  5. Fulfillment as an orchestrated saga
  6. Handling duplicates and ordering: the inbox pattern
  7. Building the order timeline from events
  8. Capacity check and stuck-order alerts
  9. What a strong order pipeline design includes
  10. Check yourself

The problem: design an order processing pipeline

An online store processes up to 2,000 orders per minute at peak. Checkout must respond within a second. After an order is placed, stock is reserved, payment captured, a shipment booked, emails sent, loyalty points awarded and analytics updated. Services each own their database. Nothing may be lost or done twice, and support needs a full timeline of every order. Design the pipeline.

What the customer waits for: the synchronous checkout

Synchronous, in the checkout request:

  1. Validate the cart and prices.
  2. Authorize the payment (not capture).
  3. Write the order with status PLACED, in one local transaction.
  4. Return "Order received" with the order number.

Everything else happens after the response.

Publishing events reliably with a transactional outbox

If the order commits but publishing OrderPlaced fails, downstream services never hear about it. If you publish first and the commit fails, they hear about an order that doesn't exist. The transactional outbox fixes both:

SQL
BEGIN;
INSERT INTO orders (id, customer_id, total, status) VALUES (991, 77, 129.00, 'PLACED');
INSERT INTO outbox (id, topic, payload) VALUES (gen_random_uuid(), 'orders', '{"type":"OrderPlaced","order_id":991}');
COMMIT;

A relay process reads new outbox rows and publishes them to the broker, marking them sent. The event exists if and only if the order does. Change data capture (Debezium reading the outbox table) is a common way to run the relay. In .NET, MassTransit, NServiceBus and Wolverine ship an outbox built on EF Core or their own storage.

Three relay details: it delivers at least once, because it can crash after publishing and before marking a row sent; it should publish rows in order per order id; and sent rows must be cleaned up, or the outbox table grows forever.

Fan-out to side effects with pub/sub

OrderPlaced goes to a topic. Independent subscribers, each with its own queue or consumer group:

  • notifications sends the confirmation email,
  • loyalty awards points,
  • analytics records the sale,
  • the fulfillment saga starts.

Each subscriber retries on its own and has a dead-letter queue. An email outage delays emails; it doesn't touch fulfillment.

Fulfillment as an orchestrated saga

An orchestrated saga, because the steps have order and compensation:

Code
ReserveStock  -> CapturePayment -> BookShipment -> mark order CONFIRMED
on failure:      void/refund       release stock     mark order FAILED, notify customer

The orchestrator persists its state per order, so restarts resume mid-saga.

Handling duplicates and ordering: the inbox pattern

Every consumer can receive a message twice. Each records processed event ids (or uses natural keys like "points for order 991") and skips repeats. Events for one order use the order id as the partition key (Day 43), so they arrive in order per order.

Record the processed id in the same transaction as the effect. If loyalty adds points and then crashes before writing "event 5501 done", the redelivered event adds them again. Writing both together, sometimes called an inbox, makes the check exact:

SQL
BEGIN;
INSERT INTO processed_events (event_id) VALUES ('5501');   -- fails if already processed
UPDATE loyalty SET points = points + 129 WHERE customer_id = 77;
COMMIT;

Building the order timeline from events

Every state change is an event: OrderPlaced, StockReserved, PaymentCaptured, ShipmentBooked. A projection builds the support team's timeline view and the customer's order page from them, and can be rebuilt if its logic changes.

The order pipeline: synchronous authorization and outbox write, event fan-out to subscribers, saga-driven fulfillment and an event-based order timeline.
The order pipeline: synchronous authorization and outbox write, event fan-out to subscribers, saga-driven fulfillment and an event-based order timeline.

Capacity check and stuck-order alerts

Code
2,000 orders/min ≈ 33 orders/s; with ~8 events per order ≈ 270 events/s

Modest for any broker. Size consumers by their slowest dependency, such as the shipping partner's rate limit, and let queues absorb bursts (Day 73).

Then watch for orders that stop moving. Alert when an order sits in PLACED for more than a few minutes, when a saga has been compensating for too long, or when a dead-letter queue isn't empty. Each one is a customer waiting for something that isn't coming.

What a strong order pipeline design includes

  • A short synchronous path with authorization, not capture.
  • The outbox, or another way events and data can't disagree.
  • Side effects as independent subscribers with retries and DLQs.
  • A saga with compensations for multi-service steps.
  • Idempotent consumers and per-order ordering.
  • A rebuildable history for support.
  • Alerts for stuck orders and sagas, not just for errors.

Check yourself

The customer gets an immediate yes or no while the funds are only reserved, not taken. If fulfillment later fails, for example because an item is out of stock, the authorization can simply be voided, which is cleaner and cheaper than charging the card and issuing a refund.

It makes saving the order and publishing its event atomic, so there's never an order without an event or an event without an order.

Make the check, the points award and recording the event id one database transaction, ideally relying on a unique constraint on the event id. Then either both the award and the record are saved or neither is, so a crash can't leave points awarded without the record that would block the duplicate.

The inbox pattern: the service records each processed event id, or that points for order 991 were already awarded, in the same transaction as the award, and skips any message whose id it has already seen. At-least-once delivery means duplicates will happen, so the consumer must make them harmless.