Catalog My learning Blog
System Design Roadmap: 100-Day Course for Engineers · Phase 8 · Lesson 80 · 5 min read

System Design: Highly Available Payment System

On this page
  1. The problem: design a highly available payment system
  2. The service tier: stateless and active-active
  3. Never charge twice: idempotency keys end to end
  4. Calling the payment processor safely
  5. Outcomes that arrive later: payment webhooks
  6. Keep card data out: tokenization and PCI DSS
  7. Payment data: synchronous replication and fencing
  8. Surviving a region failure
  9. SLOs, the SLA and chaos tests
  10. What a strong payment system design includes
  11. Check yourself

The problem: design a highly available payment system

Design the payment service for an online store. It takes payment requests from checkout and calls an external payment processor. Requirements: never charge a customer twice, never lose a successful charge, stay available through instance, zone and region failures, and publish an SLA of 99.95 percent. Peak load is 500 payment requests per second.

The service tier: stateless and active-active

The payment API is stateless, running active-active across three availability zones, sized N+1 so losing a zone still covers peak. Health checks remove bad instances; instances shut down gracefully so in-flight payments finish.

Never charge twice: idempotency keys end to end

Every request carries an idempotency key from checkout, such as order-991-attempt-1.

  1. Insert the key into payment_requests with a unique constraint, status PENDING, in the same transaction as the payment record.
  2. Call the processor, passing the same idempotency key. Major processors (Stripe, Adyen and others) accept one, so a retry to them is also safe.
  3. Store the result with the key, status SUCCEEDED or FAILED.
  4. A repeated request with the same key returns the stored result.

If the processor call times out, the outcome is unknown. Don't guess: retry with the same key, or query the processor for that key's status. A background job reconciles any requests left PENDING.

Calling the payment processor safely

  • Timeout: connect 1 s, request a few seconds, based on the processor's measured latency.
  • Retries: up to 3, exponential backoff with jitter, only for timeouts, 5xx and 429, always with the same idempotency key.
  • Circuit breaker: after repeated failures, fail fast. Fallback: if the business allows, accept the order as "payment pending" and retry from a queue; otherwise return a clear error so the customer can try another method.
  • Bulkhead: a dedicated connection pool for the processor, so its problems can't exhaust other work. Size it with Little's Law (Day 3): 500 requests per second at a typical 800 ms processor latency means about 400 calls in flight. If the processor slows to 3 seconds, that's 1,500. The bulkhead caps it, and also check the processor's own rate limits for your account.
  • Optional second processor for failover, if the business case justifies the integration cost.

Outcomes that arrive later: payment webhooks

Not every result comes back in the original call. A card that requires 3-D Secure, a bank transfer or a dispute finishes minutes or days later, and the processor tells you through a webhook. Treat webhooks like any other incoming message: verify the processor's signature so nobody can forge "payment succeeded", process each event idempotently by its event id, and reconcile with the processor periodically in case a webhook never arrives.

Keep card data out: tokenization and PCI DSS

Never let card numbers touch your servers. Use the processor's hosted payment fields or tokenization: the browser sends the card details straight to the processor and your service only ever sees a token. That keeps most of your system out of PCI DSS scope (Day 1), which is a requirement in its own right, and it removes the most valuable thing an attacker could steal.

Payment data: synchronous replication and fencing

Payment records live in a relational database with synchronous replication to a standby in another zone, so a committed payment survives losing the primary (RPO near zero within the region). Automatic failover with fencing prevents split-brain.

Surviving a region failure

Warm standby in a second region: the database replicates asynchronously, and a scaled-down copy of the service runs and is tested continuously. In a regional disaster, scale it up, promote the database and switch traffic. Target RTO 15 minutes, RPO under a minute. After failover, the reconciliation job compares local records with the processor's, using idempotency keys, to recover any payments in the replication gap.

A highly available payment system: stateless active-active service tier, replicated database, timeouts and retries with idempotency keys to the processor, and a warm standby region.
A highly available payment system: stateless active-active service tier, replicated database, timeouts and retries with idempotency keys to the processor, and a warm standby region.

SLOs, the SLA and chaos tests

  • SLI: share of valid payment requests answered successfully (charge succeeded or a clear decline) within 2 seconds.
  • SLO: 99.97 percent over 30 days, internal. SLA: 99.95 percent, external. The gap is the warning margin.
  • Chaos experiments: kill instances, inject processor latency and errors, fail over the database, and run a regional DR drill twice a year.

What a strong payment system design includes

  • Idempotency end to end, including toward the processor.
  • A clear answer for "the call timed out, did it charge?"
  • Resilience patterns with concrete settings, and a fallback decision.
  • Data durability through synchronous replication and backups.
  • A DR strategy matched to stated RTO and RPO.
  • SLOs, SLA and the tests that show the design works.
  • Webhooks verified and processed idempotently, and card data kept out of your systems.

Check yourself

You can't know. Retry with the same idempotency key, or query the processor by that key; never send a new charge without it. Reconcile any requests still pending.

So a payment is stored on the standby before success is returned to the customer. If the primary fails right after, the promoted standby already has every confirmed payment, so none are lost. Keeping it inside one region keeps the extra commit latency low.

Verify the processor's signature so the event is genuine, and record the event id so a redelivered webhook doesn't apply twice.

The SLO is an early warning. If the SLA promises 99.9 percent and the internal SLO is 99.95 percent, the team starts reacting while there's still margin, long before breaching the contract and owing credits.

Card details go straight from the customer's browser to the payment processor, which returns a token. Your systems store and use only the token, never the card number, so far fewer of your systems fall under PCI DSS requirements.