Catalog My learning Blog
CI/CD and DevOps Tutorial: Build a Delivery Pipeline · Phase 4 · Lesson 24 · 3 min read

Deployment Rollback Strategies

On this page
  1. When a rollback can't work
  2. Ways to roll back a release
  3. Expand and contract database migrations
  4. More deployment safety nets
  5. Check yourself

When a rollback can't work

A release renames a database column. The new code has a bug, so the team rolls back. The old code starts up, looks for the old column, and fails on every request. The rollback made things worse, because the database had moved forward and couldn't follow.

Ways to roll back a release

MethodHowSpeed
Feature flag offDisable the new code path (Day 15)Seconds
Traffic switchSend traffic back to the old version in blue-green or canary (Day 14)Seconds
Redeploy previous artifactDeploy the last known-good image (Day 5)Minutes
Git revertRevert the commit; the pipeline or GitOps redeploys (Day 18)Minutes, plus pipeline time

Practice them. A rollback that's never been exercised fails when you need it, just like an untested backup.

Roll back or roll forward? Rolling forward means shipping a quick fix instead. Prefer rolling back when the cause isn't certain: it restores a known state. Roll forward when a rollback isn't possible or the fix is small and well understood. Either way, the first goal is ending user impact (Day 22).

Rollback options from fastest to slowest, and expand-and-contract migrations that keep both old and new code working during a release.
Rollback options from fastest to slowest, and expand-and-contract migrations that keep both old and new code working during a release.

Expand and contract database migrations

The safe way to change a schema is in phases, so every intermediate state works with both the old and the new code. Renaming email to email_address, tested in PostgreSQL (PGlite):

Phase 1: expand. Add the new column and backfill it. Old code keeps working because nothing it uses changed.

SQL
ALTER TABLE users ADD COLUMN email_address text;
UPDATE users SET email_address = email WHERE email_address IS NULL;

Phase 2: migrate code. Deploy code that writes both columns and reads the new one. During the rollout, old and new versions run side by side. In the test, a row written by old code (id 3) is missing the new column:

Code
after both versions wrote:
  {"id":3,"email":"cy@example.com","email_address":null}
  {"id":4,"email":"di@example.com","email_address":"di@example.com"}

So run the backfill again once the old version is gone:

Code
rows missing email_address after re-backfill: [{"n":0}]

If the new code has a bug at this point, rolling back is safe: the old column is still complete enough for the old code.

Phase 3: contract. Only after every running version uses the new column (days or weeks later), tighten and remove:

SQL
ALTER TABLE users ALTER COLUMN email_address SET NOT NULL;
ALTER TABLE users DROP COLUMN email;

After this, the old code really can't run, which the test confirms: column "email" of relation "users" does not exist. That's why contracting is a separate, later release.

The same thinking applies to APIs and message formats: add new fields first, stop using old ones, remove them last.

More deployment safety nets

  • Automated rollback on alarm during canary or blue-green deploys (Days 14 and 21).
  • Deploy freezes around high-risk periods only, not as a substitute for safe deploys.
  • Backups and point-in-time recovery for data mistakes no rollback can fix.
  • Small changes: the smaller the release, the easier it is to know what to undo.

Check yourself

The old code still reads and writes the old column name. Once the migration renames it, rolling back the code puts it against a schema it doesn't understand, so the rollback fails too.

Only after every running version of the application reads and writes the new column, and you no longer need to roll back to a version that uses the old one. That's a separate, later release.

When a rollback isn't possible, for example after an irreversible data migration, or when the fix is small, well understood and can ship faster than a rollback.

A rollback that has never been exercised may fail when it's needed. Practicing it shows it works and how long it takes.

A problematic feature can be turned off in seconds without a deploy, which is faster and less risky than rolling back the whole release.