Catalog My learning Blog
CI/CD and DevOps Tutorial: Build a Delivery Pipeline · Phase 1 · Lesson 6 · 4 min read

Automated Testing in CI/CD Pipelines

On this page
  1. Why automated tests matter in CI/CD
  2. The test pyramid: unit, integration and E2E
  3. Ordering test gates in a pipeline
  4. Dealing with flaky tests
  5. Using code coverage carefully
  6. Check yourself

Why automated tests matter in CI/CD

Deploying many times a day is only safe if automated tests in your CI/CD pipelines catch broken changes for you. But a test suite that takes an hour, or fails randomly, gets skipped or ignored. The skill is choosing which tests run where.

The test pyramid: unit, integration and E2E

LayerTestsSpeedHow many
UnitOne function or class, dependencies fakedMilliseconds eachMany (hundreds to thousands)
IntegrationYour code with real dependencies: database, queue, HTTP clientSeconds eachDozens to hundreds
End-to-end (E2E)The whole system through its UI or public APISeconds to minutes eachA few critical journeys
A pyramid with many fast unit tests at the base, fewer integration tests, and a small number of end-to-end tests at the top, each gating the next stage.
A pyramid with many fast unit tests at the base, fewer integration tests, and a small number of end-to-end tests at the top, each gating the next stage.

The shape is the point: most confidence comes cheaply from unit tests, integration tests catch the mistakes mocks hide (a wrong SQL query, a misread API response), and E2E tests confirm critical paths like signup and checkout work for real. An "ice-cream cone" of mostly slow E2E tests is slow, flaky and hard to debug.

Here's the difference between the bottom two layers in a few lines. The function has a typo in its SQL. The unit test fakes the database, so it accepts any query; the integration test uses a real (in-memory) database:

Python
# test_users.py: the same bug, missed by a unit test with a mock and caught by an integration test
import sqlite3, unittest
from unittest import mock

def get_email(db, user_id):
    row = db.execute("SELECT emial FROM users WHERE id = ?", (user_id,)).fetchone()   # typo: emial
    return row[0] if row else None

class UnitTest(unittest.TestCase):
    def test_returns_email(self):
        db = mock.Mock()                                   # a fake database: accepts any SQL
        db.execute.return_value.fetchone.return_value = ("ana@example.com",)
        self.assertEqual(get_email(db, 1), "ana@example.com")

class IntegrationTest(unittest.TestCase):
    def test_returns_email(self):
        db = sqlite3.connect(":memory:")                   # a real database: checks the SQL
        db.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, email TEXT)")
        db.execute("INSERT INTO users VALUES (1, 'ana@example.com')")
        self.assertEqual(get_email(db, 1), "ana@example.com")

if __name__ == "__main__":
    unittest.main(verbosity=2)
Code
test_returns_email (__main__.IntegrationTest.test_returns_email) ... ERROR
test_returns_email (__main__.UnitTest.test_returns_email) ... ok
sqlite3.OperationalError: no such column: emial

The unit test passes, because the fake answers whatever it's asked. Only the integration test, which runs the SQL against a real database, finds the bug. That's why you need both: unit tests to check your logic quickly, integration tests to check that your code really works with the things it talks to.

Contract tests (such as Pact) sit between: they check that a service and its consumers agree on an API without running both.

Ordering test gates in a pipeline

YAML
name: Tests
on: [pull_request]

permissions:
  contents: read

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 24, cache: npm }
      - run: npm ci
      - run: npm run test:unit

  integration:
    needs: unit
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_PASSWORD: test
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s --health-timeout 5s --health-retries 10
    env:
      DATABASE_URL: postgres://postgres:test@localhost:5432/postgres
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 24, cache: npm }
      - run: npm ci
      - run: npm run test:integration

  e2e:
    needs: integration
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 24, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --grep @smoke
  • needs chains the gates, so slow stages only run once fast ones pass.
  • The service container gives integration tests a real PostgreSQL 17, and the health check makes the job wait until it accepts connections.
  • E2E runs only tests tagged @smoke: a handful of critical journeys. The full E2E suite can run on a schedule or before a release.

Order is a trade-off: chaining saves runner minutes; running the stages in parallel gives faster total feedback. Many teams chain on pull requests and parallelize on main.

Dealing with flaky tests

A flaky test sometimes passes and sometimes fails without code changes. Common causes: timing (sleep instead of waiting for a condition), shared state between tests, test order dependence, real network calls, and time zones or dates.

  • Track flakiness: most CI systems and test reporters can flag tests with mixed results.
  • Quarantine a flaky test (run it, but don't block on it) and fix it within days, with an owner.
  • Don't add automatic retries everywhere: they hide real intermittent bugs.

Using code coverage carefully

Code coverage shows untested code, which is useful. As a target ("must be 90 percent") it encourages tests that execute lines without checking anything. Watch coverage of changed code in pull requests instead.

Check yourself

Unit tests run in milliseconds, point to the exact failing code and rarely flake. E2E tests are slow, break for many unrelated reasons and are hard to debug, so keep them for a few critical journeys like signup and checkout.

Real interaction bugs: invalid SQL, a wrong column name, serialization mismatches, transaction behavior, or a misunderstanding of how a dependency really responds. Mocks only behave the way you assumed.

Retries hide the problem, which may be a real race condition in the product; quarantine the test and fix the cause.

The mock accepts any SQL and returns whatever it was told to, so the query was never checked. An integration test against a real database would have run the SQL and failed.