Testing Strategies

Tests are usually sold as bug-catching. That undersells them. A codebase without tests is one you are afraid to change — every refactor is a gamble, so the shortcuts accumulate and the design ossifies.

What a test suite actually buys is permission to change the code. Everything below follows from that: a test worth keeping is one that fails when behaviour breaks and stays quiet when you merely rearrange the implementation.

The pyramid, and what each layer buys

Three layers, differing in scope, speed and what they can prove.

The shape is a pyramid because cost and flakiness rise faster than confidence as you climb. A unit test that fails names the broken function. An end-to-end test that fails tells you something, somewhere, in a chain of eight components, is wrong — or that a button rendered 200ms late.

The common failure is the inverted pyramid: a handful of unit tests under a mass of slow browser tests. The suite takes 40 minutes, fails randomly twice a week, and the team learns to re-run rather than investigate. A suite people ignore is worse than no suite, because it costs money and provides false comfort.

Unit tests: behaviour, not implementation

The unit test that survives refactoring asserts what the code does, not how.

def test_withdrawing_more_than_the_balance_is_rejected():
    account = Account(balance=100)          # arrange

    with pytest.raises(InsufficientFunds):  # act + assert
        account.withdraw(150)

    assert account.balance == 100           # state unchanged

Three things make this useful:

The name states the rule. When it fails in CI six months from now, the failure line alone tells you what broke. test_withdraw_2 tells you nothing.

Arrange, act, assert. Set up state, perform one action, check the outcome. When a test has three acts, it has three reasons to fail and the failure no longer localises the bug.

It asserts the observable outcome. It checks the balance and the exception — not that _validate_amount was called once with 150. Assert on internal calls and every rename breaks a test that was never about correctness.

Cover the edges deliberately, because that is where the bugs are: zero, one, empty, null, negative, the maximum, the boundary itself and just past it. Off-by-one errors live at exactly these values and nowhere else.

Test doubles: fake the boundary, not the internals

Unit tests need to isolate the code under test from slow or unpredictable dependencies — clocks, networks, payment gateways.

Stubs return canned answers so the test can proceed. Mocks additionally assert that a specific call was made. Prefer stubs. A mock asserting charge() was called with exact arguments is a copy of the implementation, and it will fail on refactors that changed nothing a user can see.

Fake at architectural seams — the payment client, the clock, the mailer. Faking the internals of the class you are testing means you are testing your fakes.

Integration tests: where the real bugs are

Integration tests exercise components together against real infrastructure — a real database, a real HTTP layer:

def test_creating_a_user_persists_them(client, db):
    response = client.post("/users", json={"name": "Alice", "email": "a@example.com"})

    assert response.status_code == 201
    assert db.query("SELECT email FROM users WHERE id = %s", response.json()["id"]) == "a@example.com"

This catches an entire class of bug unit tests cannot: a wrong column name, a migration never applied, a serialiser dropping a field, a transaction that never commits. Every one of those passes a fully mocked unit suite.

Run them against the same engine you deploy on. SQLite standing in for Postgres is fast and will happily accept syntax and type behaviour that production rejects. Disposable containers make a real database cheap enough that the substitution is no longer worth making.

Each test must set up and tear down its own data. Tests that depend on rows another test created pass alone, fail in parallel, and fail differently depending on order.

End-to-end tests: few, and only for money paths

An end-to-end test drives the real system as a user does. It is the only kind that proves the whole thing is wired together, and the most expensive to own.

test("a customer can complete checkout", async ({ page }) => {
  await page.goto("/cart");
  await page.getByRole("button", { name: "Checkout" }).click();
  await page.getByLabel("Card number").fill("4242424242424242");
  await page.getByRole("button", { name: "Pay" }).click();

  await expect(page.getByText("Order confirmed")).toBeVisible();
});

Restrict them to journeys that cost you money when broken: sign-up, login, checkout, the core action of your product. Anchor selectors to roles and labels rather than CSS classes, and wait for conditions rather than sleeping for a fixed duration — timing-based waits are the single largest source of flaky tests.

Treat a flaky end-to-end test as a broken test. Fix it or delete it; a suite that fails randomly trains people to ignore red.

Coverage measures execution, not correctness

Coverage tells you which lines ran during the suite. That is genuinely useful for finding untested code, and it is not a measure of quality:

def test_divide():
    divide(10, 2)     # 100% coverage of divide, asserts nothing

Because the number is easy to game and easy to mandate, targets distort behaviour. Around 70–80% is a reasonable expectation for most codebases; chasing 100% means writing tests for trivial getters and error branches that cannot occur, which adds maintenance cost and no confidence. Look at what is uncovered, not at the percentage — an untested payment path matters, an untested __repr__ does not.

Writing the test first

Test-driven development inverts the order: red, green, refactor. Write a failing test for the behaviour you want, write the smallest code that passes, then improve the design with the test holding you steady.

The benefit most people expect is coverage. The real benefit is design pressure: code that is hard to test is usually badly coupled, and writing the test first makes you feel that before you have built on top of it. Watching the test fail first matters too — a test that never failed may not be asserting anything.

TDD suits well-specified logic with clear inputs and outputs. It fits exploratory work and UI layout much less well, and treating it as a rule for all code is how it gets abandoned.

Check your understanding

Common mistakes

Testing the implementation. Asserting on private methods and internal call sequences produces tests that break on every refactor and catch no real bugs.

Shared mutable state between tests. Tests that pass in sequence and fail in parallel almost always share a database row, a singleton or a file.

Conditionals in tests. An if in a test means it asserts different things on different runs. Write two tests.

Only testing the happy path. The bugs live in the error branches, so those are the branches worth asserting on.

Ignoring a flaky test instead of fixing it. Re-running until green teaches the team that red means nothing, and the day it means something you will re-run anyway.

Treating coverage as the goal. A number satisfied by assertion-free tests measures effort, not safety.

Practice

Key takeaways

• Tests buy permission to change the code; catching bugs is a side effect of that.
• Keep the pyramid shape — many unit tests, fewer integration, a handful of end-to-end — because cost and flakiness rise faster than confidence as you climb.
• Assert observable behaviour, not implementation, or every refactor breaks tests that were never about correctness.
• One behaviour per test, arranged as arrange–act–assert, with a name that states the rule it protects.
• Prefer stubs to mocks, and remember mocks only verify your code against your assumptions — integration tests verify the assumptions.
• Run integration tests against the same database engine you deploy on, with per-test setup and teardown.
• Coverage measures execution, not correctness. Look at what is uncovered rather than at the percentage.
• A flaky test is a broken test; tolerating red trains the team to ignore it.

Further reading

• Kent Beck, Test-Driven Development: By Example (Addison-Wesley, 2002).
• Martin Fowler, "Mocks Aren't Stubs" (2007) and "TestPyramid" (2012).
• Mike Cohn, Succeeding with Agile (Addison-Wesley, 2009) — where the pyramid was first drawn.
• John Micco, "Flaky Tests at Google and How We Mitigate Them" (Google Testing Blog, 2016).
• Gerard Meszaros, xUnit Test Patterns (Addison-Wesley, 2007) — the vocabulary of doubles: dummy, stub, spy, mock, fake.