Test Design That Catches Real Bugs

A suite of forty green tests feels like evidence. It is not evidence of anything until you have watched those tests go red for the right reason. The most expensive tests in a codebase are the ones that pass no matter what the code does: they cost review time, give back nothing, and make everyone more confident than they should be.

Everything in this lesson follows from one question you should be able to answer about every test you write: what change to the code, made by accident, would make this test fail? If the answer is "I am not sure", the test is not finished.

A test that cannot fail is worse than no test

The cheapest way to find out whether a test can fail is to break the code and run it. Do that when you write the test, not when you are debugging at midnight. If the test stays green while the function returns the wrong answer, the assertion checks something the bug does not affect.

Two shapes of this are common enough to have names. The first is the test with no assertion at all: it calls the function and inspects nothing, so it fails only if the call raises. It catches a crash, which is not nothing, but it is a weaker claim than the test name suggests. The second is the assertion on a shape rather than a value:

def test_returns_a_list():
    result = top_words("the quick brown fox")
    assert isinstance(result, list)

It passes for an empty list, a list of the wrong words, and a list of integers, and it asserts a type the function's own signature already promises. The version that carries information names the content:

def test_returns_the_most_common_word_first():
    assert top_words("the quick brown fox")[0] == ("the", 1)

That one fails when the sort order changes, when the count is wrong, when the word is not lowercased, and when the function returns nothing. Same function, same two lines, an order of magnitude more value.

Arrange, act, assert: three blocks you can point at

Almost every useful test has three parts, and in a well-written test you can point at the lines that belong to each. Arrange sets up the input and any collaborator. Act calls one thing. Assert states the expectation.

def test_totals_a_list_of_durations():
    week = {"on_call": 12, "standup": 5}                      # arrange
    total = sum(parse_duration(f"{hours}h") for hours in week.values())  # act
    assert total == 61200                                     # assert

There is no rule about blank lines, and pytest does not care. What the shape buys you is a diagnostic move. When a test fails, the first question is whether the act step did what you think it did. If the arrange block is twenty lines long, that question takes twenty lines to answer, and the failure message points into the middle of setup rather than at the behaviour under test.

When arrange gets long, move it into a fixture. Do not intersperse setup with assertions, which produces a test where the fourth assertion depends on a mutation performed by the third, and a failure on the fifth tells you little about the first four.

One reason to fail per test

If a test asserts six unrelated properties and fails, one of six things is wrong and you have to read the code to find out which. If it asserts one, the failure message is the diagnosis. Smaller tests fail faster, in both senses.

The exception is a sequence of assertions describing one behaviour from several angles, where the later ones are meaningless if the earlier ones fail. Checking that a parser returns the right object and that its fields are populated is one reason to fail, because a wrong object makes the field checks noise. Checking a parser and a separate formatter in the same test is two.

Parametrisation is usually better than a bigger test: one body, many rows, and pytest reports which row failed.

@pytest.mark.parametrize(
    ("text", "expected"),
    [
        ("", []),
        ("solitary", [("solitary", 1)]),
        ("a a a b", [("a", 3), ("b", 1)]),
        ("café CAFÉ", [("café", 2)]),
    ],
)
def test_boundaries(text, expected):
    assert top_words(text) == expected

Each row is a separate node ID, so a failure names the input. This is also the compact way to cover the ugly inputs, which is the next subject.

The boundary cases are where the bugs live

The input you imagine while writing the function is a normal one. The inputs that break it are at the edges, and five of them are worth a thought every time you write a function that takes a collection or a string.

Empty is the case people skip, because it feels like a caller error rather than an input. That is why it needs a decision rather than a default. Here is a real suite where the empty input was the only failure out of seven parametrised cases:

    @pytest.mark.parametrize(
        "text",
        ["", "1", "h", "1h30", "1H", " 1h", "1h "],
    )
    def test_rejects_malformed_input(text):
>       with pytest.raises(ValueError):
E       Failed: DID NOT RAISE ValueError

FAILED tests/test_edges.py::test_rejects_malformed_input[]
1 failed, 6 passed in 0.02s

The function raised for a trailing number, a bare unit letter, a number with no unit, and uppercase, but it returned 0 for an empty string. Whether that is a bug is a design question: an empty duration is either zero seconds or a typo, and the function has to pick one and say so in a docstring and a test. Leaving the choice to the loop's initial value, which is how 0 got there, is not a decision.

A mock assertion proves the call happened, not that anything works

A mock that records calls is the right tool when the thing you are checking is the interaction itself, such as "when the request fails, the client retries once". The mistake is using a mock when you meant to test a result. Here is a function whose only job is to compose a welcome message and hand it to a notifier. It has a real bug: it sends an empty body.

def signup(name, email, notifier):
    notifier.send(email, "")
    return {"status": "ok"}

This test passes against that code. I ran it:

from unittest.mock import Mock

from notify import signup


def test_signup_sends_a_welcome():
    notifier = Mock()
    signup("Ana", "ana@example.com", notifier)
    notifier.send.assert_called_once()

1 passed in 0.00s

assert_called_once() asks nothing about the arguments. The name of the test is a lie: nothing was welcomed. Narrow the assertion to the arguments and the bug surfaces:

def test_signup_sends_the_welcome_message():
    notifier = Mock()
    signup("Ana", "ana@example.com", notifier)
    notifier.send.assert_called_once_with("ana@example.com", "Welcome, Ana")

E           AssertionError: expected call not found.
E           Expected: send('ana@example.com', 'Welcome, Ana')
E             Actual: send('ana@example.com', '')

FAILED test_signup.py::test_signup_sends_the_welcome_message - AssertionError
1 failed in 0.05s

That is a better test, and it still has the weakness of every mock-based test: it reproduces the implementation's own idea of what it should send. Reword the welcome message and the test fails, so it is checking the call rather than the outcome.

A fake object that stores real values gets you out of that. It is a small class you write, it has the same method the production collaborator has, and the test asserts on what a recipient would see:

class RecordingNotifier:
    def __init__(self):
        self.sent = []

    def send(self, address, body):
        self.sent.append((address, body))


def test_signup_sends_a_body_a_person_can_read():
    notifier = RecordingNotifier()
    signup("Ana", "ana@example.com", notifier)
    address, body = notifier.sent[0]
    assert address == "ana@example.com"
    assert "Ana" in body

This version fails against the broken function with a message about the product rather than about a call signature:

E       AssertionError: assert 'Ana' in ''

FAILED test_signup_fake.py::test_signup_sends_a_body_a_person_can_read
1 failed in 0.01s

It also survives a reworded welcome message. Use a mock when the interaction is the behaviour; use a fake when the result is.

Test behaviour, not implementation

A test that knows how the function works internally breaks every time the internals improve; a test that knows what the function promises survives a rewrite. Asserting that a collaborator's send was called once tells you about the implementation's sequencing. Asserting that the notification reached a recorded recipient tells you about the outcome. Asserting that top_words called Counter.most_common with 3 tells you nothing a caller can observe, and it goes red the day someone replaces Counter with a faster hand-rolled loop.

The practical test is to ask whether a reviewer who rewrote the function body entirely, keeping the signature and the documentation, would still expect every test to pass. If yes, the tests describe the contract. If no, they describe this week's implementation, and they get deleted rather than updated the first time someone optimises it.

Speed is a design signal, not just an annoyance

A test suite that takes eight minutes to run gets run once a day. A suite that takes six seconds gets run between every save. The threshold is somewhere around a minute for a suite you run while thinking, and it is a design constraint on the tests, not a virtue in itself.

When one test dominates the wall clock, pytest points at it. Here is the same behavioural test against two implementations of a deduplication helper, first with a list as the membership check, then with a set:

python -m pytest tests/test_dedupe.py -q --durations=3

============================= slowest 3 durations ==============================
4.83s call     tests/test_dedupe.py::test_handles_a_large_input
3 passed in 4.84s

============================= slowest 3 durations ==============================
0.01s call     tests/test_dedupe.py::test_handles_a_large_input
3 passed in 0.01s

The test body never changed. The 4.83 seconds was the in check walking a list of 40,000 items, which is the quadratic behaviour a very-large-input case is designed to expose. Treat a slow test as information about the code before you treat it as a reason to add a marker that skips it. The numbers are from one machine and will differ on yours; the ratio is the part worth remembering.

Slow tests have two other causes and they need different answers. A test that talks to the network or a real database is slow because of the dependency, and the fix is a fake. A test that is slow because it builds an enormous fixture is usually testing two things at once. Only after those have been ruled out is a sleep in a test a real signal that the code under test has a timing dependency you have not made explicit.

import pytest

from wordcount import top_words


@pytest.mark.parametrize(
    ("text", "expected"),
    [
        ("", []),                          # empty
        ("solitary", [("solitary", 1)]),   # one item
        ("a a a b", [("a", 3), ("b", 1)]), # duplicates
        ("café CAFÉ", [("café", 2)]),      # unicode, case-folded
    ],
)
def test_boundaries(text, expected):
    assert top_words(text) == expected


def test_a_large_document_returns_one_entry():
    text = "word " * 200_000
    assert top_words(text, limit=1) == [("word", 200_000)]