Table Tests and Benchmarks

The tests you write first are the ones that are easiest to write, and that is the problem. A function with three cases gets three test functions, each with its own copy of the setup, and by the fourth case you are copying a block and hoping you changed every line that needed changing. A test that passes because it still checks the old input is worse than no test: it is a claim about the code that nobody verified.

Go's testing package has one shape that removes most of that: a slice of cases, one loop, one subtest per case. Parallel subtests, benchmarks and fuzz targets all build on it, and the tooling is designed around the names t.Run produces.

A table is the default shape, and t.Run is what makes it readable

The parser in this lesson reads a line of the form topic|partition|bytes|key=value,.... Eight behaviours matter and they are all the same kind of thing: an input, and either a result or an error. That is a table:

func TestParse(t *testing.T) {
	tests := []struct {
		name       string
		line       string
		wantTopic  string
		wantParts  int
		wantErr    error
	}{
		{name: "full line", line: line, wantTopic: "orders.created", wantParts: 3},
		{name: "no tags", line: "metrics|0|64", wantTopic: "metrics"},
		{name: "empty", line: "", wantErr: ErrShortLine},
		{name: "two fields", line: "topic|1", wantErr: ErrShortLine},
		{name: "partition is not a number", line: "topic|x|4", wantErr: ErrNotANumber},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			got, err := Fast(tt.line)
			if !errors.Is(err, tt.wantErr) {
				t.Fatalf("Fast(%q) error = %v, want %v", tt.line, err, tt.wantErr)
			}
			if tt.wantErr != nil {
				return
			}
			if got.Topic != tt.wantTopic {
				t.Errorf("Topic = %q, want %q", got.Topic, tt.wantTopic)
			}
		})
	}
}

Two decisions in that snippet carry most of the value. The cases name themselves, so a failure points at the case rather than a line number. And the expected error is a sentinel compared with errors.Is, not a string compared with ==. A test asserting on the exact text of a strconv error breaks the day somebody adds context to it, and breaks for the wrong reason: the behaviour did not change, the message did.

Passing cases print one line each when you ask for them, which is how a table tells you it covered what you thought it covered:

$ go test ./internal/feed/ -run TestParse -v
=== RUN   TestParse
=== RUN   TestParse/full_line
=== RUN   TestParse/no_tags
=== RUN   TestParse/empty
=== RUN   TestParse/two_fields
--- PASS: TestParse (0.00s)
    --- PASS: TestParse/full_line (0.00s)
    --- PASS: TestParse/no_tags (0.00s)
    --- PASS: TestParse/empty (0.00s)
    --- PASS: TestParse/two_fields (0.00s)
PASS

That same name is a filter. -run 'TestParse/two_fields' runs one case and skips the rest. -run 'TestParse/partition' runs every case whose name contains partition, so a table with sixty cases can be bisected by naming part of the input you are chasing. The separator is a slash and the pattern is a regular expression, not a glob.

Subtests also give each case its own *testing.T, which is what makes t.Fatalf correct inside the loop. Without t.Run, a Fatal in case three abandons cases four through sixty and the report never says which input did it.

t.Parallel plus a captured loop variable is the trap vet still checks

Cases that touch no shared state are worth running at the same time, and the marker for that is t.Parallel() as the first statement inside the subtest. The subtests pause, the parent function finishes building the table, and the parallel cases then run together at the end of the parent.

The mistake that used to come with it is worth understanding even though the language changed. Before Go 1.22 the loop variable was one variable reused by every iteration, so a closure that ran later saw the last value it held, which under t.Parallel() is the last case for every subtest. go vet caught it and still checks it. Point vet at a module that declares go 1.21:

$ go vet ./...
catalog/catalog_test.go:58:19: loop variable tt captured by func literal
catalog/catalog_test.go:58:34: loop variable tt captured by func literal
catalog/catalog_test.go:59:40: loop variable tt captured by func literal
catalog/catalog_test.go:59:52: loop variable tt captured by func literal

From Go 1.22 onward, with the module's go line at 1.22 or higher, each iteration of a for loop declares fresh variables, so the capture is safe and the same code reports nothing. It is a change in the language, and it only applies to modules that declare it: a module on go 1.21 keeps the old semantics under a newer toolchain. Tests written before 1.22 often carry tt := tt as the first line of the subtest; vet does not flag it and there is no reason to touch it.

The other half of t.Parallel is that it makes a data race in the code under test far more likely to appear, which is a feature. Run the suite with -race and a table whose subtests share a parser or a map reports the race instead of hiding it behind a lucky schedule.

-benchmem turns a benchmark into two numbers instead of one

A benchmark is a function whose name starts with Benchmark and which takes *testing.B. The framework calls it repeatedly, raising b.N until the run is long enough to measure, and reports time per iteration. A benchmark that reports only time cannot see what usually causes the time:

func BenchmarkParse(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		m, err := parse(line)
		if err != nil {
			b.Fatal(err)
		}
		consume(m) // keep the result observable or the compiler may remove the call
	}
}

$ go test ./internal/feed/ -run '^$' -bench '^BenchmarkParse$' -benchmem -count=8
BenchmarkParse-8   	 3254022	       364.4 ns/op	     448 B/op	       4 allocs/op

Four allocations per parse, 448 bytes. The number after the benchmark name is GOMAXPROCS, so BenchmarkParse-8 ran on a machine configured for eight threads, and two runs on different machines are not comparable. -count=8 shows the spread rather than one number that might be the good one. Nothing in a benchmark is trustworthy until you have seen it repeat.

The allocation count is the number to watch when a profile says the collector is busy: 4 allocations per call on a hot path is 4 million per million requests, and each one is work the collector has to undo. Cutting allocations moves flat CPU time in a way that hand-tuning instructions rarely does.

A benchmark without b.ResetTimer after setup measures your setup

The timer starts before the benchmark function is called, so everything it does before the loop is charged to the measurement. Code that reads a fixture, opens a file or waits for a connection is setup, and leaving it inside the timed region makes the reported number a property of the fixture. At -benchtime=1x, which runs the loop exactly once and is the honest way to see this, the difference is the whole number:

$ go test ./internal/feed/ -run '^$' -bench 'ParseWithSetup' -benchtime=1x
BenchmarkParseWithSetup-8        	       1	 201206625 ns/op
BenchmarkParseWithSetupReset-8   	       1	     27375 ns/op

Both benchmarks sleep 200 milliseconds and then parse one line. The first reports 201 milliseconds because the sleep is inside the timed region; the second reports 27 microseconds because b.ResetTimer() sits between the setup and the loop. Twenty-seven microseconds is a sensible number for parsing a 60-byte line with a map allocation, and 201 milliseconds is a number about a sleep.

The subtler version of the same bug is a benchmark whose setup is small, so the number looks plausible. At -benchtime=5x the same two benchmarks report around 40 milliseconds and 5.5 microseconds, and the first now looks like a slow parser rather than a broken measurement. The fix is always the same three lines: build the fixture, call b.ResetTimer(), enter the loop.

Two smaller pieces are worth knowing. b.StopTimer() and b.StartTimer() bracket work inside the loop that is not part of what you are measuring, such as regenerating an input. And if the compiler can prove a result is unused it deletes the call, so assign to a package-level variable or hash it.

benchstat compares two runs; your eyes compare two moods

benchstat is not in the Go distribution. It is a command from the golang.org/x/perf module and it is installed with the standard go install form:

$ go install golang.org/x/perf/cmd/benchstat@latest
$ go test ./internal/feed/ -run '^$' -bench '^BenchmarkParse$' -benchmem -count=8 > before.txt
# change the implementation
$ go test ./internal/feed/ -run '^$' -bench '^BenchmarkParse$' -benchmem -count=8 > after.txt
$ benchstat before.txt after.txt

        │ before-parse.txt │          after-parse.txt           │
        │      sec/op      │   sec/op     vs base               │
Parse-8        378.0n ± 5%   183.3n ± 3%  -51.49% (p=0.000 n=8)

        │ before-parse.txt │         after-parse.txt          │
        │       B/op       │   B/op    vs base                │
Parse-8         448.0 ± 0%   0.0 ± 0%  -100.00% (p=0.000 n=8)

        │ before-parse.txt │          after-parse.txt           │
        │    allocs/op     │ allocs/op   vs base                │
Parse-8         4.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=8)

The change was the parser: instead of splitting the line into a []string and building a fresh map per call, the new version walks the string by index and reuses a map the caller owns. Allocations per parse went from 4 to 0, bytes from 448 to nothing, and the time halved, from 378 nanoseconds to 183.

What benchstat adds is the second number on each line. ± 5% is the spread across the eight runs, p=0.000 the probability of a difference this large if the two runs were the same, and n=8 the sample count. A result of -3% (p=0.42) is noise; -3% (p=0.001) is a real three percent. Without that column, the only thing separating a real improvement from a warm laptop is which run you did second, and that is usually the one you believe.

A fuzzer finds the input you did not put in the table

A table is a list of the cases you thought of, which is its weakness. Fuzzing generates inputs instead, from a function that takes any supported type plus a seed corpus of examples you already know about. string and []byte are the two types that matter for a parser.

The target worth writing is differential: two implementations of one contract, which must agree on which lines are valid.

func FuzzFastAgainstSlow(f *testing.F) {
	f.Add(line)
	f.Add("metrics|0|64")
	f.Add("a|1|2|x=1,y=2")
	f.Add("||")

	f.Fuzz(func(t *testing.T, s string) {
		if n := strings.Count(s, "|"); n != 2 && n != 3 {
			t.Skip() // not a line the feed can send
		}
		want, werr := Slow(s)
		got, gerr := Fast(s)
		if (werr == nil) != (gerr == nil) {
			t.Fatalf("disagree on error for %q: Slow=%v Fast=%v", s, werr, gerr)
		}
		if werr != nil {
			return
		}
		if got.Topic != want.Topic || got.Partition != want.Partition || got.Bytes != want.Bytes {
			t.Fatalf("disagree for %q: Slow=%+v Fast=%+v", s, want, got)
		}
	})
}

The t.Skip() is not hiding a failure: the wire format has a fixed number of separators, so an input the feed cannot produce is outside the contract. Without the guard the fuzzer spends its budget on strings with forty pipes in them.

Run it for a fixed time, not forever:

$ go test ./internal/feed/ -run '^$' -fuzz FuzzFastAgainstSlow -fuzztime 30s
fuzz: elapsed: 0s, gathering baseline coverage: 0/6 completed
fuzz: elapsed: 0s, gathering baseline coverage: 6/6 completed, now fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 433579 (144521/sec), new interesting: 108 (total: 114)
fuzz: elapsed: 30s, execs: 6558593 (219828/sec), new interesting: 195 (total: 201)
PASS
ok  	lab.example/m6/internal/feed	30.380s

Six and a half million inputs and no disagreement, which is itself a result. The target that found something checked a property of one implementation: Fast calls strconv.Atoi on two fields, and an early draft discarded the error with _, so a malformed line came back as a message with partition zero. The property is direct: if the parser returns no error, its fields must be numbers that were in the input. The fuzzer minimised the failure to four bytes.

--- FAIL: FuzzFastKeepsAtoiErrors (0.00s)
    --- FAIL: FuzzFastKeepsAtoiErrors/56b5842c3bd02ba4 (0.00s)
        fuzz_test.go:105: Fast("|0|A") returned no error for bytes "A", which is not a number

    Failing input written to testdata/fuzz/FuzzFastKeepsAtoiErrors/56b5842c3bd02ba4

Four bytes, and the file it wrote is the regression test. A failing input is stored under testdata/fuzz/<TargetName>/, it replays on every later go test run without the -fuzz flag, and it should be committed. The bug cannot come back, because the case that found it is now part of the suite.

Show the solution and the reasoning

The first part is mechanical, but the failing case is the informative one. If the output names TestParse/partition_is_not_a_number, the table is doing its job. If it names a line number, the case is missing its name or the assertions are outside t.Run.

For the second part, the map-per-call version reports one more allocation per operation than the reuse version. What matters is the habit: two files, one tool, no conclusion from a single run. If the p-value comes back above 0.05, the honest reading is that your change did not move the metric, which is worth learning before shipping it.

The third part has a real trap: proving a negative. A fuzzer that runs for 60 seconds and finds nothing has shown the property held for the inputs it tried, not that it is true. Two things raise confidence without raising the runtime. Seed the corpus with the awkward inputs you already know, such as an empty tag list, a tag with no =, a value containing a comma, or a whitespace key, because the seed corpus is where the fuzzer starts and a target that never reaches a branch never tests it. And make the property specific enough to fail: "the parse succeeds" is nearly unfalsifiable, while "re-serialising the pairs produces the same set" fails the moment a tag is dropped.

One detail is worth checking by hand before you trust the result. If two tags share a key, the map keeps the last one and the round trip loses a pair, so either the property is "same set of pairs" for maps or the parser has to reject duplicate keys. Deciding which you mean is the actual work, and the fuzzer will say so as soon as it generates a=1,a=2.