Building a Benchmark That Repeats

Somebody says the site feels slow. You open it, press reload three times, and it feels fine, so the problem is declared fixed or imaginary. Nothing was measured. The next person to complain gets the same treatment, and the site stays slow for whoever is actually being served slowly.

A benchmark is the opposite of that. It is a fixed procedure that produces a number you can compare against another number from a different day. The hard part is not the tooling. It is deciding what to hold still so the comparison means anything, and writing down enough about the run that somebody else can repeat it.

A performance claim needs more than one observation

Requests to the same URL on the same server do not take the same amount of time. Nothing is broken about that; it is what a machine with other work to do looks like. Run a benchmark on a WordPress 7.0.7 install with 500 posts, PHP 8.5.6 and MySQL 9.6, all on one laptop, and you get this across ninety requests to the front page:

n=90  min=0.044216  median=0.04713  p95=0.051759  mean=0.0479  max=0.073748  non-200=0

Every one of those requests hit the same URL, on the same process, with the same data. The fastest took 44 ms and the slowest took 74 ms, which is a 68% spread. A single request picked at random from that set could have told you the page takes 44 ms or 74 ms, and you would have had no way to know which story you got.

That is the whole argument for a loop. Not precision, repetition. One request is an anecdote about one moment, and the moment you happened to measure is a poor sample of the moments your users get.

Two runs only compare if you held the same things still

Before you can compare two numbers you have to know what was allowed to change between them. The list is short and every item on it has burned somebody:

Write those six things down next to the number, every time. A result without them is not a measurement, it is a rumour with decimal places.

Cold cache and warm cache are two different experiments

This is the mistake that invalidates the most benchmarks. A cold-cache request is one where the cache does not have the answer and the application has to build it. A warm-cache request is one where the cache answers and the application does less. They have different code paths, different query counts, and different costs.

Run them into the same bucket and you get a number that is a weighted average of two populations, where the weights depend on the order you happened to fire the requests in. Change the order and the number changes, which makes the benchmark useless for the only thing a benchmark is for: telling you whether a change made things better or worse.

So decide which experiment you are running, and say so:

• Warm-cache benchmark. You want the latency a typical visitor sees. Fire the requests you are discarding as warm-up, then measure, and do not clear anything in between.
• Cold-cache benchmark. You want the cost of a cache miss, which is what a deploy, a purge or a cache eviction hands your users. Clear the cache before every measured request, or measure the very first request after a controlled flush, and never mix it with warm numbers.

The reason to care about both is that they fail differently. A site with a warm median of 47 ms is fine until a popular cache entry expires, at which point the cold path is the only path that exists and nobody has measured it. Module 6 is about what happens when that expiry arrives under load. For now, keep the two experiments apart so you can see both numbers.

Report the median and the p95, not the mean

The mean is the number everybody reaches for and the one that lies most often. It is the only summary that a single slow request can move by a visible amount, and slow requests are exactly what you are hunting.

Go back to the ninety requests above. The mean is 47.9 ms and the median is 47.1 ms. They look similar, so the difference seems academic until you look at the batches separately. One batch of thirty had a median of 47.3 ms and a p95 of 51.8 ms, with a single outlier at 73.7 ms. The mean for that batch was 48.2 ms: half a millisecond above the median, entirely because of one request out of thirty. If one request in thirty can move the mean by half a millisecond here, imagine what it does to a batch where a garbage collection pause or a database checkpoint lands.

Two numbers do the job better:

• Median (p50). The typical request. Half the requests were faster, half slower. It barely moves when an outlier arrives.
• p95. The slow request. One request in twenty was slower than this. It is the number that tells you whether the tail is under control, and the tail is where users notice.

Report both, and report n. Thirty requests gives you a p95 that is the second slowest request in the batch; three hundred gives you something closer to the fifteenth. Either is fine for comparing two versions of the same page on the same machine, which is all you are doing. Neither is a capacity plan.

Discard the warm-ups before you start counting

The first requests after a restart are not representative, and they are not interesting. PHP compiles files into OPcache, MySQL loads the tables into its buffer pool, the filesystem caches the template files, and the theme's transients are built for the first time. All of that is real work, and none of it happens on the thousandth request of the day.

So throw the first requests away and say how many you threw away. Five is enough for a small WordPress site. If you are measuring anything with a large first-request cost, such as a full-page cache miss that has to render a twenty-thousand-post archive, ten warm-ups cost you seconds and remove a whole class of nonsense from your baseline.

The rule to write in your notes: warm-ups are not measurements, and a benchmark whose first request is its fastest is not a benchmark.

Record one row per request, then summarise

The temptation is to let the loop print a running average. Do not. Write one row per request to a file, unsummarised, and compute percentiles afterwards. When a number looks wrong you will want to see the distribution, not a summary of it, and you cannot recover the distribution from an average.

One more thing worth knowing before you trust any number: WordPress keeps the time it measured for each query in $wpdb->queries, and that value includes the whole call through $wpdb, not just the time MySQL spent executing the statement. It is good enough for ranking queries against each other. It is not a stopwatch on the database, and lesson 2 is about the difference.

#!/usr/bin/env bash
set -euo pipefail

label="${1:?usage: bench.sh <label> <url>}"
url="${2:?usage: bench.sh <label> <url>}"
out="runs/$label.txt"
warmups=5
measured=30

mkdir -p runs
: > "$out"

for i in $(seq 1 $warmups); do
  curl -s -o /dev/null "$url"
done

for i in $(seq 1 $measured); do
  curl -s -o /dev/null \
    -w "%{time_total} %{time_starttransfer} %{http_code} %{size_download}\n" \
    "$url" >> "$out"
done

echo "wrote $out (discarded $warmups warm-ups)"

#!/usr/bin/env bash
set -euo pipefail

file="${1:?usage: summarise.sh <file>}"
sorted=$(sort -n -k1,1 "$file")

median=$(echo "$sorted" | awk '{v[NR]=$1} END {print (NR%2) ? v[(NR+1)/2] : (v[NR/2]+v[NR/2+1])/2}')
p95=$(echo "$sorted"    | awk '{v[NR]=$1} END {i=int(0.95*NR+0.999); print v[i]}')
mean=$(echo "$sorted"    | awk '{s+=$1} END {printf "%.4f", s/NR}')
min=$(echo "$sorted"     | head -1 | awk '{print $1}')
max=$(echo "$sorted"     | tail -1 | awk '{print $1}')
errors=$(awk '$3 != 200' "$file" | wc -l | tr -d ' ')

printf 'n=%s  min=%s  median=%s  p95=%s  mean=%s  max=%s  non-200=%s\n' \
  "$(wc -l < "$file" | tr -d ' ')" "$min" "$median" "$p95" "$mean" "$max" "$errors"

n=30  min=0.044216  median=0.0472805  p95=0.051759  mean=0.0482  max=0.073748  non-200=0
n=30  min=0.045374  median=0.0470055  p95=0.050890  mean=0.0476  max=0.052267  non-200=0
n=30  min=0.044821  median=0.0473905  p95=0.051947  mean=0.0480  max=0.055701  non-200=0