Graceful Degradation and Capacity Planning

There is a moment in every capacity incident when the site is slower than it should be and still up. Requests are being served, the p95 has trebled, and every additional visitor makes it worse for everyone. What happens next is decided by code that somebody wrote, and the only question is whether they wrote it in advance or discovered it live.

Everything before this lesson was about being fast at the traffic you have. This one is about the traffic you do not have yet: what the site gives up first, in what order, and how to put a number on the point where it has to. The arithmetic is the easy half. The order requires somebody to admit which features are optional.

Degradation is a list written before the incident, not a policy discovered during it

Under capacity, a WordPress site does not choose. Requests arrive, PHP workers pick them up, and whatever is at the front of the queue consumes the pool. The default degradation order is the arrival order, which is to say there is no order: an admin-ajax call from one editor can occupy a worker a checkout needed. The queue is fair and the work is not equally valuable.

Writing the list down means answering three questions for each feature. Does it have to be correct, or is an old answer acceptable? Can it be done later instead of now? And what is the cheapest version of it that still looks like the feature? A recommendation block fails all three tests and is the first thing to turn off. A search results page can be stale by a minute. An order confirmation can be queued. A payment authorisation can be neither stale nor queued, which is why it is the thing everything else is protected for.

A workable order for a shop, from first to last, is: analytics and tracking beacons, personalisation and recommendation blocks, related products, non-critical admin-ajax endpoints, search and filtering, the cart, checkout, then the product and category pages that bring the traffic in. Write your own in that shape and put it in the repository. The value of the exercise is not the list; it is that somebody has to say out loud that recommendations rank below search.

A shed response is cheap, honest and measurable, and it is not a 500

When a site is over capacity, the two bad options are to keep accepting work you cannot finish and to fail with a 500. The first turns a slow site into an unavailable one, because a PHP worker blocked on a query that will never finish is not serving a page that would. The second tells the client the request was wrong, which is false, and tells every cache and load balancer in the path that something is broken, which invites retries.

The response you want instead is a shed: a 503 with a short Retry-After, produced by code that does almost nothing, in the smallest number of lines it can be. The cost of that response is the whole point. Here is what WordPress produces for such a response in the lab, using wp_die() with a 503 argument from a request that had already booted WordPress:

HTTP/1.1 503 Service Unavailable
Retry-After: 5
Content-Type: text/html; charset=UTF-8

<title>Service Unavailable</title>
<p>Busy. Try again in a few seconds.</p>

That response came after WordPress loaded, so it cost whatever a page load costs even though it serves no page. A shed that boots the application competes for the workers it is trying to protect. The honest version belongs earlier: a check on a connection counter or a Redis key before the application loads, answered with a static body. What this lab cannot show is how a real front-end behaves when the origin sheds, because that depends on the proxy. The useful thing to verify there is that the 503 passes through with its header rather than being rewritten into a 200.

Read-only mode is for the requests that change data, not for reading

A read-only mode is usually described as protecting the database. The more useful way to think about it is that writes are the requests you can defer and reads are the requests you cannot, so under pressure the site keeps serving pages and stops changing rows. Product pages, category pages and search all keep working from cache and from the replica or primary they already use. Add to cart, checkout, comments, post publishing and anything that writes a session stop.

The implementation that survives load is a flag that is cheap to read and checked in the few places that write. Read it from a constant or an option already in memory rather than a fresh query, because a read-only mode that costs a database round trip to consult adds the load it is meant to avoid:

function shop_is_read_only(): bool {
    // A constant is a file deploy; a filter is an operator flipping it live.
    $default = defined( 'SHOP_READ_ONLY' ) && SHOP_READ_ONLY;
    return (bool) apply_filters( 'shop_read_only', $default );
}

add_action( 'template_redirect', function () {
    if ( ! shop_is_read_only() ) {
        return;
    }

    // Product and category pages keep rendering. The pages that write stop.
    if ( function_exists( 'is_checkout' ) && ( is_cart() || is_checkout() ) ) {
        if ( ! headers_sent() ) {
            header( 'Retry-After: 300' );
        }
        wp_die(
            'The shop is not accepting new orders for a few minutes. Your basket is safe.',
            'Temporarily closed for maintenance',
            array( 'response' => 503, 'back_link' => false )
        );
    }
} );

Two details matter more than the code. The first is the message: a read-only mode that says "error" loses the order, and one that says the basket is safe keeps the session for the retry. The second is that refusing a write is not refusing a request. A visitor who can read the product and see that checkout is closed for four minutes behaves differently from one who gets a blank 503 on the whole site, and the difference shows up in support tickets rather than in metrics.

For a shop, failing open and failing closed are different decisions per endpoint, and the wrong one either loses money or breaks the law

When a dependency is down, code has to choose between two defaults. Failing open means treating the failed dependency as an empty or permissive answer and continuing. Failing closed means refusing to proceed. Neither is a good default in general, because the right answer depends on what is being protected.

Fail open: a recommendation block that cannot reach its service renders nothing; a shipping-rate lookup that times out falls back to a flat rate clearly labelled as an estimate; a search index that is unavailable falls back to a database query with no relevance ranking. The visitor gets a slightly worse site, and the shop keeps selling.

Fail closed: a payment authorisation that cannot reach the gateway does not complete, and the order is not created; a tax calculation that cannot determine the rate does not invent one; a stock check that cannot confirm availability does not promise a delivery date. A coupon validation that cannot reach its service does not apply the coupon. In each case the cost of a wrong yes is a refund, a chargeback, a tax filing or a regulatory problem, and the cost of a no is one abandoned attempt.

An availability rule and a correctness rule pull in opposite directions, and the plugin that fails open to keep the funnel moving is the plugin that lets a £0 order through at 3am. Write the decision per dependency, next to the code that calls it, and state what the failure looks like to the customer. If a dependency's failure mode is "tell the customer to try again", that is a correctness decision and it belongs in the degradation list, not in a try/catch somebody added during an incident.

Capacity is arithmetic once you have three numbers and a mix

The arithmetic is Little's law applied to the pool you have, and it needs a worker count, a service time, and the share of requests served from cache. Do not use a single-threaded latency as the service time at every concurrency level; the ramp from the previous lesson showed the same request at 56 ms with one client and 113 ms with eight, because the pool was already contending with itself.

Here is the model with plausible production numbers, and a script that computes it. Sixteen PHP workers, a cached request at 30 ms, an uncached request at 250 ms, a 90 percent hit rate, and a database that answers 2000 queries per second while an uncached request sends eight queries.

service time
  cache hit                   30.0 ms
  cache miss                  250.0 ms
  average at hit rate 0.90    52.0 ms

worker ceiling
  16 workers / 52 ms average =  308 req/s
  keeping 50% spare          =  154 req/s

database ceiling
  queries per request          0.80
  database can answer       2000 queries/s
  database allows           2500 req/s at this mix

first limit is the workers
  ceiling                      308 req/s
  plan for                     154 req/s

what the hit rate is worth
  hit rate 0.50 ->    114 req/s from workers,    500 req/s from the database
  hit rate 0.80 ->    216 req/s from workers,   1250 req/s from the database
  hit rate 0.90 ->    308 req/s from workers,   2500 req/s from the database
  hit rate 0.95 ->    390 req/s from workers,   5000 req/s from the database
  hit rate 0.99 ->    497 req/s from workers,  25000 req/s from the database

Four things fall out of that table. The site can hold 308 requests per second while the mix holds, and the pool is the limit. The plan is 154, because keeping half the ceiling unused is how a spike of unknown size is absorbed. The hit rate is worth more than the worker count: dropping from 90 percent to 50 percent removes two-thirds of the capacity without anyone touching the infrastructure. And the database is not the limit here, which is a claim you can only make because you counted the queries per uncached request. Raise that count from 8 to 40, which is what a page with an unindexed filter and a plugin that queries per item will do, and the database ceiling falls to 500 requests per second, still above the pool. Give the database a measured ceiling of 150 queries per second, which is what a small instance with a cold buffer pool and a busy schema will do, and the two limits swap: 188 requests per second, limited by MySQL, and adding PHP workers then does nothing at all.

Two inputs are the ones people guess. The database ceiling is not from a specification; it is the rate at which the slow query log starts filling and the p95 starts rising, measured on your schema with your data. The hit rate is not a constant either: it is a property of the mix and the time of day, and the hour it falls is usually the hour the site falls over. Both belong in the model as ranges, and the useful output is a sentence rather than a number: "the pool holds 200 to 300 requests per second, and the database starts to matter below a 70 percent hit rate".

def ceiling(workers, hit_ms, miss_ms, hit_rate,
            db_qps=None, queries_per_miss=8.0, queries_per_hit=0.0,
            headroom=0.5):
    hit, miss = hit_rate, 1.0 - hit_rate
    average_ms = hit * hit_ms + miss * miss_ms
    worker_rps = workers / (average_ms / 1000.0)

    limit, which = worker_rps, "workers"
    if db_qps:
        per_request = miss * queries_per_miss + hit * queries_per_hit
        db_rps = db_qps / per_request if per_request else float("inf")
        if db_rps < worker_rps:
            limit, which = db_rps, "database"

    return {
        "average_ms": average_ms,
        "worker_rps": worker_rps,
        "which_binds": which,
        "ceiling_rps": limit,
        "plan_rps": limit * (1.0 - headroom),
    }


print(ceiling(16, 30, 250, 0.9, db_qps=2000, queries_per_miss=8))