Debugging a Live Store When You Cannot Attach a Debugger

A customer's store starts returning a white page on roughly one order in forty. Support has a screenshot and a timestamp. You have no debugger attached, no staging clone current enough to trust, and a business that would prefer you not restart PHP-FPM during trading hours.

Production debugging goes badly for a reason that has nothing to do with a shortage of tools. The tools you use locally are unavailable, and the ones that remain are quiet enough that people stop trusting them and start guessing.

The method is the one a debugger enforces: form a hypothesis, decide in advance what evidence would confirm it, collect that evidence, then remove the apparatus. Production only changes the size of the apparatus and how fast you take it out.

Your local debugging tools become liabilities on a live site

WP_DEBUG is the first casualty. Turning it on in production does not merely print notices into the response; with WP_DEBUG_LOG it writes them to wp-content/debug.log, which lives inside the document root on a default install unless the host has moved it or denied access to it. That file accumulates full filesystem paths and plugin directory names. On a 40,000 product catalogue, one mistyped template variable generates it faster than you can read it, and the paths tell anyone who fetches the file where your plugins live.

SAVEQUERIES is the second. It keeps every query for the request in $wpdb->queries, useful on a page running 60 queries and dangerous on a page running 30,000. The array holds the SQL, the calling function and the elapsed time for each one. On a WooCommerce store with a large catalogue and an unindexed lookup, one page can push that array past the PHP memory limit and turn a slow page into a fatal error, adding a second bug to the one you were investigating.

The interactive debugger is the third, and its failure mode is worse because it looks like it is working. Xdebug in single-step mode halts the request and waits for you. On your laptop that is invisible. On a live store, every second a session is paused is a second a PHP-FPM worker is occupied and doing nothing. Ten browser tabs, or one customer hitting the same code path, and you have consumed a share of the worker pool while investigating a latency problem you are now causing yourself.

The instruments you actually have are quiet and cheap

The web server's error log is the first place to look, and it rarely gets looked at first because it is boring. PHP fatal errors land there, and so does the upstream timed out line that explains why the load balancer returned a 502 even though WordPress never saw the request. A single Allowed memory size of 268435456 bytes exhausted line dated inside the incident window answers the question outright.

The slow query log is the second. Turn it on at the server rather than in PHP, set long_query_time to something honest like 0.5 seconds, and read the entries that fall inside the failure window. On a 12,000 order store running MariaDB 10.6, a missing index on the order meta table shows up here as near-identical entries with a large Rows_examined value.

Then there is wp profile, which is a WP-CLI command and therefore runs from the shell rather than from inside a visitor's request. You drive it yourself, so you can run it once, on your own schedule, and stop. A stage profile gives you the share of the request spent in bootstrap, main query, template and shutdown.

# One-off profiling run against production, over SSH, on a quiet minute.
wp profile stage --url=https://example.com/shop/

# Then the same synthetic request, grouped by the hooks that cost the most.
wp profile hook --url=https://example.com/shop/

The output is a table of stage names and elapsed seconds for one synthetic request. Treat it as one sample, not as a measurement of your traffic. One request from the server's own network is not the same object as ten thousand requests from shoppers on mobile connections.

A hypothesis is a decision about what you will throw away

The expensive failure in production work is not a wrong guess. It is instrumentation that answers nothing. Before you touch the server, write one sentence: "Checkout fails intermittently because the shipping rate lookup throws when the carrier returns an empty response." Then write the second sentence, which most people skip: "If that is true, the error log will contain a fatal on the shipping request, and the failures will cluster on orders shipping abroad."

That second sentence is what makes the log readable. If the log shows nothing at the incident times and the failing orders are geographically uniform, your hypothesis is dead, and you spent twenty minutes instead of a weekend.

The smallest possible instrumentation follows from that. Read a log that already exists, then enable a server log that already exists, then write a targeted logger that emits two lines and nothing else. A general purpose plugin that logs everything is the same mistake as turning on WP_DEBUG.

Mechanics of a logger you can leave on for ten minutes

A must-use plugin is the right container. It loads before ordinary plugins, and deleting one file removes it entirely, with no database residue. Keep it to one file with one observable event.

// wp-content/mu-plugins/checkout-trace.php
add_action( 'woocommerce_checkout_order_processed', function ( $order_id ) {
    $request_id = bin2hex( random_bytes( 4 ) );
    $line       = sprintf(
        "%s order=%d uri=%s\n",
        $request_id,
        $order_id,
        substr( $_SERVER['REQUEST_URI'] ?? '', 0, 80 )
    );

    // APPEND to its own file, never the site's debug.log.
    file_put_contents(
        WP_CONTENT_DIR . '/checkout-trace.log',
        $line,
        FILE_APPEND | LOCK_EX
    );
}, 10, 1 );

Four decisions in that snippet are worth naming, because each one prevents a specific disaster.

Its own file, outside the request's error handling. Writing to error_log() mixes your lines with PHP's own warnings, and on hosts that point log_errors at a shared file you will be reading other sites' noise. A dedicated file also means rm is a complete rollback.

A request identifier on every line. Without one, a log from a busy minute is an interleaved mess you cannot attribute to a single request, and you will waste an afternoon convincing yourself that two customers' lines are one sequence. Four random bytes is enough. A correlation ID from your reverse proxy is better: it also gets you the edge logs.

No logging inside a loop. A hook that fires once per order is fine. One that fires once per cart line, or once per product in a catalogue of 40,000, will write a file fast enough to matter. Count per-item events and write one line at the end of the request instead.

Assume the disk will fill. file_put_contents with FILE_APPEND has no size ceiling. On a store doing twenty orders a minute, that snippet writes roughly a megabyte an hour, which is survivable; logging inside a loop can write a gigabyte in an afternoon and take the database down when the partition fills.

The database is the last instrument and the most underused. You will not learn why an order totals the wrong amount by reading code alone, but you can learn what the stored values were. Read the order, its line items and its meta by primary key, and compare them against what the customer saw. A code path that cannot be reproduced is easier to explain when you can see the inputs it was handed.

The observability you should have had before the incident

Three records turn a debugging session into an afternoon rather than a week, and all three are things you can create now, before the next incident.

A record of what is deployed. Which commit, which plugin versions, which theme. wp core version, wp plugin list --fields=name,version,status and wp theme list --fields=name,version,status piped into a dated file gives you a comparable state to diff against. Without it, you cannot tell whether the problem started with a deploy or with a traffic change.

The timestamps of those deploys. A log line is only meaningful relative to change. "The error rate went up at 14:20" becomes a lead when a release landed at 14:18, and a dead end when the last release was on Tuesday. Keep deploys in the same timeline you keep incidents in.

An error rate baseline. One fatal error per hour is an emergency on a site that normally sees zero, and a Tuesday on a site that normally sees thirty. Count them from the web server's log so the number does not depend on your application, and record it on a normal week.

If it only reproduces under load, no quiet-server logging will find it

This is the limit, and it invalidates a whole class of otherwise reasonable work. Some bugs are not latent defects sitting in a code path waiting to be observed. They are interactions: two requests racing for the same cache key, a connection pool exhausted at 300 concurrent users, a lock that is free at 20 requests per second and contended at 200. Logging a production site during a quiet period changes the conditions that produce the bug, so the bug stops happening and the log stays empty. You will conclude, wrongly, that the problem was never real.

What finds that class of problem is a load test against a clone that can tolerate being hammered, or a staging environment seeded with production data shapes: the same post meta distribution, the same order volume in the relevant tables, the same number of active cron events. A staging copy of a 40,000 product store containing 200 products will not reproduce an indexing problem, because the query planner will choose a table scan and finish in milliseconds.

The corollary is uncomfortable. If you cannot load test and cannot clone at the right data volume, some of these bugs cannot be diagnosed from the outside at all. Reduce their blast radius, add a guard around the suspected interaction, and watch the error rate, rather than adding logging to a system that will not misbehave while you watch it.

What to do on Monday

1. Write the hypothesis and the refuting evidence as one paragraph each, before opening a shell. If you cannot write the second paragraph, you do not have a hypothesis yet.
2. Read the error log and the slow query log for the incident window first. Both are already collecting.
3. If you need instrumentation, add one must-use file that logs one event with a request ID to its own file, and note when you will delete it.
4. Delete it, then confirm the file is gone and that WP_DEBUG is off. Grep the web root for debug.log and any file your instrumentation created.
5. Capture the deployed versions and the deploy timestamps into the incident notes while you still have shell access.
6. Record the error rate for the week, so the next incident has something to compare against.