Webhook Retries, and Why They Must Not Create a Second Order

A courier's tracking system sends a delivery event at 02:14. Your receiver writes an order note, emails the customer and moves the order to completed. At 02:19 the same event arrives again. Then again at 02:34, and once more after breakfast. By nine o'clock the customer has four "your parcel has been delivered" emails and the order notes read like a stuck record.

Nothing is broken in the usual sense. Every request was valid, the signature checked out, and the handler did exactly what it was told. The receiver simply had no concept of an event it had already seen, and the sender had no way of learning that the first delivery worked.

Webhooks are push notifications sent over a channel that cannot confirm delivery, so senders that take reliability seriously retry. The retry is not a bug in their system or a symptom of yours. It is the mechanism. The only thing you control is what your receiver does on the fifth copy.

Every correct sender retries, so at-least-once is the contract

A sender fires a webhook and waits for a response. If the connection fails, or the response takes longer than its timeout, or the status is a 5xx, the sender cannot tell whether your handler ran. A dropped connection looks exactly like a handler that crashed after committing its work. Faced with that ambiguity, a sender that wants reliable delivery does what a well-written client does: it sends the event again.

The schedule usually follows a backoff. The first retry comes seconds after the original, then the gaps widen into minutes, then into hours, with attempts thinning out until the sender gives up somewhere between twelve and seventy-two hours later. Some senders are more patient than that. The shape of the curve is what matters, because it tells you the size of the window you have to survive: a duplicate can arrive a day after the original, from a process that ran for two seconds.

Some implementations add a header describing the attempt number or the next scheduled try. Treat that as diagnostic information and nothing more. Your receiver has to be correct whether the header is present, absent or wrong, because the guarantee on offer is at-least-once delivery.

Acknowledge in milliseconds, then do the work somewhere else

The fastest way to turn one delivery into three is to process the event inline and take longer than the sender's timeout. The sender sees a timeout and retries while your handler is still busy with the first copy.

A webhook receiver therefore does two things on the request path: verify the request, and write a row that says the event arrived. Everything else, the emails, the order notes, the calls to the warehouse, happens after the response has been sent. I would rather answer in a few milliseconds and do the work a minute later than keep a PHP worker busy while the sender counts down its timeout.

In WordPress the handoff is usually wp_schedule_single_event() with a near-future timestamp. It returns immediately and lets the HTTP response close. Be honest about what that is: WP-Cron is triggered by incoming requests, so the job only runs when something else hits the site. On a quiet store at 02:00 with no traffic, "near future" can mean twenty minutes. If the work has to start promptly, hand off to a real queue or a system cron worker and treat WP-Cron as the fallback.

One ordering detail is not arbitrary. The dedupe row is written before the response and before the background job is scheduled. If you schedule first and record later, two retries arriving a second apart can both queue the same work, which puts you back where you started with an extra table to clean up.

Verify the signature over the raw body

An unsigned webhook endpoint is an open write API. Anyone who learns the URL can mark orders as delivered, so the sender signs the payload with a shared secret and sends the digest in a header. You compute the same digest over the bytes you received and compare.

The part people get wrong is which bytes. The signature covers the exact body as sent, so you have to hash the raw request body before anything parses, normalises or re-encodes it. Decode the JSON, re-encode it with wp_json_encode() and hash that, and the digests will not match: key order, whitespace and unicode escaping all change the bytes.

$_POST is not a substitute. PHP fills $_POST for form-encoded bodies, and for a JSON body it stays empty, which is the trap. Reading file_get_contents( 'php://input' ) gives you the bytes the sender signed. Inside a REST callback, $request->get_body() is the same bytes after WordPress has captured them, and it is the cleaner option there.

Compare with hash_equals() rather than ==. Ordinary string comparison returns as soon as two bytes differ, which leaks the length of the matching prefix to anyone who can time the responses. hash_equals() takes the same time for every input.

A timestamp window is what makes a replay useless

A valid signature proves the payload came from the sender. It does not prove the payload is new. Anyone who captures one signed request can post the same bytes again, forever, and the signature will verify every time because nothing about the bytes has changed.

So the sender includes a timestamp, in the body or in a header, and you reject anything outside a tolerance. Five minutes is a common window and it is wide enough for ordinary clock skew. Widen it only if you have measured the skew, because every extra minute is another minute in which a captured request is still usable.

Check your own clock before you widen anything. A receiver whose time has drifted rejects good events, and that failure looks like a sender problem, which is how an afternoon disappears. Log the delta you computed for every rejected request. That number tells you immediately whether the sender is late or your server is.

The timestamp window and event-ID dedupe overlap without replacing each other. The window limits how long a captured request stays useful. Dedupe limits how many times a valid event can be acted on, and it has no time limit at all.

Dedupe on the event ID, and let the index decide

Most senders give every event an identifier and send that identifier on every retry. Put it in a table with a unique index and insert before you process. The insert either succeeds, which means this is the first time you have seen the event, or it fails with a duplicate-key error, which means you already have it.

CREATE TABLE wp_webhook_events (
  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  event_id VARCHAR(191) NOT NULL,
  topic VARCHAR(100) NOT NULL DEFAULT '',
  received_at DATETIME NOT NULL,
  processed_at DATETIME NULL,
  PRIMARY KEY  (id),
  UNIQUE KEY event_id (event_id),
  KEY received_at (received_at)
);

On the schema, the 191-character limit on event_id comes from the days when a utf8mb4 index could only be 767 bytes long. Modern MySQL with the default row format allows more, and 191 is still a safe habit. The received_at index is what makes the purge job cheap instead of a full scan.

Return 200 for the duplicate, not an error. A 409 tells the sender something went wrong, and some senders respond by retrying harder or by disabling the endpoint after enough failures. From the sender's side the event was delivered. That you had already seen it is your business.

add_action( 'rest_api_init', function () {
    register_rest_route(
        'wps/v1',
        '/courier-events',
        array(
            'methods'             => 'POST',
            'callback'            => 'wps_handle_courier_event',
            // A cookie or nonce is not the authentication here; the HMAC is.
            // WordPress 5.5 made this argument mandatory for every route.
            'permission_callback' => '__return_true',
        )
    );
} );

function wps_handle_courier_event( WP_REST_Request $request ) {
    global $wpdb;

    $secret = defined( 'WPS_COURIER_SECRET' ) ? WPS_COURIER_SECRET : '';
    $raw    = $request->get_body();
    $sent   = preg_replace( '/^sha256=/', '', (string) $request->get_header( 'x-wps-signature' ) );
    $stamp  = (int) $request->get_header( 'x-wps-timestamp' );

    if ( '' === $secret || '' === $sent ) {
        return new WP_Error( 'wps_unsigned', 'Missing signature.', array( 'status' => 401 ) );
    }

    if ( abs( time() - $stamp ) > 300 ) {
        return new WP_Error( 'wps_stale', 'Timestamp outside the tolerance window.', array( 'status' => 401 ) );
    }

    // Sign the timestamp and the exact bytes received, in that order.
    $expected = hash_hmac( 'sha256', $stamp . '.' . $raw, $secret );

    if ( ! hash_equals( $expected, $sent ) ) {
        return new WP_Error( 'wps_bad_signature', 'Signature mismatch.', array( 'status' => 401 ) );
    }

    $event = json_decode( $raw, true );

    if ( ! is_array( $event ) || empty( $event['id'] ) ) {
        return new WP_Error( 'wps_bad_event', 'Event carries no id.', array( 'status' => 400 ) );
    }

    $table = $wpdb->prefix . 'webhook_events';

    $wpdb->suppress_errors( true );
    $stored = $wpdb->insert(
        $table,
        array(
            'event_id'    => (string) $event['id'],
            'topic'       => isset( $event['type'] ) ? (string) $event['type'] : '',
            'received_at' => current_time( 'mysql', true ),
        ),
        array( '%s', '%s', '%s' )
    );
    $wpdb->suppress_errors( false );

    if ( ! $stored ) {
        // Already seen. Acknowledge so the sender stops retrying.
        return new WP_REST_Response( array( 'status' => 'duplicate' ), 200 );
    }

    wp_schedule_single_event( time() + 5, 'wps_process_courier_event', array( (string) $event['id'] ) );

    return new WP_REST_Response( array( 'status' => 'accepted' ), 200 );
}

The handler answers 200 rather than 202 in both branches. Some senders treat only 200 as success, and a receiver that gets its event accepted has no reason to care which 2xx it received.

Where event-ID dedupe stops being enough

Dedupe by event ID rests on one assumption: the sender reuses the same ID for every retry of the same event. Most senders do. Some generate a new ID per attempt, and some omit IDs entirely, and against either of those a unique index never fires and every retry creates work.

Verify it before you rely on it. Their documentation is the first place to look, and a test is the second: force your endpoint to fail once, capture two deliveries of the same logical event, and compare the IDs. Ten minutes of work, and it is the difference between a design and a guess.

When the IDs are not usable, fall back to a natural key derived from the payload, such as order number, new status and the timestamp the event occurred, hashed into one value. It is a heuristic. Two genuinely distinct events that share those fields will collapse into one, and a sender that omits the timestamp from the payload leaves you with less to work from. Log every time the fallback runs, because a fallback that fires constantly is a signal that the design needs revisiting.

The handler also has to be idempotent in its own right, for a reason that has nothing to do with the sender. The background job that reads the dedupe row can run twice if the queue redelivers, so "set the order status to delivered" is safe to repeat and "append a delivery note" is not. Write the job as though it will run twice, because eventually it will.

The dedupe table also grows forever unless you purge it, and the purge window has to be at least as long as the sender's longest retry. On a store receiving 20,000 courier events a day, a ninety-day window is under two million rows, which any MySQL instance will carry, so there is little reason to be aggressive here.

What to do on Monday

1. Log every webhook delivery for a week, including the event ID and the timestamp. Two days of that log tells you whether the sender's IDs repeat.
2. Force a failure on one delivery and watch what arrives next. Note the gap between attempts.
3. Create the events table with the unique index and confirm it with SHOW INDEX FROM wp_webhook_events.
4. Move every side effect out of the request path. If the handler sends mail, it is doing too much.
5. Add the timestamp check and log the computed delta for a week before you enforce a narrow window.
6. Write the purge job, and alert when a duplicate arrives, because a rising duplicate rate means a sender is failing more often than you think.