From WP-Cron to a Job Queue With Claims, Retries and Dead Letters

A store's nightly reconciliation ran twice on a Saturday and wrote the same order adjustments again. Nothing raised an error. The job code was careful, the schedule was correct on paper, and the numbers were wrong in a way nobody noticed until the invoices were reconciled a week later. The hook had been registered with wp_schedule_event(), which means it ran when a visitor arrived rather than when the clock said so.

Two problems produced that, and they need different fixes. WP-Cron only decides what is due while a request is shutting down, so the schedule bends toward your traffic pattern. And the work itself had no way to tell a second run from a first one. Moving the trigger to system cron solves the first. The second is a design question about the job, and no scheduler will answer it for you.

WP-Cron is a traffic meter, not a clock

Every scheduled event lives in one autoloaded option named cron, a serialised array keyed by timestamp. Once a request has produced its response, WordPress walks that array and runs whatever has come due. No request means no walk, so a job scheduled for 03:00 on a site with no night traffic runs at the next visit.

Two details about that array cause most of the confusion. The keys are GMT, so a timestamp built from local time in a non-UTC timezone lands at the wrong hour. And because a plugin can register a recurrence with any interval it likes, nothing stops the array from holding four due timestamps at once, all of which run back to back in one PHP process when a trigger arrives.

// The trap: strtotime() builds a local timestamp, and the cron array is GMT.
wp_schedule_event( strtotime( 'tomorrow 03:00' ), 'daily', 'my_store_reconcile' );

That call is also the reason a job that must run at a specific moment cannot be trusted to cron in any form, WordPress or system. Both are best-effort triggers with no memory of a missed run, and neither will tell you a run was skipped.

DISABLE_WP_CRON moves the trigger and leaves every other problem in place

The standard first step is to stop letting visitors trigger the schedule at all. Define the constant in wp-config.php above the line that requires wp-settings.php, then call the trigger from the system scheduler instead:

*/2 * * * * /usr/local/bin/php /var/www/example.com/wp-cron.php >/dev/null 2>&1

The interval is a promise you make to every plugin on the site, and it has to be shorter than the shortest scheduled task. A plugin that schedules work every five minutes behind a fifteen minute trigger produces three due timestamps that all run in one request, which is the burst you were trying to avoid. Two minutes is a defensible default for a small store.

What the constant does not change is the shape of a run. One PHP process still executes everything that is due, in order, inside one execution time limit. That leaves three failures you will meet within a month of moving to system cron.

• A long task blocks the next run. While the runner holds the cron lock, the next trigger finds the lock and returns, so a six minute sync silently skips two minutes of triggers.
• Overlapping runs duplicate work. If the lock is released because the process died rather than because it finished, the next trigger picks the same event up again and runs it a second time.
• A fatal error in one task ends the run. Everything queued behind the failure never executes, and the exit code is the only evidence, which is exactly the evidence nobody reads.

These are properties of running a schedule inside a web request, not bugs in WP-Cron. The answer is to stop putting the work in the schedule and put it in a queue that a system cron trigger drains.

A job queue is a table with a status column

The states that matter are pending, running, complete and dead, and the columns are the ones a worker needs in order to decide what to do next.

CREATE TABLE wp_wps_jobs (
  id            BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  hook          VARCHAR(191)    NOT NULL,
  args          LONGTEXT        NULL,
  status        VARCHAR(20)     NOT NULL DEFAULT 'pending',
  attempts      SMALLINT UNSIGNED NOT NULL DEFAULT 0,
  max_attempts  SMALLINT UNSIGNED NOT NULL DEFAULT 5,
  available_at  DATETIME        NOT NULL,
  claim_id      CHAR(32)        NULL,
  claimed_at    DATETIME        NULL,
  last_error    TEXT            NULL,
  created_at    DATETIME        NOT NULL,
  PRIMARY KEY (id),
  KEY status_available ( status, available_at ),
  KEY claim_id ( claim_id )
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Three of those columns carry the design. status is why a job can be counted and re-queued by hand instead of decoded out of a serialised array. available_at is both the schedule and the retry delay, which is what makes backoff easy. claim_id is how a worker proves the row it holds is still the row it claimed.

Store identifiers in args, never values. A job that spends forty minutes in a retry backoff and then trusts an order status it read when it was enqueued will act on a world that has moved. Pass the order ID and re-read it inside the job.

State retention belongs in the schema, not in an afterthought. Completed rows are useful for a week and then they are the new autoload bloat, so delete them on a schedule and keep dead rows until somebody has read last_error.

The claim is one atomic statement, or two workers take the same job

The naive worker selects a pending row, checks the status in PHP, then updates it to running. Two workers that read the same row between the check and the write both believe they own it. The fix is to let the database decide, with a single conditional UPDATE whose affected-row count is the answer.

// One worker tick. Run from system cron, one process per worker.
$claim = wp_generate_uuid4();

$ids = $wpdb->get_col( $wpdb->prepare(
    "SELECT id FROM {$wpdb->prefix}wps_jobs
     WHERE status = 'pending' AND available_at <= UTC_TIMESTAMP()
     ORDER BY available_at ASC
     LIMIT %d",
    25
) );

foreach ( $ids as $id ) {
    $claimed = $wpdb->query( $wpdb->prepare(
        "UPDATE {$wpdb->prefix}wps_jobs
         SET status = 'running', claim_id = %s, claimed_at = UTC_TIMESTAMP()
         WHERE id = %d AND status = 'pending'",
        $claim,
        $id
    ) );

    if ( 1 !== $claimed ) {
        continue; // Another worker got there first.
    }

    my_store_run_job( (int) $id );
}

The WHERE status = 'pending' clause is the whole mechanism. InnoDB takes a row lock for the duration of the UPDATE, so only one statement can match a pending row, and $wpdb->query() returns the number of affected rows: one means you own the job, zero means somebody else does. The outer SELECT is not atomic and does not need to be, because it only produces candidates.

Claims expire. A worker that is killed mid-job leaves a row in running that nothing will ever finish, so a second statement returns stale claims to the queue, and the job checks that it still holds its own claim_id before writing a result:

UPDATE wp_wps_jobs
SET status = 'pending', claim_id = NULL
WHERE status = 'running'
  AND claimed_at < UTC_TIMESTAMP() - INTERVAL 15 MINUTE;

Choose that timeout from the slowest realistic job, not from the fastest. Too short and a merely slow worker loses its claim while it is still working, which hands you the duplicate you were trying to prevent.

Retries need a count, a delay and a place to die

A failed job has to remember that it failed. The attempt count lives in the row, the delay is an exponential backoff with jitter, and the terminal state is dead, which means the queue stopped trying and a human needs to look.

function my_store_fail_job( $id, $error ) {
    global $wpdb;

    $job      = $wpdb->get_row( $wpdb->prepare(
        "SELECT attempts, max_attempts FROM {$wpdb->prefix}wps_jobs WHERE id = %d",
        $id
    ) );

    if ( ! $job ) {
        return; // The row is gone, so there is nothing to retry.
    }

    $attempts = (int) $job->attempts + 1;

    if ( $attempts >= (int) $job->max_attempts ) {
        $wpdb->update(
            "{$wpdb->prefix}wps_jobs",
            array( 'status' => 'dead', 'attempts' => $attempts, 'last_error' => $error ),
            array( 'id' => $id )
        );
        return;
    }

    // 1, 2, 4, 8 minutes, plus up to 30 seconds of jitter.
    $delay = ( 2 ** ( $attempts - 1 ) ) * MINUTE_IN_SECONDS + wp_rand( 0, 30 );

    $wpdb->update(
        "{$wpdb->prefix}wps_jobs",
        array(
            'status'       => 'pending',
            'attempts'     => $attempts,
            'available_at' => gmdate( 'Y-m-d H:i:s', time() + $delay ),
            'claim_id'     => null,
            'last_error'   => $error,
        ),
        array( 'id' => $id )
    );
}

Without a backoff, an upstream API that is down for ten minutes absorbs three hundred retries in that window and stays down for longer. The jitter matters more than it looks: a hundred jobs that fail together would otherwise retry on the same second, and the queue invents a spike out of an outage that has already passed.

The dead-letter state is the part teams skip, and skipping it is why queues get a reputation for losing work. A dead job is not a deleted job. It keeps its arguments and its last error, stops being selected, and produces one number worth alerting on: rows where status = 'dead' and created_at is older than an hour. When the upstream recovers, fix the cause, reset attempts to zero, and set the status back to pending.

Action Scheduler already implements this pattern

Most stores on WooCommerce already have a queue installed, because Action Scheduler ships with WooCommerce. It keeps actions in its own tables and gives each one a status, a scheduled time, an attempt count and a claim, with a log row per attempt. Arguments are stored as JSON rather than in an option, and failed actions sit in a state you can list. That is the pattern above, with the parts most people get wrong already handled.

Reach for it when the work is order-adjacent, when you want scheduling and queueing in one primitive, or when somebody else will be paged and needs a screen that shows what failed. The function names follow the WP-Cron ones with an as_ prefix, so the mental model carries over, and running its queue from system cron rather than from its own WP-Cron event is the difference between a queue and a schedule with extra steps.

Roll your own table when the constraints genuinely differ: when the worker has to run outside WordPress, when a non-PHP service consumes the same queue, or when you need claim and backoff semantics the library does not expose. Those tables are worth reading and not worth writing to. Query them for monitoring and leave the writes to the code that owns the state machine.

A queue will not make your work safe to run twice

This is the limit of everything above, and it survives every choice of library. A claim stops two workers from running the same row at the same moment. It does not stop the same logical job from being enqueued twice, and it does not stop a job from running twice when a claim expires because the worker was slow rather than dead. A queue delivers at least once, so design every job as though it will eventually run twice.

Idempotent shapes are not exotic. Write a destination value such as an order status, not a delta such as an incrementing counter. Insert with a unique key and treat the duplicate-key error as success. Check for a marker row before sending a notification, and write that marker in the same transaction that decides to send it. For anything that moves money, the idempotency key belongs at the payment provider, because your queue cannot know what happened across a network timeout.

What a queue does not fix is the work being wrong in the first place. It fixes the timing, the overlap and the silence, which were three of the four problems in the story at the top of this piece.

1. Define DISABLE_WP_CRON and add one system cron trigger. Run it for a week before changing anything else.
2. List every recurring hook with wp cron event list and its interval. Anything shorter than the trigger, or with a deadline, is queue work.
3. Write the retry policy before the job body: max attempts, backoff schedule, and one alert query for dead rows.
4. Make the job safe to run twice using a unique key or a destination write, not a flag you hope nobody resets.
5. If WooCommerce is installed, use Action Scheduler and alert on its failed actions instead of building a table.
6. Delete completed rows on a schedule. A queue that grows forever is the options table with a better name.