Kinetis Queue Failed Jobs 1.0: Keep the Jobs Your Worker Gives Up On#

When a job fails on its last attempt, the queue worker logs it and deletes it. Until now that was the end of it: if the failure came from a bug you have since fixed, or from an API that was down for an hour, the job was gone. kinetis/queue-failed-jobs 1.0.0 keeps those jobs in a database table, so you can look at them and run them again.

Getting started#

Install the package:

composer require kinetis/queue-failed-jobs

Copy the migration stub for your database from vendor/kinetis/queue-failed-jobs/resources/migrations/ into your migrations/ directory, then run vendor/bin/kinetis migrate. That’s all. Once the package is installed, the worker records failed jobs. There is no setting to turn it on.

It works with every queue backend: Redis, SQL, SQS and RabbitMQ. The records go to MySQL or Postgres through kinetis/database-bridge, whatever backend the queue itself uses. To keep them on a database other than your default one, set FAILED_JOBS_DB_CONNECTION_NAME.

Four commands#

vendor/bin/kinetis queue:failed --queue=mail
vendor/bin/kinetis queue:retry 3f9c0b2a64d14e8f9a0c51d7e2b46a18
vendor/bin/kinetis queue:retry --all
vendor/bin/kinetis queue:forget 3f9c0b2a64d14e8f9a0c51d7e2b46a18
vendor/bin/kinetis queue:prune --hours=168

  • queue:failed lists the records, oldest first.

  • queue:retry pushes a job back onto the queue it failed on, then deletes its record. If the push fails, the record stays and the command exits 1.

  • queue:forget deletes a record without retrying the job.

  • queue:prune deletes old records. Nothing is pruned automatically.

Never lost, sometimes twice#

The worker writes the record before it asks the queue backend to delete the job. If the write fails, the worker stops and the backend keeps the job, so a job you gave up on is never lost.

The flip side is that a crash at the wrong moment can leave a job both recorded and still on the queue, and the next delivery can record it again. This is the same at-least-once guarantee the queue already gives, so a handler that can safely run twice can also be safely retried.

Things to know#

  • Arguments are stored in full. A retry rebuilds the job from its arguments, so the table holds #[Sensitive] values too. Protect it the way you protect the queue’s own storage. queue:failed prints sensitive values as [redacted].

  • The worker depends on the failed-jobs database. If that database refuses a write, queue:work stops, and the job stays on its queue.

  • Some failures are never recorded. A job that crashes its worker process never reaches the failure path. On SQS, a redrive policy covers that case.

The failed jobs guide covers the details.

Upgrading#

Install kinetis/queue-failed-jobs 1.0.0. It requires kinetis/queue 1.8.0, the release that adds the hook the worker records through. JobFailedPermanently now carries the record’s ID as failedJobId.

Kinetis RoadRunner Adapter 1.3: New RoadRunner Package Names   Kinetis Vector 0.14: A Queue Dashboard in the Admin