Queue (Redis)

Note

Not part of core. Install it separately:

composer require kinetis/queue-redis

Adds Redis as a backend for Queue. Switching to it changes configuration, not application code.

QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
QUEUE_VISIBILITY_TIMEOUT_SECONDS=300

vendor/bin/kinetis queue:work --queue=high,default

Redis transport and Cluster

kinetis/queue-redis sends every command through kinetis/redis (Redis), which Composer installs with it: a transport that never re-sends a command whose reply it did not receive.

kinetis/redis supports Redis Cluster for application use: its ClusterClient routes commands by slot (see Redis Cluster), and kinetis/cache-redis uses it under REDIS_CLUSTER=true (see Appendix: Satellite Packages). This queue backend’s supported connection is a single Redis node. Its scripts name several keys with no shared hash tag, so it reads its connection’s REDIS_CLUSTER and rejects true with an InvalidArgumentException naming that key, before any client is built. It requires REDIS_URL or REDIS_HOST and follows no cluster redirect, so point it at a standalone Redis server, not at a cluster. The check reads only the queue connection’s own key: an application whose cache uses a cluster gives the queue its own server through a named connection, whose REDIS_JOBS_CLUSTER is unset and so false:

REDIS_CLUSTER=true
REDIS_CLUSTER_SEEDS=10.0.0.1:6379,10.0.0.2:6379,10.0.0.3:6379

QUEUE_CONNECTION_NAME=jobs
QUEUE_JOBS_CONNECTION=redis
REDIS_JOBS_HOST=queue-redis.internal

Configuring

Besides QUEUE_VISIBILITY_TIMEOUT_SECONDS, the backend reads the REDIS_* keys the cache reads, scoped by the queue connection’s name (see Queue’s “Named connections”): REDIS_URL, or REDIS_HOST with REDIS_PORT and REDIS_DATABASE; REDIS_PASSWORD; REDIS_TIMEOUT; and REDIS_TLS, REDIS_TLS_VERIFY_PEER and REDIS_TLS_CA_FILE. Configuration lists their defaults. The queue opens its own connection rather than sharing the cache’s, so REDIS_TIMEOUT is its own per-command budget.

Lease timeout

QUEUE_VISIBILITY_TIMEOUT_SECONDS (default 300, at least 1) is how long a popped job stays leased to its worker. When a worker dies before settling a job, any worker’s next pop() on that queue takes the job back once the lease expires, with its attempt count increased. No separate reaper process runs.

queue:work renews the lease automatically while the job runs, at half this window, so the setting sizes how long a crashed worker’s job waits to come back rather than how long a job may take. A handler that never yields to the event loop cannot be renewed, and neither can one whose worker has died — delivery stays at-least-once either way. Redis mechanisms describes the lease algorithm and Reservation renewal what the worker does with it.

When a Redis command fails

A command whose reply never arrives raises Kinetis\Redis\Exception\OutcomeUnknown, and the transport never sends it again. From push(), the job may be queued. From pop(), a job may stay leased to no worker until its lease expires. From ack(), release() or fail(), the settlement may or may not have happened and the job may run again. Kinetis\Redis\Exception\ConnectionFailed means the command never reached Redis. Raised inside queue:work, either exception stops the worker.

Connection lifetime

The queue opens a Kinetis\Redis\Client of its own rather than sharing the cache’s, and RedisQueueFactory::fromConfig() hands the queue that client’s close(). A queue the bootstrap binds, or one queue:work --connection=<name> builds, has its connection closed when the worker ends, with no wiring of yours; build the backend yourself and registering $app->onDispose($queue->dispose(...)) is yours too. A RedisQueue constructed around an Amp\Redis\RedisClient you built closes nothing — see Appendix: Queue Contracts’s “Connection ownership”.

Clearing a queue

RedisQueue declares ClearableQueueInterface (see Queue’s “Clearing is a separate capability”). Clearing removes a queue’s pending and delayed jobs and reports how many; jobs leased to a running worker are untouched. queue:stats counts pending jobs, delayed jobs and expired leases.

Delays and retries

A delayed job becomes available on the first pop() sweep after its delay elapses, so it runs late while every worker is busy. The delay is measured with the clocks of the pushing and popping hosts, so keep them synchronized; lease expiry uses the Redis server’s clock. Retries follow Queue: maxAttempts, QUEUE_MAX_ATTEMPTS and QUEUE_RETRY_BASE_DELAY_SECONDS. A delayed retry goes into the same delayed sorted set a delayed push uses, written inside the one fenced script that removes the lease.

See also

  • Queue — jobs, workers, retries and delivery guarantees.

  • Appendix: Queue Contracts — the lease algorithm and delivery contracts.

  • Redis — the transport, Redis Cluster, and what a failed command’s outcome means.

  • Appendix: Satellite Packages — kinetis/cache-redis, the cluster-capable cache that reads the same REDIS_* keys.

  • Configuration — named connections and every REDIS_* key.