Notes from the terminal

Job queues without the drama

backendAugust 2, 2026 · 8 min read

At some point every web service has a request that takes too long. It sends an e-mail, resizes an image, calls a slow partner API. The usual answer is to stop doing the work inside the request and put it on a queue instead. It sounds like an afternoon of work. In practice it is the start of a long relationship, and it is worth going in with open eyes.

A client puts jobs on a queue; two workers take them off
The basic shape: the request puts a job on the queue and returns immediately.

Start with what you already run

The first temptation is to pick a dedicated message broker. Sometimes that is the right call. More often, the database you already operate can hold a table of pending jobs just fine, and one less moving part means one less thing to be paged about at night.

CREATE TABLE jobs (
  id         bigserial PRIMARY KEY,
  kind       text        NOT NULL,
  payload    jsonb       NOT NULL,
  attempts   int         NOT NULL DEFAULT 0,
  run_at     timestamptz NOT NULL DEFAULT now(),
  locked_at  timestamptz
);

-- A worker claims one job without blocking the others:
SELECT * FROM jobs
 WHERE run_at <= now() AND locked_at IS NULL
 ORDER BY run_at
 LIMIT 1
 FOR UPDATE SKIP LOCKED;

That single query, FOR UPDATE SKIP LOCKED, is the reason this approach works at all. Workers never wait for each other, and a job is claimed by exactly one of them.

Assume every job runs twice

Workers crash. Networks drop the acknowledgement after the work is already done. Sooner or later the same job will be delivered a second time, and the only question is whether that matters.

The cure is idempotency: design each job so that running it again produces the same result. Send the e-mail only if a record says it has not been sent. Write the file under a name derived from the job id. Check before you charge. This is the single most useful habit I know for background work, and it costs almost nothing when done from the start.

Retries need a ceiling and a delay

A job that fails should be tried again, but not immediately and not forever. A simple exponential delay with some random jitter handles most cases:

  • First retry after a few seconds, then double the delay each time.
  • Add a little randomness so a thousand failed jobs do not all return at once.
  • After a fixed number of attempts, move the job to a separate place for a human to look at.

That last point matters. A dead-letter table that someone actually reads is worth more than any amount of clever automatic recovery.

Watch the queue, not just the workers

Workers being up tells you very little. What you want to know is how old the oldest waiting job is. If that number grows, something is wrong, even if every process looks healthy. I put that single metric on a graph and alert when it crosses a threshold. Nothing else has caught as many real problems.

A queue hides slowness. It does not remove it. Somebody still has to do the work.

When the boring option stops being enough

There are real reasons to move to a dedicated broker: very high throughput, fan-out to many consumers, or strict ordering across machines. If you have those problems, you will know. Until then, a table, an index and a well-behaved worker loop will carry you much further than people expect.

← Back to all posts