Background Jobs in Node.js: Reliable Task Queues with BullMQ and Redis

Every web app eventually hits the same wall: a request needs to do something slow — send an email, resize an image, call a flaky third‑party API, generate a PDF — and the user is left staring at a spinner while your event loop is tied up. Do enough of that under load and your p99 latency falls apart, one timeout cascades into another, and a single downstream outage takes your whole API with it.

The fix is old and boring in the best way: get the slow work off the request path. Accept the request, hand the work to a durable queue, return 202 Accepted, and let a pool of background workers churn through it — with retries, backoff, and rate limits handled for you.

In the Node.js world the pragmatic default for this is BullMQ, a Redis‑backed queue that's fast, battle‑tested, and actively maintained (this guide targets BullMQ 5.x as of 2026). If you're already running Redis for caching, you have everything you need.

When you actually need a queue

Reach for a background queue when work is any of:

• Slow — image/video processing, PDF generation, data exports, ML inference.
• Unreliable — anything that calls an external API you don't control (payment providers, email, webhooks) and therefore needs retries.
• Bursty — sign‑up spikes, batch imports, fan‑out notifications that would otherwise hammer a downstream service.
• Scheduled — nightly reports, cache warmers, cleanup jobs, reminder emails.
• Fan‑out — one event that triggers many independent side effects (this pairs naturally with an event‑driven architecture).

If the work is fast, deterministic, and must be visible to the user immediately, keep it inline. A queue is not free — it adds Redis, a second process to deploy, and a new set of failure modes to reason about. Add it when the request path is the wrong place for the work, not before.

The mental model

BullMQ has three moving parts:

1. Queue — the producer‑side handle. Your web process calls queue.add() to enqueue jobs.
2. Redis — the durable store and coordination layer. Jobs live here until they're processed.
3. Worker — a separate process that pulls jobs off the queue and runs your processor function.

The key architectural point: the Queue and the Worker usually run in different processes. Your API deploys and scales independently from your workers. That separation is the whole point — a traffic spike on the API doesn't starve your background processing, and a slow job doesn't block an HTTP handler.

Setup

Install the library and make sure you have a Redis instance reachable:

npm install bullmq

BullMQ connects to Redis via ioredis. You can pass a plain connection object, but there is one non‑obvious rule that trips up almost everyone the first time:

// connection.js
export const connection = {
  host: process.env.REDIS_HOST ?? '127.0.0.1',
  port: Number(process.env.REDIS_PORT ?? 6379),
  // Required for Workers. If you pass your own ioredis instance,
  // set this to null or BullMQ's blocking commands can throw and
  // break the worker. BullMQ applies this default for you when it
  // creates the connection, but be explicit if you manage it yourself.
  maxRetriesPerRequest: null,
};

If you hand BullMQ a shared ioredis client with a non‑null maxRetriesPerRequest, workers will misbehave under transient Redis blips. Let BullMQ own the worker's blocking connection, and reuse a separate client for producers.

Producing jobs

The Queue object is cheap to construct and safe to keep around for the lifetime of your app. Create it once and enqueue from your request handlers:

// queue.js
import { Queue } from 'bullmq';
import { connection } from './connection.js';

export const emailQueue = new Queue('email', {
  connection,
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 1000 },
    // Keep Redis from growing unbounded: retain a small window
    // of completed/failed jobs for debugging, drop the rest.
    removeOnComplete: { count: 1000 },
    removeOnFail: { count: 5000 },
  },
});

Then, from an HTTP handler, enqueue and return immediately:

// routes/signup.js
import { emailQueue } from '../queue.js';

app.post('/signup', async (req, res) => {
  const user = await createUser(req.body);

  // Hand the slow part to the queue and respond right away.
  await emailQueue.add(
    'welcome-email',
    { userId: user.id, email: user.email },
    { jobId: `welcome:${user.id}` }, // idempotency: dedupes retries
  );

  res.status(202).json({ id: user.id });
});

Two things worth calling out:

• Keep the payload small. Store an ID, not the whole object. The worker should re‑fetch fresh data from your database — the record may have changed between enqueue and execution.
• Use a deterministic jobId for idempotency. If the same logical event can be enqueued twice (a retried request, an at‑least‑once webhook), a stable jobId means BullMQ ignores the duplicate instead of sending two welcome emails.

Consuming jobs

A Worker is a long‑lived process. Give it the queue name and a processor function:

// worker.js
import { Worker } from 'bullmq';
import { connection } from './connection.js';
import { sendWelcomeEmail } from './email.js';

const worker = new Worker(
  'email',
  async (job) => {
    switch (job.name) {
      case 'welcome-email':
        return sendWelcomeEmail(job.data.userId);
      default:
        throw new Error(`Unknown job: ${job.name}`);
    }
  },
  {
    connection,
    concurrency: 10, // process up to 10 jobs in parallel per worker
  },
);

worker.on('completed', (job) => {
  console.log(`job ${job.id} done`);
});

worker.on('failed', (job, err) => {
  console.error(`job ${job?.id} failed: ${err.message}`);
});

Tuning concurrency

concurrency controls how many jobs a single worker runs at once. A sane starting point for I/O‑bound work is somewhere around half your CPU count, then measure and adjust:

import os from 'node:os';
const concurrency = Math.max(1, Math.floor(os.cpus().length / 2));

The one rule you must not break: never do synchronous, CPU‑heavy work inside a processor. A tight for loop crunching numbers blocks the event loop and stalls every concurrent job on that worker (the same event‑loop discipline covered in Node.js Performance Optimization). Offload real CPU work to BullMQ's sandboxed processors (a separate file run in a child process) or a dedicated service.

Need more throughput? Run more worker processes — horizontally, across machines or pods. BullMQ coordinates through Redis, so ten worker replicas each with concurrency: 10 gives you ~100 jobs in flight with no extra code.

Retries and backoff

This is the feature that justifies the whole exercise. Configure attempts and a backoff strategy, and BullMQ automatically re‑queues failed jobs with increasing delay:

await paymentQueue.add(
  'charge',
  { orderId },
  {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 },
  },
);

With attempts: 5 and exponential backoff seeded at 2s, a job that keeps failing is retried after roughly 2s, 4s, 8s, and 16s before it's finally marked failed. attempts counts the total tries, including the first — so attempts: 5 means one initial run plus four retries.

Exponential backoff matters because the thing you're retrying is usually a struggling downstream service. Retrying immediately, in a tight loop, is how you turn a brief hiccup into a self‑inflicted outage. Back off and give it room to recover.

For failures you know are permanent (a validation error, a 4xx from an API), don't waste retries — throw an UnrecoverableError to fail the job immediately:

import { UnrecoverableError } from 'bullmq';

if (response.status === 422) {
  throw new UnrecoverableError('Invalid payload, retrying will not help');
}

Jobs that exhaust all attempts land in the failed set, where they act as a dead‑letter queue — you can inspect them, alert on them, and re‑enqueue after a fix.

Rate limiting

When your worker fans out to an API with a quota (say, 100 requests/second), let BullMQ throttle for you instead of getting yourself rate‑limited or banned. Set a limiter on the worker:

const worker = new Worker('outbound-webhooks', processor, {
  connection,
  limiter: {
    max: 100,      // at most 100 jobs
    duration: 1000, // per 1000 ms
  },
});

While the limit is saturated, idle workers simply stop fetching new jobs until the window rolls over — no busy‑waiting, no thundering herd. This is the clean, centralized alternative to sprinkling setTimeout calls through your code, and it complements request‑side API rate limiting on the way in.

Scheduled and recurring jobs

For cron‑style work — nightly reports, hourly cache warmers, reminder sweeps — use Job Schedulers. Note that the older repeat option on add() was deprecated in BullMQ 5.16 in favor of upsertJobScheduler, which is idempotent: calling it again with the same scheduler ID updates the schedule instead of creating a duplicate.

// Run every day at 03:00, producing a "daily-report" job each time.
await reportQueue.upsertJobScheduler(
  'daily-report',                       // stable scheduler id
  { pattern: '0 3 * * *' },             // cron expression
  { name: 'daily-report', data: { kind: 'summary' } },
);

Because it's idempotent, it's safe to run this on every boot — you won't accumulate a new schedule each deploy, which was a classic footgun with the old repeatable API.

Flows: jobs that depend on other jobs

Some work is a graph, not a list: process a batch of images, and only when all of them finish, stitch them into a single PDF. BullMQ's FlowProducer models these parent/child dependencies. The parent job won't run until every child completes:

import { FlowProducer } from 'bullmq';

const flow = new FlowProducer({ connection });

await flow.add({
  name: 'build-report',
  queueName: 'reports',
  children: [
    { name: 'render-page', queueName: 'render', data: { page: 1 } },
    { name: 'render-page', queueName: 'render', data: { page: 2 } },
    { name: 'render-page', queueName: 'render', data: { page: 3 } },
  ],
});

The parent processor can read its children's return values via job.getChildrenValues(), giving you a clean fan‑out/fan‑in without hand‑rolling coordination in Redis.

Graceful shutdown — the part people forget

Here's the failure mode that bites teams in production: you deploy, Kubernetes sends SIGTERM, your worker process exits instantly — and every job that was mid‑flight is abandoned. Depending on your settings it either silently vanishes or gets reprocessed from scratch, sometimes double‑charging a customer.

The fix is to handle the shutdown signal and call worker.close(), which is graceful by design: it stops fetching new jobs and waits for the currently active ones to finish before closing connections.

async function shutdown() {
  console.log('draining worker...');
  await worker.close(); // stops new jobs, waits for active ones
  process.exit(0);
}

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

Then give the platform enough time to let jobs drain — set your orchestrator's terminationGracePeriodSeconds (or equivalent) to comfortably exceed your longest expected job. Long‑running jobs should also periodically call job.updateProgress() and check job.isActive() so a stalled worker can be detected and the job re‑queued rather than lost. This single pattern is the difference between a queue that's reliable and one that quietly drops work on every deploy.

Observability

A queue you can't see into is a queue you'll eventually distrust. BullMQ exposes rich events (completed, failed, stalled, progress) and, in recent 5.x releases, first‑class OpenTelemetry instrumentation — so job spans slot straight into the traces and dashboards from your monitoring and observability setup. At minimum, track:

• Queue depth (waiting + delayed) — the leading indicator that workers can't keep up.
• Failure rate and the size of the failed set — your dead‑letter early warning.
• Job duration (p50/p99) — regressions here often mean a downstream is degrading.

A dashboard like Bull Board or Taskforce gives you a live view of waiting, active, completed, and failed jobs, plus one‑click retries — invaluable when something goes sideways at 2am.

Production checklist

Before you ship a BullMQ setup, confirm:

• Workers run as a separate process/deployment from your API.
• Redis connection uses maxRetriesPerRequest: null for workers.
• Every job type sets sensible attempts and backoff.
• removeOnComplete / removeOnFail are configured so Redis memory stays bounded.
• Job payloads are small (IDs, not blobs); workers re‑fetch fresh data.
• Idempotent handlers (stable jobId or dedupe logic) — assume at‑least‑once delivery.
• SIGTERM / SIGINT handled with worker.close() and a matching grace period.
• Permanent failures throw UnrecoverableError instead of burning retries.
• Failed set is monitored and alertable; a dashboard is wired up.

Wrapping up

Background jobs are one of those patterns that feel like overkill right up until the moment they save you — the first time a payment provider has a bad ten minutes and your queue quietly retries every charge instead of dropping them, you'll be glad it's there. BullMQ gives you durable queues, retries with backoff, rate limiting, scheduling, and flows on top of the Redis you're probably already running.

Start small: pick the single slowest or flakiest thing in your request path — usually an email or a third‑party call — and move just that behind a queue. Get graceful shutdown and idempotency right from day one, and you'll have a foundation you can keep leaning on as your system grows.