Running Celery in Production Without Losing Tasks

Celery is easy to start and easy to run badly. The default configuration is tuned for throughput on work you do not mind losing, which is the opposite of what most people actually queue: invoices, notifications, report generation, payment reconciliation.

A queue that silently drops one job in a thousand is worse than no queue, because you stop checking. Here is what to change, and why.

Default acknowledgement loses tasks

By default a worker acknowledges a message the moment it receives it, before running it. If that worker is killed mid-task — a deploy, an OOM kill, a machine reboot — the broker has already forgotten the message. The task is gone, with no error anywhere.

app.conf.update(
    # acknowledge after the task finishes, not when it is received
    task_acks_late=True,
    # if the worker dies mid-task, requeue it
    task_reject_on_worker_lost=True,
    # don't let one worker hoard the queue: with long tasks, a
    # prefetch of 4 means three jobs sit idle behind a slow one
    worker_prefetch_multiplier=1,
)

Those three settings turn Celery from at-most-once into at-least-once delivery. Which immediately creates the next problem.

At-least-once means your tasks must be idempotent

If a task can run twice, it will run twice. That is fine for "regenerate a cached report" and catastrophic for "charge the customer".

The fix is not to avoid retries. It is to make a second execution harmless, usually by writing a record that a duplicate can detect:

@shared_task(bind=True, max_retries=5)
def charge_invoice(self, invoice_id, idempotency_key):
    # get_or_create is atomic at the database level: the second
    # execution loses the race and exits instead of charging again
    attempt, created = ChargeAttempt.objects.get_or_create(
        key=idempotency_key,
        defaults={"invoice_id": invoice_id, "state": "pending"},
    )
    if not created and attempt.state in ("succeeded", "pending"):
        return attempt.id

    try:
        result = gateway.charge(invoice_id, idempotency_key=idempotency_key)
    except TransientGatewayError as exc:
        # exponential backoff with jitter — without jitter, a
        # gateway blip makes every retry arrive simultaneously
        raise self.retry(exc=exc, countdown=2 ** self.request.retries + random.uniform(0, 3))

    attempt.state = "succeeded"
    attempt.save(update_fields=["state"])
    return attempt.id

Pass the idempotency key through to the payment gateway too, if it supports one. Then even a duplicate that slips past your own check is deduplicated on their side.

Never pass objects, always pass ids

Task arguments are serialised, usually as JSON, and the worker deserialises them possibly seconds later on another machine.

# wrong: serialises a snapshot that is stale by the time it runs
send_invoice.delay(invoice)

# right: the worker fetches the current row
send_invoice.delay(invoice.id)

The failure this prevents is subtle. A stale object means a worker emails an invoice showing a total that was corrected two seconds after the task was queued, and nothing in the logs will look wrong.

Queue after commit, not inside the transaction

This one catches everybody exactly once:

with transaction.atomic():
    order = Order.objects.create(...)
    # The worker can pick this up before the transaction commits,
    # then fail with DoesNotExist on a row that is about to exist.
    # It only reproduces under load, which makes it maddening.
    process_order.delay(order.id)

with transaction.atomic():
    order = Order.objects.create(...)
    transaction.on_commit(lambda: process_order.delay(order.id))

The second version queues nothing if the transaction rolls back, which is also what you want.

Separate queues, or one slow job blocks everything

A single default queue means a twenty-minute report generation sits in front of the password-reset email a user is staring at a spinner for.

app.conf.task_routes = {
    "billing.tasks.generate_monthly_report": {"queue": "slow"},
    "billing.tasks.rebuild_search_index":    {"queue": "slow"},
    "notifications.tasks.*":                 {"queue": "fast"},
}

# two workers, sized for what they run
celery -A proj worker -Q fast --concurrency=8 -n fast@%h
celery -A proj worker -Q slow --concurrency=2 -n slow@%h

Split by latency expectation, not by application. Anything a human is waiting for goes in fast. Anything measured in minutes goes in slow.

Always set a time limit

A task with no timeout that hangs on a socket read occupies a worker slot forever. With prefetch_multiplier=1 and a concurrency of 2, three hung tasks mean the queue has silently stopped.

app.conf.update(
    task_time_limit=600,       # hard kill at 10 minutes
    task_soft_time_limit=540,  # SoftTimeLimitExceeded first, so you can clean up
)

Catch SoftTimeLimitExceeded in anything holding a lock or a partially written file, and release it there.

Beat schedules duplicate silently

Celery beat does not coordinate. Run two beat processes — easy to do accidentally during a rolling deploy, or by leaving one running on an old machine — and every periodic task fires twice.

Run exactly one beat process, and make periodic tasks defend themselves anyway with a short-lived lock:

@shared_task
def nightly_reconciliation():
    # NX + timeout: whoever sets it first runs; the lock expires
    # on its own if the worker dies holding it
    got_lock = cache.add("lock:nightly_reconciliation", "1", timeout=3600)
    if not got_lock:
        return "skipped: already running"
    try:
        reconcile()
    finally:
        cache.delete("lock:nightly_reconciliation")

Set the timeout longer than the task can plausibly take. A lock that expires mid-run is worse than no lock.

What to monitor

Two signals catch nearly everything:

• Queue depth over time. A queue that grows and never drains means your workers cannot keep up, or they are all hung. Alert on the trend, not the absolute number.
• Task failure rate by name. Retries make failures invisible in aggregate — a task failing four times and succeeding on the fifth looks fine from the outside and is telling you something is wrong.

And route dead tasks somewhere a person will see them. A task that exhausts its retries and vanishes into a log file is a bug report you will receive from a customer instead.