Background Jobs and Webhooks You Can Trust

Users click once; your system should do the right thing once—even when networks flap, workers crash, or Stripe sends the same event twice. Background jobs and webhooks are where “mostly works” becomes revenue incidents and support tickets.

This post covers queue semantics, retry policies, idempotent handlers, and verifying inbound webhooks. It builds on Postgres basics for job tables, error contracts for surfacing failures to humans, and API design for idempotency keys on write paths.

Separate interactive from asynchronous work

HTTP request handlers should finish quickly: validate, persist intent, enqueue work, respond. Heavy PDF generation, email bursts, and third-party sync belong in workers. Define a maximum handler time aligned with your load balancer timeouts; anything longer belongs in a job.

Choose a queue you can operate—Redis-backed (BullMQ, Sidekiq patterns), Postgres SKIP LOCKED job tables, or managed services. The tool matters less than consistent enqueue semantics and observability.

Job design: payload, attempts, and poison messages

Store job payloads as versioned JSON with ids referencing domain rows—avoid embedding stale snapshots when possible. Include attempt, maxAttempts, and runAt for scheduling. After max failures, move jobs to a dead-letter queue and alert—silent loss is worse than noisy alerts.

Workers should be idempotent: running the same job twice must not double-apply effects. Use unique constraints (processed_events(provider, event_id)) or compare-and-set on status columns.

Retries with backoff

Transient failures (network blips, 503 from partners) deserve exponential backoff with jitter. Permanent failures (400 validation from partner) should fail fast without endless retry. Classify errors in code—string matching on message text rots.

Stripe documents webhook best practices including quick 2xx responses and async processing—treat their guidance as a template for any payment or billing integration.

Webhook verification first, parsing second

Inbound webhooks must verify signatures before JSON parsing drives business logic. Use the provider’s documented HMAC or signing secret scheme; constant-time comparison for signatures. Reject replayed events with old timestamps when the spec provides tolerance windows. OWASP’s REST security guidance covers webhook authentication and replay risks in more depth.

Parse into typed structures after verification—Zod at the edges works well for internal event shapes once trust is established. Never trigger side effects on unverified bodies.

Outbound webhooks you send

When your SaaS notifies customers, sign payloads, document retry behavior, and offer a dashboard for delivery logs. Customers debug from your logs; opaque failure wastes everyone’s time. Version event types additively; consumers should ignore unknown type values.

Scheduling and cron

Cron jobs are jobs with calendars—still need idempotency when a deploy overlaps the tick. Use advisory locks or lease rows so only one worker runs nightly_reconcile per tenant shard. Clock skew matters; prefer UTC in schedulers.

Observability

Log jobId, jobType, orgId, duration, and outcome—not full PII payloads. Trace across enqueue and worker when using OpenTelemetry traces. Metrics: queue depth, age of oldest job, success rate by type. Alerts on DLQ growth beat paging on every single retry.

Database patterns for outbox

The transactional outbox pattern writes domain changes and an outbox row in one transaction; a relay publishes to the queue. That avoids “committed order but never enqueued email” races. Implementations vary; the invariant is atomicity with your business write.

Concurrency and fair queuing

Noisy tenants can flood queues. Per-tenant rate limits or fair scheduling prevent one customer from starving others. Bulk imports especially belong in throttled workers with progress APIs polled by the frontend—see caching and revalidation for how UIs refresh status.

Security

Workers use service credentials, not user cookies. Scope API keys to actions workers need. Webhook endpoints should authenticate inbound callers and rate-limit by IP or signature id to reduce abuse.

Replay attacks on webhooks are mitigated by timestamps in signed payloads and rejecting events older than a few minutes. Rotate signing secrets with dual-secret verification windows so providers can cut over without downtime. Never log raw signing secrets or full payment payloads in application logs.

Testing

Integration tests should enqueue real jobs against test Redis or transactional job tables. Simulate duplicate webhook delivery in tests; assert single side effect. Chaos-test worker restarts mid-job when stakes are high.

Graceful shutdown

Workers finishing in-flight jobs on deploy reduce duplicate processing when combined with visibility timeouts. Document how long shutdown waits; killing mid-transaction without rollback poisons data.

Exactly-once is a lie; aim for effectively-once

Distributed systems rarely guarantee exactly-once side effects. Your design goal is at-least-once delivery with idempotent consumers so business outcomes behave like exactly-once. Document that invariant for new engineers so they do not try to “fix” duplicate retries by disabling them without idempotency keys.

Priority queues and deadlines

Not all jobs are equal: password email can wait; fraud checks cannot. Separate queues or priority fields prevent bulk exports from blocking transactional email. runAt scheduling supports “remind user in three days” without cron scanning entire tables—index (run_at) WHERE status = 'pending'.

Partner SLAs and your outbound retries

When you call partner APIs from workers, respect their rate limits and idempotency headers. When partners call you, document your retry schedule so they do not assume infinite backoff. Misaligned expectations cause duplicate provisioning on both sides. Stripe’s idempotent requests doc is a practical template even outside payments.

Local development ergonomics

Run workers in dev with the same serialization format as production. Stub external webhooks with signed test fixtures checked into the repo. Developers who only test happy-path HTTP miss duplicate delivery scenarios entirely.

Compliance and audit trails

Billing and access-control jobs should append audit log rows in the same transaction as state changes when possible. Regulators and enterprise customers ask what happened when; job logs with redacted payloads answer without replaying secrets.

Time zones and scheduled jobs

Cron expressions in UTC avoid daylight-saving surprises. Document how “daily at 9am user local” is implemented—usually per-user timezone stored on profile plus scheduled jobs, not one global cron. Missed ticks during deploys should be safe to rerun idempotently.

Capacity planning

Queue depth trending up over days signals understaffed workers or slower downstream dependencies—not something to fix only by scaling workers without profiling job handlers. Profile database time per job type before throwing hardware at the problem.

Human operations runbooks

When DLQ volume spikes, runbooks should list which job types are safe to replay in bulk versus which need manual review (payments, provisioning). On-call engineers should not infer semantics from job class names alone—link each type to owner team and expected side effects.

Monitoring webhook endpoints

Track signature verification failures separately from application errors—a spike often means a misconfigured secret rotation or an attacker probing endpoints. Alert on sustained verification failure rates, not on every single bad request.

Versioning job payloads

When job shapes change, bump a payloadVersion field and support readers for old versions until the queue drains. Deploy workers that understand both schemas before enqueueing new shapes—same expand-contract playbook as HTTP APIs. Dropping version handling mid-queue causes silent drops or poison messages that are hard to replay safely. Keep a short compatibility matrix in your internal docs listing which worker versions accept which payload versions.

Closing

Jobs and webhooks you can trust are idempotent, verified, observable, and split from interactive latency. Treat duplicate delivery as normal, retries as a deliberate policy decision, and dead letters as product surface for support. Boring reliability here is a competitive advantage for SaaS billing, integrations, and customer trust.