Skip to content
All work

Systems / Backend

Reliable Webhook Processor

Event processing that survives duplicates, retries and worker death

Organisation
Personal project
Role
Design and implementation
Period
2026

Context

What the system is

Any integration that receives webhooks eventually hits duplicate delivery, partial processing and provider retries. The naive handler quietly corrupts state.

Webhooks arrive more than once, out of order, and sometimes while your worker is dying. This project handles those cases explicitly rather than hoping they do not happen.

Problem

What needed solving

Processing an event twice must not produce two effects, and a worker crashing mid-job must not lose the event or leave it locked forever.

  • Providers guarantee at-least-once delivery, never exactly-once
  • Acknowledgement must be fast; the work must not happen in the request
  • Failures must be visible, not swallowed

Architecture

How it's structured

  • 01

    Receive endpoint persists the raw event with a provider-supplied idempotency key, then acknowledges immediately

  • 02

    PostgreSQL holds the event ledger and its processing state — no in-memory queue as the source of truth

  • 03

    Background workers claim events atomically, so a crashed worker's claim expires and the event is reprocessed

  • 04

    Retries use exponential backoff with an explicit attempt ceiling and a dead-letter state

Approach

What I worked on

Idempotency as storage, not as a check

The unique constraint on the idempotency key is what enforces exactly-once effect. A duplicate insert fails loudly and is treated as already-received.

Claiming, not locking

Workers claim work with a timestamped atomic update. If the worker dies, the claim ages out and another picks it up — no manual unlock.

Visible failure

Exhausted retries land in a dead-letter state with the last error attached, and a small Next.js view makes the ledger inspectable.

Outcome

What changed

  • Duplicate deliveries produce a single effect
  • Worker failure mid-processing recovers without intervention or data loss
  • Failed events are inspectable and replayable rather than silently dropped

Learnings

What I took from it

Most reliability comes from the database, not the application code. Once state lives in one durable place with the right constraints, recovery becomes almost boring.