Systems / Backend
Reliable Webhook Processor
Event processing that survives duplicates, retries and worker death
- Organisation
- Personal project
- Role
- Design and implementation
- Period
- 2026
Context
What the system is
Any integration that receives webhooks eventually hits duplicate delivery, partial processing and provider retries. The naive handler quietly corrupts state.
Webhooks arrive more than once, out of order, and sometimes while your worker is dying. This project handles those cases explicitly rather than hoping they do not happen.
Problem
What needed solving
Processing an event twice must not produce two effects, and a worker crashing mid-job must not lose the event or leave it locked forever.
- Providers guarantee at-least-once delivery, never exactly-once
- Acknowledgement must be fast; the work must not happen in the request
- Failures must be visible, not swallowed
Architecture
How it's structured
- 01
Receive endpoint persists the raw event with a provider-supplied idempotency key, then acknowledges immediately
- 02
PostgreSQL holds the event ledger and its processing state — no in-memory queue as the source of truth
- 03
Background workers claim events atomically, so a crashed worker's claim expires and the event is reprocessed
- 04
Retries use exponential backoff with an explicit attempt ceiling and a dead-letter state
Approach
What I worked on
Idempotency as storage, not as a check
The unique constraint on the idempotency key is what enforces exactly-once effect. A duplicate insert fails loudly and is treated as already-received.
Claiming, not locking
Workers claim work with a timestamped atomic update. If the worker dies, the claim ages out and another picks it up — no manual unlock.
Visible failure
Exhausted retries land in a dead-letter state with the last error attached, and a small Next.js view makes the ledger inspectable.
Outcome
What changed
- Duplicate deliveries produce a single effect
- Worker failure mid-processing recovers without intervention or data loss
- Failed events are inspectable and replayable rather than silently dropped
Learnings
What I took from it
Most reliability comes from the database, not the application code. Once state lives in one durable place with the right constraints, recovery becomes almost boring.
Next
Enterprise / Cloud / AI
Vertiv One Cloud
A multi-tenant cloud platform for centralised management of customers, sites, assets, components and gateways — with a query layer designed for AI assistants.
Enterprise / Monitoring
Vertiv Critical Insight
Alarm workflows, point data visualisation and statistics configuration inside a long-running enterprise monitoring platform.