Payment providers deliver events over HTTP, and HTTP delivery fails in ordinary ways: timeouts, deploys that drop connections, retries after a slow response. A webhook receiver that assumes each event arrives exactly once, on time and in order will eventually corrupt state. This article walks through a receiver design that treats duplicates, delays and reordering as normal conditions rather than exceptions. It leans on public documentation from Stripe, Adyen and PostgreSQL for concrete claims, and it flags where the sources do not say enough to be certain.
Why webhook handlers must assume duplicates
Webhooks exist because many payment flows are asynchronous. Stripe's webhook documentation (https://docs.stripe.com/webhooks) describes pushing real-time data to your HTTPS endpoint when events occur, for example when a customer's bank confirms a payment, a customer opens a dispute, or a recurring payment succeeds. Adyen's documentation (https://docs.adyen.com/development-resources/webhooks/) makes the same point: the final status of a resource is often unknown right after the API request, so the provider notifies you when the outcome is available, and webhooks also carry events that originate outside your own requests, such as chargebacks and report availability.
Notice what these descriptions imply. Events are pushed to you by a third party on its own schedule. The excerpts we reviewed do not specify exact retry schedules or ordering guarantees, so a safe design must not depend on any particular behavior. If you need those details for a specific provider, read its current delivery and retry documentation directly rather than inferring them from a general guide like this one.
Three failure modes drive the design:
- Duplicates: the same event is delivered more than once, for instance because your acknowledgement was lost or slow. Without protection, you might ship an order twice or credit an account twice.
- Delays: an event arrives long after the state it describes. Your system may already have moved on.
- Out-of-order delivery: a later event arrives before an earlier one. If handlers blindly overwrite state with whatever arrives last, an old status can replace a newer one.
Idempotency, meaning that processing the same input twice has the same effect as processing it once, is the core property that neutralizes the first failure and helps with the other two.
Verify the signature before you trust the payload
A webhook endpoint is a public URL. Stripe requires registered endpoints to be publicly accessible HTTPS URLs (https://docs.stripe.com/webhooks), which means anyone who discovers the URL can send it requests. The provider therefore gives you a way to prove authenticity. In Stripe's setup flow, creating a webhook destination yields a signing secret that begins with whsec_, which you use when building the handler. Adyen's documentation tree includes dedicated pages on securing webhooks and verifying HMAC signatures (https://docs.adyen.com/development-resources/webhooks/).
Practical rules that follow from this:
1. Verify before parsing business logic. Compute or check the signature against the raw request body exactly as received. Re-serializing parsed JSON can change bytes and break verification. 2. Reject on failure. Return a 4xx status for requests whose signature does not validate, and do not enqueue them. 3. Keep secrets out of code and logs. Store the signing secret in your secrets manager and rotate it according to your provider's process. 4. Verify first, deduplicate second. Never write an unverified event ID into your dedup table; an attacker could otherwise poison it and cause genuine events to be discarded as duplicates.
The exact verification algorithm and tolerance windows are provider-specific. Use the provider's official library where one exists rather than hand-rolling the comparison.
Acknowledge fast, process later
Adyen's guidance is direct: accept the webhook with a 2xx HTTP status code, store the message, and then process its contents (https://docs.adyen.com/development-resources/webhooks/). That sequence is the backbone of a resilient receiver. The HTTP request should do as little as possible: verify the signature, persist the raw event durably, and return success. Everything expensive happens afterward in a worker.
This split matters because slow synchronous processing creates its own duplicates. If your handler calls three downstream services before responding and one of them stalls, the provider may treat the delivery as failed and send the event again while your first attempt is still running. Two copies of the same event are now in flight concurrently. Acknowledging after durable storage removes most of that window. The excerpts we reviewed do not state the provider's timeout values, so treat the exact threshold as something to confirm in your provider's documentation.
A reasonable shape for the receiver:
1. Read the raw body. 2. Verify the signature; return 4xx on failure. 3. Insert the event into a durable table (details in the next section). 4. Return 2xx. 5. A separate worker claims unprocessed rows and performs the business logic.
If step 3 fails because your database is down, return a 5xx so the provider retries. Returning 2xx before the event is safely stored defeats the purpose.
Deduplicate with a unique constraint in PostgreSQL
The most reliable duplicate defense is one the database enforces. Create an events table with a unique constraint on the provider's event ID:
CREATE TABLE webhook_events ( id bigserial PRIMARY KEY, provider text NOT NULL, event_id text NOT NULL, payload jsonb NOT NULL, received_at timestamptz NOT NULL DEFAULT now(), processed_at timestamptz, UNIQUE (provider, event_id) );
Then the receiver stores events with:
INSERT INTO webhook_events (provider, event_id, payload) VALUES ($1, $2, $3) ON CONFLICT (provider, event_id) DO NOTHING RETURNING id;
The PostgreSQL INSERT documentation (https://www.postgresql.org/docs/current/sql-insert.html) explains why this works. ON CONFLICT specifies an alternative action to raising a unique constraint violation error, and DO NOTHING is one of the available conflict actions. The RETURNING clause returns values only for rows that were actually inserted, or updated when DO UPDATE is used. So if RETURNING yields a row, this delivery is new; if it yields nothing, a copy already exists and you can still acknowledge with 2xx. Because the uniqueness check is enforced by the index inside the database, two concurrent deliveries of the same event cannot both insert, which an application-level SELECT-then-INSERT cannot guarantee.
A few design decisions deserve explanation:
- Why include provider in the key? Event IDs are only meaningful within one provider's namespace. Scoping avoids accidental collisions if you integrate several.
- Why DO NOTHING rather than DO UPDATE? Re-delivery should not overwrite the stored payload or reset processing state. The documentation notes that DO UPDATE additionally requires UPDATE privilege and SELECT privilege on columns it reads, so DO NOTHING also keeps privileges simpler.
- Why store the raw payload? It lets you replay processing after a bug fix and gives you an audit trail.
One caveat from the PostgreSQL page: inserts into tables with unique indexes can block when concurrent sessions lock or modify rows matching the same unique values. For a webhook table this is brief, but it is worth knowing if you see lock waits under bursty duplicate delivery.
Dedup at the event level protects you from the same delivery twice. It does not by itself make your business logic safe: the worker can crash after performing a side effect but before setting processed_at, and a retry will repeat it. That is where idempotency on outbound calls comes in.
Use idempotency keys on outbound API calls
Workers often call the provider back, for example to create a refund, capture a payment or create an object. If a connection error occurs mid-call, you cannot know whether the operation happened. Stripe's idempotent requests reference (https://docs.stripe.com/api/idempotent_requests) addresses exactly this: you supply an idempotency key so a request can be retried without performing the same operation twice.
Key details from that page worth designing around:
- The client generates the key. Stripe suggests V4 UUIDs or another random string with enough entropy to avoid collisions, up to 255 characters, and advises against sensitive data such as email addresses.
- For API v1, Stripe saves the status code and body of the first request made with a key once endpoint execution begins, and retries with the same key return the saved response, including 500 errors.
- Incoming parameters are compared with the original request, and a mismatch produces an error, which protects against accidental key reuse.
- Results are saved only after execution begins; requests that fail validation or conflict with a concurrently executing request are not saved and can be retried.
- For API v1, keys can be pruned once they are at least 24 hours old, after which reuse creates a new request.
- All POST requests accept idempotency keys; GET and DELETE are idempotent by definition and the key has no effect.
- API v2 behaves differently: retries do not repeat successful operations, and Stripe retries failed or partial work to completion where possible.
The crucial design choice is deriving the key deterministically from your own state rather than generating a fresh random value on every attempt. A new random key per retry defeats the mechanism. Instead, persist the key alongside the work item, or derive it from something stable such as the event row's ID plus the action name (for example, refund for event 4812). The same logical operation then always presents the same key, even across worker crashes and redeliveries. Given the 24-hour pruning note for v1, do not rely on a key to protect a retry that happens days later; record completion in your own database as well.
Adyen also documents API idempotency in its resource list (https://docs.adyen.com/development-resources/webhooks/ links to it), but the excerpt we reviewed does not describe its behavior, so check that page before assuming it matches Stripe's.
Handle delayed and out-of-order events
Deduplication answers whether you have seen an event; it does not answer whether the event is still relevant. Adyen notes that webhook messages contain a snapshot of the resource so your system can update its state accordingly (https://docs.adyen.com/development-resources/webhooks/). Snapshots are convenient but dangerous when reordered: applying an older snapshot after a newer one regresses your state.
Strategies, from simplest to most robust:
1. Monotonic state machines. Define allowed transitions (for example, pending to paid to refunded) and ignore any event that would move backward. An out-of-order event is then a harmless no-op. 2. Compare timestamps or version fields where the provider supplies them. Only apply an event if it is newer than what you have stored. Confirm in the provider's schema which field is authoritative; the excerpts we reviewed do not specify one. 3. Re-fetch the current object. When an event arrives, treat it as a signal and read the latest state from the provider's API before updating. This trades an extra call for resilience against stale payloads, and the GET is idempotent by nature. 4. Per-entity ordering in the worker. Process events for the same payment or customer serially so two workers never race on one object.
Choose based on risk. For money-moving transitions, prefer a state machine plus a re-fetch; for low-stakes notifications, a simple guard is enough.
Monitor the event backlog
Once acknowledgement and processing are decoupled, a new failure mode appears: the provider sees success while your worker silently falls behind. Monitoring has to cover your side of the pipeline.
Useful signals, all computable from the events table above:
- Backlog depth: count of rows where processed_at is null.
- Oldest unprocessed age: now() minus the earliest received_at among unprocessed rows. This is usually the best alert signal, because it measures user-visible lag.
- Failure and retry counts per event, so a poison event that always crashes the worker is quarantined rather than blocking the queue.
- Signature-failure rate: a spike can indicate a rotated secret, a misconfigured endpoint or probing.
- Duplicate rate: how often ON CONFLICT suppresses an insert. A sudden jump may mean your acknowledgements are slow or failing.
- Delivery visibility from the provider: Adyen's documentation lists webhook event logs and a troubleshooting section for delivery failures (https://docs.adyen.com/development-resources/webhooks/), so compare provider-side delivery records with your own received counts when numbers disagree.
Set alert thresholds from your own traffic patterns; the sources do not provide universal numbers, and inventing them would not help you.
Implementation checklist
- Endpoint is HTTPS and publicly reachable; secrets are stored securely.
- Raw body is used for signature verification; invalid signatures get a 4xx and are never stored.
- Event is inserted into a durable table with a unique (provider, event_id) constraint using ON CONFLICT DO NOTHING.
- 2xx is returned only after durable storage; database failure returns 5xx.
- Duplicate deliveries are acknowledged, not errored.
- Workers claim rows safely and mark completion; crashes leave rows reprocessable.
- Outbound POST calls use stable, deterministic idempotency keys tied to your own work items.
- Completion of side effects is recorded in your database, not only in the provider's key cache.
- State transitions are monotonic or guarded by version checks; critical paths re-fetch current state.
- Backlog depth, oldest-event age, failure counts and duplicate rate are graphed and alerted.
- A replay procedure exists for reprocessing stored payloads after a fix.
- Provider-specific retry, timeout and ordering behavior has been read from current official docs.
Limitations
This article is built from a limited set of documentation excerpts. Several of those excerpts were truncated, so it does not cover Stripe's handler code samples, thin versus snapshot payload formats in depth, or Adyen's HMAC verification steps. It does not state provider retry schedules, timeout thresholds or ordering guarantees because the reviewed material did not specify them. Idempotency semantics differ between providers and API versions, so confirm behavior for the exact endpoints you use. The SQL shown is a starting pattern; adapt schema, indexing and worker claiming to your own load and operational standards, and test failure scenarios such as duplicate concurrent delivery and mid-processing crashes before relying on the design in production.