Technology3 min read
Reliable webhooks: design for retries, duplicates and delays
A customer pays, but the order remains unpaid. Elsewhere, one purchase creates two orders. Both failures expose the same distinction: receiving a notification and completing the right business operation are separate responsibilities.
A webhook is a notification one system sends another about an event. It might announce a payment confirmation, order change or new record. The receiver must establish the sender, recognise previous processing and handle interruptions.
Plan beyond the successful delivery
Notifications may arrive late, more than once or in an unexpected order. Stripe’s webhook documentation, for example, describes duplicate events and delivery ordering that is not guaranteed. Check the actual provider’s contract rather than assuming all services behave alike.
Decide what determines the final order state, when a fresh API lookup is required and which actions need their own records. Opening a success page in the customer’s browser is not a trustworthy payment confirmation. Follow the payment provider’s server-side verification process.
Verify the sender and message integrity
HTTPS is a necessary foundation, but it does not by itself identify the sender. When a provider signs notifications, validate the signature using its documented method and appropriate library. Stripe signature verification, for example, requires the unchanged raw request body.
Store secrets on the server rather than in public code or URLs. Separate test and production configuration. Reject invalid signatures before business processing, and record diagnostic information without exposing sensitive payload contents.
Store reliably before doing the slower work
The receiver should not keep a request open while every downstream system responds. GitHub’s webhook recommendations also describe prompt acknowledgement and asynchronous queues. Deadlines and redelivery behaviour depend on the provider.
A practical design verifies the source, stores the event durably and then acknowledges receipt. A background process performs the remaining work. If storage fails, do not hide the failure behind a success response. Agree how provider retries and your own recovery process will handle that case.
Prevent a repeated event from repeating the outcome
A unique event identifier helps recognise redelivery. A simple “check, then insert” sequence can still race when requests arrive together. Include a suitable database uniqueness rule and safe repetition of the business operation in the implementation.
Receiving a confirmed payment again should not create another order or fulfilment request. Track receipt, processing attempts and completion separately. After a partial failure, the system needs to continue without repeating steps that have already succeeded.
Test failures and assign an owner
Useful records include event and order identifiers, receipt time, attempt count and a comprehensible failure reason. Do not log complete payment details or secret keys. Repeated failures need an alert and a person responsible for responding.
- Deliver the same test event twice and concurrently.
- Change event order and inspect the final state.
- Temporarily disconnect the receiver or a downstream test service.
- Try an invalid signature and missing required data.
- Check retries, safe continuation and reconciliation against the provider’s records.
During integration development, these scenarios define what happens when a connected service is slow or unavailable. After launch, technical support should include monitoring those outcomes and investigating failures.
