A webhook is an HTTP callback you do not control the timing of. The sender retries. Your process restarts. The same event arrives twice, late, or not at all. If the handler is "parse JSON and update the row," you will ship a double charge, a double email, or a support ticket you cannot replay.
This is the bar we use on systems integration. It pairs with API rate limiting (the sender will burst) and OpenTelemetry on Node (you need the event id in the trace). A webhook is not a simpler queue. It is a queue you do not own.
The Contract, in One Page
Write this down before the first endpoint exists. Both sides should be able to read it without a call.
- Event name and version (
invoice.paid.v1), not a blob that changes shape - Id that is stable across retries
- Signature over the raw body, with a timestamp, using a secret that is not in the repo (secrets stay out of git)
- Response rule:
2xxmeans "I have durably accepted this," not "I finished every side effect" - Retry schedule the sender actually uses, and when they give up
Verify the signature before you parse into objects you trust. Reject stale timestamps so a captured request cannot be replayed next week. Rotate the signing secret without a deploy freeze: accept the previous secret for a short overlap.
If you are the sender, sign the same way. Do not ask the receiver to "just trust the source IP." IPs change. Signatures do not, until you rotate them on purpose.
Idempotency Is the Product
The sender will deliver duplicates. Your handler must treat that as normal.
- Persist the event id before you call a payment API, send mail, or write the business row.
- If the id exists, return
2xxand stop. - Do the side effect once. If the side effect fails, record the failure and let a retry or a worker finish it โ do not pretend the first attempt's HTTP 500 means the sender should invent a new id.
"Check then insert" without a unique constraint will race. Use a unique key on the event id (or a dedupe table) so two workers cannot both win.
Side effects that are not naturally idempotent (send this email, capture this payment) get their own idempotency key passed to the downstream API. Your dedupe row is not enough if Stripe or the mail provider also retries.
Retries, Timeouts, and the Poison Pile
Respond fast. The work can finish later. A handler that holds the request open for a PDF render will trip the sender's timeout, earn a retry, and do the work twice if you skipped the section above.
| Symptom | What to do |
|---|---|
| Timeout then retry | Ack after durable save; process async |
| Same id, new body | Reject or store as a conflict; do not silently overwrite |
| 4xx from you | Sender should stop; you own the bug |
| 5xx or timeout | Sender retries with backoff; you must be idempotent |
| Still failing after N tries | Poison queue or dead letter you can open, not an infinite loop |
Log the event id, the attempt, and the outcome. Alert on age (oldest unprocessed) and poison depth, not on every retry. Retries are healthy. A pile that never drains is an incident.
Give on-call a replay button that re-runs from the stored body, with the same id, in a non-production mode first if the side effect is money or a customer email. "We will resend from their dashboard" is not a plan when their dashboard is down.
What On-Call Needs on Day One
- A page that lists recent events by id and status (accepted, done, failed, poison)
- The signature failure count (clock skew vs wrong secret vs attack)
- A link from the trace to the event id
- Who to call on the other side, and their status page
- A written note: which events are safe to replay
This is smaller than a platform. It is the difference between a 10-minute answer and a night of guessing which row moved.
Rate-limit the endpoint anyway. A buggy sender is a client. 429 with a story is kinder than falling over. Do not put the signing secret in the rate-limit key.
Outbound webhooks you send deserve the same design: stable ids, signatures, backoff, and a log the customer can quote in a ticket. If you cannot show "we delivered id abc at this time and got this status," you will argue about it in email.
FAQ
Should we use a message queue instead? If both sides are yours, yes โ a queue with a consumer is easier to operate than HTTP between two deployables. Webhooks are for crossing a trust boundary (a vendor, a customer, a bank). Do not webhook yourself to look event-driven.
Is SSE or a WebSocket a substitute? Those are for a client you hold open. A webhook is for a server that was not listening at the moment you had news. The tradeoff is in WebSocket vs SSE; it does not remove the need for idempotency when money moves.
Can the handler return 200 and forget? Only if "forget" still means the event is stored. A 200 with no record is how you lose the one retry that would have saved the order.
The Bottom Line
Webhooks you can operate have a versioned event, a signature, a unique id stored before side effects, a fast 2xx, and a poison pile a person can replay. Retries are part of the design, not a failure of it.
If you cannot explain one event id to on-call, you are not ready to point a vendor at the URL.
Wiring a vendor or customer callback into a product? See systems integration or get a free estimate.