Replaying failed webhooks after an outage, without losing a submission

To replay failed webhooks after an outage, fix the receiving endpoint first, then re-send every delivery that gave up during the outage, keeping each delivery's original id so the receiver can de-duplicate, but signing each one fresh so timestamp checks still pass. Replay only deliveries that exhausted their retries, within a bounded window, and never replay email.

Why aren't automatic retries enough?

Retries are designed for blips: a deploy, a timeout, a 502 from a load balancer. They space out over seconds and minutes and then stop, because hammering a server that is down for hours helps nobody. What is left after the last retry is a delivery that gave up, and without a replay it stays given up. An endpoint that was down from lunch until four o'clock silently loses every submission from lunch until four.

What makes a replay safe?

• Same id, fresh signature. The delivery id identifies the event and must not change, so the receiver can recognise it. The signature and timestamp prove freshness and must be new, or a receiver that rejects old timestamps (as it should) will reject the replay.
• Only what gave up. Deliveries that are still retrying will finish on their own; replaying them too creates races.
• A bounded window. Thirty days covers any realistic outage. Replaying months of history into a system that has moved on causes more harm than the gap did.
• Never email. One failed email is final: a repeat to a mailbox that refused mail damages deliverability for everyone sending from the same infrastructure.

What should the receiver do?

Treat every webhook as at-least-once. A delivery can arrive twice even without a replay: if your server processed it but the response timed out on the way back, the sender sees a failure and tries again. The fix is the same for both cases.

export async function POST(req: Request) {
  const raw = await req.text();
  if (!verifySignature(req.headers, raw)) return new Response("bad signature", { status: 401 });

  const deliveryId = req.headers.get("x-webforms-delivery-id")!;
  if (await alreadyProcessed(deliveryId)) return new Response("ok"); // replay or duplicate

  await handleSubmission(JSON.parse(raw));
  await markProcessed(deliveryId); // only after success
  return new Response("ok");
}

Record the id after processing, not before. If you record it first and then crash, the replay that would have rescued the event is thrown away as a duplicate.

An outage checklist

1. Confirm the endpoint is healthy: send a test submission and watch it arrive.
2. Look at the delivery log for the outage window: how many gave up, and with what errors.
3. Replay the failed deliveries for that form or destination.
4. Watch the log again. Anything that fails twice has a different problem than the outage.

Replaying in WebForms

When deliveries have given up, a form's Deliveries tab shows a Replay failed button with the count. One click sends every webhook, Slack, Discord, Telegram and automation delivery that gave up in the last 30 days again, within a minute, with fresh retries. Each row has its own Replay button too. Replays keep the original x-webforms-delivery-id and get a fresh signature. Email notifications are never replayed. From code it is one call:

curl -X POST https://api.webforms.to/v1/deliveries/replay \
  -H "Authorization: Bearer wf_live_your_key" \
  -H "Content-Type: application/json" \
  -d '{"formId": "frm_xxx"}'