Designing Webhooks: Delivery, Retries, Signatures
A customer completes a payment. Your billing provider fires a webhook to POST /webhooks/stripe, your server is mid-deploy, and the connection resets. The provider does not know if you got it, so it sends the same event again. And again. If your handler is naive, you just provisioned the same subscription twice and emailed two receipts.
Now flip it around. You are the one sending webhooks to your customers’ endpoints. One of them is down for an hour, another returns 200 but silently drops the event, and a third is a spoofed URL an attacker registered to harvest your payloads. Here is the uncomfortable truth: a webhook is a distributed system with no shared transaction, and both sides have to defend themselves. This post covers how to build a sender that is reliable and a receiver that is safe.
At-least-once delivery, so make receivers idempotent
There is no exactly-once delivery over HTTP. The sender cannot tell a lost request apart from a lost response, so any correct sender retries, which means every real webhook system is at-least-once. Duplicates are not an edge case, they are the contract.
That pushes the burden onto the receiver: processing the same event twice must produce the same result as processing it once. The clean way is a dedupe key. Standard Webhooks, Stripe, and Svix all ship a unique message id in a header (webhook-id, or event.id in Stripe’s payload). Store it, and reject repeats before you touch business state.
create table processed_webhooks (
event_id text primary key,
handled_at timestamptz not null default now()
);
-- Inside the handler, in a transaction with the business write:
insert into processed_webhooks (event_id) values ($1)
on conflict (event_id) do nothing;
-- rowCount === 0 => already processed, skip the side effectThis is the same discipline as request-level idempotency. If that pattern is new to you, see idempotency keys prevent double charges. And do not confuse retrying with fixing a broken consumer, because retries are not a fix for a handler that is not idempotent.
The sender: retries with exponential backoff and jitter
A good sender treats a delivery as a durable job, not a fire-and-forget fetch. Success is a 2xx response inside a short timeout. Anything else (5xx, timeout, connection reset) is retryable. A 4xx that is not 408 or 429 usually means the endpoint is misconfigured, so failing fast and dead-lettering is often better than hammering it.
Retry on a schedule that backs off exponentially, and add jitter so a thousand endpoints recovering at once do not all retry on the same tick (a thundering herd).
import crypto from "node:crypto";
const MAX_ATTEMPTS = 8;
const SECRET = process.env.WEBHOOK_SECRET; // "whsec_<base64>"
// Full jitter: random point in [0, exp), capped at 1 hour.
function backoffMs(attempt) {
const exp = Math.min(3_600_000, 1000 * 2 ** attempt);
return Math.random() * exp;
}
function sign(secret, id, timestamp, body) {
const key = Buffer.from(secret.split("_")[1], "base64");
const signedContent = `${id}.${timestamp}.${body}`;
const sig = crypto.createHmac("sha256", key).update(signedContent).digest("base64");
return `v1,${sig}`;
}
async function deliver(job) {
const body = JSON.stringify(job.payload);
const timestamp = Math.floor(Date.now() / 1000); // fresh each attempt
const signature = sign(SECRET, job.id, timestamp, body);
try {
const res = await fetch(job.url, {
method: "POST",
headers: {
"content-type": "application/json",
"webhook-id": job.id,
"webhook-timestamp": String(timestamp),
"webhook-signature": signature,
},
body,
signal: AbortSignal.timeout(10_000),
});
if (res.status >= 200 && res.status < 300) return; // delivered
throw new Error(`non-2xx: ${res.status}`);
} catch (err) {
if (job.attempt + 1 >= MAX_ATTEMPTS) return deadLetter(job, err);
await scheduleRetry(job, backoffMs(job.attempt)); // re-enqueue, attempt + 1
}
}Two details that matter. Sign each attempt with a fresh timestamp, otherwise a retry an hour later fails the receiver’s replay window. And after the last attempt, move the job to a dead-letter queue with the failure reason so you can alert, inspect, and replay manually. A webhook that has silently exhausted its retries is a data-loss incident waiting to be discovered.
Where do the events come from in the first place? If you enqueue delivery jobs in the same transaction that writes your business state, you avoid the classic “committed the order but crashed before queuing the webhook” gap. That is the outbox pattern.
sequenceDiagram
participant DB as Outbox (DB)
participant S as Sender worker
participant R as Receiver
DB->>S: pick pending event
S->>R: POST + webhook-id/timestamp/signature
R-->>S: connection reset (no response)
Note over S: attempt 1 failed -> backoff + jitter
S->>R: POST (retry, same webhook-id)
R->>R: verify sig, check timestamp, dedupe by id
R-->>S: 200 OK
Note over S: mark delivered
The receiver: HMAC-SHA256 signature verification
Never trust an unauthenticated webhook. Anyone who learns your URL can POST a fake “payment succeeded”. The standard defense is a shared secret and an HMAC-SHA256 signature over the payload. The sender and receiver both hold the secret; only a party with the secret can produce a valid signature.
Three rules make or break this.
Verify the raw body, not parsed JSON. You must HMAC the exact bytes that were signed. If you let Express parse the JSON and then re-serialize it, key ordering, whitespace, and unicode escaping can differ, the hash changes, and every verification fails. Stripe is explicit that any manipulation of the raw body breaks verification. Capture the raw buffer.
Compare in constant time. A naive === on strings can leak the correct signature through timing differences, one byte at a time. Use crypto.timingSafeEqual, which requires equal-length buffers, so check the length first.
Enforce a timestamp tolerance. A valid-but-old payload can be replayed. Stripe’s libraries default to a 5-minute window, and the Svix reference implementation of Standard Webhooks uses the same: reject anything whose signed timestamp is too far from now. Never set the tolerance to zero, which disables the recency check.
import express from "express";
import crypto from "node:crypto";
const app = express();
const SECRET = process.env.WEBHOOK_SECRET; // "whsec_<base64>"
const TOLERANCE_SECONDS = 5 * 60;
// express.raw keeps req.body as the untouched Buffer.
app.post("/webhooks", express.raw({ type: "application/json" }), (req, res) => {
const id = req.get("webhook-id");
const timestamp = req.get("webhook-timestamp");
const header = req.get("webhook-signature");
if (!id || !timestamp || !header) return res.status(400).send("missing headers");
const now = Math.floor(Date.now() / 1000);
const ts = Number(timestamp);
if (!Number.isFinite(ts) || Math.abs(now - ts) > TOLERANCE_SECONDS) {
return res.status(400).send("timestamp out of tolerance");
}
const rawBody = req.body.toString("utf8");
const key = Buffer.from(SECRET.split("_")[1], "base64");
const signedContent = `${id}.${ts}.${rawBody}`;
const expected = Buffer.from(
crypto.createHmac("sha256", key).update(signedContent).digest("base64")
);
// The header is a space-delimited list of "v1,<sig>" for key rotation.
const ok = header.split(" ").some((part) => {
const [version, sig] = part.split(",");
if (version !== "v1" || !sig) return false;
const got = Buffer.from(sig);
return got.length === expected.length && crypto.timingSafeEqual(got, expected);
});
if (!ok) return res.status(401).send("invalid signature");
// Signature valid. Dedupe on `id`, enqueue, and return 2xx fast.
res.status(200).send("ok");
});Return 2xx as soon as the event is safely persisted or enqueued, and do the heavy work asynchronously. Providers treat a slow handler as a failed delivery and retry, so blocking on your database migration inside the request just multiplies load.
Standard Webhooks, Stripe, and Svix
You do not have to invent this. The Standard Webhooks spec (created by Svix, and aligned with how Stripe already works) defines exactly the scheme above: the webhook-id, webhook-timestamp, and webhook-signature headers; a signed content string of id.timestamp.payload; HMAC-SHA256 with a base64 secret prefixed whsec_; and a base64 signature carrying a v1, version prefix so you can rotate secrets with zero downtime. Stripe’s scheme is the same idea with its own header shape (Stripe-Signature: t=...,v1=...), a signed payload of {timestamp}.{raw_body}, and a hex-encoded signature computed with the whsec_ secret used directly as the HMAC key (no base64 decode). Svix uses svix-id / svix-timestamp / svix-signature. Learn one, you understand all three.
Takeaway
A webhook is two problems, not one. As a sender, make delivery a durable, retried, jittered, dead-lettered job and sign every attempt fresh. As a receiver, verify the raw body with HMAC-SHA256 in constant time, reject stale timestamps, and dedupe by message id because delivery is at-least-once. Build both halves defensively and a dropped connection becomes a retry, not a double charge or a forged event.