The silent webhook: 41 hours of stale statuses nobody reported

Nobody called. That is the symptom. Axlebrook Parts pushed a change to their order-management system (OMS) on Wednesday evening; since May 6, 9:04 PM ET their webhook endpoint — the address on their side where our status messages land — has answered every delivery-event POST with an HTTP 500, the answer a server gives when it crashes. Their deliveries are unaffected — 371 have completed normally since the deploy — but the status updates describing those deliveries stopped reaching their OMS 41 hours ago.

Their ops team has been quoting stale delivery states to callers for a day and a half without knowing it. The board’s webhook-health column has read failing since Wednesday night; this morning’s board review turned it into a case. The customer still hasn’t noticed. The case below is the live investigation.

The webhook request log, at the boundary. Three healthy deliveries, then the deploy lands and every attempt — including all retries — dies with a 500 in under 100 ms:

One event’s retry ladder on the clock: when a message fails we send it again, each wait longer than the last. Burq’s blog checklist confirms retries exist; the exact schedule below is illustrative (ASSUMPTIONS.md A1: 0s · +1m · +5m · +30m · +2h · +6h). After the final attempt the event is dropped — thrown away for good. It will not come back on its own once the endpoint recovers, which is why a dead endpoint doesn’t heal itself, and why this needed a phone call rather than patience:

The gap stale data opens. Left: what Burq knows. Right: the last status Axlebrook Parts’s OMS ever received for the same delivery:

The whole diagnosis is two log reads, because the question has exactly two suspects: our events stopped firing or your endpoint stopped accepting.

Read one: Burq-side status events kept appending all day — deliveries created, picked up, delivered, on schedule. So the pipeline that generates events is healthy. Read two: the webhook attempt log shows us POSTing on schedule the entire time, every attempt answered with a 500 in under 100 ms. The speed is the tell. A dead host or a broken network makes the sender wait and then give up with nothing; an answer that fast means something on their side picked the message up, tried to handle it, and fell over. Their application is crashing on our request — not a network problem, not silence on our side. If events had stopped firing, that table would be empty, not red.

The timestamp seals it: last 2xx at May 6, 9:01 PM ET, first 500 at 9:04 PM — squarely inside their Wednesday-evening deploy window.

Resolution is on their side: roll back or patch the Wednesday deploy. Once the endpoint returns 2xx, we send the list of 379 affected orders and they re-sync — ask us for each order’s current state and catch their own records up — via GET /v2/orders/{id} with expand=latest_delivery — the documented read path. Same-day recovery.

Prevention is on ours. This gap was caught by a threshold, not a human — but 41 hours is still too slow. The alert below pages after 6 consecutive terminal failures (messages that used up every retry), which for an account at this volume means minutes, not days. This failure mode is the reason the board carries a webhook endpoint health column at all.