case 1 of 3 · webhook integrity
The silent webhook: 41 hours of stale statuses nobody reported
sev2Axlebrook Partsopened May 8, 9:12 AM ETdetected via: Board webhook-health column: consecutive-failure trip wire (A9)
01Symptom
Nobody called. That is the symptom. Axlebrook Parts pushed a change to their order-management system on Wednesday evening; since May 6, 9:04 PM ET their webhook endpoint has answered every delivery-event POST with an HTTP 500. Their deliveries are unaffected — 371 have completed normally since the deploy — but the status updates describing those deliveries stopped reaching their OMS 41 hours ago.
Their ops team has been quoting stale delivery states to callers for a day and a half without knowing it. The board’s webhook-health column has read failing since Wednesday night; this morning’s board review turned it into a case. The customer still hasn’t noticed. The case below is the live investigation.
Last 2xx from endpoint
May 6, 9:01 PM
Consecutive failures
5,928
Events dropped (retries exhausted)
2,412
Orders affected
379
02Evidence trail
The webhook request log, at the boundary. Three healthy deliveries, then the deploy lands and every attempt — including all retries — dies with a 500 in under 100 ms:
| ts (ET) | event | type | attempt | http | latency |
|---|---|---|---|---|---|
| May 06 20:59:16 | evt_ixjd72 | delivery.updated | 1/6 | 200 | 141 ms |
| May 06 21:00:03 | evt_ixjd7g | delivery.updated | 1/6 | 200 | 272 ms |
| May 06 21:00:50 | evt_ixjd7u | delivery.updated | 1/6 | 200 | 138 ms |
| May 06 21:04:25 | evt_ixjd88 | delivery.courier_location_updated | 1/6 | 500 | 75 ms |
| May 06 21:05:04 | evt_ixjd8m | delivery.updated | 1/6 | 500 | 97 ms |
| May 06 21:05:26 | evt_ixjd88 | delivery.courier_location_updated | 2/6 | 500 | 51 ms |
| May 06 21:06:08 | evt_ixjd8m | delivery.updated | 2/6 | 500 | 96 ms |
| May 06 21:06:35 | evt_ixjd90 | delivery.courier_location_updated | 1/6 | 500 | 61 ms |
| May 06 21:07:31 | evt_ixjd9e | delivery.updated | 1/6 | 500 | 51 ms |
| May 06 21:07:44 | evt_ixjd90 | delivery.courier_location_updated | 2/6 | 500 | 42 ms |
| May 06 21:08:32 | evt_ixjd9s | delivery.courier_location_updated | 1/6 | 500 | 70 ms |
| May 06 21:08:33 | evt_ixjd9e | delivery.updated | 2/6 | 500 | 76 ms |
| May 06 21:09:19 | evt_ixjda6 | delivery.updated | 1/6 | 500 | 76 ms |
| May 06 21:09:29 | evt_ixjd88 | delivery.courier_location_updated | 3/6 | 500 | 44 ms |
| May 06 21:09:37 | evt_ixjd9s | delivery.courier_location_updated | 2/6 | 500 | 64 ms |
| May 06 21:10:08 | evt_ixjd8m | delivery.updated | 3/6 | 500 | 62 ms |
| May 06 21:10:20 | evt_ixjda6 | delivery.updated | 2/6 | 500 | 55 ms |
| May 06 21:11:43 | evt_ixjd90 | delivery.courier_location_updated | 3/6 | 500 | 70 ms |
| May 06 21:12:35 | evt_ixjd9e | delivery.updated | 3/6 | 500 | 65 ms |
| May 06 21:13:35 | evt_ixjd9s | delivery.courier_location_updated | 3/6 | 500 | 51 ms |
| May 06 21:14:19 | evt_ixjda6 | delivery.updated | 3/6 | 500 | 91 ms |
Showing the boundary window; the pattern repeats for every event since. Event shapes per the documented envelope and event types (burq.readme.io/reference/webhooks).
One event’s retry ladder on the clock. Burq’s blog checklist confirms retries exist; the exact schedule below is illustrative (ASSUMPTIONS.md A1: 0s · +1m · +5m · +30m · +2h · +6h). After the final attempt the event is dropped — which is why a dead endpoint doesn’t heal itself when it comes back:
- attempt 1first attemptMay 06 21:04:25HTTP 500
- attempt 2+1mMay 06 21:05:25HTTP 500
- attempt 3+5mMay 06 21:09:25HTTP 500
- attempt 4+30mMay 06 21:34:25HTTP 500
- attempt 5+2hMay 06 23:04:25HTTP 500
- attempt 6+6hMay 07 03:04:25HTTP 500→ event dropped
The divergence that stale data creates. Left: what Burq knows. Right: the last status Axlebrook Parts’s OMS ever received for the same delivery:
| order | burq status | at (ET) | customer last received | at (ET) |
|---|---|---|---|---|
| AXL-20260506-1115 | delivered | May 06 23:10:00 | delivery_created | May 06 21:55:00 |
| AXL-20260507-5601 | delivered | May 07 16:09:00 | delivery_created | May 07 14:41:00 |
| AXL-20260507-2483 | delivered | May 07 16:15:00 | delivery_created | May 07 14:43:00 |
| AXL-20260507-1748 | delivered | May 07 16:34:00 | delivery_created | May 07 15:40:00 |
| AXL-20260507-2403 | delivered | May 07 17:57:00 | delivery_created | May 07 17:00:00 |
| AXL-20260507-8980 | delivered | May 07 17:40:00 | delivery_created | May 07 16:53:00 |
| AXL-20260508-7398 | delivered | May 08 09:36:00 | delivery_created | May 08 08:36:00 |
| AXL-20260508-5877 | delivered | May 08 09:43:00 | delivery_created | May 08 08:43:00 |
-- Which endpoint stopped accepting, and when?
SELECT
account_id,
MAX(ts) FILTER (WHERE http_status BETWEEN 200 AND 299) AS last_2xx_at,
COUNT(*) FILTER (WHERE http_status >= 400
AND ts > now() - INTERVAL '24 hours') AS failed_24h,
COUNT(*) FILTER (WHERE ts > now() - INTERVAL '24 hours') AS total_24h
FROM webhook_attempts
GROUP BY account_id
ORDER BY last_2xx_at ASC;03Diagnosis
The whole diagnosis is two log reads, because the question has exactly two suspects: our events stopped firing or your endpoint stopped accepting.
Read one: Burq-side status events kept appending all day — deliveries created, picked up, delivered, on schedule. So the pipeline that generates events is healthy. Read two: the webhook attempt log shows us POSTing on schedule the entire time, every attempt answered with a 500 in under 100 ms. A connection that answers quickly with a server error is an application accepting the request and crashing on it — not a network problem, not a dead host, and not silence on our side. If events had stopped firing, that table would be empty, not red.
The timestamp seals it: last 2xx at May 6, 9:01 PM ET, first 500 at 9:04 PM — squarely inside their Wednesday-evening deploy window.
-- Read 1: is Burq-side delivery state still moving? (yes → pipeline healthy)
SELECT date_trunc('hour', occurred_at) AS hr,
COUNT(*) AS status_events
FROM delivery_status_events
WHERE account_id = 'acct_axlebrook'
AND occurred_at > now() - INTERVAL '48 hours'
GROUP BY 1
ORDER BY 1;
-- Read 2: are we still trying to deliver events? (yes — and every attempt
-- since the deploy dies with a 500 in under 100 ms)
SELECT http_status,
COUNT(*) AS attempts,
MIN(ts) AS first_seen,
MAX(ts) AS last_seen,
ROUND(AVG(latency_ms)) AS avg_latency_ms
FROM webhook_attempts
WHERE account_id = 'acct_axlebrook'
AND ts > now() - INTERVAL '48 hours'
GROUP BY http_status
ORDER BY http_status;what the operator did not do
- Page Burq engineering — read one shows the delivery pipeline green. Paging would burn engineering time on a customer-side fault.
- Blind-replay dropped events — the endpoint is still down; replays would die the same way. Fix first, re-sync second.
- Wait for the customer to notice — 41 hours of silence is the argument for calling them, not for waiting longer.
04Resolution & prevention
Resolution is on their side: roll back or patch the Wednesday deploy. Once the endpoint returns 2xx, we send the list of 379 affected orders and they re-sync current state via GET /v2/orders/{id} with expand=latest_delivery — the documented read path. Same-day recovery.
Prevention is on ours. This gap was caught by a threshold, not a human — but 41 hours is still too slow. The alert below pages after 6 consecutive terminal failures, which for an account at this volume means minutes, not days. This failure mode is the reason the board carries a webhook endpoint health column at all.
-- Prevention: page when the last 6 attempts to any endpoint all failed (A9).
WITH last_n AS (
SELECT account_id, http_status,
ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY ts DESC) AS rn
FROM webhook_attempts
)
SELECT account_id
FROM last_n
WHERE rn <= 6
GROUP BY account_id
HAVING COUNT(*) = 6
AND COUNT(*) FILTER (WHERE http_status BETWEEN 200 AND 299) = 0;the outreach — same facts, two altitudes
To: Sara Jennings (Integrations, Axlebrook)
Subject: Your Burq webhook endpoint has been returning 500s since May 6, 9:04 PM ET