Case 2 · The order that never moved

01Symptom

It is Fri, May 8, 2026 — two days before Mother’s Day, the weekend a florist marketplace exists for. At 7:31 AM ET, Petalfront’s primary provider, DashNow, ran out of courier capacity in two Tampa ZIPs: 33611 and 33647. In the other four zones that would have been a non-event — the quote declines, the fallback chain picks it up, nobody notices. In these two, the decline was the end of the line.

Orders kept arriving; dispatch kept asking DashNow; DashNow kept saying no; nothing was configured behind it. By the time the desk flagged the pile-up at 8:47 AM, 26 orders of cut flowers had sat — or would sit — past their pickup windows. Manual re-dispatch ran 11:20 AM–1:15 PM and recovered 23; 3 could not be saved. The incident is still open because the weekend hasn’t happened yet.

7:31 AMcapacity drop
8:47 AMdesk flagged
8:55 AMtriage start
11:20 AMre-dispatch start
1:15 PMre-dispatch done

The 76-minute gap between capacity drop and detection is the old process’s blind spot — closed by the alert in step 04.

Orders stuck

26

Recovered

23

Refunded

3

Basket at risk

$1,667.54

02Evidence trail

Start where dispatch decisions are logged. The sample below is the morning in miniature: DashNow declines, and hours later the same orders dispatch through CourierCo the moment a human routes them there. Read the reason codes — they do the diagnostic work:

Dispatch attempt log sample around the capacity drop and re-dispatch
ts (ET)	order	zip	provider	action	reason_code
May 08 08:51:00	PTF-20260508-2380	33647	DashNow	quote_requested	—
May 08 08:52:00	PTF-20260508-2380	33647	DashNow	quote_declined	capacity_limit_reached
May 08 09:42:00	PTF-20260508-1067	33647	DashNow	quote_requested	—
May 08 09:43:00	PTF-20260508-1067	33647	DashNow	quote_declined	capacity_limit_reached
May 08 11:27:00	PTF-20260508-3334	33611	DashNow	quote_requested	—
May 08 11:28:00	PTF-20260508-3334	33611	DashNow	quote_declined	capacity_limit_reached
May 08 11:29:00	PTF-20260508-2380	33647	CourierCo	quote_requested	—
May 08 11:30:00	PTF-20260508-2380	33647	CourierCo	dispatched	—
May 08 11:54:00	PTF-20260508-1067	33647	CourierCo	quote_requested	—
May 08 11:55:00	PTF-20260508-1067	33647	CourierCo	dispatched	—
May 08 13:01:00	PTF-20260508-3334	33611	CourierCo	quote_requested	—
May 08 13:02:00	PTF-20260508-3334	33611	CourierCo	dispatched	—

Provider decline reason codes are illustrative courier-API vocabulary (ASSUMPTIONS.md A3); the docs document dispatch errors, not per-provider decline reasons.

Now the configuration those declines fell into. Six zones, one difference — the two onboarded later never got a fallback chain (per-zone chains are an illustrative mechanism, ASSUMPTIONS.md A11):

Zone coverage: fallback chain and DashNow quote responses today
zip	area	fallback chain	accepted	declined	coverage
33602	Downtown Tampa	DashNow → CourierCo → FleetX	73	0	covered
33606	Hyde Park	DashNow → CourierCo → FleetX	49	0	covered
33609	Westshore	DashNow → CourierCo → FleetX	87	0	covered
33611	South Tampa	DashNow	2	11	chain ends here
33629	Palma Ceia	DashNow → CourierCo → FleetX	68	0	covered
33647	New Tampa	DashNow	3	15	chain ends here

Accepted / declined are DashNow quote responses today, per zone.

Where and when orders went stuck. Both affected ZIPs, nowhere else, starting the hour capacity dropped:

Stuck orders by zone and hour of day
zip	07:00	08:00	09:00	10:00	11:00	12:00	13:00
33611	·	·	3	2	·	5	1
33647	·	1	3	3	2	3	3

Hour labels are ET; cell shading scales with the count in that cell (darkest = worst hour).

The orders themselves, with what each one was worth and how it ended:

Stuck orders: pickup window, time stuck, basket value, resolution
order	zip	pickup window end (ET)	min stuck	basket	resolution
PTF-20260508-2380	33647	May 08 09:35:00	94	$26.84	redispatched_courierco
PTF-20260508-3334	33611	May 08 12:11:00	30	$101.78	redispatched_courierco
PTF-20260508-1067	33647	May 08 10:26:00	68	$72.21	redispatched_courierco
PTF-20260508-1548	33611	May 08 10:20:00	200	$49.47	refunded
PTF-20260508-7969	33611	May 08 12:00:00	42	$49.90	redispatched_courierco
PTF-20260508-3693	33611	May 08 11:58:00	37	$89.61	redispatched_courierco
PTF-20260508-5829	33647	May 08 12:16:00	39	$71.07	redispatched_courierco
PTF-20260508-3055	33647	May 08 09:40:00	82	$53.68	redispatched_courierco
PTF-20260508-1553	33611	May 08 13:30:00	0	$55.47	redispatched_courierco
PTF-20260508-5080	33611	May 08 10:20:00	85	$20.63	redispatched_courierco

+16 more with the same shape.

the query that surfaced them — now the step-04 alertops warehouse · illustrative schema (A8)

-- Stuck orders: unassigned past their pickup window (A12: 20-minute grace).
SELECT
  order_ref,
  zip,
  pickup_window_end,
  ROUND(EXTRACT(EPOCH FROM (now() - (pickup_window_end + INTERVAL '20 minutes'))) / 60) AS minutes_stuck,
  basket_cents
FROM deliveries
WHERE account_id = 'acct_petalfront'
  AND status IN ('request', 'delivery_created', 'driver_not_assigned')
  AND pickup_window_end < now() - INTERVAL '20 minutes'
ORDER BY pickup_window_end;

03Diagnosis

A pile of stuck orders has three plausible causes, and the reason codes separate them cleanly. A provider outage declines or times out everywhere — but DashNow accepted 277 quotes across the other four zones today with 0 declines. A coverage misconfig returns zone_not_served — nothing did. What the log actually shows is capacity_limit_reached (reason codes illustrative, A3) in exactly two ZIPs: 11 of 13 quotes declined in 33611, 15 of 18 quotes declined in 33647. That is a localized capacity drop — real, external, and ordinarily survivable.

Ordinarily. What made it an incident is on our side of the ledger. South Tampa and New Tampa were onboarded after the original four zones, and the fallback chain was never configured for them (A11). Everywhere else, a DashNow decline converts into a CourierCo dispatch automatically; in these two, the decline was terminal. The capacity drop was the weather. The missing chain was the hole in the roof.

what the operator did not do

Treat it as a DashNow problem alone — the acceptance numbers in the other four zones prove the provider was healthy everywhere the config was. Filing a provider ticket would have fixed nothing.
Keep re-dispatching one order at a time past the threshold — at 10+ stuck in a zone you stop clicking and call the customer (A12), because at that point their substitution decisions — which shops, which arrangements — beat your routing decisions.
Wait for DashNow capacity to recover on its own — with cut flowers on the counter, every hour of waiting converts recoverable orders into refunds.

04Resolution & prevention

Resolution was manual and unglamorous: re-dispatch every stuck order through the chain that should have been there. 23 of 26 orders went out through CourierCo and FleetX between 11:20 AM and 1:15 PM ET. 3 could not be saved — shops closed or stems past holding — and were refunded in full: $251.15.

Prevention shipped the same day. 33611 and 33647 now carry the same fallback chain as the other four zones (A11), FleetX is holding standby capacity in both ZIPs through Sunday, and the alert below pages on the pattern this morning showed us — before a human would have noticed it.

Fallback chain configuration before and after today’s change
zip	area	chain before	chain after
33611	South Tampa	DashNow	DashNow → CourierCo → FleetX
33647	New Tampa	DashNow	DashNow → CourierCo → FleetX

the alert that now pages usops warehouse · illustrative schema (A8)

-- Prevention: page when >= 5 orders go stuck in one zone within 60 min (A12).
SELECT
  account_id,
  zip,
  COUNT(*)               AS stuck_orders,
  MIN(pickup_window_end) AS oldest_window
FROM deliveries
WHERE status IN ('request', 'delivery_created', 'driver_not_assigned')
  AND pickup_window_end < now() - INTERVAL '20 minutes'
  AND pickup_window_end + INTERVAL '20 minutes' > now() - INTERVAL '60 minutes'
GROUP BY account_id, zip
HAVING COUNT(*) >= 5;

the thresholds, and why (A12)

20 min past pickup window = stuck. Long enough to absorb ordinary courier jitter, short enough to leave a rescue window for perishables.
5 stuck in one zone within an hour = page. One stuck order is an accident; five in the same zone is a pattern with a common cause.
10 stuck in a zone = stop re-dispatching and call the customer. Past that point their substitution decisions beat our routing decisions.

the outreach — same facts, two altitudes

To: Jordan Pike (Platform Eng, Petalfront)

Subject: RCA: this morning’s stuck dispatches in 33611/33647 + the config change that prevents a repeat

Hi Jordan, Root cause in one line: DashNow lost courier capacity in 33611 and 33647 at 7:31 AM this morning and those two zones had no fallback chain configured, so declined quotes had nowhere to go. The reason codes separate the failure modes cleanly. DashNow returned capacity_limit_reached in exactly two ZIPs while accepting normally in your other four — so this was a localized capacity drop, not a provider outage (that would decline everywhere) and not a coverage misconfig (that returns zone_not_served). 26 orders exceeded their pickup windows before we intervened. What changed today: 33611 and 33647 now carry the same fallback chain as your other zones (DashNow → CourierCo → FleetX), and we set a stuck-order alert — 5+ orders unassigned 20 min past pickup window in any zone pages us. Both were live by 1:15 PM. For the weekend: we’ve asked FleetX to hold standby capacity in both ZIPs through Sunday. Full dispatch log attached.