Driver Dispatch Under Timeouts

Overview

The dispatch pipeline behind ride requests on a multi-tenant mobility platform. A request searches outward in expanding rings, offers the ride to drivers in batches with a fixed acceptance window, and — when a full sweep finds nobody — hands the request to a retry queue rather than declaring failure. Radius, batch size, acceptance window, and retry policy are all per-client configuration.

The Problem

Dispatch looks like a lookup and behaves like a negotiation. The obvious implementation — find nearby drivers, notify them, wait — fails in several directions at once. Notify everyone and the nearest driver competes with one twice as far away, and every driver in the city gets a notification for a ride they won't take. Notify one at a time and a request that should take fifteen seconds takes four minutes. Query once and act on that list for the next three minutes, and you are dispatching to a snapshot of where drivers used to be.

Underneath all of it is the fact that a customer is sitting in the app watching a spinner, and the honest answer for most of that time is 'still looking'.

Requirements

• Nearer drivers get the offer before further ones — proximity should decide who is asked first
• No driver receives the same request twice, however many rings the search passes through
• Each client configures radius growth, batch size, and acceptance window for their own market
• Filters applied consistently: schedule conflicts, driver wallet limits, favourite and blocked relationships
• A failed sweep must not be terminal — driver availability changes minute to minute
• Any money held against the request is released if dispatch ultimately fails

Architecture

The search grows outward in rings. Each hop re-queries for drivers within the current radius, then narrows to those in the new ring only, excluding anyone already notified in a previous hop. The re-query per hop is deliberate: drivers move, go offline, and accept other rides during a sweep, so a list assembled at the start of the search is progressively less true the longer the search runs.

Within a ring, drivers pass through the eligibility filters and are split into batches. Every driver in a batch is offered the ride simultaneously — an engagement row per driver, a push notification, then a fixed wait for the acceptance window to elapse. Whoever accepts first takes the ride; the remaining engagements in that batch are timed out. If nobody accepts, the next batch is offered, and when the ring is exhausted the radius grows.

The customer's request is answered over a streamed HTTP response rather than a single reply. An acknowledgement goes out as soon as the request is created, and the connection stays open through the sweep so the app can be told what actually happened — accepted, queued for retry, or exhausted — instead of being left to poll a request whose duration is unknown in advance.

The Design Decision

The most consequential choice was what happens when a full sweep finds nobody. Treating that as failure is the simple option and it is usually wrong: the sweep proves no driver was available during those few minutes, not that no driver will be available. Supply in a mobility marketplace changes on exactly that timescale — a driver ends a trip, another comes on shift.

So exhausting the radius enqueues the request onto a Redis-backed job queue with a configured delay, and a worker re-runs the entire dispatch when the delay elapses. The request stays alive across attempts, up to a per-client retry limit. This separates two things that look identical from inside a single sweep: nobody is free right now, and nobody is going to be.

Retries are enqueued explicitly rather than relying on the queue's own retry mechanism. A retry here is a fresh business decision — the request may have been cancelled, may have been accepted through another path, and the driver landscape has changed — so each attempt starts by re-checking that the request is still alive before doing anything. Automatic retries would treat a deliberate re-attempt as a failed job being replayed, which is a different thing with different semantics.

Trade-offs

• Batched offers mean several drivers are notified for a ride only one can take. Sequential offers would eliminate that entirely and multiply time-to-assignment — accepted, because an unassigned rider is a worse outcome than a declined notification
• Re-querying every hop costs more database work than one query up front, and buys accuracy that grows more valuable the longer the search runs
• A fixed acceptance window per batch is simple to reason about and means a batch of unresponsive drivers costs the full window before the search moves on
• Holding a streamed connection open for the duration of a sweep ties up a connection per in-flight request, in exchange for the app never having to guess how long to poll
• Retrying keeps requests alive longer, which is better for assignment rates and means the system must be scrupulous about releasing held funds when a request finally does die

Challenges

The hardest correctness problem is that a request can be resolved from several directions while a sweep is running — the customer cancels, a driver accepts, the retry worker picks it up. Every stage re-checks that the request is still live before creating engagements or sending notifications, because acting on a stale view of a request means notifying drivers about a ride that no longer exists.

The failure path carries more responsibility than the success path. When dispatch finally gives up, wallet debits have to be refunded and pre-authorised card holds released — and that has to happen exactly once regardless of which of several exit paths the request took. Getting money released reliably on the unhappy path is harder than assigning a driver on the happy one.

Running the same dispatch logic in two execution contexts — inside a streamed HTTP request and inside a queue worker — is a standing source of drift. The two paths must apply identical filters, or a retry silently offers the ride to a different set of drivers than the first attempt did.

Lessons Learned

'Not found' and 'not yet' are different answers and deserve different handling. Most of dispatch's value came from refusing to collapse them.

Freshness beats efficiency when the underlying data is in motion. Re-querying each ring looked wasteful and was the difference between dispatching to drivers and dispatching to where drivers had been.

Long-running operations need a channel to say 'still working'. A request that legitimately takes minutes is not a slow request; it is an operation with progress, and pretending otherwise pushes the problem into the client as polling and guesswork.

Logic that runs in two places will drift. Shared behaviour needs to be shared code, not copied code with a comment saying it was copied.