The 10-Second Fix: Surviving a Payment Callback Race in Production

Some production bugs are in your code. Some are in your provider's code. The worst ones live in neither — they live in the timing between two systems that are each behaving correctly. This is a story about one of those, how it made real successful payments look like failures to real users, and how a deliberately boring 10-second wait fixed it.

The setup

I work on a mobility platform where users buy subscription shuttle passes — a reserved seat across a 30–45 day window. Seats are contested, so booking runs on a Redis-based distributed hold: selecting a seat places an atomic hold with a 2-minute TTL, the user pays through Razorpay, and on success Razorpay's callback hits our booking API carrying a hold token. The API validates the hold, creates the booking, writes the SQL entries, and releases the hold. The hold, not the payment, is the source of truth for who gets the seat.

This flow was solid. Holds were atomic, release paths covered back-outs and failures, and double-booking was structurally impossible. The bug, when it came, wasn't in any of that.

The symptom

Support tickets started arriving with a specific, alarming shape: a user pays for a pass — these are meaningful amounts, not impulse purchases — the payment succeeds at the bank, and our app tells them it failed. No booking. From the user's chair: the app took my money and gave me an error.

Except it hadn't. Checking those payments on the Razorpay side showed them as successful, and the bookings usually existed too — created seconds after the user had already seen the failure screen and closed the app.

The race

Here's the sequence. The user completes payment inside the Razorpay SDK. The SDK exits back into our app. The app immediately calls our payment status API to decide what to show. That status is driven by Razorpay's server-to-server callback to us — and for larger payment amounts, that callback window would occasionally stretch. Not fail; stretch. So the timeline looked like this:

t+0.0s  user completes payment in SDK
t+0.5s  SDK exits, app regains control
t+0.7s  app polls status API → payment_status = 0 (pending)
t+0.7s  app renders: "payment failed"  ← user sees this
t+4.0s  Razorpay callback lands → payment_status = 1, booking created
t+4.0s  nobody is looking anymore

Every component here is doing its job. Razorpay delivered the callback — a few seconds late, which its contract permits. Our status API answered truthfully — the payment wasn't confirmed yet. The app rendered what the API said. The failure was emergent: we had wired a synchronous user expectation ("tell me now") to an asynchronous source of truth (a callback with variable latency), and treated a transient state as a final answer.

The fix

The instinct is to chase the latency — push the provider for faster callbacks, poll harder, add retries. But the latency isn't ours to fix, and hammering the status API just asks the same premature question more often. The actual fix was to change what the app does with the gap: absorb it instead of exposing it.

We introduced a keyed grace window on the app side. When the SDK exits after a payment attempt, the app doesn't immediately poll and render a verdict. It shows a 'confirming your payment' state and waits 10 seconds — keyed from our side so it applies precisely to this window between SDK exit and callback settlement — and then checks status. Ten seconds comfortably covers the stretched callback windows we observed, and 'confirming' is an honest description of what is actually happening.

Why this is the right shape of fix

• It respects the source of truth. The callback-driven status remains authoritative; we didn't invent a second path that could disagree with it.
• It converts a lie into the truth. 'Failed' was false the moment we showed it. 'Confirming' is accurate for the entire window.
• It handles the distribution, not the average. Most callbacks were fast; the fix exists for the tail. Tail latencies are where user trust actually lives.
• It contains the blast radius. No backend changes, no provider dependency, no new failure modes — a UX-layer acknowledgment of an eventual-consistency reality.

After shipping, the false-failure tickets stopped. Payments that took four seconds to confirm now looked like exactly what they were: payments taking a moment to confirm.

The takeaway

Third-party eventual consistency is your problem, not your provider's. Any time your UI renders a verdict sourced from an asynchronous confirmation — payment callbacks, webhook-driven state, replicated reads — ask what the user sees during the gap, and whether you're presenting a transient state as a final one. Sometimes the most effective distributed-systems fix in the codebase is ten honest seconds of 'confirming.'