Resolving a Peak-Hour GraphQL Subscription Incident

The challenge

During launch and peak traffic hours at GoPato, our operations agents noticed that customer orders were not being picked up and delivery drivers could not accept incoming orders.

Inspecting our monitoring dashboard showed that the order service was severely overloaded, even though the rest of the platform remained operational thanks to our microservices boundaries. Because we were in the middle of launching, every minute of degraded order flow risked leaving customers and drivers with a poor first impression.

My contribution

As CTO, I led both the immediate incident response and the live root-cause investigation:

To buy breathing room for the live system, I first scaled up the number of order service instances. However, resource usage continued climbing rapidly—horizontal scaling alone was not going to hold for long.

Using Apollo's GraphQL monitoring tooling, I inspected live connection and operation metrics while the overload was happening. The telemetry revealed that certain GraphQL subscriptions were firing at an unexpectedly high rate. Tracing those subscriptions in the codebase uncovered the root cause: the subscription filter logic was not evaluating as originally expected and was creating a feedback loop where every new order triggered a cascade of additional connections and subscription updates.

I wrote and deployed a targeted patch to fix the subscription filter behavior and break the feedback loop, then worked with the team post-incident to integrate stress testing into our workflow.

Decisions and trade-offs

When the overload hit during peak launch traffic, the immediate decision was to scale order service instances first to reduce customer impact while keeping the system live so we could observe real telemetry rather than guessing at the cause offline.

Having service isolation (microservices) and dedicated GraphQL connection observability (Apollo tooling) proved decisive: the rest of the platform stayed up while telemetry pointed directly to subscription fan-out rather than database or REST endpoint bottlenecks.

Outcomes and lessons

Deploying the subscription filter patch resolved the overload within minutes and restored normal order pickup and driver acceptance. Adding stress testing afterward ensured that high-concurrency subscription behavior was validated before future releases reached production.