Status: Resolved

Severity: High

Incident window (ET): Feb 25, 2026 07:00 – Feb 26, 2026 13:00

Summary

On Feb 25, we deployed updates to our Tracker API related to order processing and executor integration. Later that day, trading-related endpoints began experiencing increasing latency. Some API users saw slow responses, and some UI users experienced timeouts.

The issue was fully resolved on Feb 26 after we identified a contention problem in the routing layer used for order execution and shipped a fix.

Customer impact

• API users: Increased latency on trading and order-related endpoints
• UI users: Increased latency and some timeouts
• At peak (p90):
  • Order insert latency increased from ~10ms to ~400ms
  • Order match latency increased from ~60ms to ~2s

What we did during the incident

• Investigated the regression using production metrics and traces.
• Applied mitigations that temporarily improved latency, but the issue recurred under load.
• Disabled the new routing path to eliminate the contention while we prepared a permanent fix.

Root cause (high level)

A change introduced additional contention in a routing component on the critical path for trading/order processing. Under load, this caused requests to queue and latency to rise over time.

Resolution

• Immediate mitigation: routed all traffic away from the new execution routing path (reducing contention).
• Permanent fix: updated the routing mechanism to remove the contention, and reduced overhead in our tracing instrumentation.

What we’re improving