Resolved
High API Latencies

Started
March 09, 2023 at 4:04 PM
Status
Resolved after about 3 hours

Impact

Degraded performance
Affected
Ledgers API
Platform/API
  • Resolved
    March 09, 2023 at 6:50 PM

    The Modern Treasury API latency has recovered and all operations are normal.

    This incident was caused by resource contention in our shared database cluster that started at around 0100 PT. Our on-calls were paged within minutes, and began investigating the issue.

    Our API latencies began sporadically exceeding our internal SLOs at 0200 PT, and then were consistently above our SLOs at 0700 PT. Ultimately, the incident response team resolved the performance degradation through a combination of workload management and query optimizations. The API performance returned to SLO levels at around 0950 PT.

  • Monitoring
    March 09, 2023 at 6:14 PM

    The Modern Treasury API latency has recovered. We are currently monitoring.

  • Investigating
    March 09, 2023 at 4:04 PM

    The Modern Treasury API may be experiencing degraded performance. We are investigating.