Modern Treasury - High API Latencies – Incident details

All systems operational

High API Latencies

Resolved
Degraded performance
Started over 3 years agoLasted about 3 hours

Affected

Platform/API

Degraded performance from 4:04 PM to 6:50 PM

Ledgers API

Degraded performance from 4:04 PM to 6:50 PM

Updates
  • Resolved
    UTC
    Resolved

    The Modern Treasury API latency has recovered and all operations are normal.

    This incident was caused by resource contention in our shared database cluster that started at around 0100 PT. Our on-calls were paged within minutes, and began investigating the issue.

    Our API latencies began sporadically exceeding our internal SLOs at 0200 PT, and then were consistently above our SLOs at 0700 PT. Ultimately, the incident response team resolved the performance degradation through a combination of workload management and query optimizations. The API performance returned to SLO levels at around 0950 PT.

  • Monitoring
    UTC
    Monitoring

    The Modern Treasury API latency has recovered. We are currently monitoring.

  • Investigating
    UTC
    Investigating

    The Modern Treasury API may be experiencing degraded performance. We are investigating.