March 09, 2023 at 6:50 PMThe Modern Treasury API latency has recovered and all operations are normal.
This incident was caused by resource contention in our shared database cluster that started at around 0100 PT. Our on-calls were paged within minutes, and began investigating the issue.
Our API latencies began sporadically exceeding our internal SLOs at 0200 PT, and then were consistently above our SLOs at 0700 PT. Ultimately, the incident response team resolved the performance degradation through a combination of workload management and query optimizations. The API performance returned to SLO levels at around 0950 PT.
