Elevated Error Rates & Response Times
Incident Report for Modern Treasury
Postmortem

While adding database indices to improve query performance on our time-series data pages (Events, Request Logs, Audit Records), we introduced an index that caused a regression on load times for some organizations.

We reverted that change and dropped the newly created index. Due to the volume of time series data constantly processed and persisted by Modern Treasury, the creation of new data collided with index removal lock, causing high database latency on some endpoints.

We were able to mitigate the high API latency by temporarily pausing our background worker fleet and allowing the index removal to proceed uninterrupted. Once the index was dropped, the background workers were released and behavior returned to normal.

Posted Mar 17, 2021 - 09:29 PDT

Resolved
From 16:13 PDT to 16:28 PDT, we saw elevated error rates and response times
Posted Mar 16, 2021 - 16:13 PDT