Resolved
API Performance Degradation

Started
April 13, 2022 at 11:00 PM
Status
Resolved after 7 days
Affected
Platform/API

Impact

Operational
  • Resolved
    April 20, 2022 at 11:00 PM

    This issue was resolved on April 20 at 7:00 PM

  • Identified
    April 13, 2022 at 11:00 PM

    Starting at around Apr 13 1600 PST, several of our endpoints began suffering from periodic performance degradation every day at 0000, 0600, 1200, and 1800 PST. This latency issue was most evident in our ledgers API endpoints. We found the root cause in our data warehouse integration, and this has been fixed as of Apr 20, 1600 PST.

    Modern Treasury had already been working on new latency alerts for our API endpoints, and these new alerts were actually triggered, but we took some time to calibrate our response since they were new. Going forward, we now have the confidence that these alerts have high signal-to-noise ratio and will initiate a faster response to similar incidents. In about a month, we will also move our data warehouse replication to an isolated setup to further insulate our customers from their side-effects.