Starting at around Apr 13 1600 PST, several of our endpoints began suffering from periodic performance degradation every day at 0000, 0600, 1200, and 1800 PST. This latency issue was most evident in our ledgers API endpoints. We found the root cause in our data warehouse integration, and this has been fixed as of Apr 20, 1600 PST.

Modern Treasury had already been working on new latency alerts for our API endpoints, and these new alerts were actually triggered, but we took some time to calibrate our response since they were new. Going forward, we now have the confidence that these alerts have high signal-to-noise ratio and will initiate a faster response to similar incidents. In about a month, we will also move our data warehouse replication to an isolated setup to further insulate our customers from their side-effects.