Starting at 9:03 am PT we experienced errors affecting ~3% of APIs and Web requests. This issue was caused by a manual invocation of a batch of jobs which overwhelmed our database. By 9:42 am PT, we had removed the jobs and service had returned to normal. The issue has been resolved and the service is operating normally.

We can confirm errors affecting APIs and Web. We have identified a potential root cause of the errors and are attempting mitigations. Early signs are this mitigation is reducing error rates. We continue to work towards a full root case and recovery.