Modern Treasury - Partial API disruption – Incident details

All systems operational

Partial API disruption

Resolved
Operational
Started over 2 years agoLasted 41 minutes

Affected

Platform/API

Operational from 4:22 PM to 5:03 PM

Ledgers API

Operational from 4:22 PM to 5:03 PM

Push to Warehouse

Operational from 4:22 PM to 5:03 PM

Marketing Website

Operational from 4:22 PM to 5:03 PM

Third Party Services

Operational from 4:22 PM to 4:43 PM, Degraded performance from 4:43 PM to 5:03 PM, Operational from 4:43 PM to 5:03 PM

Login: Auth0 US-3

Operational from 4:22 PM to 5:03 PM

Updates
  • Resolved
    Resolved

    Starting at 9:03 am PT we experienced errors affecting ~3% of APIs and Web requests. This issue was caused by a manual invocation of a batch of jobs which overwhelmed our database. By 9:42 am PT, we had removed the jobs and service had returned to normal. The issue has been resolved and the service is operating normally.

  • Identified
    Identified

    We can confirm errors affecting APIs and Web. We have identified a potential root cause of the errors and are attempting mitigations. Early signs are this mitigation is reducing error rates. We continue to work towards a full root case and recovery.

  • Investigating
    Investigating

    We are currently investigating an incident that's impacting a percentage of requests to APIs starting from 9:03 am PT.