Resolved
Partial API disruption

Started
September 29, 2023 at 4:22 PM
Status
Resolved after 41 minutes

Impact

Operational
Affected
Platform/API
Marketing Website
Ledgers API
Compliance API
Push to Warehouse
Third Party Services
Payments: Lob API
Compliance: KYC Data Provider
Payments: Plaid API
Login: Auth0 US-3
Compliance: KYB Data Provider
Compliance: Device Data Provider
  • Resolved
    Resolved

    Starting at 9:03 am PT we experienced errors affecting ~3% of APIs and Web requests. This issue was caused by a manual invocation of a batch of jobs which overwhelmed our database. By 9:42 am PT, we had removed the jobs and service had returned to normal. The issue has been resolved and the service is operating normally.

  • Identified
    Identified

    We can confirm errors affecting APIs and Web. We have identified a potential root cause of the errors and are attempting mitigations. Early signs are this mitigation is reducing error rates. We continue to work towards a full root case and recovery.

  • Investigating
    Investigating

    We are currently investigating an incident that's impacting a percentage of requests to APIs starting from 9:03 am PT.