Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

View current status

Jul 2022

Elevated Error Rates
ResolvedDegraded performance6 hours 52 minutes
  • Resolved
    UTC
    Resolved

    We saw a higher than normal level of errors due to a service disruption with some of our partners today between approximately 10:15 and 11:15 PST. After fixes were implemented on their systems, we saw this these elevated rates decrease. However, we continued to monitor our own platform for any anomalies. Finding none, we are now considering this issue resolved and service fully restored.

  • Monitoring
    UTC
    Monitoring

    The service disruptions with some of our partners appears to have been resolved. Between approximately 10:15 and 11:15 PST, you may have experience increased authentication errors or slow page loads when accessing our platform. However, these should now been returning to normal levels.

    We will continue to monitor our platform for any anomalies.

  • Investigating
    UTC
    Investigating

    We're seeing a higher than usual rate of errors due to a service disruption with some of our partners. You may experience delays or disruptions when accessing pages or sending requests to our API. We are currently looking into the issue and will respond with more information as it becomes available. More information on our partner's statuses are available at the links below.

    • Auth0: https://status.auth0.com/incidents/53ghn3t00tr7
Degraded Availability on APIs
ResolvedMajor outage23 minutes
  • Resolved
    UTC
    Resolved

    At around 11:40 AM (PT) on July 25, a faulty change to our tracing and metrics caused extended downtime on our API. For about 23 minutes the Modern Treasury API was in a degraded state, with 100% unavailability for 13 minutes within that period. Going forward, we commit to improving our CI/CD processes to ensure that similar faults are caught earlier in our pipelines and prevented from deployment.

  • Investigating
    UTC
    Investigating

    We are experiencing a high number of http status code 500s on many of the APIs. We've identified the issue and working on restoring service.

JUL16
Partner Maintenance: Wells Fargo API Scheduled Maintenance
CompletedMaintenance8 hours
  • Completed
    July 16, 2022 at 10:00 AMUTC
    Completed
    July 16, 2022 at 10:00 AMUTC

    Maintenance has completed successfully

  • In progress
    July 16, 2022 at 2:00 AMUTC
    In progress
    July 16, 2022 at 2:00 AMUTC

    Maintenance is now in progress

  • Planned
    July 16, 2022 at 2:00 AMUTC
    Planned
    July 16, 2022 at 2:00 AMUTC

    The Wells Fargo API will undergo scheduled maintenance from this Friday, 2022-07-15 22:00 to Saturday, 2022-07-16 06:00 Eastern Time.

    Account balances and transactions will not update for Wells Fargo accounts during this time. RTP and ACH payments originated from Wells Fargo accounts during this time will be automatically retried when the API is back online.

    Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Elevated Error rates
Resolved55 minutes
  • Resolved
    UTC
    Resolved

    Incident has been resolved

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We're experiencing an elevated level of errors and are currently looking into the issue.

Temporary Degraded Availability
ResolvedDegraded performance9 minutes
  • Resolved
    UTC
    Resolved

    We experienced a brief period of degraded availability between approximately 9:10 and 9:50 PST. You may have experienced a higher than usual rate of 503 responses during this period.

  • Investigating
    UTC
    Investigating

    We're experiencing an elevated level of errors and are currently looking into the issue.

Jun 2022

ACH payments effective June 21st, 2022 delayed
Resolved20 hours 51 minutes
  • Resolved
    UTC
    Resolved

    Payments that were delayed were successfully transmitted to the appropriate payment networks and will are undergoing standard reconciliation processes. Delayed payments have had their effective date field updated to the appropriate date.

  • Monitoring
    UTC
    Monitoring

    Due to a processing error, ACH payments effective for June 21st, 2022 may be delayed by one day.

    The addition of the observed Federal holiday for June 19th resulted in ACH payments that were not transmitted to the ACH network on the appropriate date. Any delayed ACH payments will be automatically transmitted and updated for a June 22nd effective date.

Temporary Degraded Availability
ResolvedMinor outage5 minutes
  • Resolved
    UTC
    Resolved

    By 9:15am PST, the app fully recovered.

  • Monitoring
    UTC
    Monitoring

    Due to unexpectedly high request volume this morning, we experienced a brief period of degraded availability between approximately 9:10am and 9:15am PST. You may have experienced a higher than usual rate of 503 responses during this period. Going forward, we will continue to invest in the scalability of our systems to handle higher request volumes and support the growth of our customers’ businesses.

Temporary Degraded Availability
ResolvedMinor outage5 minutes
  • Resolved
    UTC
    Resolved

    By 9:10am PST, the app fully recovered.

  • Monitoring
    UTC
    Monitoring

    Due to unexpectedly high request volume this morning, we experienced a brief period of degraded availability between approximately 9:05am and 9:10am PST. You may have experienced a higher than usual rate of 503 responses during this period. Going forward, we will continue to invest in the scalability of our systems to handle higher request volumes and support the growth of our customers’ businesses.

Elevated Error Rates
ResolvedMinor outage10 minutes
  • Update
    UTC
    Update

    We have determined the cause to be related to one of our scaling systems. The issue triggered an automatic restart causing a short period of unavailability, however we are working on a long-term fix to prevent this from happening in the future.

  • Resolved
    UTC
    Resolved

    The Modern Treasury API experienced a brief period of unavailability between approximately 9:10am and 9:20am PST and has since recovered. We are investigating.

  • Monitoring
    UTC
    Monitoring

    The Modern Treasury API experienced a brief period of unavailability between approximately 9:10am and 9:20am PST and has since recovered. We are investigating.

Partner Issue: Plaid is experiencing API Downtime impacting Payment Order and External Account Creation
ResolvedDegraded performance
  • Resolved
    UTC
    Resolved

    The Modern Treasury application and API have recovered. For more details see: https://status.plaid.com/incidents/82fmcmd3d42x

  • Identified
    UTC
    Identified

    For more details see: https://status.plaid.com/incidents/82fmcmd3d42x

May 2022

An automated up-time check has failed. (False alarm due to over-sensitive check.)
Resolved8 minutes
  • Resolved
    UTC
    Resolved

    The Modern Treasury application and API have recovered.

  • Investigating
    UTC
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Partner Issue: Plaid is experiencing API Downtime impacting Payment Order and External Account Creation
ResolvedMajor outage2 hours 44 minutes
  • Resolved
    UTC
    Resolved

    The Modern Treasury application and API have recovered.

  • Identified
    UTC
    Identified

    For more details see: https://status.plaid.com/incidents/f8bndk3h48ry

Partner Issue: Delayed transaction importing for Increase accounts.
ResolvedDegraded performance1 hour 35 minutes
  • Resolved
    UTC
    Resolved

    On May 20, 2022, the issue was resolved at 10:00 AM PT.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    UTC
    Identified

    We have identified the issue and are working on a fix.

  • Investigating
    UTC
    Investigating

    Transaction importing with Increase accounts is experiencing degraded performance. We are investigating.

An automated up-time check has failed. (False Positive.)
Resolved28 minutes
  • Resolved
    UTC
    Resolved

    This incident was a false alarm.

  • Investigating
    UTC
    Investigating

    The Modern Treasury website may be experiencing degraded performance. We are investigating.

An automated up-time check has failed.
ResolvedMajor outage8 minutes
  • Resolved
    UTC
    Resolved

    The Modern Treasury application and API have recovered.

  • Investigating
    UTC
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Previous

May 2022 to Jul 2022

Next