Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Aug 2022

High Application and API Latencies
  • Resolved
    Resolved

    We have resolved the underlying issue at the source of the latencies.

  • Monitoring
    Monitoring

    The high latencies have subsided. API performance is returning to normal. We are continuing to monitor.

  • Investigating
    Investigating

    We are currently investigating this incident.

    The previous incidents were automatically generated from one of our monitoring tools. We apologize for the numerous emails. The check had entered a middling state where it passed and then failed several times as the latency of the responses changed.

Partial API and Application Outage
  • Monitoring
    Monitoring

    We have implemented a fix and are monitoring our system for stability. More information will be provided as it becomes available.

  • Resolved
    Resolved

    At around 10:13 PT on Aug 3, a faulty database upgrade caused downtime on our application and API. For approximately 13 minutes (until 10:26 PT), a large amount of calls against Modern Treasury’s platform failed.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Jul 2022

Elevated Error Rates
  • Resolved
    Resolved

    We saw a higher than normal level of errors due to a service disruption with some of our partners today between approximately 10:15 and 11:15 PST. After fixes were implemented on their systems, we saw this these elevated rates decrease. However, we continued to monitor our own platform for any anomalies. Finding none, we are now considering this issue resolved and service fully restored.

  • Monitoring
    Monitoring

    The service disruptions with some of our partners appears to have been resolved. Between approximately 10:15 and 11:15 PST, you may have experience increased authentication errors or slow page loads when accessing our platform. However, these should now been returning to normal levels.

    We will continue to monitor our platform for any anomalies.

  • Investigating
    Investigating

    We're seeing a higher than usual rate of errors due to a service disruption with some of our partners. You may experience delays or disruptions when accessing pages or sending requests to our API. We are currently looking into the issue and will respond with more information as it becomes available. More information on our partner's statuses are available at the links below.

    • Auth0: https://status.auth0.com/incidents/53ghn3t00tr7
Degraded Availability on APIs
  • Resolved
    Resolved

    At around 11:40 AM (PT) on July 25, a faulty change to our tracing and metrics caused extended downtime on our API. For about 23 minutes the Modern Treasury API was in a degraded state, with 100% unavailability for 13 minutes within that period. Going forward, we commit to improving our CI/CD processes to ensure that similar faults are caught earlier in our pipelines and prevented from deployment.

  • Investigating
    Investigating

    We are experiencing a high number of http status code 500s on many of the APIs. We've identified the issue and working on restoring service.

Partner Maintenance: Wells Fargo API Scheduled Maintenance
  • Completed
    July 16, 2022 at 10:00 AM
    Completed
    July 16, 2022 at 10:00 AM

    Maintenance has completed successfully

  • In progress
    July 16, 2022 at 2:00 AM
    In progress
    July 16, 2022 at 2:00 AM

    Maintenance is now in progress

  • Planned
    July 16, 2022 at 2:00 AM
    Planned
    July 16, 2022 at 2:00 AM

    The Wells Fargo API will undergo scheduled maintenance from this Friday, 2022-07-15 22:00 to Saturday, 2022-07-16 06:00 Eastern Time.

    Account balances and transactions will not update for Wells Fargo accounts during this time. RTP and ACH payments originated from Wells Fargo accounts during this time will be automatically retried when the API is back online.

    Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Elevated Error rates
  • Resolved
    Resolved

    Incident has been resolved

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Investigating
    Investigating

    We're experiencing an elevated level of errors and are currently looking into the issue.

Temporary Degraded Availability
  • Resolved
    Resolved

    We experienced a brief period of degraded availability between approximately 9:10 and 9:50 PST. You may have experienced a higher than usual rate of 503 responses during this period.

  • Investigating
    Investigating

    We're experiencing an elevated level of errors and are currently looking into the issue.

Jun 2022

ACH payments effective June 21st, 2022 delayed
  • Resolved
    Resolved

    Payments that were delayed were successfully transmitted to the appropriate payment networks and will are undergoing standard reconciliation processes. Delayed payments have had their effective date field updated to the appropriate date.

  • Monitoring
    Monitoring

    Due to a processing error, ACH payments effective for June 21st, 2022 may be delayed by one day.

    The addition of the observed Federal holiday for June 19th resulted in ACH payments that were not transmitted to the ACH network on the appropriate date. Any delayed ACH payments will be automatically transmitted and updated for a June 22nd effective date.

Temporary Degraded Availability
  • Resolved
    Resolved

    By 9:15am PST, the app fully recovered.

  • Monitoring
    Monitoring

    Due to unexpectedly high request volume this morning, we experienced a brief period of degraded availability between approximately 9:10am and 9:15am PST. You may have experienced a higher than usual rate of 503 responses during this period. Going forward, we will continue to invest in the scalability of our systems to handle higher request volumes and support the growth of our customers’ businesses.

Temporary Degraded Availability
  • Resolved
    Resolved

    By 9:10am PST, the app fully recovered.

  • Monitoring
    Monitoring

    Due to unexpectedly high request volume this morning, we experienced a brief period of degraded availability between approximately 9:05am and 9:10am PST. You may have experienced a higher than usual rate of 503 responses during this period. Going forward, we will continue to invest in the scalability of our systems to handle higher request volumes and support the growth of our customers’ businesses.

Elevated Error Rates
  • Update
    Update

    We have determined the cause to be related to one of our scaling systems. The issue triggered an automatic restart causing a short period of unavailability, however we are working on a long-term fix to prevent this from happening in the future.

  • Resolved
    Resolved

    The Modern Treasury API experienced a brief period of unavailability between approximately 9:10am and 9:20am PST and has since recovered. We are investigating.

  • Monitoring
    Monitoring

    The Modern Treasury API experienced a brief period of unavailability between approximately 9:10am and 9:20am PST and has since recovered. We are investigating.

Partner Issue: Plaid is experiencing API Downtime impacting Payment Order and External Account Creation
  • Resolved
    Resolved

    The Modern Treasury application and API have recovered. For more details see: https://status.plaid.com/incidents/82fmcmd3d42x

  • Identified
    Identified

    For more details see: https://status.plaid.com/incidents/82fmcmd3d42x

Jun 2022 to Aug 2022

Next