All systems operational

Platform/API

Operational

Ledgers API

Operational

Compliance API

Operational

Push to Warehouse

Operational

Marketing Website

Operational

Notice history

Apr 2024

Scheduled Maintenance: Push to Warehouse
Scheduled for April 17, 2024 at 12:00 AM – 2:00 AM about 2 hours
  • Planned
    April 17, 2024 at 12:00 AM
    Planned
    April 17, 2024 at 12:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 4/16, from 5 to 7 PM PST.

    During this time, we plan to make the following change:

    1. Introduce the type and receiving_account_id fields to the payment_orders table, which existing recipients of Modern Treasury Payments tables will receive. The payment_orders table will be re-synced to ensure these fields are populated on historical records.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Increased API Error rate
  • Resolved
    Resolved

    We experienced a 2.5% error rate on API calls from 8:28 AM PST - 8:37 AM.

  • Monitoring
    Monitoring

    We are no longer seeing API errors and have identified and implemented a fix. We are currently monitoring the results.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

False positive from monitoring tool
  • Resolved
    Resolved

    We have determined a monitoring tool reported a false positive and API availability was unaffected during this time.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Push to Warehouse: Transfers are not occurring as scheduled for some customers
  • Resolved
    Resolved

    Scheduled transfers have resumed to all destinations impacted by this issue.

  • Monitoring
    Monitoring

    We confirmed that some worker tasks responsible for conducting transfers entered an degraded state where they were unable to process new transfers after losing connectivity to our replica database yesterday evening.

    We have cycled the impacted workers and are monitoring to confirm that transfers resume completing on schedule for all impacted customers.

  • Identified
    Identified

    Following an earlier upgrade of the database instance from which customer Push to Warehouse data is sourced, transfers to some destinations are not occurring as scheduled.

    We are working to confirm the source of the problem, and will share more information as it becomes available.

Scheduled Maintenance: Database Upgrades
  • Completed
    April 07, 2024 at 7:02 AM
    Completed
    April 07, 2024 at 7:02 AM

    Database maintenance is complete.

  • In progress
    April 07, 2024 at 6:03 AM
    In progress
    April 07, 2024 at 6:03 AM

    Modern Treasury is performing database upgrades as part of routine maintenance.

    During this time, access to the platform will be inaccessible. All operations will resume immediately after the maintenance window.

  • Planned
    April 07, 2024 at 6:00 AM
    Planned
    April 07, 2024 at 6:00 AM

    Modern Treasury is planning to perform database upgrades on Saturday, April 6th at 11:00PM PDT.

    During this time, access to the platform will be inaccessible. All operations will resume immediately after the maintenance window.

Mar 2024

Intermittent errors: Routing Validation
  • Resolved
    Resolved

    We are seeing routing information correctly validated for all customers.

  • Monitoring
    Monitoring

    We identified that a new caching setting led to outdated access tokens used to access our third-party routing verifier. Caching settings have been adjusted and we are now monitoring that routing information requests are valid.

  • Investigating
    Investigating

    There are a subset of requests for payment endpoints experiencing intermittent errors when validating new ABA routing numbers. Our team is actively investigating these occurrences. Retry attempts are confirmed as processing successfully.

Push to Warehouse: Transfers are not occurring as scheduled for some customers
  • Resolved
    Resolved

    We are seeing consistent successful transfers across all customers.

  • Monitoring
    Monitoring

    We have implemented a fix and are seeing successful transfers for all affected customers. We will continue to closely monitor transfers over the next few hours to validate this fix.

  • Identified
    Identified

    We are aware of an issue causing delays in the delivery of data to some customer warehouse destinations. We discovered this afternoon that some of our transfer agent tasks appear to have entered an unresponsive state around 6 PM PT Thursday, blocking regularly-scheduled transfers for some customers.

    We are investigating the source of the problem and working to rectify the issue.

Elevated API Error rate
  • Resolved
    Resolved

    We experienced elevated API error rates from 11:45 PM - 12:10 AM PST. We implemented a fix to reduce the error rate, and are monitoring currently.

Push to Warehouse: Data is Delayed
  • Resolved
    Resolved

    We have re-enabled the Push to Warehouse staging ETL process, and customer warehouse data should now be up-to-date.

    Incremental transfers for Balance Reports (introduced during yesterday's maintenance window) remain disabled at this time while we investigate the root cause of poor staging performance.

  • Identified
    Identified

    We experienced an issue with the ETL process involved in staging customer Push to Warehouse data following our maintenance window yesterday evening, and decided to temporarily disable it to avoid impacting the availability of our web API and UI.

    As a result, changes occurring since ~11:30 PT yesterday evening are not yet reflected in customer warehouse destinations.

    We are working to re-enable the ETL process as soon as possible.

Scheduled Maintenance: Push to Warehouse
  • Update
    March 20, 2024 at 7:52 PM
    Completed
    March 20, 2024 at 7:52 PM

    Shortly after the completion of this maintenance window we experienced an incident that impacted our ETL process. This incident has since been resolved.

    As part of the incident remediation process, transfers for the balance_reports table have been paused since ~11:30 PM PT and data may be stale. We are actively working to re-enable transfers for balance_reports and will update this maintenance update when we have a timeline estimate.

    No other tables are affected, including the newly added payment_references table.

  • Completed
    March 20, 2024 at 2:00 AM
    Completed
    March 20, 2024 at 2:00 AM
    Maintenance has completed successfully
  • In progress
    March 20, 2024 at 12:00 AM
    In progress
    March 20, 2024 at 12:00 AM
    Maintenance is now in progress
  • Planned
    March 20, 2024 at 12:00 AM
    Planned
    March 20, 2024 at 12:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 3/19, from 5 to 7 PM PST.

    During this time, we plan to make the following changes:

    1. Introduce new balance_reports and payment_references tables, which existing recipients of Modern Treasury Payments tables will receive.

    2. Re-sync historical data for the transaction_line_items table. This is a one-time measure that will allow us to improve the speed and efficiency of transfers moving forward.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Feb 2024

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completed
    February 17, 2024 at 10:00 AM
    Completed
    February 17, 2024 at 10:00 AM
    Maintenance has completed successfully
  • In progress
    February 17, 2024 at 6:20 AM
    In progress
    February 17, 2024 at 6:20 AM
    Maintenance is now in progress
  • Planned
    February 17, 2024 at 6:20 AM
    Planned
    February 17, 2024 at 6:20 AM

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Saturday, February 17th from 01:20 ET to 05:00 ET (3hrs 40m).

    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available.

    Balances and transactions will not be affected. Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Elevated Error Rates
  • Resolved
    Resolved

    2.5% of API calls returned 500 at 5:32 PM PST due to abnormally high database load. Load has returned to normal, and we are investigating to prevent this from happening in the future.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

We received a false positive notification from one of our monitoring tools.
  • Resolved
    Resolved

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Feb 2024 to Apr 2024