Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Apr 2024

Delayed Reconciliation of Transactions
  • Resolved
    Resolved

    The incident has been resolved. By 1:28 PM PDT, all transaction reconciliation paused between 8:31 AM PDT and 12:31 PM PDT has been completed successfully.

  • Update
    Update

    At 12:31 PM PDT, we unpaused reconciliation and are currently monitoring the result.

  • Monitoring
    Monitoring

    At around 8:31 AM PDT, we paused reconciliation due to a poorly performing database query. As a result, reconciliation-related status changes & webhooks were delayed. We implemented a fix and are currently monitoring the result.

    Please note that APIs and payment processing are not affected.

Scheduled Maintenance: CSV Bulk Imports
  • Completed
    April 21, 2024 at 7:00 AM
    Completed
    April 21, 2024 at 7:00 AM
    Maintenance has completed successfully
  • In progress
    April 21, 2024 at 6:00 AM
    In progress
    April 21, 2024 at 6:00 AM
    Maintenance is now in progress
  • Planned
    April 21, 2024 at 6:00 AM
    Planned
    April 21, 2024 at 6:00 AM

    Modern Treasury is planning an upgrade to our CSV bulk import functionality on Saturday, April 20th at 11:00PM PDT.

    During this time, our CSV bulk import feature will be unavailable for all bulk import resources (Payment Orders, Expected Payments, Counterparties, and Invoices). Bulk imports that are still processing during the upgrade will be interrupted and fail.

    Previously completed bulk imports will not be affected. The API and other UI features will also not be affected.

Degraded performance: MT application
  • Resolved
    Resolved

    This incident has been resolved. We identified a spike in memory which impacted application performance. Application functionality has been fully restored for affected organizations in cell us001.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating

    We are currently investigating this incident. Organizations within cell us001 may experience slower than normal loading of pages within the MT application. API performance is unaffected.

Scheduled Maintenance: Push to Warehouse
  • Completed
    April 17, 2024 at 12:43 AM
    Completed
    April 17, 2024 at 12:43 AM

    Maintenance has completely successfully.

  • In progress
    April 17, 2024 at 12:00 AM
    In progress
    April 17, 2024 at 12:00 AM
    Maintenance is now in progress
  • Planned
    April 17, 2024 at 12:00 AM
    Planned
    April 17, 2024 at 12:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 4/16, from 5 to 7 PM PST.

    During this time, we plan to make the following change:

    1. Introduce the type and receiving_account_id fields to the payment_orders table, which existing recipients of Modern Treasury Payments tables will receive. The payment_orders table will be re-synced to ensure these fields are populated on historical records.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Push to Warehouse transfers are failing intermittently
  • Resolved
    Resolved

    We are seeing 100% success rates for transfers for the past 2 hours.

  • Monitoring
    Monitoring

    We have identified and resolved the root cause behind these transfer failures and are seeing a 100% transfer success rate in the past hour. We will continue to monitor data freshness and provide an update once we are certain this issue is fully resolved. 

  • Update
    Update

    We are continuing to investigate the source of occasional transfer failures. We plan to continue to monitor data freshness and proactively initiate ad-hoc transfers to customers as needed while the issue persists.

  • Investigating
    Investigating

    We are experiencing intermittent failures for PtW transfers. Around 20% of transfers appear to be impacted, across all customers. We are actively working to identify the source of the issue, and are proactively initiating transfers as needed to ensure data freshness.

Mar 2024

Intermittent errors: Routing Validation
  • Resolved
    Resolved

    We are seeing routing information correctly validated for all customers.

  • Monitoring
    Monitoring

    We identified that a new caching setting led to outdated access tokens used to access our third-party routing verifier. Caching settings have been adjusted and we are now monitoring that routing information requests are valid.

  • Investigating
    Investigating

    There are a subset of requests for payment endpoints experiencing intermittent errors when validating new ABA routing numbers. Our team is actively investigating these occurrences. Retry attempts are confirmed as processing successfully.

Push to Warehouse: Transfers are not occurring as scheduled for some customers
  • Resolved
    Resolved

    We are seeing consistent successful transfers across all customers.

  • Monitoring
    Monitoring

    We have implemented a fix and are seeing successful transfers for all affected customers. We will continue to closely monitor transfers over the next few hours to validate this fix.

  • Identified
    Identified

    We are aware of an issue causing delays in the delivery of data to some customer warehouse destinations. We discovered this afternoon that some of our transfer agent tasks appear to have entered an unresponsive state around 6 PM PT Thursday, blocking regularly-scheduled transfers for some customers.

    We are investigating the source of the problem and working to rectify the issue.

Elevated API Error rate
  • Resolved
    Resolved

    We experienced elevated API error rates from 11:45 PM - 12:10 AM PST. We implemented a fix to reduce the error rate, and are monitoring currently.

Push to Warehouse: Data is Delayed
  • Resolved
    Resolved

    We have re-enabled the Push to Warehouse staging ETL process, and customer warehouse data should now be up-to-date.

    Incremental transfers for Balance Reports (introduced during yesterday's maintenance window) remain disabled at this time while we investigate the root cause of poor staging performance.

  • Identified
    Identified

    We experienced an issue with the ETL process involved in staging customer Push to Warehouse data following our maintenance window yesterday evening, and decided to temporarily disable it to avoid impacting the availability of our web API and UI.

    As a result, changes occurring since ~11:30 PT yesterday evening are not yet reflected in customer warehouse destinations.

    We are working to re-enable the ETL process as soon as possible.

Scheduled Maintenance: Push to Warehouse
  • Update
    March 20, 2024 at 7:52 PM
    Completed
    March 20, 2024 at 7:52 PM

    Shortly after the completion of this maintenance window we experienced an incident that impacted our ETL process. This incident has since been resolved.

    As part of the incident remediation process, transfers for the balance_reports table have been paused since ~11:30 PM PT and data may be stale. We are actively working to re-enable transfers for balance_reports and will update this maintenance update when we have a timeline estimate.

    No other tables are affected, including the newly added payment_references table.

  • Completed
    March 20, 2024 at 2:00 AM
    Completed
    March 20, 2024 at 2:00 AM
    Maintenance has completed successfully
  • In progress
    March 20, 2024 at 12:00 AM
    In progress
    March 20, 2024 at 12:00 AM
    Maintenance is now in progress
  • Planned
    March 20, 2024 at 12:00 AM
    Planned
    March 20, 2024 at 12:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 3/19, from 5 to 7 PM PST.

    During this time, we plan to make the following changes:

    1. Introduce new balance_reports and payment_references tables, which existing recipients of Modern Treasury Payments tables will receive.

    2. Re-sync historical data for the transaction_line_items table. This is a one-time measure that will allow us to improve the speed and efficiency of transfers moving forward.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Feb 2024

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completed
    February 17, 2024 at 10:00 AM
    Completed
    February 17, 2024 at 10:00 AM
    Maintenance has completed successfully
  • In progress
    February 17, 2024 at 6:20 AM
    In progress
    February 17, 2024 at 6:20 AM
    Maintenance is now in progress
  • Planned
    February 17, 2024 at 6:20 AM
    Planned
    February 17, 2024 at 6:20 AM

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Saturday, February 17th from 01:20 ET to 05:00 ET (3hrs 40m).

    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available.

    Balances and transactions will not be affected. Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Elevated Error Rates
  • Resolved
    Resolved

    2.5% of API calls returned 500 at 5:32 PM PST due to abnormally high database load. Load has returned to normal, and we are investigating to prevent this from happening in the future.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

We received a false positive notification from one of our monitoring tools.
  • Resolved
    Resolved

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Feb 2024 to Apr 2024

Next