Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

View current status

Apr 2024

Delayed Reconciliation of Transactions
Resolved5 hours 3 minutes
  • ResolvedUTC

    The incident has been resolved. By 1:28 PM PDT, all transaction reconciliation paused between 8:31 AM PDT and 12:31 PM PDT has been completed successfully.

  • UpdateUTC

    At 12:31 PM PDT, we unpaused reconciliation and are currently monitoring the result.

  • MonitoringUTC

    At around 8:31 AM PDT, we paused reconciliation due to a poorly performing database query. As a result, reconciliation-related status changes & webhooks were delayed. We implemented a fix and are currently monitoring the result.

    Please note that APIs and payment processing are not affected.

APR21
Scheduled Maintenance: CSV Bulk Imports
CompletedMaintenance1 hour
  • CompletedApril 21, 2024 at 7:00 AMUTC
    Maintenance has completed successfully
  • In progressApril 21, 2024 at 6:00 AMUTC
    Maintenance is now in progress
  • PlannedApril 21, 2024 at 6:00 AMUTC

    Modern Treasury is planning an upgrade to our CSV bulk import functionality on Saturday, April 20th at 11:00PM PDT.

    During this time, our CSV bulk import feature will be unavailable for all bulk import resources (Payment Orders, Expected Payments, Counterparties, and Invoices). Bulk imports that are still processing during the upgrade will be interrupted and fail.

    Previously completed bulk imports will not be affected. The API and other UI features will also not be affected.

Degraded performance: MT application
ResolvedDegraded performance1 hour 11 minutes
  • ResolvedUTC

    This incident has been resolved. We identified a spike in memory which impacted application performance. Application functionality has been fully restored for affected organizations in cell us001.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • InvestigatingUTC

    We are currently investigating this incident. Organizations within cell us001 may experience slower than normal loading of pages within the MT application. API performance is unaffected.

APR17
Scheduled Maintenance: Push to Warehouse
CompletedMaintenance43 minutes
  • CompletedApril 17, 2024 at 12:43 AMUTC

    Maintenance has completely successfully.

  • In progressApril 17, 2024 at 12:00 AMUTC
    Maintenance is now in progress
  • PlannedApril 17, 2024 at 12:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 4/16, from 5 to 7 PM PST.

    During this time, we plan to make the following change:

    1. Introduce the type and receiving_account_id fields to the payment_orders table, which existing recipients of Modern Treasury Payments tables will receive. The payment_orders table will be re-synced to ensure these fields are populated on historical records.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Push to Warehouse transfers are failing intermittently
ResolvedDegraded performance69 hours 10 minutes
  • ResolvedUTC

    We are seeing 100% success rates for transfers for the past 2 hours.

  • MonitoringUTC

    We have identified and resolved the root cause behind these transfer failures and are seeing a 100% transfer success rate in the past hour. We will continue to monitor data freshness and provide an update once we are certain this issue is fully resolved. 

  • UpdateUTC

    We are continuing to investigate the source of occasional transfer failures. We plan to continue to monitor data freshness and proactively initiate ad-hoc transfers to customers as needed while the issue persists.

  • InvestigatingUTC

    We are experiencing intermittent failures for PtW transfers. Around 20% of transfers appear to be impacted, across all customers. We are actively working to identify the source of the issue, and are proactively initiating transfers as needed to ensure data freshness.

Mar 2024

Intermittent errors: Routing Validation
Resolved
  • ResolvedUTC

    We are seeing routing information correctly validated for all customers.

  • MonitoringUTC

    We identified that a new caching setting led to outdated access tokens used to access our third-party routing verifier. Caching settings have been adjusted and we are now monitoring that routing information requests are valid.

  • InvestigatingUTC

    There are a subset of requests for payment endpoints experiencing intermittent errors when validating new ABA routing numbers. Our team is actively investigating these occurrences. Retry attempts are confirmed as processing successfully.

Push to Warehouse: Transfers are not occurring as scheduled for some customers
ResolvedPartial outage20 hours 31 minutes
  • ResolvedUTC

    We are seeing consistent successful transfers across all customers.

  • MonitoringUTC

    We have implemented a fix and are seeing successful transfers for all affected customers. We will continue to closely monitor transfers over the next few hours to validate this fix.

  • IdentifiedUTC

    We are aware of an issue causing delays in the delivery of data to some customer warehouse destinations. We discovered this afternoon that some of our transfer agent tasks appear to have entered an unresponsive state around 6 PM PT Thursday, blocking regularly-scheduled transfers for some customers.

    We are investigating the source of the problem and working to rectify the issue.

Elevated API Error rate
ResolvedPartial outage

  • We experienced elevated API error rates from 11:45 PM - 12:10 AM PST. We implemented a fix to reduce the error rate, and are monitoring currently.

    UTC
Push to Warehouse: Data is Delayed
Resolved8 hours 58 minutes
  • ResolvedUTC

    We have re-enabled the Push to Warehouse staging ETL process, and customer warehouse data should now be up-to-date.

    Incremental transfers for Balance Reports (introduced during yesterday's maintenance window) remain disabled at this time while we investigate the root cause of poor staging performance.

  • IdentifiedUTC

    We experienced an issue with the ETL process involved in staging customer Push to Warehouse data following our maintenance window yesterday evening, and decided to temporarily disable it to avoid impacting the availability of our web API and UI.

    As a result, changes occurring since ~11:30 PT yesterday evening are not yet reflected in customer warehouse destinations.

    We are working to re-enable the ETL process as soon as possible.

MAR20
Scheduled Maintenance: Push to Warehouse
CompletedMaintenance2 hours
  • UpdateMarch 20, 2024 at 7:52 PMUTC

    Shortly after the completion of this maintenance window we experienced an incident that impacted our ETL process. This incident has since been resolved.

    As part of the incident remediation process, transfers for the balance_reports table have been paused since ~11:30 PM PT and data may be stale. We are actively working to re-enable transfers for balance_reports and will update this maintenance update when we have a timeline estimate.

    No other tables are affected, including the newly added payment_references table.

  • CompletedMarch 20, 2024 at 2:00 AMUTC
    Maintenance has completed successfully
  • In progressMarch 20, 2024 at 12:00 AMUTC
    Maintenance is now in progress
  • PlannedMarch 20, 2024 at 12:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 3/19, from 5 to 7 PM PST.

    During this time, we plan to make the following changes:

    1. Introduce new balance_reports and payment_references tables, which existing recipients of Modern Treasury Payments tables will receive.

    2. Re-sync historical data for the transaction_line_items table. This is a one-time measure that will allow us to improve the speed and efficiency of transfers moving forward.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Feb 2024

FEB17
Partner Scheduled Maintenance: Wells Fargo Gateway API
CompletedMaintenance3 hours 40 minutes
  • CompletedFebruary 17, 2024 at 10:00 AMUTC
    Maintenance has completed successfully
  • In progressFebruary 17, 2024 at 6:20 AMUTC
    Maintenance is now in progress
  • PlannedFebruary 17, 2024 at 6:20 AMUTC

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Saturday, February 17th from 01:20 ET to 05:00 ET (3hrs 40m).

    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available.

    Balances and transactions will not be affected. Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Elevated Error Rates
ResolvedPartial outage12 minutes
  • ResolvedUTC

    2.5% of API calls returned 500 at 5:32 PM PST due to abnormally high database load. Load has returned to normal, and we are investigating to prevent this from happening in the future.

  • InvestigatingUTC

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

We received a false positive notification from one of our monitoring tools.
Resolved4 minutes
  • ResolvedUTC

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • InvestigatingUTC

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Previous

Feb 2024 to Apr 2024

Next