Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Dec 2023

Delays in KYB processing
  • Resolved
    Resolved

    This incident has been resolved. We have confirmed that all KYB data has been successfully processed.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    We have identified the issue and are working to implement a fix now.

  • Investigating
    Investigating

    We are currently investigating a delay in processing of KYB compliance data.

Scheduled Maintenance: Push to Warehouse
  • Completed
    December 20, 2023 at 3:00 AM
    Completed
    December 20, 2023 at 3:00 AM

    Maintenance has completed successfully. We opted not to make the planned change to the transaction_line_items table this time, as we observed behavior inconsistent with our staging environment. We plan to revisit this change in January.

  • In progress
    December 20, 2023 at 1:00 AM
    In progress
    December 20, 2023 at 1:00 AM

    Maintenance is now in progress

  • Planned
    December 20, 2023 at 1:00 AM
    Planned
    December 20, 2023 at 1:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 12/19, from 5 to 7 PM PST.

    Below is a summary of planned changes:

    • Add the following new table columns. These changes require re-syncing historical data for the affected tables. We expect this to take around 10 minutes in total, during which time data may be unavailable.
      • Add "payment_order_id" column to the "reversals" table.
      • Add "ledger_transaction_id" column to "incoming_payment_details" table.
    • Re-sync historical data for the "transaction_line_items" table. This is a one-time measure that will allow us to improve the speed and efficiency of transfers moving forward.
    • Introduce a new "ledger_account_settlements" table. As we recently announced via direct email, Modern Treasury is deprecating "ledger_account_payouts" effective 12/15/23, and we have advised customers to begin using "ledger_account_settlements" instead. The underlying data is the same. Modern Treasury will continue to sync "ledger_account_payouts" to customer warehouses through at least March 15, 2024.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of these changes.

Scheduled Maintenance: Database Upgrades
  • Completed
    December 17, 2023 at 8:38 AM
    Completed
    December 17, 2023 at 8:38 AM

    Maintenance has completed successfully.

  • Planned
    December 17, 2023 at 8:00 AM
    Planned
    December 17, 2023 at 8:00 AM

    Modern Treasury is planning to perform database upgrades on Sunday, December 17th at 12:00am PT.

    During this time, the platform will be inaccessible. All operations will resume immediately after the maintenance window.

We received a false positive notification from one of our monitoring tools.
  • Resolved
    Resolved

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Payment Order transmission is failing at JPMC API for Wires, EFTs, and ACHs
  • Resolved
    Resolved

    This incident has been resolved and Payment transmission has been restored.

  • Monitoring
    Monitoring

    We have deployed the fix and are closely monitoring for any issues. We are seeing successful payment transmission.

  • Identified
    Identified

    We have identified a recent deploy that has affected the Payment Order transmission for JPMC API Wires, EFTs, and ACHs. We are deploying the fix now.

    Payments at all other banks are unaffected. Payments for JPMC SFTP transmission is also unaffected.

Nov 2023

Elevated Error Rates in us0
  • Resolved
    Resolved

    Starting from 11/27 9:07 PM ledgers api experienced elevated API timeout errors which affected a single customer in cell us000. This issue was caused by outdated statistics of specific tables in the database that resulted in incorrect query plans. By 11/29 1:21 PM, we had updated and fixed the statistics and query plan and service had returned to normal. The issue has been resolved and the service is operating normally.

  • Monitoring
    Monitoring

    We implemented a fix and will monitor the results for a few hours.

  • Identified
    Identified

    We're experiencing an elevated level of API timeout errors for a single customer in one cell and are currently working on a fix to mitigate the database-related issue. We will update in 30 minutes.

Scheduled Maintenance: Database Upgrades on us0, us5
  • Completed
    November 28, 2023 at 2:46 AM
    Completed
    November 28, 2023 at 2:46 AM

    Maintenance has completed successfully.

  • In progress
    November 28, 2023 at 2:30 AM
    In progress
    November 28, 2023 at 2:30 AM

    Maintenance is now in progress

  • Planned
    November 28, 2023 at 2:30 AM
    Planned
    November 28, 2023 at 2:30 AM

    Modern Treasury is planning to perform database upgrades on our us0 and us5 databases on Monday, November 27th at 6:30pm PT. We expect that this operation will affect only the customers in these cells, and will have minimal impact to our API and app.

    Most customers will not be affected. You can use the Organization Settings page to determine the cell that your organization is in.

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completed
    November 26, 2023 at 7:00 PM
    Completed
    November 26, 2023 at 7:00 PM

    Maintenance has completed successfully

  • In progress
    November 26, 2023 at 5:00 AM
    In progress
    November 26, 2023 at 5:00 AM

    Maintenance is now in progress

  • Planned
    November 26, 2023 at 5:00 AM
    Planned
    November 26, 2023 at 5:00 AM

    [Planned]​ Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on ​Sunday, November 26th​ from ​12:00 AM ET​ to ​02:00 PM ET​ (14 hrs).
    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available. Balances and transactions will not be affected.

    Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Scheduled Maintenance: Push to Warehouse
  • Completed
    November 22, 2023 at 3:27 AM
    Completed
    November 22, 2023 at 3:27 AM

    Maintenance is complete for all destination types except Databricks, where we're continuing to work on remediating an issue that is preventing incremental transfers of payment_orders, returns, and expected_payments.

    We are concluding our maintenance window at this time. We will follow up on resolution of the outstanding issue with Databricks customers.

  • In progress
    November 22, 2023 at 3:00 AM
    In progress
    November 22, 2023 at 3:00 AM

    Maintenance is now in progress

  • Planned
    November 22, 2023 at 1:00 AM
    Planned
    November 22, 2023 at 1:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 11/21, from 5 to 7 PM PST.

    Below is a summary of planned changes:

    • Add the following new table columns. With the exception of the "ledger_entries" change, these will be applied by re-syncing all historical table data. We expect this to take around 10 minutes in total, during which time tables may be unavailable.
      • Add "effective_at" column to the "ledger_entries" table.
      • Add "discarded_at" column to the "returns" table.
      • Add "ledger_transaction_id" column to "payment_orders", "expected_payments", and "returns".
    • Drop the "paper_items" table. Paper Items are deprecated and unused by Push to Warehouse customers.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of these changes.

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completed
    November 19, 2023 at 11:00 AM
    Completed
    November 19, 2023 at 11:00 AM

    Maintenance has completed successfully

  • In progress
    November 19, 2023 at 7:00 AM
    In progress
    November 19, 2023 at 7:00 AM

    Maintenance is now in progress

  • Planned
    November 19, 2023 at 7:00 AM
    Planned
    November 19, 2023 at 7:00 AM

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Sunday, November 19th from 02:00 ET to 06:00 ET (4hrs).

    The following Wells Fargo APIs will be unavailable:

    • ACH Payments
    • Instant Payments
    • RTP Payments
    • Wire Payments

    ACH, RTP, and Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available. Balances and transactions will not be affected.

    Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Oct 2023

Intermittent authentication errors
  • Resolved
    Resolved

    auth0 has confirmed the incident is resolved. The Modern Treasury web app is operational.

  • Update
    Update

    auth0 has confirmed an issue causing increased error rates. We are continuing to monitor the situation, and will update every 60 minutes or as we have new information.

  • Monitoring
    Monitoring

    We have confirmed a connectivity issue with auth0. We are continuing to monitor the situation, and will update every 30 minutes or as we have new information.

  • Investigating
    Investigating

    We are seeing intermittent issues with authentication to the web app. The APIs are still operational. We are currently investigating.

Oct 2023 to Dec 2023

Next