Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Jan 2024

Scheduled Maintenance: Push to Warehouse - Source Database
  • Completed
    January 31, 2024 at 3:11 AMUTC
    Completed
    January 31, 2024 at 3:11 AMUTC

    Maintenance complete.

  • In progress
    January 31, 2024 at 3:00 AMUTC
    In progress
    January 31, 2024 at 3:00 AMUTC

    Maintenance is now in progress

  • Planned
    January 31, 2024 at 3:00 AMUTC
    Planned
    January 31, 2024 at 3:00 AMUTC

    We plan to upgrade the database instance where customer Push to Warehouse data resides tonight (1/30) at 7pm PT. We expect this will require a few minutes of downtime, as a result of which regularly-scheduled data transfers may be delayed by a few minutes.

Degraded Performance for 40 minutes in us001
  • Resolved
    UTC
    Resolved

    From 11:40am to 12:20pm PT, the us001 database was under high load due to data processing jobs.

    During this time, the p95 latency for API requests were elevated, peaking at 1.6seconds. We did not see any API errors during this time period. There were around eleven total 500s due to timeouts from our web UI in the same time period.

    We are currently making changes to our data processing jobs to tackle the root cause.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We're experiencing an elevated level of API errors and are currently looking into the issue.

Scheduled Maintenance: Push to Warehouse
  • Completed
    January 17, 2024 at 2:40 AMUTC
    Completed
    January 17, 2024 at 2:40 AMUTC

    Maintenance has completed successfully.

  • In progress
    January 17, 2024 at 1:00 AMUTC
    In progress
    January 17, 2024 at 1:00 AMUTC

    Maintenance is now in progress

  • Planned
    January 17, 2024 at 1:00 AMUTC
    Planned
    January 17, 2024 at 1:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 1/16, from 5 to 7 PM PST.

    During this time, we plan to perform some internal housekeeping to update our Push to Warehouse product groupings. We do not expect this to impact customers — neither the tables received nor data availability.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Dec 2023

Delays in KYB processing
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We have confirmed that all KYB data has been successfully processed.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    UTC
    Identified

    We have identified the issue and are working to implement a fix now.

  • Investigating
    UTC
    Investigating

    We are currently investigating a delay in processing of KYB compliance data.

Scheduled Maintenance: Push to Warehouse
  • Completed
    December 20, 2023 at 3:00 AMUTC
    Completed
    December 20, 2023 at 3:00 AMUTC

    Maintenance has completed successfully. We opted not to make the planned change to the transaction_line_items table this time, as we observed behavior inconsistent with our staging environment. We plan to revisit this change in January.

  • In progress
    December 20, 2023 at 1:00 AMUTC
    In progress
    December 20, 2023 at 1:00 AMUTC

    Maintenance is now in progress

  • Planned
    December 20, 2023 at 1:00 AMUTC
    Planned
    December 20, 2023 at 1:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 12/19, from 5 to 7 PM PST.

    Below is a summary of planned changes:

    • Add the following new table columns. These changes require re-syncing historical data for the affected tables. We expect this to take around 10 minutes in total, during which time data may be unavailable.
      • Add "payment_order_id" column to the "reversals" table.
      • Add "ledger_transaction_id" column to "incoming_payment_details" table.
    • Re-sync historical data for the "transaction_line_items" table. This is a one-time measure that will allow us to improve the speed and efficiency of transfers moving forward.
    • Introduce a new "ledger_account_settlements" table. As we recently announced via direct email, Modern Treasury is deprecating "ledger_account_payouts" effective 12/15/23, and we have advised customers to begin using "ledger_account_settlements" instead. The underlying data is the same. Modern Treasury will continue to sync "ledger_account_payouts" to customer warehouses through at least March 15, 2024.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of these changes.

Scheduled Maintenance: Database Upgrades
  • Completed
    December 17, 2023 at 8:38 AMUTC
    Completed
    December 17, 2023 at 8:38 AMUTC

    Maintenance has completed successfully.

  • Planned
    December 17, 2023 at 8:00 AMUTC
    Planned
    December 17, 2023 at 8:00 AMUTC

    Modern Treasury is planning to perform database upgrades on Sunday, December 17th at 12:00am PT.

    During this time, the platform will be inaccessible. All operations will resume immediately after the maintenance window.

We received a false positive notification from one of our monitoring tools.
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • Investigating
    UTC
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Payment Order transmission is failing at JPMC API for Wires, EFTs, and ACHs
  • Resolved
    UTC
    Resolved

    This incident has been resolved and Payment transmission has been restored.

  • Monitoring
    UTC
    Monitoring

    We have deployed the fix and are closely monitoring for any issues. We are seeing successful payment transmission.

  • Identified
    UTC
    Identified

    We have identified a recent deploy that has affected the Payment Order transmission for JPMC API Wires, EFTs, and ACHs. We are deploying the fix now.

    Payments at all other banks are unaffected. Payments for JPMC SFTP transmission is also unaffected.

Nov 2023

Elevated Error Rates in us0
  • Resolved
    UTC
    Resolved

    Starting from 11/27 9:07 PM ledgers api experienced elevated API timeout errors which affected a single customer in cell us000. This issue was caused by outdated statistics of specific tables in the database that resulted in incorrect query plans. By 11/29 1:21 PM, we had updated and fixed the statistics and query plan and service had returned to normal. The issue has been resolved and the service is operating normally.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and will monitor the results for a few hours.

  • Identified
    UTC
    Identified

    We're experiencing an elevated level of API timeout errors for a single customer in one cell and are currently working on a fix to mitigate the database-related issue. We will update in 30 minutes.

Scheduled Maintenance: Database Upgrades on us0, us5
  • Completed
    November 28, 2023 at 2:46 AMUTC
    Completed
    November 28, 2023 at 2:46 AMUTC

    Maintenance has completed successfully.

  • In progress
    November 28, 2023 at 2:30 AMUTC
    In progress
    November 28, 2023 at 2:30 AMUTC

    Maintenance is now in progress

  • Planned
    November 28, 2023 at 2:30 AMUTC
    Planned
    November 28, 2023 at 2:30 AMUTC

    Modern Treasury is planning to perform database upgrades on our us0 and us5 databases on Monday, November 27th at 6:30pm PT. We expect that this operation will affect only the customers in these cells, and will have minimal impact to our API and app.

    Most customers will not be affected. You can use the Organization Settings page to determine the cell that your organization is in.

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completed
    November 26, 2023 at 7:00 PMUTC
    Completed
    November 26, 2023 at 7:00 PMUTC

    Maintenance has completed successfully

  • In progress
    November 26, 2023 at 5:00 AMUTC
    In progress
    November 26, 2023 at 5:00 AMUTC

    Maintenance is now in progress

  • Planned
    November 26, 2023 at 5:00 AMUTC
    Planned
    November 26, 2023 at 5:00 AMUTC

    [Planned]​ Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on ​Sunday, November 26th​ from ​12:00 AM ET​ to ​02:00 PM ET​ (14 hrs).
    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available. Balances and transactions will not be affected.

    Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Scheduled Maintenance: Push to Warehouse
  • Completed
    November 22, 2023 at 3:27 AMUTC
    Completed
    November 22, 2023 at 3:27 AMUTC

    Maintenance is complete for all destination types except Databricks, where we're continuing to work on remediating an issue that is preventing incremental transfers of payment_orders, returns, and expected_payments.

    We are concluding our maintenance window at this time. We will follow up on resolution of the outstanding issue with Databricks customers.

  • In progress
    November 22, 2023 at 3:00 AMUTC
    In progress
    November 22, 2023 at 3:00 AMUTC

    Maintenance is now in progress

  • Planned
    November 22, 2023 at 1:00 AMUTC
    Planned
    November 22, 2023 at 1:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 11/21, from 5 to 7 PM PST.

    Below is a summary of planned changes:

    • Add the following new table columns. With the exception of the "ledger_entries" change, these will be applied by re-syncing all historical table data. We expect this to take around 10 minutes in total, during which time tables may be unavailable.
      • Add "effective_at" column to the "ledger_entries" table.
      • Add "discarded_at" column to the "returns" table.
      • Add "ledger_transaction_id" column to "payment_orders", "expected_payments", and "returns".
    • Drop the "paper_items" table. Paper Items are deprecated and unused by Push to Warehouse customers.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of these changes.

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completed
    November 19, 2023 at 11:00 AMUTC
    Completed
    November 19, 2023 at 11:00 AMUTC

    Maintenance has completed successfully

  • In progress
    November 19, 2023 at 7:00 AMUTC
    In progress
    November 19, 2023 at 7:00 AMUTC

    Maintenance is now in progress

  • Planned
    November 19, 2023 at 7:00 AMUTC
    Planned
    November 19, 2023 at 7:00 AMUTC

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Sunday, November 19th from 02:00 ET to 06:00 ET (4hrs).

    The following Wells Fargo APIs will be unavailable:

    • ACH Payments
    • Instant Payments
    • RTP Payments
    • Wire Payments

    ACH, RTP, and Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available. Balances and transactions will not be affected.

    Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Nov 2023 to Jan 2024

Next