Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Feb 2024

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completed
    February 17, 2024 at 10:00 AMUTC
    Completed
    February 17, 2024 at 10:00 AMUTC
    Maintenance has completed successfully
  • In progress
    February 17, 2024 at 6:20 AMUTC
    In progress
    February 17, 2024 at 6:20 AMUTC
    Maintenance is now in progress
  • Planned
    February 17, 2024 at 6:20 AMUTC
    Planned
    February 17, 2024 at 6:20 AMUTC

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Saturday, February 17th from 01:20 ET to 05:00 ET (3hrs 40m).

    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available.

    Balances and transactions will not be affected. Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Elevated Error Rates
  • Resolved
    UTC
    Resolved

    2.5% of API calls returned 500 at 5:32 PM PST due to abnormally high database load. Load has returned to normal, and we are investigating to prevent this from happening in the future.

  • Investigating
    UTC
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

We received a false positive notification from one of our monitoring tools.
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • Investigating
    UTC
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Jan 2024

Scheduled Maintenance: Push to Warehouse - Source Database
  • Completed
    January 31, 2024 at 3:11 AMUTC
    Completed
    January 31, 2024 at 3:11 AMUTC

    Maintenance complete.

  • In progress
    January 31, 2024 at 3:00 AMUTC
    In progress
    January 31, 2024 at 3:00 AMUTC

    Maintenance is now in progress

  • Planned
    January 31, 2024 at 3:00 AMUTC
    Planned
    January 31, 2024 at 3:00 AMUTC

    We plan to upgrade the database instance where customer Push to Warehouse data resides tonight (1/30) at 7pm PT. We expect this will require a few minutes of downtime, as a result of which regularly-scheduled data transfers may be delayed by a few minutes.

Degraded Performance for 40 minutes in us001
  • Resolved
    UTC
    Resolved

    From 11:40am to 12:20pm PT, the us001 database was under high load due to data processing jobs.

    During this time, the p95 latency for API requests were elevated, peaking at 1.6seconds. We did not see any API errors during this time period. There were around eleven total 500s due to timeouts from our web UI in the same time period.

    We are currently making changes to our data processing jobs to tackle the root cause.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We're experiencing an elevated level of API errors and are currently looking into the issue.

Scheduled Maintenance: Push to Warehouse
  • Completed
    January 17, 2024 at 2:40 AMUTC
    Completed
    January 17, 2024 at 2:40 AMUTC

    Maintenance has completed successfully.

  • In progress
    January 17, 2024 at 1:00 AMUTC
    In progress
    January 17, 2024 at 1:00 AMUTC

    Maintenance is now in progress

  • Planned
    January 17, 2024 at 1:00 AMUTC
    Planned
    January 17, 2024 at 1:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 1/16, from 5 to 7 PM PST.

    During this time, we plan to perform some internal housekeeping to update our Push to Warehouse product groupings. We do not expect this to impact customers — neither the tables received nor data availability.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Dec 2023

Delays in KYB processing
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We have confirmed that all KYB data has been successfully processed.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    UTC
    Identified

    We have identified the issue and are working to implement a fix now.

  • Investigating
    UTC
    Investigating

    We are currently investigating a delay in processing of KYB compliance data.

Scheduled Maintenance: Push to Warehouse
  • Completed
    December 20, 2023 at 3:00 AMUTC
    Completed
    December 20, 2023 at 3:00 AMUTC

    Maintenance has completed successfully. We opted not to make the planned change to the transaction_line_items table this time, as we observed behavior inconsistent with our staging environment. We plan to revisit this change in January.

  • In progress
    December 20, 2023 at 1:00 AMUTC
    In progress
    December 20, 2023 at 1:00 AMUTC

    Maintenance is now in progress

  • Planned
    December 20, 2023 at 1:00 AMUTC
    Planned
    December 20, 2023 at 1:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 12/19, from 5 to 7 PM PST.

    Below is a summary of planned changes:

    • Add the following new table columns. These changes require re-syncing historical data for the affected tables. We expect this to take around 10 minutes in total, during which time data may be unavailable.
      • Add "payment_order_id" column to the "reversals" table.
      • Add "ledger_transaction_id" column to "incoming_payment_details" table.
    • Re-sync historical data for the "transaction_line_items" table. This is a one-time measure that will allow us to improve the speed and efficiency of transfers moving forward.
    • Introduce a new "ledger_account_settlements" table. As we recently announced via direct email, Modern Treasury is deprecating "ledger_account_payouts" effective 12/15/23, and we have advised customers to begin using "ledger_account_settlements" instead. The underlying data is the same. Modern Treasury will continue to sync "ledger_account_payouts" to customer warehouses through at least March 15, 2024.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of these changes.

Scheduled Maintenance: Database Upgrades
  • Completed
    December 17, 2023 at 8:38 AMUTC
    Completed
    December 17, 2023 at 8:38 AMUTC

    Maintenance has completed successfully.

  • Planned
    December 17, 2023 at 8:00 AMUTC
    Planned
    December 17, 2023 at 8:00 AMUTC

    Modern Treasury is planning to perform database upgrades on Sunday, December 17th at 12:00am PT.

    During this time, the platform will be inaccessible. All operations will resume immediately after the maintenance window.

We received a false positive notification from one of our monitoring tools.
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • Investigating
    UTC
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Payment Order transmission is failing at JPMC API for Wires, EFTs, and ACHs
  • Resolved
    UTC
    Resolved

    This incident has been resolved and Payment transmission has been restored.

  • Monitoring
    UTC
    Monitoring

    We have deployed the fix and are closely monitoring for any issues. We are seeing successful payment transmission.

  • Identified
    UTC
    Identified

    We have identified a recent deploy that has affected the Payment Order transmission for JPMC API Wires, EFTs, and ACHs. We are deploying the fix now.

    Payments at all other banks are unaffected. Payments for JPMC SFTP transmission is also unaffected.

Dec 2023 to Feb 2024

Next