Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

View current status

Feb 2024

FEB17
Partner Scheduled Maintenance: Wells Fargo Gateway API
CompletedMaintenance3 hours 40 minutes
  • CompletedFebruary 17, 2024 at 10:00 AMUTC
    Maintenance has completed successfully
  • In progressFebruary 17, 2024 at 6:20 AMUTC
    Maintenance is now in progress
  • PlannedFebruary 17, 2024 at 6:20 AMUTC

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Saturday, February 17th from 01:20 ET to 05:00 ET (3hrs 40m).

    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available.

    Balances and transactions will not be affected. Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Elevated Error Rates
ResolvedPartial outage12 minutes
  • ResolvedUTC

    2.5% of API calls returned 500 at 5:32 PM PST due to abnormally high database load. Load has returned to normal, and we are investigating to prevent this from happening in the future.

  • InvestigatingUTC

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

We received a false positive notification from one of our monitoring tools.
Resolved4 minutes
  • ResolvedUTC

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • InvestigatingUTC

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Jan 2024

JAN31
Scheduled Maintenance: Push to Warehouse - Source Database
CompletedMaintenance11 minutes
  • CompletedJanuary 31, 2024 at 3:11 AMUTC

    Maintenance complete.

  • In progressJanuary 31, 2024 at 3:00 AMUTC

    Maintenance is now in progress

  • PlannedJanuary 31, 2024 at 3:00 AMUTC

    We plan to upgrade the database instance where customer Push to Warehouse data resides tonight (1/30) at 7pm PT. We expect this will require a few minutes of downtime, as a result of which regularly-scheduled data transfers may be delayed by a few minutes.

Degraded Performance for 40 minutes in us001
ResolvedDegraded performance26 minutes
  • ResolvedUTC

    From 11:40am to 12:20pm PT, the us001 database was under high load due to data processing jobs.

    During this time, the p95 latency for API requests were elevated, peaking at 1.6seconds. We did not see any API errors during this time period. There were around eleven total 500s due to timeouts from our web UI in the same time period.

    We are currently making changes to our data processing jobs to tackle the root cause.

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result.

  • InvestigatingUTC

    We're experiencing an elevated level of API errors and are currently looking into the issue.

JAN17
Scheduled Maintenance: Push to Warehouse
CompletedMaintenance1 hour 40 minutes
  • CompletedJanuary 17, 2024 at 2:40 AMUTC

    Maintenance has completed successfully.

  • In progressJanuary 17, 2024 at 1:00 AMUTC

    Maintenance is now in progress

  • PlannedJanuary 17, 2024 at 1:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 1/16, from 5 to 7 PM PST.

    During this time, we plan to perform some internal housekeeping to update our Push to Warehouse product groupings. We do not expect this to impact customers — neither the tables received nor data availability.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Dec 2023

Delays in KYB processing
Resolved1 hour 19 minutes
  • ResolvedUTC

    This incident has been resolved. We have confirmed that all KYB data has been successfully processed.

  • MonitoringUTC

    We implemented a fix and are currently monitoring the result.

  • IdentifiedUTC

    We have identified the issue and are working to implement a fix now.

  • InvestigatingUTC

    We are currently investigating a delay in processing of KYB compliance data.

DEC20
Scheduled Maintenance: Push to Warehouse
CompletedMaintenance2 hours
  • CompletedDecember 20, 2023 at 3:00 AMUTC

    Maintenance has completed successfully. We opted not to make the planned change to the transaction_line_items table this time, as we observed behavior inconsistent with our staging environment. We plan to revisit this change in January.

  • In progressDecember 20, 2023 at 1:00 AMUTC

    Maintenance is now in progress

  • PlannedDecember 20, 2023 at 1:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 12/19, from 5 to 7 PM PST.

    Below is a summary of planned changes:

    • Add the following new table columns. These changes require re-syncing historical data for the affected tables. We expect this to take around 10 minutes in total, during which time data may be unavailable.
      • Add "payment_order_id" column to the "reversals" table.
      • Add "ledger_transaction_id" column to "incoming_payment_details" table.
    • Re-sync historical data for the "transaction_line_items" table. This is a one-time measure that will allow us to improve the speed and efficiency of transfers moving forward.
    • Introduce a new "ledger_account_settlements" table. As we recently announced via direct email, Modern Treasury is deprecating "ledger_account_payouts" effective 12/15/23, and we have advised customers to begin using "ledger_account_settlements" instead. The underlying data is the same. Modern Treasury will continue to sync "ledger_account_payouts" to customer warehouses through at least March 15, 2024.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of these changes.

DEC17
Scheduled Maintenance: Database Upgrades
CompletedMaintenance38 minutes
  • CompletedDecember 17, 2023 at 8:38 AMUTC

    Maintenance has completed successfully.

  • PlannedDecember 17, 2023 at 8:00 AMUTC

    Modern Treasury is planning to perform database upgrades on Sunday, December 17th at 12:00am PT.

    During this time, the platform will be inaccessible. All operations will resume immediately after the maintenance window.

We received a false positive notification from one of our monitoring tools.
Resolved9 minutes
  • ResolvedUTC

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • InvestigatingUTC

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Payment Order transmission is failing at JPMC API for Wires, EFTs, and ACHs
Resolved3 hours 29 minutes
  • ResolvedUTC

    This incident has been resolved and Payment transmission has been restored.

  • MonitoringUTC

    We have deployed the fix and are closely monitoring for any issues. We are seeing successful payment transmission.

  • IdentifiedUTC

    We have identified a recent deploy that has affected the Payment Order transmission for JPMC API Wires, EFTs, and ACHs. We are deploying the fix now.

    Payments at all other banks are unaffected. Payments for JPMC SFTP transmission is also unaffected.

Previous

Dec 2023 to Feb 2024

Next