Modern Treasury - Notice history

Payments: Partner Bank under maintenance

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Jun 2024

Role Based Access-Control: Backend Upgrade
  • Completed
    June 06, 2024 at 10:42 PMUTC
    Completed
    June 06, 2024 at 10:42 PMUTC
    Maintenance has completed successfully.
  • In progress
    June 06, 2024 at 10:00 PMUTC
    In progress
    June 06, 2024 at 10:00 PMUTC
    Maintenance is now in progress
  • Planned
    June 06, 2024 at 10:00 PMUTC
    Planned
    June 06, 2024 at 10:00 PMUTC

    On Thursday 6/6 at 3 PM PT, we are migrating to a new backend system for the Roles Based Access-Control system. We do not anticipate any downtime, we will be actively monitoring for any access issues. There are no UI changes at this time.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this change.

Push to Warehouse: Data Store Upgrade
  • Completed
    June 05, 2024 at 12:00 AMUTC
    Completed
    June 05, 2024 at 12:00 AMUTC
    Maintenance has completed successfully
  • In progress
    June 04, 2024 at 10:00 PMUTC
    In progress
    June 04, 2024 at 10:00 PMUTC
    Maintenance is now in progress
  • Planned
    June 04, 2024 at 10:00 PMUTC
    Planned
    June 04, 2024 at 10:00 PMUTC

    On Tuesday 6/4 at 3 PM PT, we plan to migrate to a new data store for customer Push to Warehouse data. We expect regularly-scheduled transfers to be paused for ~2 hours during this process. The migration will also require a re-sync of historical data for the following tables:

    • ledger_account_balances

    • ledger_account_settlements

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this change.

May 2024

Partner Issue: Evolve Bank
  • Resolved
    UTC
    Resolved

    This issue has been resolved by our Partner. Please see their status page for further details.

  • Monitoring
    UTC
    Monitoring

    Monitoring partner updates: https://status.getevolved.com/

  • Identified
    UTC
    Identified

    Modern Treasury is actively monitoring a service disruption for one of our partners. Additional details can be located at https://status.getevolved.com/

    Impact: Missing or delayed transactions for Evolve Bank customers

Intermittent Connection Errors:API
  • Resolved
    UTC
    Resolved

    Modern Treasury has identified a network connectivity issue between two AWS regions is likely the cause of the errors a subset of customers experienced May 23rd. We’ve updated our systems to improve performance and reliability for our API services. We are seeing consistent and successful activity at this time.

  • Monitoring
    UTC
    Monitoring

    Currently monitoring as functionality appears stable.

  • Investigating
    UTC
    Investigating

    Modern Treasury is currently investigating reports of intermittent connection errors for a subset of API calls affecting US001

Scheduled Maintenance: Database Upgrades
  • Completed
    May 12, 2024 at 7:32 AMUTC
    Completed
    May 12, 2024 at 7:32 AMUTC

    Database maintenance is complete.

  • In progress
    May 12, 2024 at 6:01 AMUTC
    In progress
    May 12, 2024 at 6:01 AMUTC

    Database maintenance in cell us001 has begun. Expect intermittent API and web platform errors for the duration of the maintenance.

  • Planned
    May 12, 2024 at 6:00 AMUTC
    Planned
    May 12, 2024 at 6:00 AMUTC

    Modern Treasury is planning to perform database maintenance in cell us001 on Saturday, May 11th at 11:00PM PDT.

    During this time, access to the platform for customers in cell us001 will be inaccessible. All operations will resume immediately after the maintenance window.

Increased API Failures
  • Resolved
    UTC
    Resolved

    Between 18:11 and 18:37 we saw an increase in API error rates in us000. This mostly impacted operations related to ledgers listing. We have completed reverting the suspected database operation.

  • Monitoring
    UTC
    Monitoring

    We've identified a potential fix involving the rollback of a database optimization. Error levels are steadily decreasing and monitoring is ongoing.

  • Identified
    UTC
    Identified

    Only affecting customers in cell us000

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Push to Warehouse: Transfers are not occurring as scheduled for some customers
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    All affected customers should have fresh data as of 8:30am PT. Regularly-scheduled Transfers have resumed and are succeeding. We are continuing to carefully monitor transfer health to validate the service is fully operational.

  • Identified
    UTC
    Identified

    We have identified the workload responsible for causing transfers to become stuck and deployed a fix to mitigate the issue. We have begun triggering catch-up transfers for affected warehouse destinations.

  • Investigating
    UTC
    Investigating

    The service responsible for conducting Push to Warehouse transfers is becoming stuck in some cases, delaying scheduled transfers. Customers may observe some data less fresh than usual.

Apr 2024

Delayed Reconciliation of Transactions
  • Resolved
    UTC
    Resolved

    The incident has been resolved. By 1:28 PM PDT, all transaction reconciliation paused between 8:31 AM PDT and 12:31 PM PDT has been completed successfully.

  • Update
    UTC
    Update

    At 12:31 PM PDT, we unpaused reconciliation and are currently monitoring the result.

  • Monitoring
    UTC
    Monitoring

    At around 8:31 AM PDT, we paused reconciliation due to a poorly performing database query. As a result, reconciliation-related status changes & webhooks were delayed. We implemented a fix and are currently monitoring the result.

    Please note that APIs and payment processing are not affected.

Scheduled Maintenance: CSV Bulk Imports
  • Completed
    April 21, 2024 at 7:00 AMUTC
    Completed
    April 21, 2024 at 7:00 AMUTC
    Maintenance has completed successfully
  • In progress
    April 21, 2024 at 6:00 AMUTC
    In progress
    April 21, 2024 at 6:00 AMUTC
    Maintenance is now in progress
  • Planned
    April 21, 2024 at 6:00 AMUTC
    Planned
    April 21, 2024 at 6:00 AMUTC

    Modern Treasury is planning an upgrade to our CSV bulk import functionality on Saturday, April 20th at 11:00PM PDT.

    During this time, our CSV bulk import feature will be unavailable for all bulk import resources (Payment Orders, Expected Payments, Counterparties, and Invoices). Bulk imports that are still processing during the upgrade will be interrupted and fail.

    Previously completed bulk imports will not be affected. The API and other UI features will also not be affected.

Degraded performance: MT application
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We identified a spike in memory which impacted application performance. Application functionality has been fully restored for affected organizations in cell us001.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating

    We are currently investigating this incident. Organizations within cell us001 may experience slower than normal loading of pages within the MT application. API performance is unaffected.

Scheduled Maintenance: Push to Warehouse
  • Completed
    April 17, 2024 at 12:43 AMUTC
    Completed
    April 17, 2024 at 12:43 AMUTC

    Maintenance has completely successfully.

  • In progress
    April 17, 2024 at 12:00 AMUTC
    In progress
    April 17, 2024 at 12:00 AMUTC
    Maintenance is now in progress
  • Planned
    April 17, 2024 at 12:00 AMUTC
    Planned
    April 17, 2024 at 12:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 4/16, from 5 to 7 PM PST.

    During this time, we plan to make the following change:

    1. Introduce the type and receiving_account_id fields to the payment_orders table, which existing recipients of Modern Treasury Payments tables will receive. The payment_orders table will be re-synced to ensure these fields are populated on historical records.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Push to Warehouse transfers are failing intermittently
  • Resolved
    UTC
    Resolved

    We are seeing 100% success rates for transfers for the past 2 hours.

  • Monitoring
    UTC
    Monitoring

    We have identified and resolved the root cause behind these transfer failures and are seeing a 100% transfer success rate in the past hour. We will continue to monitor data freshness and provide an update once we are certain this issue is fully resolved. 

  • Update
    UTC
    Update

    We are continuing to investigate the source of occasional transfer failures. We plan to continue to monitor data freshness and proactively initiate ad-hoc transfers to customers as needed while the issue persists.

  • Investigating
    UTC
    Investigating

    We are experiencing intermittent failures for PtW transfers. Around 20% of transfers appear to be impacted, across all customers. We are actively working to identify the source of the issue, and are proactively initiating transfers as needed to ensure data freshness.

Apr 2024 to Jun 2024

Next