Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Nov 2023

Elevated Error Rates in us0
  • Resolved
    Resolved

    Starting from 11/27 9:07 PM ledgers api experienced elevated API timeout errors which affected a single customer in cell us000. This issue was caused by outdated statistics of specific tables in the database that resulted in incorrect query plans. By 11/29 1:21 PM, we had updated and fixed the statistics and query plan and service had returned to normal. The issue has been resolved and the service is operating normally.

  • Monitoring
    Monitoring

    We implemented a fix and will monitor the results for a few hours.

  • Identified
    Identified

    We're experiencing an elevated level of API timeout errors for a single customer in one cell and are currently working on a fix to mitigate the database-related issue. We will update in 30 minutes.

Scheduled Maintenance: Database Upgrades on us0, us5
  • Completedsafojaskfaokf
    November 28, 2023 at 2:46 AM
    Completed
    November 28, 2023 at 2:46 AM

    Maintenance has completed successfully.

  • In progresssafojaskfaokf
    November 28, 2023 at 2:30 AM
    In progress
    November 28, 2023 at 2:30 AM

    Maintenance is now in progress

  • Plannedsafojaskfaokf
    November 28, 2023 at 2:30 AM
    Planned
    November 28, 2023 at 2:30 AM

    Modern Treasury is planning to perform database upgrades on our us0 and us5 databases on Monday, November 27th at 6:30pm PT. We expect that this operation will affect only the customers in these cells, and will have minimal impact to our API and app.

    Most customers will not be affected. You can use the Organization Settings page to determine the cell that your organization is in.

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completedsafojaskfaokf
    November 26, 2023 at 7:00 PM
    Completed
    November 26, 2023 at 7:00 PM

    Maintenance has completed successfully

  • In progresssafojaskfaokf
    November 26, 2023 at 5:00 AM
    In progress
    November 26, 2023 at 5:00 AM

    Maintenance is now in progress

  • Plannedsafojaskfaokf
    November 26, 2023 at 5:00 AM
    Planned
    November 26, 2023 at 5:00 AM

    [Planned]​ Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on ​Sunday, November 26th​ from ​12:00 AM ET​ to ​02:00 PM ET​ (14 hrs).
    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available. Balances and transactions will not be affected.

    Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Scheduled Maintenance: Push to Warehouse
  • Completedsafojaskfaokf
    November 22, 2023 at 3:27 AM
    Completed
    November 22, 2023 at 3:27 AM

    Maintenance is complete for all destination types except Databricks, where we're continuing to work on remediating an issue that is preventing incremental transfers of payment_orders, returns, and expected_payments.

    We are concluding our maintenance window at this time. We will follow up on resolution of the outstanding issue with Databricks customers.

  • In progresssafojaskfaokf
    November 22, 2023 at 3:00 AM
    In progress
    November 22, 2023 at 3:00 AM

    Maintenance is now in progress

  • Plannedsafojaskfaokf
    November 22, 2023 at 1:00 AM
    Planned
    November 22, 2023 at 1:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 11/21, from 5 to 7 PM PST.

    Below is a summary of planned changes:

    • Add the following new table columns. With the exception of the "ledger_entries" change, these will be applied by re-syncing all historical table data. We expect this to take around 10 minutes in total, during which time tables may be unavailable.
      • Add "effective_at" column to the "ledger_entries" table.
      • Add "discarded_at" column to the "returns" table.
      • Add "ledger_transaction_id" column to "payment_orders", "expected_payments", and "returns".
    • Drop the "paper_items" table. Paper Items are deprecated and unused by Push to Warehouse customers.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of these changes.

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completedsafojaskfaokf
    November 19, 2023 at 11:00 AM
    Completed
    November 19, 2023 at 11:00 AM

    Maintenance has completed successfully

  • In progresssafojaskfaokf
    November 19, 2023 at 7:00 AM
    In progress
    November 19, 2023 at 7:00 AM

    Maintenance is now in progress

  • Plannedsafojaskfaokf
    November 19, 2023 at 7:00 AM
    Planned
    November 19, 2023 at 7:00 AM

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Sunday, November 19th from 02:00 ET to 06:00 ET (4hrs).

    The following Wells Fargo APIs will be unavailable:

    • ACH Payments
    • Instant Payments
    • RTP Payments
    • Wire Payments

    ACH, RTP, and Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available. Balances and transactions will not be affected.

    Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Oct 2023

Intermittent authentication errors
  • Resolved
    Resolved

    auth0 has confirmed the incident is resolved. The Modern Treasury web app is operational.

  • Update
    Update

    auth0 has confirmed an issue causing increased error rates. We are continuing to monitor the situation, and will update every 60 minutes or as we have new information.

  • Monitoring
    Monitoring

    We have confirmed a connectivity issue with auth0. We are continuing to monitor the situation, and will update every 30 minutes or as we have new information.

  • Investigating
    Investigating

    We are seeing intermittent issues with authentication to the web app. The APIs are still operational. We are currently investigating.

Sep 2023

Partial API disruption
  • Resolved
    Resolved

    Starting at 9:03 am PT we experienced errors affecting ~3% of APIs and Web requests. This issue was caused by a manual invocation of a batch of jobs which overwhelmed our database. By 9:42 am PT, we had removed the jobs and service had returned to normal. The issue has been resolved and the service is operating normally.

  • Identified
    Identified

    We can confirm errors affecting APIs and Web. We have identified a potential root cause of the errors and are attempting mitigations. Early signs are this mitigation is reducing error rates. We continue to work towards a full root case and recovery.

  • Investigating
    Investigating

    We are currently investigating an incident that's impacting a percentage of requests to APIs starting from 9:03 am PT.

Network Connectivity Issues
  • Resolved
    Resolved

    AWS experienced network latencies and errors which started around 10:20am PT that were caused by resource contention within the subsystem responsible for propagation of network mappings within the Amazon Virtual Private Cloud.

    This affected the elastic nature of our services as we typically scale-in and scale-out API resources due to demand. Early creation and movement of our services, in an attempt to restore operation, introduced latency and manifested in errors when attempting to perform health checks and connectivity timeouts with other supporting services.

    By around 9:15pm PT, we were no longer experiencing any resource contention and our systems operated normally.

  • Update
    Update

    We continue to monitor the health of our systems. The error rates have fallen significantly and performance has stabilized. AWS is beginning to operate normally.

    We will provide a final update when all has been confirmed.

  • Update
    Update

    AWS continues to work on their network mapping propagation. Our current plan is to monitor and investigate any additional mitigations in conjunction with the AWS mitigation phases.

    We expect partial outage on all Modern Treasury endpoints with elevated rates of 504s.

    We will continue to provide updates every 90 minutes, or as we have additional information to share.

    The Modern Treasury API offers idempotent requests to prevent accidental duplication of API calls. This feature is particularly useful when initiating actions such as money transfers, entity creation, or resource modifications. For instance, if a gateway timeouts (504) occur while creating a Payment Order, you can safely retry the request using the same idempotency key to ensure that only one payment order is created.

  • Monitoring
    Monitoring

    We continue to monitor the health of our services. The error rates have continued to fall and have stabilized at a very low rate.

    We expect to be back at healthy operation levels in a few hours, after AWS has completed their updates to address the network connectivity issues and errors affecting the Availability Zones in the us-west-2 region.

    You can see their AWS's status updates here: https://health.aws.amazon.com/health/status

  • Update
    Update

    We are continuing to work on mitigations. From our monitoring, our error rate has been steadily going down. From the beginning of the incident until now, our error rate has averaged around 1%, and our worst minute window hit a max of 7.2% error rate.

    AWS is actively working on fixing the us-west-2 Availability Zone issues, you can see their status update here: https://health.aws.amazon.com/health/status

  • Identified
    Identified

    AWS is experiencing network latencies and errors in multiple availability zones in the us-west-2 region.

    We currently have limited impact due to our high availability network layout within the us-west-2 region.

    We are currently looking into additional mitigations to further limit impact.

Partial API outage
  • Resolved
    Resolved

    The Modern Treasury API has recovered and all operations are normal.

    This incident was due to a resource-intensive database operation on our replica that caused us to return errors to ~2% of list and retrieve requests from 6:27 PDT to 6:29 PDT.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Partial API Outrage
  • Resolved
    Resolved

    This incident has been resolved as of 8:30 AM PT.

    Between 7:55 AM PT and 8:29 AM PT, approximately 0.03% of API requests failed due to a modification to our database instance. We have remediated the offending modification and will take steps to ensure it doesn't occur again.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Sep 2023 to Nov 2023

Next