Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Mar 2024

Intermittent errors: Routing Validation
  • Resolved
    Resolved

    We are seeing routing information correctly validated for all customers.

  • Monitoring
    Monitoring

    We identified that a new caching setting led to outdated access tokens used to access our third-party routing verifier. Caching settings have been adjusted and we are now monitoring that routing information requests are valid.

  • Investigating
    Investigating

    There are a subset of requests for payment endpoints experiencing intermittent errors when validating new ABA routing numbers. Our team is actively investigating these occurrences. Retry attempts are confirmed as processing successfully.

Push to Warehouse: Transfers are not occurring as scheduled for some customers
  • Resolved
    Resolved

    We are seeing consistent successful transfers across all customers.

  • Monitoring
    Monitoring

    We have implemented a fix and are seeing successful transfers for all affected customers. We will continue to closely monitor transfers over the next few hours to validate this fix.

  • Identified
    Identified

    We are aware of an issue causing delays in the delivery of data to some customer warehouse destinations. We discovered this afternoon that some of our transfer agent tasks appear to have entered an unresponsive state around 6 PM PT Thursday, blocking regularly-scheduled transfers for some customers.

    We are investigating the source of the problem and working to rectify the issue.

Elevated API Error rate
  • Resolved
    Resolved

    We experienced elevated API error rates from 11:45 PM - 12:10 AM PST. We implemented a fix to reduce the error rate, and are monitoring currently.

Push to Warehouse: Data is Delayed
  • Resolved
    Resolved

    We have re-enabled the Push to Warehouse staging ETL process, and customer warehouse data should now be up-to-date.

    Incremental transfers for Balance Reports (introduced during yesterday's maintenance window) remain disabled at this time while we investigate the root cause of poor staging performance.

  • Identified
    Identified

    We experienced an issue with the ETL process involved in staging customer Push to Warehouse data following our maintenance window yesterday evening, and decided to temporarily disable it to avoid impacting the availability of our web API and UI.

    As a result, changes occurring since ~11:30 PT yesterday evening are not yet reflected in customer warehouse destinations.

    We are working to re-enable the ETL process as soon as possible.

Scheduled Maintenance: Push to Warehouse
  • Update
    March 20, 2024 at 7:52 PM
    Update
    March 20, 2024 at 7:52 PM

    Shortly after the completion of this maintenance window we experienced an incident that impacted our ETL process. This incident has since been resolved.

    As part of the incident remediation process, transfers for the balance_reports table have been paused since ~11:30 PM PT and data may be stale. We are actively working to re-enable transfers for balance_reports and will update this maintenance update when we have a timeline estimate.

    No other tables are affected, including the newly added payment_references table.

  • Completed
    March 20, 2024 at 2:00 AM
    Completed
    March 20, 2024 at 2:00 AM
    Maintenance has completed successfully
  • In progress
    March 20, 2024 at 12:00 AM
    In progress
    March 20, 2024 at 12:00 AM
    Maintenance is now in progress
  • Planned
    March 20, 2024 at 12:00 AM
    Planned
    March 20, 2024 at 12:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 3/19, from 5 to 7 PM PST.

    During this time, we plan to make the following changes:

    1. Introduce new balance_reports and payment_references tables, which existing recipients of Modern Treasury Payments tables will receive.

    2. Re-sync historical data for the transaction_line_items table. This is a one-time measure that will allow us to improve the speed and efficiency of transfers moving forward.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Feb 2024

Partner Scheduled Maintenance: Wells Fargo Gateway API
  • Completed
    February 17, 2024 at 10:00 AM
    Completed
    February 17, 2024 at 10:00 AM
    Maintenance has completed successfully
  • In progress
    February 17, 2024 at 6:20 AM
    In progress
    February 17, 2024 at 6:20 AM
    Maintenance is now in progress
  • Planned
    February 17, 2024 at 6:20 AM
    Planned
    February 17, 2024 at 6:20 AM

    Modern Treasury has been notified that the Wells Fargo Gateway API will be undergoing scheduled maintenance on Saturday, February 17th from 01:20 ET to 05:00 ET (3hrs 40m).

    The following Wells Fargo APIs will be unavailable:

    • Wire Payments

    Wire payments originated from Wells Fargo accounts during that window may be delayed. Modern Treasury will automatically hold and retry payments when the API becomes available.

    Balances and transactions will not be affected. Customers using file based transmission at Wells Fargo will not be affected. Accounts at other banking partners will not be affected.

Elevated Error Rates
  • Resolved
    Resolved

    2.5% of API calls returned 500 at 5:32 PM PST due to abnormally high database load. Load has returned to normal, and we are investigating to prevent this from happening in the future.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

We received a false positive notification from one of our monitoring tools.
  • Resolved
    Resolved

    This incident has been resolved. We have determined that it was a false positive, and production API calls were not affected.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Jan 2024

Scheduled Maintenance: Push to Warehouse - Source Database
  • Completed
    January 31, 2024 at 3:11 AM
    Completed
    January 31, 2024 at 3:11 AM

    Maintenance complete.

  • In progress
    January 31, 2024 at 3:00 AM
    In progress
    January 31, 2024 at 3:00 AM

    Maintenance is now in progress

  • Planned
    January 31, 2024 at 3:00 AM
    Planned
    January 31, 2024 at 3:00 AM

    We plan to upgrade the database instance where customer Push to Warehouse data resides tonight (1/30) at 7pm PT. We expect this will require a few minutes of downtime, as a result of which regularly-scheduled data transfers may be delayed by a few minutes.

Degraded Performance for 40 minutes in us001
  • Resolved
    Resolved

    From 11:40am to 12:20pm PT, the us001 database was under high load due to data processing jobs.

    During this time, the p95 latency for API requests were elevated, peaking at 1.6seconds. We did not see any API errors during this time period. There were around eleven total 500s due to timeouts from our web UI in the same time period.

    We are currently making changes to our data processing jobs to tackle the root cause.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We're experiencing an elevated level of API errors and are currently looking into the issue.

Scheduled Maintenance: Push to Warehouse
  • Completed
    January 17, 2024 at 2:40 AM
    Completed
    January 17, 2024 at 2:40 AM

    Maintenance has completed successfully.

  • In progress
    January 17, 2024 at 1:00 AM
    In progress
    January 17, 2024 at 1:00 AM

    Maintenance is now in progress

  • Planned
    January 17, 2024 at 1:00 AM
    Planned
    January 17, 2024 at 1:00 AM

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 1/16, from 5 to 7 PM PST.

    During this time, we plan to perform some internal housekeeping to update our Push to Warehouse product groupings. We do not expect this to impact customers — neither the tables received nor data availability.

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this changes.

Jan 2024 to Mar 2024

Next