Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Aug 2023

Duplicate incoming transactions and IPDs at Increase Bank
  • Resolved
    Resolved

    This incident has been resolved and all duplicate transactions have been removed.

  • Monitoring
    Monitoring

    We have deployed a fix and identified all duplicate transactions and IPDs. We are now working with impacted customers to discard duplicates.

    All systems are fully operational.

  • Investigating
    Investigating

    We are currently seeing duplicate incoming transactions and Incoming Payment Details (IPDs) at Increase Bank. This does not impact outgoing transactions or create duplicate payments.

    We have identified the issue with the Increase API. We are working on deploying a fix shortly.

    This does not impact customers at other banks.

Partial API outage
  • Resolved
    Resolved

    The Modern Treasury API has recovered and all operations are normal.

    This incident was due to a spike of high database load causing us to return errors to ~2% of list and retrieve requests from 2:45 PM PDT to 2:50 PM PDT.

  • Monitoring
    Monitoring

    We saw a low level of intermittent failures on list and retrieve endpoints between 2:45 PST to 2:50 PST. We are currently investigating but believe the situation is resolved.

Jul 2023

Scheduled Maintenance: Postgres Database Upgrade
  • Completed
    July 30, 2023 at 7:43 AM
    Completed
    July 30, 2023 at 7:43 AM

    We completed our database updated at 12:37am PT.

    We are currently monitoring and performing integrity checks.

  • Update
    July 30, 2023 at 7:15 AM
    In progress
    July 30, 2023 at 7:15 AM

    Maintenance is now in progress

  • In progress
    July 30, 2023 at 7:15 AM
    In progress
    July 30, 2023 at 7:15 AM

    We are starting our planned database maintenance at 12:15am PT.

  • Planned
    July 30, 2023 at 7:15 AM
    Planned
    July 30, 2023 at 7:15 AM

    This is due to a scheduled upgrade of a Postgres database to a newer software version. Such planned upgrades are part of our maintenance routine as we stay ahead of EOL schedules and ensure that we are always up-to-date on security patches.

    During this time, the platform will be inaccessible. All operations will resume immediately after the maintenance window.

Partial API Outage
  • Resolved
    Resolved

    The Modern Treasury API has recovered and all operations are normal.

    This incident was due to a spike of high database load causing us to return errors to ~4% of list and retrieve requests from 7:26 PDT to 7:28 PDT. The increase in database load came from an internal job that has now been fixed.

  • Monitoring
    Monitoring

    We saw a low level of intermittent failures on list and retrieve endpoints. We are currently investigating but believe the situation is resolved.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Intermittent connection timeouts
  • Resolved
    Resolved

    After a day of monitoring, we can confirm that customers are no longer seeing the connection timeouts which were affecting ≈0.002% of API calls. The defects causing this incident have been resolved by the proxy rollback and TLS 1.3 upgrade.

    We are continuing to work with a few customers who are able to connect to our API servers but are experiencing high response latencies that are specific to their orgs. We are not seeing any performance regressions at this time.

  • Monitoring
    Monitoring

    After working closely with a few customers, we were able to verify that the initial issue with intermittent connection timeouts has been resolved after we rolled back one of our proxies.

    We are continuing the investigation into a possible latency issue, which is distinct from the initial issue with intermittent connection timeouts.

  • Update
    Update

    We are continuing to work closely with a few customers on reproduction and resolution. Meanwhile, we have published a guide describing recommended practices that can be used today to configure API clients for improved reliability and defend against network weather.

  • Update
    Update

    We have reason to believe that a subset of the issues our API clients are seeing is related to TLS handshakes. Following our cloud provider's recommendations, we have upgraded our public load balancers to add support for TLS 1.3 while maintaining backwards compatibility with TLS 1.2.

    TLS 1.3 is optimized for performance and security by using one round trip (1-RTT) TLS handshakes. This change is expected to mitigate some, but not all, of the connection timeouts.

  • Identified
    Identified

    We have observed intermittent timeouts affecting approximately 0.002% of the requests. We are actively addressing this issue as our top priority and will continue to provide updates.

    If your API requests received 2xx status codes in the responses, you are not affected by this issue.

    The Modern Treasury API offers idempotent requests to prevent accidental duplication of API calls. This feature is particularly useful when initiating actions such as money transfers, entity creation, or resource modifications. For instance, if a network error occurs while creating a Payment Order, you can safely retry the request using the same idempotency key to ensure that only one payment order is created.

Jun 2023

Intermittent issue validating routing numbers during Payment Order creation
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    Some customers are experiencing valid routing numbers being marked as invalid. We are currently investigating this incident.

Resolved
  • Resolved
    Resolved

    This incident has been resolved.

Partner Issue: SVB
  • Resolved
    Resolved

    Modern Treasury determined that the errors were a result of a change made by SVB to their SSH connection configuration. For further details on this change, we recommend that customers reach out to their client service team at the bank. Modern Treasury has implemented a patch to account for the new configuration. We have confirmation from SVB that all files were processed correctly and on time. All payment flows are operating as normal.

  • Update
    Update

    We are continuing to investigate. We are still not able to successfully connect to bank servers.

  • Investigating
    Investigating

    We are currently investigating an issue with SVB. We are unable to connect to SVB servers to transmit payments.

Jun 2023 to Aug 2023

Next