Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Jul 2023

Scheduled Maintenance: Postgres Database Upgrade
  • Completed
    July 30, 2023 at 7:43 AMUTC
    Completed
    July 30, 2023 at 7:43 AMUTC

    We completed our database updated at 12:37am PT.

    We are currently monitoring and performing integrity checks.

  • Update
    July 30, 2023 at 7:15 AMUTC
    Update
    July 30, 2023 at 7:15 AMUTC

    Maintenance is now in progress

  • In progress
    July 30, 2023 at 7:15 AMUTC
    In progress
    July 30, 2023 at 7:15 AMUTC

    We are starting our planned database maintenance at 12:15am PT.

  • Planned
    July 30, 2023 at 7:15 AMUTC
    Planned
    July 30, 2023 at 7:15 AMUTC

    This is due to a scheduled upgrade of a Postgres database to a newer software version. Such planned upgrades are part of our maintenance routine as we stay ahead of EOL schedules and ensure that we are always up-to-date on security patches.

    During this time, the platform will be inaccessible. All operations will resume immediately after the maintenance window.

Partial API Outage
  • Resolved
    UTC
    Resolved

    The Modern Treasury API has recovered and all operations are normal.

    This incident was due to a spike of high database load causing us to return errors to ~4% of list and retrieve requests from 7:26 PDT to 7:28 PDT. The increase in database load came from an internal job that has now been fixed.

  • Monitoring
    UTC
    Monitoring

    We saw a low level of intermittent failures on list and retrieve endpoints. We are currently investigating but believe the situation is resolved.

  • Investigating
    UTC
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Intermittent connection timeouts
  • Resolved
    UTC
    Resolved

    After a day of monitoring, we can confirm that customers are no longer seeing the connection timeouts which were affecting ≈0.002% of API calls. The defects causing this incident have been resolved by the proxy rollback and TLS 1.3 upgrade.

    We are continuing to work with a few customers who are able to connect to our API servers but are experiencing high response latencies that are specific to their orgs. We are not seeing any performance regressions at this time.

  • Monitoring
    UTC
    Monitoring

    After working closely with a few customers, we were able to verify that the initial issue with intermittent connection timeouts has been resolved after we rolled back one of our proxies.

    We are continuing the investigation into a possible latency issue, which is distinct from the initial issue with intermittent connection timeouts.

  • Update
    UTC
    Update

    We are continuing to work closely with a few customers on reproduction and resolution. Meanwhile, we have published a guide describing recommended practices that can be used today to configure API clients for improved reliability and defend against network weather.

  • Update
    UTC
    Update

    We have reason to believe that a subset of the issues our API clients are seeing is related to TLS handshakes. Following our cloud provider's recommendations, we have upgraded our public load balancers to add support for TLS 1.3 while maintaining backwards compatibility with TLS 1.2.

    TLS 1.3 is optimized for performance and security by using one round trip (1-RTT) TLS handshakes. This change is expected to mitigate some, but not all, of the connection timeouts.

  • Identified
    UTC
    Identified

    We have observed intermittent timeouts affecting approximately 0.002% of the requests. We are actively addressing this issue as our top priority and will continue to provide updates.

    If your API requests received 2xx status codes in the responses, you are not affected by this issue.

    The Modern Treasury API offers idempotent requests to prevent accidental duplication of API calls. This feature is particularly useful when initiating actions such as money transfers, entity creation, or resource modifications. For instance, if a network error occurs while creating a Payment Order, you can safely retry the request using the same idempotency key to ensure that only one payment order is created.

Jun 2023

Intermittent issue validating routing numbers during Payment Order creation
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Investigating
    UTC
    Investigating

    Some customers are experiencing valid routing numbers being marked as invalid. We are currently investigating this incident.

Resolved
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

Partner Issue: SVB
  • Resolved
    UTC
    Resolved

    Modern Treasury determined that the errors were a result of a change made by SVB to their SSH connection configuration. For further details on this change, we recommend that customers reach out to their client service team at the bank. Modern Treasury has implemented a patch to account for the new configuration. We have confirmation from SVB that all files were processed correctly and on time. All payment flows are operating as normal.

  • Update
    UTC
    Update

    We are continuing to investigate. We are still not able to successfully connect to bank servers.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue with SVB. We are unable to connect to SVB servers to transmit payments.

May 2023

Partner Issue: Lob
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Update
    UTC
    Update

    Error rates have dropped, and we are currently monitoring the result. Customers should be able to create check payment orders.

  • Monitoring
    UTC
    Monitoring

    One of our partners is experiencing a problem. You can see more information https://lob.statuspage.io/ for updates.

    Customers are unable to create check payment orders.

May 2023 to Jul 2023

Next