Modern Treasury - Notice history

Modern Treasury has identified a network connectivity issue between two AWS regions is likely the cause of the errors a subset of customers experienced May 23rd. We’ve updated our systems to improve performance and reliability for our API services. We are seeing consistent and successful activity at this time.

Currently monitoring as functionality appears stable.

Between 18:11 and 18:37 we saw an increase in API error rates in us000. This mostly impacted operations related to ledgers listing. We have completed reverting the suspected database operation.

We've identified a potential fix involving the rollback of a database optimization. Error levels are steadily decreasing and monitoring is ongoing.

Only affecting customers in cell us000

All affected customers should have fresh data as of 8:30am PT. Regularly-scheduled Transfers have resumed and are succeeding. We are continuing to carefully monitor transfer health to validate the service is fully operational.

We have identified the workload responsible for causing transfers to become stuck and deployed a fix to mitigate the issue. We have begun triggering catch-up transfers for affected warehouse destinations.

The incident has been resolved. By 1:28 PM PDT, all transaction reconciliation paused between 8:31 AM PDT and 12:31 PM PDT has been completed successfully.

At 12:31 PM PDT, we unpaused reconciliation and are currently monitoring the result.

This incident has been resolved. We identified a spike in memory which impacted application performance. Application functionality has been fully restored for affected organizations in cell us001.

We are seeing 100% success rates for transfers for the past 2 hours.

We have identified and resolved the root cause behind these transfer failures and are seeing a 100% transfer success rate in the past hour. We will continue to monitor data freshness and provide an update once we are certain this issue is fully resolved. 

We are continuing to investigate the source of occasional transfer failures. We plan to continue to monitor data freshness and proactively initiate ad-hoc transfers to customers as needed while the issue persists.

We are seeing routing information correctly validated for all customers.

We identified that a new caching setting led to outdated access tokens used to access our third-party routing verifier. Caching settings have been adjusted and we are now monitoring that routing information requests are valid.

We are seeing consistent successful transfers across all customers.

We have implemented a fix and are seeing successful transfers for all affected customers. We will continue to closely monitor transfers over the next few hours to validate this fix.

We have re-enabled the Push to Warehouse staging ETL process, and customer warehouse data should now be up-to-date.

Incremental transfers for Balance Reports (introduced during yesterday's maintenance window) remain disabled at this time while we investigate the root cause of poor staging performance.