Platform/API
Ledgers API
Compliance API
Push to Warehouse
Marketing Website
Notice history
Sep 2023
- ResolvedResolved
AWS experienced network latencies and errors which started around 10:20am PT that were caused by resource contention within the subsystem responsible for propagation of network mappings within the Amazon Virtual Private Cloud.
This affected the elastic nature of our services as we typically scale-in and scale-out API resources due to demand. Early creation and movement of our services, in an attempt to restore operation, introduced latency and manifested in errors when attempting to perform health checks and connectivity timeouts with other supporting services.
By around 9:15pm PT, we were no longer experiencing any resource contention and our systems operated normally.
- MonitoringUpdate
We continue to monitor the health of our systems. The error rates have fallen significantly and performance has stabilized. AWS is beginning to operate normally.
We will provide a final update when all has been confirmed.
- MonitoringUpdate
AWS continues to work on their network mapping propagation. Our current plan is to monitor and investigate any additional mitigations in conjunction with the AWS mitigation phases.
We expect partial outage on all Modern Treasury endpoints with elevated rates of 504s.
We will continue to provide updates every 90 minutes, or as we have additional information to share.
The Modern Treasury API offers idempotent requests to prevent accidental duplication of API calls. This feature is particularly useful when initiating actions such as money transfers, entity creation, or resource modifications. For instance, if a gateway timeouts (504) occur while creating a Payment Order, you can safely retry the request using the same idempotency key to ensure that only one payment order is created.
- MonitoringMonitoring
We continue to monitor the health of our services. The error rates have continued to fall and have stabilized at a very low rate.
We expect to be back at healthy operation levels in a few hours, after AWS has completed their updates to address the network connectivity issues and errors affecting the Availability Zones in the us-west-2 region.
You can see their AWS's status updates here: https://health.aws.amazon.com/health/status
- IdentifiedUpdate
We are continuing to work on mitigations. From our monitoring, our error rate has been steadily going down. From the beginning of the incident until now, our error rate has averaged around 1%, and our worst minute window hit a max of 7.2% error rate.
AWS is actively working on fixing the us-west-2 Availability Zone issues, you can see their status update here: https://health.aws.amazon.com/health/status
- IdentifiedIdentified
AWS is experiencing network latencies and errors in multiple availability zones in the us-west-2 region.
We currently have limited impact due to our high availability network layout within the us-west-2 region.
We are currently looking into additional mitigations to further limit impact.
Aug 2023
Jul 2023
- ResolvedResolved
After a day of monitoring, we can confirm that customers are no longer seeing the connection timeouts which were affecting ≈0.002% of API calls. The defects causing this incident have been resolved by the proxy rollback and TLS 1.3 upgrade.
We are continuing to work with a few customers who are able to connect to our API servers but are experiencing high response latencies that are specific to their orgs. We are not seeing any performance regressions at this time.
- MonitoringMonitoring
After working closely with a few customers, we were able to verify that the initial issue with intermittent connection timeouts has been resolved after we rolled back one of our proxies.
We are continuing the investigation into a possible latency issue, which is distinct from the initial issue with intermittent connection timeouts.
- IdentifiedUpdate
We are continuing to work closely with a few customers on reproduction and resolution. Meanwhile, we have published a guide describing recommended practices that can be used today to configure API clients for improved reliability and defend against network weather.
- IdentifiedUpdate
We have reason to believe that a subset of the issues our API clients are seeing is related to TLS handshakes. Following our cloud provider's recommendations, we have upgraded our public load balancers to add support for TLS 1.3 while maintaining backwards compatibility with TLS 1.2.
TLS 1.3 is optimized for performance and security by using one round trip (1-RTT) TLS handshakes. This change is expected to mitigate some, but not all, of the connection timeouts.
- IdentifiedIdentified
We have observed intermittent timeouts affecting approximately 0.002% of the requests. We are actively addressing this issue as our top priority and will continue to provide updates.
If your API requests received 2xx status codes in the responses, you are not affected by this issue.
The Modern Treasury API offers idempotent requests to prevent accidental duplication of API calls. This feature is particularly useful when initiating actions such as money transfers, entity creation, or resource modifications. For instance, if a network error occurs while creating a Payment Order, you can safely retry the request using the same idempotency key to ensure that only one payment order is created.