After a day of monitoring, we can confirm that customers are no longer seeing the connection timeouts which were affecting ≈0.002% of API calls. The defects causing this incident have been resolved by the proxy rollback and TLS 1.3 upgrade.

We are continuing to work with a few customers who are able to connect to our API servers but are experiencing high response latencies that are specific to their orgs. We are not seeing any performance regressions at this time.

After working closely with a few customers, we were able to verify that the initial issue with intermittent connection timeouts has been resolved after we rolled back one of our proxies.

We are continuing the investigation into a possible latency issue, which is distinct from the initial issue with intermittent connection timeouts.

We are continuing to work closely with a few customers on reproduction and resolution. Meanwhile, we have published a guide describing recommended practices that can be used today to configure API clients for improved reliability and defend against network weather.

We have reason to believe that a subset of the issues our API clients are seeing is related to TLS handshakes. Following our cloud provider's recommendations, we have upgraded our public load balancers to add support for TLS 1.3 while maintaining backwards compatibility with TLS 1.2.

TLS 1.3 is optimized for performance and security by using one round trip (1-RTT) TLS handshakes. This change is expected to mitigate some, but not all, of the connection timeouts.

This incident has been resolved.

Error rates have dropped, and we are currently monitoring the result. Customers should be able to create check payment orders.