Platform/API - Operational
Platform/API
Ledgers API - Operational
Ledgers API
Push to Warehouse - Operational
Push to Warehouse
Marketing Website - Operational
Marketing Website
Notice history
Nov 2023
- CompletedNovember 28, 2023 at 2:46 AMCompletedNovember 28, 2023 at 2:46 AM
Maintenance has completed successfully.
- In progressNovember 28, 2023 at 2:30 AMIn progressNovember 28, 2023 at 2:30 AM
Maintenance is now in progress
- PlannedNovember 28, 2023 at 2:30 AMPlannedNovember 28, 2023 at 2:30 AM
Modern Treasury is planning to perform database upgrades on our us0 and us5 databases on Monday, November 27th at 6:30pm PT. We expect that this operation will affect only the customers in these cells, and will have minimal impact to our API and app.
Most customers will not be affected. You can use the Organization Settings page to determine the cell that your organization is in.
- CompletedNovember 22, 2023 at 3:27 AMCompletedNovember 22, 2023 at 3:27 AM
Maintenance is complete for all destination types except Databricks, where we're continuing to work on remediating an issue that is preventing incremental transfers of payment_orders, returns, and expected_payments.
We are concluding our maintenance window at this time. We will follow up on resolution of the outstanding issue with Databricks customers.
- In progressNovember 22, 2023 at 3:00 AMIn progressNovember 22, 2023 at 3:00 AM
Maintenance is now in progress
- PlannedNovember 22, 2023 at 1:00 AMPlannedNovember 22, 2023 at 1:00 AM
A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 11/21, from 5 to 7 PM PST.
Below is a summary of planned changes:
- Add the following new table columns. With the exception of the "ledger_entries" change, these will be applied by re-syncing all historical table data. We expect this to take around 10 minutes in total, during which time tables may be unavailable.
- Add "effective_at" column to the "ledger_entries" table.
- Add "discarded_at" column to the "returns" table.
- Add "ledger_transaction_id" column to "payment_orders", "expected_payments", and "returns".
- Drop the "paper_items" table. Paper Items are deprecated and unused by Push to Warehouse customers.
Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of these changes.
- Add the following new table columns. With the exception of the "ledger_entries" change, these will be applied by re-syncing all historical table data. We expect this to take around 10 minutes in total, during which time tables may be unavailable.
Sep 2023
- ResolvedResolved
AWS experienced network latencies and errors which started around 10:20am PT that were caused by resource contention within the subsystem responsible for propagation of network mappings within the Amazon Virtual Private Cloud.
This affected the elastic nature of our services as we typically scale-in and scale-out API resources due to demand. Early creation and movement of our services, in an attempt to restore operation, introduced latency and manifested in errors when attempting to perform health checks and connectivity timeouts with other supporting services.
By around 9:15pm PT, we were no longer experiencing any resource contention and our systems operated normally.
- UpdateUpdate
We continue to monitor the health of our systems. The error rates have fallen significantly and performance has stabilized. AWS is beginning to operate normally.
We will provide a final update when all has been confirmed.
- UpdateUpdate
AWS continues to work on their network mapping propagation. Our current plan is to monitor and investigate any additional mitigations in conjunction with the AWS mitigation phases.
We expect partial outage on all Modern Treasury endpoints with elevated rates of 504s.
We will continue to provide updates every 90 minutes, or as we have additional information to share.
The Modern Treasury API offers idempotent requests to prevent accidental duplication of API calls. This feature is particularly useful when initiating actions such as money transfers, entity creation, or resource modifications. For instance, if a gateway timeouts (504) occur while creating a Payment Order, you can safely retry the request using the same idempotency key to ensure that only one payment order is created.
- MonitoringMonitoring
We continue to monitor the health of our services. The error rates have continued to fall and have stabilized at a very low rate.
We expect to be back at healthy operation levels in a few hours, after AWS has completed their updates to address the network connectivity issues and errors affecting the Availability Zones in the us-west-2 region.
You can see their AWS's status updates here: https://health.aws.amazon.com/health/status
- UpdateUpdate
We are continuing to work on mitigations. From our monitoring, our error rate has been steadily going down. From the beginning of the incident until now, our error rate has averaged around 1%, and our worst minute window hit a max of 7.2% error rate.
AWS is actively working on fixing the us-west-2 Availability Zone issues, you can see their status update here: https://health.aws.amazon.com/health/status
- IdentifiedIdentified
AWS is experiencing network latencies and errors in multiple availability zones in the us-west-2 region.
We currently have limited impact due to our high availability network layout within the us-west-2 region.
We are currently looking into additional mitigations to further limit impact.