Modern Treasury - Notice history

All systems operational

Platform/API - Operational

Ledgers API - Operational

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

View current status

Oct 2024

CSV Export Downloads Unavailable
Resolved4 hours 48 minutes
  • ResolvedUTC

    This incident has been resolved. Access to exports are re-enabled.

  • IdentifiedUTC

    We have identified a problem with out export system and are deploying a fix.

  • InvestigatingUTC

    We are currently investigating an issue with CSV Exports. Exporting is unavailable while we are investigating.

OCT16
Role Based Access Control: API Key Migration
CompletedMaintenance22 minutes
  • CompletedOctober 16, 2024 at 11:22 PMUTC
    Maintenance has completed successfully.
  • In progressOctober 16, 2024 at 11:04 PMUTC
    Maintenance is now in progress.
  • UpdateOctober 16, 2024 at 11:00 PMUTC

    Modern Treasury will be conducting maintenance on October 16th at 4pm PDT. As part of the RBAC system upgrade, all API keys, production and sandbox, that currently have "manage" access to all resources will be migrated to the new "API Manage All" role. We expect no downtime.

    Additional details about the new RBAC system can be found here.

  • PlannedOctober 16, 2024 at 12:11 PMUTC
    Maintenance didn't start yet.
  • In progressOctober 16, 2024 at 11:00 AMUTC
    Maintenance is now in progress
OCT16
Scheduled Maintenance: Push to Warehouse
CompletedMaintenance2 hours
  • CompletedOctober 16, 2024 at 2:00 AMUTC
    Maintenance has completed successfully
  • In progressOctober 16, 2024 at 12:00 AMUTC
    Maintenance is now in progress
  • PlannedOctober 16, 2024 at 12:00 AMUTC

    We will be conducting our regularly scheduled Push to Warehouse maintenance window this coming Tuesday (10/15), from 5 PM to 7 PM PST.

    During this maintenance window we will introduce two new columns (as_of_date and as_of_time) to the balance_reports and transactions tables. These supersede the existing as_of_datetime columns on each table, which we are deprecating and plan to drop in a future release. We will communicate the timing of that change in a future maintenance notice.

    Please note that regularly-scheduled warehouse transfers may be delayed during the maintenance window. If you have any questions or concerns regarding this change, please reach out to support@moderntreasury.com.

Modern Treasury API: Elevated error rate
ResolvedPartial outage
  • UpdateUTC

    This issue has been resolved.

  • ResolvedUTC

    We've identified a ~10-minute period of elevated API errors impacting customers in cell us001 starting at ~2:01pm PDT. We have determined this was temporary and is now resolved.

    You can find your cell number on the settings page of the Modern Treasury app.

Sep 2024

Modern Treasury App: Degraded Performance
ResolvedDegraded performance12 minutes
  • ResolvedUTC

    This incident has been resolved.

  • InvestigatingUTC

    We are investigating elevated errors rates from the Modern Treasury app impacting customers. Customers may experience degraded performance while using the Modern Treasury platform.

CSV Export Downloads Unavailable
ResolvedPartial outage1 hour 54 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    We have released a fix for the CSV export issue and are currently monitoring. The export download functionality should be working as expected for all customers.

  • IdentifiedUTC

    We are working to resolve an issue preventing the download of CSV exports that is impacting all customers. Estimated resolution time: <1 hour.

Elevated Reconciliation Delays
ResolvedDegraded performance2 hours 22 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    With the increased processing rate the reconciliation backlog should be processed ~1 hour.

  • IdentifiedUTC

    We have identified and quarantined the issue. The reconciliation processing rate is improving.

  • InvestigatingUTC

    We have detected delays in our reconciliation system and are working to resolve the issue.

SEP1
Partner Scheduled Maintenance: Wells Fargo Gateway API
CompletedMaintenance2 hours
  • CompletedSeptember 01, 2024 at 11:00 PMUTC
    Maintenance has completed successfully
  • In progressSeptember 01, 2024 at 9:00 PMUTC
    Maintenance is now in progress
  • PlannedSeptember 01, 2024 at 9:00 PMUTC
    We are planning for a scheduled maintenance during that time.

Aug 2024

Elevated Reconciliation Delays
ResolvedDegraded performance2 hours 38 minutes
  • ResolvedUTC

    This incident has been resolved.

  • UpdateUTC

    We are continuing to work on a fix for this incident. With the increased processing rate the reconciliation backlog should be processed ~1 hour.

  • IdentifiedUTC

    We have identified and quarantined the issue. We are taking steps to increase our processing rate in order to process the reconciliation backlog.

AUG21
Scheduled Maintenance: Push to Warehouse
CompletedMaintenance2 hours
  • CompletedAugust 21, 2024 at 2:00 AMUTC
    Maintenance has completed successfully
  • In progressAugust 21, 2024 at 12:00 AMUTC
    Maintenance is now in progress
  • PlannedAugust 21, 2024 at 12:00 AMUTC

    A Push to Warehouse maintenance window will take place per our regular schedule next week, on Tuesday 8/20, from 5 to 7 PM PST.


    We plan to make the following changes during this time:

    • Cease syncing ledger_transactions records with a ledgerable_type of PaymentOrderAttempt. New ledger_transactions records associated with a Modern Treasury Payment Order will have a ledgerable_type of PaymentOrder, and their respective ledgerable_id values will identify the relevant Payment Order. By default, this change will apply only to newly-synced records. Please contact Modern Treasury Support if you would like for us to re-sync historical ledger_transactions records to ensure they also reflect this change.

    • Cease syncing the ledger_account_balances table, which does not contain meaningful Ledger balance information. ledger_entries is and will remain authoritative with respect to account balances.

    • Perform a one-off resync of the payment_orders table, to address a bug that made it possible in rare cases for transaction_ids to become out-of-sync with our application database.

    • Add a transaction_id column to the incoming_payment_details table, and perform a one-off re-sync of the table to ensure the field is populated correctly on historical records.

    Regularly-scheduled warehouse transfers may be delayed for the duration of the maintenance window (up to two hours).

    Please reach out to support@moderntreasury.com with any questions or concerns regarding the scope of this change.

Push to Warehouse transfers of Incoming Payment Detail data are failing
ResolvedDegraded performance55 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    We've implemented a fix, and transfers of incoming_payment_details are now succeeding.

    We're waiting for a few remaining transfers to complete to confirm that the issue is resolved for all subscribed destinations.

  • IdentifiedUTC

    Push to Warehouse transfers are occurring as scheduled, but transfers of the incoming_payment_details table are failing due to an error. We have identified the source of the issue and are working on resolving it.

Payment order API failures
ResolvedPartial outage2 hours 32 minutes
  • ResolvedUTC

    This incident has been resolved and the Payment Order API is operating as expected.

  • MonitoringUTC

    The Payment Order API is now operating as expected and we are continuing to monitor.

  • InvestigatingUTC

    We are currently investigating an issue with the Payment Orders API. We will provide updates as we learn more.

API POST & UI Creation workflows in US001
ResolvedMajor outage6 hours 1 minute
  • ResolvedUTC

    This incident has been resolved. We identified a spike in memory which impacted API POST & UI creation workflows. POST & UI creation functionality has been fully restored for affected organizations in cell us001.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • InvestigatingUTC

    We are currently investigating this incident impacting customers in US001. Errors may be experienced when submitting API POST requests or using creation based workflows in the web application.

Previous

Aug 2024 to Oct 2024

Next