Modern Treasury - Notice history

System Under Maintenance

Platform/API - Under maintenance

Ledgers API - Under maintenance

Push to Warehouse - Operational

Marketing Website - Operational

Notice history

Nov 2022

API and Application Errors.
  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Resolved
    Resolved

    API error rates and latencies have been mitigated.

  • Monitoring
    Monitoring

    We believe we have mitigated the cause of the errors and will continue to monitor.

  • Investigating
    Investigating

    The Modern Treasury Ledger API may be experiencing degraded performance. We are investigating.

Most Ledger Transaction updates are failing
  • Resolved
    Resolved

    We have resolved the issue.

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    We've identified the fix for this issue.

  • Investigating
    Investigating

    We are currently investigating this incident.

Scheduled Maintenance: Hardware Upgrade
  • Completed
    November 20, 2022 at 9:13 AM
    Completed
    November 20, 2022 at 9:13 AM

    Maintenance has been completed successfully. Actual API downtime was 77 seconds.

  • In progress
    November 20, 2022 at 8:15 AM
    In progress
    November 20, 2022 at 8:15 AM

    Maintenance is now in progress

  • Planned
    November 20, 2022 at 8:15 AM
    Planned
    November 20, 2022 at 8:15 AM

    This is due to an upcoming hardware retirement scheduled by our cloud provider for a Redis cluster. During maintenance time, the platform will be inaccessible. All operations will resume immediately after the maintenance window.

<0.2% of API calls returned 502s and 504s
  • Resolved
    Resolved

    We have recovered and now operating normally.

  • Monitoring
    Monitoring

    We have put the fix in place and should be recovering.

  • Identified
    Identified

    We have identified the cause and working on mitigation strategy.

  • Investigating
    Investigating

    We are observing increased error rates on some API calls. Our team is investigating on this issue.

Partial outage of API calls
  • Resolved
    Resolved

    The Modern Treasury Ledger API has recovered.

  • Investigating
    Investigating

    The Modern Treasury Ledger API may be experiencing degraded performance. We are investigating.

Oct 2022

Elevated Error Rates
  • Resolved
    Resolved

    This issue was resolved.

  • Investigating
    Investigating

    We're experiencing an elevated level of API errors and are currently looking into the issue.

High API and application latencies
  • Resolved
    Resolved

    The issue is resolved and we've solved the underlying cause of the contention.

  • Monitoring
    Monitoring

    The issue is resolved and we've solved the underlying cause of the contention.

  • Identified
    Identified

    We have identified the cause of the contention. Working to mitigate and ultimately resolve this issue.

  • Investigating
    Investigating

    The Modern Treasury Ledger API may be experiencing degraded performance. We are investigating.

We are experiencing API and UI issues due to database contention.
  • Resolved
    Resolved

    We have addressed the underlying database contention and will follow-up with further improvements to the system to prevent reoccurrence.

  • Identified
    Identified

    We have identified the issue inside our database.

  • Investigating
    Investigating

    We are experiencing API and UI issues due to database contention.

Modern Treasury UI temporarily unavailable
  • Resolved
    Resolved

    All services are operational and healthy.

  • Identified
    Identified

    We had deployed a bad commit which fails to render UI. This downtime did not affect API calls.

Sep 2022

An automated up-time check has failed. (False positive)
  • Resolved
    Resolved

    This automated check has been resolved. No customer was impacted.

  • Investigating
    Investigating

    We are in the process of changing the application and triggered a false positive alarm. No customer was impacted.

API outage due to load balancer misconfiguration
  • Resolved
    Resolved

    At 9:55 PM UTC, we recovered from 4 minutes outage due to load balancer misconfiguration. This incident has been resolved and services are now healthy to continue normal traffic.

  • Investigating
    Investigating

    The Modern Treasury Ledger API may be experiencing degraded performance. We are investigating.

An automated up-time check has failed. (False Positive.)
  • Resolved
    Resolved

    The issue is determined to be related to the service degradation of our integration with Plaid. For more details, please see Observability issues in Development and Sandbox with potential increased API errors rates.

    The Modern Treasury production application and API are NOT impacted by this.

  • Investigating
    Investigating

    The Modern Treasury application and API may be experiencing degraded performance. We are investigating.

Partner Issue: app.moderntreasury.com is experiencing login issues
  • Resolved
    Resolved

    The authentication issue has been resolved by Auth0. Modern Treasury web application at app.moderntreasury.com is now fully functional.

  • Monitoring
    Monitoring

    `app.moderntreasury.com` is having authentication issues with Auth0. Some users may have trouble logging into the Modern Treasury web application. Our API, which does not depend on Auth0, remains 100% functional.

    For more information, please refer to the Auth0 status page: Elevated errors in Auth0.

Partial API and Application Outage
  • Resolved
    Resolved

    At around 9:22am PT on September 1, we were alerted about unexpected resource utilization across our system. For about 23 minutes, we experienced intermittent failures of API calls. Roughly 75% of incoming API calls failed.

  • Investigating
    Investigating

    The Modern Treasury Ledger API may be experiencing degraded performance. We are investigating.

Sep 2022 to Nov 2022

Next