Reported Issue: Between August 19, 2:28 AM UTC and 3:56 AM UTC, all customers using the TCW platform across all regions experienced failures and delays when attempting to export selected location data from the My Locations feature. Export jobs remained in an “In Progress” state and could not be completed or canceled via the user interface.
Duration: 1 hour and 28 minutes (August 19, 2:28 AM UTC – 3:56 AM UTC)
Cause: The issue was caused by excessive system resource consumption due to multiple large export jobs running concurrently, along with a high volume of identical exports initiated by a single user. This overwhelmed the system and delayed the processing of smaller export jobs. The RabbitMQ message broker, responsible for routing export tasks, routed large jobs into queues not designed to handle such load, contributing further to system contention.
Solution: The problem was mitigated by manually terminating duplicate and long-running export jobs, which allowed the system to begin processing the backlog of queued exports normally. All export operations resumed functioning as expected by 3:56 AM UTC.
Posted Aug 20, 2025 - 12:54 EDT
Monitoring
We are continuing to monitor for any further issues.
Posted Aug 19, 2025 - 23:59 EDT
Investigating
We are continuing to monitor the results.
Posted Aug 18, 2025 - 23:59 EDT
Monitoring
A fix has been implemented and we are monitoring the results.