On May 15, at approximately 08:57 UTC, we identified an issue with our CloudWorks service that impacted customers in Anaplan Data Centers - US East, US West, Germany, and the Netherlands. This issue caused CloudWorks integrations to take longer than expected to complete.
Initial investigations identified that integrations were being picked up but were not progressing to completion. We restarted a sub-component which initially had a positive impact; however, the issue continued to persist. Additional investigation indicated that the issue was linked to a recent change that was completed on Thursday, May 11, 2023. As a result, the change was rolled back. However, this did not resolve the underlying issue that was impacting the CloudWorks integrations.
Further investigation into upstream components identified high resource utilization. Although the resource utilization had not reached capacity, we identified that there was an unbalance of load between the underlying sub-components. As a result, we isolated the impacted sub-component to restart the service. This resolved the underlying issue, and normal service levels resumed at 20:03 UTC. Though, due to the duration of the incident, some customers may have experienced some backlog jobs taking up to an additional 8 hours to complete.
We are currently reviewing the change and its impact on the underlying component, to identify what caused this unbalance to occur. We have also identified an initiative to improve the resource usage and steps to tune our monitoring to avoid a similar issue from reoccurring.
We deeply apologize for the impact this had on your business operations. We understand the disruption these issues can cause to your business and users, and we are continuously strengthening our systems and procedures to avoid similar issues from happening again in the future.
If you have any further questions or concerns, please contact Anaplan Support. Thank you for your patience during this situation and thank you for being an Anaplan customer.