On October 25, 2022, at approximately 04:54 UTC, we were notified that some end-users were experiencing degraded service and were not able to load workspaces in Amazon Single Tenants (AST). Initial investigation indicated an issue with our Widget Data Services service, which is responsible for access to data in Anaplan Models. We performed a rolling restart of this service, observed a decrease in errors and at 07:21 UTC, end-users were able to load their workspaces. We therefore closed the incident while we continued to work on identifying the root cause of the issue.

At approximately 19:41 UTC, it was again reported that end users could not open workspaces in AST. We immediately repeated the steps completed during the earlier incident however, the issue persisted. Further investigation indicated an issue with a vendor component that we use for storage. The component in question was in an unhealthy state which caused the service degradation. We performed recovery actions on the storage to restore regular service at 23:36 UTC.

The vendor subsequently advised us of a data bug affecting our current version of the storage component. We are currently testing the new version to prevent reoccurrence and will deploy it as soon as testing is complete. Additionally, we have added extra detection measures to the service to allow immediate mitigations before end-user impact.

We sincerely apologize for any impact this had on your business operations. We understand the disruption these issues can cause to your business and users, and we are continuously strengthening our systems and procedures to avoid similar issues in the future. If you have any further questions or concerns, please contact Anaplan Support.

Thank you for your patience during this situation and thank you for being an Anaplan customer.