On May 23, 2024, at 12:14 UTC, we started to receive errors indicating that models were inaccessible in Anaplan Amazon Cloud – Europe.   

Our team established that a change to authentication interactions within the platform had been deployed around 10:35 UTC. This change was identified as the probable cause of the incident and was rolled back. The rollback was completed by 12:47 UTC. All local testing and associated monitoring were confirmed as healthy, and the incident was closed at 12:56 UTC.  

At 13:10 UTC, we received feedback that some customer models remained inaccessible. The incident was re-opened, and we found that a workspace migration had automatically started once the change had been reverted. This migration was required to move workspaces to fresh components; however, workspaces were inaccessible until the migration was completed.   

 We closely monitored the migration and by 13:24 UTC only two workspaces remained in progress. These workspaces were reviewed and at 13:43 UTC it was decided that manual intervention was required to expedite the process. The migration job was manually restarted and by 14:03 UTC all workspaces were available. We continued to monitor the platform to ensure no further issues were identified. The incident was resolved at 14:31 UTC.  

 To prevent a recurrence of this issue, we are updating our approach when implementing these types of changes. Our aim is to make the process more efficient, as well as, improving the rollback mechanism to avoid disruption in future. In addition, we are implementing additional alerting and monitoring for this scenario to ensure we know about this even earlier if the same scenario were to arise in future.  

 We apologize for any impact this issue may have had on your business operations. We are continuously strengthening our systems and procedures to ensure we avoid future disruptions to your business and users.  

 If you have further questions or concerns, please visit the Anaplan Support website. Thank you for your patience during this situation and thank you for being an Anaplan customer.