Platform Alerts
Incident Report for Anaplan
Postmortem

On October 5, 2022, at approximately 08:24 UTC, we received multiple synthetic alerts that indicated Definition Service, which is used to store and retrieve metadata for pages used by Anaplan, was degraded in our Anaplan Data Center – US-East. This resulted in some customers experiencing prolonged platform response times and, in some cases, errors when trying to use some functionalities within the New User Experience pages.

Initial investigation revealed a high CPU and maxed-out connection to the database by a specific pod within the node. This would have impacted some customers’ traffic routed to the unhealthy pod. We restarted the suspected pod and service was restored at 09:38 UTC.

Further investigation identified that an end-user ran multiple attempts to clone an extremely large report page. Due to the sheer size, this task was CPU and memory-intensive, which maxed out connections to the database for a specific pod.

To prevent a reoccurrence, we have added additional monitoring to increase our visibility of requests with durations that exceed normal thresholds. We are also implementing extra measures to limit the size of pages users can clone.

We sincerely apologize for the impact this had on your business operations. We understand the disruption these issues can cause to your business and users, and we are continuously strengthening our systems and procedures to avoid similar problems from happening again in the future. If you have any further questions or concerns, please contact Anaplan Support. Thank you for your patience during this situation and thank you for being an Anaplan customer.

Posted Oct 10, 2022 - 20:48 UTC

Resolved
We have confirmed that the issue is now resolved. We deeply apologize for any impact this issue may have caused. We appreciate your patience and partnership as we worked through this issue. We will follow up within 7 business days with a detailed root cause analysis (RCA) that will be shared on our Status Page. If you have any questions, concerns, or continue to experience errors, please do not hesitate to contact us at Anaplan Support.
Posted Oct 05, 2022 - 11:03 UTC
Monitoring
Service has now been restored; you should now be able to resume normal activities. Our engineers will continue to monitor the platform to ensure no additional issues arise. If you have any questions, concerns, or continue to experience issues, please do not hesitate to contact Anaplan Support.

We will provide a final update to you when we consider this situation fully resolved.
Posted Oct 05, 2022 - 09:27 UTC
Update
We are continuing to investigate this issue.
Posted Oct 05, 2022 - 09:25 UTC
Update
We are currently experiencing issues impacting customers’ ability to see apps pages.
Our engineering team is working to identify the root cause and implement a solution.
We will continue to provide updates to you as we work to resolve this issue as quickly as possible.
Posted Oct 05, 2022 - 09:12 UTC
Investigating
We are investigating a potential issue impacting Anaplan Platform.
We will provide regular updates on this page until the issue is resolved.
Posted Oct 05, 2022 - 09:08 UTC
This incident affected: Anaplan Data Center - US East (Planning Platform), Anaplan Data Center - Netherlands (Planning Platform), Anaplan Amazon Cloud Public - US (Planning Platform), Anaplan Google Cloud Public - US East (Planning Platform), Anaplan Data Center - US West (Planning Platform), Anaplan Data Center - Germany (Planning Platform), and Anaplan Google Cloud Public - Japan (Planning Platform).