Platform Alerts
Incident Report for Anaplan
Postmortem

On December 21, 2022, at approximately 09:30 UTC, we noticed unusually elevated API errors within Google Single Tenant (GST), which can cause degraded service with Anaplan connect integration workspaces.

An initial investigation revealed a decrease in request rate to the API service at 01:00 UTC. The decrease was likely due to a delay in saving model tasks, and it resulted in our API service running with fewer threads. Subsequently, our API service began returning timeout errors (status 524) as it could not process some incoming requests.

Our engineering team restarted the API service to clear the queue and allow requests to be processed again. That resolved the issue at 10:45 UTC.

To limit the likelihood of this issue recurring, we have added extra detection measures to the API service to allow immediate mitigations. At the same time, we continue to work with our vendor to resolve the issue with occasional delays to saving models.

We sincerely apologize for any impact this situation had on your business operations. We understand the disruption these issues can cause to your business and users, and we are continuously strengthening our systems and procedures to avoid similar issues in the future. If you have any further questions or concerns, please contact Anaplan Support.

Thank you for your patience during this situation and thank you for being an Anaplan customer.

Posted Jan 04, 2023 - 11:45 UTC

Resolved
We have confirmed that the issue is now resolved. We deeply apologize for any impact this issue may have caused.We appreciate your patience and partnership as we worked through this issue. We will follow up within 7 business days with a detailed root cause analysis (RCA) that will be shared on our Status Page. If you have any questions, concerns, or continue to experience errors, please do not hesitate to contact us at Anaplan Support.
Posted Dec 21, 2022 - 10:49 UTC
Identified
The process to restore service is in progress. We will provide further updates in 30 minutes or on resolution.
Posted Dec 21, 2022 - 10:40 UTC
Investigating
We are experiencing an elevated level of API errors, which may be impacting some of your integrations.

Our engineering team is working to identify the root cause and implement a solution.We will continue to provide updates to you as we work to resolve this issue as quickly as possible.
Posted Dec 21, 2022 - 10:34 UTC
This incident affected: Anaplan Google Cloud - Dedicated (Planning Platform).