Platform Alerts
Incident Report for Anaplan
Postmortem

On August 2nd, 2022, at 02:42 UTC, it was determined that the Anaplan Google Cloud Dedicated platform was inaccessible. Investigation identified that the database cluster was not functioning correctly because of missing data. A restart of the cluster was completed, but this did not resolve the issue. Therefore, further action was taken to use a suitable backup and restore to the production cluster, which restored service.

Post restoration, further investigation identified that at around 02:30 UTC a database pod was provisioned on a worker node with intermittent network issues. The pod attempted to join multiple times without success which resulted in the data issues. The cluster was subsequently scaled down and back up, using the affected pod as the “source of truth”, which resulted in all pods to be provisioned without data.

To avoid a reoccurrence of this issue, we have validated the configuration of the database cluster and ensured that appropriate monitoring is in place. We have also commenced a deep dive investigation into the cause of the network issues and will take the appropriate action(s).

We deeply apologize for the impact this had on your business operations. We understand the disruption these issues can cause to your business and users, and we are continuously strengthening our systems and procedures to avoid similar issues from happening again in the future. 

If you have any further questions or concerns, please contact support@anaplan.com. Thank you for your patience during this situation and thank you for being an Anaplan customer.

Posted Aug 04, 2022 - 17:55 UTC

Resolved
We have confirmed that the issue is now resolved. We deeply apologize for any impact this issue may have caused. We appreciate your patience and partnership as we worked through this issue. We will follow up within 7 business days with a detailed root cause analysis (RCA) that will be shared on our Status Page. If you have any questions, concerns, or continue to experience errors, please do not hesitate to contact us at Anaplan Support.
Posted Aug 02, 2022 - 06:19 UTC
Monitoring
Service has now been restored; you should now be able to resume normal activities. Our engineers will continue to monitor the platform to ensure no additional issues arise. If you have any questions, concerns, or continue to experience issues, please do not hesitate to contact Anaplan Support.

We will provide a final update to you when we consider this situation fully resolved.
Posted Aug 02, 2022 - 05:57 UTC
Update
We have identified the likely cause of the issue, and we are focused right now on restoring your service as quickly as possible.

ESTIMATED TIME TO RESOLUTION: We estimate service will be restored in approximately 45 minutes.

*Please note the estimated time to resolution is an approximation based on information we have at this stage and is subject to change.
Posted Aug 02, 2022 - 05:23 UTC
Identified
We have identified the likely cause of the issue, and we are focused right now on restoring your service as quickly as possible. We will provide further updates in 30 minutes. Thank you for your patience while we work through this issue.
Posted Aug 02, 2022 - 04:53 UTC
Investigating
Thank you for your patience as we continue to investigate this issue.
We will continue to provide updates to you as we work to resolve this issue as quickly as possible.
If you have any questions or concerns, please do not hesitate to contact us at Anaplan Support.
Posted Aug 02, 2022 - 04:34 UTC
Update
We are diligently working to restore service related to recent customer reported issues. Additional updates will be posted to this page until service is restored or there is an estimated time to resolution.
Posted Aug 02, 2022 - 04:01 UTC
Identified
We have identified the issue preventing customers accessing the Anaplan Platform. Our engineers are actively working to restore service and determine the root cause. Updates will be posted to this page as they become available.
Posted Aug 02, 2022 - 03:32 UTC
Update
We are continuing to investigate the reported issues regarding Anaplan accessibility. We will provide regular updates on this page as we diligently work to resolve this issue(s). Please contact Anaplan Customer Care at support@anaplan.com if you continue to experience issues. Thank you for bearing with us.
Posted Aug 02, 2022 - 02:57 UTC
Investigating
We are investigating a potential issue impacting Anaplan Platform.
We will provide regular updates on this page until the issue is resolved.
Posted Aug 02, 2022 - 02:42 UTC
This incident affected: Anaplan Google Cloud - Dedicated (Authentication and Login).