On November 8, 2024, at 11:12 UTC, the Anaplan platform experienced an issue, preventing access to the platform. It was communicated that this issue impacted the below regions:  

• Anaplan Data Center — U.S. East 

• Anaplan Data Center — U.S. West 

• Anaplan Data Center — Germany 

• Anaplan Data Center — Netherlands 

• Anaplan Google Cloud Public — U.S. East 

• Anaplan Google Cloud Public — Japan 

• Anaplan Amazon Cloud Public — U.S. 

• Anaplan Amazon Cloud Public — Europe  

However, during the recovery activities, we completed testing and validation. We confirmed that the below regions hadn't been impacted and remained operational:  

• · Anaplan Google Cloud Public — U.S. East  

• · Anaplan Google Cloud Public — Japan  

• · Anaplan Amazon Cloud Public — U.S.  

• · Anaplan Amazon Cloud Public — Europe   

We quickly found that the issue was caused by a change made at 11:05 UTC to add three domain names into the domain name system (DNS). We immediately began to roll back the change. However, the automated rollback process couldn't be completed because of the issue. We quickly identified a manual approach, involving directly removing the new domain names for each region.   

By 12:03 UTC, Anaplan Data Center — U.S. East and Anaplan Data Center — Netherlands were accessible but experiencing degraded performance. At 12:13, Anaplan Data Center — Germany and Anaplan Data Center — U.S. West were both accessible but experiencing degraded performance.  

Due to the issue, CloudWorks service experienced a disruption, which caused a sub-service not to initialize on recovery in Anaplan Data Center - U.S. West and Anaplan Data Center — Netherlands. The CloudWorks sub-service was restarted to restore functionality.  

Full service was restored on the below regions at 12:23 UTC:  

• Anaplan Data Center — U.S. East   

• Anaplan Data Center — Netherlands   

• Anaplan Data Center — Germany  

• Anaplan Data Center — U.S. West at 12:37 UTC.  

As validations steps were being completed, we identified that the issue resulted in an error to a back-end component. This affected Google Cloud — Dedicated and Anaplan Google Cloud Public — Saudi Arabia. As a result, customers couldn't open new models in these regions. Any models that were already opened weren't affected. The error showed that a request to an internal service was failing. We restarted the affected component, and service was restored. The errors started to occur at 11:12 UTC and subsided at 12:52 UTC.  

We have conducted a thorough review of the incident. We identified that one of the new domain names had a faulty entry. To prevent a recurrence of this issue, the DNS service is currently undergoing an end-to-end review. This review will include how updates are made to the service and the testing procedure. We are also evaluating CloudWorks initialization process to prevent delays in recovery. In addition, we have reviewed the error in the back-end service seen in Anaplan Google Cloud — Dedicated and Anaplan Google Cloud Public — Saudi Arabia. It was identified that model openings in these regions were impacted by the back-end service caching the failed domain names. Monitoring is being enhanced to capture this faster in the future. A fix is also being made to stop this type of cache of failed domain names from happening.  

We sincerely apologize for the impact this has had on your customers' business operations. We know these issues can cause trouble for your customers' business and users. We are always improving our systems and procedures to prevent similar issues from happening again.   

If you have any further questions or concerns, please contact Anaplan Customer Care. Thank you for your patience during this situation and thank you for being an Anaplan customer.