On October 09, 2024, at 01:28 UTC, we saw an influx of API alerts across several internal microservices. This issue primarily affected Anaplan Data Center - U.S. East. However, other regions may also have experienced degradation: Anaplan Data Center - U.S. West, Anaplan Data Center — Netherlands, Anaplan Data Center — Germany, Anaplan Google Cloud Public — U.S. East, Anaplan Google Cloud Public — Japan, Anaplan Amazon Cloud Public — U.S. and Anaplan Amazon Cloud Public — Europe.
Initially, the impact seemed to be limited; however, at 03:20 UTC, platform performance had degraded, and a major incident was declared. During the incident, customers may have experienced slow behavior when interacting with API integrations and modeling services.
Our investigation established that there were errors presented from the infrastructure and network layers. Both areas were reviewed, and it was identified that the issue was caused by a faulty infrastructure component that wasn't handling traffic correctly. The component was isolated from the infrastructure, and full service was restored at 05:29 UTC.
We worked with our infrastructure vendor and found that the component had an unstable network link, which caused the intermittent errors. To prevent this issue from reoccurring, the vendor has reviewed the infrastructure and is in the process of replacing the faulty component. We are also working with the vendor to implement a new monitoring system. This will help to identify and resolve this type of issue before it results in degraded performance.
We sincerely apologize for the impact this has had on your business operations. We know these issues can cause trouble for your business and users. We are always improving our systems and procedures to prevent similar issues from happening again.
If you have any further questions or concerns, please contact Anaplan Customer Care. Thank you for your patience during this situation and thank you for being an Anaplan customer.