On October 30, 2023, at 20:03 UTC, we identified an issue causing new authentication requests to fail when using the Anaplan platform across all regions. Customers already logged in to the Anaplan platform were not impacted.
Initial investigation identified that our third-party vendor, Cloudflare, was experiencing an outage that was impacting their customers across multiple regions. Cloudflare confirmed that the issue was resolved at 20:35 UTC. Once we completed internal testing to confirm that new authentication requests were working, we confirmed that service was restored by 20:50 UTC.
However, one of the backend services in the Anaplan Amazon Cloud – Europe region didn’t recover as expected. This impacted our customers’ ability to run API jobs within this region. We completed a rolling restart of the backend service and confirmed API service was restored at 11:36 UTC on October 31.
Further investigation identified that one of the backend components that triggered a failsafe mechanism during the vendor outage (intended to protect the platform in Anaplan Amazon Cloud – Europe) had stayed open and didn’t recover. We determined that this was a result of the default configuration failing to capture the lower level of traffic requests.
We apologize for any impact this issue might have had on your business operations. We're continuously strengthening our systems and procedures to ensure we avoid future disruption to your business and users.
To avoid a reoccurrence of this issue, we’ve enhanced our monitoring to ensure we proactively detect similar issues. We’ll also be making a change to the backend service to accommodate the low traffic levels in Anaplan Amazon Cloud – Europe to enable the failsafe mechanism to gracefully recover. And we’re working closely with our vendor to ensure they conduct the necessary mitigation to prevent such an outage in the future.
If you have further questions or concerns, please visit our Anaplan Support website. Thank you for your patience during this situation and thank you for being an Anaplan customer.