On March 3, 2026, at 14:20 UTC, customers in our us1: Data Center - US East, us2: Data Center - US West, eu1: Data Center - Netherlands, and eu2: Data Center - Germany regions experienced issues with CloudWorks™ integrations. Customers experienced varying levels of impact depending on how frequently their integrations ran and when those runs occurred during the recovery period.
Root cause
During a CloudWorks update on March 3, a background process used for previous maintenance work was unintentionally triggered. This process cleared workflow configuration records from a back-end system. The workflow files remained intact, but the system needed to rebuild the configuration records before integrations could run.
Recovery
Our engineering team identified the issue and took immediate action across all affected regions:
us2: Data Center - US West - The system self-recovered as the workflow configuration records were rebuilt automatically. Resolved at 15:30 UTC.
eu1: Data Center - Netherlands - We ran a manual synchronization process to restore the missing workflow configuration records. Resolved at 18:08 UTC.
eu2: Data Center - Germany - We adjusted system settings and increased processing capacity to accelerate recovery. Resolved at 19:21 UTC.
us1: Data Center - US East - We applied multiple configuration changes and scaled our processing systems to handle the restoration workload. Resolved at 01:08 UTC on March 4.
Corrective and preventative actions
We are implementing the following actions to prevent recurrence:
We have deactivated the maintenance process that caused this issue and updated our procedures to ensure it only runs when explicitly needed. This prevents workflow configuration records from being cleared unintentionally.
We are implementing additional monitoring and alerts to detect when workflow configuration records are missing or incomplete. This enables us to identify and restore CloudWorks integrations before customers are impacted.
We are creating detailed recovery procedures for our teams to follow if similar configuration issues occur in the future. This enables faster restoration of CloudWorks integrations across all regions.
We are conducting a comprehensive assessment of our back-end systems to optimize how they handle configuration record restoration. This reduces recovery time if similar rebuilding processes are needed in the future.
We apologize for the impact this issue has had on your operations. We are committed to the improvements outlined above to prevent similar disruptions. If you have questions or concerns, please contact our Support team.