On June 13th at 19:09 UTC we were alerted by our internal systems that the unprocessed message queue was above threshold. We were also unable to log in to the Secure Email Threat Defense portal in the US region.
A major outage of AWS Lambda services in the us-east-1 region created an outage in the Secure Email Threat Defense service (Message processing and UI). During this time, we were also unable to log in to the AWS console.
We issue a sincere apology for the disruption this issue caused.
| Date/Time (UTC) | Comments |
|---|---|
| 19:09 June 13, 2023 | Issue detected by internal systems and alerted Secure Email Threat Defense engineer, who then acknowledged and started working on the issue. |
| 20:07 June 13, 2023 | Secure Email Threat Defense team identified the root cause and sent a customer email notification. |
| 20:30 June 13, 2023 | As there was no clear ETA from the AWS side on the resolution, Secure Email Threat Defense team initiated the Business Continuity Plan (BCP). |
| 20:56 June 13, 2023 | Secure Email Threat Defense team was able to log back into the Secure Email Threat Defense portal in the US region. We observed an improvement in Lambda error rates and processing Queues. |
| 21:23 June 13, 2023 | Secure Email Threat Defense team sent an updated email notification to customers. |
AWS experienced a region-level failure which impacted ETD's ability to process and remediate emails.
We are actively investigating the option of implementing an immediate failover strategy to a backup region in order to bolster the resilience of our systems. By doing so, we aim to effectively mitigate similar issues that may arise in the future.