Asia Pacific - R3
Asia Pacific - API
: 33
Asia Pacific - Contact Imports
Asia Pacific - Dynamics Connector
: 92
Asia Pacific - Email to SMS
: 5
Asia Pacific - Integration Hub
: 192
Asia Pacific - Mail Sending
Asia Pacific - Open and Link Tracking
: 77
Asia Pacific - Pages and Forms
Asia Pacific - Reporting
Asia Pacific - SMS
Asia Pacific - Salesforce Connector
: 73
Asia Pacific - Surveys and Forms
: 95
Asia Pacific - Transactional Email
: 4
Asia Pacific - Web Application
: 50
Europe - R1
Europe - API
: 141
Europe - Contact Imports
Europe - Dynamics Connector
: 149
Europe - Email to SMS
: 9
Europe - Integration Hub
: 1025
Europe - Mail Sending
Europe - Open and Link Tracking
: 38
Europe - Pages and Forms
Europe - Product image resize
: 434
Europe - Reporting
Europe - Salesforce Connector
: 128
Europe - Surveys and Forms
: 199
Europe - Transactional Email
: 8
Europe - Web Application
: 96
Global CPaaS
CPaaS - API
: 173
CPaaS - Portal
: 208
North America - R2
North America - API
: 93
North America - Contact Imports
North America - Dynamics Connector
: 42
North America - Email to SMS
: 6
North America - Integration Hub
: 1343
North America - Mail Sending
North America - Open and Link Tracking
: 278
North America - Pages and Forms
North America - Reporting
North America - SMS
North America - Salesforce Connector
: 40
North America - Surveys and Forms
: 106
North America - Transactional Email
: 7
North America - Web Application
: 107
Global - Image Hosting
: 148
Global - Login Page
: 221
Global - Website
: 1428

Asia Pacific - R3 100 %
Asia Pacific - API 100 %
Asia Pacific - Contact Imports 100 %
Asia Pacific - Dynamics Connector 100 %
Asia Pacific - Email to SMS 100 %
Asia Pacific - Integration Hub 100 %
Asia Pacific - Mail Sending 100 %
Asia Pacific - Open and Link Tracking 100 %
Asia Pacific - Pages and Forms 100 %
Asia Pacific - Reporting 100 %
Asia Pacific - SMS 100 %
Asia Pacific - Salesforce Connector 100 %
Asia Pacific - Surveys and Forms 100 %
Asia Pacific - Transactional Email 100 %
Asia Pacific - Web Application 100 %
Europe - R1 100 %
Europe - API 100 %
Europe - Contact Imports 100 %
Europe - Dynamics Connector 100 %
Europe - Email to SMS 100 %
Europe - Integration Hub 100 %
Europe - Mail Sending 100 %
Europe - Open and Link Tracking 100 %
Europe - Pages and Forms 100 %
Europe - Product image resize 100 %
Europe - Reporting 100 %
Europe - Salesforce Connector 100 %
Europe - Surveys and Forms 100 %
Europe - Transactional Email 100 %
Europe - Web Application 100 %
Global CPaaS 100 %
CPaaS - API 100 %
CPaaS - Portal 100 %
North America - R2 100 %
North America - API 100 %
North America - Contact Imports 100 %
North America - Dynamics Connector 100 %
North America - Email to SMS 100 %
North America - Integration Hub 100 %
North America - Mail Sending 100 %
North America - Open and Link Tracking 100 %
North America - Pages and Forms 100 %
North America - Reporting 100 %
North America - SMS 100 %
North America - Salesforce Connector 100 %
North America - Surveys and Forms 100 %
North America - Transactional Email 100 %
North America - Web Application 100 %
Global - Image Hosting 100 %
Global - Login Page 100 %
Global - Website 100 %

Sep 25, 2023

Sep 24, 2023

Sep 23, 2023

Sep 22, 2023

Sep 21, 2023

Sep 20, 2023

Sep 19, 2023

Sep 18, 2023

Sep 17, 2023

Sep 16, 2023

Sep 15, 2023

Sep 14, 2023

Sep 13, 2023

Sep 12, 2023

Sep 11, 2023

Sep 10, 2023

Sep 09, 2023

Sep 08, 2023

Sep 07, 2023

Sep 06, 2023

Sep 05, 2023

Sep 04, 2023

Sep 03, 2023

Sep 02, 2023

Sep 01, 2023

Aug 31, 2023

Aug 30, 2023

Intermittent access to dotdigital in APAC region

( )

Asia Pacific - API(Asia Pacific - R3), Asia Pacific - Web Application(Asia Pacific - R3), Asia Pacific - Open and Link Tracking(Asia Pacific - R3), Asia Pacific - Surveys and Forms(Asia Pacific - R3), Asia Pacific - Mail Sending(Asia Pacific - R3), Asia Pacific - Reporting(Asia Pacific - R3), Asia Pacific - Pages and Forms(Asia Pacific - R3), Asia Pacific - Contact Imports(Asia Pacific - R3), Asia Pacific - SMS(Asia Pacific - R3), Asia Pacific - Transactional Email(Asia Pacific - R3), Asia Pacific - Integration Hub(Asia Pacific - R3), Asia Pacific - Email to SMS(Asia Pacific - R3), Asia Pacific - Dynamics Connector(Asia Pacific - R3), Asia Pacific - Salesforce Connector(Asia Pacific - R3)

Summary of impact:

At approximately 10:51 UTC on Wednesday 30th August 2023, services in our Asia Pacific region (R3) began to fail as a result of an Azure datacenter outage. We restored services at 16:30 UTC the same day.

During this time, customers in R3 will have been unable to use all aspects of our platform, including, but not limited to:

  • Issues logging into the platform.
  • Delayed:
    • Program execution.
    • Email and SMS sending.
  • A loss of reporting data from email, SMS and web behaviour.
  • Email and SMS links failing to load for contacts.
  • Pages and Forms were offline.
  • New Transactional Email couldn’t be submitted.
  • API failures.

Root Cause:

The Dotdigital platform is hosted by Microsoft’s Azure cloud and uses the Australia East and Australia Southeast regions. The Dotdigital platform began to fail after a power surge damaged cooling units in the Azure datacenter. This had the knock-on impact of Azure’s compute and storage systems being taken offline causing widespread disruption for their customers. You can read more on Azure’s status page.

During this incident, Dotdigital engineers attempted to restore service to our alternate location in Azure’s Australia Southeast region. Together Australia East and Southeast form a cross-region pair which is the recommended approach from Azure to achieve high availability. Our engineers discovered deployments in the Southeast region failed due to the Azure outage impacting the second region. Although this wasn’t confirmed on Azure’s public status page, it was reported on their customer-facing system health report.


Mitigation:

The timeline (in UTC) for resolving this issue was:
10:51 - Our monitoring detected the first signs of failing services.
11:11 - We raised an incident and began investigations.
11:24 - We created a status Page highlighting the issue with Azure.
11:31 - We disabled the send pipeline to protect the integrity of sends.
11:53 - We attempted to scale the healthy pool which failed as the Azure management node was unresponsive.
12:23 - We opened a high-priority ticket with the Azure support team.
12:31 - Azure confirmed the issue was related to their cooling issue.
14:20 - Our attempts to restore services via the Southeast region failed due to a secondary Azure issue.
14:45 - We opened a second high-priority ticket with the Azure support team regarding the Southeast region also being unresponsive.
15:09 - We received verbal confirmation from the Azure support team Australia Southeast was also degraded.
16:31 - The Azure issue was resolved and we started all Dotdigital services. Any scheduled sends submitted prior to the incident were immediately dispatched.


Next Steps:

We’ll be working with Azure to understand the events that took place on their side and in particular how two regions suffered in tandem. In addition, we’ll investigate the adoption of Azure’s Availability Zones and reduce the need for infrastructure deployments in our secondary regions.


Statement from John Adkins, Platform Engineering Director: 

We're really sorry for the trouble the outage may have caused you. We know service reliability is super important. We'll be working hand in hand with Azure to learn from this incident and take action to prevent it from happening again. Thanks for your patience while we resolved the issue, and we promise to keep working hard to provide you with great service.

Problems with Azure's datacentre have been resolved and all Dotdigital services are now back online. Queued sends are being dispatched immediately.

Our service is still being impacted by problems in Azure's Australia East region. In addition attempts to move systems to our alternate site in Australia Southeast have also failed due to Azure outages in this region. You can track the Azure issue via the following link https://azure.status.microsoft/en-us/status

We continue to work with our cloud provider Microsoft Azure to mitigate the issue. Azure have reported some success in their mitigation and is looking to restore services over the next few hours. You can track progress via the following link https://azure.status.microsoft/en-us/status

Issue identified with cloud provider. Users will continue to experience disruption to all dotdigital services including Sends, API, Programs and Web Behaviour Tracking.

Issue identified with cloud provider. Users will continue to experience intermittent disruption to all dotdigital services including Sends, API, Programs and Web Behaviour Tracking.

Aug 29, 2023

Aug 28, 2023

Aug 27, 2023

OR

OR

Privacy Policy / Terms of Use / Cookie Use Policy / Modern Slavery Act Transparency Statement / Dotdigital Investor Relations

Dotdigital is a trading name of dotdigital EMEA Limited (company number: 03762341) whose registered office is at No. 1 London Bridge, London, SE1 9BG.