Summary of impact:
At approximately 07:30 UTC on 29th November 2023, we experienced issues processing unsubscribes and resubscribes in all regions. We restored services at 15:00 UTC on 30th November 2023.
During this time, customers may have experienced some of the following issues:
- Unable to resubscribe via confirmation email sent to a contact’s inbox
- Unable to unsubscribe via unsubscribe links and List-Unsubscribe headers
Root Cause:
During the bi-weekly platform updates, we updated how the tracking site processes some requests. This change was introduced to help us support more performant means of handling requests in the future.
Resubscribes:
The result of the change meant resubscribe emails were sent out with an invalid URL. Contacts were directed to an incorrect page when attempting to confirm their resubscription. Resubscribes were working incorrectly between 07:30 UTC 29th November and 15:00 UTC 30th November.
Unsubscribes:
A separate issue with that code meant we needed to roll back the tracking site on 29th November at 11:31 UTC. Other changes had modified the format of an unsubscribe message which is exchanged between the tracking app service and a background service. With those two services now out of sync, the message was processed incorrectly by the background service leading to failed unsubscribes. Unsubscribes were working incorrectly between 11:54 UTC 29th November and 12:32 UTC 30th November.
Mitigation:
The timeline (in UTC) for resolving this issue was:
29th November:
- 07:30 - Our regular deployment introduced the resubscribe issue
- 11:31 - We noticed a small number of errors related to product notification verification and chat verification. We decided to roll back the Tracking app
- 11:54 - We completed the rollback of the Tracking app, which introduced the unsubscribe issue
30th November:
- 12:09 - We raised an internal incident when we noticed it wasn’t possible to unsubscribe via signup forms
- 12:32 - We rolled forward the Tracking app to restore unsubscribes
- 13:03 - We noticed the resubscribe issue
- 13:29 - We rolled back the tracking app and event parser to restore resubscribes. This didn’t resolve the issue.
- 13:45 - We identified the underlying problem and began creating a fix
- 14:36 - We validated the fix in a staging environment
- 15:00 - We deployed the fix for the resubscribe issue to production
Next Steps:
We’ve exported a list of requests likely to have resulted in a failed unsubscribe action. Customers can request a list of the requests relevant to their accounts by contacting our Privacy Team at privacy@dotdigital.com.
To prevent a reoccurrence of these issues, we’ll:
- Improve our documentation on how to safely alter serialized data to make sure services can be deployed and rolled back independently without issue
- Introduce testing for this functionality in QA pre-release test cases
