Email Sending Error
Incident Report for OpenSRS
Postmortem

Incident Date: April 29, 2021
Incident Number: PR-1940

On April 29, 2021, at 6:30 PM ET. Tucows’ hosted email platform experienced service interruption impacting outbound emails for a subset of users in both Prod A and Prod B.

The service interruption was due to a scheduled and approved low risk change.

At 8:51 PM ET, The engineering team recovered the impacted services by partially rolling back the change.

Tucows is to revise and improve the change management process.

Tucows is to enhance the monitoring for better visibility. 

Thank you,

Tucows Engineering Team

Posted May 05, 2021 - 20:47 UTC

Resolved
The constant issue for email: 550 5.7.1 Message detected as spam under both of our clusters A and B is now resolved.


Incident Start Time: 04-29-2021 14:30:00
Incident End Time:04-29-2021 16:51:00
Total Duration:2 hours, 21 minutes
Posted Apr 30, 2021 - 01:47 UTC
Monitoring
Our team implemented a fix and we are currently monitoring the results.
Posted Apr 30, 2021 - 01:25 UTC
Update
Our team has implemented a fix and we are monitoring results from both clusters.

We will continue to provide updates as they become available.
Posted Apr 30, 2021 - 01:07 UTC
Update
Our team is still looking into the cause and we'll continue to update you on our progress.
Posted Apr 30, 2021 - 00:31 UTC
Investigating
It has been brought to our attention that a subset of users is seeing an error message when attempting to send an email: 550 5.7.1 Message detected as spam. We are working on isolating the issue and will give more information shortly.
Posted Apr 30, 2021 - 00:00 UTC
This incident affected: Hosted Email (Cluster A, Cluster B, Webmail).