Cluster A Sending Issue

Incident Report for OpenSRS

Postmortem

https://www.opensrsstatus.com/incidents/6xcwcsjzj3z2

Incident Date: October 19, 2021 Incident Number: PR-2474

On October 19, 2021, at 1:32 PM ET, Tucows’ Hosted Email platform experienced service interruption impacting outbound emails for customers in Prod A.

The service interruption was caused due to human error where configurations were updated on a load balancer during migration efforts.

At 2:44 PM ET, The engineering team reverted the incorrect configuration and all the services were restored successfully.

Tucows is working on implementing a permanent fix to prevent future interruptions. Thank you,
Tucows Engineering Team

Posted Oct 22, 2021 - 14:04 UTC

Resolved

Our engineering team has resolved the matter and we are no longer reporting errors with sending mail.

Start Time: 17:32 PM UTC
Completed Time: 18:44 PM UTC
Total Duration: 1 hour and 12 minutes

Posted Oct 19, 2021 - 18:52 UTC

Investigating

We are experiencing a degradation in service for Hosted Email customers on cluster A. Users may experience issues with sending with error "SMTP Error (554)". Our Engineering team has been engaged and they are currently investigating the issue.

Posted Oct 19, 2021 - 18:19 UTC

This incident affected: Hosted Email (Cluster A).