Cluster B connection issue

Incident Report for OpenSRS

Postmortem

Incident Date: March 2, 2022
Incident Number: PR-2896

On March 2, 2022 at 12:48 PM EST, Tucows’ Hosted Email platform experienced service interruption impacting Webmail in Prod B. Tucows’ Engineering team was engaged to investigate the issue.

The service interruption was caused due to a DDoS attack. The abusive traffic was mitigated however it caused unexpected behaviour with one of our load balancers.

The engineering team failover the services to the secondary load balancer and restarted webmail services in a controlled manner to restore all the services successfully.

Tucows is to continue working with existing vendors to improve DDoS mitigation services to address and rectify DDoS attacks in a timely manner.

Tucows is to work with the vendor to investigate the root cause of the unexpected behaviour on the load balancer.

Thank you,

Tucows Engineering Team

Posted Mar 23, 2022 - 19:36 UTC

Resolved

All services have been restored to normal after restarting the webmail services. This incident has been resolved.

Incident Start Time: 03-02-2022 17:48:00 UTC
Incident End Time: 03-02-2022 19:08:00 UTC
Total Duration: 1 hr 20 minutes

Posted Mar 02, 2022 - 20:43 UTC

Monitoring

All services have been recovered and we are monitoring the situation.

Posted Mar 02, 2022 - 19:29 UTC

Update

The engineering team has restarted the webmail services. We will provide further updates as they become available.

Posted Mar 02, 2022 - 19:13 UTC

Investigating

We are currently experiencing an issue that is impacting Hosted email (Webmail, IMAP). The engineering team has been engaged and is currently investigating the issue. During this time, all email services in cluster B will be impacted during the outage.

Posted Mar 02, 2022 - 18:14 UTC

This incident affected: Hosted Email (Cluster B).