Cluster B email authentication

Incident Report for OpenSRS

Postmortem

Incident Date: April 28, 2020
Incident Number: PR-1074

On April 28, 2020, at 10:56 AM ET, Tucows’ HostedEmail platform experienced issues with authentication service impacting IMAP, POP and Webmail in Cluster B. Tucows’ Engineering team was engaged to investigate the issue.

Upon investigation, Tucows’ Engineering team observed that the issue was due to an attack that exhausted the authentication connections impacting new logins and causing them to fail.

At 11:04 AM ET, the Engineering team started the mitigation process by restarting the authentication service and by blocking the offending traffic. Due to the continuous changes in the attack patterns, Tucows had to continuously amend the rules to block the new sources.

As part of the mitigation efforts, Tucows Engineering executed multiple emergency maintenance to improve the performance during the incident.

At 8:56 PM ET, The Engineering team was able to successfully block all the offending traffic and clear the backlog of login requests to restore the services successfully.

Tucows is to continue working on improving the overall security mitigation services to address and rectify future attacks in a timely manner.

‌

Thank you,

Tucows Engineering Team

Posted Apr 30, 2020 - 21:44 UTC

Resolved

Our team has implemented a solution and this issue should now be resolved.

Posted Apr 29, 2020 - 01:11 UTC

Update

Our operation team is working on the authentication subsystem and implemented a fix on the issue. Users should be able to login to the account via webmail/POP/IMAP now. However, some users may still experience slow logins. Our operation team continues to perform checks and monitor its status.

Posted Apr 28, 2020 - 22:25 UTC

Identified

The issue was identified and our team is working on a solution at this moment.

We will continue to provide further updates as they becomes available.

Posted Apr 28, 2020 - 20:45 UTC

Update

The Operations team is implementing a change that should help free up some connections slots for new connection requests.

Updates will be provided as soon as possible

Posted Apr 28, 2020 - 19:18 UTC

Update

As our Operations department continues to investigate the authentication issues on Cluster B, they have implemented some mitigation measures in place to help with the problem.

Updates will be provided as soon as they are available

Posted Apr 28, 2020 - 18:53 UTC

Update

Our Operations department is continuing to investigate the authentication issues users on Cluster B are experiencing.

Updates will be posted as soon as possible

Posted Apr 28, 2020 - 17:04 UTC

Investigating

We are currently experiencing an issue that is impacting (IMAP, POP, Cluster B, Webmail). We're experiencing authentication problems in cluster B that may affect end users, our operations team is investigating.

Client Impact: Some users might experience authentication issues when attempting to access POP, IMAP and webmail services.

Updates will be provided as soon as possible

Posted Apr 28, 2020 - 15:30 UTC

This incident affected: Hosted Email (Cluster B, Webmail).