Cluster B email authentication
Incident Report for OpenSRS

Incident Date: April 28, 2020
Incident Number: PR-1074

On April 28, 2020, at 10:56 AM ET, Tucows’ HostedEmail platform experienced issues with authentication service impacting IMAP, POP and Webmail in Cluster B. Tucows’ Engineering team was engaged to investigate the issue.

Upon investigation, Tucows’ Engineering team observed that the issue was due to an attack that exhausted the authentication connections impacting new logins and causing them to fail.

At 11:04 AM ET, the Engineering team started the mitigation process by restarting the authentication service and by blocking the offending traffic. Due to the continuous changes in the attack patterns, Tucows had to continuously amend the rules to block the new sources.

As part of the mitigation efforts, Tucows Engineering executed multiple emergency maintenance to improve the performance during the incident.

At 8:56 PM ET, The Engineering team was able to successfully block all the offending traffic and clear the backlog of login requests to restore the services successfully.

Tucows is to continue working on improving the overall security mitigation services to address and rectify future attacks in a timely manner.

Thank you,

Tucows Engineering Team

Posted Apr 30, 2020 - 21:44 UTC

Our team has implemented a solution and this issue should now be resolved.
Posted Apr 29, 2020 - 01:11 UTC
Our operation team is working on the authentication subsystem and implemented a fix on the issue. Users should be able to login to the account via webmail/POP/IMAP now. However, some users may still experience slow logins. Our operation team continues to perform checks and monitor its status.
Posted Apr 28, 2020 - 22:25 UTC
The issue was identified and our team is working on a solution at this moment.

We will continue to provide further updates as they becomes available.
Posted Apr 28, 2020 - 20:45 UTC
The Operations team is implementing a change that should help free up some connections slots for new connection requests.

Updates will be provided as soon as possible
Posted Apr 28, 2020 - 19:18 UTC
As our Operations department continues to investigate the authentication issues on Cluster B, they have implemented some mitigation measures in place to help with the problem.

Updates will be provided as soon as they are available
Posted Apr 28, 2020 - 18:53 UTC
Our Operations department is continuing to investigate the authentication issues users on Cluster B are experiencing.

Updates will be posted as soon as possible
Posted Apr 28, 2020 - 17:04 UTC
We are currently experiencing an issue that is impacting (IMAP, POP, Cluster B, Webmail). We're experiencing authentication problems in cluster B that may affect end users, our operations team is investigating.

Client Impact: Some users might experience authentication issues when attempting to access POP, IMAP and webmail services.

Updates will be provided as soon as possible
Posted Apr 28, 2020 - 15:30 UTC
This incident affected: Hosted Email (Cluster B, Webmail).