Cluster B - IMAP/POP/Webmail
Incident Report for OpenSRS

Incident Date: August 27, 2020
Incident Number: PR-1273

On August 27, 2020, at 2:34 PM ET, the Tucows Hosted Email platform experienced service interruptions impacting IMAP, POP and Webmail in cluster B.

The engineering team identified the issue with an unknown bug on a stable kernel version causing file system lockups. 

At 3:56 PM ET, The Engineering team successfully restored the services by manually failing-over and upgrading the impacted device's file system.

At 5:36 PM ET, A second service interruption was observed and lasted for 12 minutes due to the same kernel systems bug. 

On August 28, 2020 at 1:00 AM ET, Tucows performed emergency maintenance in order to further stabilize the email systems in cluster B.

Tucows is in contact with external vendors to investigate the cause and develop a plan to roll out a permanent solution to address the identified systems bug.


Thank you,

Tucows Engineering Team

Posted Aug 31, 2020 - 15:06 UTC

The engineering team has successfully migrated the impacted services to the upgraded file system. Users should no longer experience any issues with regards to Cluster B

Incident Start Time: 08-27-2020 18:34:00 UTC
Incident End Time: 08-27-2020 19:56:00 UTC
Total Duration: 1 h 22 mins
Posted Aug 27, 2020 - 19:56 UTC
The investigation into this Cluster B issue continues. We are examining the problem and solutions and hope to have some more answers in the near future.

Next Update: Within 15 mins
Posted Aug 27, 2020 - 19:50 UTC
Our engineering team is continuing to investigate the root cause of this email issue. Customers should experience some service restoration.

Next Update: Within 15 mins
Posted Aug 27, 2020 - 19:22 UTC
We are currently experiencing an issue where users on Cluster B may notice they are unable to log in or access their mailbox.

We are investigating and will provide updates as soon as they are available
Posted Aug 27, 2020 - 18:52 UTC
This incident affected: Hosted Email (Cluster B, Webmail).