OpenSRS Status - Cluster B - IMAP/POP/Webmail

Cluster B - IMAP/POP/Webmail

Incident Report for OpenSRS

Postmortem

Incident Date: August 27, 2020
Incident Number: PR-1273

On August 27, 2020, at 2:34 PM ET, the Tucows Hosted Email platform experienced service interruptions impacting IMAP, POP and Webmail in cluster B.

The engineering team identified the issue with an unknown bug on a stable kernel version causing file system lockups.

At 3:56 PM ET, The Engineering team successfully restored the services by manually failing-over and upgrading the impacted device's file system.

At 5:36 PM ET, A second service interruption was observed and lasted for 12 minutes due to the same kernel systems bug.

On August 28, 2020 at 1:00 AM ET, Tucows performed emergency maintenance in order to further stabilize the email systems in cluster B.

Tucows is in contact with external vendors to investigate the cause and develop a plan to roll out a permanent solution to address the identified systems bug.

Thank you,

Tucows Engineering Team

Posted Aug 31, 2020 - 15:06 UTC

Resolved

The engineering team has successfully migrated the impacted services to the upgraded file system. Users should no longer experience any issues with regards to Cluster B

Incident Start Time: 08-27-2020 18:34:00 UTC
Incident End Time: 08-27-2020 19:56:00 UTC
Total Duration: 1 h 22 mins

Posted Aug 27, 2020 - 19:56 UTC

Update

The investigation into this Cluster B issue continues. We are examining the problem and solutions and hope to have some more answers in the near future.

Next Update: Within 15 mins

Posted Aug 27, 2020 - 19:50 UTC

Update

Our engineering team is continuing to investigate the root cause of this email issue. Customers should experience some service restoration.

Next Update: Within 15 mins

Posted Aug 27, 2020 - 19:22 UTC

Investigating

We are currently experiencing an issue where users on Cluster B may notice they are unable to log in or access their mailbox.

We are investigating and will provide updates as soon as they are available

Posted Aug 27, 2020 - 18:52 UTC

This incident affected: Hosted Email (Cluster B, Webmail).