Cluster B: Email issues

Incident Report for OpenSRS

Postmortem

Incident Date: July 30, 2021
Incident Number: PR-2213

On July 30, 2021, at 12:53 AM ET, Tucows’ hosted email platform experienced service interruption impacting IMAP/POP/Webmail in Prod B.

The service interruption was due to a file system error on the network storage pair.

Tucows’ Engineering team increased the severity of the incident when we observed the external impact.

At 2:26 AM ET, The engineering team restarted the affected network storage and started to restore all the services.

At 2:50 PM ET, the engineering team restored all the services to stabilize the hosted email environment.

The Tucows Engineering team is to upgrade the core software on the affected systems to prevent this incident from happening again.

Tucows has revised and updated the triage and troubleshooting documentation to better identify the severity of the issue.

Thank you,

Tucows Engineering Team

Posted Aug 02, 2021 - 14:40 UTC

Resolved

Our engineering team have completed checks and confirmed all services are fully restored.

Incident Start Time: 07-30-2021 04:53:00 UTC
Incident End Time: 07-30-2021 06:50:00 UTC
Total Duration: ~1hr 57mins

Posted Jul 30, 2021 - 07:34 UTC

Monitoring

All mail services have now recovered properly after a successful manual restoration of mail service. We are currently monitoring the Cluster B mail service and follow with any updates or to close this incident.

Posted Jul 30, 2021 - 07:12 UTC

Identified

We are manually bringing services back online and will continue to post updates on this. Thank you for your patience.

Posted Jul 30, 2021 - 06:52 UTC

Update

We are continuing to investigate this issue.

Posted Jul 30, 2021 - 05:20 UTC

Investigating

We are experiencing an issue affecting Webmail, Pop and IMAP connections for users on Cluster B. Our engineering team is currently investigating this, and we will post updates as they are available.

Posted Jul 30, 2021 - 05:10 UTC

This incident affected: Hosted Email (Cluster B).