OpenSRS Status - Cluster B

Cluster B - Webmail Login

Incident Report for OpenSRS

Postmortem

Incident Date: June 24, 2020
Incident Number: PR-1185

On June 24, 2020, at 3:45 PM ET, the Tucows HostedEmail platform experienced service interruptions impacting IMAP, Inbound/outbound mail deliveries, POP and Webmail accessibility in cluster B.

The service interruption was caused by a software bug on the newly installed hardware that was not exposed during testing.

At 7:43 PM ET, The Engineering team successfully executed an emergency maintenance to implement the fixes by upgrading the core software on the affected systems. All the services were successfully restored.

Tucows is in the process of planning additional maintenance on new hardware with the similar software bug and implement the fixes to prevent future interruptions.

Thank you,

Tucows Operations

Posted Jun 26, 2020 - 19:34 UTC

Resolved

Email services continue to run smoothly and services are fully restored.

Incident Start Time: 06-24-2020 15:45:00 EST
Incident End Time:06-24-2020 19:43:00 EST

Total Duration: 4 hours

Posted Jun 24, 2020 - 23:54 UTC

Monitoring

Our operation team has successfully completed the emergency maintenance. The services are back online and users will be able to successfully log in. The users will experience delays in their inbound email delivery as the engineering team continues to enable email delivery services.

Posted Jun 24, 2020 - 22:42 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jun 24, 2020 - 21:25 UTC

Update

Mitigation efforts continue as maintenance is underway. We should expect downtime for users in Cluster B impacted by this outage for the next 20 minutes.

More updates to follow shortly thereafter.

Posted Jun 24, 2020 - 21:14 UTC

Identified

Our operations team is performing emergency maintenance at this time to mitigate the impact for users on Cluster B affected by this outage.

Posted Jun 24, 2020 - 20:49 UTC

Investigating

We are investigating an issue preventing some users on Cluster B from logging into Webmail. Our operations team has been engaged.

We will provide an update once we have additional information.

Posted Jun 24, 2020 - 20:07 UTC

This incident affected: Hosted Email (Cluster B, Webmail).