Hosted Email | Cluster B inbound mail delay.
Incident Report for OpenSRS
Postmortem

Incident Date: February 17, 2020
Incident Number: PR-890

On February 17, 2020, at 10:20 AM ET, Tucows HostedEmail platform experienced service degradation, email delivery delays, impacting inbound traffic in prod B.

The service degradation was caused by a high load on a network storage device due to a disk failure.

At 4:33 PM ET, The Operations team restored all the services and stabilized the load in the email environment. At 9:21 PM ET Operations team successfully completed the disk rebuild.

Preventive measures: As part of the ongoing stabilization efforts in PROD B; Tucows will expedite the migration of mail stores on the affected hardware onto high-performance storage to prevent further client impact.

Thank you,

Tucows Operations

Posted Feb 24, 2020 - 21:45 UTC

Resolved
No further issues have been reported. Thank you for your patience.
Posted Feb 17, 2020 - 22:47 UTC
Monitoring
The delays with inbound Email as well as the intermittent slowness on cluster B have been resolved. We will continue to monitor.
Posted Feb 17, 2020 - 22:13 UTC
Update
Our investigation continues regarding this Cluster B hosted email issue

Some users may also face intermittent slowness to login to Webmail/Imap/pop services on Cluster B

Updates will be provided as soon as possible
Posted Feb 17, 2020 - 19:39 UTC
Update
We are continuing to work on the issue affecting users on Cluster B

Updates will be provided as soon as possible
Posted Feb 17, 2020 - 18:20 UTC
Update
We are currently experiencing an issue with our Cluster B hosted email server.

Client Impact: Inbound mail will be delayed for Prod B users on
Posted Feb 17, 2020 - 16:08 UTC
Investigating
Degradation of service
Posted Feb 17, 2020 - 16:04 UTC
This incident affected: Hosted Email (Cluster B).