Cluster B webmail/pop/imap not working

Incident Report for OpenSRS

Postmortem

Incident Date: March 9, 2020
Incident Number: PR-985

On March 9, 2020, at 2:00 PM ET, the Tucows HostedEmail platform experienced service degradation, impacting mailbox accessibility in prod B.

The service degradation was caused due to a high load on a network storage device.

At 4:40 PM ET, The Operations team restored all the services by stopping the replication process to alleviate and stabilize the load in the email environment.

‌

Preventive measures: As part of the ongoing stabilization efforts in PROD B; Tucows will continue the migration of mail stores on the affected hardware onto high-performance storage to prevent further client impact.

‌

Thank you,

Tucows Operations

Posted Mar 23, 2020 - 15:31 UTC

Resolved

This incident has been resolved.

Posted Mar 09, 2020 - 20:58 UTC

Monitoring

The load on our network file system has been fixed and users should no longer experience issues on Cluster B.

We will continue monitoring the situation to ensure the problem has been resolved.

Posted Mar 09, 2020 - 20:54 UTC

Identified

Our operations team has identified the root cause of the problem affecting Cluster B.

Service degradation is currently due to extreme load on one of our network file systems. Users on Cluster B that have their accounts on this file system will experience a service degradation.

Our operations teams are currently investigating the problems and looking for ways to alleviate this load. Some users should experience some service restoration already.

Updates will be provided as soon as possible

Posted Mar 09, 2020 - 20:42 UTC

Update

Our operations team continues to investigate an issue where some users on Cluster B may experience some trouble accessing their mailbox via webmail/pop/imap

Client Impact: Some customers will be unable to access their mailboxes via webmail/pop/imap.

Updates will be provided as soon as possible

Posted Mar 09, 2020 - 19:37 UTC

Investigating

We are currently investigating an issue where some users on Cluster B may have trouble loading their webmail

Client impact: Some users on Cluster B may experience slowness or trouble loading their webmail

Posted Mar 09, 2020 - 18:11 UTC

This incident affected: Hosted Email (Cluster B, Webmail).