Cluster A - Webmail Login, IMAP and POP
Incident Report for OpenSRS
Postmortem

Incident Date: August 23, 2019
Tracking Number: PR-527

On Aug 23, 2019, at 05:00 am EST Tucows experienced network degradation that partially impacted all of Tucows’ services running from the Canadian datacenter. The network incident was due to failure on a set of core network gear. The team engaged the vendor and performed a firmware update to mitigate the incident. The network incident impacted some of the traffic to the Hosted Email Prod A platform.

On August 23, 2019, at 11:59 PM EST, the network engineering team performed emergency maintenance to fully stabilize the network including the email platform services in prod A.

Tucows is currently testing new network gear and will complete hardware upgrade on the impacted network equipment to prevent this incident from happening again.

‌

Tucows Operations Team

Posted Aug 30, 2019 - 16:04 UTC
Resolved
Our Operations team has completed the configuration change to our network infrastructure. And we are pleased to report all services have been restored for Cluster A

In regards to the ability to access The Reseller Control Panel, opensrs.com and WHOis services.
The errors seen were false negatives due to some security measures put in place to protect our whois service.
This has now been resolved and all services have been restored.
Posted Aug 24, 2019 - 10:14 UTC
Update
We have also confirmed reports of some reseller being unable to load manage.opensrs.com and opensrs.com webpages which are related to the same issue earlier reported. The RCP for a small number of reseller may fail to load or time out. We will provide additional details as they become available.
Posted Aug 24, 2019 - 07:05 UTC
Update
Our Operations team has completed the configuration change to our network infrastructure
We are currently monitoring to verify these changes have corrected the intermittent connectivity issues experienced today. We will provide a further update shortly.
Posted Aug 24, 2019 - 05:53 UTC
Update
Our Operations team will be performing configuration change to our network infrastructure at 04:00 UTC. These changes should address the intermittent connectivity issues experienced today. We will continue to monitor and will provide further update accordingly.
Posted Aug 23, 2019 - 20:50 UTC
Monitoring
We are continuing to investigate this issue with Cluster A. Further updates will be provided as they become available.
Posted Aug 23, 2019 - 20:33 UTC
Update
Our investigation into users having issues with Cluster A continues.

The next update will be in approx. 60 mins or sooner if new information becomes available.
Posted Aug 23, 2019 - 19:19 UTC
Update
We are continuing to investigate this issue with Cluster A.

Our operations team are diagnosing the problem.

We will provide another update in approx. 60 mins.
Posted Aug 23, 2019 - 18:14 UTC
Update
We are still investigating this issue regarding Cluster A and users unable to access webmail and other email services.

We will provide our next update in approx. 60 mins or sooner if there is new information.
Posted Aug 23, 2019 - 17:11 UTC
Investigating
We are investigating an issue which some users on Cluster A may have trouble connecting to our Webmail, IMAP and POP services.

Our operations team is investigating and we will provide an update once more information is available.
Posted Aug 23, 2019 - 16:02 UTC
This incident affected: Hosted Email (Cluster A) and Control Panels (Reseller Control Panel).
← Current Status Powered by Atlassian Statuspage