OpenSRS Reseller Control Panel / Registrations / Transfers

Incident Report for OpenSRS

Postmortem

Incident Date: February 27, 2020
Incident Number: PR-1755

‌

On February 27, 2021 at 3:21 AM ET, we experienced service interruption with Tucows’ Domains platform impacting accessibility issues with Resellers Control Panel.

The service interruption was caused due to a high load on a network storage device due to a disk failure.

At 6:37 AM ET, The engineering team gracefully restarted the affected server and replaced the failed drive to restore all the services successfully.

Tucows is to review and enhance monitoring for better visibility.

Tucows is to review and upgrade the system to ensure stability and mitigate further issues.

Thank you,

Tucows Engineering Team

Posted Mar 04, 2021 - 20:41 UTC

Resolved

Thank you for your patience. All services have now been restored.

Start Time 02-27-2021 08:21:00 UTC
End Time 02-27-2021 11:37:00 UTC
Total Duration:3 hour and 16 minutes

Posted Feb 27, 2021 - 11:41 UTC

Update

Our engineering team is still in process of correcting the issue. They are currently rebooting the database to rectify the situation. Further details will be provided once available. Thank you again for your patience.

Posted Feb 27, 2021 - 11:08 UTC

Update

We are continuing to work on a fix for this issue.

Posted Feb 27, 2021 - 10:52 UTC

Identified

Our engineering team has identified the problem and is working to resolve the issue. We thank you for your patience and will update as soon as we have more information.

Posted Feb 27, 2021 - 10:02 UTC

Investigating

We are currently experiencing an incident that is impacting OpenSRS Domains. We are experiencing issues with our OSRS/HRS environment.
This is impacting the ability to log in to the Reseller Control Panel as well as registrations and transfers.

We have already engaged our engineering team to investigate. We will provide an update as soon possible.

Posted Feb 27, 2021 - 08:49 UTC

This incident affected: Control Panels (Reseller Control Panel, Classic RWI, End User Control Panel, Storefront, Payment Gateway) and Reseller Support.