OpenSRS API Connection Errors

Incident Report for OpenSRS

Postmortem

Incident Date: September 26, 2021
Incident Number: PR-2392

On September 26, 2021 at 4:38 PM ET Tucows’ Domains platform experienced intermittent issues while accessing hover.com. Tucows Engineers were engaged and started investigating the issue.

The service interruption was caused due to an increase in the number of domain lookups performed.

On September 27, 2021 at 10:27 AM ET, Tucows’ engineering team increased the severity of the incident after we observed the external impact.

At 2:20 PM ET, The Engineering team recovered the services by rate-limiting the increased domain lookup traffic to stabilize the environment.

Tucows has implemented a permanent solution to prevent further interruption.

Tucows is to revise the architecture to further improve the overall security against high volume of traffic.

Thank you,

Tucows Engineering Team

Posted Oct 15, 2021 - 17:13 UTC

Resolved

We can confirm that services have been restored. This incident is now resolved.

Incident Start Time: 09-26-2021 20:38:15
Incident End Time: 09-27-2021 18:20:00
Total Duration: 21 hours, 42 minutes

Posted Sep 27, 2021 - 18:48 UTC

Identified

We have identified that this issue is related to domain lookups. We are continuing to investigate. Customers may experience issues with .UK renewals, registrations and transfers for the time being.

Posted Sep 27, 2021 - 17:21 UTC

Update

Engineering teams are still investigating the root cause of the issue still, we've implemented measures to allow orders to continue to pass. Some orders may still fail. Please retry when possible.

Posted Sep 27, 2021 - 16:26 UTC

Update

The Registry has confirmed there was no planned maintenance or ongoing issues. We have engaged our dev team for an investigation.

Posted Sep 27, 2021 - 14:49 UTC

Update

The Registry has engaged their technical teams for further investigation. we are awaiting a response from them. More details will be provided once we have them available.

Posted Sep 27, 2021 - 14:33 UTC

Update

We are continuing to investigate this issue.

Posted Sep 27, 2021 - 13:18 UTC

Update

We have reached out to the registry again and requested to investigate on their end. We will post updates once we have them available.

Posted Sep 27, 2021 - 13:16 UTC

Update

Please note that Registrations, renewals, transfers in and out would be impacted by this issue. We are still waiting to hear back from the registry.

Posted Sep 27, 2021 - 13:04 UTC

Update

The Engineering team has been engaged and we have also reached out to Nominet to further investigate this issue.

Posted Sep 27, 2021 - 11:57 UTC

Investigating

We are currently investigating intermittent errors when placing new registrations or renewal orders for .co.uk ccTLDs. We will add additional information once it is available.

Posted Sep 27, 2021 - 11:00 UTC

This incident affected: APIs (OpenSRS API) and Domain Services (Core ccTLDs).