Intermittent DNS service interruption
Incident Report for OpenSRS
Postmortem

Incident Date: January 19, 2020
Incident Number: PR-839

On January 19 at 2:33 AM ET, Tucows’ domain platform experienced intermittent issues with the Domain Name System (DNS). The service degradation was due to a DDoS attack.

Tucows Engineering teams mitigated the issue by blocking the offending traffic. Due to the continuous changes in the attack patterns, Tucows had to continuously amend the rules to block the new traffic.

At 3:49 PM ET, the traffic subsided and the DNS services recovered.

Tucows is to continue working with existing vendors to improve DDoS mitigation services to address and rectify DDoS attacks in a timely manner.

Tucows is to revise the DNS architecture to further improve the overall security against malicious traffic.

Thank you,

Tucows Engineering Team

Posted Jan 27, 2020 - 16:56 UTC

Resolved
The SystemDNS issue has now been resolved. Domains using our nameservers should now properly resolve.
Posted Jan 19, 2020 - 22:35 UTC
Monitoring
A fix has been implemented on the issue and we will continue to monitor the situation to ensure no further issues arise
Posted Jan 19, 2020 - 21:33 UTC
Update
Our network operations team is working to resolve this situation. At this time some customers may experience intermittent DNS outage.
Posted Jan 19, 2020 - 17:31 UTC
Update
We have implemented some measures to mitigate the DNS resolution issues. Some customers may still see intermittent resolution issues while we continue to work.
Posted Jan 19, 2020 - 14:26 UTC
Update
Our Operations team continues their work to resolve the intermittent issues with our DNS service.

We will post updates as they become available.
Posted Jan 19, 2020 - 12:16 UTC
Update
DNS lookups continue to intermittently fail for customers. Our operations team will continue to try and isolate the root of the problem and provide updates when they are available.
Posted Jan 19, 2020 - 10:52 UTC
Update
We have attempted to mitigate the issue with small improvements, DNS lookups continue to intermittently fail for customers and our operations teams are still investigating at the moment.
Posted Jan 19, 2020 - 10:00 UTC
Investigating
We are currently investigating an issue impacting OpenSRS DNS service, it is intermittent and is effecting systemdns services and DNS lookup will fail intermittently for customers. We will provide updates as soon as possible.
Posted Jan 19, 2020 - 08:20 UTC
This incident affected: DNS (SystemDNS).