The engineering team identified a high volume of abusive domain lookups. They have blocked the abusive traffic and all the services have recovered successfully.
Incident Start Time: 09-24-2021 20:20:00 UTC Incident End Time: 09-25-2021 02:42:00 UTC Total Duration: 6 hours 22 minutes
Posted Sep 25, 2021 - 03:22 UTC
Update
We have added additional worker nodes to the platform and the team continues to investigate the issue.
Posted Sep 25, 2021 - 01:23 UTC
Update
The engineering team continues to investigate the intermittent issues and will provide more updates shortly.
Posted Sep 25, 2021 - 00:10 UTC
Identified
The engineering team has identified the source of the excessive traffic and implemented measures to limit the impact on our platform.
Posted Sep 24, 2021 - 23:08 UTC
Update
The engineering team is still investigating this issue. Further updates will be provided once available.
Posted Sep 24, 2021 - 21:43 UTC
Investigating
We are currently investigating reports of connection drops in our infrastructure. We will provide updates as they come. Customers might experience intermittent delays and connection failures while accessing OpenSRS and HRS control panels. Some API connections may also be impacted.
Posted Sep 24, 2021 - 20:43 UTC
This incident affected: APIs (OpenSRS API, OpenHRS API) and Control Panels (Reseller Control Panel, Classic RWI, End User Control Panel, Storefront).