The root cause has been identified as a missing Trust to OpenSRS connection which has been restored by Domains Engineering and the pending/stuck Trust orders have been resubmitted to the vendors. We are marking this incident as resolved.
Incident Start Time: 01-31-2024 02:43:00 UTC Incident End Time: 01-31-2024 19:50:00 UTC Total Duration: 17 hours, 7 minutes
Posted Jan 31, 2024 - 21:31 UTC
Investigating
Our engineers continue to investigate the possible root cause of stuck payments in the Reseller Control Panel. We will post updates once we know more.
Posted Jan 31, 2024 - 18:16 UTC
Update
The investigation by our engineers continues. Further updates will be provided.
Posted Jan 31, 2024 - 16:19 UTC
Update
While the Nomad resource usage has been fixed, customers are still seeing issues. On-call OSRS devs have been escalated to, and are currently investigating the issue further.
Posted Jan 31, 2024 - 04:49 UTC
Identified
Engineers have found a nomad job consuming too much CPU resources, which is negatively affecting the other jobs running on the nomad cluster. They are currently trying to bring CPU usage in line.
Posted Jan 31, 2024 - 03:56 UTC
Investigating
We are currently experiencing a Class III incident that is impacting Domains, (OpenSRS). We are getting reports of OpenSRS customers unable to submit Sectigo & Digicert certificate orders and adding funds to their accounts. We are escalating to engineers for further investigation.
Posted Jan 31, 2024 - 03:23 UTC
This incident affected: Control Panels (Reseller Control Panel, Payment Gateway) and SSL.