OpenSRS Status

We resolved the issue which was causing provisioning delays and failures for OpenSRS Cluster B. All of your requests are now being processed. Our Ops team identified and addressed it. We continue our investigation, but will provide you with an incident summary by end of business day today.

All other services including mailbox access and mail flow were not affected.

Incident Summary (added 17:09 ET)

Our Network Operations Center (NOC) monitoring system received alerts for the provisioning system at starting at approximately 11:12 ET.  Our testing and log evaluation confirmed that there were issues with provisioning related to high connections.  We notified you at 11:56 ET.  Provisioning requests via the API were unavailable. Changes via the MAC were working. All other email services were functioning.

Operations identified a potential cause of the load and temporarily blocked an IP. The issue did not resolve. After more digging they identified and blocked an additional IP. These IPs were temporarily blocked help us further isolate the source. The high connection load on the provisioning system reduced significantly. Further analysis helped us determine the problem to be database related. Our engineering team successfully resolved it. The temporarily blocked IPs were restored. Service was tested and fully restored by 13:41 ET.

Our DBA team will continue their investigation into the root cause.

This update is related to Incident 7375