Updates related to Incident 7375
Storefront is Online
Storefront services are online. We resolved the provisioning system issue which was preventing email forwarding provisioning requests. We re-processed the queue of these requests and all are now functioning well.
All other Storefront services were unaffected by this issue.
Incident Summary (added 17:07 ET)
We identified an increase of provisioning errors for email forwarding requests. Customers were informed at 13:07 ET. There was an issue with our provisioning system which also affected this one item with Storefront. All other Storefront services were online.
Operations identified a potential cause of the load and temporarily blocked an IP. The issue did not resolve. After more digging they identified and blocked an additional IP. These IPs were temporarily blocked help us further isolate the source. The high connection load on the provisioning system reduced significantly. Further analysis helped us determine the problem to be database related. Our engineering team successfully resolved it. The temporarily blocked IPs were restored. Service was tested and fully restored by 13:46 ET.
Our DBA team will continue their investigation into the root cause.
This update is related to Incident 7375
Email Cluster B is Online
We resolved the issue which was causing provisioning delays and failures for OpenSRS Cluster B. All of your requests are now being processed. Our Ops team identified and addressed it. We continue our investigation, but will provide you with an incident summary by end of business day today.
All other services including mailbox access and mail flow were not affected.
Incident Summary (added 17:09 ET)
Our Network Operations Center (NOC) monitoring system received alerts for the provisioning system at starting at approximately 11:12 ET. Our testing and log evaluation confirmed that there were issues with provisioning related to high connections. We notified you at 11:56 ET. Provisioning requests via the API were unavailable. Changes via the MAC were working. All other email services were functioning.
Operations identified a potential cause of the load and temporarily blocked an IP. The issue did not resolve. After more digging they identified and blocked an additional IP. These IPs were temporarily blocked help us further isolate the source. The high connection load on the provisioning system reduced significantly. Further analysis helped us determine the problem to be database related. Our engineering team successfully resolved it. The temporarily blocked IPs were restored. Service was tested and fully restored by 13:41 ET.
Our DBA team will continue their investigation into the root cause.
This update is related to Incident 7375
Storefront is Degraded
Storefront (Shopco.com) provisioning requests for email forwarding are not working. We are queuing your requests until we resolve an issue with the provisioning system.
All other Storefront services are online and working well.
This update is related to Incident 7375
Email Cluster B is Degraded
Our Operations team is working to identify the cause of high connections on OpenSRS Cluster B provisioning system. We addressed one issue, but the primary issue continues. At this time, provisioning changes via your API may not work. You can make changes directly in the MAC.
This update is related to Incident 7375
Email Cluster B is Degraded
Our Operations team isolated the issue which was causing provisioning change delays. We took action to address it and are closely monitoring the systems for further issues. We will keep you posted.
Again, all mailbox and mail flow is working well. You can make provisioning changes directly via the MAC.
This update is related to Incident 7375
Email Cluster B is Degraded
We are investigating a high load on the OpenSRS Cluster B provisioning system. Depending on how you have integrated your provisioning API, you may notice delays or timeouts for your provisioning requests. Changes may be queued based on your use of the provisioning system. Provisioning changes directly done in the Mail Administration Center (MAC) are being accepted.
All mailbox access via webmail, POP and IMAP is functioning. Mail flow (inbound and outbound) is also unaffected.
This update is related to Incident 7375