Updates related to Incident 11874
Email Cluster B is Online
The issue with provisioning services on Cluster B has cleared. NOC has fully tested and confirmed that services are fully available.
Inbound and outbound mail was unaffected. Mailbox access was fully available.
Incident Summary: (updated Thursday, April 15, 2010: 17:30 ET / 21:30 UTC )
OpenSRS Email Services Cluster B had a brief provisioning issue from 16:55 ET/20:55 UTC - 17:16/21:16 ET. Resellers may have experienced provisioning timeouts.
Our Network Monitoring system (Nagios) alerted of the high queue of provisioning jobs. This large queue of provisioning changes resulted in threads to be overworked. This caused the timeouts. We cleared the provisioning queues. We tested services and confirmed online by 17:16/21:16 ET.
We have an existing project to restructure our provisioning systems. Part of this process is to improve the length of time required for all provisioning tasks.
Inbound/outbound mail and mailbox access (IMAP, POP, and Webmail) were unaffected.
This update is related to Incident 11874
Email Cluster B is Degraded
We are investigating an issue with our provisioning system. Resellers may encounter a server error.
All other email services including inbound/outbound mail and mailbox access are working well.
We will have more details shortly.
This update is related to Incident 11874