Email Cluster B Status

Online

Updated Wednesday, April 14th, 2010 at 5:16 PM ET
2010-04-14 at 21:16 UTC - Other time zones

The issue with provisioning services on Cluster B has cleared. NOC has fully tested and confirmed that services are fully available.

Inbound and outbound mail was unaffected. Mailbox access was fully available.

Incident Summary: (updated Thursday, April 15, 2010:    )
OpenSRS Email Services Cluster B had a brief provisioning issue from 16:55 ET/20:55 UTC - 17:16/21:16 ET.  Resellers may have experienced provisioning timeouts.
Our Network Monitoring system (Nagios) alerted of the high queue of provisioning jobs. This large queue of provisioning changes resulted in threads to be overworked. This caused the timeouts.  We cleared the provisioning queues. We tested services and confirmed online by 17:16/21:16 ET.
We have an existing project to restructure our provisioning systems. Part of this process is to improve the length of time required for all provisioning tasks.
Inbound/outbound mail and mailbox access (IMAP, POP, and Webmail) were unaffected.

Incident Summary: (updated Thursday, April 15, 2010:  17:30 ET / 21:30 UTC  )

OpenSRS Email Services Cluster B had a brief provisioning issue from 16:55 ET/20:55 UTC - 17:16/21:16 ET.  Resellers may have experienced provisioning timeouts.

Our Network Monitoring system (Nagios) alerted of the high queue of provisioning jobs. This large queue of provisioning changes resulted in threads to be overworked. This caused the timeouts.  We cleared the provisioning queues. We tested services and confirmed online by 17:16/21:16 ET.

We have an existing project to restructure our provisioning systems. Part of this process is to improve the length of time required for all provisioning tasks.

Inbound/outbound mail and mailbox access (IMAP, POP, and Webmail) were unaffected.

This update is related to