Email Cluster B Status Archive
Email Cluster B is Online
The issue with provisioning services on Cluster B has cleared. NOC has fully tested and confirmed that services are fully available.
Inbound and outbound mail was unaffected. Mailbox access was fully available.
Incident Summary: (updated Thursday, April 15, 2010: 17:30 ET / 21:30 UTC )
OpenSRS Email Services Cluster B had a brief provisioning issue from 16:55 ET/20:55 UTC - 17:16/21:16 ET. Resellers may have experienced provisioning timeouts.
Our Network Monitoring system (Nagios) alerted of the high queue of provisioning jobs. This large queue of provisioning changes resulted in threads to be overworked. This caused the timeouts. We cleared the provisioning queues. We tested services and confirmed online by 17:16/21:16 ET.
We have an existing project to restructure our provisioning systems. Part of this process is to improve the length of time required for all provisioning tasks.
Inbound/outbound mail and mailbox access (IMAP, POP, and Webmail) were unaffected.
This update is related to Incident 11874
Email Cluster B is Degraded
We are investigating an issue with our provisioning system. Resellers may encounter a server error.
All other email services including inbound/outbound mail and mailbox access are working well.
We will have more details shortly.
This update is related to Incident 11874
Email Cluster B is Online
Email Cluster B is Online
OpenSRS Email Service is fully available however statistics provided in the MAC control panel are not accurate. We are aware of the issue and are working to correct it. We will update Status when the data is corrected.
We apologize for the inconvenience.
Email Cluster B is Online
OpenSRS Email Services - Cluster B is fully online.
All provisioning requests via the Mail Administration Center (MAC) and Application protocol (APP) are being accepted and processed in normal time. The previous queue of change requests have been cleared.
Customer mailbox access and inbound/outbound mailbox were fully available during this incident.
Incident Summary (Updated 16:47 ET/20:47 UTC)
At about 02:15 ET our network monitoring system alerted us to issues with mailbox provisioning on Cluster B of our Email Service.
Analysis showed that provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) were degraded due to a high load on the provisioning system. Customer mailboxes remained accessible via IMAP, POP, and Webmail. Inbound and outbound mail were fully available.
By 04:07 ET we isolated the cause of the issue as a misconfigured provisioning script. The script put an unusually high load on the provisioning servers that caused normal provisioning requests to be queued.
At 04:24 ET we stopped all provisioning requests to investigate further. We cleared a invalid provisioning requests and restarted the provisioning system at 05:02 ET. All queued and new provisioning requests were then processed normally.
The provisioning queue cleared by 05:19 ET.
Our investigation shows that an incorrectly configured script caused this abnormally high load on the provisioning system. While we normally run such scripts during off-peak hours we are now refining our internal processes to ensure that these types of scripts receive greater scrutiny before being used.
We apologize for any inconvenience to you and your customers.
This update is related to Incident 8469
Email Cluster B is Degraded
We restarted the Provisioning system. All previously issued new and change requests are being processed. There is a backlog of provisioning requests. As it is not prime time business hours, these are processing fast. New change provisioning requests via the Mail Administration Center (MAC) and APP are being accepted for processing.
We identified the cause of the provisioning issue and addressed it. We will continue our investigation.
Customer mailboxes are fully available. Mailbox access was not affected by this issue. Inbound and outbound mail are also working well.
This update is related to Incident 8469
Email Cluster B is Degraded
OpenSRS - Email Services Cluster B provisioning services are offline. All other Email Services including mailbox access and mailflow are fully available and online. Customers can access their mailboxes via IMAP, POP and Webmail. Inbound and outbound mail flow are unaffected.
Provisioning System Unavailable (Offline)
Our technical teams continue to investigate a high load on the provisioning system. All provisioning changes will be queued. We have temporarily stopped the provisioning system. Changes via the Mail Administration Center (MAC) and Application protocol (APP) are unavailable at this time.
This update is related to Incident 8469
Email Cluster B is Degraded
OpenSRS Email Services - Cluster B provisioning services are degraded. Our operations team is investigating a high load of provisioning requests which is causing all new and change requests to be delayed. Provisioning changes will be queued for processing and be completed once this backlog of requests is flushed.
All mailboxes are available via IMAP, POP and Webmail. Inbound and Outbound mail are also unaffected.
Update 07:07 UTC:
We have identified the cause of the high load on the provisioning system.
Update 07:19 UTC:
The provisioning change request queue is slowly processing. There continues to be a backlog of change requests queued. Change requests submitted will be processed with some delay. We continue to work on addressing the cause of the high load on the provisioning system.
Update 07:56 UTC:
Our Technical teams are investigating the best solution to address the high load on the provisioning systems.
This update is related to Incident 8469
Email Cluster B is Online
The OpenSRS Email Services Cluster B maintenance is complete.
Our Technical Operations team finished the necessary work early. All Email Services are fully available. (The 3-hour scheduled window was announced to end at 11:00 UTC.)
Email Cluster B is In Maintenance
There is 3-hour network maintenance at our data center for OpenSRS Email Services Cluster B starting now (08:00 UTC).
Service Impact for Resellers:
We will use this time to upgrade our core routers to accommodate 10 Gigabit cards. To minimize the impact on your services, traffic will be routed through secondary routers. This action should limit the actual down time to approximately 20 - 30 minutes within the 3 hour window. OpenSRS Email Services via Webmail, IMAP and POP, inbound/outbound mail flow Provisioning via the Mail Administration Center (MAC) and the Application Protocol Interface (API) will be unavailable during that time.
Service Impact for End-users:
Within the window, customer will experience 20 - 30 minutes with no access to their mailbox via Webmail, IMAP, or POP. Their mail will be temporarily remotely queued for delivery during that same period.