Updates related to Incident 4743
Email Cluster A is Degraded
Currently, 50% of customer mailboxes on this cluster are fully available and those customers are able to send and receive email normally. Additionally, forward-only and filter-only accounts are also functioning and mail is being delivered. However, some filter-only users will be unable to log into the Spam quarantine.
The remaining 50% of customer mailboxes are offline and those users have no access to their mailboxes. Those customers are unable to send or receive mail at this time. Inbound mail is being queued for these customers and will be delivered once service is restored.
Affected users logging into webmail may see a "Service Unavailable" error message. Users with email clients will not be able to send or receive new mail. They will receive timeout errors.
We continue to monitor the rebuild of the affected mailstores. The rebuild of the first of the mailstores has completed as expected. Rebuilding of the next mailstore is now underway. We still anticipate that load across the system will prevent us from restoring those users on the rebuilt mailstore, but we will confirm this before making a final decision.
The current estimate for completion of the rebuild for those remaining mailstores is 10 P.M. EST.
We will keep you updated. The next update will be provided at approximately 2:30 P.M. EST.
This update is related to Incident 4743
Email Cluster A is Degraded
Currently, 50% of customer mailboxes on this cluster are fully available and those customers are able to send and receive email normally. Additionally, forward-only and filter-only accounts are also functioning and mail is being delivered. However, some filter-only users will be unable to log into the Spam quarantine.
The remaining 50% of customer mailboxes are offline and those users have no access to their mailboxes. Those customers are unable to send or receive mail at this time. Inbound mail is being queued for these customers and will be delivered once service is restored.
Affected users logging into webmail may see a "Service Unavailable" error message. Users with email clients will not be able to send or receive new mail. They will receive timeout errors.
We continue to monitor the rebuild of three affected mailstores. Currently, we estimate that sometime around 12 P.M. (noon) EST the first of the three mailstores will be restored. Based on load, we will make a decision as to whether to bring those users (an additional 25%) back online. We anticipate that load across the system will prevent us from restoring those users, but we will confirm this before making a final decision.
After that point, the rebuild will continue for the remaining mailstores. The current estimate for completion of the rebuild for those remaining mailstores is approximately 10 hours from the noon milestone (10 P.M. EST).
We will keep you updated. The next update will be provided at approximately 12:00 P.M. (noon) EST, but may be delayed slightly depending on the status of the mailbox restore.
This update is related to Incident 4743
Email Cluster A is Degraded
Cluster A is degraded. We were able to restore access to half of the customer mailboxes on this cluster. These customers are able to send and receive email. Forward-only accounts are functioning. The remaining half of the customers do not have access to their mailboxes. Inbound mail is being queued for these customers.
We continue to monitor the rebuild of the three affected mailstores. Currently, we estimate that between 11:00 - Noon EST another 25% of the mailstore will be restored, but will not be activated. Based on the load, we will make a decision as to whether to bring them back online or wait. We need to ensure that the rebuild speed is unaffected by this decision. The next stage of the rebuild, currently estimated to complete within 10 hours from the noon milestone, may put us in a position to fully restore the cluster to online status.
This update is related to Incident 4743
Email Cluster A is Offline
Cluster A is offline. All email services for Cluster A are unavailable. Incoming mail is being remotely queued. The rebuild of the three affected mailstores continues.
Our technical teams are working closely with our vendor, Netapp, to troubleshoot this issue. As mentioned previously, to alleviate stress on one functioning disk head, we are keeping all the mailstores offline for the time being. We are currently testing the unaffected mailstores and expect to have a more detailed up date within the hour about this progress. Our senior executive is engaged.
This update is related to Incident 4743
Email Cluster A is Offline
Cluster A is offline.
During our scheduled maintenance, the firmware upgrade of our NetApp caused a failure of the controlling disk head in the storage pool. The controlling disk head crashed which handed control over to the second head which triggered a rebuild of the disks.
At this point, all email services for Cluster A are offline. All incoming mail is being remotely queued.
We're working to rebuild the disks on three of the mailstores. To alleviate stress on one functioning head, we are keeping all the mailstores offline for the time being. Our technical leadership is engaged in this issue and working closely with NetApp, the storage device vendor. Our teams are in the data center actively working on this issue. We're doing everything we can to be bring users online as soon as possible.
This update is related to Incident 4743