Updates related to Incident 13843
Email Cluster A is Online
The event related to provisioning services on Mail Cluster A has been resolved. The root cause appears to have related to a power event at the data centre. We are still waiting for the data centre facilities team to identify the cause of the event and correct it.
In the meantime, we have connected alternative power to restore service. We may perform an emergency maintenance in the near future. Thank you for your patience, we apologize for the inconvenience.
Incident Summary:
Our Network Operations Centre monitoring received alarms for power failures at our Toronto data center. Our technical teams began their investigation at approximately 05:00 UTC and determined that email provisioning services for Cluster A were also degraded. Mailboxes via IMAP, POP and Webmail and inbound/outbound mail flow were unaffected. Provisioning changes such as adding new accounts and password changes were affected.
Our data center provider was engaged to investigate. We confirmed the scope of the issue and notified customers at 05:34 UTC. There was power failure in the data center which affected a rack of equipment. Email provisioning services were restored by 07:43 UTC. Due to the built in systems redundancy, email provisioning was the only affected service. There was a software defect which created an unnecessary dependency between the provisioning subsystem and some of the other subsystems which were affected by the power failure. This software defect will be corrected in one of the future email software releases.
This update is related to Incident 13843
Email Cluster A is Degraded
Our operations team is currently investigating an issue related to degraded performance. We're speaking with our data centre techs and expect to have some more details on this event soon.
Update:
06:16 UTC: a power event in our Toronto data centre has affected provisioning related to mail cluster A. Mail delivery is unaffected, but provisioning-related functions are unavailable. We've escalated the issue and are waiting on the DC facilities team to further investigate onsite.
6:57 UTC: one of our engineers has arrived at the data centre and investigating further. We expect a DC facilities technician to arrive within the hour (08:00 UTC) to further troubleshoot the electrical power issue.
This update is related to Incident 13843