Email Cluster A Status

Online

Updated Thursday, July 23rd, 2009 at 9:45 AM ET
2009-07-23 at 13:45 UTC - Other time zones

OpenSRS Email Cluster A is fully online.

The provisioning queue for Cluster A has cleared. All new changes via the API (Application) or the Mail Administration Center (MAC) are now processing as per normal.

Inbound/Outbound mail flow and mailbox access were not affected by this.

Incident Summary: (added at 17:05ET)
At 07:58 ET we began noticing unusual behaviour with the provisioning system. There were corresponding alerts on our Network Operations Center (NOC) monitoring. Our Operations team began investigating to determine the scope and cause. Application hosts were restarted and this did not resolve the issue. We advised you of a provisioning delay at 08:54ET. All changes via the API were being queued for approximately 45 minutes. There were two types of errors noted: pending and failing. Both types of requests were being queued.

Our Development team identified an issue with the Oracle libraries that provide connectivity to the provisioning database. The back-end hosts were restarted and this resolved the issue. There were approximately 600 provisioning requests queued. We fixed the errored provisioning requests. The provisioning queue cleared. Network Operations Center (NOC) and Technical Support staff fully tested and confirmed that corresponding alerts had stopped on our monitoring system and that all new provisioning requests were successful. The service was fully restored by 09:45ET and we posted it online.

This update is related to