Updates related to Incident 7521
Email Cluster A is Online
OpenSRS Email Cluster A is fully online.
The provisioning queue for Cluster A has cleared. All new changes via the API (Application) or the Mail Administration Center (MAC) are now processing as per normal.
Inbound/Outbound mail flow and mailbox access were not affected by this.
Incident Summary: (added at 17:05ET)
At 07:58 ET we began noticing unusual behaviour with the provisioning system. There were corresponding alerts on our Network Operations Center (NOC) monitoring. Our Operations team began investigating to determine the scope and cause. Application hosts were restarted and this did not resolve the issue. We advised you of a provisioning delay at 08:54ET. All changes via the API were being queued for approximately 45 minutes. There were two types of errors noted: pending and failing. Both types of requests were being queued.
Our Development team identified an issue with the Oracle libraries that provide connectivity to the provisioning database. The back-end hosts were restarted and this resolved the issue. There were approximately 600 provisioning requests queued. We fixed the errored provisioning requests. The provisioning queue cleared. Network Operations Center (NOC) and Technical Support staff fully tested and confirmed that corresponding alerts had stopped on our monitoring system and that all new provisioning requests were successful. The service was fully restored by 09:45ET and we posted it online.
This update is related to Incident 7521
Email Cluster A is Degraded
Our Development and Operations team identified the provisioning problem to be related to behaviour of a particular provisioning device. They restarted it. Your provisioning changes via the API, which were previously queued, are being processed. We are closely monitoring all the provisioning systems. There is a queue of requests which need to be processed.
You can make provisioning changes via the Mail Administration Center (MAC) and API. Please note that there are still delays with changes via the API.
All other Email Cluster A services are working well.
This update is related to Incident 7521
Email Cluster A is Degraded
Our Development team is looking into the issue. Provisioning via the API was working after the maintenance window earlier this morning. Your provisioning requests sent in the last 30 minutes have been queued and are delayed.
Resellers can make provisioning changes such as password changes and creating new accounts via the Mail Administration Center (MAC).
This update is related to Incident 7521
Email Cluster A is Degraded
OpenSRS Cluster A provisioning changes via the APP have been delayed for the last 20 minutes. You can make changes via the Mail Adminstration Center (MAC) Our Operations team is investigating.
All other services for Email including mailflow (inbound and outbound) and mailbox access are working well.
This update is related to Incident 7521