Status archive for June, 2008
Email Cluster B is Online
OpenSRS Email Service customers on cluster B can now provision normally. All previously queued provisioning jobs are processing and will clear over the next few hours. We apologize for the inconvenience.
Incident Summary:
This problem was caused by an abnormally high number of jobs in the provisioning queue due to a customer migration into the cluster. Unfortunately when provisioning is done rapidly the provisioning database query doesn't complete in a reasonable amount of time. This is due to high load on servers, and the large number of rows.
To fix this, the development team temporarily stopped provisioning. Welcome messages for new customers were delayed until the provisioning queues were at a reasonable load. The DEV team will work with our professional services and product management teams to update our migration scripts to ensure this doesn't happen with future migrations into the cluster.
This update is related to Incident 306
Email Cluster B is Degraded
We have noted an issue with provisioning. It was escalated first to our Operations then Development team for handling. The queue of provisioning requests is higher than normal due to a customer migration. We are working to address the high load. No requests will be lost as we work to reduce the load.
This update is related to Incident 306