Updates related to Incident 12933
Storefront is Online
All OpenSRS Storefront services are online.
Incident Summary
2010-06-13 15:45 UTC: Our monitoring systems first alerted our Network Operations Centre staff to the presence of slow connections to a number of services and network systems.
2010-06-13 16:00 UTC: Operations staff became involved and began investigating the cause of the incident.
2010-06-13 16:15 UTC: A large number of alerts continued to show that core network components were overloaded, affecting connections to Email Cluster A, Domains/SSL Provisioning and Management, Managed DNS, Storefront. Monitoring probes where registering significant spikes of abnormal traffic across a number of our services. This spike in traffic saturated a number of core network components. The incident was labeled Critical and was escalated to the Executive level.
2010-06-13 16:15 - 20:00 UTC: Using various traffic mitigation strategies, our Operations staff managed to balance all incoming data streams to our datacenter, easing pressure on a number of core network components. After identification and analysis, network elements were configured to drop unwanted traffic at the network edges.
2010-06-13 20:00 UTC: Email Cluster A service returning to normal operation.
2010-06-13 20:15 UTC: Domain Service was set to degraded status, while further traffic mitigation work was performed.
2010-06-13 21:00 UTC: All services online.
This update is related to Incident 12933
SSL Service is Online
All OpenSRS SSL services are online.
Incident Summary
2010-06-13 15:45 UTC: Our monitoring systems first alerted our Network Operations Centre staff to the presence of slow connections to a number of services and network systems.
2010-06-13 16:00 UTC: Operations staff became involved and began investigating the cause of the incident.
2010-06-13 16:15 UTC: A large number of alerts continued to show that core network components were overloaded, affecting connections to Email Cluster A, Domains/SSL Provisioning and Management, Managed DNS, Storefront. Monitoring probes where registering significant spikes of abnormal traffic across a number of our services. This spike in traffic saturated a number of core network components. The incident was labeled Critical and was escalated to the Executive level.
2010-06-13 16:15 - 20:00 UTC: Using various traffic mitigation strategies, our Operations staff managed to balance all incoming data streams to our datacenter, easing pressure on a number of core network components. After identification and analysis, network elements were configured to drop unwanted traffic at the network edges.
2010-06-13 20:00 UTC: Email Cluster A service returning to normal operation.
2010-06-13 20:15 UTC: Domain Service was set to degraded status, while further traffic mitigation work was performed.
2010-06-13 21:00 UTC: All services online.
This update is related to Incident 12933
Domain Service is Online
All OpenSRS services are online.
Incident Summary
2010-06-13 15:45 UTC: Our monitoring systems first alerted our Network Operations Centre staff to the presence of slow connections to a number of services and network systems.
2010-06-13 16:00 UTC: Operations staff became involved and began investigating the cause of the incident.
2010-06-13 16:15 UTC: A large number of alerts continued to show that core network components were overloaded, affecting connections to Email Cluster A, Domains/SSL Provisioning and Management, Managed DNS, Storefront. Monitoring probes where registering significant spikes of abnormal traffic across a number of our services. This spike in traffic saturated a number of core network components. The incident was labeled Critical and was escalated to the Executive level.
2010-06-13 16:15 - 20:00 UTC: Using various traffic mitigation strategies, our Operations staff managed to balance all incoming data streams to our datacenter, easing pressure on a number of core network components. After identification and analysis, network elements were configured to drop unwanted traffic at the network edges.
2010-06-13 20:00 UTC: Email Cluster A service returning to normal operation.
2010-06-13 20:15 UTC: Domain Service was set to degraded status, while further traffic mitigation work was performed.
2010-06-13 21:00 UTC: All services online.
This update is related to Incident 12933
SSL Service is Degraded
OpenSRS SSL Services may continue to experience slowness or intermittent timeouts. Our technical teams continue to monitor and test the cause of the high loads which affected our service availability today. We will have another update within the hour.
We apologize for the inconvenience to you and your customers. We are working hard to resolve this as quickly as possible.
This update is related to Incident 12933
Storefront is Degraded
OpenSRS Storefront Services may continue to experience slowness or intermittent timeouts. Our technical teams continue to monitor and test the cause of the high loads which affected our service availability today. We will have another update within the hour.
We apologize for the inconvenience to you and your customers. We are working hard to resolve this as quickly as possible.
This update is related to Incident 12933
Domain Service is Degraded
OpenSRS SSL Services may continue to experience slowness or intermittent timeouts. Our technical teams continue to monitor and test the cause of the high loads which affected our service availability today. We will have another update within the hour.
We apologize for the inconvenience to you and your customers. We are working hard to resolve this as quickly as possible.
This update is related to Incident 12933
Domain Service is Degraded
OpenSRS Services may continue to experience slowness or intermittent timeouts. Our technical teams continue to monitor and test the cause of the high loads which affected our service availability today. We will have another update within the hour.
Affected services are:
Domains Provisioning and Management
SSL
Storefront
DNS
We apologize for the inconvenience to you and your customers. We are working hard to resolve this as quickly as possible.
This update is related to Incident 12933
Email Cluster A is Online
OpenSRS Email Services - Cluster A are available.
Incident Summary
2010-06-13 15:45 UTC: Our monitoring systems first alerted our Network Operations Centre staff to the presence of slow connections to a number of services and network systems.
2010-06-13 16:00 UTC: Operations staff became involved and began investigating the cause of the incident.
2010-06-13 16:15 UTC: A large number of alerts continued to show that core network components were overloaded, affecting connections to Email Cluster A, Domains/SSL Provisioning and Management, Managed DNS, Storefront. Monitoring probes where registering significant spikes of abnormal traffic across a number of our services. This spike in traffic saturated a number of core network components. The incident was labeled Critical and was escalated to the Executive level.
2010-06-13 16:15 - 20:00 UTC: Using various traffic mitigation strategies, our Operations staff managed to balance all incoming data streams to our datacenter, easing pressure on a number of core network components. After identification and analysis, network elements were configured to drop unwanted traffic at the network edges.
2010-06-13 20:00 UTC: Email Cluster A service returning to normal operation.
2010-06-13 20:15 UTC: Domain Service was set to degraded status, while further traffic mitigation work was performed.
2010-06-13 21:00 UTC: All services online.
This update is related to Incident 12933
Email Cluster A is Degraded
OpenSRS Cluster A are currently degraded. We have been experiencing a high traffic load for approximately 3 hours and 30 minutes. All of our technical teams are fully engaged and investigating.
Email Services including mailbox access (IMAP, POP, SMTP and webmail) are available. However,if you are using our DNS for your email services, you may experience degraded services. Provisioning may also be affected. Email Cluster B is unaffected.
Our senior executive team is engaged and working to obtain details. We apologize for the inconvenience to you and your customers.
PLEASE NOTE: Our main communication channels, including http://www.opensrsstatus.com, were affected during this issue. We will be providing messages via email and our twitter account http://twitter.com/@opensrsstatus will have regular updates.
The OpenSRS Team
This update is related to Incident 12933
Storefront is Offline
OpenSRS services are currently unavailable. We have been experiencing a high traffic load for approximately 3 hours and 30 minutes. All of our technical teams are fully engaged and investigating.
Services affected include:
Domains Provisioning and Management
Managed DNS
SSL
Storefront
The following services were affected, but are now working well. We will post these as degraded as a precaution while we closely monitor services.
Email Cluster A
If you are using our DNS for your email services, you may experience degraded services. Provisioning may also be affected. Email Cluster B is unaffected.
Our senior executive team is engaged and working to obtain details. We apologize for the inconvenience to you and your customers.
PLEASE NOTE: Our main communication channels, including http://www.opensrsstatus.com, were affected. We will be providing messages via email and our twitter account http://twitter.com/@opensrsstatus will have regular updates.
The OpenSRS Team
This update is related to Incident 12933