OpenSRS Status

All OpenSRS services are online.

Incident Summary

2010-06-13 15:45 UTC: Our monitoring systems first alerted our Network Operations Centre staff to the presence of slow connections to a number of services and network systems.

2010-06-13 16:00 UTC: Operations staff became involved and began investigating the cause of the incident.

2010-06-13 16:15 UTC: A large number of alerts continued to show that core network components were overloaded, affecting connections to Email Cluster A, Domains/SSL Provisioning and Management, Managed DNS, Storefront. Monitoring probes where registering significant spikes of abnormal traffic across a number of our services. This spike in traffic saturated a number of core network components. The incident was labeled Critical and was escalated to the Executive level.

2010-06-13 16:15 - 20:00 UTC: Using various traffic mitigation strategies, our Operations staff managed to balance all incoming data streams to our datacenter, easing pressure on a number of core network components. After identification and analysis, network elements were configured to drop unwanted traffic at the network edges.

2010-06-13 20:00 UTC: Email Cluster A service returning to normal operation.

2010-06-13 20:15 UTC: Domain Service was set to degraded status, while further traffic mitigation work was performed.

2010-06-13 21:00 UTC: All services online.

This update is related to Incident 12933