Updates related to Incident 12784
Domain Service is Online
OpenSRS Domains Service is now fully restored.
Incident Summary:
2010-06-05 17:15 UTC: Our monitoring systems first alerted our Network
Operations Centre staff to the presence of slow connections to a
number of services and network systems .
2010-06-05 17:30 UTC: Operations staff become involved and started
working on the incident.
2010-06-05 17:45 UTC: A large number of alerts continued to show that
core network components were overloaded. Incoming traffic to our
Canadian datacenter location was more than quadruple the normal peak
traffic level. This spike in traffic saturated our upstream links. The
incident was labeled Critical and was escalated to the Executive
level.
2010-06-05 18:00 - 20:00 UTC: Using various traffic routing strategies
over primary and backup data links, our Operations staff managed to
balance all incoming data streams to our datacenter, easing pressure
on a number of core network components. After identification and
analysis, network elements were configured to drop unwanted traffic at
the network edges.
2010-06-05 20:00 - 22:00 UTC: Services delivered out of our Canadian
Toronto Datacenter were returning to normal.
2010-06-05 22:00 - 00:00 EDT: Operations staff noted that the
intensity of the unwanted incoming traffic was decreasing.
2010-06-06 01:00 UTC: Operations staff noted that the attack signature
disappeared completely from incoming data streams.
This update is related to Incident 12784
Domain Service is Degraded
OpenSRS Domains Service is now restored with the exception of Domain Forwarding.
We are keeping service Status at Degraded at this time while we work on fully restoring the complete service.
An incident report will be provided by 21:00 UTC Monday.
This update is related to Incident 12784
Storefront is Online
OpenSRS Storefront is fully online. We have resolved the issue and will continue to monitor the service.
Incident Summary:
2010-06-05 17:15 UTC: Our monitoring systems first alerted our Network
Operations Centre staff to the presence of slow connections to a
number of services and network systems .
2010-06-05 17:30 UTC: Operations staff become involved and started
working on the incident.
2010-06-05 17:45 UTC: A large number of alerts continued to show that
core network components were overloaded. Incoming traffic to our
Canadian datacenter location was more than quadruple the normal peak
traffic level. This spike in traffic saturated our upstream links. The
incident was labeled Critical and was escalated to the Executive
level.
2010-06-05 18:00 - 20:00 UTC: Using various traffic routing strategies
over primary and backup data links, our Operations staff managed to
balance all incoming data streams to our datacenter, easing pressure
on a number of core network components. After identification and
analysis, network elements were configured to drop unwanted traffic at
the network edges.
2010-06-05 20:00 - 22:00 UTC: Services delivered out of our Canadian
Toronto Datacenter were returning to normal.
2010-06-05 22:00 - 00:00 EDT: Operations staff noted that the
intensity of the unwanted incoming traffic was decreasing.
2010-06-06 01:00 UTC: Operations staff noted that the attack signature
disappeared completely from incoming data streams.
This update is related to Incident 12784
SSL Service is Online
All OpenSRS SSL Service is fully online. We have resolved the issue and will continue to monitor the service.
Incident Summary:
2010-06-05 17:15 UTC: Our monitoring systems first alerted our Network
Operations Centre staff to the presence of slow connections to a
number of services and network systems .
2010-06-05 17:30 UTC: Operations staff become involved and started
working on the incident.
2010-06-05 17:45 UTC: A large number of alerts continued to show that
core network components were overloaded. Incoming traffic to our
Canadian datacenter location was more than quadruple the normal peak
traffic level. This spike in traffic saturated our upstream links. The
incident was labeled Critical and was escalated to the Executive
level.
2010-06-05 18:00 - 20:00 UTC: Using various traffic routing strategies
over primary and backup data links, our Operations staff managed to
balance all incoming data streams to our datacenter, easing pressure
on a number of core network components. After identification and
analysis, network elements were configured to drop unwanted traffic at
the network edges.
2010-06-05 20:00 - 22:00 UTC: Services delivered out of our Canadian
Toronto Datacenter were returning to normal.
2010-06-05 22:00 - 00:00 EDT: Operations staff noted that the
intensity of the unwanted incoming traffic was decreasing.
2010-06-06 01:00 UTC: Operations staff noted that the attack signature
disappeared completely from incoming data streams.
This update is related to Incident 12784
Email Cluster A is Online
All OpenSRS Cluster A is fully online. We have resolved the issue and will continue to monitor the service.
Incident Summary:
2010-06-05 17:15 UTC: Our monitoring systems first alerted our Network
Operations Centre staff to the presence of slow connections to a
number of services and network systems .
2010-06-05 17:30 UTC: Operations staff become involved and started
working on the incident.
2010-06-05 17:45 UTC: A large number of alerts continued to show that
core network components were overloaded. Incoming traffic to our
Canadian datacenter location was more than quadruple the normal peak
traffic level. This spike in traffic saturated our upstream links. The
incident was labeled Critical and was escalated to the Executive
level.
2010-06-05 18:00 - 20:00 UTC: Using various traffic routing strategies
over primary and backup data links, our Operations staff managed to
balance all incoming data streams to our datacenter, easing pressure
on a number of core network components. After identification and
analysis, network elements were configured to drop unwanted traffic at
the network edges.
2010-06-05 20:00 - 22:00 UTC: Services delivered out of our Canadian
Toronto Datacenter were returning to normal.
2010-06-05 22:00 - 00:00 EDT: Operations staff noted that the
intensity of the unwanted incoming traffic was decreasing.
2010-06-06 01:00 UTC: Operations staff noted that the attack signature
disappeared completely from incoming data streams.
This update is related to Incident 12784
Storefront is Degraded
Beginning on 2010-06-05 at 17:51 UTC UTC the OpenSRS network experienced extremely high load that overloaded our network and one of our upstream providers. Since then we have re-distributed load across a number of providers and we believe we have identified the cause of the load and mitigated its impact.
We have restored intermittent service at this time but are keeping service Status at Degraded at this time while we continue to monitor and make adjustments to fully restore service.
An incident report will be provided by 21:00 UTC Monday.
This update is related to Incident 12784
SSL Service is Degraded
Beginning on 2010-06-05 at 17:51 UTC UTC the OpenSRS network experienced extremely high load that overloaded our network and one of our upstream providers. Since then we have re-distributed load across a number of providers and we believe we have identified the cause of the load and mitigated its impact.
We have restored intermittent service at this time but are keeping service Status at Degraded at this time while we continue to monitor and make adjustments to fully restore service.
An incident report will be provided by 21:00 UTC Monday.
This update is related to Incident 12784
Domain Service is Degraded
Beginning on 2010-06-05 at 17:51 UTC UTC the OpenSRS network experienced extremely high load that overloaded our network and one of our upstream providers. Since then we have re-distributed load across a number of providers and we believe we have identified the cause of the load and mitigated its impact.
We have restored intermittent service at this time but are keeping service Status at Degraded at this time while we continue to monitor and make adjustments to fully restore service.
An incident report will be provided by 21:00 UTC Monday.
This update is related to Incident 12784
Storefront is Offline
Beginning on 2010-06-05 at 17:51 UTC UTC the OpenSRS network experienced extremely high load that overloaded our network and one of our upstream providers. Since then we have re-distributed load across a number of providers and we believe we have identified the cause of the load and mitigated its impact.
We have restored intermittent service at this time but are keeping service Status at Offline at this time while we continue to monitor and make adjustments to fully restore service.
An incident report will be provided by 21:00 UTC Monday.
This update is related to Incident 12784
SSL Service is Offline
Beginning on 2010-06-05 at 17:51 UTC UTC the OpenSRS network experienced extremely high load that overloaded our network and one of our upstream providers. Since then we have re-distributed load across a number of providers and we believe we have identified the cause of the load and mitigated its impact.
We have restored intermittent service at this time but are keeping service Status at Offline at this time while we continue to monitor and make adjustments to fully restore service.
An incident report will be provided by 21:00 UTC Monday.
This update is related to Incident 12784