Updates related to Incident 13107

SSL Service is Online

Updated Monday, June 21st, 2010 at 8:21 PM ET
2010-06-22 at 0:21 UTC - Other time zones

All OpenSRS SSL services are fully online.

Update: June 22, 2010: 16:16ET/20:13 UTC

Incident Summary:

Summary: A Database unit suffered a memory corruption resulting in a kernel panic. This caused OpenSRS SSL Services to be offline. We addressed the issue, restarted services, and tested. All services were fully available within the hour. We moved the database to a stand-by platform while the hardware problems on the original system are resolved.

Monday, June 21, 2010 19:20 ET/23:20 UTC
Our Network Monitoring system alerted of an issue with a core database unit. OpenSRS Domains, SSL and Storefront services were unavailable.

19:25 ET/23:25 UTC
Technical teams were engaged and identified the issue. A database unit suffered a hardware failure. OpenSRS services were restarted and teams began testing.

19:51 ET/23:51 UTC
Services were available. We changed the status to degraded while we continued to closely monitor availability and continue our testing regime.

20:02 ET/Tuesday, June 22, 2010 00:02 UTC
Services were working well.

20:20 ET/Tuesday, June 22, 2010 00:20 UTC
All OpenSRS services were online.

This update is related to

Storefront is Online

Updated Monday, June 21st, 2010 at 8:21 PM ET
2010-06-22 at 0:21 UTC - Other time zones

All OpenSRS Storefront services are fully online.

Update: June 22, 2010: 16:16ET/20:13 UTC

Incident Summary:

Summary: A Database unit suffered a memory corruption resulting in a kernel panic. This caused OpenSRS Storefront Services to be offline. We addressed the issue, restarted services, and tested. All services were fully available within the hour. We moved the database to a stand-by platform while the hardware problems on the original system are resolved.

Monday, June 21, 2010 19:20 ET/23:20 UTC
Our Network Monitoring system alerted of an issue with a core database unit. OpenSRS Domains, SSL and Storefront services were unavailable.

19:25 ET/23:25 UTC
Technical teams were engaged and identified the issue. A database unit suffered a hardware failure. OpenSRS services were restarted and teams began testing.

19:51 ET/23:51 UTC
Services were available. We changed the status to degraded while we continued to closely monitor availability and continue our testing regime.

20:02 ET/Tuesday, June 22, 2010 00:02 UTC
Services were working well.

20:20 ET/Tuesday, June 22, 2010 00:20 UTC
All OpenSRS services were online.

This update is related to

Domain Service is Online

Updated Monday, June 21st, 2010 at 8:20 PM ET
2010-06-22 at 0:20 UTC - Other time zones

All OpenSRS Domains services are fully online.

Update: June 22, 2010: 16:16ET/20:13 UTC

Incident Summary:

Summary: A Database unit suffered a memory corruption resulting in a kernel panic. This caused OpenSRS Services to be offline (Domains, SSL and Storefront). We addressed the issue, restarted services, and tested. All services were fully available within the hour. We moved the database to a stand-by platform while the hardware problems on the original system are resolved.

Monday, June 21, 2010 19:20 ET/23:20 UTC
Our Network Monitoring system alerted of an issue with a core database unit. OpenSRS Domains, SSL and Storefront services were unavailable.

19:25 ET/23:25 UTC
Technical teams were engaged and identified the issue. A database unit suffered a hardware failure. OpenSRS services were restarted and teams began testing.

19:51 ET/23:51 UTC
Services were available. We changed the status to degraded while we continued to closely monitor availability and continue our testing regime.

20:02 ET/Tuesday, June 22, 2010 00:02 UTC
Services were working well.

20:20 ET/Tuesday, June 22, 2010 00:20 UTC
All OpenSRS services were online.

This update is related to

Storefront is Degraded

Updated Monday, June 21st, 2010 at 7:53 PM ET
2010-06-21 at 23:53 UTC - Other time zones

OpenSRS Storefront services are online. We continue to closely monitor and test services while we investigate the cause.

This update is related to

SSL Service is Degraded

Updated Monday, June 21st, 2010 at 7:52 PM ET
2010-06-21 at 23:52 UTC - Other time zones

OpenSRS SSL services are online. We continue to closely monitor and test services while we investigate the cause.

This update is related to

Domain Service is Degraded

Updated Monday, June 21st, 2010 at 7:51 PM ET
2010-06-21 at 23:51 UTC - Other time zones

OpenSRS Domains services are online. We continue to closely monitor and test services while we investigate the cause.

Update Tuesday 00:02 UTC
All services are working well. Our technical teams are testing functionality. We will continue to keep services in degraded status as a precaution.

This update is related to

SSL Service is Offline

Updated Monday, June 21st, 2010 at 7:24 PM ET
2010-06-21 at 23:24 UTC - Other time zones

We are investigating a database issue. At this time, OpenSRS SSL services are unavailable. Our Technical teams are engaged and investigating. We will have more details shortly.


Update 23:35 UTC:
We restarted services and are testing.

This update is related to

Storefront is Offline

Updated Monday, June 21st, 2010 at 7:23 PM ET
2010-06-21 at 23:23 UTC - Other time zones

We are investigating a database issue. At this time, all OpenSRS Storefront services are unavailable. Our Technical teams are engaged and investigating. We will have more details shortly.


Update 23:35 UTC:
We restarted services and are testing.

This update is related to

Domain Service is Offline

Updated Monday, June 21st, 2010 at 7:23 PM ET
2010-06-21 at 23:23 UTC - Other time zones

We are investigating a database issue. At this time, all OpenSRS Domains services are unavailable. Our Technical teams are engaged and investigating. We will have more details shortly.


Update 23:35 UTC:
We restarted services and are testing.

This update is related to