Status archive for July, 2009
.com is Online
The .COM registry maintenance is complete.
.net is Online
The .NET registry maintenance is complete.
.net is In Maintenance
There is a 45-minute .NET registry maintenance starting now (01:00 UTC).
Service Impact:
Domain provisioning and Whois will be unavailable. Customers who have enabled provisioning queuing will have their orders queued until this complete. Domains will continue to resolve.
.com is In Maintenance
There is a 45-minute .COM registry maintenance starting now (01:00 UTC).
Service Impact:
Domain provisioning and Whois will be unavailable. Customers who have enabled provisioning queuing will have their orders queued until this complete. Domains will continue to resolve.
Email Cluster A is Online
OpenSRS Email Cluster A is fully online.
The provisioning queue for Cluster A has cleared. All new changes via the API (Application) or the Mail Administration Center (MAC) are now processing as per normal.
Inbound/Outbound mail flow and mailbox access were not affected by this.
Incident Summary: (added at 17:05ET)
At 07:58 ET we began noticing unusual behaviour with the provisioning system. There were corresponding alerts on our Network Operations Center (NOC) monitoring. Our Operations team began investigating to determine the scope and cause. Application hosts were restarted and this did not resolve the issue. We advised you of a provisioning delay at 08:54ET. All changes via the API were being queued for approximately 45 minutes. There were two types of errors noted: pending and failing. Both types of requests were being queued.
Our Development team identified an issue with the Oracle libraries that provide connectivity to the provisioning database. The back-end hosts were restarted and this resolved the issue. There were approximately 600 provisioning requests queued. We fixed the errored provisioning requests. The provisioning queue cleared. Network Operations Center (NOC) and Technical Support staff fully tested and confirmed that corresponding alerts had stopped on our monitoring system and that all new provisioning requests were successful. The service was fully restored by 09:45ET and we posted it online.
This update is related to Incident 7521
Email Cluster A is Degraded
Our Development and Operations team identified the provisioning problem to be related to behaviour of a particular provisioning device. They restarted it. Your provisioning changes via the API, which were previously queued, are being processed. We are closely monitoring all the provisioning systems. There is a queue of requests which need to be processed.
You can make provisioning changes via the Mail Administration Center (MAC) and API. Please note that there are still delays with changes via the API.
All other Email Cluster A services are working well.
This update is related to Incident 7521
Email Cluster A is Degraded
Our Development team is looking into the issue. Provisioning via the API was working after the maintenance window earlier this morning. Your provisioning requests sent in the last 30 minutes have been queued and are delayed.
Resellers can make provisioning changes such as password changes and creating new accounts via the Mail Administration Center (MAC).
This update is related to Incident 7521
Email Cluster A is Degraded
OpenSRS Cluster A provisioning changes via the APP have been delayed for the last 20 minutes. You can make changes via the Mail Adminstration Center (MAC) Our Operations team is investigating.
All other services for Email including mailflow (inbound and outbound) and mailbox access are working well.
This update is related to Incident 7521
.es is Online
Our Network Operations Center (NOC) monitoring alerts for connections to the .ES registry indicate that the issue has cleared. Whois lookups and your .ES requests (changes and new orders) can be processed now.
There has been an intermittent connection issue with the .ES registry for a few days. Each time we contact them during and after business hours to follow-up. We will do so again during their business hours today.
We do apologize for the intermittent .ES issues you have been encountering. We will keep this incident number open to track the responses and connection issues while we continue our investigation with the registry.
Incident Summary(posted 17:11ET)
The .ES registry advises that an application server crashed yesterday. The connection issues which we experienced may have been related. OpenSRS and the .ES registry are continuing investigations into the recurring intermittent issues.
This update is related to Incident 7483
Email Cluster A is Online
The OpenSRS Email Cluster A maintenance is complete and all services are
fully online.
Our Operations team made the hardware changes by 12:30 ET. The storage device was brought up with meticulous attention to detail. Each component was thoroughly tested. Once they confirmed that all was up, they requested our Network Operations Center (NOC) and Technical Support team test each part of your services.
All services are fully available:
*Webmail, IMAP, POP
*Inbound/Outbound mail
*Provisioning, LDAP
We originally scheduled 3 hours for this window, but we successfully completed our work early.