Status archive for September, 2009
.de is Online
The .DE maintenance is complete. (The registry finished their work ahead of schedule)
.de is In Maintenance
.DE has 4-hour registry maintenance starting now (12:00 UTC)
Service Impact:
Domain provisioning and Whois will be unavailable. Customers who have enabled provisioning queuing will have their orders queued until this complete. Domains will continue to resolve.
Email Cluster B is Online
OpenSRS Email Services - Cluster B is fully online.
All provisioning requests via the Mail Administration Center (MAC) and Application protocol (APP) are being accepted and processed in normal time. The previous queue of change requests have been cleared.
Customer mailbox access and inbound/outbound mailbox were fully available during this incident.
Incident Summary (Updated 16:47 ET/20:47 UTC)
At about 02:15 ET our network monitoring system alerted us to issues with mailbox provisioning on Cluster B of our Email Service.
Analysis showed that provisioning services via the Mail Administration Center (MAC) and Application Protocol (APP) were degraded due to a high load on the provisioning system. Customer mailboxes remained accessible via IMAP, POP, and Webmail. Inbound and outbound mail were fully available.
By 04:07 ET we isolated the cause of the issue as a misconfigured provisioning script. The script put an unusually high load on the provisioning servers that caused normal provisioning requests to be queued.
At 04:24 ET we stopped all provisioning requests to investigate further. We cleared a invalid provisioning requests and restarted the provisioning system at 05:02 ET. All queued and new provisioning requests were then processed normally.
The provisioning queue cleared by 05:19 ET.
Our investigation shows that an incorrectly configured script caused this abnormally high load on the provisioning system. While we normally run such scripts during off-peak hours we are now refining our internal processes to ensure that these types of scripts receive greater scrutiny before being used.
We apologize for any inconvenience to you and your customers.
This update is related to Incident 8469
Email Cluster B is Degraded
We restarted the Provisioning system. All previously issued new and change requests are being processed. There is a backlog of provisioning requests. As it is not prime time business hours, these are processing fast. New change provisioning requests via the Mail Administration Center (MAC) and APP are being accepted for processing.
We identified the cause of the provisioning issue and addressed it. We will continue our investigation.
Customer mailboxes are fully available. Mailbox access was not affected by this issue. Inbound and outbound mail are also working well.
This update is related to Incident 8469
Email Cluster B is Degraded
OpenSRS - Email Services Cluster B provisioning services are offline. All other Email Services including mailbox access and mailflow are fully available and online. Customers can access their mailboxes via IMAP, POP and Webmail. Inbound and outbound mail flow are unaffected.
Provisioning System Unavailable (Offline)
Our technical teams continue to investigate a high load on the provisioning system. All provisioning changes will be queued. We have temporarily stopped the provisioning system. Changes via the Mail Administration Center (MAC) and Application protocol (APP) are unavailable at this time.
This update is related to Incident 8469
Email Cluster B is Degraded
OpenSRS Email Services - Cluster B provisioning services are degraded. Our operations team is investigating a high load of provisioning requests which is causing all new and change requests to be delayed. Provisioning changes will be queued for processing and be completed once this backlog of requests is flushed.
All mailboxes are available via IMAP, POP and Webmail. Inbound and Outbound mail are also unaffected.
Update 07:07 UTC:
We have identified the cause of the high load on the provisioning system.
Update 07:19 UTC:
The provisioning change request queue is slowly processing. There continues to be a backlog of change requests queued. Change requests submitted will be processed with some delay. We continue to work on addressing the cause of the high load on the provisioning system.
Update 07:56 UTC:
Our Technical teams are investigating the best solution to address the high load on the provisioning systems.
This update is related to Incident 8469
Tucows Website Builder is Online
The Tucows Website Builder maintenance is complete. Our technical teams finished their work ahead of schedule and confirm all services are online.
Tucows Blogware is Online
The Tucows Blogware maintenance is complete. Our technical teams finished their work ahead of schedule and confirm all services are online.
Personal Names Service is Online
The OpenSRS Personal Names Service maintenance is complete. Our technical teams finished their work ahead of schedule and confirm all services are online.
Storefront is Online
The OpenSRS Storefront maintenance is complete. Our technical teams finished their work ahead of schedule and confirm all services are online.