Update - Our customer base is reporting that a majority of the imapacted users have now recovered.

We're closely monitoring the situation and continuing to work on the remaining users that cannot access their services at this time.
Apr 4, 21:24 UTC

Update - We are continuing with the migration, and users can still experience short periods where there is a loss of connection or the inability to log on. We'll keep you informed as we work to reconfigure the account's connections to our servers.
Apr 2, 08:55 UTC

Update - We have now unblocked all users on Cluster A to be able to access and use mail services.

There are still remaining accounts to be migrated to the new mailstore, and we will post updates on this incident to keep you informed as we work to complete the migration.

To be kept up to date, please subscribe either to the System Status page or this specific case.
Mar 27, 01:31 UTC

Update - The operations team has already enabled roughly 60% of users out of the subset of users (approx 2k) that was under maintainance and progressing towards enabling remaining users.
Mar 26, 23:18 UTC

Update - The affected mailstore is now enabled again. With the exception of the small subset of users (approx 2k) placed in maintenance for additional investigation. All other users on Cluster A should be accessible without issue
Mar 26, 20:13 UTC

Update - The previously disabled mailstores have been re-enabled again. Our Operations department has narrowed down the potential issue in the last affected mailstore to a subset of users. Currently, we are bringing the last affected mailstore online with those users disabled so we can monitor performance.

Updates will be provided when possible
Mar 26, 19:44 UTC

Update - While re-enabling the affected mailstore an issue regarding it was discovered. To mitigate that, our Operations department will be disabling some other mailstores. They will be brought back up one at a time to monitor performance. Approx 70k users will be impacted.

Updates will be provided as soon as possible
Mar 26, 19:01 UTC

Update - We have re-enabled all mailstores in our system except one since our last update. Some users on the affected mailstore will be offline until we completed the migration. We will provide an update as soon as the migration is done.
Mar 26, 18:33 UTC

Update - Most of the affected mailstores have been re-enabled. While the operations team continues to investigate another to identify and rectify the issue on it.

Updates will be provided as soon as possible
Mar 26, 12:05 UTC

Update - After re-enabling our mailstores to resolve this ongoing incident we have encountered a different issue and needed to bring the mailstores offline again in order to resolve this problem. We will likely be bringing them back up shortly once we have completed testing. These affected users will be unable to login to the Webmail/IMAP or POP until the mailstores have been re-established.
Mar 26, 11:10 UTC

Update - The mailstores have been enabled with 12521 users migrated. The CR and migration are stopped for today. At this time service is fully online and we will be monitoring for any impact come business hours EST which is currently expected to be better than previous days.
Mar 26, 10:00 UTC

Update - The emergency maintenance on Cluster A will be targeted for specific users. Those targeted users will be unable to login to their account via POP/IMAP/Webmail or an email client during the maintenance window. This would affect approx. 6.6% of users on Cluster A

Start time: Mar 25, 2020 at 10:00 PM UTC

End time: Mar 26, 2020, at 10:00 AM UTC
Mar 25, 20:58 UTC

Update - We will be performing an emergency maintenance of targeted users. Cluster A users will be unable to login to their account via POP/IMAP/Webmail or an email client during the maintenance window.

Start time: Mar 25, 2020 at 10:00 PM UTC

End time: Mar 26, 2020, at 10:00 AM UTC
Mar 25, 19:54 UTC

Update - The operations team will be rotating mail flow by enabling and disabling different smtpin nodes at a time to ensure mails are delivered to all users.

Updates will be provided as soon as possible
Mar 25, 17:25 UTC

Update - Our Operations team have enabled IMAP/POP/Webmail. Email deliveries will be resumed shortly.

Updates will be provided as soon as possible
Mar 25, 15:18 UTC

Update - We will again be stopping access to 70k users and start them again over the next hour.

Updates will be provided as soon as possible
Mar 25, 14:24 UTC

Update - We are seeing high load again. Users may notice slowness or intermittent issues to login.

Updates will be provided as soon as possible
Mar 25, 14:03 UTC

Update - Our operations team has ended the maintenance on Cluster A by enabling all of the mailstores.

A small number of accounts will still not have webmail or email client access as we prepare for another batch migration to be done during a lull period.
Mar 25, 07:35 UTC

Update - The maintenance is nearing completion. Currently 4 of the 5 mailstores have been enabled again and we are awaiting storage load to drop before the 5th mailstore is enabled.

At this time, only users on the still disabled mailstore are expected to be completely unable to utilize hosted email. Users from the other 4 mailstores which were enabled will still see degraded performance at this time.

We will continue to update, as updates become avaialble. We thank you for your patience.
Mar 25, 07:06 UTC

Update - Our operations team is working on an emergency migration process to help restore access.
A handful of users will be unable to login to their account via webmail or their email clients.
Mar 24, 23:56 UTC

Update - The operations team is still observing a high load on the NAS. They are monitoring it to ensure it stabilizes before they resume email deliveries.

Updates will be provided as soon as possible
Mar 24, 20:26 UTC

Update - Our operations team has turned on all 5 mailstores and is currently observing the performance. Once the NAS stabilizes, mail deliveries will be resumed one mailstore at a time.

Updates will be provided as soon as possible
Mar 24, 18:55 UTC

Update - Our operations team has turned on 4 mailstores out of 5 mailstores and is currently observing the performance of the NAS. We have around ~56k users online. Once the load on the NAS settles down, the remaining mailstore will be turned on as well.
Mar 24, 17:36 UTC

Update - The operations team has turned off all mailstores on the affected NAS and disabled mail services for ~70k users to stabilize the network storage device. The mailstore migration processes are still running. Once migration processes finish successfully, the Operations Team will be turning mailstores online one at a time and will observe the performance on the NAS

Updates will be provided as soon as possible
Mar 24, 15:36 UTC

Update - Update: The problem has not fully recovered after last night’s maintenance. The operations team is taking subset of users offline to stabilize the hardware. The plan is to take down around 15k users out of the 70k at a time until we notice improvement. Users on that mail store may experience problems accessing their mailbox including Tucows users. For now, email is still being delivered to users inboxes, even if they cannot login at this time.
Mar 24, 13:37 UTC

Update - Our OPS dept has stopped the migration for now. 6396 users have been migrated so far and OPS is planning for the next step for today. Webmail is degraded, as well as IMAP and POP
Mar 24, 12:54 UTC

Update - The migration appears to be progressing well. A further estimate on the completion time has not yet been made. We will update again once we have this information.
Mar 24, 09:20 UTC

Update - The migration continues. So far 5100 users have been migrated for the maintenance and the mailstores will be enabled again in approx. 1 hour. We will continue to provide updates as they become avaialble/
Mar 24, 06:01 UTC

Update - The migration appears to be progressing well. A further estimate on the completion time has not yet been made. We will update again once we have this information.
Mar 24, 03:26 UTC

Update - We are currently performing emergency maintenance on the cluster to expedite the migration of the affected users.
During the maintenance, users may be unable to access our email services. We will post the updates as we receive.
Mar 24, 01:37 UTC

Update - Our operation team is still working on the mailstore migration. We will provide updates as we receive them.
Mar 23, 23:23 UTC

Update - Our operation team is diligently working on migrating the affected users into newly created mailstores.
We appreciate your continued patience.
Mar 23, 21:32 UTC

Update - We are currently having emergency maintenance to bring service back online.
Mar 23, 20:46 UTC

Update - There is no change in the status of service impact, however, the failover is almost complete.

Some POP3 users may also experience some trouble connecting.

The next update will be within the next 60 minutes.
Mar 23, 19:40 UTC

Update - Our investigation continues into this issue.

Webmail is still down and imap still under heavy load currently.

Next Update: Within 30 minutes
Mar 23, 18:08 UTC

Update - Our Ops dept are still investigating the issue. Webmail is still down and imap still under heavy load currently.

Next update in 30 minutes.
Mar 23, 17:22 UTC

Update - We are continuing to investigate this issue. Webmail is still down completely and many users are still unable to login at this time. Currently, our Ops dept is taking down mailstores one at a time to try and mitigate the impact as a whole. At this time, 5 mailstores have been taken down and inbound smtp delivery for these users is being queued locally for later delivery.

Next update within 30 mins
Mar 23, 16:38 UTC

Update - Webmail is now down completely and many users are still unable to login at this time. We have upgraded this issue and our operations team is still working to mitigate this impact.

Updates will be provided as soon as possible
Mar 23, 15:53 UTC