Cluster A
Incident Report for OpenSRS
Resolved
This incident has been resolved.
Posted Jul 29, 2020 - 13:15 UTC
Update
Engineering team has successfully migrated all the (beta testing) mailboxes to the legacy infrastructure and users are able to access all their emails. Users might notice incorrect dates and timestamps on the emails. Engineering team is currently working on resolving issues. Further updates will be provided once the issue is corrected.

Cluster A remains operational for all production services.

Next update: Based on progress.
Posted Jul 27, 2020 - 14:30 UTC
Update
Engineering team has successfully brought all services online and started prioritizing migrations for customers' (beta testing) mailboxes. Further updates will be provided once all the mailboxes are merged and users are able to view all their emails.

Cluster A remains operational for all production services.

Next update: Based on progress.
Posted Jul 24, 2020 - 19:05 UTC
Update
Our engineering team is performing final cleanup and data integrity checks before re-enabling the service for the affected (beta testing) customers. Cluster A remains operational for all production services.

Next update: Based on progress.
Posted Jul 22, 2020 - 20:47 UTC
Update
Our engineering team continues to work with our solution provider towards restoring the service for those affected (beta testing) customers. The recovery effort is in the final phase and we should have more information to report tomorrow. Cluster A remains operational for all production services.

Next update: Wednesday
Posted Jul 21, 2020 - 21:04 UTC
Update
Our engineering team continues to work with our solution provider towards restoring the service for those affected (beta testing) customers. Cluster A remains operational for all production services.

Next update: Tuesday
Posted Jul 20, 2020 - 20:18 UTC
Update
Our engineering team continues to work towards restoring the service for those affected (beta testing) customers. Cluster A remains operational for all production services.

Next update: Monday
Posted Jul 19, 2020 - 18:18 UTC
Update
We are continuing to work on a fix for this issue.

Next Update: Sunday
Posted Jul 18, 2020 - 21:04 UTC
Update
Our team continues to work towards restoring the service to a familiar state. Otherwise, the users participating in the beta can still use the service with the following in mind:

- Should you be presented an error on webmail login then log out and try logging in again
- Webmail timestamps will default to Pacific Time, you can adjust this in your webmail settings

Next Update: Saturday
Posted Jul 17, 2020 - 19:24 UTC
Update
We have implemented a workaround for those affected customers beta testing new features and their email services have been restored. We will continue to work for a permanent fix.
Posted Jul 17, 2020 - 15:42 UTC
Update
We are continuing to work on a fix for this issue. This issue only affected a few identified customers.

Next update: 1 hour
Posted Jul 17, 2020 - 14:00 UTC
Identified
We are currently experiencing partial outage on Cluster A which affects specific customers beta testing new features. We are working directly with the small number of customers who are affected and this does not affect the operation of our production services. Resellers and email users who are not explicitly beta testing these new features are unaffected by this issue.

Next update: 1 hour
Posted Jul 17, 2020 - 12:57 UTC
This incident affected: Hosted Email (Cluster A, Webmail).