Cluster A - inbound email delays

Incident Report for OpenSRS

Resolved

All-clear and sorry about this! Please do let us know if you notice anything else unusual.

Incident Start Time: 03-13-2021 02:35:00 UTC
Incident End Time: 03-13-2021 14:57:00 UTC
Total Duration: 12 hours 22 mins
Posted Mar 13, 2021 - 18:17 UTC

Monitoring

A fix has been implemented for this issue. We're monitoring the results now.
Posted Mar 13, 2021 - 16:28 UTC

Identified

We found the problem and are working on a fix. It should be fixed shortly.
Posted Mar 13, 2021 - 15:28 UTC

Update

Our teams are working to diagnose the problem. We want to get the system back up as soon as possible. Please stay tuned.
Posted Mar 13, 2021 - 14:07 UTC

Update

Our technician has been engaged. We are still working on this and will update again when there is more progress.
Posted Mar 13, 2021 - 12:55 UTC

Update

We are continuing to investigate this issue. Currently our Network Engineering team is investigating packet loss between bra1 and bra2 clusters for PROD A. We are engaging Data Centre Technician to investigate the link between the clusters.
We will provide updates as they become available, and appreciate your patience.
Posted Mar 13, 2021 - 11:54 UTC

Update

Our engineering team is still investigating the root cause of inbound email delays on PROD A. We will post updates once we have them available.
Posted Mar 13, 2021 - 09:53 UTC

Investigating

We are currently experiencing an incident that is impacting Hostedemail, (Prod A, Inbound/Outbound Mail). Users on prod A will experience inbound email delays. The engineering team is currently investigating the issues.
Posted Mar 13, 2021 - 09:08 UTC
This incident affected: Hosted Email (Cluster A, Webmail).