Cluster B - Inbound Delays
Incident Report for OpenSRS
Resolved
This incident has been resolved.
Posted Aug 30, 2019 - 00:04 UTC
Monitoring
We are pleased to report all backlog mails have been cleared and all services are now back to operational for Cluster B.

Thank you for your patience as we worked to resolve this.
Posted Aug 29, 2019 - 23:22 UTC
Update
Operations team has made significant progress on the backlog and all new mail is now being delivered immediately. We are actively working on the remaining backlog and provide more update when it becomes available.

Next update: Based on the progress
Posted Aug 29, 2019 - 20:09 UTC
Update
We are making progress with the mail delivery backlog and the queue is reducing. Operations team is adding more service nodes to reduce the remaining backlog.

We will provide more update when it becomes available.

Next update: Based on the progress
Posted Aug 29, 2019 - 17:57 UTC
Update
We have successfully deployed new service nodes to the pool to help reducing the inbound emails backlog. We will continue to monitor the progress and will provide update accordingly.

Next update: 17:45 UTC
Posted Aug 29, 2019 - 16:45 UTC
Update
Operations team, with the help of the Security team, aggressively looking for more ways to filter out the intrusive spam emails. While working with the spam vendor on the side, our Operations team is going forward with increasing the capacity of the system to keep up with the delivery time of the incoming emails. We will provide more update when it becomes available.

Next update: 17:15 UTC
Posted Aug 29, 2019 - 16:18 UTC
Identified
We have escalated the issue to our spam vendor for a spam detection issue. The additional spam load is delaying the delivery of legit emails. We will provide more update when it becomes available.

Next update: 16:00 UTC
Posted Aug 29, 2019 - 15:04 UTC
Update
We are continuing to investigate the issue. The inbound mail delay can be up to an hour. We will provide an update once we have additional information.

Next update: 15:20 UTC
Posted Aug 29, 2019 - 14:18 UTC
Investigating
We are currently experiencing inbound mail delays on Cluster B. Our operations team is investigating the issue.

We will provide an update in 30 minutes.
Posted Aug 29, 2019 - 13:34 UTC
This incident affected: Hosted Email (Cluster B).