Major Outage

[russellw]

Good morning

Apologies to all for the 15 hour outage from ~17:00 Sunday afternoon.

The hosting company we use (and have used since Day 1) had a major outage during a planned maintenance of their network infrastructure which started at midnight Sunday night their time which unfortunately was 17:00 here. Apparently after 5 hours they decided to abandon the change and start the remediation process.

I don't have the full details of the root cause but from what I gather every server connected to the particular switch stack lost connectivity that needed manual intervention to resolve.

I didn't discover the outage until this morning unfortunately and when I contacted them they were still working their way through the list of servers to 'fix' and were only escalating the priority when someone contacted them so I suspect the change took out a fair portion of the data centre.

That it coincided with a day when the bride and I decided to binge watch something and was thus away from the computer and no-one rang me to let me know is Murphy's Law at it's best because we could have probably been back up by midnight.