Hey All, Hope your enjoying this wonderful weekend. I just wanted to quickly discuss the main reason why the Ember Box has been having outages.
Outage 1) Fiber Amplifier went offline: The main fiber link between the datacenter and the telehouse went down for 4 hours after a Amplifier went offline. A part was quickly found and connectivity was restored as soon as possible. The part had to be sourced from outside of London so the additional hours where due to hauling of the item.
Outage 2) Power Outage: Normally when the local grid has a power outage the datacenter is able to go straight onto battery UPS (Un-delayed Power Supplies)/Diesel, Unfortunately the duel setup of two generators failed at the same time which caused a major power outage, Our host’s datacenter provider went offline for 6 hours, Finally after this outage all the servers came back online however this did have a knock on effect with Ember as we discovered later on in the week. (The NS2 server went down), We manually rebooted the machine this morning to fix this. The datacenter has sent off an RFO on the generators and engineers have fixed the problem.
Outage/At Risk 3) 09/03/2014 at 12pm there will be maintenance carried out on the fiber link to the telehouse, The backup circuit has been tested and hopefully only minor packet loss will be detected, The repairs are needed as the Fiber Link has been damaged by a vehicle. Traffic packet-loss was noticeable thought out the maintenance as the backup circuit couldn’t carry all the traffic at once. The traffic is now being carried over the mixture of the two fiber pools, by 11pm the traffic will be rerouted to the normal fiber line, you may notice some minor latency as this switch happens.
Phew, Hopefully things will get back to normal now.
Please note all other services including Teamspeak, Chivalry, flobname.net and this website where not affected by these outages.