Successful unscheduled UPS and diesel generator test!
There was a major power cut on the northside of Cork city yesterday. According to ESB Networks the problem occurred when a digger cut through a cable at 12:06 yesterday afternoon. 2,500 homes in the area were affected and power was restored at 13:02.
CIX was completely unaffected by the power cut because as soon as the power dropped, CIX’s UPS’s and batteries automatically took over running the facility. Within 15 seconds the diesel generator had come on line and commenced powering the data centre.
We run scheduled tests on the diesel generator every weekend where we simulate outages and log the results but there is nothing like an actual unplanned outage to really test your system.
April 16th, 2008 at 11:46 pm
What would have been your run time if the generator failed to kick in?
April 17th, 2008 at 11:18 am
A very good question Alan, thanks for asking.
The batteries are rated to run the facility for 20 minutes when the facility is full.
At the moment because we are just starting out and many customers haven’t moved their equipment in yet, the batteries could comfortably power the facility for a number of days, I suspect!
April 24th, 2008 at 12:30 pm
Hi Alan, Currently we’re at 10% of our UPS load capacity so we have 3 hours and 20 minutes of battery run time. Tom’s point is very valid: every week this reduces at load is added.
Just before we get to 20 minutes run time is when we need to order more UPSs. Interestingly this will occur when the UPSs will only be at just under 50% capacity because above that point we would loose our 2N failover capability.
I am currently investigating best practice configurations for sites with clustered UPSs. I am surprised by how complex the issue is. There is an interesting article in the Jan/Feb edition of Zerodowntime entitled ‘Advantages of MFR (multiple feed redundancy) for datacenters. This article discusses a configuration I had not considered before.
April 25th, 2008 at 10:58 am
Hi Guys – good to see things work as intended. Just a question on the weekly tests – do you do a full simulated outage and run everything of the generator or is it an off-load generator test.
I assume you keep sufficient oil in the tanks to run for a few days.
May 5th, 2008 at 6:10 pm
Hi Michael,
We start the diesel generator once per week. This is an offload test. The prime purpose of this test is to make sure the diesel generator actually starts. There are two reasons why this is necessary.
1) The diesel generator is connected to the mains supply to keep the engine block warm(ish) and the battery fully charged. We are testing that these mains supported features are working.
2) Unlike petrol engines that use evaporated fuel, diesel engines use liquid fuel that is ‘injected’ into the cylinders. If an airlock comes in the fuel pipework the engine will never start no matter how long you turn the engine. We start the engine every week to make sure it is not airlocked. and to expel any small air bubbles that may have formed.
Our also do quarterly full load tests. We have a mains failure simulation circuit in the low voltage (LV) switchgear to enable us to do this. Because of what happened we now have proof that a real mains failure is handled in the same way as a simulated failure.
We don’t need to do a full failover test now until the end of July. It will be done at a weekend. If any customer wants to attend they are more than welcome.
Incidentally, from November to February inclusive at times from 5:00pm to 7:00pm it is more expensive for us to purchase electricity than to generate it. So we are planning to go off-grid for 10 hours each week for those four months.
May 5th, 2008 at 11:35 pm
Also woundering what would happen if you had a failure during the weekly generator test. I know this is a very rare occurance but when we are talking about %99.999+.
May 6th, 2008 at 3:21 pm
Hi Alan,
If the mains failed during the weekly test then the changeover to generator power would be faster than usual because the twelve second start time for the diesel generator would not be required. There is no loss of redundancy during this test.
May 30th, 2008 at 9:10 pm
You should hire the guy with the digger and make him part of your test crew. Nothing like the real thing, as you say.
October 22nd, 2008 at 12:38 pm
Can you recommend a vendor who would implement such a solution? We are looking at exactly this kind of electricity failover in our company (we already have UPS).