Underfloor vs in-row cooling in data centres
Stephen McCarron, the Managing Director of Hosting365 (Ireland’s largest hosting compny), asked a very pertinent question of us yesterday. His question was in relation to my post on the installation of our raised floor. Stephen asked:
Would in-row cooling not be more ‘modern’ an approach – underfloor will limit you to not much more than 5KW per cab?
Stephen, in phase 1 of our data centre build we are installing 120 usable cabs with 4 x 120kW computer room air handling systems (CRAHs). Allowing for N+1 redundancy we therefore have 360kW of cooling in the room. So our design limit for the room cooling is 3kW average per rack.
Trying to achieve a heat density anything higher than this is difficult with a raised floor, we agree. However the benefits of full room cooling are that it is extremely cost effective, incredibly energy efficient, and it offers very good redundancy. The four CRAHs maintain a constant positive pressure of cool air under the floor. If one fails, or needs to be taken out of service for maintenance, there is no localised heating. The huge open expanse under the floor allows the operating CRAHs to compensate.
As you point out, an alternative approach to room cooling is point cooling. There are three common point cooling techiques, in-row (promoted by APC), overhead (promoted by Liebert), and in-rack promoted by the rack manufacturers such as Knurr and Schroff.
APC promote in-row cooling as part of their InfrastruXure offering. This utilises a relatively new concept called hot aisle containment. This is ideal for smaller data rooms especially where there are height restrictions preventing underfloor air flow. The real efficiency here though comes from the hot aisle containment not from putting the CRAH in-row. We have adopted a process called cold aisle containment which is the inverse of APC’s process and is even more efficient. CIX will be the first data centre in Ireland using cold aisle containment.
We will have many customers that will use less than 3kW per rack. That’s about 15 x Dell 2850 servers. For them our cooling solution is perfect. There is no reason why we can’t add point cooling at any time in the future to meet specific customer demands.
However our preferred solution will be overhead rather than in-row because we don’t lose a rack space every time we install an in-row CRAH. If a point solution fails then the room cooling is still available and may give you time to correct the situation before being forced to shut down equipment. We are fortunate that we have a building with over four metres of clearance from floor to roof. Many companies build data centres in office blocks and therefore have to make compromises regarding air handling. APCs InfrastruXure is ideal for data centres that are constrained in this manner.
In summary room cooling with cold aisle containment is the most cost effective, efficient, and resilient method of aircon for a data centre. It does have a defined capacity limit. We will add point cooling solutions (probably overhead but possibly in-row) as required later for specific customer loads without losing our cold aisle containment efficiencies and still maintaining most of our resilience.
I hope this answers your question Stephen. Thanks for bringing it up.
May 11th, 2007 at 10:55 am
Hi Tom,
I do wonder who has told you about the efficiencies of air cooling? Obviously you guys haven’t run into this with your heads down. You’ve taken time to plan it out and investigate every possibility. Hats off to you, it looks like its going very well. Good luck with it.
I’d have to question your comment on it being the most cost effective and efficient design for a data centre. Air is not known to be the best method of cooling in any design. Water however is far more efficient at dispersing heat than air. Also won’t it cost a lot in electrical power to keep the air pressure constant for the entire data floor? I’ve looked at in cab heat exchangers which employ water cooling in the back door that is controled by a central cooling system that monitors the temperature at various points in the cabinet. This would be a cold isle, cold isle solution.
That said Tom, YMMV as they say. There would be a certain level of environment cooling involved in the above, but it’d certainly be much less significant than trying to keep the air pressure under the raised floor that would be required to cool the cabinets above.
Also having used several data centres in Dublin, most do the same thing. They have 800mm or 900mm raised floors and pump tonnes of cool air and keep constant air pressure there which forces cold air up into the rows. Or maybe I missed something?
– just wondering is all. Also a point to note, from experience this is, that rows of cabs cooled with your design get warmer as you move toward the middle of the row. It’s more difficult to cool because of the heat be generated as you move in.
Paul
May 11th, 2007 at 4:52 pm
Hey Paul,
Thanks for the interest and the comment.
Paul, I’m going to post a separate blog entry over the next few days specifically on aircon efficiency as it is a huge topic in its own right.
Regarding cost effectiveness, we were quoted €350 per kilowatt for APC in-row CRAHs versus €140 per kilowatt for standard EDPAC CRAHs.
The resilience comes from the fact that the CRAHs are loosely coupled to the load and if one fails the others take over.
Most data centres do use underfloor distribution of cold air. What is different about our layout is the fact that we will be using cold aisle containment. More details of why this is so good will be covered in the next blog post we are planning around aircon efficiency.
It is a mathematical fact that the energy required to pump air is proportional to the cube of the velocity of the air. The heat transfer capacity is proportional to the volume of air transferred. Therefore the design objective is to maximise volume while reducing speed.
This is done by having large air paths. Hence the large underfloor cavity and the high ceiling above the cabinets. During our research we found out that Intel built an experimental data centre in Arizona that has an underfloor cavity of twelve feet (3.6m for us Europeans). Service engineers could walk freely underneath to lay cable. Because of the height of the air velocity was very low. The normal convection currents of the circulating air was nearly enough to keep the airflow going without any assistance.
Point solution CRAHs such as in-row, or rear door mounted, or overhead use more electricity for air circulation fans per kilowatt of cooling than do large CRAHs because they tend to create fast air flows.
And, remember that this electricity is burned in the data centre and therefore has to be taken out by the aircon system capacity.
I hope I have clarified things for you
May 12th, 2007 at 9:19 am
Hi Tom,
Wow! Wasn’t expecting a whole post in response to my comment !
You clearly seem to have done your homework, and the 4m floor to ceiling in your room will certainly help, but, to give you an example, we’re currently fitting out a new suite in our data centre – we’re installing 10x 40KW CCUs in-row on the floor for 52 usable cabinets of space. This was the only design we could use to achieve over 6kw per cabinet usable cooling with redundancy . We’ve decided not to use the APC solution in this suite, but will be using it in our high-density suite for blade/SAN type infrastructure. All the units will be chilled water cooled also. We’re designating this suite our ‘colocation suite’ as it is designed to a lower power density to our new ‘managed suite’.
More and more we are seeing customers installing dual and quad core servers, heavily loaded with ram and drives, Storage arrays, attached disk and – these days – more and more blade servers. A single 42U rack can now physically accommodate 4 blade enclosures, pulling over 20KW (if allowed!)
Our current suite uses 7x 35KW units with just over 80 cabinets (approx 3KW per rack) and works well, even though our floor void is relatively small ( downflow units with a 15-20cm void) – that said we run only lv power underfloor with all networking, etc, in overhead tray runs.
As for unit pricing, APC will of course argue that while the units cost more at install (like their UPS’s for example) you gain that back in their efficiency.
May 12th, 2007 at 9:22 am
Just to add – I’m very intrigued with the idea of ‘cold aisle containment’ – if you contain the cold aisle, does that mean the ambient in the data centre will be extremely warm? (ie: all the hot air from the racks vents into the room?) Probably very good for the machines, but perception counts for a lot in colocation – and no one likes a datacentre that ‘feels’ very warm.
May 13th, 2007 at 6:39 pm
Hey Stephen,
Your idea of moving the high power density equipment to an area specifically designed to deal with those type of loads is excellent.
We are starting with a 3,000 sq ft data room with 120 cab spaces. Depending on the speed of occupancy of this phase we will begin kit out of a second 3,000 sq ft room. We have the empty room ready to roll. We won’t even need to apply for planning permission and the services area we are building right now is designed to handle 1MW of redundant power and cooling with the addition of another transformer and chiller and diesel generator for which the cabling and piping will already be laid.
This will probably occur in the late 2008 time frame. We will build higher power density technology into part or all of that room in a similar way to how you are now building out. Your reply has been very good at stimulating our thought process on how we might tackle this Phase 2.
Cold aisle containment is, as you put it, intriguing. The CIX data centre is being designed from the outset as the most energy efficient data centre possible.
Our biggest expense over its lifetime will be our electricity bills. And energy is becoming a very scarce commodity. We thought long and hard about the perception issue of a walk in ambient temperature for a data centre of about 30°C and we acknowledge that this will come as a surprise (and maybe a shock) to visitors to the DC.
We have decided to try to deal with this perception issue from an educational standpoint rather than to shy away from it. There are big benefits. We need to convince our customers of this. We believe that in a few years time cold aisle containment will become normal in data centres.
May 13th, 2007 at 11:24 pm
[...] http://www.cix.ie/underfloor-vs-in-row-cooling-in-data-centres/ Stephen, in phase 1 of our data centre build we are installing 120 usable cabs with 4 x 120kW computer room air handling systems (CRAHs). Allowing for N+1 redundancy we therefore have 360kW of cooling in the room. So our design limit for the room cooling is 3kW average per rack. [...]
May 19th, 2007 at 6:10 pm
Cold aisle containment is definitely a very efficient system and its great to see a Data Centre using it in Ireland.
Other things to factor in would be to to reduce by-pass air loss through cable openings by using Koldloks http://www.datacleaneurope.co.uk/blog/?cat=24
We are also seeing a lot of companies starting to use the pleanform under floor air barrier system to direct the cold air to exactly where it is needed. http://www.Plenaform.com
May 20th, 2007 at 1:51 pm
We’re already using something similar to those grommets Fintan, albeit round rather than square. We use those for data, and another type of sealable grommet for power, 2x.
We’ve discussed using barriers to direct airflow under the floor, but I think it’s something we’ve decided to come back to. There will of course be an element of fine-tuning.
Thanks for your comments.
adam
May 31st, 2007 at 7:46 pm
InRowTM Cooling Architecture is not being recognized for its efficiency gains over raised floor perimeter cooling. At the end of the day the connected fan power for InRow cooling is substantially less than conventional raise floor computer room air-handlers. Please consider that InRow cooling may only require 0.34 watts (or less) per CFM versus 0.6 watts per CFM for typically perimeter air-conditioning. The major drivers for the additional fan power are: system effect on fan discharging into floor plenum, static losses of fan, and significantly higher internal cabinet losses for perimeter air-conditioning. Beyond this perimeter air-conditioning will circulate more CFM per kW of IT load than InRow solutions, assuming containment is not used.
Regarding the containment of the cold aisle, ask yourself would you rather contain the cold aisle with the room flooded with hot air, or contain the hot aisle with the room flooded with cool air. If you consider the volume of air at the desired temperature it becomes obvious that containment of the hot aisle is more favorable.
Furthermore InRow architectures are not susceptible to all of those under floor obstruction and problems associated with achieving desired airflow balance of perforated tiles.
June 25th, 2007 at 10:33 am
John, thanks for your comment. I have to take issue, though with what you are saying above. We have a large room with a high ceiling. The mean ambient temperature in Cork is 10C – the temperature of the air exiting the back of the servers is 30C-40C.
Dumping the hot air to the room allows for cooling by the ambient air free. This is extremely energy efficient.
The only negative to this approach is that people entering the data room will be surprised the first time they enter to find it is warm. This is an issue of perception which we will address with all customers and potential customers before allowing them onto the floor.
July 4th, 2007 at 3:15 pm
We’re in the cold winter and quite few hot days in West Galille, Israel. Brandstetter Systems Ltd installed in 1994 saves us approx a third of consumption bill. The small company is the brainchild of Dr A.Brandstetter who worked in Australia many years + his two sons.
May 23rd, 2008 at 1:25 pm
[...] we’re talking about traveling less, heating buildings using excess energy from data centers, only buying power when its cheap because the local wind farms are working efficiently, dealing [...]
July 30th, 2008 at 5:04 pm
We are in the starting phases of a 400-600kW datacenter in Madrid where energy efficiency is our main concern. We have severe space limitations in the clean room (3,5m/10,5ft high, 200m2/1800ft2), but not in the floor below. That means HUGE mean power density (8-10kW/rack), therefore APC InRow is right now our winning horse.
However specialized datacenter design firms consulted warn against this technology, and previous installations in Spain are scarse.
Are really risks so high? Should we go for a traditional solution and reduce power density? What is your opinion?
TIA
July 15th, 2009 at 9:38 pm
Hi Jose,
I’m interested why datacenter design consultants have warned against APC’s InRow/hot aisle containment noting high risks -
There are many options of course and with your particular set of circumstances noted I personally see the APC solution a viable option.
Data Centre designs need to consider high density for the mid-longer term future unless is it built for a short term period.
You can attempt to reduce power densities with even throttling back using more energy efficient ICT hardware technologies.
However vendors are still going moving towards squeezing more into a rack (general blade technology- i.e. HP adaptive infrastructure, CISCO unified computing) and 10 – 20kW will start becoming standard in 5+ years time -
Data Centre design should be factored for longevity and if you are hitting 8kW/rack now then opt for a scalable high density solution now (underfloor is definitely not an option).
Just my humble option – keen to hear from others (btw great blog Tom and excellent comments from all…) Cheers!
Vince
August 30th, 2009 at 9:29 pm
Hi,
Ever considerated free outside air cooling ? that is a really efficiënt way of cooling. Trends are going in this direction, and server manufactorers are currently redesigning to meet the required ETSI qualifications. Telco operators have been doing this for years…. and yes 8kw a rack is more than low density, but certainly not a constraint to kick out underfloor cooling by fact. we’ve designed many datacenters with underfloor cooling going to 10-12kw a rack, in a very efficiënt model (both investment as the energy bill). grtz. Erwin Born.