3

I'm investigating different ways to safely and efficiently cool down one of our servers that uses two GPU cards for intense renderings that takes hours to complete.

The overall system wattage is about 1600 Watts per server and it generates a decent amount of heat.

  • Considering that my goal is to keep a server cool, do it safely, and efficiently, how should I use mineral oil to accomplish this?

  • What fire precautions should I take?

  • Is there a better (more efficient, safer) technique that I could use instead of using a submerged server? (example being water cooling system with mineral oil instead of H2O)

makerofthings7
  • 8,911
  • 34
  • 121
  • 197
  • I don't think immersion is a viable or feasible solution for a single server. You can't just go get a container, fill it with mineral oil and dunk your server in it - http://www.grcooling.com/product/faqs – joeqwerty Apr 05 '13 at 17:56
  • @joeqwerty Why not? (Ghetto factor aside) – Ryan Ries Apr 05 '13 at 17:59
  • just looking at those pictures kinda wierds me out... – tony roth Apr 05 '13 at 18:15
  • What is the overall problem you're trying to solve here: Is it "conventional cooling methods won't work because *x*" or is it "I heard that cooling stuff with mineral oil is kewl, how do I do that?". The reason I ask is because I'd suggest the most efficient method of cooling *a* server is to do so conventionally - advanced cooling methods only make sense once you have enough servers to cool to make the investment in a proper environment worthwhile. – Rob Moir Apr 05 '13 at 18:24
  • @RobM I'm looking to achieve a balance of energy efficiency, cooling for hot servers in a branch office. I dislike the mineral oil technique because it voids a warranty, but maybe it's a good substitute for water in a water cooled device. My primary concern is fire safety. – makerofthings7 Apr 05 '13 at 18:30
  • 3
    If you're just looking to play with mineral oil cooling, go for it, it could be fun. Make sure you have a way to circulate the oil past the hottest components. But if you're looking for a way to keep your production server running reliably, air cooling is the way to go. PenguinComputing sells a quad GPU 2U compute server (and an 8 GPU 4U server) that uses air cooling, so air cooling is not impossible for such a server. – Johnny Apr 05 '13 at 18:43
  • 1
    "but maybe it's a good substitute for water in a water cooled device." - Or in a branch office, maybe air cooled is a better choice again. Good rule of thumb for any coding/sysadmin project: If your design bypasses conventional in favour of interesting and *its not absolutely neccessary* then you're doing it wrong. And air-cooled will work fine here, so the more exotic solutions are not neccessary. – Rob Moir Apr 05 '13 at 19:14

1 Answers1

4

Water cooling kits for GPUs are very common in the enthusiast PC market. That would be much easier and safer than dunking your server in a giant vat full of oil.

If you did submerge your server in oil, you'd want to remove all the moving parts (ie, the fans,) from everything. And you'd have to use external storage such as a SAN. Mechanical hard drives have breather holes that can't be covered up. (And I doubt their seals are tested under submersed conditions either.) You'd want to remove any chassis cover and any airflow covers (like the kind GPUs typically have) so that the oil can actually reach the components and circulate around them.

If your giant vat of oil caught fire? Uhm... evacuate the building because that mofo is burning down. I mean honestly, you'd probably need to displace all the air in the room with halon to put that out in a timely manner. At least... keep a fire extinguisher handy. But mineral oil has such a high smoke point, it's very unlikely that it'll catch fire. They use it in 10KVA distribution transformers and those don't catch fire too often.

Edit to answer your comment: I would just use distilled water. There are "Water Wetter" type additives, but they are designed for car radiators and much different temperatures and their use in this type of application would not likely show any measurable results. You might consider an additive that will prevent algae growth, but don't use any chemical that might compromise the integrity of the seals or hoses. Using oil in a system designed to carry water would not work. Not only would it not work but it doesn't even make sense, even if you got a really strong pump that could pump the oil quickly. Water is desired because it can absorb the heat quickly as it whisks over the components, then it is taken to a radiator where it is cooled back down. Oil does not absorb heat as quickly, but it can absorb much more heat overall before bad things happen.

Ryan Ries
  • 55,481
  • 10
  • 142
  • 199
  • Can the water be swapped out for a different liquid? Is it a good idea to use Mineral oil instead of H2O or distilled water for that GPU coolant? – makerofthings7 Apr 05 '13 at 18:32
  • @makerofthings7: No, it can't. Water cooling is for *water*. The flow speed of oil will not be high enough due to the much higher viscosity. – Sven Apr 05 '13 at 18:43
  • Edited my post to answer your follow up questions. – Ryan Ries Apr 05 '13 at 18:51