Traditional and Modern Data Center Cooling Systems
Today’s continuously increasing computing capacities are changing server standards as well as data center infrastructure standards. The constant development and increase in computing capacity are causing important hardware components such as CPUs, GPUs, and DPUs to change and evolve. The most significant obstacle in the development of these components is the increasing energy and heat. This increase in energy and heat demand requires a lot of cooling capacity within data centers. The rising need for cooling also necessitates a substantial amount of energy. Due to this cycle, data center white space, operation, and consumption costs are progressively increasing.
Apart from the cost, if this heat is not managed effectively, it can lead to interruptions, malfunctions, and even data loss within the data center. Therefore, modern data center cooling systems are increasingly becoming the preferred choice. When planning new data centers, it will be very beneficial to analyze and integrate modern data center cooling system infrastructures to meet this increasing heat load.
The main reasons for the need for modern cooling systems include;
- High Cooling Efficiency
Liquid cooling systems can transfer heat much more effectively compared to cooling mediums like air. They can remove heat directly from server components more quickly and efficiently than air cooling. - Higher Density of Computing Power
Data centers, aiming to optimize space usage and increase energy efficiency, increasingly need more dense computing power. Liquid cooling supports higher density computing loads, allowing data centers to make better use of their existing spaces. - Energy Efficiency and Cost Savings
Liquid cooling systems are generally more energy-efficient compared to air-based cooling systems. This translates to lower cooling costs and thus lower operational costs. Reducing energy consumption is a significant saving, especially for large-scale data centers. - Reduction of Environmental Impact
The increase in energy efficiency contributes to the reduction of the carbon footprint. The use of liquid cooling systems helps reduce the overall energy consumption, thereby reducing the environmental impact of data centers. - Heat Recovery Potential
Some liquid cooling systems can recover heat collected during cooling and use this heat for other purposes, such as heating buildings or providing hot water. This further enhances energy efficiency and offers a sustainable solution. - Extended Equipment Lifespan
Effective cooling prevents the overheating of server and storage components, which extends the life of the hardware and reduces failure rates. This results in cost savings in the long term and increases system reliability.
Below you can see the current energy requirements of CPUs and GPUs and the computing power they provide. The cost of high computing power is increased energy and heat.
Key Differences
Traditional Data Center Cooling
Air-Based Cooling: Predominantly relies on CRAC (Computer Room Air Conditioner) and CRAH (Computer Room Air Handler) units. These systems circulate cooled air within the data center and maintain temperature by creating hot and cold air corridors.
High Energy Consumption: Traditional air cooling systems are generally energy-inefficient because they require significant energy to cool and circulate large volumes of air.
Low Environmental Sustainability: High energy consumption increases the carbon footprint and raises concerns about environmental sustainability.
Temperature and Humidity Control: Designed to cool large areas, but their effectiveness may decrease with increased server density.
Modern Data Center Cooling
Liquid Cooling Technologies: Modern data centers are turning to liquid cooling solutions, such as direct liquid cooling (DLC), in-row and in-aisle cooling, and immersion cooling technologies. These techniques remove heat more directly and effectively.
Energy Efficiency: Modern cooling solutions significantly reduce energy consumption and provide higher energy efficiency. This contributes to reduced operational costs and decreased environmental impact.
Environmentally Friendly Coolants: The use of coolants with a low global warming potential (GWP), such as hydrocarbons, has increased, enhancing environmental sustainability.
Scalability and Modularity: Modern cooling systems are designed to accommodate the growth and changing needs of data centers. These systems typically offer more flexible and scalable solutions.
Heat Recovery Potential: Some modern cooling systems can recover waste heat produced during cooling and use it for other purposes, thereby further increasing energy efficiency.
How Traditional and Modern Data Center Cooling Systems Work
Legacy Air-Cooling (CRACs/CRAHs) (Maximum 15 kW per Cabinet)
Legacy air-cooling systems, commonly referred to as Computer Room Air Conditioners (CRAC) and Computer Room Air Handlers (CRAH), are traditional air-cooling technologies widely used in data centers and computer rooms. These systems provide a flow of cold air that spans the room to cool electronic equipment. The operating principles of CRAC and CRAH units are similar, but they show some differences in their cooling mechanisms.
CRAC (Computer Room Air Conditioner)
Cooling Mechanism: CRAC units use a compressor to compress and expand refrigerant. This process absorbs heat, which is then expelled to the atmosphere with the help of an external condenser. After the cooling process is complete, the cooled air is blown back into the room.
Control and Management: CRAC units are usually equipped with thermostats to monitor and adjust room temperature. These units manage air temperature by controlling the pressure and temperature changes of the refrigerant.
CRAH (Computer Room Air Handler) Units
Cooling Mechanism: CRAH units use external cold water for cooling. This water cools the warm air taken from the room as it passes through a heat exchanger (coil). As a result, the cooled air is blown back into the computing room.
Water Cooling Cycle: The cooling water is typically chilled by a chiller and then pumped to the CRAH units. These systems leverage the cooling capacity of water to reduce air temperature.
General Operating Principle
Both types of systems draw warm air from the room, cool it with their respective cooling mechanisms, and then blow the cooled air back into the room. This process helps prevent electronic equipment from overheating.
Cold air is usually distributed through raised flooring, so it directs cold air directly to the fronts of servers and other equipment. Warm air exits from the back of the equipment, rises to the ceiling, and returns to the cooling system.
Advantages and Disadvantages
Advantages: CRAC and CRAH systems are widely used, and there is extensive knowledge about their installation and maintenance in many data centers and computer rooms.
Disadvantages: Traditional air-cooling systems can be less effective in terms of energy efficiency compared to modern cooling techniques (e.g., liquid cooling or in-line air cooling systems). They also require careful arrangement to prevent the mixing of hot and cold air flows.
In-Row Air-Cooling (Maximum 25 kW per Cabinet)
In-Row Air-Cooling is a cooling method designed for temperature control in data centers. This system consists of cooling units placed between the cabinets (racks), usually near the hot air corridors. In-Row Cooling aims to more efficiently manage the heat produced by equipment in data centers and helps reduce energy consumption compared to traditional centralized cooling systems.
- Layout and Design
Cooling units are placed right next to server cabinets to optimize hot and cold air corridors. This design ensures that the cooling capacity is close to the sources of heat, capturing and managing the heat more effectively. - Hot Air Capture
In-Row cooling units directly capture the hot air emitted from the back of server cabinets. This approach minimizes the mixing of hot and cold air and enhances cooling efficiency. - Air Cooling and Circulation
The captured hot air is passed through cooling coils within the cooling units. These coils are typically cooled with water or refrigerant gas. After the air is cooled, it is directed back into the cold air corridor, where it is reabsorbed by the fronts of the server cabinets. This process ensures efficient circulation of cold air. - Heat Exchange
In-Row cooling units perform the necessary heat exchange on-site. This is ideal for quickly and directly removing heat. The cooling medium (water or refrigerant gas) can be recooled through a central cooling system or an external cooling tower. - Modularity and Scalability
In-Row cooling systems are often modular and scalable. This means that additional cooling capacity can be easily added as the data center grows or as cooling needs change.
Advantages
Energy Efficiency: By placing cooling capacity directly next to heat sources, it reduces energy consumption.
Reduced Air Mixing: Minimizes the mixing of hot and cold air, thus increasing cooling effectiveness.
Flexibility and Scalability: Adapts to the current and future cooling needs of the data center.
Rear Door Heat Exchanger Cooling (RDHx) (Maximum 55 kW per Cabinet)
The Rear Door Heat Exchanger (RDHx) cooling system is a cooling solution mounted to the back of cabinets (racks) in data centers. This system utilizes the cooling capacity of water to effectively remove the heat produced by equipment inside the cabinet. RDHx units aim to increase energy efficiency in cooling high-density server and network equipment and provide better thermal management compared to air-cooled systems.
- Heat Emission
Server and network equipment in data center cabinets generate heat while operating. This heat is dispersed through the air inside the cabinet. - Role of the Cooling Door
The RDHx system is a cooling door mounted on the back of the cabinet. This door contains one or more heat exchangers (radiators). The hot air exiting the back of the cabinet is cooled as it passes over these heat exchangers. - Water Cycle
There is a closed circuit through which water circulates in the heat exchangers of the cooling door. This water is typically cooled by a cooling tower or another cooling system located outside the data center. The cold water circulating in the system cools the hot air coming from inside the cabinet as it passes over the heat exchangers, thereby absorbing the heat. - Heat Removal
During this thermal interaction between hot air and cold water, the water absorbs the heat and warms up. The heat-carrying water then returns to the cooling tower or central cooling system to be recooled. - Energy Efficiency
RDHx systems are more energy-efficient compared to traditional air-cooled systems because they directly target the hot air at the back of the cabinet and do not require additional external airflow for cooling. This reduces the overall cooling needs of the data center and lowers energy consumption.
Advantages
Low Energy Consumption: RDHx reduces the energy required for cooling and increases the energy efficiency of the data center.
Space Saving: Since it is mounted to the back of the cabinet, it does not require additional space and can be easily integrated into the existing layout of the data center.
Flexibility: It can accommodate different cabinet sizes and configurations, thus responding to various cooling needs.
Liquid-to-Chip (Maximum 75 kW per Cabinet)
Liquid-to-Chip server cooling systems use liquid cooling technology to efficiently manage heat through cooling blocks placed directly on server components. These systems are an advanced cooling solution commonly used in high-performance computing (HPC), data centers, and other energy-intensive computing infrastructures.
- Installation of Cooling Blocks
Cooling blocks are mounted onto each processor or high heat-generating component using a thermally conductive material (usually thermal paste). These blocks are designed to transfer heat from the component to the cooling liquid. - Circulation of Cooling Liquid
A pump activates the flow of the cooling liquid within the system. This liquid, often water or a special non-conductive fluid, absorbs heat from the cooling blocks and carries it through the system. - Heat Transfer to the Cooling Liquid
As the cooling liquid flows over the cooling blocks, it absorbs the heat produced by the processors or other components. This liquid makes direct contact with the surfaces of the components, absorbing heat much more efficiently. - Removal of Heat
The heat-saturated cooling liquid is directed to a heat exchanger (radiator) to be expelled from the system. Here, the heat carried by the liquid is dispersed into the environment through air flow over the radiator’s surface or through a water cooling system. - Recirculation of the Cooling Liquid
The cooled liquid is returned to the system, and this cycle repeats continuously, ensuring that the components are consistently cooled effectively.
Advantages
High Cooling Efficiency: Liquid carries and disperses heat much more efficiently compared to air cooling.
Quiet Operation: Liquid cooling systems operate more quietly compared to air-cooled systems.
Improved Heat Management: Reduction in hot spots and better overall control of system temperature.
Immersion Cooling
Single-Phase Immersion Cooling (Maximum 184 kW per Cabinet)
Single-Phase Immersion Cooling is a method of cooling electronic components or entire servers by immersing them in a non-electrically conductive liquid. This cooling method is particularly used in energy-intensive technologies such as high-performance computing (HPC), data centers, and blockchain mining.
Key Features and Operating Principle
Immersion of Electronic Components in Liquid: In this system, heat-generating electronic components (CPUs, GPUs, memory modules, etc.) are directly immersed in a cooling liquid. This liquid is specially formulated to be non-conductive, meaning it does not carry electrical current and does not damage the electronic components.
Heat Transfer: There is direct heat transfer from the electronic components to the liquid. As the components operate, they transfer their heat to the liquid, which absorbs the heat and helps stabilize the temperature within the system.
Single-Phase Process: As the name suggests, in the single-phase immersion cooling process, the cooling liquid does not evaporate or change to a gas phase. The liquid remains in a liquid state throughout and the heated liquid is circulated through the cooling system and cooled via a heat exchanger.
Energy Efficiency and Effectiveness: This method has much higher cooling efficiency compared to traditional air cooling systems. The cooling liquid quickly and directly absorbs heat from the surfaces of the components, facilitating effective heat removal.
Advantages
High Cooling Capacity: Single-phase immersion cooling can effectively manage intense heat loads and prevent overheating in high-performance systems.
Energy Efficiency: This cooling method enhances energy efficiency because the amount of energy required for cooling is less compared to traditional air-cooling systems.
Reduced Noise Levels: Since there are no fans or other moving parts, single-phase immersion cooling systems operate much more quietly.
Longer Component Lifespan: The cooling liquid protects components from dust and oxidation, thereby extending the hardware’s lifespan.
Heat Recovery Potential: The waste heat collected during cooling can be used as a heat source in other systems, further enhancing energy efficiency.
Two-Phase Immersion Cooling (Maximum 250 kW per Cabinet)
Two-Phase Immersion Cooling is a cooling technology that cools electronic components or complete servers by utilizing the processes of evaporation and condensation. This method is specifically designed for high-density computing applications, making it an ideal cooling solution for data centers, high-performance computing (HPC), and blockchain mining.
Key Features and Operating Principle
Direct Liquid Contact: In this system, electronic components are directly immersed in a non-electrically conductive liquid, specifically chosen to absorb the heat produced by the components.
Phase Change through Heat Transfer: The heat transferred from the components to the liquid causes the liquid to evaporate. This evaporation process absorbs a large amount of heat, effectively removing it from the electronic components.
Condensation and Heat Dissipation: The evaporated liquid is transported to a condenser, which is part of the cooling system. In the condenser, the vapor returns to a liquid state (condenses), releasing heat during this process. The cooled liquid is then recirculated back to the electronic components to continue in the system.
Energy Efficiency and Cooling Performance: Two-phase immersion cooling has high energy efficiency and cooling capacity, capable of directly and effectively removing heat from components.
Advantages
High Cooling Capacity: The phase change has a high heat dissipation capacity, allowing the system to manage very intense heat loads.
Low Energy Consumption: It is a passive cooling method that does not require active cooling components (e.g., fans), significantly reducing operational energy consumption.
Extended Equipment Lifespan: The liquid protects electronic components from dust, oxidation, and other environmental harms, potentially extending the hardware’s lifespan.
Quiet Operation: Compared to active air cooling systems, two-phase immersion cooling systems operate nearly silently, reducing noise pollution.
Heat Recovery Potential: The heat expelled from the system can be used for other purposes through heat recovery, further enhancing energy efficiency.
Key Differences
Energy Efficiency: Two-phase immersion cooling is generally more energy efficient than single-phase immersion cooling because it absorbs more heat during the phase change process.
Cooling Capacity: Two-phase systems can more effectively manage intense heat loads due to their ability to exploit the latent heat of vaporization in the cooling process.
Complexity and Cost: Two-phase immersion cooling systems require more complex and costly equipment to manage the phase change process. This includes specialized components like condensers and controlled pressure environments to ensure efficient phase transitions.
Cooling Liquid Properties: Both systems use cooling liquids with distinct properties; however, two-phase systems generally prefer liquids with a lower boiling point to facilitate the rapid vaporization and condensation cycles necessary for effective heat removal.
Differences Between Hydrocarbon Liquid (Single-Phase) and Fluorocarbon Liquid (Two-Phase)
Chemical Structure
Hydrocarbons are organic compounds consisting solely of carbon and hydrogen atoms. Common hydrocarbons used as cooling fluids include propane (R290), isobutane (R600a), and ethane. These are typically utilized in single-phase immersion cooling systems due to their efficient thermal conductivity and minimal environmental impact.
Fluorocarbons contain carbon, fluorine, and sometimes hydrogen. These compounds are often classified into categories such as CFCs (Chlorofluorocarbons), HCFCs (Hydrochlorofluorocarbons), HFCs (Hydrofluorocarbons), and HFOs (Hydrofluoroolefins), commonly used in two-phase immersion cooling systems.
Environmental Impact
Hydrocarbons have very low Global Warming Potential (GWP) values and do not damage the ozone layer, making them more environmentally favorable compared to fluorocarbons. This makes hydrocarbons a preferred choice in applications where environmental impact is a significant concern.
Fluorocarbons especially CFCs and HCFCs, have the potential to harm the ozone layer and contribute to global warming with high GWP values. HFCs do not harm the ozone layer but still possess high GWP values. HFOs are seen as a more environmentally friendly alternative due to their low GWP and lack of ozone-depleting characteristics.
Flammability and Safety
Hydrocarbons are highly flammable, which limits their use and requires additional safety measures. However, when properly managed, they can be used effectively and safely, especially in small-scale and residential cooling systems.
Fluorocarbons are generally non-flammable or have low flammability, providing a safety advantage, particularly in large-scale cooling systems. This characteristic makes them a safer option for applications where risk mitigation is crucial.
Performance and Energy Efficiency
Hydrocarbons offer excellent thermal conductivity and high energy efficiency. This makes them particularly appealing for applications where energy costs are significant.
Fluorocarbons provide stable cooling performance across a wide temperature range. However, some modern HFO compounds and HFCs stand out by offering high energy efficiency.
Application Areas
Hydrocarbons are preferred in small-scale cooling systems, residential refrigerators, and air conditioning systems, especially in regions with stricter regulations concerning energy efficiency and environmental impacts.
Fluorocarbons are commonly used in commercial and industrial refrigeration, air conditioning systems, and certain specialized cooling applications.
Disadvantages of Two-Phase Immersion Cooling Compared to Single-Phase Cooling
Higher Initial Costs
Two-phase immersion cooling systems generally require more complex and costly infrastructure. These systems need additional equipment to manage the liquid’s vaporization and condensation processes, such as condensers. In contrast, single-phase systems are typically simpler and less expensive.
Complex System Design and Maintenance
The design and maintenance of two-phase systems are more complex because they need to manage the processes of evaporation and condensation, ensure the circulation of the liquid, and optimize temperature control. In single-phase systems, the liquid continuously stays in the liquid phase, making the system simpler and easier to maintain.
Cooling Liquid Selection and Management
Two-phase immersion cooling systems require special cooling liquids that usually need to have high latent heat of vaporization. The cost of these liquids can be high, and managing the liquid (e.g., refilling, replacing, waste management) can be more complex.
Leakage and Safety Concerns
Although both systems take measures against leakage risks, two-phase immersion cooling systems’ processes of vaporization and condensation require additional safety and leakage prevention measures. Leaks can damage electronic components and pose safety risks in the operating environment.
Energy Management and Heat Recovery
In two-phase immersion cooling systems, managing and recovering the heat expelled during the condensation process may require more thought and equipment. This can impact the overall energy efficiency of the system.
General Disadvantages of Immersion Cooling
- High Initial Costs
The installation of immersion cooling systems requires specialized design and materials. The investment cost for these systems can be higher compared to traditional air cooling or liquid cooling systems. - Cost of Cooling Liquid
The special liquids used for cooling are often expensive and may require regular replacement or maintenance. Additionally, the storage and handling of these liquids can create extra costs and logistical challenges. - Maintenance and Repair Difficulties
When issues arise in the system, components need to be removed from the liquid, cleaned, and reinstalled. These processes can make maintenance and repair complex and time-consuming. - Compatibility Issues
Not all hardware components and materials may be suitable for immersion cooling. For instance, some components may need special coatings or designs to allow for immersion in liquid. - Leakage and Spillage Risk
Leakage or spillage of the cooling liquid is a significant concern. These incidents can damage hardware and pose safety risks in the work environment. - Environmental and Health Safety Concerns
The potential impacts of the cooling liquids on human health and the environment must be carefully considered. Disposal and recycling of the liquids must also be managed in accordance with environmental regulations.
Examples of Immersion Cooling
Differences Between Liquid Cooling and Immersion Cooling Systems
Liquid Cooling
Direct Cooling: Liquid cooling circulates the cooling fluid through pipes and radiators close to the heat sources (e.g., processors and graphics cards) to absorb heat. In this system, the cooling fluid passes over components through cooling blocks, absorbs heat, and then transports it to a radiator for heat dissipation.
Closed-Loop and Open-Loop Systems: Liquid cooling systems can be found in both closed-loop (maintenance-free, pre-assembled systems) and open-loop (customizable and expandable systems) configurations.
High Energy Efficiency: By directly removing heat from the components, it effectively dissipates heat, typically offering better performance than air cooling.
Immersion Cooling
Full Immersion: Immersion cooling submerges electronic components or entire servers into a non-electrically conductive liquid that absorbs heat. This liquid directly absorbs the heat produced by the components and can either evaporate or be cooled through a heat exchanger.
Two-Phase and Single-Phase Systems: Two-phase immersion cooling systems involve the liquid evaporating due to the heat from the components and then the vapor condensing back into a liquid. In single-phase systems, the liquid remains in a liquid state and circulates with pumps to absorb heat.
Superior Cooling Capacity: The ability to evenly absorb heat from the entire surface area of components makes immersion cooling extremely effective, especially in applications with high thermal density.
Key Differences
Application Method: Liquid cooling targets specific components, whereas immersion cooling submerges all electronic components or devices in a liquid.
Cooling Efficiency and Capacity: Immersion cooling typically has higher cooling capacity and heat dissipation efficiency compared to liquid cooling because it can directly absorb heat from the components and distribute heat more uniformly.
Installation and Maintenance: Liquid cooling systems are generally more traditional and familiar, while immersion cooling systems are newer and come with potentially higher initial costs and special maintenance requirements.
What questions should you ask manufacturers for Immersion Cooling?
Can I use fiber optics?
Ask about the compatibility of optical cables and modules for connection and signal stability.
Is it usable for all IT systems?
Ensure that server, storage, and network hardware manufacturers certify their compatibility with immersion cooling systems to avoid any issues.
What about hardware warranties?
Verify that server and hardware manufacturers support their products for use in immersion cooling systems to maintain warranty coverage.
How easy is it to service and technically intervene with the hardware?
Although it may not be as easy as with air cooling or traditional liquid cooling, confirm that technical interventions and servicing can still be efficiently performed.
Can it be operated in every data center?
Confirm that once the appropriate infrastructure is prepared, there should be no reason why it couldn’t operate in any data center.
What are the floor requirements?
Discuss whether the system can be installed on both raised floors and concrete slabs. Immersion cooling systems do not structurally require raised floors, but if used, ensure that the weight distribution is adequately balanced.
Are there any certifications to comply with?
Check if immersion cooling systems are compliant with common data center certifications. It is also wise to consult with manufacturers about any specific certifications needed for the equipment to be included in the system.
Comparative Representation of Air Cooling and Immersion Cooling Systems
Finally, modern cooling systems will become a necessity over time for certain workloads. Manufacturers compatible with Liquid Cooling are currently quite numerous, with nearly all server manufacturers supporting Liquid Cooling compatible systems and producing the appropriate hardware. Immersion Cooling, on the other hand, is still somewhat new and it may take some time for all server and hardware manufacturers to fully integrate this system. Planning according to the workload used is very important from the beginning.
See you soon.
