Article from DLRmagazine 173: DLR's CARA and CARO supercomputers support research projects with their exceptional computing power
High-speed nodes
High Performance Computing (HPC)
This interconnected system is commonly referred to as a cluster, in which the computers work together seamlessly and function as a single, powerful supercomputer, processing vast quantities of data at impressive speeds.
In 1937, Konrad Zuse developed the world's first fully programmable computer. Powered by the electric motor from a vacuum cleaner, and with a clock speed of approximately one hertz, it went down in history as the first digital computer. The era of vacuum cleaner motors has long since faded into history; motherboards, central processing units, graphics processors and disc drives have become vital components. Nowadays, computers can perform several billion calculations per second, but even these modern systems are frequently pushed to their limits by scientific researchers. Simulating phenomena such as the airflow around aircraft wings or the behaviour of the propellants inside rocket engines demands computers with much higher performance. DLR operates two such supercomputers at its sites in Dresden and Göttingen.
The HPC-Supercomputer CARA
CARA is located in the Lehmann Center Data Center at the Dresden University of Technology.
High Performance Computing (HPC) surpasses the power of conventional personal computers by leveraging a network of individual computers referred to as nodes. This interconnected system is commonly referred to as a cluster, in which the computers work together seamlessly and function as a single, powerful supercomputer, processing vast quantities of data at impressive speeds. A typical cluster processes data approximately 10,000 times faster than a standard laptop. DLR's two HPC supercomputers are named CARA and CARO after their official title – 'Computer for Advanced Research in Aerospace'. CARA is located in Dresden and CARO in Göttingen. They are operated by the DLR Institute of Software Methods for Product Virtualization, which functions as DLR's HPC competence centre. "Not everyone who would benefit from DLR's HPC clusters is familiar with them yet. The competence centre supports DLR's institutes by providing advice to new users who are interested in accessing the systems," says Daniel Molka from the Institute of Software Methods for Product Virtualization. The competence centre is run by the High-Performance Computing Department. The operation of the cluster is carried out in cooperation with the Center for Information Services and High Performance Computing (Zentrum für Informationsdienste und Hochleistungsrechnen; ZIH) at TU Dresden and the Society for Scientific Data Processing (Gesellschaft für wissenschaftliche Datenverarbeitung Göttingen; GWDG).
CARA and CARO by the numbers
The HPC-Supercomputer CARO
The CARO high-performance computer is located in Göttingen. Among other things, it is used for the numerical simulation of aircraft and wind turbines.
The CARA supercomputer is equipped with almost 150,000 processor cores. CARO in Göttingen has approximately 175,000 cores. In the course of an upgrade, some nodes were equipped with graphics cards (GPUs). When it entered operation in July 2022, CARO secured a place among the top 150 fastest computer systems in the world. This cluster achieves a computing speed of 3.46 petaflops, meaning it can execute 3460 trillion floating point operations per second.
Formidable computing power at DLR
Researchers use the computational power of CARA and CARO to carry out complex simulations or work with artificial intelligence methods. Currently, approximately 25 DLR institutes are utilising the capabilities of the two supercomputers and others have shown interest. In particular, institutes specialising in aeronautics – such as the Institute of Aerodynamics and Flow Technology and the Institute of Aeroelasticity – make extensive use of these supercomputers.
Flow simulation of a wind turbine
CARO and CARA are used to simulate the aerodynamics and flow behaviour around wind turbines.
CARA and CARO often come into play to conduct elaborate simulations. By employing physical and mathematical models, various components of aircraft or wind turbines, for example, can be translated into the virtual world. To gain a better understanding of flow physics as a basis for technology development, these simulations can become exceptionally complex. "When we model a wind turbine, we go into the smallest details." explains Cornelia Grabe from the Institute of Aerodynamics and Flow Technology. This approach enables the researchers to conduct virtual tests on flow behaviour or structural mechanics and to investigate and evaluate important performance parameters such as efficiency, durability or sound generation. As the level of detail increases and additional phenomena are taken into consideration, the demand for computing power rises. "The advantage is that we can use such simulations to test physical experiments in advance – in the past we had to build prototypes and carry out tests. Today, much of this is done virtually," says Dirk Schneider from the Institute of Space Propulsion. However, despite the significant advancements in simulations, experiments remain essential, as, ultimately, the results of the simulations must be confirmed in the real world.
Flow simulation
This flow simulation illustrates the behaviour of air flowing over an aircraft wing. The flow pattern changes depending on the aircraft's velocity and angle of attack.
Flow simulations take up the largest share of CARA's and CARO's computing time. In one application, researchers at the DLR Institute of Aerodynamics and Flow Technology conducted simulations of the airflow around an aircraft wing during take-off. Depending on how various factors such as the angle of attack or the speed of the aircraft varied, the flow phenomena that occurred around the wing also changed. The Japanese space agency JAXA carried out measurements in a wind tunnel using a model and was able to compare the results of the simulation and their experiment.
Simulation of a rocket in reverse flight
The Spacecraft Department in Göttingen is conducting research into the aerodynamics of reusable launch vehicles. This image shows the thermal load on a rocket's exterior during a braking manoeuvre in reverse flight, when three of its nine engines are running. Normally, the rocket would heat up considerably during re-entry into Earth's atmosphere. The exhaust plume created by the three central engines shields the underside of the rocket from the hot air flow and prevents this.
Simulations also play an important role in space exploration. Researchers at the DLR Institute of Aerodynamics and Flow Technology are developing simulation methods for this purpose in order to predict the complex physical processes in rocket engine combustion chambers efficiently and accurately. Understanding how acoustics and heat release are related is important for predicting combustion instabilities. These are strong acoustic oscillations that can occur in combustion chambers and can even lead to the destruction of the propulsion system and loss of the spacecraft. Due to the complex physical processes in the combustion chamber and the small, turbulent vortices, these simulations require the very high computing power of an HPC cluster.
Cabin with thermally representative mannequins
This HPC simulation shows a cabin in which thermally representative mannequins are sitting. It has been used, for example, to investigate various ventilation concepts. Among other things, the researchers investigated how exhaled particles are distributed in the surrounding environment.
The combustion and heat transfer processes in liquid-propellant rocket engines are being modelled at the DLR Institute of Space Propulsion in Lampoldshausen. When these engines are being developed, it is important to be able to predict the thermodynamic states in the combustion chamber as well as the cooling processes. For these simulations, the researchers take up considerable resources at the CARO HPC cluster in Göttingen. "The simulations are becoming increasingly complex," says Sabine Roller, Director of the Institute of Software Methods for Product Virtualization, adding: "This is because with increasing computing capacity, more detailed representations are possible in the simulations." In addition, institutes with a focus on artificial intelligence are increasingly using the computing power of supercomputers. Research areas that work with Big Data are also turning to CARO and CARA. This is especially the case when extensive statistical models that incorporate large amounts of data are being created.
Software options for applications
Certain computer software is required for a simulation to run on CARA or CARO. This software is typically developed by researchers from the various DLR departments. Another option is to access open-source software. Here, the source code is freely accessible, so that changes can be made to suit the project and the research. Commercial software typically allows for individual parameter adjustments, but otherwise often has limited flexibility. For this reason, DLR scientists typically use software that they have written themselves or that can be adapted. They are supported in this by experts from the Institute of Software Methods for Product Virtualization. These specialists possess in-depth expertise in computer science and programming and have a comprehensive understanding of the computers themselves.
High energy consumption put to good use
Simulations can support research projects
Developing aircraft and their engines is extremely complex. Simulations accelerate and facilitate this process.
The computing power of CARA and CARO requires a lot of energy. With a power consumption of up to 1200 kilowatts (CARA) and 800 kilowatts (CARO), the high-performance computers are among the major power consumers at DLR. The two computers use a hot-water cooling system with a flow temperature of 35 degrees Celsius – for comparison, cooling water temperatures are usually between 15 and 18 degrees Celsius and are significantly more energy-intensive. The Dresden data centre uses the waste heat from the water cooling to heat the surrounding buildings. There are also plans to integrate the data centre into Dresden's district heating network. At CARO in Göttingen, the surplus heat is used to warm the greenhouses operated by the Faculty of Agricultural Sciences.
The first computers
Konrad Zuse next to the Z3
The Z3 was the world's first functioning automated program-controlled calculating machine.
Credit:
Deutsches Museum
The use of computers in aeronautical research dates back to the 1940s. Konrad Zuse (1910-1995) developed his first electronic calculating machines during the 1930s. In 1939, Zuse introduced the experimental Z2 computer to experts at the German Aviation Research Institute (Deutsche Versuchsanstalt für Luftfahrt; DVL) in Berlin-Adlershof – one of DLR's predecessor organisations. Recognising the potential of this innovative computing machine, the DVL provided support for Zuse's work. By 1941, Zuse had unveiled the Z3, the world's first operational, programmable and fully automatic calculating machine. This could be used for the analysis of flutter phenomena in aircraft – a difficult and time-consuming task to carry out manually. In early 1945, Zuse successfully transported the improved Z4 to the Aerodynamic Research Institute (Aerodynamische Versuchsanstalt; AVA) in Göttingen, another of DLR's predecessors, where it was temporarily stored. The Institute's directors were astounded by the machine's performance during a demonstration.
Typically, the simulations are executed concurrently on the supercomputers. By doing so, the DLR experts aim for a continuous and high utilisation of the computers. Most of the calculations only use a small part of the machines, while a few take up a quarter or half of their current capacities. In addition to the varying number of nodes used, the calculations performed by the users also differ greatly in their runtimes. These range from a few minutes to several days. The resources must be distributed as fairly as possible among the participating institutes and their users. Keeping the utilisation high requires good planning.
Processing huge amounts of data
CARA in Dresden has almost 150,000 processor cores.
When we purchase a new laptop, it is certain that this model will soon become obsolete due to further technological developments. Supercomputers are no different. Keeping the high-performance computers up to date with the latest technologies is vital for research work with increasingly complex simulation requirements. Therefore, DLR intends to operate each supercomputer for a service life of approximately five to six years before they receive the hardware upgrades necessary to meet the demand for increasingly complex simulations. "At present, there is no better alternative to high-performance computers. We anticipate that people will still be working with them in 30 years' time," says Daniel Molka. The increasing complexity of aircraft simulation models means that the HPC clusters will continue to be needed in the future. The current emphasis on climate friendly air transport may lead to future aircraft designs that differ significantly from those that are in use today. However, before these new designs can be built and tested, numerous simulations will have to be run on DLR's supercomputers.
