A new era in geoinformation with 'terrabyte'
The German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt; DLR) and the Leibniz Supercomputing Centre (Leibniz-Rechenzentrum; LRZ) of the Bavarian Academy of Sciences and Humanities have commenced operations with 'terrabyte' – one of Europe's largest scientific platforms for analysing Earth observation data. For the first time, the platform enables researchers to efficiently analyse global data sets from Earth observation satellites at the highest resolution. With regard to global change, researchers can thus also determine the effects on humans and the environment more precisely and develop suitable courses of action. This applies, for example, to the drastic changes in the polar regions, the increasing frequency of extreme weather events and natural disasters, and the growing urbanisation that is occurring worldwide. terrabyte is secure and independent of commercial operators. The launch of the large-scale computing facility took place on 14 June 2023 at the LRZ in Garching, near Munich, with guests from government and academia.
terrabyte provides current and historical Earth observation data collected over five decades as well as intelligent Big Data analysis methods using cloud services. In addition to extensive data from the German TerraSAR-X/TanDEM-X missions and the US Landsat Program, these also include all products from the Sentinels of the European Copernicus Earth observation programme. In addition, users can integrate information from social networks into their analysis.
"terrabyte is a big step forward for geoinformation research. Big Data, the latest AI-based processing methods and high-performance computing are combined in a unique way. terrabyte not only forms the foundation for our science but is also a significant economic driver for Bavaria and Germany. The basis for this is the cooperation with our partners and the associated knowledge transfer," said Anke Kaysser-Pyzalla, Chair of the DLR Executive Board.
Stefan Dech, Director of the German Remote Sensing Data Center at DLR added: "With the Leibniz Supercomputing Centre, we have found the ideal cooperation partner for this challenging project. With terrabyte, we have an autonomous and secure platform with comprehensive global datasets from Earth observation that we can access together with our partners. At the same time, terrabyte forms a technological bridge between our Oberpfaffenhofen and Garching sites."
"terrabyte meets the diverse requirements for efficient data storage, data science, AI and cloud services, and even high-performance computing and supercomputing. With our many years of know-how in these areas, we combine all these technologies into a customised service for Earth observation. It is not only DLR researchers who benefit from the huge amount of historical and current satellite data via terrabyte, but also scientists in Bavaria and Germany. A very exciting, but also challenging project which was made possible thanks to the intensive and excellent cooperation between the colleagues from DLR and the LRZ," says Dieter Kranzlmüller, Chairman of the Board of Directors at the LRZ.
Top technological performance, yet user friendly
terrabyte has a total of approximately 50 petabytes of online storage capacity. (One petabyte corresponds to 1015 bytes or the content of about 223,000 DVDs.) The system is integrated into the existing infrastructure of the LRZ in Garching and connected to the German Satellite Data Archive operated by DLR in Oberpfaffenhofen. There, all data are archived securely and for the long-term, and can be uploaded to terrabyte as required. A network connection with a data rate of 100 gigabits per second ensures the rapid transfer of archive data to the terrabyte production system. The latest generation of CPU and GPU resources enable efficient operations in the cloud environment. For processing and analysing the data, the terrabyte platform is equipped with the latest AI hardware and delivers 1.3 petaflops per second of computing power – that is 1.3 quadrillion floating-point calculations per second.
terrabyte offers a range of software-based services so that the research community can make use of the data resources. A large part of the data are pre-processed so that they can immediately be incorporated into applications (Analysis Ready Data; ARD). Software tools and systems make it easier for scientists to implement their own algorithms and use them in the terrabyte environment. The researchers at the DLR Earth Observation Center (EOC), other DLR institutes and their partners are developing algorithms that incorporate AI and machine learning. This will enable a wide range of global geoscientific analyses to be carried out, which were previously only possible on commercial platforms due to the volumes of data and limited computing power available.
The future of Big Data in Earth observation
The potential of Big Data analysis of Earth observation data is demonstrated by global products such as DLR's 'World Settlement Footprint'. This data set, which shows the development of the populated areas on the planet, is now a standard used by the United Nations, the World Bank and many others. The map was generated using many petabytes of data from the Copernicus and Landsat missions, in combination with the global digital elevation model created using data acquired by the TanDEM-X mission. Another example is AI-based global analysis of offshore wind farms under construction or in operation. These are an important indicator for assessing how successful efforts to achieve carbon dioxide neutrality are. The evaluations also provide information on the extent to which economic independence can be achieved through this energy supply technology.
The routine operation of terrabyte has now started. terrabyte offers the project partners a platform to continue their research work in the field of high-performance computing and cloud computing. Together with the Leibniz Supercomputing Centre, DLR intends to take advantage of the opportunity to continue initiating new IT developments – to both enhance scientific investigations and to strengthen Germany's position as a prime location for research.
About the project
'terrabyte' is a cooperation between the Leibniz Supercomputing Centre (Leibniz-Rechenzentrum; LRZ) of the Bavarian Academy of Sciences and Humanities (Bayerische Akademie der Wissenschaften; BAdW) and the German Remote Sensing Data Center (Deutsches Fernerkundungsdatenzentrum; DFD) of the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt; DLR). Its goal is to make current and historical Earth observation data centrally and publicly accessible, and usable for scientific research.
DLR has invested approximately 19 million euros in the computing infrastructure, which was specifically adapted to the requirements of Earth observation, and in the development of the software environment. The LRZ is responsible for the operation of the High Performance Data Analytics Platform, bears the running costs and, in addition to considerable personnel resources, is contributing its many years of expertise to the cooperation.