Big Data from space - seeking a solution for the flood of data from space

Earth observation satellites supply us with important data so that we can rapidly identify and record processes like environmental and climate change, or the movement or disappearance of glaciers. Up-to-date maps can be produced for emergency relief teams in cases of disasters like floods or earthquakes.

However, such information is based on huge mountains of data. The satellites of the European Copernicus programme are among the world's largest data producers. They are currently producing with their high-resolution instruments a daily volume of some 20 terabytes. That would be equivalent to a HD film that runs about a year and a half. To this total must be added the data treasures from national missions like TerraSAR-X and TanDEM-X, and increasingly from other sources as well, like the internet or measurement stations. Processing and analysing such a large amount of heterogeneous data is one of the "big data" tasks posing a challenge for the future of our digital society.

In order to address it, some 650 experts are meeting at the "Big Data from Space" conference being held from 19 to 21 February 2019 and hosted by the German Aerospace Center (DLR) in Munich. It is jointly organised by the European Space Agency (ESA), the European Union Satellite Centre, and the Joint Research Center. Support is being provided by the European Space Imaging, OHB, GAF AG and Quantum companies.

During the reception at the Munich Residence, Bavaria's minister of economic affairs, Hubert Aiwanger, stated, "It is remarkable, how much space technology is also affecting daily life: navigation, head-up displays and the entire area of assistance robotics are just three of many examples. Today the subjects of climate change and environmental protection are at the top of the political agenda, not only in Bavaria. Both are global concerns, and the latest technology is required for both. Research on the climate system, the atmosphere, tropical rain forests, the oceans and the poles—in all these areas space pursuits can supply crucial data for better understanding and protecting ecosystems. Some 65,000 engineers, technicians and specialists work in Bavaria's flourishing aerospace sector and generate an annual turnover of ten billion euro."

Artificial intelligence controls data selection

New ideas and concepts are required to process the large amounts of data and convert them into information. Artificial intelligence plays a large role because it is a very powerful tool for dealing with big data in particular. For example, Prof. X. Zhu, a DLR scientist at Munich Technical University (TUM) is investigating the application of such methodologies. Together with her team, Prof. Zhu is creating explorative algorithms that combine signal processing and artificial intelligence, especially machine learning. This should significantly improve the harvesting of global geoinformation from satellite data and make innovations possible in the environmental and geosciences. The fusion of petabytes of complementary data from sources with geo-relevance, from earth observation satellites to social networks, is the next step being made possible by pioneering data-science algorithms. The results have the potential to be useful for addressing major challenges that could not be well tackled before, such as recording and mapping urbanisation worldwide—one of the most important megatrends of global change.

Not only satellite earth observation is confronted with this big-data challenge. Viewing in the other direction, from Earth into space, also generates huge amounts of data. Telescopes like the Square Kilometre Array (SKA) in South Africa and Australia and ESA space telescopes such as GAIA and EUCLID produce enormous amounts of data. Systematic analysis of archived data with self-learning AI programmes is becoming increasingly important for astronomers.

Intelligent search portals supply the desired information

The yield from earth observation satellites and modern telescopes is evidence that data science and artificial intelligence methodologies are essential. In order to manage and evaluate the mass of data, suitable storage and efficient access possibilities are required. For potential users, having easy access is the decisive factor. This is now usually in the form of online portals like the CODE-DE (Copernicus Data and Exploitation Platform - Deutschland) of DLR's Space Administration division. Users can directly access and process all the data there. It is a comfortable online workspace for the satellite data that is made available without cost in the Copernicus programme. That frees users from having to provide their own local storage space and computer capacity.

The data portals themselves are also a subject of research. What should an ideal platform be like and what search options should users have? Taking Internet search machines as a model, images of distant planets and of Earth's surface should be easy to find after an information query: "On which heavenly body is there apt to be water?" or "Which parts of Germany are likely to suffer drought in summertime?" Modern portals should be able to provide the right data sets as answers to such queries. Long-term preservation of this global information is also essential. As with conventional libraries, it must be assured that satellite data can be safeguarded, found, and retrieved for a long time. This is being done, for example, at the German Satellite Data Archive (D-SDA) of the Earth Observation Center at DLR in Oberpfaffenhofen.

Time series - curse and blessing in one