Machine learning provides novel pathways to assess the skill of climate models
April 2, 2020
Machine learning provides novel pathways to assess the skill of climate models
Machine learning, one of the important approaches of artificial intelligence (AI) and already successfully applied in many scientific disciplines, also has large potential to advance Earth system science. Together with the Climate Informatics group at the DLR Institute for Data Sciences (DLR-DW) in Jena and with the three other host institutions– the Max Planck Institute for Biogeochemistry in Jena, the University of Valencia and Columbia University New York - of the recently awarded European Research Council (ERC) Grant on “Understanding and Modelling the Earth System with Machine Learning (USMILE)”, DLR-IPA develops and applies AI methods for Earth system research.
Sophisticated computer models of the Earth’s climate project how climate change will affect regional trends such as the location and severity of rainfall, making them crucial tools to inform the work of decision makers in governments, civic planning and businesses. A wide variety of such models are under development in more than 40 research centres worldwide and participate in the World Climate Research Programme’s Coupled Model Intercomparison Project Phase 6 (CMIP6), currently chaired by DLR-IPA [1]. A new study published in Nature Communication led by the Imperial College London in collaboration with DLR-DW and DLR-IPA has developed a new approach to evaluate their skill using ML techniques [2].
This new approach – called Causal Model Evaluation - relies on recently developed causal inference algorithms [3]. The method allowed to create a digital ‘fingerprint’ of each climate model, where each fingerprint characterizes causal pathways through which different world regions are coupled within a given model. These coupling mechanisms in turn are known for being important drivers of regional weather patterns.
Data set-characteristic causal fingerprints, which can be used for model evaluation and intercomparison. Node colours indicate the level of autocorrelation (auto-MCI) as the selflinks of each component and link colours the interdependency strength (cross-MCI). Link-associated time lags (unit = 3 days) are indicated by small labels. Only the around 200 most significant links each for the reanalysis and for data from four climate models are shown. Links with lag zero, for which directions cannot be easily causally resolved, are not shown. (Graphics: [2] )
An appealing feature of this method is that it can also be applied to measurements of the real climate system. Comparing this true fingerprint with the modelled fingerprints, the authors were able to deduce how well each model captures reality. Excitingly, models with greater skill also performed better in terms of modelling rainfall patterns over Africa, the Americas, China, Europe or India. Probably even more importantly, ranking models by skill also led to a clearer picture concerning the expected level of rainfall changes over land areas during the 21st century.
Historical network comparison. Scatter plots showing the individual network comparison score, with HadGEM2-ES models taken as reference. F1 is a score that varies between 0 and 1 (perfect network match). Black crosses (red for reference) mark average results, grey dashed lines the average score excluding the reference itself. Causal model evaluation detects similarities between certain model groups. (Graphics: [2] )
“Modern data science approaches can be incredibly useful for climate science, and our work is a demonstration of this idea. The climate system with all its sub-processes is highly complex. Here we show, somewhat counterintuitively, that we can use this complexity to our advantage: we could only define those valuable skill metrics through the combination of system complexity and the application of causal inference methods. In contrast, we found that simpler ways of measuring model performance did not achieve the same goals.”, Dr Nowack explains. “This new approach provides a novel and objective pathway for process-oriented evaluation of the models with observations and for the identification of interdependencies in the CMIP ensemble” says Prof Eyring , Chair of the CMIP Panel and DLR-IPA coauthor of the study.
“The study is also highly interesting from a methodological point of view. Causal inference methods are particularly promising in climate science as we are often interested in inferring cause and effect relationships, meaning that that our approach could also accelerate our understanding of the climate system”, says Dr Runge from DLR-DW.
The findings of this study are promising news for more accurate climate change projections that are required for planning and investment to protect lives and livelihoods in a world of climate change. Dr Nowack says: “Our results imply that we can develop new data-driven ways to improve how well we can model regional climate change. I see great opportunities in using artificial intelligence approaches to study future changes in the most impactful extreme events such as droughts and floods.”
