AI brainstorms weather prediction

For as long as humans have wanted to know what the weather was going to do next, we’ve looked to technology to help us work out which atmospheric dynamics are portents of changing conditions.

From the 17th-century invention of the barometer to 21st-century Earth observation satellite radar, forecasting the weather has developed broadly apace with advances in engineering and science.

Generations of supercomputers have for decades been crunching through the massive datasets needed to inform weather forecasts, typically using Numerical Weather Prediction (NWP) models – objective calculations of changes to the mapped weather founded on physics-based mathematical equations. However, even the most advanced petaflop-powered compute platforms have limitations and are not necessarily the best way to meet the growing and diversifying demand for advance notice of impending weather events.

NWP processes take their time, and in some use cases, a faster forecast delivery is preferred to one that’s super-accurate. This is one of the reasons why researchers are developing ways in which artificial intelligence (AI) and its subsidiary discipline machine learning (ML) can assist – or improve on – routine NWP techniques.

Although most projects are still in development – AI and ML are, of course, continuously being ‘trained’ – early results from such initiatives indicate that AI and ML, working separately or in concert with existing computing systems, open new ways to determine an informed prognosis on how weather conditions will play out.

Here, E&T focuses on three notable projects: Colorado State University’s Machine Learning Probabilities platform, the Lenovo/University of Connecticut AI Nowcasting Project, and Microsoft Research’s DeepMC initiative.

Excessive rainfall, and the flash flooding it causes, remains one of the most difficult forecasting challenges for high-tech weather watchers. Sudden deluges often occur without advance changes in meteorological conditions that alert forecasters to an impending surge in precipitation – leaving them no time to warn those who’ll be impacted.

Because they’re so unpredictable, flash floods can be devastatingly destructive. An additional complication for predicters is that, unlike other weather hazards, the definition of ‘excessive’ rainfall varies from location to location, depending on the local and regional climatology, as well as drainage conditions at surface level.

It was to address this predicament that a team at Colorado State University developed a probabilistic forecast system for excessive rainfall. The Colorado State University Machine Learning Probabilities (CSU-MLP) platform uses ensemble reforecasts, precipitation observations, and ML algorithms, specifically random forests (a method that functions by constructing and combining the outputs from a multitude of decision trees – a decision-support tool that uses a tree-like model of decisions and their possible consequences).

CSU-MLP’s forecasts were designed to emulate the Excessive Rainfall Outlook generated and issued by the US Weather Prediction Center forecasters for the contiguous US at lead times of one-to-three days. They serve as a primary information tool for forecasters to use to inform a ‘first guess’ in the rolling ERO process.

“We first demonstrated an ML-based tool for excessive rainfall prediction at the Weather Prediction Center back in 2017,” says Professor Russ S Schumacher, Department of Atmospheric Science at Colorado State University. “Forecasters were excited about its potential, but there were also some deficiencies. Forecasts of heavy rain in the North American Monsoon, for example, were not very good.” (The North American Monsoon is a pattern of increased thunderstorms and rainfall over large areas of the south-western US and north-western Mexico between June and mid-September.)

Schumacher adds: “Over the last five years, we’ve experimented with improvements, gotten feedback from forecasters to see what they think, and made further improvements.”

CSU-MLP has since become the very first AI/ML-powered weather prediction system being used by a national weather service bureau.
The CSU-MLP model’s AI is trained on a very large dataset containing some nine years of detailed historical weather observations over the continental US. The data is combined with meteorological retrospective forecasts, which are model ‘re-forecasts’ created from outcomes of past weather events.

CSU researchers pulled the environmental factors from those model forecasts and associated them with past events of severe weather, such as tornadoes and hail. The result is a model that can run in real-time with current weather events and produce a probability of those types of hazards with a four-to-eight-day lead time, based on present environmental factors, such as temperature and wind.

A key characteristic of weather for ancient forecasters was that many of its patterns recur – often in sequence with one another. One of the earliest known approaches to using repeatable weather phenomena to predict weather outcomes was practised by the ancient Babylonians around 650 BCE, who studied the appearance of cloud patterns as the basis of short-term forecasts.

It’s an approach that has been revisited by researchers led by system solutions experts from Lenovo and the University of Connecticut (UConn). The team is developing an AI-based approach to plotting weather futures that applies 21st-century image extrapolation techniques to radar data from Earth observation satellites, in a modern version of the Babylonians’ fascination with the predictive potential of cloud formation.

‘The practice of weather forecasting is, to a significant extent, one of weather interpretation.’

“The practice of weather forecasting is, to a significant extent, one of weather interpretation,” says Dr Zaphiris Christidis, weather segment leader, Infrastructure Solutions Group at Lenovo. “Human forecasters interpret constantly updated information supplied by the weather bureaus and agencies. They use their forecasting experience and intelligence to reach an informed judgement on what the weather will likely do.

“With UConn, we’re developing a platform that uses AI and neural networks to deliver forecasts but using machine intelligence instead of human acumen.”

‘Nowcasting’ is weather forecasting on a very short-term meteorology period of between two and six hours (definitions vary). Such forecasts rely on methods of extrapolation over time of known weather parameters, rather than mathematics-based models. The Lenovo/UConn AI Nowcasting Project leverages a neural network and draws on spatio-temporal modelling techniques developed to perform unsupervised video anomaly detection for use-cases in surveillance video.

“There are several different ways in which AI can be deployed to aid weather forecasters, including to augment NWP,” says Christidis. “To understand the advantages of our approach, you first have to understand the characteristics of the traditional prediction methodologies.”

NWP models are proficient at foretelling typical weather systems, but they have their limitations and blindsides, adds Christidis. Perhaps foremost of these limitations imposed on ‘time-to-prediction’ is duration required for data processing. Manipulating the vast datasets and crunching the complex calculations necessary to most NWP is time-consuming, even for the most powerful supercomputers used by top meteorological bureaus.

The AI Nowcasting Project’s rationale is predicated in part on shortening time-to-prediction by using AI to come up with basic information based on observable phenomena, rather than the massively scaled data crunching of NWP. What fine detail might be lacking is made up for in its speed of delivery.

“Many use-cases exist where it is most important to know quickly what the weather is going to do in the imminent future – the next three to four hours, say – rather than in detail over a longer timescale,” Christidis points out.

In view of this, the AI Nowcasting Project uses observed reflectivity radar data images – past and present – as its primary source data. Reflectivity radar data images present a high-definition picture of the weather from the energy reflected back to radar receivers. These are the radar images that most often appear in TV weather updates.

The AI Nowcasting Project first analyses a set of reflectivity radar data images from actual past weather events that occurred over a geographical region. These have come from UConn’s extensive database of radar images of weather events. By doing this, the system’s AI component is ‘trained’ in how those weather events evolved and unfolded over set start-to-finish points.

The AI then applies this knowledge to real-time radar images from current weather patterns for the same geographical area, using a convolutional transformer methodology to derive an extrapolative view of how it ‘expects’ the weather will develop.

“Our prototype interprets ongoing weather movements in the context of past specific weather events it has been trained on and posits a ‘view’ on how the current situation will unfold,” explains Christidis. “It is interpreting current weather patterns on the basis of past weather patterns and outputting an informed view – or forecast – of how things will turn out.”

Lenovo and UConn are currently developing a demonstration version of the platform that will extend its portability to other US locations, reports Christidis. “We have had good results that are extremely promising. The system is even capable of learning from its mistakes and making corrective adjustments for future forecasts. At this stage we believe that forecasts based on AI analysis of cloud formations are probably no more or no less accurate than equivalent predictions based on NWP.”

Mainstream meteorology tracks climate dynamics over broad geographical regions, but producing more localised forecasts with accuracy presents greater difficulties. This has been a bugbear for industrial sectors like agriculture, which look for weather projections that can zero-in on areas encompassing hectares rather than hemispheres. Even then, meteorological conditions on a large farm can differ markedly from field to field, hour to hour.

Microclimates are the radiation, air temperature, wind speed and humidity as well as the dynamics of soil temperature and soil moisture (rain, sleet, snow) conditions that exist within a metre or so above and below ground level. A microclimate is the differentiated climate of a small-scale designated geographical area, such as a garden, an area of town or upland, coastal or forested, or agriculture.
Weather variables – humidity, rainfall, temperature, wind – in a microclimate may vary from the conditions that prevail over the wider area within which it is located. Indeed, it is the agglomeration of constituent slightly different microclimates that actually makes up the climate for a given urban or rural territory.

The benefits of microclimate predictions to agriculture, forestry and ecological conservation, have been long recognised. But predicting the weather over reduced areas at ground level has proved a technological challenge that scientists have only just started to address.
DeepMC, from Microsoft Research, is a framework that uses AI to predict the constituent dynamics of microclimates. DeepMC predicts various microclimate parameters using IoT sensors deployed at surface-level locations.

DeepMC uses AI and ML to localise its predictions related to weather and climate in real time. It combines two different sources of data: one from on-site microclimate sensors, and the other from standard local weather forecast data (as supplied via APIs by sources like the National Oceanic and Atmospheric Administration and the National Weather Service). It then uses a fusion mechanism to combine these two signals for further real-time analysis.

“Training an ML model on the dissimilarity between the two data streams is more efficient than training the model on just the microclimate data,” explains Peeyush Kumar, senior research scientist at Microsoft Research. “Most commercial weather forecasts come from sophisticated weather models that capture the underlying relationship between climatic parameters based on the physics of meteorology.”

Kumar adds: “By capturing that relationship – rather than having to use [existing] data to learn that from scratch – we can reduce the required amount of data to train and more efficiently construct the underlying model to capture those sophisticated relationships.”
DeepMC trains its AI to precisely find the ‘error’ between the local weather forecast and the microclimate weather conditions. The system uses historical data on both weather forecasts and local sensor data for training, and predicts each weather parameter, like temperature and wind speed, individually.

“DeepMC is the base model that allows prediction of individual microclimatic parameters. We provide access to the base model to enable higher configurability for the end user,” explains Kumar. “The end user can configure the output of microclimate predictions to suit their requirements within the constraints of data quality and data quantity from their IoT sensors.”

Kumar adds: “As part of a cloud-based offering that will soon be available, we are working to enable the end user to set up a more conventional, holistic microclimate forecast ‘personalised’ to their region of interest, such as a particular field on the farm.”

The Microsoft Research system uses a method called ‘decomposition’ to find both short-term and long-term trends and patterns in weather data. “DeepMC decomposes, or separates, the incoming weather signal into various ‘scales’, which are repeating weather patterns across different time windows,” Kumar explains.

He continues: “For example, if we take temperature microclimate predictions, the temperature signal has daily variations caused by the Earth’s rotation (colder in the night, warmer in the day), seasonal variations (caused by changing weather seasons – warmer in summer, colder in winter), and long-term patterns caused by more longer variations of Earth’s global weather movements, among others. DeepMC uses tools from signal analysis to separate the weather data into various scales automatically.” DeepMC forms part of Microsoft’s FarmBeats initiative that enables data-driven farming (see box below).