Post by Benfxmth on Nov 15, 2023 7:35:14 GMT -5
This is going to be a gamechanger for weather forecasting within the next few years or so. A new AI-based weather model developed by Google (DeepMind), and a few others - evidently - are already outperforming the traditional deterministic weather models and more quickly.
www.science.org/content/article/ai-churns-out-lightning-fast-forecasts-good-weather-agencies
Experiemtally, it was also shown to predict a NS landfall by Hurricane Lee three days earlier than all other models anticipated: www.technologyreview.com/2023/11/14/1083366/google-deepminds-weather-ai-can-forecast-extreme-weather-quicker-and-more-accurately/
Meteorologists call it the “quiet revolution”: a gradual but steady improvement in weather forecasting. Today, the 6-day forecast is about as good as the 3-day forecast from 30 years ago. Rarely do severe storms or heat waves catch people unaware. This revolution has saved lives and money, but it also comes with a cost: billions of dollars’ worth of energy-hungry supercomputers that must run 24/7 just to produce a few forecasts a day.
Artificial intelligence (AI) is now spurring another revolution within numerical weather prediction, as the field is known. In mere minutes on cheap desktop computers, trained AI systems can now make 10-day forecasts that are as good as the best traditional models—and in some cases even better. The world’s top weather agency, the European Centre for Medium-Range Weather Forecasts (ECMWF), has embraced the technology: Last month it began to generate its own experimental AI forecasts. The algorithms could enable more frequent forecasts and free up computing resources for other thorny problems. “It’s very, very exciting to know we can generate global predictions that are skillful, really cheaply,” says Maria Molina, an AI-focused research meteorologist at the University of Maryland.
Some of the world’s biggest tech giants are jockeying to claim the most skillful model, including Google DeepMind, which describes its GraphCast model in Science this week, and Huawei, which published a similar model, called Pangu-Weather, in Nature earlier this year. Google also has a short-term AI weather model that makes rolling 24-hour predictions that are more accurate than nearly any weather agency’s. It’s incredible progress on a task that was thought infeasible just a few years ago, says Aditya Grover, an AI researcher at the University of California, Los Angeles. “From a technology standpoint, we have all the ingredients in place.”
Traditional weather models start by feeding a snapshot of current conditions, based on observations from satellites, weather stations, and buoys, into a gridlike computer model that divides the atmosphere into millions of boxes. The snapshot is run forward in time by applying the physical laws of fluid dynamics to each box—at great computational expense. The models can take several hours to run on supercomputers with 1 million processors, and weather agencies typically produce updates just four times a day.
The new AI models skip the expense of solving equations in favor of “deep learning.” They identify patterns in the way the atmosphere naturally evolves, after training on 40 years of ECMWF “reanalysis” data—a combination of observations and short-term model forecasts that represents modelers’ best and most complete picture of past weather. When fed a starting snapshot of the atmosphere based on the same combination of observations and modeling, GraphCast can outperform the ECMWF forecast out to 10 days on 90% of its verification targets, including hurricane tracks and extreme temperatures. Although it took 32 computers 4 weeks to train the AI model, the resulting algorithm is lightweight enough to work in less than 1 minute on a single desktop computer, says Rémi Lam, lead author of the GraphCast paper. “It is fast, accurate, and useful.”
These benefits seem to hold even in more realistic settings. Earlier this year, ECMWF researchers ran Pangu, feeding it only the observations that go into its operational weather model. Those observations offer a more limited picture of the atmosphere than the reanalysis snapshots used to test GraphCast. The skill of Pangu’s forecast was similar to ECMWF’s main model, although its predictions of rainfall and other fine-scale features were slightly fuzzier. “It was an even playing field,” says Zied Ben Bouallègue, who led the analysis, released as an arXiv preprint in July. “We were surprised to see the good results.”
These advances came startlingly fast. A key step came in 2020, when a group led by Stephan Rasp, now also at Google, created WeatherBench, which made the ECMWF reanalysis data easy to digest and also, to provoke competition, provided a benchmark for measuring forecast skill. In 2022, after a few months of work during a sabbatical, Ryan Keisler, a physicist now at KoBold Metals, a mineral exploration company, published a preprint describing a simple model with considerable skill in 6-day forecasts. “Given how much historical data there was to learn from, it just had to work at some level,” Keisler says.
A next step will be to produce ensemble results, a forecasting innovation that helps capture uncertainty by running a model multiple times to create a range of possible outcomes. AI researchers could follow the traditional technique of tweaking initial weather conditions just slightly before each model run, or they could adapt the AI generative techniques making waves in text and image generation to create tweaked conditions on the fly. “I’m pretty sure every group is working on that,” Rasp says. Such ensemble forecasts could help the AI models better predict extreme events, such as strong hurricanes, that they currently underestimate in intensity.
To improve further, the AI models could be weaned off the reanalysis data, which carry the biases of traditional models. Instead, they could learn directly from the petabytes of raw observation data held by weather agencies, Keisler says. Google’s short-term weather model already does so, training itself on data from weather stations, radar, and satellites.
The potential for these models doesn’t stop at weather prediction, says Christopher Bretherton, an atmospheric scientist at the Allen Institute for AI. They cannot project climate on their own, because the 40-year training data sets are not long enough to capture global warming trends, which are subject to complex feedbacks from clouds, gases, and aerosols that can accelerate or slow climate change. But they could assist a new generation of high-resolution climate models being developed to run on exascale computers, the latest ultrafast machines. Once those models produce enough output for the AIs to be trained on, the AIs could take over. “We can make emulators of these models and then run them 100 times faster,” Bretherton says.
Few expect traditional forecasts to disappear anytime soon, but AI is “rapidly approaching the point where it could be a useful complement,” says Matthew Chantry, who coordinates ECMWF’s AI work. Adoption might be slowed by unease about the black-box nature of the AI: Researchers often can’t say how such systems reach their conclusions. But that concern can be overstated, says Chantry, who notes that traditional models are also so complicated that “there’s a degree of opaqueness already built into them.”
Ultimately, it will come down to users, Grover says. “If you’re a farmer in the field, would you care about the more accurate forecast, or the one you can write down with physical equations?”
Artificial intelligence (AI) is now spurring another revolution within numerical weather prediction, as the field is known. In mere minutes on cheap desktop computers, trained AI systems can now make 10-day forecasts that are as good as the best traditional models—and in some cases even better. The world’s top weather agency, the European Centre for Medium-Range Weather Forecasts (ECMWF), has embraced the technology: Last month it began to generate its own experimental AI forecasts. The algorithms could enable more frequent forecasts and free up computing resources for other thorny problems. “It’s very, very exciting to know we can generate global predictions that are skillful, really cheaply,” says Maria Molina, an AI-focused research meteorologist at the University of Maryland.
Some of the world’s biggest tech giants are jockeying to claim the most skillful model, including Google DeepMind, which describes its GraphCast model in Science this week, and Huawei, which published a similar model, called Pangu-Weather, in Nature earlier this year. Google also has a short-term AI weather model that makes rolling 24-hour predictions that are more accurate than nearly any weather agency’s. It’s incredible progress on a task that was thought infeasible just a few years ago, says Aditya Grover, an AI researcher at the University of California, Los Angeles. “From a technology standpoint, we have all the ingredients in place.”
Traditional weather models start by feeding a snapshot of current conditions, based on observations from satellites, weather stations, and buoys, into a gridlike computer model that divides the atmosphere into millions of boxes. The snapshot is run forward in time by applying the physical laws of fluid dynamics to each box—at great computational expense. The models can take several hours to run on supercomputers with 1 million processors, and weather agencies typically produce updates just four times a day.
The new AI models skip the expense of solving equations in favor of “deep learning.” They identify patterns in the way the atmosphere naturally evolves, after training on 40 years of ECMWF “reanalysis” data—a combination of observations and short-term model forecasts that represents modelers’ best and most complete picture of past weather. When fed a starting snapshot of the atmosphere based on the same combination of observations and modeling, GraphCast can outperform the ECMWF forecast out to 10 days on 90% of its verification targets, including hurricane tracks and extreme temperatures. Although it took 32 computers 4 weeks to train the AI model, the resulting algorithm is lightweight enough to work in less than 1 minute on a single desktop computer, says Rémi Lam, lead author of the GraphCast paper. “It is fast, accurate, and useful.”
These benefits seem to hold even in more realistic settings. Earlier this year, ECMWF researchers ran Pangu, feeding it only the observations that go into its operational weather model. Those observations offer a more limited picture of the atmosphere than the reanalysis snapshots used to test GraphCast. The skill of Pangu’s forecast was similar to ECMWF’s main model, although its predictions of rainfall and other fine-scale features were slightly fuzzier. “It was an even playing field,” says Zied Ben Bouallègue, who led the analysis, released as an arXiv preprint in July. “We were surprised to see the good results.”
These advances came startlingly fast. A key step came in 2020, when a group led by Stephan Rasp, now also at Google, created WeatherBench, which made the ECMWF reanalysis data easy to digest and also, to provoke competition, provided a benchmark for measuring forecast skill. In 2022, after a few months of work during a sabbatical, Ryan Keisler, a physicist now at KoBold Metals, a mineral exploration company, published a preprint describing a simple model with considerable skill in 6-day forecasts. “Given how much historical data there was to learn from, it just had to work at some level,” Keisler says.
A next step will be to produce ensemble results, a forecasting innovation that helps capture uncertainty by running a model multiple times to create a range of possible outcomes. AI researchers could follow the traditional technique of tweaking initial weather conditions just slightly before each model run, or they could adapt the AI generative techniques making waves in text and image generation to create tweaked conditions on the fly. “I’m pretty sure every group is working on that,” Rasp says. Such ensemble forecasts could help the AI models better predict extreme events, such as strong hurricanes, that they currently underestimate in intensity.
To improve further, the AI models could be weaned off the reanalysis data, which carry the biases of traditional models. Instead, they could learn directly from the petabytes of raw observation data held by weather agencies, Keisler says. Google’s short-term weather model already does so, training itself on data from weather stations, radar, and satellites.
The potential for these models doesn’t stop at weather prediction, says Christopher Bretherton, an atmospheric scientist at the Allen Institute for AI. They cannot project climate on their own, because the 40-year training data sets are not long enough to capture global warming trends, which are subject to complex feedbacks from clouds, gases, and aerosols that can accelerate or slow climate change. But they could assist a new generation of high-resolution climate models being developed to run on exascale computers, the latest ultrafast machines. Once those models produce enough output for the AIs to be trained on, the AIs could take over. “We can make emulators of these models and then run them 100 times faster,” Bretherton says.
Few expect traditional forecasts to disappear anytime soon, but AI is “rapidly approaching the point where it could be a useful complement,” says Matthew Chantry, who coordinates ECMWF’s AI work. Adoption might be slowed by unease about the black-box nature of the AI: Researchers often can’t say how such systems reach their conclusions. But that concern can be overstated, says Chantry, who notes that traditional models are also so complicated that “there’s a degree of opaqueness already built into them.”
Ultimately, it will come down to users, Grover says. “If you’re a farmer in the field, would you care about the more accurate forecast, or the one you can write down with physical equations?”
www.science.org/content/article/ai-churns-out-lightning-fast-forecasts-good-weather-agencies
Experiemtally, it was also shown to predict a NS landfall by Hurricane Lee three days earlier than all other models anticipated: www.technologyreview.com/2023/11/14/1083366/google-deepminds-weather-ai-can-forecast-extreme-weather-quicker-and-more-accurately/
