Fourier transforms reveal how AI learns complicated physics

The black field of scientific AIs isn’t any match for a 200-year-old technique

Fourier transforms reveal how effectively the deep neural community learns complicated physics.

Considered one of computational physics’ oldest instruments, a 200-year-old mathematical method often known as Fourier evaluation, can reveal essential insights into how a type of synthetic intelligence known as a deep neural community learns. carry out duties involving complicated physics like local weather and turbulence modeling, in accordance with a brand new examine.

The invention by mechanical engineering researchers at Rice College is described in an open-access examine revealed within the journal Nexus PNASa sister publication of Proceedings of the Nationwide Academy of Sciences.

It’s the first rigorous framework to clarify and information the usage of deep neural networks for complicated dynamic methods resembling local weather, mentioned the examine’s corresponding creator, Pedram Hassanzadeh. This might dramatically speed up the usage of deep scientific studying in local weather science and result in far more dependable local weather change projections.

The Rice College researchers educated a type of synthetic intelligence known as a deep studying neural community to acknowledge complicated flows of air or water and predict how the flows will change over time. This visible illustrates the substantial variations within the scale of options the mannequin shows throughout coaching (high) and the options it learns to acknowledge (backside) to make its predictions. Credit score: Picture courtesy of P. Hassanzadeh/Rice College

Within the article, Hassanzadeh, Adam Subel, and Ashesh Chattopadhyay, each former college students, and Yifei Guan, a postdoctoral analysis affiliate, detailed their use of Fourier evaluation to review a deep studying neural community that has been educated to acknowledge the complicated airflows within the ambiance. or water within the ocean and predict how these flows would change over time. Their evaluation not solely revealed what the neural community had realized, however it additionally allowed us to instantly join what the community had realized to the physics of the complicated system it was modeling, Hassanzadeh mentioned.

Deep neural networks are sadly obscure and are sometimes considered black bins, he mentioned. This is without doubt one of the foremost considerations in utilizing deep neural networks in scientific purposes. The opposite is generalizability: these networks can’t work for a system completely different from the one for which they had been educated.

Coaching state-of-the-art deep neural networks requires a considerable amount of information, and the retraining burden, with present strategies, remains to be vital. After coaching and retraining a deep studying community to carry out completely different duties involving complicated physics, the Rice College researchers used Fourier evaluation to match the 40,000 cores from the 2 iterations and located that greater than 99% had been related. This illustration reveals the Fourier spectra of the 4 nuclei that differed probably the most earlier than (left) and after (proper) retraining. The outcomes show the potential of the strategies to determine extra environment friendly recycling pathways that require considerably much less information. Credit score: Picture courtesy of P. Hassanzadeh/Rice College

Hassanzadeh mentioned the analytical framework his workforce presents within the paper opens up the black field, permits us to look inside to grasp what the networks have realized and why, and likewise permits us to narrate this to the physics of the system that has been realized.

Subel, the examine’s lead creator, started the analysis as an undergraduate at Rice and is now a graduate pupil at

” data-gt-translate-attributes=”[{” attribute=””>New York University. He said the framework could be used in combination with techniques for transfer learning to enable generalization and ultimately increase the trustworthiness of scientific deep learning.

While many prior studies had attempted to reveal how deep learning networks learn to make predictions, Hassanzadeh said he, Subel, Guan and Chattopadhyay chose to approach the problem from a different perspective.

Pedram Hassanzadeh. Credit: Rice Universit

The common

He said Fourier analysis, which was first proposed in the 1820s, is a favorite technique of physicists and mathematicians for identifying frequency patterns in space and time.

People who do physics almost always look at data in the Fourier space, he said. It makes physics and math easier.

For example, if someone had a minute-by-minute record of outdoor temperature readings for a one-year period, the information would be a string of 525,600 numbers, a type of data set physicists call a time series. To analyze the time series in Fourier space, a researcher would use trigonometry to transform each number in the series, creating another set of 525,600 numbers that would contain information from the original set but look quite different.

Instead of seeing temperature at every minute, you would see just a few spikes, Subel said. One would be the cosine of 24 hours, which would be the day and night cycle of highs and lows. That signal was there all along in the time series, but Fourier analysis allows you to easily see those types of signals in both time and space.

Based on this method, scientists have developed other tools for time-frequency analysis. For example, low-pass transformations filter out background noise, and high-pass filters do the inverse, allowing one to focus on the background.

Adam Subel. Credit: Rice University

Hassanzadehs team first performed the Fourier transformation on the equation of its fully trained deep-learning model. Each of the models approximately 1 million parameters act like multipliers, applying more or less weight to specific operations in the equation during model calculations. In an untrained model, parameters have random values. These are adjusted and honed during training as the algorithm gradually learns to arrive at predictions that are closer and closer to the known outcomes in training cases. Structurally, the model parameters are grouped in some 40,000 five-by-five matrices, or kernels.

When we took the Fourier transform of the equation, that told us we should look at the Fourier transform of these matrices, Hassanzadeh said. We didnt know that. Nobody has done this part ever before, looked at the Fourier transforms of these matrices and tried to connect them to the physics.

And when we did that, it popped out that what the neural network is learning is a combination of low-pass filters, high-pass filters and Gabor filters, he said.

The beautiful thing about this is, the neural network is not doing any magic, Hassanzadeh said. Its not doing anything crazy. Its actually doing what a physicist or mathematician might have tried to do. Of course, without the power of neural nets, we did not know how to correctly combine these filters. But when we talk to physicists about this work, they love it. Because they are, like, Oh! I know what these things are. This is what the neural network has learned. I see.

Subel said the findings have important implications for scientific deep learning, and even suggest that some things scientists have learned from studying machine learning in other contexts, like classification of static images, may not apply to scientific machine learning.

We found that some of the knowledge and conclusions in the machine learning literature that were obtained from work on commercial and medical applications, for example, do not apply to many critical applications in science and engineering, such as climate change modeling, Subel said. This, on its own, is a major implication.

Reference: Explaining the physics of transfer learning in data-driven turbulence modeling by Adam Subel, Yifei Guan, Ashesh Chattopadhyay and Pedram Hassanzadeh, 23 January 2023, PNAS Nexus.
DOI: 10.1093/pnasnexus/pgad015

Chattopadhyay received his Ph.D. in 2022 and is now a research scientist at the Palo Alto Research Center.

The research was supported by the Office of Naval Research (N00014- 20-1-2722), the National Science Foundation (2005123, 1748958) and the Schmidt Futures program. Computational resources were provided by the National Science Foundation (170020) and the National Center for Atmospheric Research (URIC0004).