Understanding Neural Operators: The Next Evolution in Machine Learning

Have you ever wondered how we could teach computers to understand continuous processes, like weather patterns or fluid dynamics, rather than just working with fixed data points? Enter neural operators - a fascinating breakthrough in machine learning that's changing how we approach complex scientific problems. Unlike the traditional neural networks, neural operators learn mappings between function spaces. In some sense, we can view the traditional neural networks as calculators that can only work with specific numbers you input. Neural operators, on the other hand, are more like mathematical wizards that can understand and work with entire functions - continuous patterns that exist across space and time. Imagine being able to predict weather patterns at any location, not just where weather stations exist!

Key Aspects of Neural Operators

Breaking Free from Resolution Constraints

One of the most exciting aspects of neural operators is their "resolution agnostic" nature. A neural operator, with a fixed set of parameters, can be applied to input functions given at any discretization. This means that as the discretization of input functions is refined, the outputs converge to the true solution, differing only by a discretization error. This property is a significant advantage over standard neural networks, as they do not have guarantees of generalizing to other resolutions, and often perform poorly when interpolated to higher resolutions. 

The Secret Sauce: Integral Operators

Neural operators are built using linear integral operators, followed by non-linear pointwise activations. The linear integral operator involves a learnable kernel that maps between input and output domains. 

o The integral operation is given by: ∫ k(x, y)a(y)dy ≈ ∑ k(x, yi)a(yi)∆yi, where a(·) is the input function, and k(x, y) is a learnable kernel between any two points x and y.

o The query point x in the output domain does not need to be limited to the discrete training grid and can be any point in the continuous domain.

Zero-Shot Super-resolution and Super-evaluation

Due to the discretization convergence property, trained neural operators can perform zero-shot super-resolution, where the output is predicted at a higher resolution than what was seen during training, and zero-shot super-evaluation, where the operator can be evaluated on a new, finer discretization than seen during training. 

Fourier Neural Operator (FNO)

An example of neural operators is the Fourier Neural Operator (FNO). FNO consists of one or multiple Fourier layers that learns and emulates the interactions among the variables of interest in Fourier space. These layers are sandwiched by two linear transformation layers that convert the dimensions between inputs, hidden, and output layers. Fourier layers are analogus to convolution layers in the convolutional neural networks (CNN). However, filtering in convolution neural networks are usually local as shown in the figure below. They are good to capture local patterns such as edges and shapes. The filtering by FNOs are global sinusoidal functions. They are better for representing continuous functions and thus for learning mappings between continuous functions. 

CNN filters versus Fourier filters 
Image courtesy of https://zongyi-li.github.io/blog/2020/fourier-pde/

Physics-Informed Neural Operators (PINOs) 

This is another example of popular neural operators. These operators incorporate physics constraints, such as PDEs, into the training process. This leads to improved generalization, extrapolation capabilities and reduced training data requirements compared to purely data-driven neural operators. Please see my earlier post on Physics-informed Neural Networks.

Neural Operators Applications

Neural operators have demonstrated success in various scientific and engineering applications, such as solving PDEs, fluid dynamics, weather forecasting, climate modeling, and inverse problems. As we push the boundaries of scientific discovery, neural operators represent a significant leap forward in our ability to model and understand complex systems. They're not just another machine learning tool - they're a bridge between discrete digital computing and the continuous nature of our physical world. For further exploration, please visit the neural operators library where you can download jupyter notebooks illustrating the their use.