A radial basis function(RBF) is a real-valued function whose value depends only on the input and its distance from some fixed point (c) or the origin. The function satisfies the criteria below:
The distance is usually the Euclidean distance between two points. The Euclidean distance is calculated as follows:
The sum of radial basis functions is typically used to approximate the given function.
But what does this mean?
||x – xn|| is the radial part of the function, since its value depends on some distance from a fixed center. The basis function is the mathematical representation of the gaussian function. Now, you have points (xn,yn) ∈ D, where D is a dataset. All the points in D affect a hypothesis h(x). This means that all the points in the dataset affect and influence a machine learning model. But in our case, it affects h(x) specifically based on ||x – xn|| , i.e the distance
What do we do with the equation?
We find the wns or weights that minimize the error between h(x) and yn, so we basically try to equate them. This is called interpolation.
This is a representation of the gaussian distribution. You can observe above, an RBF is symmetric around the center if we take the topmost point as the center.
Applications of Radial Basis Function
Radial basis functions serve as a glue for many topics of machine learning. RBFs are often used for classification.
Relationship to nearest neighbor method
In the nearest neighbor(NN) method, you classify points according to their features. But is not going to be a smooth process. You thus consider only K nearest neighbors(KNN) to smoothen the classification boundaries. When we use RBF with K centers, we can assign more weight to the nearest points and less weight to the points that are far away. This makes the classification smoother than NN and KNN.
RBF with K centers
How to choose the centers?
We choose K number of clusters, where K << N. We get K number of centers for K clusters.
Our main objective is to minimize the distance between xn and the closed center uk. We choose a cluster around the center uk and split the data points D into K clusters. We choose points that are nearest to the center. We minimize ||xn-uk||2 of all the points in each cluster where xn are points nearest to the center uk.
Mathematically, we express the equation as follows:
w is the weight we aim to calculate.
How to choose the weights?
We minimize the equation for h(x) and approximately equate it to yn.
We can treat gamma as a learnable parameter and use the EM approach(see wiki) to solve it. i.e
- We fix the value of gamma and solve for wn
- We fix wn, and try to minimize the error by adjusting gamma. We can calculate different gamma for different centers
Relationship to Neural Networks
Radial Basis Function network is an artificial neural network with an input layer, a hidden layer, and an output layer. It is similar to 2-layer networks, but we replace the activation function with a Gaussian radial basis function. RBF centers can be selected randomly from a set of data by using supervised learning, or they can be defined by using k-means clusters using unsupervised learning. We take each input vector and feed it into each basis. Then, we do a simple weighted sum to get our approximated function value at the end. This is the approximation is compared to the training data and then assigned a value of similarity. We train these using backpropagation like any neural network.
Some points about RBF Networks:
- Radial basis function networks are distinguished from other neural networks due to their universal approximation and faster learning speed
- They are used in image classification
- It is used for the non-linear classification of data
You can see the code in python in this Python Machine Learning post.
RBF in Kernels
Radial Basis Function kernels are popular kernels in machine learning for non-linear data. It is the most commonly used kernel in Support Vector Machines(see our post).
 Xanthoula Eirini Pantazi, Dimitrios Moshou, Dionysis Bochtis, Chapter 2 – Artificial intelligence in agriculture, Intelligent Data Mining and Fusion Systems in Agriculture, Academic Press, 2020, Pages 17-101, ISBN 9780128143919, https://doi.org/10.1016/B978-0-12-814391-9.00002-9.