Radial Basis Function

A radial basis function(RBF) is a real-valued function whose value depends only on the input and its distance from some fixed point (c) or the origin. The function satisfies the criteria below:

Function Criteria
Radial Function Criteria

The distance is usually the Euclidean distance between two points. The Euclidean distance is calculated as follows:

Calculation of Euclidean Distance (Source: Wiki)
Calculation of Euclidean Distance

The sum of radial basis functions is typically used to approximate the given function.

Gaussian RBF Formula
Gaussian Radial Basis Function

But what does this mean?

||x – xn|| is the radial part of the function, since its value depends on some distance from a fixed center. The basis function is the mathematical representation of the gaussian function. Now, you have points (xn,yn) D, where D is a dataset. All the points in D affect a hypothesis h(x). This means that all the points in the dataset affect and influence a machine learning model. But in our case, it affects h(x) specifically based on ||x – xn|| , i.e the distance

What do we do with the equation?

We find the wns or weights that minimize the error between h(x) and yn, so we basically try to equate them. This is called interpolation.

Gaussian RBF
Gaussian RBF

This is a representation of the gaussian distribution. You can observe above, an RBF is symmetric around the center if we take the topmost point as the center.

Applications of Radial Basis Function

Radial basis functions serve as a glue for many topics of machine learning. RBFs are often used for classification.

Relationship to nearest neighbor method

In the nearest neighbor(NN) method, you classify points according to their features. But is not going to be a smooth process. You thus consider only K nearest neighbors(KNN) to smoothen the classification boundaries. When we use RBF with K centers, we can assign more weight to the nearest points and less weight to the points that are far away. This makes the classification smoother than NN and KNN.

RBF with K centers

How to choose the centers?

We choose K number of clusters, where K << N. We get K number of centers for K clusters.

K centers
K centers

Our main objective is to minimize the distance between xn and the closed center uk. We choose a cluster around the center uk and split the data points D into K clusters. We choose points that are nearest to the center. We minimize ||xn-uk||2 of all the points in each cluster where xn are points nearest to the center uk.

Mathematically, we express the equation as follows:

RBF with K centers
RBF with K centers

w is the weight we aim to calculate.

How to choose the weights?

We minimize the equation for h(x) and approximately equate it to yn.

We can treat gamma as a learnable parameter and use the EM approach(see wiki) to solve it. i.e

  • We fix the value of gamma and solve for wn
  • We fix wn, and try to minimize the error by adjusting gamma. We can calculate different gamma for different centers
Nearest neighbor vs Nonparametric RBF(smoother version of KNN) (Source: Learning from Data)
Nearest neighbor vs Nonparametric RBF(smoother version of KNN) (Source: Learning from Data)

Relationship to Neural Networks

Radial Basis Function network is an artificial neural network with an input layer, a hidden layer, and an output layer. It is similar to 2-layer networks, but we replace the activation function with a Gaussian radial basis function. RBF centers can be selected randomly from a set of data by using supervised learning, or they can be defined by using k-means clusters using unsupervised learning[1]. We take each input vector and feed it into each basis. Then, we do a simple weighted sum to get our approximated function value at the end. This is the approximation is compared to the training data and then assigned a value of similarity. We train these using backpropagation like any neural network.

RBF network vs Neural Network
RBF network vs Neural Network (Source: Caltech MOOC Course)

Some points about RBF Networks:

  1. Radial basis function networks are distinguished from other neural networks due to their universal approximation and faster learning speed
  2. They are used in image classification 
  3. It is used for the non-linear classification of data

You can see the code in python in this Python Machine Learning post.

RBF in Kernels

Radial Basis Function kernels are popular kernels in machine learning for non-linear data. It is the most commonly used kernel in Support Vector Machines(see our post).

References

[1] Xanthoula Eirini Pantazi, Dimitrios Moshou, Dionysis Bochtis, Chapter 2 – Artificial intelligence in agriculture, Intelligent Data Mining and Fusion Systems in Agriculture, Academic Press, 2020, Pages 17-101, ISBN 9780128143919, https://doi.org/10.1016/B978-0-12-814391-9.00002-9.

Links to Resources


Similar Posts

2 Comments

  1. I’m truⅼy enjoyіng the ɗesign and layoᥙt of youг site.

    It’s a very easy on the eyes wһich makes it much more
    pleasant for me to come here and visit mⲟre often. Did you hire out a designer to create your theme?
    Оսtstanding work!

Leave a Reply

Your email address will not be published.