Kernel (Covariance) Function Options

In supervised learning, it is expected that the points with similar predictor values $$x_i$$, naturally have close response (target) values $$y_i$$. In Gaussian processes, the covariance function expresses this similarity [1]. It specifies the covariance between the two latent variables $$f\left(x_i\right)$$ and $$f\left(x_j\right)$$, where both $$x_i$$ and $$x_j$$ are d-by-1 vectors. In other words, it determines how the response at one point $$x_i$$ is affected by responses at other points $$x_j$$, i ≠ j, i = 1, 2, ..., n. The covariance function $$k\left(x_i,x_j\right)$$ can be defined by various kernel functions. It can be parameterized in terms of the kernel parameters in vector $$\theta$$. Hence, it is possible to express the covariance function as $$k\left(x_i,x_j|\theta \right)$$.

For many standard kernel functions, the kernel parameters are based on the signal standard deviation $${\sigma }_f$$ and the characteristic length scale $${\sigma }_l$$. The characteristic length scales briefly define how far apart the input values $$x_i$$ can be for the response values to become uncorrelated. Both $${\sigma }_l$$ and $${\sigma }_f$$ need to be greater than 0, and this can be enforced by the unconstrained parametrization vector $$\theta$$, such that

$${\theta }_1=\log {\sigma }_l,{\theta }_2=\log {\sigma }_f.$$

The built-in kernel (covariance) functions with same length scale for each predictor are:

• Squared Exponential Kernel

  This is one of the most commonly used covariance functions and is the default option for fitrgp. The squared exponential kernel function is defined as

  $$k\left(x_i,x_j|\theta \right)={\sigma }_f^2\text{exp}\left[-\frac{1}{2}\frac{{(x_i-x_j)}^T(x_i-x_j)}{{\sigma }_l^2}\right].$$

  where $${\sigma }_l$$ is the characteristic length scale, and $${\sigma }_f$$ is the signal standard deviation.

• Exponential Kernel

  You can specify the exponential kernel function using the 'KernelFunction','exponential' name-value pair argument. This covariance function is defined by

  $$k(x_i,x_j|\theta )={\sigma }_f^2\exp \left(-\frac{r}{{\sigma }_l}\right),$$

  where $${\sigma }_l$$ is the characteristic length scale and

  $$\begin{array}{l}r=\sqrt{{(x_i-x_j)}^T(x_i-x_j)}\hfill \end{array}$$

  is the Euclidean distance between $$x_i$$ and $$x_j$$.

• Matern 3/2

  You can specify the Matern 3/2 kernel function using the 'KernelFunction','matern32' name-value pair argument. This covariance function is defined by

  $$\begin{array}{l}k(x_i,x_j|\theta )={\sigma }_f^2\left(1+\frac{\sqrt{3}r}{{\sigma }_l}\right)\text{exp}\left(-\frac{\sqrt{3}r}{{\sigma }_l}\right)\hfill \end{array},$$

  where

  $$\begin{array}{l}r=\sqrt{{(x_i-x_j)}^T(x_i-x_j)}\hfill \end{array}$$

  is the Euclidean distance between $$x_i$$ and $$x_j$$.

• Matern 5/2

  You can specify the Matern 5/2 kernel function using the 'KernelFunction','matern52' name-value pair argument. The Matern 5/2 covariance function is defined as

  $$\begin{array}{l}k(x_i,x_j)={\sigma }_f^2\left(1+\frac{\sqrt{5}r}{{\sigma }_l}+\frac{5r^2}{3{\sigma }_l^2}\right)\text{exp}\left(-\frac{\sqrt{5}r}{{\sigma }_l}\right)\hfill \end{array},$$

  where

  $$\begin{array}{l}r=\sqrt{{(x_i-x_j)}^T(x_i-x_j)}\hfill \end{array}$$

  is the Euclidean distance between $$x_i$$ and $$x_j$$.

• Rational Quadratic Kernel

  You can specify the rational quadratic kernel function using the 'KernelFunction','rationalquadratic' name-value pair argument. This covariance function is defined by

  $$k(x_i,x_j|\theta )={\sigma }_f^2{\left(1+\frac{r^2}{2\alpha {\sigma }_l^2}\right)}^{-\alpha },$$

  where $${\sigma }_l$$ is the characteristic length scale, $$\alpha$$ is a positive-valued scale-mixture parameter, and

  $$\begin{array}{l}r=\sqrt{{(x_i-x_j)}^T(x_i-x_j)}\hfill \end{array}$$

  is the Euclidean distance between $$x_i$$ and $$x_j$$.

It is possible to use a separate length scale $${\sigma }_m^{}$$ for each predictor m, m = 1, 2, ...,d. The built-in kernel (covariance) functions with a separate length scale for each predictor implement automatic relevance determination (ARD) [2]. The unconstrained parametrization $$\theta$$ in this case is

$$\begin{array}{l}{\theta }_m=\log {\sigma }_m,\text{for}m=1,2,\mathrm{...},d\\ {\theta }_{d+1}=\log {\sigma }_f.\end{array}$$

The built-in kernel (covariance) functions with separate length scale for each predictor are:

• ARD Squared Exponential Kernel

  You can specify this kernel function using the 'KernelFunction','ardsquaredexponential' name-value pair argument. This covariance function is the squared exponential kernel function, with a separate length scale for each predictor. It is defined as

  $$k(x_i,x_j|\theta )={\sigma }_f^2\text{exp}\left[-\frac{1}{2}{\displaystyle \sum _{m=1}^d\frac{{(x_{im}-x_{jm})}^2}{{\sigma }_m^2}}\right].$$

• ARD Exponential Kernel

  You can specify this kernel function using the 'KernelFunction','ardexponential' name-value pair argument. This covariance function is the exponential kernel function, with a separate length scale for each predictor. It is defined as

  $$k(x_i,x_j|\theta )={\sigma }_f^2\exp \left(-r\right),$$

  where

  $$r=\sqrt{{\displaystyle \sum _{m=1}^d\frac{{(x_{im}-x_{jm})}^2}{{\sigma }_m^2}}}.$$

• ARD Matern 3/2

  You can specify this kernel function using the 'KernelFunction','ardmatern32' name-value pair argument. This covariance function is the Matern 3/2 kernel function, with a different length scale for each predictor. It is defined as

  $$k(x_i,x_j|\theta )={\sigma }_f^2\left(1+\sqrt{3}r\right)\text{exp}\left(-\sqrt{3}r\right),$$

  where

  $$r=\sqrt{{\displaystyle \sum _{m=1}^d\frac{{(x_{im}-x_{jm})}^2}{{\sigma }_m^2}}}.$$

• ARD Matern 5/2

  You can specify this kernel function using the 'KernelFunction','ardmatern52' name-value pair argument. This covariance function is the Matern 5/2 kernel function, with a different length scale for each predictor. It is defined as

  $$\begin{array}{l}k(x_i,x_j|\theta )={\sigma }_f^2\left(1+\sqrt{5}r+\frac{5}{3}{r}^2\right)\text{exp}\left(-\sqrt{5}r\right)\hfill \end{array},$$

  where

  $$r=\sqrt{{\displaystyle \sum _{m=1}^d\frac{{(x_{im}-x_{jm})}^2}{{\sigma }_m^2}}}.$$

• ARD Rational Quadratic Kernel

  You can specify this kernel function using the 'KernelFunction','ardrationalquadratic' name-value pair argument. This covariance function is the rational quadratic kernel function, with a separate length scale for each predictor. It is defined as

  $$k(x_i,x_j|\theta )={\sigma }_f^2{\left(1+\frac{1}{2\alpha }{\displaystyle \sum _{m=1}^d\frac{{(x_{im}-x_{jm})}^2}{{\sigma }_m^2}}\right)}^{-\alpha }.$$

You can specify the kernel function using the KernelFunction name-value pair argument in a call to fitrgp. You can either specify one of the built-in kernel parameter options, or specify a custom function. When providing the initial kernel parameter values for a built-in kernel function, input the initial values for signal standard deviation and the characteristic length scale(s) as a numeric vector. When providing the initial kernel parameter values for a custom kernel function, input the initial values the unconstrained parametrization vector $$\theta$$. fitrgp uses analytical derivatives to estimate parameters when using a built-in kernel function, whereas when using a custom kernel function it uses numerical derivatives.

References

[1] Rasmussen, C. E. and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press. Cambridge, Massachusetts, 2006.

[2] Neal, R. M. Bayesian Learning for Neural Networks. Springer, New York. Lecture Notes in Statistics, 118, 1996.