Deep Gaussian Processes

Deep GPs as neural networks

A GP \(\mathcal{G}\) is defined by the mean and covariance function \(\mu(\cdot)\) and \(k(\cdot, \cdot')\), respectively. A \(H\)-layer DGP \(f\) is defined by the composition of \(H\) multi-variate GPs as follows:

\[f(\cdot) = \mathcal{G}^{(H)} \circ \cdots \circ \mathcal{G}^{(2)} \circ \mathcal{G}^{(1)}(\cdot),\]

where \(f \circ g\) denotes the composition function \(f(g(\cdot))\). Each layer \(\mathcal{G}^{(i)}\) is a GP with mean function \(\mu^{(i)}(\cdot)\) and kernel function \(k^{(i)}(\cdot, \cdot')\), \(i=1,\ldots,H\). The input of the first layer is the input of the DGP, and the output of the \(i\)-th layer is the input of the \((i+1)\)-th layer. The output of the \(H\)-th layer is the output of the DGP.

We approximate GP \(\mathcal{G}^{(i)}\) by the finite-rank approximation as a one-layer neural network:

\[\begin{split}\begin{align*} \hat{\mathcal{G}}^{(i)}(\cdot) := & \mu + k(\cdot, \mathbf{U}) [ k(\mathbf{U}, \mathbf{U})]^{-1} \mathcal{G}^{(i)}(\mathbf{U}), \\ = & \mu + k(\cdot, \mathbf{U}) R^{-1}_{\mathbf{U}} \mathbf{Z} \\ = & \mu + \phi^{T}(\cdot) \mathbf{Z} \end{align*}\end{split}\]

where \(\mathbf{U}=\{ \mathbf{u}_i \}_{i=1}^{m}\) are the inducing points. \(R_{\mathbf{U}}\) is the Cholesky decomposition of the kernel matrix \(k(\mathbf{U}, \mathbf{U})\), and \(\mathbf{Z} = [R^{T}_{\mathbf{U}}]^-1 \mathcal{G}(\mathbf{U})\) are independently distributed weights. \(\phi(\cdot) = k(\cdot, \mathbf{U}) R^{-1}_{\mathbf{U}}\) is the activation feature map.

Note

This notebook is not necessarily intended to teach the mathematical background of sparse DGPs, but to provide a simple tutorial of how to use Sparse DGP module in machine learning applications. For a mathematical treatment of sparse DGPs, please refer to the original paper: A Sparse Expansion For Deep Gaussian Processes.

Deep GPs with the sparse grid

Sparse grid

Inducing points \(\mathbf{U}^{SG}_l\) are selected on a level-\(l\) sparse grid. A sparse grid is a set of grid points in a d-dimensional input space.

Deep Tensor Markov GP (DTMGP)

Deep Tensor Markov GP (DTMGP) is a DGP with the sparse grid structure and Markov kernel. The hidden layer architecture of DTMGP is as follows:

Deep GPs with the additive structure

Deep Additive Markov GP (DAMGP) is a DGP with the additive structure and Markov kernel. Each layer is a composition of GPs with 1-D sparse grid. The hidden layer architecture of DAMGP is as follows: