Demo 4: Multivariate Normal Distributions

If you have problems displaying the page, try zooming out (Ctrl+- or Cmd+-) and refreshing the page (F5 or Cmd+R). Remember that you can ask questions or leave a comment on the last text slide.

The following demo will introduce what a multivariate normal distribution is, and what the parameters of the distribution mean. To the right is an figure you can interact with to change the underlying distribution; you can tap (select mean or direction) or double-tap (deselect). The numerical values of parameters are shown on the slide "Parameters". You can get a guided tour by navigating these text slides (press the right-left buttons, click the entries underneath, or swipe).

Mean of the distribution

The plot shows 100 samples drawn from a multivariate normal distribution. The samples are shown as black dots. One parameter that defines the distribution is the mean, visualized as a blue dot.

Try tapping the mean (blue dot) and move the mean of the distribution. New samples will be draw based on the new mean. Try moving the mean around slightly in the same place to see different samples from (roughly) the same distribution. Deselect the mean to stop moving it by double tapping.

Directions

Another parameter of the distribution is the covariance matrix. The covariance matrix is controlled by changing two directions/vectors, visualized as the red and green line.

Try selecting one of the directions by tapping it. You can deselect the line by double tapping. Try moving the vector around. You'll notice that the direction and length of the vector changes the sample draw.

The length of the vectors in red and green define the axes of two filled ellipses in purple (one at 1 times the length and one at 2 times the length). The lengths of the vectors are shown at the end of the vector. Before playing further with the plot, have a look at the quick recap of the underlying math on the next slide.

The multivariate normal distribution

The multivariate normal distribution is introduced in the chapter on densities and models (Chapter 6 on page 91, start of chapter). An example of a probability density function for a multivariate normal distribution is shown on Figure 6.5 on page 97. In general, we consider a point $\boldsymbol{x}\in\mathbb{R}^d$ of $d$ attributes (superscript $T$ denotes transpose):

$\boldsymbol{x} = \begin{bmatrix}x_1 & x_2 & \cdots & x_d\end{bmatrix}^T$

The normal distribution density function is defined as:

$\boldsymbol{\mathcal N}\left(\boldsymbol{x} | \boldsymbol{\mu}, \boldsymbol{\Sigma}\right) = \left( \sqrt{ (2\pi)^d |\boldsymbol{\Sigma}|} \right)^{-1}\mathrm{exp}\left(-\frac{1}{2}(\boldsymbol{x}- \boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1} (\boldsymbol{x}-\boldsymbol{\mu})\right), $

Here $\boldsymbol{\Sigma}$ is known as the covariance matrix which must be symmetric and positive definite and $|\boldsymbol{\Sigma}|$ is the determinant of $\boldsymbol{\Sigma}$ and $\boldsymbol{ \mu} $ is known as the mean of the multivariate normal distribution.

Eigenvalue decomposition of the covariance matrix

The covariance matrix can be thought of it in terms of its eigendecomposition (or spectral decomposition):

$\boldsymbol{\Sigma} = \boldsymbol{Q} \boldsymbol{\Lambda} \boldsymbol{Q}^{-1}$

where $\boldsymbol{Q}$ is a square $d \times d$ matrix whose columns are the eigenvectors of the covariance matrix, and $\boldsymbol{\Lambda}$ is a diagonal matrix with diagonal elements corresponding to the eigenvalues of the covariance matrix. With this decomposition, the eigenvectors define the directions that the multivariate normal distribution "varies in", and the eigenvalues determines the degree of variation in that direction. Go the next slide to see what this means in terms of the plot.

Parameters

For the plot, we're looking at $d=2$, and so $\boldsymbol{x} = \begin{bmatrix}x_1 & x_2\end{bmatrix}^T$. By changing the blue dot, and red and green arrows, you're changing the parameters of the distribution. The mean $\boldsymbol{\mu}$ (the blue dot), and the covariance matrix ($\boldsymbol{\Sigma}$) is shown below the plot.
The covariance matrix is based on your choice of the length and direction of the red and green vectors - you're changing the eigenvalue decomposition of $\boldsymbol{\Sigma}$. The length of the vector changes the root of the eigenvalue (shown in $\boldsymbol{\Lambda}$), and the direction changes the eigenvectors ($\boldsymbol{Q}$). Notice that the $\boldsymbol{\Lambda}$ contains the lengths of the vectors squared; the length of the vectors define the standard deviation in the given direction, and so the squared value is the variance in the direction. The ellipses then define one standard deviation and two standard deviations away from the mean of the distribution.

Changing the covariance matrix

Try setting the mean in the origin (tap the mean and move to the origin, then deselect by double tap) and make one of the vectors point directly up and have length approximately 4. Can you identify which column in $\boldsymbol{\Lambda}$ and $\boldsymbol{Q}$ corresponds to that vector? How did it change $\boldsymbol{\Sigma}$, and lastly, what does this mean in terms of the samples draw (the black dots)?

The diagonal elements in $\boldsymbol{\Sigma}$ signifies variation for the corresponding attribute. Similarly, the off-diagonals determine the amount of covariance between the two attributes.

Try to change the vectors so the distribution has negative covariance between the two attributes. Similarly, try to change the vectors so $\boldsymbol{\Sigma}$ shows positive covariance.