STAT 238 - Bayesian Statistics Lecture Twenty Nine

Spring 2026, UC Berkeley

Gibbs Sampler for Mixture Models¶

Known variances and proportion¶

We observe real-valued data $y_1, \dots, y_n$ and consider the model:

with $w$ known and fixed (e.g., $w = 0.3$ or 0.7 as in the simulations) and $\mu_0, \mu_1$ being unknown. We will describe the Gibbs sampler algorithm in this setting first, and then generalize it to the case where $w$ is unknown and we have variance parameters $\sigma_0^2$ and $\sigma_1^2$.

The log-likelihood corresponding to (1) is:

where $\phi(x, \mu, \sigma^2)$ is the normal density with mean $\mu$, variance $\sigma^2$ evaluated at $x$.

We will take a standard prior for $\mu_0, \mu_1$ e.g., $\mu_0, \mu_1 \overset{\text{i.i.d}}{\sim} N(0, C)$ for a large $C$. The posterior of $\mu_0, \mu_1$ is then:

This posterior cannot be evaluated in closed form and numerical methods need to be used. A standard approach is to use the Gibbs sampler with augmentation. First observe that the model (1) can be rewritten in the following way:

It should be clear that, under the above model, the marginal distribution of $y_i$ coincides with (1). $z_1, \dots, z_n$ can be thought of as unobserved latent variables which represent which of the two populations (corresponding to the distributions $N(\mu_0, 1)$ and $N(\mu_1, 1)$ respectively) the observation $y_i$ comes from.

Gibbs sampler is implemented for jointly sampling from the posterior of $\mu_0, \mu_1, z_1, \dots, z_n$ given the data. This requires being able to sample from the full conditionals

where $y = (y_1, \dots, y_n)$ is the data and $z = (z_1, \dots, z_n)$. It is easy to see that these full conditionals can be written in closed form as follows. Given $\mu_0, \mu_1, y_1, \dots, y_n$, the variables $z_1, \dots, z_n$ are independent with

On the other hand, given $z_1, \dots, z_n, y_1, \dots, y_n$, the variables $\mu_0, \mu_1$ are independent with

where $n_0$ is the number of $z_i$’s that are equal to 0, and $n_1$ is the number of $z_i$’s that are equal to 1. Note that in the limit as $C \rightarrow \infty$, we can write

Based on these full conditional distributions, the Gibbs sampler algorithm takes the following form:

1. Initialize $\mu_0^{(0)}, \mu_1^{(0)}$.

2. Repeat the following for $t = 0, 1, 2, \dots$:

   1. Generate $z_1^{(t)}, \dots, z_n^{(t)}$ via:

      $$\begin{align*} z_i^{(t)} \sim \text{Bernoulli} \left(\frac{w \phi(y_i, \mu^{(t)}_1, 1)}{w \phi(y_i, \mu^{(t)}_1, 1) + (1 - w) \phi(y_i, \mu^{(t)}_0, 1)}\right). \end{align*}$$

      (9)

   2. Generate $\mu_0^{(t+1)}, \mu_1^{(t+1)}$ via:

      $$\begin{align*} & \mu^{(t+1)}_1 \sim N \left(\frac{\sum_{i=1}^n y_i z^{(t)}_i}{\sum_{i=1}^n z^{(t)}_i}, \frac{1}{\sum_{i=1}^n z^{(t)}_i} \right) \\ & \mu^{(t+1)}_0 \sim N \left(\frac{\sum_{i=1}^n y_i (1 - z^{(t)}_i)}{\sum_{i=1}^n (1 -z^{(t)}_i)}, \frac{1}{\sum_{i=1}^n (1 - z^{(t)}_i)} \right). \end{align*}$$

      (10)

As we saw in the simulations in the last couple of lectures, initialization is very important. The log-likelihood can have multiple modes only one of which is the correct mode (in the sense of having large likelihood). If the Gibbs sampler is not properly initialized, the algorithm would sample from a spurious peak.

Unknown variances and proportion¶

Now consider the model:

The unknown parameters are now $w, \mu_0, \mu_1, \sigma^2_0, \sigma_1^2$. We place the following independent priors on these variables:

The proportion parameter $w$ is constrained to lie in $[0, 1]$ so it is natural to take the Beta prior for it. A standard choice here is $a = b = 1$ (which corresponds to the uniform prior). For $\mu_0, \mu_1$, we are using the $N(m, s^2)$ prior. Standard choice is $m = 0$ and $s$ very large. The Inverse Gamma prior $IG(\alpha, \beta)$ corresponds to the density:

The standard uninformative prior for $\sigma$ is $\propto \sigma^{-1} I\{\sigma > 0\}$. The corresponding prior density for $\sigma^2$ is also $\propto x^{-1} I\{x > 0\}$. This is a special case of $IG(\alpha, \beta)$ corresponding to $\alpha = \beta = 0$.

The prior (12) is therefore uses conjugate families while including the standard uninformative choices as special cases. The reason for conjugacy is that the full conditional distributions corresponding to the posterior distribution can be written in closed form. We shall look at the formulae in the next lecture.