STAT 238 - Bayesian Statistics Lecture Thirty Six

Spring 2026, UC Berkeley

Variational Inference¶

Variational Inference is a method for approximating a posterior distribution by a simpler tractable family of distributions. It uses optimization to select a distribution $q$ that is closest to the true posterior in some metric. In practice, variational inference can be much faster than MCMC, though sometimes less accurate in representing posterior uncertainty.

The basic idea behind variational inference is very simple. Consider the standard Bayesian setup with data given by $y$ and parameters given by $\theta$. The prior is $f_{\theta}(\theta)$ and the likelihood is $f_{y \mid \theta}(y)$. The posterior is then:

The denominator is the marginal density $f_{y}(y)$ of $y$. The marginal density $f_{y}(y)$is also known as the Evidence. Using the marginal density for the denominator, we get

The posterior is usually intractable because of the integration involved in the computation of the Evidence (denominator). In variational inference, one chooses a simpler family of distributions $\Qcal$ and then employs optimization techniques to choose a distribution in $\Qcal$ that is as close as possible to the posterior $f_{\theta \mid y}(\theta)$. The distance-measure used to measure closeness here is most commonly the following Kullback-Leibler divergence:

Recall that the KL divergence between two densities $q$ and $p$ is given by $\int q \log (q/p)$. It is always nonnegative and it equals zero if and only if $q = p$. It is not symmetric in general (i.e., $KL(q \| p) \neq KL(p \| q)$).

The optimization that we need to solve therefore is:

The choice of this specific KL divergence is mostly for technical convenience as it makes the resulting optimization tractable. Observe that

The second term above (which involves the usually intractable $f_y(y)$) does not depend on $q$ so the optimization (4) is equivalent to:

The above objective function is called the Variational Free Energy, or simply, Free Energy $F(q)$:

So the main goal in Variational Inference is to minimize the Free Energy. The negative of the Free Energy is called the Evidence Lower Bound (ELBO):

So the main goal in Variational Inference can also be said to be the maximization of the ELBO. The name ELBO comes from the fact that $\text{ELBO}(q)$ always satisfies:

This is because of the expression (5) which says that the difference between $\log f_{y}(y)$ and $\text{ELBO}(q)$ equals the Kullback Leibler divergence $KL(q \| f_{\theta \mid y})$ which is nonnegative. Because $f_y(y)$ is called the evidence and $\text{ELBO}(q)$ gives a lower bound for the logarithm of the evidence, it is given the name Evidence Lower Bound. It is also easy to see that if we maximize $\text{ELBO}(q)$ over all probability densities $q$, then we obtain equality in (9):

The above follows from (5) because

Because $f_{y, \theta}(y, \theta) = f_{\theta}(\theta) f_{y \mid \theta}(y)$, the ELBO has the following alternative expression:

In other words, $\text{ELBO}(q)$ represents the average likelihood (with respect to $q$) minus the KL divergence between $q$ and the prior.

Usually, the main tasks in Variational Inference are (a) to choose a class of distributions $\Qcal$, and (b) to maximize $\text{ELBO}(q)$ (or, equivalently, to minimize $F(q)$) over $q \in \Qcal$. We will see how to do this via some examples.

Bayesian Logistic Regression¶

Consider again the logistic regression setting where we observe data $(x_i, y_i), i = 1, \dots, n$ with $x_i \in \R^d$ and $y_i \in \{0, 1\}$. The likelihood model is:

and we take the improper uniform prior for $\beta$. We then have:

where $\ell(\beta)$ is the log-likelihood is

Note that, because the prior is one, $f_{y, \beta}(y, \beta)$ equals the likelihood $f_{y \mid \beta}(y)$. We can thus calculate the ELBO as:

The quantity $-\int q \log q$ is known as the entropy $H(q)$ of $q$. We thus have

Let us now take $\Qcal$ to be the class of all normal distributions $N(m, \Sigma)$ as $m \in \R^p$ and $\Sigma$ varies over the class of all $p \times p$ positive definite matrices $\Sigma$. The entropy of the multivariate normal distribution (see Multivariate normal distribution) is:

where $|\Sigma|$ is the determinant of $\Sigma$. We thus have:

This can be maximized over $m$ and $\Sigma$ to obtain the posterior approximation $N(\hat{m}, \hat{\Sigma})$. Here $m$ can be any vector over $\R^p$ but $\Sigma$ is constrained to be positive definite. Constrained optimization is usually difficult (for gradient-based methods) so we convert this to unconstrained optimization by taking

where $L$ is a $p \times p$ lower-triangular matrix with non-zero diagonal entries. We can also, without loss of generality, assume that $L$ has strictly positive diagonal entries (given any $L$, we can replace it by $L D$ where $D$ is a diagonal matrix with entries +1 if the corresponding entry of $L$ is positive and -1 otherwise). In code, we can represent the diagonal entries of $L$ by $\exp(a_1), \dots, \exp(a_p)$. With this parametrization, we have

and thus

In order to maximize the above function, we need to make the term $\E_{\beta \sim N(m, L L^T)} \ell(\beta)$ more explicit. For this, we use the reparametrization trick (see Reparameterization trick) and write

This gives

The expectation above can be approximated by Monte Carlo. Specifically generate

and use the approximation:

This can be maximized using standard gradient-based optimization software such as PyTorch.