STAT 238 - Bayesian Statistics Lecture Nineteen

Spring 2026, UC Berkeley

A High-Dimensional Linear Regression Model¶

We have a response variable $y$ and a single covariate $x$. In our example, $y$ denotes weekly earnings and $x$ denotes years of experience. The covariate $x$ takes the values $0, 1, \dots, m$ for some fixed integer $m$.

Our data is $(x_i, y_i), i = 1, \dots, n$. The model is:

where $\relu(u) := \max(u, 0)$, and $\epsilon_i \overset{\text{i.i.d}}{\sim} N(0, \sigma^2)$. Note we did not include $\relu(x_i - m)$ because it always equals 0.

This linear regression equation can be written in the usual notation as:

where $y = (y_1, \dots, y_n)^T$ and $X$ is the $n \times (m+1)$ matrix with columns 1, $x$, $\relu(x - j)$ for $j = 1, \dots, m-1$.

Recap: Linear Regression with default priors¶

As we saw before, the default prior on $\beta_0, \dots, \beta_m$ in the linear regression model (2) is:

with $C \rightarrow \infty$. With this prior, the posterior of $\beta$ given $\sigma$ becomes (see e.g., Problem 7(a) in Homework Three)

which means that the posterior mean of $\beta$ is just the least squares estimator $(X^T X)^{-1} X^T y$ (note that this posterior mean of $\beta$ does not depend on $\sigma$, so is both the conditional posterior mean given $\sigma$, as well as the unconditional posterior mean).

The fact that the posterior mean of $\beta$ coincides with the least squares estimator is shared by some priors other than (3) as well. For example, this is also true for the following prior:

as $C \rightarrow \infty$. This can be seen by the following general result (which follows, for example, from Problem 9 in Homework Two):

From here, it is clear that if $Q$ is chosen so that $Q^{-1} \approx 0$, then we get the same posterior as (4). The prior (5) corresponds to $Q = C I$ so that $Q^{-1} = (1/C)I$ which goes to zero when $C \rightarrow \infty$. So the prior (5) leads to the same posterior (4).

Regularization¶

In the earnings-experience regression example, we have seen in the last class that the least squares estimator for (2) is not interpretable. We want the fitted curve

to be smooth for it to be interpretable. This can be achieved in the Bayesian setting if we use the prior:

for some small $\tau$. The above prior is equivalent to $\beta \sim N(0, Q)$ where $Q$ is the $(m+1) \times (m+1)$ diagonal matrix with diagonal entries $C, C, \tau^2, \dots, \tau^2$. So using the formula (6), the posterior mean corresponding to this prior is:

Because $Q$ is diagonal with diagonal entries $C, C, 1/\tau^2, \dots, 1/\tau^2$, it is easy to see that $Q^{-1}$ is also diagonal with diagonal entries $1/C, 1/C, \tau^{-2}, \dots, \tau^{-2}$. Because $C$ is large, we can approximate $Q^{-1}$ by the matrix with diagonal entries $0, 0, \tau^{-2}, \dots, \tau^{-2}$. In other words:

Note that $J$ is an $(m+1) \times (m+1)$ diagonal matrix. Thus the posterior mean becomes:

We use the notation $\tau^2 = \sigma^2 \gamma^2$ or equivalently $\gamma^2 = \tau^2/\sigma^2$. This gives

It can be verified that this quantity $(X^TX + J/\gamma^2)^{-1} X^T y$ coincides with a ridge regularized least squares estimator. This is shown in the next section.

Connection to Ridge Regularization¶

For the model (1), the ridge regression estimator $\ridge$ for $\beta$ is given by the minimizer of:

The objective function above can also be written as

It turns out that $\ridge$ can be written in closed form using matrix notation. To see this, note first that the gradient of the above objective function with respect to $\beta$ is given by

Recalling that $J$ is the $(m+1) \times (m+1)$ diagonal matrix with diagonal entries $0, 0, 1, \dots, 1$, we can write

Setting this gradient equal to zero, we get

which gives

Note that $\lambda = 0$ corresponds to least squares, and $\lambda \rightarrow \infty$ leads to linear regression. When $\lambda$ is set to be neither very close to 0 nor very large, we should get a smooth estimate that still gives a good fit to the data.

Comparing (14) to (12), it is clear that they are identical when $\lambda = 1/\gamma^2$. In other words, when we use the prior (8), then the posterior mean (given $\tau$ and $\sigma$) coincides with ridge regularized least squares with regularization parameter $\lambda = 1/\gamma^2 = \sigma^2/\tau^2$.

Bayesian inference for $\gamma$ and $\sigma$¶

The big advantage of Bayesian inference is that it also allows inference on $\gamma$ and $\sigma$. We shall use the prior:

Recall again that $\tau = \gamma \sigma$. The joint prior on all the parameters $(\beta, \gamma, \sigma)$ is:

Here $Q$ is the $(m+1) \times (m+1)$ diagonal matrix with diagonal entries $C, C, \gamma^2 \sigma^2, \dots, \gamma^2 \sigma^2$.

With this prior, we shall argue in the next lecture that the posterior is given by:

and

and

With this, inference can be carried out by first taking a grid of $\gamma$ values and computing the above posterior (on the logarithmic scale) at the grid points. This posterior can be used to obtain posterior samples of $\gamma$. For each sample of $\gamma$, we can then sample $\sigma$ using the distribution (18). Given samples from both $\gamma$ and $\sigma$, we can then sample $\beta$ using (17).