STAT 238 - Bayesian Statistics Lecture Sixteen

Spring 2026, UC Berkeley

Linear Regression¶

In the last lecture, we discussed Bayesian inference for multiple linear regression. The setting is as follows: we have one response variable $y$ and $m$ covariates $x_1, \dots, x_m$ ($m = 1$ corresponds to simple linear regression). We observe data on $n$ instances or subjects for all these variables: $(y_i, x_{i1}, \dots, x_{im})$ for $i = 1, \dots, n$. The multiple linear regression model (with normal errors) is given by:

Using the matrix vector notation:

the model can be written as:

Below we review standard frequentist and Bayesian approaches to linear regression. Both lead to identical answers.

Frequentist Approach¶

Here the focus is on the least squares estimate which is given by:

To perform inference, the key is to compute the distribution of $\hat{\beta}$. The dependence of the above on the $X$ matrix is quite complicated. On the other hand, the dependence on $y$ is linear. If we assume that $X$ is fixed (non-random), then it is easy to write the distribution of $\hat{\beta}$. Indeed because $y \sim N(X \beta, \sigma^2 I_n)$, it is easy to check that

This implies that

where $(X^T X)^{j+1, j+1}$ is the $(j+1, j+1)$-th diagonal entry of $(X^T X)^{-1}$. The above is equivalent to:

Because $\sigma$ is unknown, it is natural to replace it with a suitable estimate $\sigma$. The standard estimate used is the residual standard error:

where $S(\beta) = \|y - X \beta\|^2$ is the sum of squares and $S(\hat{\beta})$ is the smallest possible value of the sum of squares. Replacing $\sigma$ by $\hat{\sigma}$ in (6) will change the distribution. In fact, it can be shown that $N(0, 1)$ will be replaced by the standard $t$ distribution with $n-m-1$ degrees of freedom:

This can be used to derive confidence intervals for $\beta_j$. If $t_{n-m-1, \alpha/2}$ is the point beyond which the $t$-distribution (with $n-m-1$ degrees of freedom) assigns probability $\alpha/2$, then

which is same as:

This interval:

is the $100(1-\alpha)\%$ confidence interval for $\beta_j$.

The quantity $\hat{\sigma} \sqrt{(X^T X)^{j+1, j+1}}$ is known as the standard error corresponding to $\beta_j$.

Bayesian Approach¶

The default prior for linear regression is:

The likelihood is given by:

To obtain this likelihood, we can assume that $X$ is fixed (and that $\epsilon \sim N(0, \sigma^2 I_d)$). Or we can assume that $\epsilon$ is independent of $X$ and that the distribution of $X$ does not depend on the parameters $\beta, \sigma$. There could be other assumptions on $\epsilon, X$ which can lead to the same (or very similar) likelihood (see Section ??).

Under this prior and likelihood, we calculated (see last lecture notes) that the posterior density of $\beta$ is given by:

Note that, in (14), the quantity $S(\hat{\beta})/(n-m-1)$ is the frequentist unbiased estimator for $\sigma^2$, which is denoted by $\hat{\sigma}^2$. $\hat{\sigma}$ can also be justified as a Bayesian estimator of $\sigma$ (this will be a question in Homework three).

With the notation for $\hat{\sigma}$, the posterior (14) becomes:

By one of the facts mentioned about the multivariate $t$-distribution, the posterior of each individual $\beta_j$ is also $t$:

where $(X^T X)^{j+1, j+1}$ is the $(j+1)^{th}$ diagonal entry of $(X^T X)^{-1}$ (note that we are using the $(j+1)$th diagonal entry of $X^TX$ because $\beta_j$ is the $(j+1)$th component of $\beta$). This implies that

This can be used to obtain uncertainty intervals for $\beta_j$. If $t_{n-m-1, \alpha/2}$ is the point beyond which the $t$-distribution (with $n-m-1$ degrees of freedom) assigns probability $\alpha/2$, then

which is same as:

This interval:

is the $100(1-\alpha)\%$ Bayesian Credible interval for $\beta_j$.

From (8) and (14), it is clear that the Bayesian and frequentist $100(1 - \alpha)\%$ intervals for $\beta_j$ coincide.

When $n$ is large¶

The degrees of freedom corresponding to the $t$-density in (14) is $n - m - 1$ where $n$ is the number of observations, and $m$ is the number of covariates. Thus if $n-m-1$ is large, then the posterior distribution (which is actually $t$) is approximately normal:

In other words, when $n-m-1$ is large, the $t$-density (14) is approximately equal to the $N_{m+1}(\hat{\beta}, \hat{\sigma}^2 (X^T X)^{-1})$. Further, when $n-m-1$ is large, the distribution (16) will be close to the normal distribution $N(\hat{\beta}_j, \hat{\sigma}^2 (X^T X)^{j+1, j+1})$. In such cases, $t_{n-m-1, \alpha/2}$ can be replaced by $z_{\alpha/2}$ in the intervals.

Bayesian vs Frequentist¶

We mentioned above that the Bayesian uncertainty interval for $\beta_j$ exactly coincides with the standard frequentist confidence interval. But the Bayesian reasoning that led to the interval differs fundamentally from the frequentist reasoning. The Bayesian reasoning is based on the fact (13) while frequentist reasoning is based on (7).

The only difference between (7) and (13) is in the conditioning. In (13), the random variable is $\beta_j$ and we are computing its distribution conditional on the observed data. In contrast, in the frequentist statement (7), the random variable is $\hat{\beta}_j$ (which is a function of $y_1, \dots, y_n$) and we are computing its distribution the parameters $\beta, \sigma$ are fixed. Note that $X$ is assumed to be fixed here.

The fact that both Bayesian and frequentist reasoning lead to the same interval is a matter of coincidence. By changing the setting slightly, this coincidence can be broken. One such example involves AR models and is discussed next.

AutoRegression¶

We observe a time series $\{y_t, t = 1, \dots, n\}$. The AutoRegressive Model of order 1 (in short, AR(1)) is given by:

In the above equation, $y_1$ does not appear on the right hand side, so we can treat $y_1$ as a constant. The equation (15) is simply a linear regression model with covariate given by $x_t = y_{t-1}$. Frequentist and Bayesian inference now give different answers (although in practice the answers will be quite similar) and with very different justifications. This is explained below.

Frequentist Inference¶

We start with the least squares estimate of $\hat{\beta}$:

where

Unlike in the previous case, we cannot compute the distribution of $\hat{\beta}$ assuming that $X$ is fixed. This is because $X$ involves depends on $y_1, \dots, y_{n-1}$. Instead, we need to use other arguments. For example, note that

where $\bar{y} = \bar{y}_{2:n} := (y_2 + \dots + y_n)/(n-1)$ and $\bar{x} = \bar{y}_{1:(n-1)} := (y_1 + \dots + y_{n-1})/(n-1)$. The distribution of $\hat{\beta}_1$ is actually quite complicated. Asymptotic arguments can be used to derive a limiting normal distribution. For the full argument, see, for example, ShumwayStoffer which uses a dependent Central Limit Theorem.

Bayesian Inference¶

For Bayesian inference, we need to write the prior and likelihood. The prior will obviously be the same as in usual linear regression:

For the likelihood, we write:

We can now assume that $\epsilon_t$ is independent of $y_1, \dots, y_{t-1}$ for each $t = 2, \dots, n$. This gives

For $f_{y_1 \mid \beta, \sigma}(y_1)$, the simplest thing is to assume that it does not depend on the parameters $\beta, \sigma$ so that this term can be ignored. This assumption leads to

This is exactly the same likelihood as in usual linear regression (see (9)) except that $n$ in (9) is now replaced by $n-1$ which is the number of observations in the regression model (15). Because the likelihood is of the same form as in (9), the posterior of $\beta$ follows the same calculation as in usual linear regression and we obtain:

This is the same as (14) with $n-1$ replacing $n$ and $m = 2$.

Therefore, Bayesian inference gives the same posterior $t$-distribution in AutoRegression as in usual linear regression. Unlike frequentist AutoRegression, there is no need for asymptotics assuming $n \rightarrow \infty$. This example is useful for highlighting the differences between Bayesian and frequentist inference.