STAT 238 - Bayesian Statistics Lab One

More on Example 1 from Lecture¶

In Lectures 1 and 2, we studied the following problem.

Let $\Theta$ denote the binary parameter which represents whether I truly have Covid or not ($\Theta = 1$ when I have Covid and $\Theta = 0$ when I don’t). Let $X$ denote the binary outcome of the Covid test so that $X = 1$ represents the positive test. We need to calculate the probability:

where test data is simply $X = 1$, and the background information refers to things like “I have been strictly quarantining for the past 3 weeks”, “I do not have symptoms such as fever” etc.

We used the probability model (below $B$ stands for background information):

With these probability assignments, we use the Bayes rule to compute (1) as

This probability is not very high even though the test has very good false positive and false negative rates. This is because the prior probability $\P(\Theta = 1 \mid B)$ is very low (0.02). So, even with the positive test result, it is more likely than not that we are covid-free.

On the other hand, if we pose the problem as hypothesis testing:

and calculate the $p$-value as

we get a different result (if we use the standard cutoff 0.05 on the $p$-value). This leads to rejection of the null hypothesis and declaring that I have Covid.

The use of $p$-values has been linked to serious issues such as lack of reproducibility (see for example the paper "The reproducibility of research and the misinterpretation of $p$-values" by David Colquhoun). In this context, we can calculate the probability of reproducibility of the positive test as follows. Let $X_2$ denote the outcome of the second test (and $X_1 = X$ will now denote the outcome of the first test):

To calculate the probabilities in the right hand side above, we make the following assignment:

This assumption means that conditional on my Covid status $\Theta$, the two test outcomes $X_1$ and $X_2$ are independent. Using this assignment, it is straightforward to calculate the reproducibility probability as follows (note that we already calculated $\P(\Theta = 1 \mid X_1 = 1, B) = 1 - \P( \Theta = 0 \mid X_1 = 1, B) = 0.3356$)

Thus the positive test will be reproducible with probability only 35.88% which means that it is more likely to get a negative test the second time.

Law of Total Probability and Bayes Rule¶

In Bayesian statistics, the rules of probability are used mostly for the following:

1. to compute the marginal distribution of $X$ based on knowledge of the conditional distribution of $X$ given $\Theta = \theta$ (i.e., the likelihood) as well as the marginal distribution of $\Theta$ (i.e., the prior)

2. to compute the conditional distribution of $\Theta$ given $X = x$ (i.e., the posterior) based on the same knowledge of the conditional distribution of $X$ given $\Theta = \theta$ (i.e., the likelihood) as well as the marginal distribution of $\Theta$ (i.e., the prior).

The formula for the first item above is sometimes called the Law of Total Probability (LTP), while the formula for the second item is called the Bayes Rule. The precise formulae differ according to whether $X$ and $\Theta$ are discrete or continuous. It is natural to consider the following four separate cases.

$X$ and $\Theta$ are both discrete¶

The LTP is

and the Bayes rule is

$X$ and $\Theta$ are both continuous¶

Here LTP is

and Bayes rule is

$X$ is discrete while $\Theta$ is continuous¶

LTP is

and Bayes rule is

$X$ is continuous while $\Theta$ is discrete¶

LTP is

and Bayes rule is

Next is a simple application for the case where $\Theta$ is discrete and $X$ is continuous.

A Simple Model Selection Application¶

Suppose $\Theta$ has the $Ber(0.5)$ distribution i.e.,

Next assume that $X_1, \dots, X_n$ have the following distributions conditional on $\Theta = \theta$:

and

$f_0$ is the standard normal density and $f_1$ is a Laplace (double-exponential) density. Both densities have the same maximal value of $1/\sqrt{2 \pi}$. Based on the information given, calculate the conditional distribution of $\Theta$ given $X_1 = x_1, X_2 = x_2, \dots, X_6 = x_6$ (i.e., $n = 6$) where

Here is the statistical context for this question. We observe data $x_1, \dots, x_n$ with $n = 6$. We want to use one of the models $f_0$ or $f_1$ for this data. The random variable $\Theta$ is used to describe the choice of the model. We want to treat both the models on an equal footing so we assumed that $\Theta$ has the uniform prior distribution on $\{0, 1\}$.

To calculate the conditional distribution of $\Theta$ given the data, we use the formula (6) because $\Theta$ is discrete and the data $X_1, \dots, X_n$ are continuous. This gives

Similarly

Plugging in the above formula the data values given in (7) for $x_1, \dots, x_6$, we obtain

Thus, conditioning on the data, we have a $72 \%$ probability for the normal model compared to $28\%$ probability for the Laplace model. So we would prefer to use the normal distribution here.

Now suppose that we add in an additional observation $x_7 = 5$. It can be checked that

Now there is overwhelming preference for the Laplace model. This is because $x_7 = 5$ is an outlying observation to which the Laplace model gives much higher probability compared to the Normal model owing to heavy tails of the Laplace density.