Preliminaries

Definition.

  • A parameter space, denoted by Θ\Theta, is the set of all possible parameter values. Each element θ\theta of Θ\Theta is called a parameter, which is a numerical value representing population characteristics.
  • A sample space, denoted by Y\Y, is a set of all possible datasets. Each element yy of Y\Y is a possible dataset which describes the population.

In the Bayesian view, the parameter θ\theta is nothing but a modeled random variable that reflects our uncertainty, contrary to the frequentist approach, which considers the parameter as a fixed but unknown value.

Definition.

  • For each θ∈Θ\theta \in \Theta, the prior distribution p(θ)p(\theta) describes our belief that θ\theta represents the true population characteristics.
  • For a given θ∈Θ\theta \in \Theta, the sampling model p(y∣θ)p(y\mid\theta) describes the variability of each y∈Yy\in \mathcal{Y}.
  • For a given y∈Yy \in \mathcal{Y}, the posterior distribution p(θ∣y)p(\theta\mid y) describes our belief about each θ∈Θ\theta \in \Theta.

Preliminaries

Given θ∈Θ\theta \in \Theta, the sampling model p(y∣θ)p(y\mid\theta) evaluates the suitability of each y∈Yy \in \Y. By Bayes’ Rule, we observe that the posterior distribution p(θ∣y)p(\theta\mid y) can be expressed by the prior distribution and the sampling model.

p(θ∣y)=p(y∣θ)p(θ)p(y)=p(y∣θ)p(θ)∫Θp(y∣θ)p(θ)dθ.p(\theta\mid y) = \frac{p(y \mid\theta)p(\theta)}{p(y)} = \frac{p(y \mid\theta)p(\theta)}{\int_\Theta p(y\mid\theta)p(\theta) d\theta}.

The posterior distribution p(θ∣y)p(\theta \mid y) reflects our updated belief for a given y∈Yy \in \Y.

Beliefs

Definition.

A belief function Be()\Be() is a function that satisfies the following belief axioms.

Indeed a probability function satisfies the belief axioms, so once we use a probability function Pr⁡()\Pr(), it ensures the belief axioms.

Conjugacy

Definition.

  • A conjugate family for a sampling model p(y∣θ)p(y\mid\theta) is a collection \P of prior distributions satisfying
p(θ)∈P  ⟹  p(θ∣y)∈Pp(\theta) \in \Pt\implies p(\theta \mid y) \in \Pt
  • A conjugate prior for a sampling model p(y∣θ)p(y\mid\theta) is a prior such that the posterior p(θ∣y)p(\theta \mid y) is in the same probability distribution family as the prior.
  • A semiconjugate prior for a sampling model p(y∣θ)p(y\mid\theta) where θ=(θ1,⋯ ,θm)\theta = (\theta_1, \cdots, \theta_m) is a prior so that there exist closed-form full conditional distributions
p(θ1∣θ2,θ3⋯ ,θm,y),p(θ2∣θ1,θ3,⋯ ,θm,y),⋯ ,p(θm∣θ1,θ2⋯ ,θm−1,y).p(\theta_1 \mid \theta_2, \theta_3 \cdots, \theta_m, y), p(\theta_2 \mid \theta_1, \theta_3, \cdots, \theta_m, y), \cdots, p(\theta_m \mid \theta_1, \theta_2 \cdots, \theta_{m-1}, y).

Monte Carlo Methods

Monte Carlo Approximation

Introduction

If we have a conjugacy so that we could obtain a closed form of the posterior, then we could try the followings.

  • Direct Calculation of the Posterior
  • Using Monte Carlo Method

If it is hard to derive the posterior then we are not able to sample directly. So the followings are available approaches.

  • Gibbs Sampler
  • Markov Chain Monte Carlo

Posterior Inference

Monte Carlo Approximation

Assume we have a closed form of posterior distribution, then the posterior inference using the Monte Carlo approximation is performed by the following steps.

  • Obtaining finite random values of parameter
θ(1),θ(2),⋯ ,θ(S)∼i.i.d.p(θ∣y1,y2,⋯ ,yn)\theta^{(1)}, \theta^{(2)}, \cdots, \theta^{(S)} \stackrel{\text{i.i.d.}}{\sim} p(\theta \mid y_1, y_2, \cdots, y_n)
  • Approximating integration by summation as
1S∑k=1Sg(θ(k))≈E[g(θ)∣y1,y2,⋯ ,yn]=∫Θg(θ)p(θ∣y1,y2,⋯ ,yn)dθ\frac{1}{S} \sum_{k=1}^{S} g(\theta^{(k)}) \approx \E [g(\theta) \mid y_1, y_2, \cdots, y_n] = \int_\Theta g(\theta) p(\theta \mid y_1, y_2, \cdots, y_n) d\theta

Theorem.

Assume θ(1),θ(2),⋯ ,θ(S)∼i.i.d.p(θ∣y1,y2,⋯ ,yn)\theta^{(1)}, \theta^{(2)}, \cdots, \theta^{(S)} \stackrel{\text{i.i.d.}}{\sim} p(\theta \mid y_1, y_2, \cdots, y_n) and E[∣g(θ)∣]<∞\E[\mid g(\theta)\mid] < \infty then

1S∑k=1Sg(θ(k))→a.s.E[g(θ)∣y1,y2,⋯ ,yn]=∫Θg(θ)p(θ∣y1,y2,⋯ ,yn)dθ.\frac{1}{S} \sum_{k=1}^{S} g(\theta^{(k)}) \xrightarrow{\text{a.s.}} \E [g(\theta) \mid y_1, y_2, \cdots, y_n] = \int_\Theta g(\theta) p(\theta \mid y_1, y_2, \cdots, y_n) d \theta.

Proof. Since g(θ(1)),g(θ(2)),⋯ ,g(θ(S))∼i.i.d.p(g(θ)∣y1,y2,⋯ ,yn)g(\theta^{(1)}), g(\theta^{(2)}), \cdots, g(\theta^{(S)}) \stackrel{\text{i.i.d.}}{\sim} p(g(\theta) \mid y_1, y_2, \cdots, y_n), we have

1S∑k=1Sg(θ(k))→a.s.E[g(θ)∣y1,y2,⋯ ,yn]\frac{1}{S} \sum_{k=1}^{S} g(\theta^{(k)}) \xrightarrow{\text{a.s.}} \E [g(\theta) \mid y_1, y_2, \cdots, y_n]

by the strong law of large numbers.

Corollary.

  • θˉ=∑k=1Sθ(k)→PE[θ∣y1,y2,⋯ ,yn]\bar{\theta} = \sum_{k=1}^S \theta^{(k)} \xrightarrow{P} \E [\theta \mid y_1, y_2, \cdots, y_n]
  • ∑k=1S(θ(k)−θˉ)2/(S−1)→PVar(θ∣y1,y2,⋯ ,yn)\sum_{k=1}^S \left(\theta^{(k)} - \bar{\theta}\right)^2 / (S-1) \xrightarrow{P} \V (\theta \mid y_1, y_2, \cdots, y_n)

Markov Chain Monte Carlo

Markov Chain

Introduction

Now assume that our prior is p(θ)p(\theta) with θ={θ1,⋯ ,θm}\theta = \{\theta_1, \cdots, \theta_m\} and we do not have a closed-form posterior, which implies we cannot just sample from

θ∼p(θ∣y)\theta \sim p(\theta \mid y)

as we have done before.

Markov Chain

Definition.

Let XnX_n be a discrete-time stochastic process with state space Θ\Theta.

  • XnX_n is a Markov chain if for all integers n≥0n \geq 0 and states i0,i1,⋯ ,in−1,i,ji_0, i_1, \cdots, i_{n-1}, i, j,
Pr⁡(Xn+1=j∣Xn=i,Xn−1=in−1,⋯X0=i0)=Pr⁡(Xn+1=j∣Xn=i).\Pr(X_{n+1} = j \mid X_n = i, X_{n-1} = i_{n-1}, \cdots X_0 = i_0) = \Pr(X_{n+1}= j \mid X_n = i).
  • A Markov chain XnX_n is homogeneous if Pr⁡(Xn+1=j∣Xn=i)\Pr(X_{n+1}= j \mid X_n = i) is independent of nn.

Definition.

Let XnX_n be a homogeneous Markov chain with state space Θ\Theta.

  • We denote νn\nu_n a probability distribution on Θ\Theta at the time nn and K:Θ×B(Θ)→RK : \Theta \times \B(\Theta) \to \R the transition kernel of XnX_n.
  • If Θ\Theta is at most countable, the transition kernel KK is a matrix K={pij}i,j∈ΘK = \{p_{ij}\}_{i, j \in \Theta} such that
pij=Pr⁡(Xn+1=j∣Xn=i).p_{ij} = \Pr(X_{n+1} = j \mid X_n = i).

Otherwise, for all A⊂ΘA \subset \Theta

K(x,A)=Pr⁡(Xn+1∈A∣Xn=x)=∫Ap(Xn+1=y∣Xn=x)dy.K(x, A) = \Pr(X_{n+1} \in A \mid X_n = x) = \int_A p(X_{n+1}= y \mid X_n = x) dy.
  • We have
Xn∼νnandνn=ν0KnX_n \sim \nu_n \quad \text{and} \quad \nu_n = \nu_0 K^n

for all n≥0n \geq 0 where νK(A)=∫ΘK(x,A)ν(x)dx.\nu K (A) = \int_{\Theta} K (x, A) \nu (x) dx.

Definition.

Let XnX_n be a homogeneous Markov chain with state space Θ\Theta and the transition kernel KK.

  • A stationary distribution of XnX_n, denoted by π\pi is a probability distribution satisfying
π=πK.\pi = \pi K.
  • XnX_n is irreducible if, for any initial state, it has positive probability of entering any set to which assigns positive probability, that is, for all A⊂ΘA \subset \Theta with π(A)>0\pi(A) > 0 and for all y∈Θy \in \Theta, there exists a positive integer nn such that
Kn(y,A)>0.K^n (y, A) > 0.
  • XnX_n is positively recurrent if E(τ(y))<∞E(\tau(y)) < \infty for all y∈Θy \in \Theta where τ\tau denotes the number of steps required to return from yy to yy.

Ergodic Markov Chain

Theorem.

Let XnX_n be a homogeneous Markov chain with state space Θ\Theta and the transition kernel KK. Assume XnX_n satisfies the following.

  • irreducibility
  • positive recurrence

Then there exists a unique stationary distribution π\pi and

1S∑k=1Sg(Xk)→a.s.Eπ[g].\frac{1}{S}\sum_{k=1}^{S} g(X_k) \xrightarrow{\text{a.s.}} \E_\pi[g].

Note that we do not require aperidiocity of XnX_n since our claim is about almost sure convergence.

  • X→a.s.XnX \xrightarrow{\text{a.s.}} X_n if P(lim⁡n→∞Xn=X)=1.P(\lim_{n \to \infty} X_n = X) = 1.
  • X→a.s.Xn  ⟹  X→pXnX \xrightarrow{\text{a.s.}} X_n \implies X \xrightarrow{p} X_n
  • X→pXn  ⟹  X→dXnX \xrightarrow{p} X_n \implies X \xrightarrow{d} X_n

Theorem (Birkhoff).

Let (Ω,F,P)(\Omega, \F, P) be a probability space and assume φ:Ω→Ω\varphi : \Omega \to \Omega be a measure-preserving transformation on (Ω,F,P)(\Omega, \F, P), that is,

P(φ−1A)=P(A)P(\varphi^{-1} A) = P(A)

for all A∈FA \in \F. For any random variable X:Ω→RX: \Omega \to \R such that X∈L1X \in L^1,

1n∑k=0n−1X(φmω)→a.s.E(X∣I)\frac{1}{n}\sum_{k=0}^{n-1} X(\varphi^m \omega) \xrightarrow{\text{a.s.}} \E(X \mid \mathcal{I})

in L1L^1 where I={A∈F∣φ−1A=A}.\mathcal{I} = \{ A \in \F \mid \varphi^{-1} A = A \}.

Markov Chain Monte Carlo

Markov Chain Monte Carlo

Posterior inference using the Markov Chain Monte Carlo method is performed by the following steps.

  • Obtaining a finite sample of parameter values using a Markov chain.
  • Approximating integration by summation as
1S∑k=1Sg(θ(k))≈E[g(θ)∣y1,y2,⋯ ,yn]=∫Θg(θ)p(θ∣y1,y2,⋯ ,yn)dθ\frac{1}{S} \sum_{k=1}^{S} g(\theta^{(k)}) \approx \E [g(\theta) \mid y_1, y_2, \cdots, y_n] = \int_\Theta g(\theta) p(\theta \mid y_1, y_2, \cdots, y_n) d\theta

Note that Step 2 is identical to the Monte Carlo method.

MCMC Methods

Gibbs Sampling

Motivation

Now assume that our prior is p(θ)p(\theta) with θ={θ1,⋯ ,θm}\theta = \{\theta_1, \cdots, \theta_m\} and we only know the full conditional distributions. Our question is that could we sample from the joint distribution of {θ1,⋯ ,θm}\{ \theta_1, \cdots, \theta_m \} exploiting the full conditional distributions

p(θ1∣θ2,θ3⋯ ,θm,y),p(θ2∣θ1,θ3,⋯ ,θm,y),⋯ ,p(θm∣θ1,θ2⋯ ,θm−1,y).p(\theta_1 \mid \theta_2, \theta_3 \cdots, \theta_m, y), \quad p(\theta_2 \mid \theta_1, \theta_3, \cdots, \theta_m, y), \quad \cdots, p(\theta_m \mid \theta_1, \theta_2 \cdots, \theta_{m-1}, y).

This is the very motivation of the method called Gibbs sampling.

  • First suppose we were given θ(0)=(θ1(0),θ2(0),⋯ ,θm(0))\theta^{(0)} = (\theta_1^{(0)}, \theta_2^{(0)}, \cdots, \theta_m^{(0)}). Then we could sample
θ1(1)∼p(θ1∣θ2(0),⋯ ,θm(0),y).\theta_1^{(1)} \sim p(\theta_1 \mid \theta_2^{(0)}, \cdots, \theta_m^{(0)}, y).
  • From previous result, we could update the value of θ2\theta_2 by sampling
θ2(1)∼p(θ2∣θ1(1),θ3(0),⋯ ,θm(0),y).\theta_2^{(1)} \sim p(\theta_2 \mid {\theta_1^{(1)}}, \theta_3^{(0)}, \cdots, \theta_m^{(0)}, y).
  • Sample
θ3(1)∼p(θm∣θ1(1),θ2(1),θ3(0),⋯ ,θm(0),y1,⋯ ,yn).\theta_3^{(1)} \sim p(\theta_m \mid {\theta_1^{(1)}, \theta_2^{(1)},} \theta_3^{(0)}, \cdots, \theta_m^{(0)}, y_1, \cdots, y_n). ⋯\cdots θm(1)∼p(θ3∣θ1(1),⋯ ,θm−1(1),y).\theta_m^{(1)} \sim p(\theta_3 \mid {\theta_1^{(1)}, \cdots, \theta_{m-1}^{(1)}}, y).

Gibbs Sampling

The algorithm of the Gibbs sampler is as follows.

Definition.

Given a current state of the parameters θ(s)=(θ1(s),⋯ ,θm(s))\theta^{(s)} = (\theta_1^{(s)}, \cdots, \theta_m^{(s)}),

  • Sample
θi(s+1)∼p(θi∣θ1(s+1),⋯ ,θi−1(s+1),θi+1(s),⋯ ,θm(s),y)\theta_i^{(s+1)} \sim p(\theta_i \mid \theta_1^{(s+1)}, \cdots, \theta_{i-1}^{(s+1)}, \theta_{i+1}^{(s)}, \cdots, \theta_m^{(s)}, y)

sequentially for i=1,2,⋯ ,mi = 1, 2, \cdots, m.

  • Let θ(s+1)=(θ1(s+1),⋯ ,θm(s+1))\theta^{(s+1)} = (\theta_1^{(s+1)}, \cdots, \theta_m^{(s+1)})

The Gibbs sampler generates a dependent sequence of parameters {θ(1),θ(2),⋯ }.\{ \theta^{(1)}, \theta^{(2)}, \cdots\}.

Metropolis Algorithm

Motivation

Our question is how could we approximate posterior when neither the posterior nor the full conditionals have closed forms. Assume that our prior is p(θ)p(\theta) with θ=(θ1,⋯ ,θm)\theta = (\theta_1, \cdots, \theta_m).

Metropolis Algorithm

The Metropolis algorithm is as follows.

Definition.

Given a current state θ(s)\theta^{(s)},

  • Sample θ∗∼J(θ∣θ(s))\theta^* \sim J(\theta \mid \theta^{(s)}) as a candidate of θ(s+1)\theta^{(s+1)}.
  • Let r=p(θ∗∣y)p(θ(s)∣y)r = \frac{p(\theta^* \mid y)}{p(\theta^{(s)} \mid y)}
  • Set
θ(s+1)={θ∗with probability min⁡(r,1)θ(s)otherwise\theta^{(s+1)} = \begin{cases} \theta^* \quad \text{with probability }\min(r, 1)\\ \theta^{(s)} \quad \text{otherwise} \end{cases}
  • Here the proposal distribution JJ must be symmetric, that is, J(α∣β)=J(β∣α)J(\alpha \mid \beta) = J(\beta \mid \alpha) for all α,β∈Θ\alpha, \beta \in \Theta.

Metropolis Algorithm

  • J(θ∣θ(s))J(\theta \mid \theta^{(s)}) suggests θ∗\theta^* in the neighborhood of the current state θ(s)\theta^{(s)}. Our typical choices of JJ are the followings.

  • J(θ∗∣θ(s))=Uniform(θ(s)−δ,θ(s)+δ)J(\theta^* \mid \theta^{(s)}) = \Unif(\theta^{(s)} - \delta, \theta^{(s)} + \delta)
  • J(θ∗∣θ(s))=Normal(θ(s),δ2)J(\theta^* \mid \theta^{(s)}) = \Nm(\theta^{(s)}, \delta^2)

Metropolis–Hastings

Metropolis–Hastings Algorithm

The Metropolis–Hastings algorithm is as follows.

Definition.

Given a current state θ(s)\theta^{(s)},

  • Sample θ∗∼J(θ∣θ(s))\theta^* \sim J(\theta \mid \theta^{(s)}) as a candidate of θ(s+1)\theta^{(s+1)}.
  • Let r=p(θ∗∣y)p(θ(s)∣y)×J(θ(s)∣θ∗)J(θ∗∣θ(s))r = \frac{p(\theta^* \mid y)}{p(\theta^{(s)} \mid y)} \times \frac{J(\theta^{(s)} \mid \theta^*)}{J(\theta^* \mid \theta^{(s)})}
  • Set
θ(s+1)={θ∗with probability min⁡(r,1)θ(s)otherwise\theta^{(s+1)} = \begin{cases} \theta^* \quad \text{with probability }\min (r, 1)\\ \theta^{(s)} \quad \text{otherwise} \end{cases}

Note that the algorithm above is identical to the Metropolis algorithm whenever JJ is symmetric.

How It Works

Since the Gibbs sampler and Metropolis algorithm are special cases of the Metropolis–Hastings algorithm, we need only consider the case of Metropolis–Hastings.

Theorem.

Let {θ(1),θ(2),⋯ ,θ(S)}\{\theta^{(1)}, \theta^{(2)}, \cdots, \theta^{(S)}\} be a sequence of parameters generated by \cref{MH} and denote by π\pi the stationary distribution of the Markov chain θ(n)\theta^{(n)} with its state space Θ\Theta. Assume θ(n)\theta^{(n)} satisfies the conditions of \cref{Ergodic}, then

π(θ)=p(θ∣y)\pi(\theta) = p(\theta \mid y)

so we have

1S∑k=1Sg(θ(k))→a.s.E[g(θ)].\frac{1}{S}\sum_{k=1}^{S} g(\theta^{(k)}) \xrightarrow{\text{a.s.}} \E[g(\theta)].

Proof. Since we have the uniqueness of the stationary distribution, it suffices to show that the posterior p0=p(θ∣y)p_0 = p(\theta \mid y) is stationary itself, that is, p0K=p0.p_0 K = p_0.

To obtain the transition kernel K(x,A)K(x, A), recall that we have a proposal J(θ∣x)J(\theta \mid x), r=p0(θ)p0(x)×J(x∣θ)J(θ∣x)r = \frac{p_0(\theta)}{p_0(x)} \times \frac{J(x \mid \theta)}{J(\theta \mid x)}, and acceptance rate α(x,θ)=min⁡{1,r}\alpha(x, \theta) = \min \{1, r\}. So we have

K(x,A)=Pr⁡(Xn+1∈A∣Xn=x)=∫AJ(θ∣x)α(x,θ)dθ⏟acceptance+(1−∫ΘJ(θ∣x)α(x,θ)dθ)1A(x)⏟rejectance.K(x, A) = \Pr(X_{n+1} \in A \mid X_n = x)\\ = \underbrace{\int_A J(\theta \mid x) \alpha (x, \theta) d\theta}_{\text{acceptance}} + \underbrace{\left(1-\int_\Theta J(\theta \mid x) \alpha(x, \theta) d\theta \right) \1_A(x)}_{\text{rejectance}}.

That is, we indeed sample θ(s+1)\theta^{(s+1)} as

θ(s+1)∼K(θ(s),⋅).\theta^{(s+1)} \sim K(\theta^{(s)}, \cdot).

Since we have the following reversibility condition (detailed balance),

p0(x)J(θ∣x)α(x,θ)=p0(θ)J(x∣θ)α(θ,x)p_0(x) J(\theta \mid x) \alpha(x, \theta) = p_0(\theta) J(x \mid \theta) \alpha(\theta, x)

we readily check

p0K(A)=∫Θp0(x)K(x,A)dx=∫Θp0(x)∫AJ(θ∣x)α(x,θ)dθdx+∫Ap0(x)dx−∫Ap0(x)∫ΘJ(θ∣x)α(x,θ)dθdx=∫Ap0(x)dxp_0 K (A) = \int_\Theta p_0(x) K(x, A) dx\\ = \int_\Theta p_0(x) \int_A J(\theta \mid x) \alpha(x, \theta) d\theta dx + \int_A p_0(x) dx - \int_A p_0(x) \int_\Theta J(\theta \mid x) \alpha(x, \theta) d\theta dx\\ = \int_A p_0(x) dx

for all set A⊂ΘA \subset \Theta.

MCMC Diagnostics

Introduction

Suppose we want to approximate the integral

E[θ]=∫Θθp(θ)dθ=θ0\E[\theta] = \int_\Theta \theta p(\theta) d\theta = \theta_0

using the empirical distribution of {θ(1),⋯ ,θ(S)}\{\theta^{(1)} , \cdots, \theta^{(S)}\}.

Diagnosing Convergence

As we have checked before, the convergence

1S∑k=1Sg(θ(k))→a.s.E[g(θ)]\frac{1}{S}\sum_{k=1}^{S} g(\theta^{(k)}) \xrightarrow{\text{a.s.}} \E[g(\theta)]

is straightforward. In this section, we now turn our focus to the speed of convergence.

  • Stationarity: whether the chain reached its equibrilium distribution
  • Speed of mixing: how quickly θ(k)\theta^{(k)} explores around the parameter space as k→∞k \to \infty.

Lag-$$

t $$ Autocorrelation Function

Definition.

The lag-tt autocorrelation function

acf⁡t(θ)=1S−t∑k=1S−t(θk−θˉ)(θk+t−θˉ)1S−1∑k=1S(θk−θˉ)2\operatorname{acf}_t(\theta) = \frac{\frac{1}{S-t} \sum_{k=1}^{S-t}(\theta_k - \bar{\theta})(\theta_{k+t}-\bar{\theta})}{\frac{1}{S-1}\sum_{k=1}^{S}(\theta_k - \bar{\theta})^2}

estimates the correlation between elements of the sequence that are tt steps apart.

Burn-In

  • Discarding early iterations of the simulation runs
  • Convention is to discard the first half of the sequence.

Thining

  • Discarding all but kthk^{th} simulation draws
  • Reducing autocorrelation
  • There are some assertions that thining is often inefficient and not necessary. Read \cite{onthining} for the details.

Bayesian Models

The Normal Model

Bayesian Normal Model

Bayesian normal model is a method for inferring two parameters θ,σ2\theta, \sigma^2 assuming that observations

y1,y2,⋯ ,yn∼i.i.d.Normal(θ,σ2).y_1, y_2, \cdots, y_n \stackrel{\text{i.i.d.}}{\sim} \Nm(\theta, \sigma^2).

At this point, the prior distribution is p(θ,σ2)p(\theta, \sigma^2) and our goal is to consider the joint posterior

p(θ,σ2∣y)=p(y∣θ,σ2)p(θ,σ2)p(y)p(\theta, \sigma^2 \mid y) = \frac{p(y \mid \theta, \sigma^2) p(\theta, \sigma^2)}{p(y)}

and denote y=(y1,y2,⋯ ,yn)y = (y_1, y_2, \cdots, y_n). Then the following joint sampling density is

p(y,σ2)=∏i=1np(yi∣θ,σ2)=(2πσ2)−n2exp⁡{−12∑i=1n(yi−θσ)2}.p(y, \sigma^2) = \prod_{i=1}^n p(y_i \mid \theta, \sigma^2) = (2\pi\sigma^2)^{-\frac{n}{2}} \exp \left \{ -\frac{1}{2} \sum_{i=1}^n \left( \frac{y_i - \theta}{\sigma} \right)^2 \right \}.

Define yˉ=∑i=1nyin\bar{y} = \sum_{i=1}^n \frac{y_i}{n} and s2=∑i=1n(yi−yˉ)2n−1s^2 = \sum_{i=1}^n \frac{(y_i - \bar{y})^2 }{n-1}. Now we are going to show that the pair {yˉ,s2}\{ \bar{y}, s^2 \} is a two-dimensional sufficient statistic for {θ,σ2}\{ \theta, \sigma^2 \} by exploiting the following theorem.

Theorem (factorization theorem).

For a statistic T(x)T(x) to be sufficient for the parameter θ\theta, the likelihood p(x∣θ)p(x \mid \theta) should be factorized as the form of

p(x∣θ)=g(T(x),θ)×h(x).p(x \mid \theta) = g(T(x), \theta) \times h(x).

Note that it suffices to consider only the following part of our joint density.

∑i=1n(yi−θ)2=∑i=1nyi2−2θ∑i=1nyi+nθ2=(n−1)σ2+n(yˉ−θ)\sum_{i=1}^n \left(y_i - \theta \right)^2 = \sum_{i=1}^n y_i^2 - 2\theta \sum_{i=1}^n y_i + n\theta^2 = (n-1) \sigma^2 + n(\bar{y} - \theta)

The remaining part of the proof is so obvious that we left it as an exercise.

Group Comparisions and Hierarchial Modeling

Comparing Two Groups

Motivation

In the sense of frequentist, we compared two groups using pp-value, which is the method called hypothesis testing. The following is the procedure of performing such model selection.

  • Setting null hypothesis H0H_0 and alternative hypothesis H1H_1.
  • Selecting the significance level α\alpha.
  • If p<αp < \alpha, then reject H0H_0. \qquad Otherwise, accept H0H_0.

Indeed it is a neat method to compare two groups. But this frequentist method has the limitation that it treats the two populations as completely distinct or exactly identical.

Comparing Two Groups by a Bayesian Analysis

Consider the following sampling model for data from the two groups.

Xi,1=μ+δ+εi,1Yi,2=μ−δ+εi,2εi,j∼i.i.dNormal(0,σ2).X_{i, 1} = \mu + \delta + \varepsilon_{i, 1} \quad Y_{i, 2} = \mu - \delta + \varepsilon_{i, 2} \quad \varepsilon_{i, j} \stackrel{\text{i.i.d}}{\sim} \Nm (0, \sigma^2).

Using this parametrization where θ1=μ+δ\theta_1 = \mu + \delta and θ2=μ−δ\theta_2 = \mu - \delta, convenient conjugate prior distributions for the unknown parameters are

p(μ,δ,σ2)=p(μ)×p(δ)×p(σ2)p(\mu, \delta, \sigma^2) = p(\mu) \times p(\delta) \times p(\sigma^2)

where

μ∼Normal(μ0,γ02)δ∼Normal(δ0,τ02)σ2∼Inverse−Gamma(ν02,ν0σ022).\mu \sim \Nm(\mu_0, \gamma_0^2) \quad \delta \sim \Nm (\delta_0, \tau_0^2) \quad \sigma^2 \sim \Igam\left(\frac{\nu_0}{2}, \frac{\nu_0\sigma^2_0}{2}\right).

From our construction we obtain the following full conditional distributions.

μ∣y1,y2,δ,σ2∼Normal(μn,γn2)\mu \mid y_1, y_2, \delta, \sigma^2 \sim \Nm \left(\mu_n, \gamma_n^2\right)

where

$ \mu_n=\gamma_n^2 \times\left[\mu_0 / \gamma_0^2+\sum_{i=1}^{n_1}\left(y_{i, 1}-\delta\right) / \sigma^2+\sum_{i=1}^{n_2}\left(y_{i, 2}+\delta\right) / \sigma^2\right], \, \gamma_n^2=\left[1 / \gamma_0^2+\left(n_1+n_2\right) / \sigma^2\right]^{-1}. $

δ∣y1,y2,μ,σ2∼Normal(δn,τn2)\delta \mid y_1, y_2, \mu, \sigma^2 \sim \Nm \left(\delta_n, \tau_n^2\right)

where

$ \delta_n=\tau_n^2 \times\left[\delta_0 / \tau_0^2+\sum\left(y_{i, 1}-\mu\right) / \sigma^2-\sum\left(y_{i, 2}-\mu\right) / \sigma^2\right], \, \tau_n^2=\left[1 / \tau_0^2+\left(n_1+n_2\right) / \sigma^2\right]^{-1}. $

σ2∣y1,y2,μ,δ∼Inverse−Gamma(νn2,νnσn22)\sigma^2 \mid y_1, y_2 , \mu, \delta \sim \Igam\left(\frac{\nu_n}{2}, \frac{\nu_n\sigma^2_n}{2}\right)

where

$ \nu_n=\nu_0+n_1+n_2, \, \nu_n \sigma_n^2=\nu_0 \sigma_0^2+\sum\left(y_{i, 1}-[\mu+\delta]\right)^2+\sum\left(y_{i, 2}-[\mu-\delta]\right)^2. $

Comparing Multiple Groups

Hierarchical Models

Let us consider a model describing our information about a hierachical data

{Y1,⋯ ,Yn}whereYj={Y1,j,⋯ ,Ynj,j}.\{Y_1, \cdots, Y_n\} \quad \text{where} \quad Y_j = \{ Y_{1, j}, \cdots, Y_{n_j, j} \}.

Assume Y1,j,⋯ ,Ynj,jY_{1, j}, \cdots, Y_{n_j, j} are random samples from group jj, then it is reasonable to treat Y1,j,⋯ ,Ynj,jY_{1, j}, \cdots, Y_{n_j, j} as exchangeable. Further assume that group jj is large compared to the sample size njn_j, then de Finetti’s theorem implies that we can model the data within group jj as

Y1,j,⋯ ,Ynj,j∣θj∼i.i.d.p(y∣θj).Y_{1, j}, \cdots, Y_{n_j, j} \mid \theta_j \stackrel{\text{i.i.d.}}{\sim} p(y \mid \theta_j).

Moreover, suppose the groups themselves are samples from larger population of groups. Then the exchangeability of the group-specific parameters θj\theta_j can be justified.

Thus applying de Finetti’s theorem once again we obtain

y1,j,⋯ ,ynj,j∣θj∼i.i.d.p(y∣θj)(within-group sampling variability) θ1,⋯ ,θm∣ψ∼i.i.d.p(θ∣ψ)(between-group sampling variability)ψ∼p(ψ)(prior distribution).y_{1, j}, \cdots, y_{n_j, j} \mid \theta_j \stackrel{\text{i.i.d.}}{\sim} p\left(y \mid \theta_j\right) \quad \text{(within-group sampling variability) } \\ \theta_1, \cdots, \theta_m \mid \psi \stackrel{\text{i.i.d.}}{\sim} p(\theta \mid \psi) \quad \text{(between-group sampling variability)} \\ \psi \sim p(\psi) \text{(prior distribution)}.

Here we notice that p(y∣θ)p(y\mid\theta) represents variability within a group and p(θ∣ψ)p(\theta\mid\psi) represents variability across groups contrary to p(ψ)p(\psi) which represents unknown quantity.

Homoscedastic Case

The Hierarchical Normal Model: Homoscedastic Case

Here we introduce the hierarchical normal model which is a popular model for describing the heterogeneity of means across several populations. First assume that the within-group sampling variability σ2\sigma^2 is constant across groups.

Definition (the hierarchical normal model with the homoscedastic assumption).

θj={θj,σ2},p(y∣θj)=Normal(θj,σ2)(within-group model)ψ={μ,τ2},p(θj∣ψ)=Normal(μ,τ2)(between-group model)\theta_j = \{ \theta_j, \sigma^2 \}, \quad p(y\mid\theta_j) = \Nm (\theta_j, \sigma^2 ) \quad \text{(within-group model)}\\ \psi = \{ \mu, \tau^2 \}, \quad p(\theta_j\mid\psi) = \Nm (\mu, \tau^2) \quad \text{(between-group model)}

Posterior Inference

The unknown quantities in our system are the followings.

  • group-specfic means: θ1,⋯ ,θm\theta_1, \cdots, \theta_m
  • within-group sampling variability: σ2\sigma^2
  • the mean and variance of the population of group-specific means: (μ,τ2)(\mu, \tau^2)

Starting from the following relations, we would like to find the full conditional distribution of each variable so that we could apply Gibbs sampling.

p(θ1,⋯ ,θm,μ,τ2,σ2∣y1,⋯ ,ym)∝p(μ,τ2,σ2)×p(θ1,⋯ ,θm∣μ,τ2,σ2)×p(y1,⋯ ,ym∣θ1,⋯ ,θm,μ,τ2,σ2)=p(μ)p(τ2)p(σ2){∏j=1mp(θj∣μ,τ2)}{∏j=1m∏i=1njp(yi,j∣θj,σ2)}p(\theta_1, \cdots, \theta_m, \mu, \tau^2, \sigma^2 \mid y_1, \cdots, y_m) \\ \propto p(\mu, \tau^2, \sigma^2) \times p(\theta_1, \cdots, \theta_m \mid \mu, \tau^2, \sigma^2) \times p(y_1, \cdots, y_m \mid \theta_1, \cdots, \theta_m, \mu, \tau^2, \sigma^2) \\ =p(\mu) p(\tau^2) p(\sigma^2)\left\{\prod_{j=1}^m p(\theta_j \mid \mu, \tau^2)\right\}\left\{\prod_{j=1}^m \prod_{i=1}^{n_j} p(y_{i, j} \mid \theta_j, \sigma^2)\right\}

We obtain

μ∣θ1,⋯ ,θm,τ2∼Normal(mθˉ/τ2+μ0/γ02m/τ2+1/γ02,[m/τ2+1/γ02]−1)\mu \mid \theta_1, \cdots, \theta_m, \tau^2 \sim \Nm \left(\frac{m \bar{\theta} / \tau^2+\mu_0 / \gamma_0^2}{m / \tau^2+1 / \gamma_0^2},\left[m / \tau^2+1 / \gamma_0^2\right]^{-1}\right) 1/τ2∣θ1,⋯ ,θm,μ∼Gamma(η0+m2,η0τ02+∑(θj−μ)22)1 / \tau^2 \mid \theta_1, \cdots, \theta_m, \mu \sim \Gam \left(\frac{\eta_0+m}{2}, \frac{\eta_0 \tau_0^2+\sum\left(\theta_j-\mu\right)^2}{2}\right) θj∣y1,j,⋯ ,ynj,j,σ2∼Normal(njyˉj/σ2+1/τ2nj/σ2+1/τ2,[nj/σ2+1/τ2]−1)\theta_j \mid y_{1, j}, \cdots, y_{n_j, j}, \sigma^2 \sim \Nm \left(\frac{n_j \bar{y}_j / \sigma^2+1 / \tau^2}{n_j / \sigma^2+1 / \tau^2},\left[n_j / \sigma^2+1 / \tau^2\right]^{-1}\right) 1/σ2∣θ,y1,⋯ ,yn∼Gamma(12[ν0+∑j=1mnj],12[ν0σ02+∑j=1m∑i=1nj(yi,j−θj)2]).1 / \sigma^2 \mid \theta, y_1, \cdots, y_n \sim \Gam \left(\frac{1}{2}\left[\nu_0+\sum_{j=1}^m n_j\right], \frac{1}{2}\left[\nu_0 \sigma_0^2+\sum_{j=1}^m \sum_{i=1}^{n_j}\left(y_{i, j}-\theta_j\right)^2\right]\right).

Linear Regression

Normal Linear Regression

Normal Linear Regression

yi=β⊤xi+εiy_i = \beta^\top x_i + \varepsilon_i

where εi∼i.i.d.Normal(0,σ2)\varepsilon_i \stackrel{\text{i.i.d.}}{\sim} \Nm (0, \sigma^2) therefore yi∼i.i.d.Normal(β⊤xi,σ2)y_i \stackrel{\text{i.i.d.}}{\sim} \Nm (\beta^\top x_i, \sigma^2).

Y∣X,β,σ2=[y1y2⋮yn]∼Normal(Xβ,σ2I)whereXβ=[x1x2⋮xn][β1β2⋮βn]=[β⊤x1β⊤x2⋮β⊤xn].Y \mid X, \beta, \sigma^2 = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} \sim \Nm (X\beta, \sigma^2 I) \quad \text{where} \quad X \beta = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} \begin{bmatrix} \beta_1 \\ \beta_2 \\ \vdots \\ \beta_n \end{bmatrix} =\begin{bmatrix} \beta^\top x_1 \\ \beta^\top x_2 \\ \vdots \\ \beta^\top x_n \end{bmatrix}.

Normal Linear Regression

SSR⁡(β)=∑i=1n(yi−β⊤xi)2\operatorname{SSR}(\beta) =\sum_{i=1}^n (y_i - \beta^\top x_i)^2

Theorem.

arg⁡min⁡βSSR⁡(β)=(X⊤X)−1X⊤y{\arg\min}_{\beta} \operatorname{SSR}(\beta) = (X^\top X)^{-1} X^\top y

Proof. ddβSSR⁡(β)=−2X⊤y+2X⊤Xβ=0  ⟺  β=(X⊤X)−1X⊤y.\frac{d}{d\beta}\operatorname{SSR}(\beta) = -2 X^\top y + 2 X^\top X \beta = 0 \iff \beta = (X^\top X)^{-1} X^\top y.

References

  • Barbu, A. and Zhu, S. (2020). Monte Carlo Methods. Springer Nature Singapore.
  • Hoff, P. (2009). A First Course in Bayesian Statistical Methods. Springer Texts in Statistics. Springer New York.