Ch 1: Linear Regression with One Predictor Variable

Outline

1. Simple Regression Model

   • Formulation by Least Square Estimation
   • Gauss-Markov theorem
   • Properties of linear regression model
   • Estimation of ${\sigma }^2=Var(E)$

2. Normal Error Regression Model

   • Formulation by Maximum Likelihood Estimation
   • Advantages of Normal error regression model

---

1. Simple Regression Model

Formulation by Least Square Estimation

Let $Y$ be dependent / response variable, and $X$ be independent / explanatory / predictor variable. Given $n$ paired data $(X_i,Y_i),i=1,\cdots ,n$, we want to find a linear function $f(x)={\beta }_0+{\beta }_{1}x$ s.t.

$$Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i,i=1,\cdots ,n$$

where ${ϵ}_i$ stands for the error between the function value $f(X_i)$ and the response $Y_i$, $E({ϵ}_i)=0$, $Var({ϵ}_i)={\sigma }^2$, ${ϵ}_i\perp {ϵ}_j$ for $i\ne j$.

There are various ways to draw a line which crosses the data points $(X_i,Y_i),i=1,\cdots ,n$. Among them, it is reasonable to select one with the least sum of squared errors $\sum _i{ϵ}_i^2$.

Define a function $Q$ parametrized by ${\beta }_0,{\beta }_1$,

$$Q=Q({\beta }_0,{\beta }_1)=\sum _i(Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2=\sum _i{ϵ}_i^2$$

Note

• $Q$ is convex.
• Every convex function has a global minimum.
• $Q$ is differentiable over ${\beta }_0,{\beta }_1$.

By differentiating the function $Q$ over ${\beta }_0,{\beta }_1$, find a critical point $(b_0,b_1)$ by setting each partial derivative = 0.

$$\frac{\partial Q}{\partial {\beta }_0}=-2\sum _i(Y_i-{\beta }_0-{\beta }_1{X}_i)=0$$ $$\frac{\partial Q}{\partial {\beta }_1}=-2\sum _i{X}_i(Y_i-{\beta }_0-{\beta }_1{X}_i)=0$$

The solution for the above simultaneous equations is

$$b_0=E(Y)-b_{1}E(X)$$ $$b_1=\frac{E[(X_i-E(X))(Y_i-E(Y))]}{\sum _i(X_i-E(X){)}^2}=\frac{Cov(X,Y)}{Var(X)}$$

Proof

1. $$n{\beta }_0=\sum _i(Y_i-{\beta }_1{X}_i)$$
2. $\sum _i{X}_i(Y_i-{\beta }_0-{\beta }_1{X}_i)=\sum _i(X_i-E(X))(Y_i-{\beta }_0-{\beta }_1{X}_i)$ $=\sum _i(X_i-E(X))(Y_i-E(Y)+{\beta }_{1}E(X)-{\beta }_1{X}_i)$

Remark

1. Once a dataset $(X_i,Y_i),i=1,\cdots ,n$ is given, $X_i$s are known constants. Also, $E(Y_i)={\beta }_0+{\beta }_1{X}_i$s are considered as constants.
2. The error term ${ϵ}_i$ is a random variable.
3. $Y_i$ is the sum of the constant ${\beta }_0+{\beta }_1{X}_i$ and random ${ϵ}_i$. Therefore, $Y_i$ is also a random variable.
4. Since $E({ϵ}_i)=0$, </br>$E(Y_i)=E({\beta }_0+{\beta }_1{X}_i+{ϵ}_i)={\beta }_0+{\beta }_1{X}_i$, and </br>$Var(Y_i)=Var({ϵ}_i)={\sigma }^2$.

Gauss Markov theorem

Statement Under the regression model, the least square estimators of regression coefficients $b_0,b_1$ are

1. unbiased (i.e. $E(b_0)={\beta }_0,E(b_1)={\beta }_1$) and
2. having minimum variance among all unbiased linear estimators. (i.e. $Var(b_i)\le Var(b_i^{\ast }),i=0,1$, $b_i^{\ast }$ unbiased, linear)

Shortly, it is said that “$b_0,b_1$ are BLUE(best linear unbiased estimator) of ${\beta }_0,{\beta }_1$, respectively.

Proof go to link

Properties of linear regression model

Notations Observation : $Y={\beta }_0+{\beta }_{1}X+ϵ$ Real line : $E(Y)={\beta }_0+{\beta }_{1}X$ (unknown) Fitted line : $\hat{Y}=b_0+b_{1}X$ (known)

residual : $e_i=Y-\hat{Y}$ (known) error : ${ϵ}_i=Y-E(Y)$ (unknown)

1. $\sum _i{e}_i=0$

   $\because \sum _i(Y_i-\widehat{Y_i})=\sum _i(Y_i-b_0-b_1{X}_i)=0$ by the choice of $b_0,b_1$

2. $\sum _i{e}_i^2$ is the minimum
3. $$\sum _i{Y}_i=\sum _i\widehat{Y_i}$$
4. $\sum _i{X}_i{e}_i=0$ (residual $e$ is orthogonal to $X$)

   $\because \sum _i{X}_i{e}_i=\sum _i(X_i-\overline{X})e_i=\sum _i(X_i-\overline{X})(Y_i-(\overline{Y}+b_1(X_i-\overline{X})))$ > $=\sum _i(X_i-\overline{X})(Y_i-\overline{Y})-b_1\sum _i(X_i-\overline{X}{)}^2$ > $=\sum _i(X_i-\overline{X})(Y_i-\overline{Y})-\frac{\sum _i(X_i-\overline{X})(Y_i-\overline{Y})}{\sum _i(X_i-\overline{X}{)}^2}\sum _i(X_i-\overline{X}{)}^2=0$

5. $\sum _i\widehat{Y_i}{e}_i=0$ (residual $e$ is orthogonal to the fitted line $\hat{Y}$)
6. $\widehat{Y_i}=b_0+b_1{X}_i$ (regression line passes through $(\overline{X},\overline{Y})$)

   $$\because \widehat{Y_i}=\overline{Y}+b_1(X_i-\overline{X})$$

Estimation of ${\sigma }^2=Var(E)$

$e_i=Y_i-\widehat{Y_i}$ Define $SSE$ (Sum of Squared Error) $=\sum _i(Y_i-\widehat{Y_i}{)}^2=\sum _i{e}_i^2$ $s^2=MSE$ (Mean Square Error) $=\frac{SSE}{n-2}=\sum _i\frac{(Y_i-\widehat{Y_i}{)}^2}{n-2}=\sum _i\frac{e_i^2}{n-2}$ ($n-2$ is the degree of freedom, df of the model)

Observation

• $E(MSE)={\sigma }^2$ ($s^2$ is an unbiased estimator of ${\sigma }^2$)
• $$Var({ϵ}_i)=\frac{\sum _i{ϵ}_i^2}{n}$$

2. Normal Error Regression Model

Formulation by Maximum Likelihood Estimation

Assuming a normal distribution to the error terms ${ϵ}_i$, we can estimate $Var({ϵ}_i)={\sigma }^2$ as well as ${\beta }_0,{\beta }_1$.

Now the formulation is given as $Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i$ where ${ϵ}_i\sim N(0,{\sigma }^2),{ϵ}_i\perp {ϵ}_j$ for $i\ne j$, (pdf of ${ϵ}_i$) $=\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{{ϵ}_i^2}{2{\sigma }^2})$

We estimate ${\beta }_0,{\beta }_1$ and ${\sigma }^2$ from Maximum Likelihood Estimation. Since $E(Y_i)={\beta }_0+{\beta }_1{X}_i$ and $Var(Y_i)={\sigma }^2$, pdf of $Y_i$ is given as $f_i=\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{(Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2}{2{\sigma }^2})$.

Using the i.i.d. condition of variables $Y_i$ and $Y_j$, the likelihood function, $L$ is given as

$$L({\beta }_0,{\beta }_1,{\sigma }^2)={\mathrm{\Pi }}_i{f}_i=\frac{1}{(\sqrt{2\pi }\sigma {)}^n}exp(-\sum _i\frac{(Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2}{2{\sigma }^2})$$

We maximize $L$ or $logL$ by calcuating the critical point. Then the estimators obtained by maximizing likelihood are given as

$$\widehat{{\beta }_0}=b_0=\overline{Y}-b_1\overline{X}$$ $$\widehat{{\beta }_1}=b_1=\frac{\sum _i(X_i-\overline{X})(Y_i-\overline{Y})}{\sum _i(X_i-\overline{X}{)}^2}$$

(same as LSE)

$$\widehat{{\sigma }^2}=\frac{\sum _i(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i{)}^2}{n}=\frac{\sum _i(Y_i-\widehat{Y_i}{)}^2}{n}$$

cf) $MSE=s^2=\frac{\sum _i(Y_i-\widehat{Y_i}{)}^2}{n-2}$ is an unbiased estimator of ${\sigma }^2$. $\widehat{{\sigma }^2}$ is not an unbiased estimator of ${\sigma }^2$.

Advantages of Normal error regression model

By introducing normal distribution on the error ${ϵ}_i$, we could obtain an estimator of ${\sigma }^2$. Also, in the following chapter, we could find out this normality condition enables to calculate confidence intervals to several variables in the linear regression model.

---

Here is the jupyter notebook script to run several practice codes using R.