# Ch 7: Multiple Regression 2

In Ch 6, we’ve seen linear regression model with multiple variables. To resolve the limitation for testing regression coefficients, we introduce a new concept, extra sum of squares.

## Outline

1. Extra sum of squares
• definition
• application 1 : Tests
• application 2 : coefficient of partial determination
2. Standardized multiple regression model
• correlation transformation and standardized regression model
• multicollinearity issue

## 1. Extra sum of squares

### definition

Recall

• $$SSTO$$ can be divided into $$SSE$$ and $$SSR$$.
• When we add prediction variables to the regression model, $$SSE$$ gets decreased, while $$SSR$$ gets increased.

Using the above concept, define extra sum of squares

$$SSR(X_2 \mid X_1) = SSE(X_1) - SSE(X_1, X_2)$$,

the marginal explanation ability by adding $$X_2$$ into the model containing $$X_1$$. $$SSR(X_2 \mid X_1) = SSR(X_1, X_2) - SSR(X_1)$$ results in the same value..

### application : Tests

Test whether a single $$\beta_k=0$$

$H_0: \beta_3 = 0$

Full model) $$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \beta_3 X_{i3} + \epsilon_i$$

Reduced model) $$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \epsilon_i^\ast$$

test statistic: $$\frac{(SSE(R)-SSE(F))/(df_f - df_r)}{SSE(F)/df_f}$$
$$= \frac{(SSE(X_1,X_2)-SSE(X_1,X_2,X_3))/(n-3 - (n-4))}{SSE(X_1,X_2,X_3)/(n-4)} = \frac{SSR(X_3|X_1,X_2)/1}{SSE(X_1,X_2,X_3)/(n-4)} = \frac{MSR(X_3|X_1,X_2)}{MSE(X_1,X_2,X_3)} \sim F(1,n-4)$$

Test whether several $$\beta_k=0$$

$H_0: \beta_2 = \beta_3 = 0$

Full model) $$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \beta_3 X_{i3} + \epsilon_i$$

Reduced model) $$Y_i = \beta_0 + \beta_1 X_{i1} + \epsilon_i^\ast$$

test statistic: $$\frac{(SSE(X_1)-SSE(X_1,X_2,X_3))/(n-2 - (n-4))}{SSE(X_1,X_2,X_3)/(n-4)}$$
$$= \frac{SSR(X_2, X_3|X_1)/2}{SSE(X_1,X_2,X_3)/(n-4)} \newline = \frac{MSR(X_2,X_3|X_1)}{MSE(X_1,X_2,X_3)} \sim F(2,n-4)$$

Test whether the slopes of $$x_k$$ and $$x_l$$ are the same

$H_0: \beta_2 = \beta_3 = \beta_c$

Full model) $$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \beta_3 X_{i3} + \epsilon_i^\ast$$

Reduced model) $$Y_i = \beta_0 + \beta_c (X_{i1} + X_{i2}) + \beta_3 X_{i3} + \epsilon_i$$ Consider $$X_{i1} + X_{i2}$$ as a new variable.

test statistic: $$\frac{SSR(X_1+X_2|X_3)/1}{SSE(X_1,X_2,X_3)/(n-4)} \newline = \frac{MSR(X_1+X_2|X_3)}{MSE(X_1,X_2,X_3)} \sim F(1,n-4)$$

### application 2 : coefficient of partial determination

We can also calculate $$R^2$$ which accounts for marginal contribution of $$X_2$$ given that $$X_1$$ is in the model. $$R_{2|1}^2 = \frac{SSR(X_2|/X_1)}{SSE(X_1)}$$ $$0 \leq R_{2|1}^2 \leq 1$$ and large value implies large contribution.

## 2. Standardized multiple regression model

When the predictor variables are not scaled properly, it leads to $$det(X^tX) \approx 0$$, which means there is multicollinearity between predictor variables. There is a cure for this problem.

### correlation transformation and standardized regression model

correlation transform $$n$$ : number of observations We use standardized variables $$\frac{Y-\bar{Y}}{s_Y}$$ and $$\frac{X_{ik}-\bar{X_k}}{s_{X_k}}$$.

When we use $$Y_i^\ast$$ and $$X_{ik}^\ast$$ as simple functions of $$\frac{Y-\bar{Y}}{s_Y}$$ and $$\frac{X_{ik}-\bar{X_k}}{s_{X_k}}$$, respectively, $$Y_i^\ast = \frac{1}{\sqrt{n-1}} \frac{Y_i-\bar{Y}}{s_Y} = \frac{Y_i-\bar{Y}}{\sqrt{\sum_i (Y_i -\bar{Y})^2}}$$ where $$s_Y^2 = \frac{\sum_i (Y_i -\bar{Y})^2}{n-1}$$ $$Y_{ik}^\ast = \frac{1}{\sqrt{n-1}} \frac{X_{ik}-\bar{X_k}}{s_{X_k}} = \frac{X_{ik}-\bar{X_k}}{\sqrt{\sum_i (X_{ik} -\bar{X_k})^2}}$$ where $${s_{X_k}}^2 = \frac{\sum_i (X_{ik} -\bar{X_k})^2}{n-1}$$

standardized regression model $$Y_i^\ast = \beta_1^\ast X_{i1}^\ast + \cdots + \beta_{p-1}^\ast X_{i,p-1}^\ast + \epsilon_i^\ast$$

Note

• $\beta_0^\ast = 0$
• $\beta_0 = \bar{Y} - \beta_1 \bar{X_1} - \cdots - \beta_{p-1} \bar{X_{p-1}}$
• $\beta_k = \frac{s_Y}{s_{X_k}} \beta_k^\ast, k=1,2,\cdots, p-1$

Property of correlation matrix of $$X$$ variables Let $$corr(X_i,X_j) = r_{ij}$$.

$$(X^\ast)^t X^\ast = r_{XX}$$ $$r_{XX}b^\ast = r_{YX}$$ $$\Rightarrow b^\ast = \frac{r_{YX}}{r_{XX}}$$

### multicollinearity issue

Why do we need to avoid multicollinearity among variables?

When we solve normal equation $$X^tX b = X^tY$$, the $$det(X^tX) \approx 0$$ condition makes the system highly singular. This leads to high round-off error while computation, resulting in severe error in $$b$$.

How to know if the predictor variables are correlated among themselves?

1. uncorrelated $$X_1$$ and $$X_2$$ $$r_{12}^2=0$$ $$SSR(X_1|X_2) = SSR(X_1)$$ or $$SSR(X_2|X_1) = SSR(X_2)$$
2. perfectly correlated $$X_1$$ and $$X_2$$ $$r_{12}^2=1$$ $$(X^tX)^{-1}$$ Does not exist. $$\Rightarrow$$ infinitely many regression lines
3. general effects of multicollinearity ($$0 < r_{12}^2 < 1$$) $$r_{12}^2 \approx 1 \Rightarrow$$ increase of explanation ability is not significant.