RA ch11 Remedial measures
Ch 11: Remedial measures
Transformation is one of the standard remedial measure for a linear model. Recall that its uses are:
- to linearize the regression relationship
- to make the error distribution more nearly normal
- to make the variances of the error terms more nearly equal.
In this chapter, we will find additional remedial measures to handle several pitfalls. Then we discuss non-parametric regression methods (which are quite different from previous regression models). A common feature of the remedial measures and alternative regression methods is that estimation procedures from the ways we’ve seen are relatively complex, so we need easier and more generic way to evaluate the precision of these complex estimators. Bootstrapping is one example.
Outline
-
Other remedial measures
- unequal error variance - weighted least squares
- multicollinearity - ridge regression
- influential cases - robust regression
-
Nonparametric regression
- regression trees
-
Bootstrap confidence intervals
1. Other remedial measures
unequal error variance - weighted least squares
Generalized multiple regression model
When we use ordinary least square estimators, then still we have these properties:
- the estimators of
are unbiased - the estimators are consistent consistency is defined that
for succiciently large ( converges to ). However, the estimators are no longer have minimum variance. This is due to the unequal error variances, making cases no longer have the same reliability. In other words, observations with small variances provide more reliable information.
A remedy to handle this problem is to use weighted least squares. First, we start from the simplest case.
when error variances are known Suppose that the errror variances
Here, we substitute
Since
This is just as the same as minimizing
when variances are known up to proportionality constant Now, relaxing the condition, assume that we only know proportions among error variances. Consider a case that the second error variance is twice larger than the first one. i.e.
when error variances are unknown We use estimators of the error variances. There are two approaches.
- Estimation by residuals
. Hence, the squared residual is an estimator of . - Use of replicates or near replicates
- In a designed experiment (such as scientific simulations), replicate observations could be made. If the number of replicates is large, use the fact that sample variance gets closer to variance.
- However, in an observational studies (such as social studies), we use near observations as replicates.
Inference procedures should be followed when weights are estimated. We want to estimate a variance-covariance matrix
Lastly, we can use the ordinary least squares leaving the error variances unequal. Regression coefficient
Remark on weighted least squares Weighted least squares can be interpreted as transforming the data
and by setting
where
multicollinearity - ridge regression
When multicollineariry is detected in a regression model, we have fixed the model by
- using centered data in polynomial regression models
- drop some redundant predictor variables
- add some observations that could break the multicollinearity
- use pricipal components instead of the current variables
- Ridge regression.
Ridge regression perturb the estimated coefficient

To obtain
Choice of biasing constant
- using Ridge trace Try several
and plot ’s according to values of . Take the point where the fluctuation in ceases.

- using
Try several and plot ’s according to values of . Take the point where the fluctuation in ceases.
influential cases - robust regression
When an outlying influential case is not clearly erroneous, we should proceed a further examination to obtain important clues to improve the model. We may find
- the omission of an important predictor variable,
- incorrect functional forms, or
- needs to dampen the influence of such cases.
Robust regression is an approach of regression to make the results less rely on outlying cases. There are several ways to conduct robust regression:
- LAR (= LAD = minimum
-error) regression Least Absolute Residuals (= Least Absolute Deviations = minimum ell-one-error regression) is done by minimizing . When there’s an outlier where its residual is large , then its squared residual gets much larger so that the regression function could overweigh this observation. This can be alleviated by using absolute values of residuals. Note that function is not differentiable. Thus, we cannot obtain an estimator using partial derivatives. Instead, this problem can be dealt by linear programming (a kind of optimization technique). - IRLS robust regression Iteratively Reweighted Least Squares. This approach uses the weighted least square procedure as introduced before, but, weights on residuals are now determined by the magnitude of how far outlying a case is. The weights are updated iteratively in a way to flatten the weights.
- LMS regression Least Median Squares. Recall that average is sensitive to outliers, but median is not. Thus, we could minimize the function
.
2. Nonparametric regression
For complex cases, it is hard to guess a right functional form (analytic expression) of the regression function. Also, as the number of parameter increases, the space of observations gets rarefied (there are fewer cases in a neighborhood), so that the regression function tends to behave erratically. For these cases, nonparametric regression could be helpful.
regression trees
Regression tree is a powerful, yet computationally simple method of nonparametric regression. It requires virtually no assumptions, and it is simple to interpret. It is popular for explanatory studies, especially for large data sets.

In each branch of a regression tree, the range of a specific predictor variable
How to find a “best” regression tree is linked with
- the number of segmented regions
, and - the split points
that optimally divides the data into two sets at each branch.
The basic idea is to choose a model which minimizes the error sum of squres for the resulting regression tree.
3. Bootstrap confidence intervals
Above regression approaches are complex so that it is hard to estimate confidence intervals using the previous analysis on
An
Enjoy Reading This Article?
Here are some more articles you might like to read next: