Lecture 7

Outline of Today

1. Intro and Announcements. Midterm will not be released until Thursday.
2. Model Selection and Regularization: 6.1,2,4
3. Lab
4. Review

Model Selection

Why not just use every variable? Why not just use least squares?

• Prediction accuracy: if \(n>>p\), and true relationship is approximately linear, OLS is fine. Otherwise, there can be issues due to variability. If \(p>n\) we can’t use OLS at all.
• Model Interpretability: Fewer parameters and less complexity make for more interpret able models

\(~\)

Three classes of methods:

• Subset Selection: identify a subset of parameters, fit a model with Least Squares
• Shrinkage: Fit a model with all parameters, shrink coefficients towards 0 (regularization)
• Dimension Reduction: Reduce p parameters through projection. e.g. PCA. Not covered today

Subset Selection

Background

• In theory: fit Least Squares regressions to every possible combination of parameters. Choose the best among \(2^p\) options
• In practice:
  1. fit all models with k predictors
  2. find the best, call it \(M_k\)
  3. compare \(M_k\)
  • Does not work for $>$40 parameters. Alternatives are going to be used in this course
  • RSS and \(R^2\) are monotonically decreasing so poor choices for comparison

Efficient option 1: Stepwise Selection

• Forward Stepwise Selection:
  0. Fit the null model \(M_0\)
  1. Fit all models with 1 predictor
  2. Find the best (RSS/\(R^2\)), add it to the model \(M_1\)
  3. Fit all p-1 models with an additional parameter, keeping all parameters in \(M_{2}\)
  4. Repeat
  5. Choose the best \(M_i\)
  • \(1+\frac{p(p+1)}{2}\) models instead of \(2^p\) (e.g. 211 vs 1048576 when p=20)
  • Greedy algorithm, may not be optimal
• Backward Stepwise Selection:
  0. Fit the model with all predictors \(M_p\)
  1. Fit all models with p-1 predictors, keep the best (based on RSS/\(R^2\)) \(M_{p-1}\)
  2. Fit all p-2 models based on \(M_{p-1}\)
  3. Repeat
  4. Choose best \(M_i\)
• Hybrid approaches:
  • some forward, some backward

Choosing the optimal model

1. Indirectly Estimate test error by making an adjustment to the training error to account for overfitting bias
2. Directly Estimate test error using a validation set or cross validation (C5, next week)

Indirect estimates

• \(C_p\): \(C_p=\frac{1}{n}(RSS+2d\hat{\sigma}^2)\)
  • \(\hat{\sigma}^2\) is an estimate of \(\epsilon\) variance
  • d is the number of variables
• AIC: Akaike Information Criterion \(AIC=\frac{1}{n}(RSS+2d\hat{\sigma}^2)\)
  • Defined for models fit by Maximum Likelihood. In LS this is the same
• BIC: Bayesian Information Criterion : \(BIC=\frac{1}{n}(RSS+\log(n)d\hat{\sigma}^2)\)
  • Larger parameter penalty
• Adjusted \(R^2\) \(1-\frac{RSS/(n-d-1)}{TSS/(n-1)}\)
  • Unlike the above 3, a higher value is good

Direct estimates

• Validation set: held out data that was not used to train the model
• Cross validation: next week

Efficient option 2: Shrinkage Methods

Instead of finding and fitting a subset of parameters, we can constrain or regularize the coefficient estimates.

We fit models that minimize a function. In OLS we try to minimize RSS.

• (ordinary) Least Squares \[RSS=\sum_{i}\left(y_i-\beta_0-\sum_{j}\beta_jx_{ij}\right)\]
  • Last week
• Ridge Regression \[RSS+\lambda\sum_{j}\beta_j^2\]
  • \(\lambda\geq0\) is a tuning parameter.
  • The second term is the shrinkage penalty, which is smaller when \(\beta\) is close to 0.
  • The intercept, \(\beta_0\), is not included in the second term
  • Advantages:
    • Computationally efficient over Subset Selection
    • Simultaneously fitting all values of \(\lambda\) are almost identical to fitting a single least squares model
  • Disadvantages:
    • It will generally include all variables. This is not a problem for prediction accuracy, but may be for interpretation.
• Lasso \[RSS+\lambda\sum_{j}|\beta_j|\]
  • Very similar to Ridge Regression
  • \(\ell_1\) vs \(\ell_2\)
  • Advantages:
    • Computationally efficient
    • Leads to sparse models (subset of variables)

Question 1

What happens when \(\lambda=0\)? When \(\lambda\rightarrow\infty\)? When \(\lambda=\infty\)?

• Part a: In Ridge Regression
• Part b: In Lasso

Question 2

In Least Squares, given \(Y=\beta_0+\beta_1 X\) how does \(\beta_1\) change if X is multiplied by a constant, c?

\(~\) \(~\) \(~\) \(~\)

PLEASE DON’T SCROLL DOWN TILL WE TALK ABOUT QUESTION 2

\(~\) \(~\) \(~\) \(~\)

Assorted Notes

OLS is scale equivariant (\(X_j\beta_j\) will always be the same). However, Ridge Regression is not. Therefore we should always standardized the predictors The book divides by the standard deviation of the predictor: \(x=\frac{x}{\sigma(x)}\). I prefer to subtract the mean as well which gives more interpretable \(\beta_0\) values. \(x=\frac{x-\bar{x}}{\sigma(x)}\)

• As \(\lambda\) increases, flexibility decreases:
  • Less variance
  • More bias
• Reframing the minimization problem allows us to show that the differences depend on the amount of “budget” we have for the model.
  • If large, OLS=Lasso=Ridge
  • If small, Lasso, Ridge are computationally feasible alternatives to subset selection
• Why does Lasso lead to variable selection when Ridge Regression doesn’t?
  • See page 245, figure 6.7
  • Intersection of contours: line vs circle
• Comparison:
  • Lasso More interpretable
  • Ridge generally slightly lower variance/MSE

Selecting \(\lambda\)

Cross validation: select lowest error. Next week

Dimension Reduction

Not today

High dimensional issues

• \(n>>p\) we are fine
• High dimensional setting: \(p>n\) or \(p\approx n\)
  • Least Squares will not work well
  • Example below/with class

\(~\)

Question 3 Generate 6 random variables of length 10. (Y and 5 X variables). Use a seed of 0 for replicability. Using only the first 5 values. Fit 6 models in order Y given nothing, Y given X1, Y given X1 and X2,… Y given everything else. Use the remaining 5 values as the test set.

1. How does the Train MSE/RSS/\(R^2\) change as you add variables?
2. How does the Test MSE/RSS/\(R^2\) change as you add variables?
3. What do you notice? Why?

\(~\)

Notes in high dimensions:

• Regression in High dimensions (6.4.3)
  1. Regularization or shrinkage plays a key role in high-dimensional problems
  2. Appropriate tuning parameter selection is crucial for good predictive performance
  3. Test error tends to increase as the dimensionality of the problem increases, unless the features are truly associated with the response (Curse of dimensionality)
• Interpreting Results (6.4.4)
  • Multicollinearity is a huge issue
  • We can never truly know the truly predictive/best parameters or coefficients
  • Never use \(R^2\), p values, or traditional measures of fit on the training data

\(~\)\(~\)\(~\)\(~\)\(~\)\(~\)

Lab: Subset Selection

Lab: Partner work

This will be a short lab (number of questions), but a potentially long lab (depth of questions). For all questions use the below dataset

library(glmnet)

## Warning: package 'glmnet' was built under R version 4.4.2

## Loading required package: Matrix

## Loaded glmnet 4.1-8

library(leaps)
set.seed(0)
V1<-rnorm(100,0,1)
df=data.frame(V1)
for(i in 1:20){
  df[,i]=rnorm(100,sample(-5:5,1),runif(1,min=0,max=4))
}
df$V21=df$V20/2
df$V22=df$V2-2
df$Y=5+df$V1-2*df$V2+3.5*df$V10-20*df$V21+10*df$V7+rnorm(100,0,1)

grid=c(0,1,10,100)

Question 4 This is the only question to turn in if you were in class today.

1. Non coding questions
   1. What is the equation of this relationship?
   2. How are the X generated (possible means, possible sd, expected sd)
2. Without normalizing your data, manually fitting the models using lm
   0. Use a linear regression with all possible values
   1. Use forward stepwise selection to fit the best model
   2. Use backward stepwise selection to fit the best model
   3. Use Ridge Regression and lambda = 0, 1, 10, 100 (hint use glmnet(alpha=0,standardize=FALSE))
   4. Use Lasso and lambda = 0, 1, 10, 100
3. With normalizing your data, manually fitting the models using lm
   1. Use forward stepwise selection to fit the best model
   2. Use backward stepwise selection to fit the best model
   3. Use Ridge Regression and lambda = 0, 1, 10, 100
   4. Use Lasso and lambda = 0, 1, 10, 100 (hint use glmnet(alpha=1,standardize=FALSE))
4. Using the normalized dataset and the leaps package
   1. Fit the best subset model of size 4, 5, 6 using regsubsets()
   2. Fit the best subset model using forward selection in regsubsets()
   3. Fit the best subset model using backward selection in regsubsets()
5. glmnet practice
   1. Using the normalized data, and standardization = TRUE
      • Fit a Ridge Regression with lambda = 0, 1, 10, 100
      • Fit a Lasso with lambda = 0, 1, 10, 100
   2. Using the unnormalized data, and standardization = TRUE
      • Fit a Ridge Regression with lambda = 0, 1, 10, 100
      • Fit a Lasso with lambda = 0, 1, 10, 100
6. What do you notice?
   1. Which methods seem better? Worse? The same?
   2. Do any models get the right answer? Why? Why not?

Lab goals:

In this lab we reviewed the following:

• Model Selection and Regularization (Theory/how it works)
• Model Selection and Regularization (how it works in R)

Course Schedule:

1. Today: Model Selection and Regularization
2. Tomorrow: Polynomial Regression
3. Next week: Resampling Methods

Reminders for next class:

• Homework is due Friday at 10pm
• Labs are due Friday at 10pm
• Midterm will be released (Thursday?)
• Reading for Thursday is due Tonight