library(tidyverse)
## Warning: package 'tidyverse' was built under R version 4.4.2
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.4 ✔ readr 2.1.5
## ✔ forcats 1.0.0 ✔ stringr 1.5.1
## ✔ ggplot2 3.5.1 ✔ tibble 3.2.1
## ✔ lubridate 1.9.3 ✔ tidyr 1.3.1
## ✔ purrr 1.0.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Outline of Today
- Announcements
- Resampling Methods
- Lab 9: Cross Validation
- Lab 10: Bootstrap
- Review
Announcements
- Labs this week will be in groups of 3 with your Midterm group
- Labs this week will be short to allow for project work time
- Both Labs will be available in this document, we will go through the
material together Tuesday
- Please turn them in separately. Be careful with your question
numbering
- You may choose to work on the project or on Lab 10 after finishing
Lab 9
- You are still expected to come to class on Thursday.
- Midterm Instructions have changed!
Resampling Methods
- Repeatedly drawing samples from a training set and refitting a
model
- Computationally expensive (although not prohibitive)
- Cross-Validation (Today)
- Estimate Test Error
- Select appropriate level of flexibility
- Model Assessment (not selection)
- Bootstrap (Thursday)
- Measure of Accuracy of parameter estimate
- Measure of Accuracy of a given statistical learning method
Class work:
Lab 9
Cross Validation
- Training Error vs Testing Error
- Training is easy to get
- Testing may not
- Holding out a subset of the training observations
- Validation set:
- Randomly divide into training and validation sets
- Train on training set
- Predict validation set: this is an estimate of Test error rate
- Issues:
- Validation estimate can be highly variable
- Reduces training data->worse fit-> may overestimate error
set.seed(9)
X <- runif(50, 0,10)
Y <- 5+X+2*X^2+rnorm(50, mean=0, sd=1)
my_df <- data.frame(X,Y)
set.seed(9)
MSE<-numeric(10)
for (i in 1:10){
temp_df<-my_df[sample(nrow(my_df)),] #random sort
train_df<-temp_df[1:30,]
val_df<-temp_df[31:50,]
temp_lm<-lm(Y~poly(X,2,raw=TRUE),train_df)
MSE[i]<-mean((predict(temp_lm,val_df)-val_df$Y)**2)
}
MSE
## [1] 0.9577663 1.2026312 1.0974934 1.1899052 1.0873191 0.8984011 1.3263010
## [8] 0.9867985 1.2864615 1.7732997
Question 1 What would we expect the MSE to be? Why?
Can you prove it?
Leave-One-Out Cross-Validation
- LOOCV
- A single observation is the validation set, the other observations
are the training set
- Repeat for all n observations
- \(CV_{(n)}=\frac{1}{n}\sum_{i=1}^nMSE_i\)
- Computationally expensive
- Except in regression!
- Fit entire dataset
- \(CV_{(n)}=\frac{1}{n}\sum_{i=1}^n\left(\frac{y_i-\hat{y_i}}{1-h_i}\right)^2\)
Question 2 Perform LOOCV on the following
dataset:
X<-c(1,2,3,4,5)
Y<-c(5.25,9,15,21.25,24.25)
k-fold Cross-Validation
- Randomly split data into k folds of approximately equal size
- A single fold is the validation set, the others make the training
set
- \(CV_{(n)}=\frac{1}{k}\sum_{i=1}^kMSE_i\)
- Potential Goals:
- Determine how the stat learning procedure can be expected to perform
on independent data
- Estimate of test MSE is sufficient
- Identify best method
- location of minimum test MSE is important, value is less
important
Question 3 Why would k=5 or 10 be useful instead of
k=n (LOOCV), k=2 (train/val), or k=1 (no split)
Bias-Variance Tradeoff
- Bias
- Validation: overestimate error
- LOOCV approximately unbiased estimate
- k-fold: intermediate bias
- in other words: LOOCV>k-fold>Validation
- Variance
- LOOCV has more variance than k-fold due to high correlation in
models
Cross Validation on Classification
- Works very similarly. MSE is replaced with Err where \(ERR_i=I(y_i\neq\hat{y_i})\)
Lab 10
The Bootstrap
the below is from the text
- Can be used to quantify the uncertainty associated with a given
estimator or statistical learning method
- Allows us to use a computer to emulate the process of obtaining new
sample sets
- Works by repeatedly sampling observations from the original
dataset
- Sampling is done With Replacement
- Estimate parameter of interest using the repeated samples
- Compute Standard Error
the below is modified from Professor Wells
- How is a statistic distributed?
- Traditional approach: Theoretical Distribution
- Formulate it as a function of sample observations
- Make (often unreasonable) assumptions
- Look up theoretical distribution
- Hope that Central Limit Theorem applies
- Alternative approach: Simulation (Optimism)
- Obtain a large number of sample set
- compute statistic of interest on each set
- Plot and summarize distribution
- Issue: unreasonable to obtain large number of sample sets
- Bootstrap:
- Assume sample is large enough to be “Representative” of the
population
- Create a bootstrap sample by sampling with replacement up to the
original sample size
- Repeat
\(~\)
Question 1 Assume You had a sample with 500 paired
observations X and Y. You want to estimate the value of the parameter
\(\beta_1\) in the standard linear
regression formula \(Y\sim \beta_0+\beta_1
X\). You decide to use bootstraping with 1000 generated bootstrap
samples. How many total observations will you be working with?
Partner work
Lab 9: Partner work
Question 4 5.3.2 in the textbook
Question 5 5.3.3 in the textbook
Lap 10: Partner work
the below is modified from Professor Wells
Question 2 Using the Model \(Y=1+2X_1+3x_2+5X1X2+\epsilon\) where \(\epsilon\sim N(0.03)\)
- Simulate 1000 sample sets (not using bootstrapping, use the
underlying formula). For each sample set:
- fit the linear model, record the interaction term coefficient
- plot the simulated coefficients
- calculate mean and sd of the coefficient
- Using the single sample below, generate 1000 bootstrap samples. For
each sample set:
- fit the linear model, record the interaction term coefficient
- plot the simulated coefficients
- calculate mean and sd of the coefficient
- Which method is more accurate? Which method is more reasonable in
the real world where you don’t know the underlying model?
set.seed(10)
X1<-runif(400,0,1)
X2<-runif(400,0,1)
Y=1+2*X1+3*X2+5*X1*X2+rnorm(400,0,0.3)
data_10<-data.frame(X1,X2,Y)
summary(lm(Y~X1*X2,data=data_10))$coefficients
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.030908 0.06584171 15.65737 2.371525e-43
## X1 1.938174 0.11177104 17.34058 1.603156e-50
## X2 2.911687 0.10836403 26.86950 2.829162e-91
## X1:X2 5.137203 0.18406815 27.90925 1.489651e-95
Question 3 5.3.4 (Estimating the Accuracy of a
Statistic of Interest)
Question 4 5.3.4 (Estimating the Accuracy of a
Linear Regression Model)
Wrap-up
Lab goals:
This week we discussed Resampling:
- Validation Sets
- Cross Validation
- Bootstrap
Course Schedule:
- Today: Resampling
- Thursday: Resampling Cont
- Next week: TBD
Reminders for next class:
- Homework is due Friday at 10pm
- Labs are due Friday at 10pm
- Midterm is due next Friday at 10pm
- Reading assigment is due tonight (ish)
- No office hours Friday