library(ggplot2)
library(dplyr)
library(tidyverse)
library(knitr)
library(ggthemes)
library(ISLR2)
theme_set(theme_bw())

Outline of Today

  1. Announcements
  2. Multiple Linear Regression (3.2)
  3. Lab
  4. Wrap up

3.2 Multiple Linear Regression

\[\hat{y}=\hat{\beta_0}+\hat{\beta_1}x_1+...+\hat{\beta_p}x_p\]

Closed form solutions are more complicated and the text omits them, but they are easily computed. We will not compute these by hand.

Questions that we can answer

  1. Is at least one of the predictors useful in predicting the response?
  2. Do all predictors help explain Y, or only a subset
  3. How well does the model fit the data
  4. Given a set of values, what response should we predict, and how accurate do we expect it to be?
  5. (not one of the text questions) We want to know what each X does, not what something that it is correlated to does (sharks, ice cream)

Relationship between Response and Predictors

\[H_0: \beta_1=\beta_2=...=\beta_p\]

\[H_a: \text{at least one }\beta\text{ is non-zero}\]

This hypothesis test uses the F-Statistic:

\[F=\frac{(TSS-RSS)/p}{RSS/(n-p-1)}\] If the linear model assumptions hold: the expected value of the denominator is \(\sigma^2\). If \(H_0\) is true: the expected value of the numerator is \(\sigma^2\). If \(H_a\) is true, we expect that the numerator is \(>\sigma^2\) so we expect \(F>1\).

We can calculate these statistics for particular subsets of coefficients. equation 3.24 and text: these are exactly equivalent to the F-test omiting a single variable allowing for the partial effect of adding a variable to the model (F and p).

Why use F

each individual p value has 5% chance to be below 0.05 by random chance (so 20 non-correlated values would have one by random chance. xkcd jellybeans). p value of F is only under .05 5% of the time no matter how many predictors/observations we have.

https://xkcd.com/882/

image = "https://imgs.xkcd.com/comics/significant.png"
knitr::include_graphics(image)

Deciding on Important Variables

Variable selection: using only the relevant variables (X). We will briefly cover some ideas today, and go into more detail next week.

Potential models (no interaction) grows quickly (\(2^p\) for p variables)

  • Forward selection:
    • start with null model
    • try each simple linear model independently
    • pick the best
    • repeat till stopping rule
  • Backward selection:
    • start with complete model
    • drop largest p value
    • repeat till stopping rule
  • Mixed selection:
    • Start null
    • add variables as in forward selection
    • drop variables when p value rises too large

Forward selection can always be used. Backward cannot be used when \(p>n\). Forward selection is a greedy approach, which Mixed selection can help.

Model Fit

\(R^2=Corr(Y,\hat{Y})^2\). Will always increase as you add new variables. RSS will always decrease. RSE can increase given the p in the denominator \(RSE=\sqrt{\frac{1}{n-p-1}RSS}\)

Graphing can help you to see interaction effects (e.g. edges, see text)

Predictions

  • We predict a least squares plane instead of a least squares line

  • There is model bias (not everything is linear), but we and the text will ignore that

  • Even if we knew f(X) we couldn’t perfectly predict Y due to \(\epsilon\)

  • Confidence interval:

    • 95% of intervals will contain the true value of f(X)

  • Prediction interval:

    • 95% sure this interval will contain the true value of Y
    • Same mean, more uncertainty, wider interval

Lab: Multiple Linear Regression

Lab: Partner work

We will continue using the lab from Tuesday.

Multiple Linear Regression

Syntax: lm(Y~X1+X2+X3,data=DataFrame): regression model y given x1, x2, and x3

Syntax: lm(y~.,data=DataFrame): regression model y given everything else

Syntax: lm(y~.-x3,data=DataFrame): regression model y given everything else except x3

Question 1 Fit a complete model (all columns) of the Boston Dataset with medv as the outcome (Y) variable. What are the coefficients?

Interaction terms

Syntax: x1:x2, creates the interaction variable

Syntax: lm(y~x1*x2) models y given x1, x2, and x1:x2

Question 2 Again using the Boston Dataset

  1. Create a linear model with medv as your outcome variable, lstat and age being your input variables. Include an interaction term
  2. Which is the most important variable based on p value?
  3. Plot the residuals, what do you notice?

Qualitative Predictors

Question 3 Using the Carseats data.

  1. Fit a linear model to predict Sales using all variables as well as a price:age interaction term
  2. Explore how contrasts() works to create dummy variables
  3. plot the residuals.

Question 4 Using the Carseats data, fit the “best” regression you can. Explain your steps, justify why it is the best, and why you used the variables that you did.

Things to think about:

  • Variable selection (forward, backward, mixed)
  • polynomial terms
  • interaction terms
  • quantitative variables
  • p value, Rsquared, Adjusted R squared, F-statistic

Wrap-up

Lab goals:

In this lab we practiced the following:

  • Multiple Linear Regression
  • Interaction variables
  • Qualitative predictors

Course Schedule:

  1. Today: Multiple Linear Regression
  2. Next week: Model Selection and Regularization

Reminders for next class:

  • Homework 2 is due Friday at 10pm
  • Labs 5,6 are dur Friday at 10pm
  • The reading assignment for Chapter 6.1,2,4 of the text is due Monday at 10pm