library(tidyverse)
library(knitr)
library(ISLR2)
theme_set(theme_bw())

Outline of Today

Announcements
Classification
Lab 11: Logistic Regression
Review

Announcements

Midterm Grades have been released. If you got less than a .66, please come talk to me.
I apologize for the delay in Homework grades, I will catch up over break
Upcoming Projects
- Project 2
  - Will be a classification competition
  - I will set up a form for you to mark students that you do/don’t want to work with during week 8
    - I will do my best to accommodate this
  - Released Week 9, Due Week 10
  - Caveats: no duplicated partners between Project groups (if I can manage that)
  - Shouldn’t be overly difficult so you have time to work on Project 3 around it
- Project 3 (Final)
  - Open ended
    - Collect Data
    - Articulate Research Question
    - Data wrangling/cleaning
    - Exploratory Data Analysis
    - Build (and assess) Statistical models
    - Technical Report
    - Presentation
  - Timeline
    - Proposed Group due this Friday (Next Monday)
      - I will assign groups if needed
    - Proposal (with Research Question and data source) due Friday of Week 9
    - Exploratory Analysis and Clean Data due Week 11
    - Draft Technical Report due Week 12
    - Presentations will be randomly assigned in class Week 14
      - If you have conflicts let me know ASAP for scheduling
    - Final Technical Report due Week 14 Friday at 5 PM

Classification

4.1 What is Classification?

Classification is the process by which we divide observations into groups. Common examples include “Is this spam”, “Is this fraudulent”, “Is this a real person”.

As with Regression we have training observations that we can use to build a classifier.

Example The book Uses

defData<-Default

ggplot(defData,aes(x=balance,y=income,color=default))+
  geom_point(alpha=.25)+
  scale_color_manual(values=c("lightblue","red"))

ggplot(defData,aes(x=balance,y=default,color=as.factor(default)))+
  geom_point()+
  scale_color_manual(values=c("lightblue","red"))

4.2 Why not use Linear Regression?

What if there are more than 2 levels?
- No natural ordering
If there are two levels:
- Is an estimate of P(Y|X), however, it does not provide meaningful estimates

Question 1 Create a Least Squares Regression and add it as an abline to the above plot to demonstrate the non-meaningful estimates.

4.3 Logistic Regression

Goal: Predict Pr(Y|X) or the Probability of Y given X.

Example: Pr(default=Yes|balance)
Common threshold: 0.5 (above=Yes/1, below=No/0)
Thresholds can change depending on context (e.g. more conservative lender might do 0.1)

4.3.1 Logistic Model

Linear Model: \(p(X)=\beta_0+\beta_1X\)
Logistic Model: \(p(x)=\frac{e^{\beta_0+\beta_1X}}{1+e^{\beta_0+\beta_1X}}\)
Produces an S shaped curve between 0 and 1
Produces more useful probabilities
Again Estimated using Maximum likelihood \[p(x)=\frac{e^{\beta_0+\beta_1X}}{1+e^{\beta_0+\beta_1X}}\implies \frac{p(X)}{1-p(X)}=e^{\beta_0+\beta_1X}\]
Odds: \(\frac{p(X)}{1-p(X)}\)
- Between 0 and Infinity
- 0 Low probability (of default)
- \(\infty\) High probability (of default)
log odds or logit: \[\log_e\left(\frac{p(X)}{1-p(X)}\right)=\beta_0+\beta_1X\]
Logistic Regression vs Linear Regression
- Linear Regression: \(\beta_1\) is the change in Y given a 1 unit change in X
- Logistic regression: \(\beta_1\) is the change in the log odds of Y given a 1 unit change in X
  - Equivalently: Multiplies the odds of Y by \(e^{\beta_1}\)

Question 2 What percent of people with the following odds will default? 1/4, 9, 49, 99, N

4.3.2 Estimating the Regression Coefficients

We could use the nonlinear least squares method to fit the model, however we prefer to use maximum likelihood
Maximum Likelihood Estimation: (MLE)
- Find estimates for \(\beta\) such that the predicted probability \(\hat{p}(x_i)\) of default is as close to the observed status as possible
- Ideal: \(p(x_i)=\hat{p}(x_i)\) which is 0 or 1
- Likelihood Function \[\ell(\beta_0,\beta_1)=\prod_{i:y_i=1}p(x_i)\prod_{i\prime:y_{i\prime}=0}(1-p(x_{i\prime}))\]
- Estimates of \(\hat{\beta}\) to maximize \(\ell\)
- MLE is useful in general
  - OLS is a specific case
  - Mathematical details are outside this course (335-336?)

4.3.3 Making Predictions

Once we have the coefficients, we can make predictions by plugging our \(\beta\) values back into the original equation \(p(x)=\frac{e^{\beta_0+\beta_1X}}{1+e^{\beta_0+\beta_1X}}\)

log_model<-glm(default~balance, defData,family = binomial)
summary(log_model)

## 
## Call:
## glm(formula = default ~ balance, family = binomial, data = defData)
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -1.065e+01  3.612e-01  -29.49   <2e-16 ***
## balance      5.499e-03  2.204e-04   24.95   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 1596.5  on 9998  degrees of freedom
## AIC: 1600.5
## 
## Number of Fisher Scoring iterations: 8

predict.glm(log_model,data.frame(balance=1000)) #logodds

##         1 
## -5.152414

predict.glm(log_model,data.frame(balance=1000),type="response") #\hat{p}(x)

##           1 
## 0.005752145

logOdds=predict.glm(log_model,data.frame(balance=1000)) #proof
exp(logOdds)/(1+exp(logOdds))

##           1 
## 0.005752145

Lab

Start Working with partner

4.3.4 Multiple Logistic regression

As with linear regression we can have multiple predictors, and it extends the same way.

\[p(x)=\frac{e^{\beta_0+\sum_i(\beta_iX_i)}}{1+e^{\beta_0+\sum_i(\beta_iX_i)}}\] Question 3

You can use dummy variables: Create a logistic model for default given student status. Are students more or less likely to default?
Create a logistic model to predict default given balance, income, and student. What do you notice?
Can you explain the difference between A and B?

4.3.4 Multinomal Logistic regression

What if there are more than 2 classes? Let there be K classes.
Without loss of generality, let the Kth class be the baseline. \[Pr(Y=k|X=x)=\frac{e^{\beta_{k0}+\sum\beta_{ki}x_i}}{1+\sum_{l=1}^{K-1}e^{\beta_{l0}+\sum_j\beta_{lj}x_j}}\]
This is true for any k\(\neq\)K, for k=K the numerator is 1.
log odds: \[log\left(\frac{pr(Y=k|X=x)}{Pr(Y=K|X=x)}\right)=\beta_{k0}+\sum_i\beta_{ki}x_i\]
Interpretation: must be done with more care
- It is an increase of odds in class k OVER class K
Softmax: works the same way only we don’t select a baseline class. It is used in Machine Learning a lot. We may cover this at the end of the semester.

Additional Problems from the book (4.7)

Question 4 4.7.1 Using the Smarket data from the ISLR2 library answer the following questions:
- What variables are there?
- What do they mean?
- What is the size of the dataset?
- Are any values highly correlated? If so, is that an issue? If not, are you surprised?
Question 5 4.7.2 Continuing to use Smarket
- Try to predict “Direction” using the other variables
- Are any of these statistically significant?
- what does predict() do if you give it a model but no data? (Print the first 10 values that you find)
Question 6 4.7.2 extended: Given a dataset (Smarket) a model (your model above) and a threshold (0.5)
- Add a new variable to the dataset “predicted” that is either 1 or 0 (or whatever levels you are working with)
- using table, create a confusion matrix. What are the FP and FN rates?

If you have additional time

Try to figure out how to run a multinomial regression on the following dataset:

Y=sample(c(1,2,3,5),100,TRUE)
X=runif(100)

Try to figure out how to run the softmax in R on the above dataset

Wrap-up

Lab goals:

This week we discussed Logistic Regression and Classification:

What is Classification
Why Logistic Regression is Superior to Linear Regression for Classification
Various parts of Logistic Regression

Course Schedule:

Today: Logistic Regression 4.1-3
Thursday: Naive Bayes Classifier 4.4.4
Next week: Break!
When we get back
- Unsupervised Classification
- Trees
- Forests
- SVM
- Other topics as time permits

Reminders:

Homework is due Friday at 10pm
Labs are due Friday at 10pm
Final Project Group is due next Friday at 10pm
Reading assignment is due tonight (ish)
Send me an email if you need to talk about midterm

\(~\)

Answers

Question 1

defData2<-defData
defData2$default<-as.integer(defData2$default)
defData2$default<-defData2$default-1
coefs=lm(default~balance,defData2)$coefficients

ggplot(defData2,aes(x=balance,y=default,color=as.factor(default)))+
  geom_point()+
  scale_color_manual(values=c("lightblue","red"))+
  geom_abline(slope=coefs[2],intercept=coefs[1])

Question 2 What percent of people with the following odds will default?

1/4: .2, 1/5, 20%
9: 90%
49: 98%
99: 99%
N: \[\frac{p(x)}{1-p(x)}=N\implies(1-p(x))N=p(x)\implies N-Np(x)=p(x)\implies N=(1+N)p(x)\implies p(x)=\frac{N}{1+N}\]

Question 3

A. You can use dummy variables: Create a logistic model for default given student status. Are students more or less likely to default? More B. Create a logistic model to predict default given balance, income, and student. What do you notice? Income doesn’t seem to matter, Student sign flips C. Can you explain the difference between A and B? Students have more debt on average, but for the same amount of debt are less risky (Confounding variables matter)

log_model<-glm(default~student, defData,family = binomial)
summary(log_model)

## 
## Call:
## glm(formula = default ~ student, family = binomial, data = defData)
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -3.50413    0.07071  -49.55  < 2e-16 ***
## studentYes   0.40489    0.11502    3.52 0.000431 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 2908.7  on 9998  degrees of freedom
## AIC: 2912.7
## 
## Number of Fisher Scoring iterations: 6

log_model<-glm(default~balance+income+student, defData,family = binomial)
summary(log_model)

## 
## Call:
## glm(formula = default ~ balance + income + student, family = binomial, 
##     data = defData)
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept) -1.087e+01  4.923e-01 -22.080  < 2e-16 ***
## balance      5.737e-03  2.319e-04  24.738  < 2e-16 ***
## income       3.033e-06  8.203e-06   0.370  0.71152    
## studentYes  -6.468e-01  2.363e-01  -2.738  0.00619 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 1571.5  on 9996  degrees of freedom
## AIC: 1579.5
## 
## Number of Fisher Scoring iterations: 8

Lecture 11

2025-03-04