library(tidyverse)
library(knitr)
library(ISLR2)
theme_set(theme_bw())

Outline of Today

  1. Announcements
  2. Classification
  3. Lab 12: Classification
  4. Review

Announcements

Classification

4.1 What is Classification?

Classification is the process by which we divide observations into groups. Common examples include “Is this spam”, “Is this fraudulent”, “Is this a real person”.

4.4 Why not use Logistic Regression for everything?

  • Substantial separation: unstable estimates
  • Better accuracy
  • More natural extensions for >2 classes
  • (what if no outcome variable)
  • KNN and Logistic regression: directly estimate P(Y=k|X)
  • Bayes: do the opposite: estimate P(X|Y=k)
    • We can use this with Bayes Rule and the Total Law of Probability to estimate P(Y=k|X)
    • Bayes Rule \(P(E|B)=\frac{P(B|E)P(E)}{P(B)}\)
    • Law of Total Probability:
      • Suppose \(E_1,...E_k\) are mutually exclusive (\(P(E_i\&E_j)\)=0)
      • And Exhaustive \(\sum_kP(E_k)=1\)
      • Then \(P(B)=\sum_k P(B|E_k)P(E_k)\)
  • Example: Two coins, one is double sided with heads, one is normal
    • Randomly select one, flip it, you observe heads
    • Question 1 What is the probability that you selected the double-headed coin?
      • Prior probability:
      • Posterior probability:

Bayes’ Theorem

  • let \(\pi_k\) represent the overal (prior) probability that a randomly chosen observation comes from the kth class
  • let \(f_k(X)=Pr(X|Y=k)\) denote the density function of k for an observation from the kth class.
    • Large if high probability observation in kth class has \(X\approx x\)
    • Low else
  • Then Bayes’ Theorem states: \[Pr(Y=k|X=x)=\frac{\pi_kf_k(x)}{\sum_{l=1}^K\pi_lf_l(x)}\]
  • \(p_k(x)=Pr(Y=k|X=x)\) is the posterior probability that an observation X=x belongs to the kth class
    • That is, the probability that the observation belongs to the kth class given the predictor value for that observation
  • Instead of directly computing \(p_k(x)\) we can estimate 2 quantities
    • \(\pi_k(x)\): easy to compute with a random sample
    • \(f_k(x)\): difficult to estimate. We will need to make some assumptions

4.4.1 Linear Discriminant Analysis for p=1

  • p=1, we only have one predictor
  • Assume f(k) is normal or Guassian: \[f_k(x)=\frac{1}{\sqrt{2\pi}\sigma_k}exp\left(-\frac{1}{2\sigma_k^2}(x-\mu_k)^2\right)\]
  • assume shared variance \(\sigma^2\)
  • Then we have that \[p_k(x)=\frac{\pi_k\frac{1}{\sqrt{2\pi}\sigma}exp\left(-\frac{1}{2\sigma^2}(x-\mu_k)^2\right)}{\sum_{l=1}^k\pi_l\frac{1}{\sqrt{2\pi}\sigma}exp\left(-\frac{1}{2\sigma^2}(x-\mu_k)^2\right)}\]
  • Math (elided): Assign the observation to which \(\delta_k(x)=x\frac{\mu_k}{\sigma^2}-\frac{\mu_k^2}{2\sigma^2}+log(\pi_k)\) is the largest

Question 1 what is the Bayes decision boundary (\(\delta_1(x)=\delta_2(x)\)) if K=2 and \(\pi_1=\pi_2\)

\(~\)

  • Linear Discriminate Analysis:
  • \(\hat{\mu_k}=\frac{1}{n_k}\sum_{i:y_i=k}x_i\)
  • \(\hat{\sigma}^2=\frac{1}{n-K}\sum_{k=1}^K\sum_{i:y_i=k}(x_i-\hat{\mu}_k)^2\)
    • n is the total training observations
    • \(n_k\) is observations in kth class
    • \(\mu_k\) is the average of all observations in the kth class
    • \(\hat{\sigma}^2\) is basically a weighted average of the sample variances
  • Sometimes we know \(\pi_k\)
    • If not, we estimate it as \(\hat{\pi}_k=\frac{n_k}{n}\)
  • assign class based on the value of k that maximizes:
    • \(\hat{\delta}_k(x)=x\frac{\hat{\mu}_k}{\hat{\sigma}^2}-\frac{\hat{\mu}_k^2}{2\hat{\sigma}^2}+\log(\hat{\pi}_k)\)
  • “Linear” Discriminant analysis:
    • discriminant functions \(\hat{\delta}_k(x)\) are linear functions of x

4.4.2 LDA for p>1

  • X is drawn from a multivariate Gaussian
    • each predictor follows a one dimensional normal distribution with some pairwise correlation
  • We will ignore much of the math here, it is in the textbook if you care

4.4.3 Quadratic Discriminant Analysis

  • Unlike LDA, each class has it’s own covariance matrix
  • Both LDA and QDA end up very comparable to logistic regression
    • So I’m going to mostly skip over them and we will move on to Naive Bayes

4.4.4 Naive Bayes

(modified from Professor Wells Slides in addition to the textbook)

  • Goal: Estimate P(Y=k|X)
  • Method: Estimate P(X|Y=k) for all levels of k, combine using Bayes Rule and Law of Total Probability
    • Assume all \(X_i\) independent
      • By multiplication rule: \(P(X|Y=k)=\prod_iP(X_i|Y=k)\)
      • Each term can be estimated individually
        • Continuous: normal distribution model
        • Categorical: proportions
  • Why make an unreasonable (naive) assumption?
    • All models are wrong, some are useful
    • Can’t always estimate a large number of relationships
      • need simplifying assumptions (high bias, low variance)
    • Provides non-linear decision boundaries (trading one flexibility for another)
    • Model accuracy: correctly predicting class of Y, not estimating the probability that Y is in that class
      • Tends to produce woefully incorrect estimates of P(Y=k|X)
      • Usually concurs with the prediction made by the true model
    • Some dependence among variables can “cancel out” in the aggregate
  • Naive Bayes in R:
library(e1071)
nb_mod <- naiveBayes(Y ~ X1 + X2, data = training_data) #Fit
my_preds <- predict(nb_mod, data = test_data) #Predict
my_probs<- predict(nb_mod, data = test_data, type = "raw") #Estimates

Lab

Work with your partner

We will continue to use the Smarket data from Tuesday

Question 3 4.7.5 Fit a Naive Bayes classifier to Direction given Lag 1 and Lag 2. Verify your solution with the text AFTER you work on this on your own.

Question 4 4.8.9 Odds

If additional Time

Question 5 If time, don’t turn in. Using the Titanic dataset

library(rsample)
Titanic=data.frame(Titanic)%>%mutate(Survived = as.factor(Survived))
Titanic_split <- initial_split(Titanic)
Titanic_train <- training(Titanic_split)
Titanic_test <- testing(Titanic_split)

Part A: Exploratory Analysis. What trends are apparent? Are the predictors independent given the response?

library(GGally)
Titanic_train %>% select(Survived, Age, Class, Sex) %>% ggpairs(aes(color = Survived))

Part B: Fit a logistic model to predict survived given age, pclass, embarked, and sex

Part C: Fit a Naive Bayes model to predict survived given age, pclass, embarked, and sex

Part D: Create a results dataframe (Survived, predictions, probabilities) for both B and C

Part E: Compute the accuracy, sensitivity, and specificity for both models

library(yardstick)
my_metrics <- metric_set(accuracy, sensitivity, specificity)
my_metrics(resultsDF, truth = obs, estimate = preds)

Wrap-up

Lab goals:

This week we discussed Logistic Regression and Classification:

  • What is Classification
  • Why Logistic Regression is Superior to Linear Regression for Classification
  • Various parts of Logistic Regression

Course Schedule:

  1. Today: Logistic Regression 4.1-3
  2. Thursday: Naive Bayes Classifier 4.4.4
  3. Next week: Break!
  4. When we get back
    • Unsupervised Classification
    • Trees
    • Forests
    • SVM
    • Other topics as time permits

Reminders:

  • Homework is due Friday at 10pm
  • Labs are due Friday at 10pm
  • Final Project Group is due next Friday at 10pm
  • Reading assignment is due tonight (ish)
  • Send me an email if you need to talk about midterm

\(~\)

Answers

Question 1 What is the probability that you selected the double-headed coin?

  • Prior probability: 1/2
  • Posterior probability: 2/3

Question 2

\[x\frac{\mu_2}{\sigma^2}-\frac{\mu_2^2}{2\sigma^2}+log(\pi)=\delta_2(x)=\delta_1(x)=x\frac{\mu_1}{\sigma^2}-\frac{\mu_1^2}{2\sigma^2}+log(\pi)\] \[\implies x\frac{\mu_2}{\sigma^2}-\frac{\mu_2^2}{2\sigma^2}=x\frac{\mu_1}{\sigma^2}-\frac{\mu_1^2}{2\sigma^2}\] \[\implies x\frac{\mu_2}{\sigma^2}-x\frac{\mu_1}{\sigma^2}=\frac{\mu_2^2}{2\sigma^2}-\frac{\mu_1^2}{2\sigma^2}\] \[\implies x\left(\frac{\mu_2}{\sigma^2}-\frac{\mu_1}{\sigma^2}\right)=\frac{1}{2}\left(\frac{\mu_2^2}{\sigma^2}-\frac{\mu_1^2}{\sigma^2}\right)\] \[\implies x(\mu_2-\mu_1)=\frac{1}{2}(\mu_2^2-\mu_1^2)\] \[\implies x=\frac{1}{2}\frac{\mu_2^2-\mu_1^2}{\mu_2-\mu_1}=\frac{1}{2}\frac{(\mu_2-\mu_1)(\mu_2+\mu_1)}{\mu_2-\mu_1}=\frac{\mu_1+\mu_2}{2}\]

Question 5 DEF

library(e1071)
nb_fit <- naiveBayes(Survived ~ Age + Class + Sex, data = Titanic_train)
my_preds <- predict(nb_fit, Titanic_test)
my_probs <- predict(nb_fit, Titanic_test, type = "raw")
nb_results <- data.frame(obs = Titanic_test$Survived, preds = my_preds, probs = my_probs)
library(yardstick)
my_metrics <- metric_set(accuracy, sensitivity, specificity)
my_metrics(nb_results, truth = obs, estimate = preds)
## # A tibble: 3 × 3
##   .metric     .estimator .estimate
##   <chr>       <chr>          <dbl>
## 1 accuracy    binary         0.375
## 2 sensitivity binary         0.25 
## 3 specificity binary         0.5