Outline of This week

library(cluster)
library(factoextra)
library(ggplot2)
library(dplyr)

Trees

Why use trees

  • Advantages
    • Easy to interpret
      • Even compared to OLS
    • Similar to human decison making
    • Easy to display graphically
    • Easy to interpret even by non experts
    • Don’t need dummy vairables
  • Disadvantages
    • Less accurate
    • non-robust
      • small changes in input data can lead to very different trees
  • Solutions to disadvantages: Next week

Decision Trees:

(See Board)

Decision Trees consist of:

  • Root node (base of the tree)
  • Branches (lines that connect nodes)
  • Internal Nodes (decision locations)
  • Leaf nodes (final outcome)

Examples:

20 Questions Game

Example 1: On the board, one at a time

Question 1 Working with your partner decide on the 4-5 most useful questions you think could be asked in the first round.

Question 2 Working with the group next to you, decide on the 2 most useful questions you think should be asked

Example 2: On the board, one set of 4 at a time.

Other Examples in life

https://en.wikipedia.org/wiki/Flowchart#/media/File:LampFlowchart.svg
https://en.wikipedia.org/wiki/Flowchart#/media/File:LampFlowchart.svg

Question 3 Using the animals dataset from last week, create a tree to classify the animals as efficiently as possible. Note, you should probably write this on paper/whiteboard and take a picture to include. Justify your choices

data("animals")
animals[animals == 1] <- "No"
animals[animals == 2] <- "Yes"
animals <- apply(animals, 2, as.factor)
colnames(animals) <- c("Warm_blooded?", "Can_fly?", "Vertebrate?", "Endangered?", "Lives_in_groups?", "Has_hair?")
rownames(animals) = c("ant","bee","cat","cephalopod", "chimp", "cow", "duck", "eagle", "elephant", "fly", "frog", "hermit", "lion", "lizard", "lobster", "human", "rabbit", "salamander", "spider", "whale")
animals
##            Warm_blooded? Can_fly? Vertebrate? Endangered? Lives_in_groups?
## ant        "No"          "No"     "No"        "No"        "Yes"           
## bee        "No"          "Yes"    "No"        "No"        "Yes"           
## cat        "Yes"         "No"     "Yes"       "No"        "No"            
## cephalopod "No"          "No"     "No"        "No"        "No"            
## chimp      "Yes"         "No"     "Yes"       "Yes"       "Yes"           
## cow        "Yes"         "No"     "Yes"       "No"        "Yes"           
## duck       "Yes"         "Yes"    "Yes"       "No"        "Yes"           
## eagle      "Yes"         "Yes"    "Yes"       "Yes"       "No"            
## elephant   "Yes"         "No"     "Yes"       "Yes"       "Yes"           
## fly        "No"          "Yes"    "No"        "No"        "No"            
## frog       "No"          "No"     "Yes"       "Yes"       NA              
## hermit     "No"          "No"     "Yes"       "No"        "Yes"           
## lion       "Yes"         "No"     "Yes"       NA          "Yes"           
## lizard     "No"          "No"     "Yes"       "No"        "No"            
## lobster    "No"          "No"     "No"        "No"        NA              
## human      "Yes"         "No"     "Yes"       "Yes"       "Yes"           
## rabbit     "Yes"         "No"     "Yes"       "No"        "Yes"           
## salamander "No"          "No"     "Yes"       "No"        NA              
## spider     "No"          "No"     "No"        NA          "No"            
## whale      "Yes"         "No"     "Yes"       "Yes"       "Yes"           
##            Has_hair?
## ant        "No"     
## bee        "Yes"    
## cat        "Yes"    
## cephalopod "Yes"    
## chimp      "Yes"    
## cow        "Yes"    
## duck       "No"     
## eagle      "No"     
## elephant   "No"     
## fly        "No"     
## frog       "No"     
## hermit     "No"     
## lion       "Yes"    
## lizard     "No"     
## lobster    "No"     
## human      "Yes"    
## rabbit     "Yes"    
## salamander "No"     
## spider     "Yes"    
## whale      "No"

Classification Trees

Much of this is modified from Professor Wells

Predict class rather than value (regression trees)

Metrics are a bit more difficult

  • Can’t use RSS since there isn’t a numeric outcome
  • Natural choice: Classification error rate (proportion of observations not in most common class)
    • Issue Insensitive to increases in node purity
      • That is, a proposed cut can better separate categories while E remains constant
  • Alternative: Gini Index G for a region with K total classes \[G=\sum_{k=1}^K\hat{p_k}(1-\hat{p_k})\]
    • \(\hat{p_k}\) is the proporition of observations in class k
    • Probability that a random element would be incorrectly labeled if it was labeled randomly according to proportions for labels in the region
    • Always between 0 and 1-\(\frac{1}{k}\)
    • Question 4 When is G smallest? Largest?
  • Alternative: Entropy D for region with K classes \[D=-\sum_{k=1}^K\hat{p_k}\log_2\hat{p_k}\]
    • measures the average amount of information gained by learning the true class of an observation, given that you already know the probabilities that it belongs to each class
    • D is always between 0 and \(\log_2(k)\)
    • Question 5 Calculate G and K for a region with 50/30/20 elements belonging to each class A/B/C respectively
    • Question 6 Calculate G and K for the same region split into two regions of sizes 45/10/5, 5/20/15
  • Overall, Both Gini Index and Entropy tend to be more accurate than Error rate E, but neither is consistently better than the other
  • Note: can also handle multilevel categorical variables however this can cause bias depending on how it is done.

Text Problems

Question 7 Work through 8.3.1

Wrap-up

Lab goals:

In this week we covered Tree based models.

Course Schedule:

  1. This week: Trees
  2. Next week: Ensemble Models

Reminders for next class:

  • Labs due Friday
  • Please do readings
  • Project 2 has been released, it is due next week
  • Final Project proposal is due this Friday
  • You have a homework on Clustering due this Friday