Outline of Today

  1. Terminology
  2. Foundations of Statistical Learning
  3. Discussion
  4. Lab: ggplot

Foundations of Statistical Learning

Important Terms:

  • Variables:
    • Input Variables:
      • predictors, independent variables, features, variables
    • Output Variables:
      • dependent variable, response variable
  • General form:
    • \(Y=f(X)+\epsilon\)
    • Y is our quantitative response variable
    • X is a vector of input variables
    • f is the systematic information that X provides about Y
    • \(\epsilon\) is a random error term (mean 0, independent of X)
  • Overfitting:
    • Follow the errors or noise too closely

Estimating f

Prediction vs Inference:

  • Prediction: Inputs (X) are easy to obtain, outputs (Y) are not
    • We can treat \(\hat{f}\) as a black box, as long as it works, it’s fine
    • \(\hat{Y}=\hat{f}(X)\)
    • Accuracy:
      • Reducible error: \(\hat{f}\) is not perfect, but can be made better
      • Irreducible error: \(\epsilon\)
      • Class exercise 1: come up with reasons why \(\epsilon\) may not be 0 and why \(\hat{f}\) may not perfectly equal f.
  • Inference: What is the association between Y and X?
    • We need to know the exact form of \(\hat{f}\)
    • Identify important predictors
    • What is the relationship between \(X_i\) and Y
    • Can we use a linear equation, or is it more complicated?

\(~\)

Parametric vs Non-Parametric methods:

  • Parametric:
    1. Make an assumption about the functional form of f
      • e.g. f is linear, f(X)=\(\Sigma_{i=0}^n\beta_iX_i\)
    2. Fit the model to the training data
      • e.g. OLS
    • Pro:
      • Easier to fit
    • Con:
      • Unlikely to match f exactly
  • Non-Parametric:
    • No explicit assumptions with respect to f
    • Pro:
      • Can potentially be more accurate to f
    • Con:
      • Significantly more data required

Lab: Data visualization with ggplot 2

Lab: Class work

Packages and Datasets

This lab will use the ggplot2 package:

# install.packages("ggplot2")
library(ggplot2)

It will also use data from The College Scorecard:

colleges <- read.csv("https://remiller1450.github.io/data/Colleges2019.csv")
  • Description: The colleges data set contains attributes and outcomes for all primarily undergraduate institutions in the United States with at least 400 full-time students for the year 2019.

\(~\)

How ggplot2 creates graphics

ggplot2 package builds graphics in a structured manner using layers. Layers are sequentially added to a graph, with each serving a particular purpose, such as:

  1. Displaying raw data
  2. Displaying a statistical summary
  3. Adding metadata (ie: annotations, context, references, etc.)

Consider the following examples:

## Example #1.1 - nothing (just a base layer)
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition))

## Example #1.2 - add a layer displaying raw data
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition)) +
  geom_point()
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

## Example #1.3 - add another layer (smoother)
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition)) +
  geom_point() +
  geom_smooth()
## `geom_smooth()` using method = 'gam' and formula = 'y ~ s(x, bs = "cs")'
## Warning: Removed 57 rows containing non-finite outside the scale range
## (`stat_smooth()`).
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).
## Example #1.4 - add another layer (reference text)
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition)) +
  geom_point() +
  geom_smooth() + 
  annotate("text", x =20000, y=40000, label = "Data from 2019")
## `geom_smooth()` using method = 'gam' and formula = 'y ~ s(x, bs = "cs")'
## Warning: Removed 57 rows containing non-finite outside the scale range
## (`stat_smooth()`).
## Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

Note: A high-quality graphic doesn’t require any particular number of layers. In fact, more layers can sometimes detract from the clarity of a data visualization.

\(~\)

More on the base layer

The mapping and data arguments provided in the base layer are carried forward to all subsequent layers (which is often desirable). However, we can avoid this behavior.

## Example #2.1
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Region)) +
  geom_point()
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).
## Example #2.2 (local override of color aesthetic)
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Region)) +
  geom_point(color = "red")
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

Specifying aesthetics locally within layers:

## Example #3.1
ggplot(data = colleges) +
  geom_point(mapping = aes(x = Cost, y = Net_Tuition)) 
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).
## Example #3.2
ggplot(data = colleges) +
  geom_point(mapping = aes(x = Cost, y = Net_Tuition)) +
  geom_point(mapping = aes(x = Cost, y = Enrollment), color = "red")
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).
## Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

Local specification is most useful when you want to add layers that involve the same parameters (“x” and “y” in the example above) but for different things. Common examples are drawing a line through several different group means of an ordinal categorical variable, or displaying both polygons and points on a map.

\(~\)

Creating Effective Visualizations

The fundamental principles of creating effective data visualizations are quite simple. In short, an effective visualization should:

  1. Clearly convey a particular message
  2. Let the data speak for itself with minimal distortion
  3. Avoid “sales tactics” and unnecessary frills

It’s helpful to understand these principles with a few examples of effective and ineffective visuals:

## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

Example #1

Consider survey data on the popularity of different internet browsers over time:

Which graph is more effective? Why?

\(~\)

Example #2

Consider the heights of men and women in the NHANES sample:

Which graph is more effective? Why?

\(~\)

Lab: Partner work

At this point you will begin working with your partner. Please read through the text/examples and make sure you both understand before attempting to answer the embedded questions.

ggplot graphics are a very expansive topic and not everything can be self-contained in this lab. Throughout the lab, I encourage you to reference the following resources if you need to figure out the proper function or syntax for a particular task:

\(~\)

Terminology

The ggplot framework is unique because all graphics are grammatically defined using the following terminology:

  • Aesthetics (or “aes”) - mappings of variables to visual cues representing their values (ie: position on the x-axis)
  • Geometric elements (or “geom”) - what you actually see in the plot (ie: points, lines, etc.)
  • Scales - guidelines for how aesthetic mappings should be displayed (ie: logarithmic x-axis, red to blue color palate, etc.)
  • Guides (or “legends”) - references to help a human reader interpret the aesthetics
  • Facets - rules specifying how to break up and separately display subsets of data

Question #1: Identify and briefly describe each term mentioned above in the graphic created by the code below.

ggplot(data = colleges, mapping = aes(x = Adm_Rate, color = Private)) + 
  geom_density() +
  scale_x_continuous(trans = "reverse") +
  facet_wrap(~Region)
## Warning: Removed 180 rows containing non-finite outside the scale range
## (`stat_density()`).

Question #2: Create a histogram of the variable “Enrollment” displayed on the log2-scale and faceted by the variable “Private”. Use the ggplot2 cheatsheet to help you identify the necessary functions and arguments.

\(~\)

Themes

Themes are pre-built style templates used to better tailor a graphic to the mode of publication.

The example below applies a black and white theme to Example 2.1 from the preamble.

## Example #2.1 w/ black and white theme
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Region)) +
  geom_point() +
  theme_bw() 
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

Other pre-built themes:

  • theme_bw()
  • theme_linedraw(), theme_light() and theme_dark()
  • theme_minimal()
  • theme_classic()
  • theme_void()

You can judge the differences in these themes below:

Any theme can be further customized using theme(). Most commonly this function is used to remove a graph’s legend:

## Example #2.1 w/ black and white theme
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Region)) +
  geom_point() +
  theme_bw() +
  theme(legend.position = "none")
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

Question #3: The code below creates a line graph that depicts (via a smoothed moving average) the approval of former presidents Jimmy Carter, Ronald Reagan, and Barrack Obama. Modify second portion of this code to try out a few different non-default themes. Then, briefly discuss (1-2 sentences) which themes you feel are most effective and least effective for this type of graph. Include the graph with your preferred theme in your lab write-up.

## Data processing
approval <- read.csv("https://bit.ly/398YR6M")
approval$Week = as.numeric(difftime(as.Date(approval$End.Date, "%m/%d/%y"), as.Date(approval$Inaug.Date, "%m/%d/%y"), units = "weeks"))
approval2 = subset(approval, President %in% c("Reagan", "Carter", "Obama"))

## Creating the graph
ggplot(data = approval2, mapping = aes(x = Week, y = Approving, color = President)) +
 geom_smooth(method = "loess", span = 0.6, se = FALSE)

\(~\)

Labels and Annotations

Labels and annotations are important aspects of well-constructed data visualizations. They are used to provide context, or draw the viewer’s attention towards particular aspects of the graphic.

Labels corresponding to aesthetics (such as x, y, color, etc.) are controlled using the labs() function:

ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Adm_Rate)) +
  geom_point() +
  labs(x = "Sticker Cost", y = "Price Paid", color = "Admissions Rate")
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

Annotations are added the graphic as a layer using the annotate() function:

ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Adm_Rate)) +
  geom_point() + 
  annotate(geom = "rect", xmin = 65000, xmax = 77000, ymin = 20000, ymax = 52000, color = "red", alpha = 0.2)
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

The example above annotates a scatter plot by drawing a red rectangle with 20% transparency (controlled by alpha) around the cluster of colleges with high costs, high net tuition, and low admissions rates.

Question #4: Use the subset() function to create a data frame containing only colleges located in the state of Iowa. Using these data, create a box plot of admissions rates, rename the x-axis label to “Admissions Rate”, and add a text annotation above the outlier saying “Grinnell”.

\(~\)

Scales

Scales map values in the data space to the aesthetic space (ie: where and how should data with Adm_Rate=0.2, Cost=40000, and Net_Tuition=10000 appear on the graphic?).

Scales can be modified by adding layers using functions whose names follow the general format: “scale_aesthetic_function()”. Shown below are a few examples:

## Default Scales
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Adm_Rate)) +
  geom_point()
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).
## Put cost (the "x" aesthetic) on the log2 scale
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Adm_Rate)) +
  geom_point() +
  scale_x_continuous(trans = "log2")
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

## Use a gradient from purple to yellow to display Adm_Rate
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Adm_Rate)) +
  geom_point() +
  scale_color_gradient(low = "purple", high = "yellow")
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).
## Use the popular "viridis" color scale, reversing the default direction via "-1"
ggplot(data = colleges, mapping = aes(x = Cost, y = Net_Tuition, color = Adm_Rate)) +
  geom_point() +
  scale_color_continuous(type = "viridis", direction = -1)
## Warning: Removed 57 rows containing missing values or values outside the scale range
## (`geom_point()`).

There are two things you should note at this point:

  1. There are variants of scale functions depending upon the type of the variable being used by that aesthetic. In these examples, scale_color_continuous() is used because “Adm_Rate” is a continuous numeric variable. However, if colors were being determined by a character string you’d want to use scale_color_brewer() instead (or one of the related color scale functions designed for discrete variables).
  2. The color aesthetic is not synonymous with all of the color you see on the graph. For example, the fill aesthetic can be used to add color to a bar chart, and you must use a function like scale_fill_brewer() to change the fill color.

Question #5: Create a 2-dimensional filled density plot (using geom_density_2d_filled()) with the aesthetics “x = Adm_Rate” and “y = Net_Tuition” using the “viridis” color scale and a “reverse” x-axis that goes from 1.00 to 0.00. Then, write a sentence or two describing the combinations of admissions rate and net tuition that occur most frequently among US colleges.

\(~\)

Stats

Sometimes we’d like to display a statistical transformation (mean, error bars, etc.) alongside the data itself. While this could be accomplished by creating a separate data frame, it’s generally better to use a stat_ function:

ggplot(data = colleges[1:30,], mapping = aes(x = Net_Tuition, y = Private)) +
  geom_point() + 
  stat_summary(fun = mean, geom = "point", color = "red", size = 3) +
  stat_summary(fun.data = mean_se, geom = "errorbar", color = "red")

The example above adds error bars at 1 standard error above and below the mean net tuition cost of for the private and public colleges in the first 30 rows of the colleges data frame.

Notice how the first instance of stat_summary() uses the fun argument (a simple option designed to return a single number/vector), while the second uses the fun.data argument (a complex option designed to return a data frame, which contains the lower and upper endpoints of the interval in this example).

Question #6: Using the data subset containing only colleges located in Iowa (you created this in Question #4), create a graph similar to the example above but using the arguments geom = "linerange", alpha = 0.3, size = 4, and color = "red" to depict 1 standard error above/below the means of private and public colleges.

\(~\)

Facets

Relying on large numbers of aesthetics to include additional variables on a graph can quickly become overwhelming. Facets allow you to display multiple side-by-side graphs according to one or more categorical variables.

  • facet_wrap() is designed to display the data broken by a single categorical variable
  • facet_grid() is designed to display the data broken by all combinations of two categorical variables

Two examples of faceting are shown below:

## create a subset for example purposes
reduced_colleges = subset(colleges, Region %in% c("Great Lakes", "Far West", "Plains"))

## facet_wrap
ggplot(data = reduced_colleges, mapping = aes(x = Cost)) +
  geom_density() +
  facet_wrap(~Region, nrow = 1)
## Warning: Removed 17 rows containing non-finite outside the scale range
## (`stat_density()`).

## facet_grid
ggplot(data = reduced_colleges, mapping = aes(x = Cost)) +
  geom_density() +
  facet_grid(Private~Region)
## Warning: Removed 17 rows containing non-finite outside the scale range
## (`stat_density()`).

The facet_wrap() function will wrap a series of plots according to a fixed number of rows or columns, while facet_grid() will construct a 2-dimensional grid whose panels correspond to the unique combinations of values in the variables used in the grid formula.

\(~\)

Practice:

Question #7 - Part A

The “babynames” package contains a data set documenting the number and frequency of all names that appear at least 5 times within a given year as recorded by the United States Social Security Administration.

The code below will load this dataset. You will likely need to install the package unless you’ve previously used it.

#install.packages("babynames")
library(babynames)
data("babynames")

Create a subset of babynames named my_subset that contains information on the names: "Ryan", "Jeff", "Shonda", "Jonathan", "Collin", "Nathan", "William"

Next, run the ggplot code given below, which seems like it should create a line chart of each name’s frequency by year. What is happening that makes this graph look so horrible? Take a look at the data frame and explain the issue in 1-2 sentences.

ggplot(my_subset, aes(x = year, y = n, color = name)) + geom_line()

\(~\)

Question #7 - Part B

Create a new graph that fixes the problem you identified in Part A and appropriately displays the frequency of each name over time.

Question #3

Consider the following ggplot code:

ggplot(data = mpg) + 
  geom_point(mapping = aes(x = displ, y = hwy, color = "blue"))

Part A: Why are the data-points in the scatter plot not colored blue? Briefly explain.

Part B: Modify the code so that the points are properly shown as blue. \(~\)

Wrap-up

Lab goals:

Hopefully this lab was mostly review. In this lab we reviewed the following:

  • Basics of ggplot2
    • Base layer
    • Themes
    • Using Stats
    • Understanding issues

Course Schedule:

  1. Today: Further Introduction to the course
  2. Next week: Foundations of Statistical Learning

Reminders for next class:

  • Homework 0 is due Friday at 10pm
  • Tuesday’s Lab is due Friday at 10pm
  • This Lab is due Friday at 10pm
  • The reading assignment for Chapter 2.1 of the text is due Monday at 10pm
  • I will likely not be here for class on Tuesday