3 The Tidyverse

Summary

  • A library or package is a collection of functions and datasets. Packages can also be collections of other packages.

  • The package dplyr has functions for manipulating and transforming data.

  • The slice function picks out a subset of observations from the whole dataset.

  • The mutate function creates new variables (columns) as functions of existing ones.

  • The select function picks out variables (columns).

  • The pipe operator |> allows the output of a function to be given as the first input for another function. This makes composition of several functions easy, and makes code much more readable.

  • The tidyverse is a collection of packages in R for doing many commons tasks in data science.

3.1 Packages

So far the commands and data used have been part of base R, which consists of things that are available when the minimal version of R is installed on a computer.

Like many computer languages, R can be extended through the use of libraries. These are also known as packages.

A library or package is a collection of functions and datasets. Packages can also be collections of other packages.

A particular library typically has a theme. For instance, the dplyr library is made for transforming and manipulating data. Before a library can be used, it must be installed on the system. This can be accomplished with the install.packages command. It takes a single argument, the name of the package in quotes. So install.packages("dplyr") will install the dplyr.

Installing a package only needs to be done once for any particular R installation. If you use the install.packages command again, R will check for any updates available for the package and install those.

3.2 Built in datasets

R contains a number of built in datasets to assist in learning how functions work. For instance, consider the cars dataset. Using ?cars reveals that this is a data frame with 50 observations (rows) on 2 variables (columns) from around 1920. The first variable, speed, is the speed of the car in miles per hour, mph. The second variable, dist, is the number of feet the car needed to stop from that speed.

To get a glimpse of the data, the head function can be used to look at the first few observations. This function takes as its first argument any data frame, and by default, it will show the first six observations.

head(cars)
##   speed dist
## 1     4    2
## 2     4   10
## 3     7    4
## 4     7   22
## 5     8   16
## 6     9   10

3.3 The slice function

While head is useful, it always starts with the first observation. To go beyond this, the slice function can be used. This function is part of the dplyr package. What it does is to take some of the observations. For instance, the third, seventh, and eleventh observations in the cars dataset can be found as follows.

dplyr::slice(cars, 3, 7, 11)
##   speed dist
## 1     7    4
## 2    10   18
## 3    11   28

The dplyr:: before the function name indicates that the slice function is part of the dplyr package. Here the :: separates the package name from the function.

The : symbol can also be used to construct sequences. For instance 4:6 translates to the three numbers 4, 5, 6. The : notation for sequences can be used inside of the slice function. The following uses this notation to place the first three observations in their own dataset first_cars.

first_cars <- dplyr::slice(cars, 1:3)
first_cars
##   speed dist
## 1     4    2
## 2     4   10
## 3     7    4

Slice can also be used to remove observations using negative index notation. There the observations that we wish to remove are given negative indices. For instance,

dplyr::slice(first_cars, -2)
##   speed dist
## 1     4    2
## 2     7    4

removes the second observation from the data set first_cars.

3.4 The library function

It gets old pretty quick typing dplyr:: before every function from the package dplyr. The library command can be used to bring all the functions and variables from a package into the environment.

library(dplyr)

You only need to use install.packages once to download and install a library in your local R installation. On the other hand, you need to use library every time you restart R or at the beginning of an R Markdown file to use the functions in a library.

Now that the library has been loaded, slice (or any other function in dplyr) can be used whenever we would like without the dplyr:: prefix.

slice(cars, 2:4)
##   speed dist
## 1     4   10
## 2     7    4
## 3     7   22

3.5 The mutate command

Another useful function in the dplyr package is the mutate function. This allows the creation of a new variable (or overwriting an existing variable) in the dataset as a function of other variables. For instance, the speed variable in cars uses miles per hour, mph. To convert to kilometers per hour, kph, multiply by 1.6. Using mutate to do this works as follows.

cars_kph <- mutate(cars, speed_kph = speed * 1.6)

In the original version of R, the $ command was used to access columns of data. So cars$speed would be used to get the speed. In the more modern version being used here, once cars is set as the data in the first argument to the mutate command, it is understood that a reference to something like speed refers to a column of cars. This can be helpful when dealing with multiple datasets that have the same column name, and tends to lead to cleaner code.

3.6 The select command

The select command in package dplyr can be used to keep some of the columns/variables in the dataset. For instance, to keep speed_kph and dist as the only columns of cars_kph, use

select(cars_kph, speed_kph, dist)
##    speed_kph dist
## 1        6.4    2
## 2        6.4   10
## 3       11.2    4
## 4       11.2   22
## 5       12.8   16
## 6       14.4   10
## 7       16.0   18
## 8       16.0   26
## 9       16.0   34
## 10      17.6   17
## 11      17.6   28
## 12      19.2   14
## 13      19.2   20
## 14      19.2   24
## 15      19.2   28
## 16      20.8   26
## 17      20.8   34
## 18      20.8   34
## 19      20.8   46
## 20      22.4   26
## 21      22.4   36
## 22      22.4   60
## 23      22.4   80
## 24      24.0   20
## 25      24.0   26
## 26      24.0   54
## 27      25.6   32
## 28      25.6   40
## 29      27.2   32
## 30      27.2   40
## 31      27.2   50
## 32      28.8   42
## 33      28.8   56
## 34      28.8   76
## 35      28.8   84
## 36      30.4   36
## 37      30.4   46
## 38      30.4   68
## 39      32.0   32
## 40      32.0   48
## 41      32.0   52
## 42      32.0   56
## 43      32.0   64
## 44      35.2   66
## 45      36.8   54
## 46      38.4   70
## 47      38.4   92
## 48      38.4   93
## 49      38.4  120
## 50      40.0   85

Even here printing 50 rows of data is hard. Many datasets have thousands, millions, or even more rows of data. So often when displaying a dataset, slice is used to keep only the first few rows.

slice(select(cars_kph, speed_kph, dist), 1:5)
##   speed_kph dist
## 1       6.4    2
## 2       6.4   10
## 3      11.2    4
## 4      11.2   22
## 5      12.8   16

The notation here is of nested functions. To figure out what happens, start at the inside and work towards the outside. Nested function can become complicated to understand as the number of nestings grows.

A better way to apply these functions is to think about starting with the cars_kph dataset, then applying select, then applying slice. An object called a pipe can be used to code in this way.

3.7 Pipes

The pipe operator |> allows easy application of more than one function to a dataset. It changes the way the code is written so as to start with our dataset, and then apply one function after another.

A pipe works by moving the first argument to a function to the left of the pipe symbol. For example, consider the following simple function that adds together its two arguments.

add <- function(a, b) return(a + b)
add(3, 8)
## [1] 11

Then with the pipe symbol, the first argument can be moved to the left hand side of the |>. That is:

3 |> add(8)
## [1] 11

At this point it might be hard to see the usefulness of pipes. With one or two functions pipes are not really necessary, but pipes really shine when many functions are being applied one after the other. For instance, consider the following code.

square <- function(a) return(a * a)
square(4)
## [1] 16

Now consider applying several iterations of add and square to some numbers.

x <- 3
add(square(add(x, 3)), -4)
## [1] 32

It is kind of hard to parse what is going on. Now look at the same code written using pipes.

x |>
  add(3) |>
  square() |>
  add(-4)
## [1] 32

Each application of a function gets its own line, which improves readability. Moreover, the operations now occur in order instead of inside-out. First take x, add 3, square the result, then add -4 to that result. For transformations of datasets, this type of notation usually makes things much simpler to read and understand.

Consider the earlier code for selecting two variables and five observations from the cars_kph dataset.

slice(select(cars_kph, speed_kph, dist), 1:5)
##   speed_kph dist
## 1       6.4    2
## 2       6.4   10
## 3      11.2    4
## 4      11.2   22
## 5      12.8   16

With pipes, this becomes:

cars_kph |> select(speed_kph, dist) |> slice(1:5)
##   speed_kph dist
## 1       6.4    2
## 2       6.4   10
## 3      11.2    4
## 4      11.2   22
## 5      12.8   16

It is the same result, but the pipe expression is more easily translated to human language (start with cars_kph, keep the variables speed_kph and dist, keep the first three observations) because it moves left to right instead of inside out.

To assign the result of several pipes to another variable the assignment operator <- can be used. Put the new variable name first, this <-, then all of the piped together commands. (There is also a -> assignment that assigns whatever is left to the name on the right, but it is almost never used in practice and should be avoided.)

For instance, to do the previous changes to cars and store it in a new variable cars2, use the following.

cars2 <-
  cars |>
  mutate(speed_kph = speed * 1.6) |>
  select(-speed)
cars2 |> head()
##   dist speed_kph
## 1    2       6.4
## 2   10       6.4
## 3    4      11.2
## 4   22      11.2
## 5   16      12.8
## 6   10      14.4

Note here in select instead of listing out the variables to keep, a - sign was put in front of the variable to be gotten rid of.

One could make the changes to cars and overwrite the variable cars instead of creating a new variable cars2. This, however, is poor practice, as it could break existing code that used cars expecting the original variable. Especially with small variables that have at most a few million observations, alterations to the data should be stored in a new variable instead of overwriting the old one.

3.8 Tibbles

A table with extra information is called a tibble in the tidyverse.

A tibble is a way of storing a table in the tidyverse, along with extra information such as the data type of each column.

An older form of table storage in R is called the data frame. Datasets such as cars were originally data frames, and functions like slice, mutate, and select can take either a data frame as their first parameter or the more recent tibble data type. All of these functions will output the result as a tibble.

It is because the first input to these functions are tibbles or data frames and the output are tibbles that allow us to use pipes to string together multiple functions easily.

3.9 The summarize command

Now consider the problem of applying a function like mean to a particular variable that is embedded within a tibble. For instance, suppose the goal is to find the mean of the distances in the cars dataset. This goal can be achieved by creating a new tibble that has only a single row and column. The entry would then contain the desired number. The summarize command can be used to do this.

The first parameter for summarize (supplied either directly or through a pipe as a data frame or tibble) is the dataset to be used. The second parameter is a function like mean, max, or min that applies to a vector of numerical data. The result is output as a tibble that applies the statistic to the variable specified.

For example, the following code finds the maximum speed in the cars dataset.

cars |>
  summarize(max_speed = max(speed))
##   max_speed
## 1        25

If two statistics are to be calculated, just list them in the order you want them to appear in the output.

cars |>
  summarize(max_speed = max(speed), avg_dist = mean(dist))
##   max_speed avg_dist
## 1        25    42.98

In general, the summarize command will have output that is a tibble with one row, and a number of columns equal to the number of statistics to be calculated.

3.10 The tidyverse package

Data is said to be tidy when it is in a table where each line contains an observation, and each column contains a variable that is something that can be measured. For instance, in the cars dataset in R, there are 50 observations, which means there are 50 rows. There are two variables (speed and dist,) which means there are two columns.

Variables in a dataset should not be confused with a variable in a programming language. In statistics, a variable is just anything that can be measured, such as height, weight, color, speed, education level, et cetera. This type of variable is also sometimes called a factor.

The tidyverse package is a collection of packages that accomplish the tasks needed in data science. These include the following that will be discussed in the rest of this text.

  • dplyr Transformation and manipulation of data.

  • readr Reading and writing data from a website or hard drive to main memory.

  • ggplot2 Visualization of data.

  • tidyr For putting data into tidy form.

  • stringr Deals with text data (aka strings.)

  • forcats Dealing with categorical data.

  • modelr Modeling data.

  • purrr Replaces loops for better efficiency.

If you use install.packages("tidyverse"), R will install every package in the tidyverse at once. Make sure you have a good Internet connection, put your feet up and relax, that could take a while!

Questions

Consider the following dataset, created using the tibble function.

simple_example <- tibble(
  change = c(-5, 3, 4, -1),
  season = c("Winter", "Summer", "Summer", "Fall")  
)

The resulting table of data is as follows.

simple_example
## # A tibble: 4 × 2
##   change season
##    <dbl> <chr> 
## 1     -5 Winter
## 2      3 Summer
## 3      4 Summer
## 4     -1 Fall

  1. Write code to add a variable (column) abs_change to simple_example that is the absolute value of the change value.

  2. Continuing the last part, write code to add a variable positive which has value TRUE if the value of change is greater than 0, and FALSE to otherwise.

  1. This can be done with

    simple_example |> mutate(abs_change = abs(change))
    ## # A tibble: 4 × 3
##   change season abs_change
##    <dbl> <chr>       <dbl>
## 1     -5 Winter          5
## 2      3 Summer          3
## 3      4 Summer          4
## 4     -1 Fall            1

  2. This can be done with

    simple_example |>
  mutate(positive = (change > 0))
    ## # A tibble: 4 × 3
##   change season positive
##    <dbl> <chr>  <lgl>   
## 1     -5 Winter FALSE   
## 2      3 Summer TRUE    
## 3      4 Summer TRUE    
## 4     -1 Fall   FALSE

Consider the following set of high and low temperature forecasts for Claremont, California during a few days in September 2022.

temps_claremont <- tibble(
  dates = c("2022-09-07",
            "2022-09-08",
            "2022-09-09",
            "2022-09-10"),
  high  = c(105, 102, 104, 77),
  low   = c(75, 77, 77, 71)
)

  1. Write code to only keep from temps_claremont the first and last observation.

  2. Write code to only keep from temps_claremont the high and low temperature data.

  3. The temperatures are given using the Fahrenheit temperature scale. Using the formula \(C = (F - 32)(5 / 9)\) to convert from Fahrenheit to Celsius, add a new variable low_celsius that holds the low forecasts in Celsius.

  1. This can be done with

    temps_claremont |> slice(c(1, 4))
    ## # A tibble: 2 × 3
##   dates       high   low
##   <chr>      <dbl> <dbl>
## 1 2022-09-07   105    75
## 2 2022-09-10    77    71

  2. This can be done with

    temps_claremont |> select(high, low)
    ## # A tibble: 4 × 2
##    high   low
##   <dbl> <dbl>
## 1   105    75
## 2   102    77
## 3   104    77
## 4    77    71

  3. This can be done with

    temps_claremont |> mutate(low_celsius = (low - 32) * (5 / 9))
    ## # A tibble: 4 × 4
##   dates       high   low low_celsius
##   <chr>      <dbl> <dbl>       <dbl>
## 1 2022-09-07   105    75        23.9
## 2 2022-09-08   102    77        25  
## 3 2022-09-09   104    77        25  
## 4 2022-09-10    77    71        21.7

Consider the mpg dataset which is in the ggplot2 package, which can be loaded with the following code.

library(ggplot2)

Currently, engine displacement in mpg is measured in liters. Convert this to cubic centimeters with the mutate command.

The median command in R calculates the sample median of a dataset. This is the middle value in a vector of values if the length of the vector is odd, and the arithmetic average of the two middle values in a vector of values if the length of the vector is even.

For instance,

median(c(3, 7, 17))
## [1] 7

and

median(c(3, 7, 10, 17))
## [1] 8.5

illustrates this sample median.

Use this command together with summarize to find the sample median of the mpg variable in the mtcars dataset built into R.

Use the summarize command to create a tibble that contains the average mpg, the median mpg, and the average of the wt variable that measures the weight of the vehicle in thousands of pounds.

Given a vector that consists of boolean values, TRUE and FALSE, when you use sum, every TRUE gets turned into a 1, and every FALSE into a 0.

The rest of this problem uses

x <- c(TRUE, TRUE, FALSE, FALSE, TRUE, FALSE, TRUE, TRUE, TRUE, FALSE)

  1. Try applying sum to this vector x.

  2. Try applying mean to this vector x.

  3. Try applying max to this vector x.

  4. Try applying min to this vector x.

Consider the variable flights in the package nycflights13. When arr_delay is zero or negative, say that a particular flight is on time.

  1. Use mutate to add a new boolean on_time that is true if a flight is one time and false otherwise.

  2. What percentage of the flights were on time?

The data set uspop is a time series data type that holds the results of the United States Census from 1790 to 1970. You can convert it to a tibble using the tibble function.

us_census <- tibble(uspop)

  1. Add to the tibble a year variable that runs from 1790 to 1970 skipping by ten years.

  2. Add to the tibble from part (a) another variable log_pop that shows the natural logarithm of the population.