12 Strings and regular expressions

Summary

  • Regular expressions (aka regex) are used to search for patterns of symbols within a list of symbols (a string).

  • str_detect returns TRUE if the pattern appears in the text, and FALSE if it does not.

  • An anchor in a regex limits the pattern matching to occur either at the beginning or end of a string.

  • The ^ anchor changes the pattern so that it only occurs at the beginning of the string.

  • The $ anchor changes the pattern so that it only occurs at the end of the string.

  • The . symbol is a wildcard. When it appears in a regex it matches any single character in a string.

  • Bracket notation is a way to create a pattern that matches more than one symbol in a string. Hyphens can designate a range of possibilities, for instance, [a-d] matches either a, b, c, or d.

  • str_extract returns the first part of the string that matches the regex pattern if it appears in the string, and NA if it does not.

  • str_extract_all returns every part of the string that matches the regex pattern if it appears in the string in a list. If the pattern does not appear, an empty character vector is returned.

  • If ^ appears as the first character inside of brackets, then it acts as a negation operator. This negated bracket pattern then matches everything that does not match the pattern in the rest of the bracketed expression. For instance, [1-5] matches the digits 1 through 5, while [^1-5] matches any symbol that is not the digits 1 through 5.

  • The + operator changes the pattern to match the text in the string if it appears one or more times.

  • The \(\ast\) operator (called the Kleene star) changes the pattern to match the text in the string if it appears zero or more times.

  • The ? operator changes the pattern to match the text in the string if it appears zero or one time.

  • The {n} operator repeats the last part of the pattern \(n\) times exactly.

12.1 Dealing with textual data

Often, variables in datasets take on textual values. For instance, consider the following snippet of data from the Global Historical Climatology Network–Daily (GHCN-Daily) dataset.

ghcn1 <- tribble(
  ~key, ~lat, ~lon, ~un1, ~station, ~type, ~gsnnum,
  "ACW00011604", 17.1167,  -61.7833,   10.1,    "ST JOHNS COOLIDGE FLD", NA, NA,    
  "ACW00011647", 17.1333,  -61.7833,   19.2,    "ST JOHNS", NA, NA,                  
  "AE000041196", 25.3330,   55.5170,   34.0,    "SHARJAH INTER. AIRP", "GSN",     41196,
  "AF000040930", 35.3170,   69.0170, 3366.0,    "NORTH-SALANG",        "GSN",     40930,
  "AG000060390", 36.7167,    3.2500,   24.0,    "ALGER-DAR EL BEIDA",  "GSN",     60390,
  "AG000060590", 30.5667,    2.8667,  397.0,    "EL-GOLEA",            "GSN",     60590,
  "AG000060611", 28.0500,    9.6331,  561.0,    "IN-AMENAS",           "GSN",     60611,
  "AG000060680", 22.8000,    5.4331, 1362.0,    "TAMANRASSET",         "GSN",     60680
)

Note that some of the stations are given the type value of GSN. This indicates that the station is part of the GCOS Surface Network. To search for only those observations with type equal to GSN, the filter could be used.

ghcn1 |> filter(type == "GSN")
## # A tibble: 6 × 7
##   key           lat   lon   un1 station           type  gsnnum
##   <chr>       <dbl> <dbl> <dbl> <chr>             <chr>  <dbl>
## 1 AE000041196  25.3 55.5     34 SHARJAH INTER. A… GSN    41196
## 2 AF000040930  35.3 69.0   3366 NORTH-SALANG      GSN    40930
## 3 AG000060390  36.7  3.25    24 ALGER-DAR EL BEI… GSN    60390
## 4 AG000060590  30.6  2.87   397 EL-GOLEA          GSN    60590
## 5 AG000060611  28.0  9.63   561 IN-AMENAS         GSN    60611
## 6 AG000060680  22.8  5.43  1362 TAMANRASSET       GSN    60680

Now suppose that the goal is to simply find a string within the data, rather than have it be the exact string. For instance, there are two stations with ST JOHNS in the name. But only one will be exactly equal to ST JOHNS.

ghcn1 |> filter(station == "ST JOHNS")
## # A tibble: 1 × 7
##   key           lat   lon   un1 station  type  gsnnum
##   <chr>       <dbl> <dbl> <dbl> <chr>    <chr>  <dbl>
## 1 ACW00011647  17.1 -61.8  19.2 ST JOHNS <NA>      NA

To find all station names that contain ST JOHNS, a regular expression will be used.

A regular expression is an encoding of a pattern to be searched for within a string of text. These are also called regex for short.

12.2 The stringr package

Regular expressions are implemented in R using the stringr package, which is loaded as part of the larger tidyverse package. This package contains several commands allowing the user to test for patterns described by regular expressions against data very efficiently.

For instance, suppose that the goal is just to find any string that contains ST JOHNS. Then the str_detect function can be used with the input "ST JOHNS" to detect if these letters appear anywhere in the string. The format is slightly different than when using ==. The first argument to str_detect is the vector of string values, and the second argument is the regex.

ghcn1 |> filter(str_detect(station, "ST JOHNS"))
## # A tibble: 2 × 7
##   key           lat   lon   un1 station           type  gsnnum
##   <chr>       <dbl> <dbl> <dbl> <chr>             <chr>  <dbl>
## 1 ACW00011604  17.1 -61.8  10.1 ST JOHNS COOLIDG… <NA>      NA
## 2 ACW00011647  17.1 -61.8  19.2 ST JOHNS          <NA>      NA

Recall that filter requires an argument that is a vector of TRUE/FALSE values, one for each observation. So what str_detect returns is exactly such a vector.

example_string1 <- c(
  "ST JOHNS",
  "ST JOHNS COOLIDGE FLD",
  "VAST JOHNS",
  "PITTSBURGH",
  "SV JOHNS AREA 51",
  "ST. JOHNS",
  "ST JOHNS AND ST JOHNS"
)
example_string1 |> str_detect("ST JOHNS")
## [1]  TRUE  TRUE  TRUE FALSE FALSE FALSE  TRUE

The first three entries and the last entry are TRUE because the string "ST JOHNS" appears somewhere within the string. Note that the regular expression still finds "ST JOHNS" in "VAST JOHNS", since it does not care about word breaks. The last entry is true because it is looking for the pattern appearing at least once. The fact that it appears twice is immaterial.

12.3 Anchors

What if the goal was only to find the characters "ST JOHNS" at the beginning of the string? Then the regex could be modified by putting an anchor that looks for the pattern either at the beginning or end of the string. In particular, the circumflex symbol ^ will only match the pattern if it occurs at the start of the string.

example_string1 |> str_detect("^ST JOHNS")
## [1]  TRUE  TRUE FALSE FALSE FALSE FALSE  TRUE

What if the goal was to find these characters at the end of the string? Then put the dollar sign symbol $ at the end of the regex.

example_string1 |> str_detect("ST JOHNS$")
## [1]  TRUE FALSE  TRUE FALSE FALSE FALSE  TRUE

If the goal was to find it at the beginning and at the end of the string, use both the ^ and $ symbols.

example_string1 |> str_detect("^ST JOHNS$")
## [1]  TRUE FALSE FALSE FALSE FALSE FALSE FALSE

This formulation of ^string$ is equivalent to searching if the text exactly equals the string in question.

But wait a minute, doesn’t ST JOHNS AND ST JOHNS both begin and end with ST JOHNS. So why is it detected as FALSE?

The answer lies in what regular expressions look for. They are looking for a pattern that satisfies all the constraints laid out in the regex. That means that it is looking for a set of contiguous characters in the string with three properties.

  1. The ^ means that the set of contiguous characters must occur at the beginning of the string.

  2. The ST JOHNS means that the set of contiguous characters must be of length eight, and match the characters exactly.

  3. The $ means that the set of contiguous characters must be at the end of the string.

For the regex to return TRUE, all three of these properties must hold for the same set of characters. You cannot use one part of the string to match some of the properties and another part of the string to match the other properties. All properties must be satisfied by the same set of characters within the string.

12.4 Wildcards

A wildcard character in a regex is a stand in that can match several possible values of a single character.

For instance, a wildcard could match the digits 0 through 9. The most general wildcard matches any single character. In R, the wildcard for matching any symbol is the period, ..

Recall that the fifth entry in example_string1 is SV JOHNS AREA 51, and so does not match ST JOHNS.

example_string1 |> str_detect("ST JOHNS")
## [1]  TRUE  TRUE  TRUE FALSE FALSE FALSE  TRUE

Now suppose that the user does not care about the second character. By replacing it with the wildcard symbol ., it can also match the V in SV JOHNS.

example_string1 |> str_detect("S. JOHNS")
## [1]  TRUE  TRUE  TRUE FALSE  TRUE FALSE  TRUE

12.5 Brackets

What if the goal is to see if the string contains any digits 0, 1, 2, up to 9? Then a pattern for digits can be formed by using the bracket construction [0-9]. This will accept any of the values 0 through 9.

example_string1 |> str_detect("[0-9]")
## [1] FALSE FALSE FALSE FALSE  TRUE FALSE FALSE

Here SV JOHNS AREA 51 is the only one with a digit.

This type of pattern can also be used with letters. The following finds any string that uses the letters A through C.

example_string1 |> str_detect("[A-C]")
## [1] FALSE  TRUE  TRUE  TRUE  TRUE FALSE  TRUE

Note this captures PITTSBURGH because it has a B in it. Here it is important to specify uppercase or lowercase letters. None of the strings have any lowercase letters, so searching for [a-c] returns all FALSE values.

example_string1 |> str_detect("[a-c]")
## [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE

What if the goal was to match either a digit 0 through 9, or the letter B? Then both could be put into the bracket.

example_string1 |> str_detect("[B0-9]")
## [1] FALSE FALSE FALSE  TRUE  TRUE FALSE FALSE

12.6 String extraction

So far str_detect has been used to generate just TRUE or FALSE. This is enough to use to filter, but sometimes the match itself is needed. This is the job of str_extract, which returns the part of the pattern that matches the regular expression.

example_string1 |> str_extract("[B0-9]")
## [1] NA  NA  NA  "B" "5" NA  NA
example_string1 |> str_extract("ST JOHNS")
## [1] "ST JOHNS" "ST JOHNS" "ST JOHNS" NA         NA         NA         "ST JOHNS"

Note that even though the last string ST JOHNS AND ST JOHNS contains the last string twice, it only shows up once in the report. This is because once it finds the string, str_extract stops looking at the string. To find all the occurrences of a pattern in a string, use str_extract_all.

example_string1 |> str_extract_all("ST JOHNS")
## [[1]]
## [1] "ST JOHNS"
## 
## [[2]]
## [1] "ST JOHNS"
## 
## [[3]]
## [1] "ST JOHNS"
## 
## [[4]]
## character(0)
## 
## [[5]]
## character(0)
## 
## [[6]]
## character(0)
## 
## [[7]]
## [1] "ST JOHNS" "ST JOHNS"

The output is a list, with the elements of the list being the multiple times that the pattern was found.

12.7 Replacing parts of a string

Earlier, it was seen that it is sometimes necessary to replace strings matched in a pattern by other strings. The str_replace function can be used to do this. Consider conserving vowels by replacing the pattern JOHNS found by JHNS. This can be done in our example strings as follows.

example_string1 |> str_replace("JOHNS", "JHNS")
## [1] "ST JHNS"              "ST JHNS COOLIDGE FLD" "VAST JHNS"           
## [4] "PITTSBURGH"           "SV JHNS AREA 51"      "ST. JHNS"            
## [7] "ST JHNS AND ST JOHNS"

In strings where the pattern is not found (like PITTSBURGH) no replacement was done. However, in the last entry only the first JOHNS was changed, and not the second. Once again, adding _all to the command gives the function str_replace_all that accomplishes both tasks.

example_string1 |> str_replace_all("JOHNS", "JHNS")
## [1] "ST JHNS"              "ST JHNS COOLIDGE FLD" "VAST JHNS"           
## [4] "PITTSBURGH"           "SV JHNS AREA 51"      "ST. JHNS"            
## [7] "ST JHNS AND ST JHNS"

Now every occurrence has changed.

12.8 Repeating patterns

If the goal is just to detect one or more digits in a string, then str_detect(string, "[0-9]") works fine. However, when str_extract is being used, often the goal is to match a consecutive string of numbers. The Kleene star can be used to do this. Stephan Kleene was a mathematician and early theoretical computer scientist who wanted a way to extend the abilities of regular expressions.

The Kleene Star, *, in a regular expression indicates that the pattern that appears previously occurs 0, 1, 2, or more times in a row.

The special character * matches zero or more instances of a character. For instance,

str_extract(c("onc", "once", "onced", "onceed", "cece"), "ce")
## [1] NA   "ce" "ce" "ce" "ce"

This finds the "ce" pattern in the last four strings. To match "c" followed by 0 or more "e" characters, use

str_extract(c("onc", "once", "oncee", "cece"), "ce*")
## [1] "c"   "ce"  "cee" "ce"

Note the * only allows zero or more of the previous character, so "c" still always must appear exactly once followed by zero or more appearances of "e".

To get one or more matches of a character, use "+".

str_extract(c("onc", "once", "oncee", "cece"), "ce+")
## [1] NA    "ce"  "cee" "ce"

To see this in a practical situation, consider the following table. This data came from a map of Epcot at Walt Disney World for May 2022.

epcot_map <- c("Spaceship Earth1",
               "Creations Shop9",
               "Living with the Land18",
               "Turtle Talk With Crush20")

Unfortunately, there is no comma separating the first part of the name from the number attached to the key at the end. To make matters more difficult, the number of digits at the end is different for different observations. The following pulls off the digits at the end

epcot_map |> str_extract("[0-9]+")
## [1] "1"  "9"  "18" "20"

The following builds a tibble with the number codes.

epcot_map_tibble <- 
  tibble(
    number = str_extract(epcot_map, "[0-9]+")
  )
epcot_map_tibble |> kable() |> kable_styling()
number
1
9
18
20

12.8.1 Negation in brackets

There is still the problem of how to extract the information other than the digits. Here, the negation operator inside brackets can be used.

That is where things get complicated. Earlier, it was shown that ^ in a regex expression makes what comes next only match the beginning of the expression. However, when ^ appears as the first character inside a bracketed expression, the pattern matches anything except what is tested for in the brackets. So [^0-9] matches anything except the digits 0 through 9. This can be used to pull out the rest of the information.

The following builds a tibble with the place name separated from the number code.

epcot_map_tibble <- 
  tibble(
    place = str_extract(epcot_map, "[^0-9]+"),
    number = str_extract(epcot_map, "[0-9]+")
  )
epcot_map_tibble |> kable() |> kable_styling()
place number
Spaceship Earth 1
Creations Shop 9
Living with the Land 18
Turtle Talk With Crush 20

12.8.2 The zero-one repeating variant

In the same vein, the character ? checks if the pattern appears either zero or one times. This can be useful when a character is optional.

12.9 Repeating a pattern a specific number of times

Sometimes (as with phone numbers), a very specific number of digits need to appear. In the U.S., a phone number (including the area code) has 10 digits. Using [0-9]{10} will match ten digits exactly.

phone_numbers <-
  c("9095553472",
    "909555",
    "909 555 3472",
    "(909)555 3472",
    "(909 555 3472",
    "909-555-3472")

phone_numbers |> str_extract("[0-9]{10}")
## [1] "9095553472" NA           NA           NA           NA          
## [6] NA

12.10 Alternations

The vertical bar | is the alternation symbol in regular expressions, and acts like a logical OR. For instance, cat|dog detects either the string "cat" or the string "dog".

str_detect(c("cat person", "dog person", "tiger person"),
           "cat|dog")
## [1]  TRUE  TRUE FALSE

Returning to the phone example from earlier putting in extra spaces, parentheses, or hyphens will mess up the extraction. Suppose that the area code should be followed by either a hyphen - or a space . Then this might look like

phone_numbers |> str_extract("[0-9]{3}( |-)[0-9]{3}( |-)[0-9]{4}")
## [1] NA             NA             "909 555 3472" NA             "909 555 3472"
## [6] "909-555-3472"

Here the ( |-) searches for either a space or a hyphen. Unfortunately, it is now demanding that there be a space or hyphen. To make it optional, use the ? operator.

phone_numbers |> str_extract("[0-9]{3}( |-)?[0-9]{3}( |-)?[0-9]{4}")
## [1] "9095553472"   NA             "909 555 3472" NA             "909 555 3472"
## [6] "909-555-3472"

This is close to working, but still has problems with parentheses around the area code. To handle this, it will be necessary to be able to match characters such as ( and ) that are special characters in the regex. How to do this will be covered in the next chapter.

Questions

Consider the regular expression 34.[a-g]. Which of the following strings does this pattern match?

  1. "345b"

  2. "34b5"

  3. "435b"

  4. "34gb"

Write down a regular expression that matches a digit, followed by any character, followed by the letter a, followed by any lowercase letter.

Consider the mtcars dataset. The row_names_to_column function can be used to make the row names the first column of data.

mtcars |>
  rownames_to_column("Model_name") |>
  head()
##          Model_name  mpg cyl disp  hp drat    wt  qsec vs am gear carb
## 1         Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## 2     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## 3        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## 4    Hornet 4 Drive 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## 5 Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## 6           Valiant 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

  1. Using str_detect and filter, find all the observations with car names that start with "Merc".

  2. Using str_detect and filter, find all the observations with car names that have at least one digit in them.

Consider the USArrests dataset.

  1. Use str_detect and filter to find all the observations from states that begin with the letter A.

  2. Use str_detect and filter to find all the observations from states that end with the letter a.

  3. Use str_detect and filter to find all the observations from states whose name begins with A and ends with a.

Consider the following code.

test <- c("Uranium Fever", "New Moon", "Ain't that a kick in the head")
str_extract(test, "[a-zA-Z]+$")
## [1] "Fever" "Moon"  "head"
str_extract(test, "^[a-zA-Z]+")
## [1] "Uranium" "New"     "Ain"
str_extract(test, "[^a-zA-Z]+")
## [1] " " " " "'"

  1. Describe in words what the regular expression [a-zA-Z]+$ is trying to find.

  2. Describe in words what the regular expression ^[a-zA-Z]+ is trying to find.

  3. Describe in words what the regular expression [^a-zA-Z]+ is trying to find.

The 18th century art critic Roger de Piles graded 54 classical painters in various abilities on a scale from 0 to 20. This dataset is as follows:

Then the dataset looks as follows.

painters_dePiles |> head()
## # A tibble: 6 × 6
##   Painter       Composition Drawing Colour Expression School
##   <chr>               <int>   <int>  <int>      <int> <fct> 
## 1 Da Udine               10       8     16          3 A     
## 2 Da Vinci               15      16      4         14 A     
## 3 Del Piombo              8      13     16          7 A     
## 4 Del Sarto              12      16      9          8 A     
## 5 Fr. Penni               0      15      8          0 A     
## 6 Guilio Romano          15      16      4         14 A

  1. Find the observations where the painter name has at least one space in it.

  2. Using your tibble from part a., create a column after_space which lists the part of the painters name after the last space in the name.

Suppose you have a vector of strings in s.

  1. Write code to detect for each string in the vector if it contains the letter a followed by one or more digits.

  2. Write code that matches the pattern given in part a, but instead of returning true or false, returns the characters that match the pattern if it exists in the string.

Consider a tibble:

phone_numbers <-
  tribble(
  ~Name, ~Phone,
  "Mr. Plow", "636-555-3226",
  "Big Mean Carla", "323-555-0129",
  "Elmo", "212-555-6666",
  "Phoenix Biogenics", "225-330-7040"
)

The area code for a U.S. phone number consists of the first three digits.

  1. Write code to add a column area_code to the tibble which consists of the first three digits of the phone number.

  2. Write code to filter phone_numbers to only include numbers where the middle three digits are 555.

  1. Give a regular expression that matches one or more digits at the beginning of a string.

  2. Give a regular expression that matches zero or more digits at the end of a string.

Consider the following tibble.

data <- 
  tibble(
    s = c("blue141car", "red314159truck", "yellow2718airplane")
  )

Using the separate command, use a regular expression for the sep parameter to separate the s variable into before_number and after_number.