13 Backslashes

Summary

  • Escape characters are special characters that are usually represented by a backslash in front of another character. Examples include \( for an escaped left parenthesis, \. for an escaped period, and \\ for an escaped backslash.

  • Escape characters are used in strings and in regular expressions.

  • Escape characters for strings include "\n" for newline, "\t" for tab, and "\\" for backslash itself.

  • Escape characters for regular expressions include \(, \), \[, \], \^, \$, \*, \+, \., \?, \{, \}, \|, and \\ because all these (unescaped) characters have special meanings within the regex.

  • The regex function can be used to explicitly turn a string into a regex, although R applies this function automatically in most places it is needed. Note that to get something like \( in a regex, you need to use the string "\\(" because the escape character in the string for a backslash is "\\".

  • Searching a string for a pattern contained in a simple regex takes time linear in the length of the string.

  • A backreference allows a regex to remember what actual string matched a pattern with wildcards. Use of backreferences can make the running time polynomial or exponential in unusual cases.

  • Another way to present a pattern is to use a glob. The glob2rx function turns a pattern given by a glob into a pattern given by a regex.

13.1 Searching for special characters

The . is a wildcard character that matches any symbol in a string (except a newline).

c("AA", "AB", "AC", "BC") |>
  str_extract("A.")
## [1] "AA" "AB" "AC" NA

Of course, this raises the question, how would you search for an actual period, . inside a string? How to match the string ST. LOUIS exactly?

13.2 Using the backslash

In order to match ., or ^, or $, or any of the other special characters in regex expressions, it is necessary to use escape characters.

Escape characters are symbols that match different symbols by adding extra indicator characters. Usually they are of the form \<character>, where \ is the backslash.

For instance, inside of a regex, \. matches against an actual period ".", rather than acting like a wildcard.

13.3 The difference between a string and a regex

Okay, so if the regex ST\. LOUIS were to be created, then it would match the string "ST. LOUIS". But how exactly is a regex created? Here it is important to talk a bit about how a regex can be constructed in R (and many other languages.)

There is an explicit function regex that converts a string to a regex.

regex("test")
## [1] "test"
## attr(,"options")
## attr(,"options")$case_insensitive
## [1] FALSE
## 
## attr(,"options")$comments
## [1] FALSE
## 
## attr(,"options")$dotall
## [1] FALSE
## 
## attr(,"options")$multiline
## [1] FALSE
## 
## attr(,"class")
## [1] "stringr_regex"   "stringr_pattern" "character"

Note that the final output gives the class as regex, pattern, and character. So this is definitely not a simple string!

Here is the thing, in order to keep things simple, R will normally apply the regex function automatically when it is needed. Consider the following.

"trying" |> str_extract("ing")
## [1] "ing"

The function str_extract needs a pattern, a regex as an argument. But it was given a string. So what R does is automatically apply the regex function to the string in order to get a pattern.

"trying" |> str_extract(regex("ing"))
## [1] "ing"

Exactly the same!

13.4 Creating a backslash inside of strings

Why is this important if R does it automatically? Well, it helps explain what comes next!

Suppose the following command is typed into R.

"a\.b"
## Error: '\.' is an unrecognized escape in character string (<input>:1:4)

This gives an error. R tells the user that \. is an unrecognized escape character. You cannot put this escape character into a string! So how can the user get it into a regex?

What is needed is a way to just pass a backslash as a string. The answer is to use the escape character for backslash, which is \\!

"a\\.b"
## [1] "a\\.b"

Now the string holds the escape character for backslash. It will be interpreted correctly when the regex function is applied.

c("a.b", "ab") |>
  str_extract("a\\.b")
## [1] "a.b" NA

13.5 Escape characters in regex

The characters ()[]^$*+?.{}|\ all have special meanings within a regex. So they all have escape characters, \(, \), \[, \], \^, \$, \*, \+, \?, \., \{, \}, \|, and \\.

And remember, when writing these within strings, every backslash \ in the regex is replaced by two backslashes "\\" in the string. So \$ in the regex becomes "\\$" in the string.

This leads to an interesting conclusion: to get the regex pattern to match a backslash, the regex is \\, making the string "\\\\". Wow!

a <- c("\\", "\\.", "abc")
print(a)
## [1] "\\"  "\\." "abc"
a |> str_extract("\\\\")
## [1] "\\" "\\" NA

13.5.1 Generating parentheses

Consider the phone number problem from earlier.

phone_numbers <-
  c("9095553472",
    "909555",
    "909 555 3472",
    "(909)555 3472",
    "(909 555 3472",
    "909-555-3472")

To allow for the area code to be surrounded by parentheses, the escape character for ( and ) can be used.

phone_numbers |> str_extract("\\([0-9]{3}\\)")
## [1] NA      NA      NA      "(909)" NA      NA

Good, but the goal here is not to require that the area code be in parentheses, but make it optional.

phone_numbers |> str_extract("(\\([0-9]{3}\\))|([0-9]{3})")
## [1] "909"   "909"   "909"   "(909)" "909"   "909"

Now add the rest of the number.

13.6 The speed of regex

At this point it might be good to question, why are regular expressions so widely used? After all, they go against the basic principles of clear, readable code. Instead, a regex encodes a program in a way that is almost impenetrable to someone who has not studied their special characters.

The notation for regular expressions was created at a time when computers were much slower than today. The compactness of the notation was a feature. Input for computers was introduced either by direct wiring or a bit later through paper card input. In these situations, keeping program sizes as small as possible is very beneficial.

Another reason to use a regex is that pattern matching with them is very fast. Not just fast for the problem, but as fast as is possible asymptotically in the worst case to search a string. Computer scientists say that a pure regex solves the pattern matching problem in linear time. For a fixed length regex, the time needed to see if a pattern written as a regex falls in the string is roughly proportional to the number of characters in the string. Since every character in the string could be needed to be read in order to do the search, this is the fastest possible.

To see why, consider a search for the regex pattern a.b in the string "safcadbr". The computer starts with the first character in the regex, a, and compares it to the first character in the string "s". They are different. Therefore, the computer starts from the beginning of the regex, and moves to the next character of the string, making that the current string character.

Next the computer looks at the first character in the regex, a, and compares it to the current character of the string "a". They are the same! Hence the computer moves to the next character in the regex, and the next character in the string.

That means the computer will compare the character in the regex . to the character in the string "f". They match (because the regex character . matches everything). Hence the computer moves to the next character of the string, and the next character of the regex.

Oh no, the next character of the regex is b, but the next character of the string is "c". They do not match. So the computer moves back to the beginning character of the regex, and back to the character which follows a in the string, which is f.

The whole search of the string by the regex is contained in the following table.

Position regex Position string Char regex Char String Result
1 1 a s No match
1 2 a a Match
2 3 . f Match
3 4 b c No match
1 3 a f No match
1 4 a c No match
1 5 a a Match
2 6 . d Match
3 7 b b Match/End search successfully

Note that as the pattern was searched for the string, each position of the string is only read for at most a number of times equal to the length of the regex. And that is why using a regex to search for a pattern in a string is so fast, and why they are still used today in data science. Languages such as Go and Rust employ this method to achieve linear time.

It should be mentioned, however, that many computer implementations (including Python, C++, Java, and stringr in R) do not just use the basic theoretical regex in their approach. Rather, they employ a method called backtracking that is often fast on commonly used regexes. Unfortunately, under certain pathological cases, this can lead to polynomial or even exponential running times. So be aware of how the particular language you are using handles regex. Is speed more important? Or is the linear time guarantee better?

13.7 Backreferences

When the wildcard character . is used, it matches any symbol. But which symbol exactly did it match? In standard regex, this information is gone for good once the wildcard is used.

The idea of a back reference (also sometimes written without the space as backreference) allows the pattern to be used again.

For instance, suppose a string has the form of a first string, then a space, then a second string.

example_strings <- c("abc abc", "ab ab", "ca ad")

In two of these strings, the first part before the space is the same as the second part after the space. But in the third string they are different. Suppose the goal is to detect when this occurs.

Using the tools presented so far, the character before the space can be pulled out.

str_extract(example_strings, "^[^ ]+")
## [1] "abc" "ab"  "ca"

Similarly, the $ can be used to anchor the pattern to the end of the strings.

str_extract(example_strings, "[^ ]+$")
## [1] "abc" "ab"  "ad"

Finally, simple comparison can check if the two strings are the same.

str_extract(example_strings, "^[^ ]+") == str_extract(example_strings, "[^ ]+$")
## [1]  TRUE  TRUE FALSE

This worked, but it is pretty clunky. Also, it does not really generalize very well.

A better way is to construct a backreference. To use a backreference, first create a capture group by putting part of the regex in parentheses.

A capture group is a part of a regex that is in parentheses.

For instance, ([^ ]+) is a capture group that looks for one or more characters that are not a space. If this regex is run on "abc 123", then it captures "abc". A backreference then allows you to retrieve the "abc" that was captured later.

A backreference refers back to the part of the string that matched a capture group.

In most languages, backreferences are numbered by capture group. The first capture group gets assigned \1, the second \2, and so on. Consider the following.

str_extract(example_strings, "^([^ ]+) \\1")
## [1] "abc abc" "ab ab"   NA

In words, the regex ^([^ ]+) \1 means that starting at the beginning of the search string, look for characters until a space is found. Record the characters that were searched until the space is found. Then look for a space. Then search to see if the exact same characters were found in the string again.

Now for the downside: even with languages that handle basic regex in linear time, the use of backreferences can make the running time grow to polynomial or exponential time. So use carefully!

13.8 Regular expressions and separate

For the separate command the separation parameter sep can be assigned a regular expression. Consider the following data.

tibble(
  name = c("Trial - by : fire", "A - dozen : eggs")
)
## # A tibble: 2 × 1
##   name             
##   <chr>            
## 1 Trial - by : fire
## 2 A - dozen : eggs

To break apart these entries into three words, the regular expression " - | : " could be used. The expression to the left of the bar matches the first hyphen surrounded by spaces used to separate the first and second columns. The expression to the right matches the colon surrounded by spaces that separates the second and third columns. The result is as follows.

tibble(
  name = c("Trial - by : fire", "A - dozen : eggs")
) |>
  separate(name, into = c("word1", "word2", "word3"), sep = " - | : ")
## # A tibble: 2 × 3
##   word1 word2 word3
##   <chr> <chr> <chr>
## 1 Trial by    fire 
## 2 A     dozen eggs

13.8.1 Example: tidying the U.S. Senate List

The U.S. Senate website maintains a list of senators. Today they do try to keep the data tidy, but that was not always the case. In 2019, the website had the data organized as in the file senate116.txt (retrieved from https://www.senate.gov/general/contact_information/senators_cfm.cfm on 2019-03-05). This dataset contains the name, party, state, class, address, phone, and contact URL for all 100 members of the 116th Congress, but it is not tidy data. Time to transform!

First read in the data.

senate_raw_data <- 
  read_delim("senate116.txt", delim = "\n", skip = 1, col_names = FALSE)
## Rows: 400 Columns: 1
## ── Column specification ────────────────────────────────────────────────────────────────
## Delimiter: "\001"
## chr (1): X1
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

A quick look at the data reveals that the column is a pattern of four repeating rows. In order, these contain what will be referred to as "stuff", "address", "phone", "contact". To break these into separate columns, mutate can be used to add a column with the proper names, then pivot_wider can move the rows into the proper position. There are 100 senators, so this pattern repeats itself 100 times. The rep command can be used to repeat this pattern for every row in the table.

The rep() command in R will repeat a vector for a specified number of times. For instance,

rep(c("a", "b"), 4)
## [1] "a" "b" "a" "b" "a" "b" "a" "b"

Or you can set how many times to repeat by giving the target length of the vector with the parameter length.out.

rep(c("a", "b"), length.out = 6)
## [1] "a" "b" "a" "b" "a" "b"

It is also useful to have an ID number for the senators, which will just be added (like the data is presented) in alphabetical order. To accomplish this, the integer division operator %/% is useful. The value of a %/% b is the number of times the integer b fits into a. For instance, 64 divided by 10 is 6 with a remainder of 4, so

64 %/% 10
## [1] 6

gives 6.

Use this to add a variable type to each row that runs through "stuff", "address", "phone", and "contact". Recall that n() returns the number of rows in the table. Note that the code is written so that the number of senators is not necessary to know.

senate_raw_data |>
  mutate(type = rep(c("stuff", "address", "phone", "contact"),  length.out = n())) |>
  mutate(id = (row_number() - 1) %/% 4 + 1, .before = 1)
## # A tibble: 400 × 3
##       id X1                                              type 
##    <dbl> <chr>                                           <chr>
##  1     1 Alexander, Lamar - (R - TN) Class II            stuff
##  2     1 455 Dirksen Senate Office Building Washington … addr…
##  3     1 (202) 224-4944                                  phone
##  4     1 Contact: www.alexander.senate.gov/public/index… cont…
##  5     2 Baldwin, Tammy - (D - WI) Class I               stuff
##  6     2 709 Hart Senate Office Building Washington DC … addr…
##  7     2 (202) 224-5653                                  phone
##  8     2 Contact: www.baldwin.senate.gov/feedback        cont…
##  9     3 Barrasso, John - (R - WY) Class I               stuff
## 10     3 307 Dirksen Senate Office Building Washington … addr…
## # ℹ 390 more rows

So now the rows have been given id values and a type. Use pivot_wider to tidy the data.

senate_raw_data |>
  mutate(type = rep(c("stuff", "address", "phone", "contact"),  length.out = n())) |>
  mutate(id = (row_number() - 1) %/% 4 + 1, .before = 1) |>
  pivot_wider(names_from = type, values_from = X1) 
## # A tibble: 100 × 5
##       id stuff                           address phone contact
##    <dbl> <chr>                           <chr>   <chr> <chr>  
##  1     1 Alexander, Lamar - (R - TN) Cl… 455 Di… (202… Contac…
##  2     2 Baldwin, Tammy - (D - WI) Clas… 709 Ha… (202… Contac…
##  3     3 Barrasso, John - (R - WY) Clas… 307 Di… (202… Contac…
##  4     4 Bennet, Michael F. - (D - CO) … 261 Ru… (202… Contac…
##  5     5 Blackburn, Marsha - (R - TN) C… 357 Di… (202… Contac…
##  6     6 Blumenthal, Richard - (D - CT)… 706 Ha… (202… Contac…
##  7     7 Blunt, Roy - (R - MO) Class III 260 Ru… (202… Contac…
##  8     8 Booker, Cory A. - (D - NJ) Cla… 717 Ha… (202… Contac…
##  9     9 Boozman, John - (R - AR) Class… 141 Ha… (202… Contac…
## 10    10 Braun, Mike - (R - IN) Class I  B85 Ru… (202… Contac…
## # ℹ 90 more rows

Finish tidying up this data with separate and unite. A string like "Alexander, Lamar - (R - TN) Class II" can be broken apart with the following separator:

tibble(
  name = "Alexander, Lamar - (R - TN) Class II"
) |>
  separate(name, 
           into = c("name", "party", "state", "class"),
           sep = "( - \\()|( - )|(\\) Class )")
## # A tibble: 1 × 4
##   name             party state class
##   <chr>            <chr> <chr> <chr>
## 1 Alexander, Lamar R     TN    II

13.9 Pattern matching with globs

One common application of pattern matching is to sort through names of files. In R, the dir function can be used to get a list of the file names in a particular directory. Suppose that this command was run and the following list of files was returned.

example_filelist <- c("weather1.txt", 
                      "weather23.txt", 
                      "weather23.csv",
                      "weather47.html")

Then to sort out the files ending in .txt, the following regex does the trick:

example_filelist |> str_extract("^.*\\.txt$")
## [1] "weather1.txt"  "weather23.txt" NA              NA

Here the .txt part is called the file extension and it is common to want to search for files with a particular file extension. For that reason, another form of pattern matching is used in many operating systems called globbing.

A glob is a pattern matching scheme that is often used with file directories. It is notable for its two wildcard characters. First, * matches any continuous sequence of characters. Second, ? matches any single character.

In the search above, the regex ^.*\.txt$ is equivalent to *.txt in glob form. If you wish to use globs directly, the glob2rx function can be used to convert a glob expression to a regex.

example_filelist |> str_extract(glob2rx("*.txt"))
## [1] "weather1.txt"  "weather23.txt" NA              NA

The ? symbol matches any single character. So ??? in a glob matches with any three character file extension.

example_filelist |> str_extract(glob2rx("*.???"))
## [1] "weather1.txt"  "weather23.txt" "weather23.csv" NA

Similarly, .???? at the end will match any four character file extension.

example_filelist |> str_extract(glob2rx("*.????"))
## [1] NA               NA               NA               "weather47.html"

Questions

What string answer01 matches the pattern (Why?)(Because.) somewhere in a string? So regex(answer01) should match with "(How?)(Why?)(Because.)" but not "(How?)(Why.)(Because?)"

Suppose that I have a regular expression \(\[[a-z]*\]\) that looks for zero or more lowercase letters surrounded by brackets and then surrounded by parentheses. What is the string answer where regex(answer) creates this regular expression?

Write a function answer03 that takes as input a vector of strings, and returns TRUE for each string that contains two digits followed by the same two digits. So "a57572b" should match, but "57725" should not.

  1. Create a function answer04a that takes as input a vector of strings, and checks if each string is both nonempty and only contains digits. So "23443" and "2" should match, but "" and "2452abc" should not. For strings in the vector that match, it should return TRUE, and strings that don’t it should return FALSE.

  2. Create a function answer04b that matches a left bracket, followed by any number (including zero) of digits, followed by a right bracket. So "[23443]" and "[]" should match, but "(23443)" and "[23gdg]" should not. Have your function extract the first pattern that matches if such exists in the string.

Using str_replace_all write a function answer05 that takes a string s as input and replaces every instance of digits surrounded by parentheses with the same digits surrounded by the vertical bar |. So "(345)(874)[273]" gets turned into "|345||874|[273]".