AEM 6850

Empirical Methods for Applied Economists

Prof. Ariel Ortiz-Bobea

16 · Text I: regex

Thursday, October 22, 2026

Plan for today

1. Text as data in economics
   Three papers · the terms
2. Regex basics
   What a regular expression is · metacharacters · character classes · quantifiers · anchors · groups
3. Validation, extraction and replacement
   The functions · checking a format · pulling out numbers · cleaning text · testing a pattern · Exercise 1
4. Patterns on 171 Fed statements
   Finding a word · false positives · word counts · Exercise 2 · net tone
5. A chatbot writes the extraction code
   Reading its pattern · comparing it with the Fed’s daily series · a look-alike hyphen · fixing the code
6. When to use regex

Before Part 4: unzip 16-text-regex.zip and open 16-text-regex.Rproj.

From text to variables

The Federal Open Market Committee decided today to establish a target range for the federal funds rate of 0 to 1/4 percent.

Since the Committee’s last meeting, labor market conditions have deteriorated, and the available data indicate that consumer spending, business investment, and industrial production have declined. Financial markets remain quite strained and credit conditions tight. Overall, the outlook for economic activity has weakened further.

• A pattern finds 0 to 1/4 percent → lower = 0 and upper = 0.25 (percent).
• A count of words on a finance word list: the statement has 403 words. 8 are negative (the 4 in red are in these two paragraphs) and 2 are positive.
• Net tone = 100 × (positive − negative) / words = 100 × (2 − 8) / 403 = −1.49.
• Run on all 171 statements from 2006 to 2026, each method gives a column you can plot and compare with other data.

The Federal Reserve (the Fed) is the US central bank. Its policy committee, the Federal Open Market Committee (FOMC), sets a target range for the federal funds rate (the interest rate banks charge each other on overnight loans). It publishes a statement after each meeting.

Paragraphs 1 and 2 of the FOMC statement of 16 December 2008, during the financial crisis.

1 · Text as data in economics

• Three papers that count words
• The terms

Install now, in the R Console:

install.packages(c("stringr", "stopwords"))

Three papers that count words

Paper	Text	What they counted	Finding
Baker, Bloom & Davis (QJE, 2016)	articles in 10 US newspapers, each month	share of articles with a word from each of three lists: economy, policy, uncertainty (an uncertainty index)	the index rises near close elections, wars, 9/11, the failure of Lehman Brothers and the 2011 debt-ceiling fight
Shapiro & Wilson (REStud, 2022)	transcripts of Fed policy meetings, 2000–2011	negative and positive words from a finance word list	the committee aimed at about 1.5 percent inflation, below the 2 percent usually assumed
Djourelova (AER, 2023)	more than 2,000 US news outlets	the phrase “illegal immigrant”, before and after the Associated Press told its reporters to stop using it in April 2013	readers of papers that relied on AP stories became less supportive of restrictive immigration policy

Lehman Brothers: a large US investment bank that failed in September 2008. Debt ceiling: the legal limit on US federal borrowing. Associated Press (AP): a US news agency whose stories many local papers reprint.

Takeaway: each paper turns text into a variable by counting words or a phrase. In the first two, people read a random sample by hand to confirm the count measures what the authors claim. Details and figures are on the session page.

Lehman Brothers: a large US investment bank that failed in September 2008. The debt ceiling: the legal limit on federal borrowing. The Associated Press is a news agency whose stories many local papers reprint.

Terms for text as data

Corpus: the 171 Fed statements, 2006–2026. Example document: the statement of 22 March 2023 (target range 4-3/4 to 5 percent).

2 · Regex basics

• What a regular expression is
• Metacharacters
• Character classes and the dot
• Quantifiers
• Anchors and word boundaries
• Groups and alternatives

What a regular expression is

• A regular expression (regex) is a pattern, written as a string, that describes a set of texts. A function reads the pattern and finds the stretches of text that fit it, called matches.
• Most characters match themselves: the pattern "rate" finds the letters r, a, t, e in that order. These are literals.
• A few characters have a special meaning, such as ?, which makes the piece before it optional. These are metacharacters.
• The same syntax works in most programming languages and text editors.

R Console; needs stringr

# install.packages(c("stringr", "stopwords"))   # once
library(stringr)               # text functions; every name starts with str_
words <- c("color", "colour", "colr", "Colour")
str_detect(words, "colou?r")   # TRUE if the string contains a match; ? makes the u optional

#> [1]  TRUE  TRUE FALSE FALSE

str_view(words, "colou?r")     # the strings that match, each match in < >

#> [1] │ <color>
#> [2] │ <colour>

str_detect(words, regex("colou?r", ignore_case = TRUE))   # upper and lower case match alike

#> [1]  TRUE  TRUE FALSE  TRUE

Matching is case-sensitive, so “Colour” does not match until ignore_case = TRUE inside regex() turns that off.

Metacharacters

Symbol	Means	Pattern	Example match
.	any one character	r.te	rate, rite
[ ]	one character from the set	[0-9]	4
[^ ]	one character not in the set	[^0-9]	the - and / in 4-3/4
?	the piece before, zero or one time	rates?	rate, rates
*	the piece before, zero or more times	10*	1, 10, 100
+	the piece before, one or more times	[0-9]+	2026
{n}	the piece before, exactly n times	[0-9]{4}	2026
^ $	the start, the end of the text	^The	“The” at the start
|	the piece on either side	raise|lower	raise, lower
( )	a group, whose match R can return alone	([0-9]+)%	4% (group: 4)
\\	the next character is plain (in R, typed twice)	\\.	a period

Shortcuts: \\d a digit, \\s a space or line break, \\w a letter, digit or underscore, \\b the edge of a word. In R each backslash is written twice: you type "\\d" and the pattern receives \d. A piece is one character, a class in brackets, or a group.

Character classes [ ] and \\d

R Console; needs stringr

# a sentence from the Fed's policy statement of 22 March 2023
x <- "the Committee decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent."
str_view(x, "\\d")

#> [1] │ the Committee decided to raise the target range for the federal funds rate to <4>-<3>/<4> to <5> percent.

str_view(x, "[0-9/-]")

#> [1] │ the Committee decided to raise the target range for the federal funds rate to <4><-><3></><4> to <5> percent.

str_view(x, "[^a-z ]")      # ^ inside brackets: any character NOT in the set

#> [1] │ the <C>ommittee decided to raise the target range for the federal funds rate to <4><-><3></><4> to <5> percent<.>

• Character class: characters in brackets. It matches any one of them.
• Inside brackets, 0-9 means any digit and a-z any lowercase letter. A hyphen placed last, as in [0-9/-], is a plain hyphen.
• [^ ] matches any one character NOT in the set. The set in [^a-z ] is the lowercase letters and the space, so it catches the capital C, the digits, the - and the /, and the final period.
• \\d: one digit, the same as [0-9].
• The Fed writes rates as a whole number and a fraction: “4-3/4 percent” is 4.75 percent.

The dot and the escape

R Console; needs stringr and x

str_view(c("rate", "rite", "r8te", "rte"), "r.te")   # . : any one character

#> [1] │ <rate>
#> [2] │ <rite>
#> [3] │ <r8te>

str_view(x, "\\.")           # \\. : a period

#> [1] │ the Committee decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent<.>

c(str_count(x, "."), str_count(x, "\\."))

#> [1] 97  1

• The dot .: any one character.
• The dot needs exactly one character, so “rte” does not match. str_count() counts the matches in each string.
• Escape: a backslash that turns a metacharacter back into a plain character: \\. a period, \\$ a dollar sign, \\? a question mark, \\( a parenthesis.
• x has 97 characters. "." matches each of them, and "\\." only the period.

Quantifiers

R Console; needs x

str_extract_all(x, "[0-9/-]+")            # + : one or more of the piece before

#> [[1]]
#> [1] "4-3/4" "5"

str_extract(x, "\\d+-\\d+/\\d+")           # a whole number, a hyphen, a fraction

#> [1] "4-3/4"

str_detect(c("1/4 percentage point", "1/2 percentage points"), "percentage points?")

#> [1] TRUE TRUE

str_extract(c("year 2026", "26"), "\\d{4}")   # {4} : exactly four digits

#> [1] "2026" NA

• Quantifier: a mark after a piece (one character, a class in brackets, or a group) saying how many times it may repeat: + one or more, * zero or more, ? zero or one, {4} exactly four.
• str_extract() returns the first match in each string, or NA. str_extract_all() returns every match, as a list with one element per string (the [[1]]).

Anchors and word boundaries

R Console; needs stringr

s <- c("rate", "the rate rose", "moderate", "rates")
str_detect(s, "rate")             # the literal also matches inside other words

#> [1] TRUE TRUE TRUE TRUE

str_detect(s, "\\brate\\b")       # \\b : the edge of a word

#> [1]  TRUE  TRUE FALSE FALSE

str_detect(s, "^rate$")           # ^ : start of the text; $ : end of the text

#> [1]  TRUE FALSE FALSE FALSE

• Anchor: ^ pins the pattern to the start of the text and $ to the end. With both, the whole string must match. Inside brackets, ^ means NOT.
• Word boundary \\b: the edge between a letter, digit or underscore and anything else: a space, a punctuation mark, or the start or end of the text.
• False positive: text the pattern catches but should skip, such as “moderate” when you want the word “rate”. False negative: text it misses but should catch, such as “rates” under \\brate\\b.

Capture groups ( )

R Console; needs stringr

rates <- c("4-3/4", "1-1/2", "5")             # rates as the Fed writes them
m <- str_match(rates, "(\\d+)-(\\d+)/(\\d+)")  # whole number, numerator, denominator
m

#>      [,1]    [,2] [,3] [,4]
#> [1,] "4-3/4" "4"  "3"  "4" 
#> [2,] "1-1/2" "1"  "1"  "2" 
#> [3,] NA      NA   NA   NA

as.numeric(m[, 2]) + as.numeric(m[, 3]) / as.numeric(m[, 4])   # 4 + 3/4 = 4.75

#> [1] 4.75 1.50   NA

• Capture group: a part of the pattern inside ( ). str_match() returns the whole match in column 1, then one column per group, in the order the groups open.
• The groups split “4-3/4” into 4, 3 and 4, so the last line can compute 4 + 3/4 = 4.75.
• “5” has no fraction, so the pattern finds no match and the whole row is NA.

Alternatives |

R Console; needs stringr

# phrases from the Fed statements of 8 Aug 2006, 22 Mar 2023 and 16 Dec 2008
phrases <- c("keep its target for the federal funds rate at 5-1/4 percent",
             "raise the target range for the federal funds rate to 4-3/4 to 5 percent",
             "establish a target range for the federal funds rate of 0 to 1/4 percent")
str_match(phrases, "(?:at|to|of) ([0-9/-]+) percent")

#>      [,1]               [,2]   
#> [1,] "at 5-1/4 percent" "5-1/4"
#> [2,] "to 5 percent"     "5"    
#> [3,] "to 1/4 percent"   "1/4"

• at|to|of matches at, to or of. (?: ) groups the three words without adding a column, so the number lands in column 2.
• Rows 2 and 3 return only the upper end of the range:
  • The pattern needs a space and “percent” right after the number.
  • After “to 4-3/4” comes “to 5”, so the search moves on and matches “to 5 percent”.
  • Getting both ends takes a second number group.

The same constructions come back in the chatbot’s pattern in Part 5.

3 · Validation, extraction and replacement

• The functions for each task
• Validating a format
• Extracting numbers
• Replacing and cleaning text
• Testing a pattern
• Exercise 1

The functions for each task

Task and result	stringr	base R
find, validate TRUE or FALSE	str_detect(x, p)	grepl(p, x)
count a number per string	str_count(x, p)	lengths(regmatches(x, gregexpr(p, x)))
extract first match, or NA	str_extract(x, p)	regmatches(x, regexpr(p, x))
groups match, then groups	str_match(x, p)	regmatches(x, regexec(p, x))
extract all a list of matches	str_extract_all(x, p)	regmatches(x, gregexpr(p, x))
replace the new strings	str_replace_all(x, p, r)	gsub(p, r, x)

Result: what the stringr function returns. x is the text, p the pattern and r the replacement. Base R puts the pattern first, and its extract functions drop strings with no match. Chatbots write either style.

Validating a format

R Console; needs stringr

fips <- c("06037", "6037", "36109", "3610A", " 36109")   # US county codes
str_detect(fips, "^\\d{5}$")         # exactly five digits, nothing before or after

#> [1]  TRUE FALSE  TRUE FALSE FALSE

str_detect(fips, "\\d{5}")           # without the anchors, " 36109" passes

#> [1]  TRUE FALSE  TRUE FALSE  TRUE

fips[!str_detect(fips, "^\\d{5}$")]  # the values that fail

#> [1] "6037"   "3610A"  " 36109"

dates <- c("2026-10-22", "10/22/2026", "2026-1-5")
str_detect(dates, "^\\d{4}-\\d{2}-\\d{2}$")   # year-month-day, two digits each

#> [1]  TRUE FALSE FALSE

• Validation: checking that every value has the format it should, before you merge or analyze the data.
• A FIPS code is the US government’s five-digit code for each US county: two digits for the state, three for the county. Read as a number, “06037” (Los Angeles County, California) becomes 6037, and a merge on it fails.

Extracting numbers

R Console; needs stringr

prices <- c("$1,250", "price: $99.99", "free", "$52,000")
str_extract(prices, "[0-9,.]+")             # digits, commas and periods

#> [1] "1,250"  "99.99"  NA       "52,000"

ages <- c("<1", "1-5", "5-10", "15+")        # age groups in a mortality table
str_match(ages, "(\\d+)(?:-(\\d+))?")         # group 1: the first number; group 2: the second

#>      [,1]   [,2] [,3]
#> [1,] "1"    "1"  NA  
#> [2,] "1-5"  "1"  "5" 
#> [3,] "5-10" "5"  "10"
#> [4,] "15"   "15" NA

• Extraction: keeping only the part of each string that matches.
• str_extract() returns the match as text.
• str_match() returns one column per group, so “5-10” splits into “5” and “10”. (?:-(\\d+))? makes the hyphen and the second number optional together; when they are missing, group 2 is NA.
• The pattern returns 1 for “<1” and 15 for “15+”. You code by hand that “<1” starts at 0 and that “15+” has no upper end.

Replacing and cleaning text

R Console; needs prices

amount <- str_extract(prices, "[0-9,.]+")       # "free" has no digits, so NA
as.numeric(str_replace_all(amount, ",", ""))    # drop the commas, then convert

#> [1]  1250.00    99.99       NA 52000.00

str_squish("  Federal   Reserve \n Board ")     # \n: a line break; result: one space between words

#> [1] "Federal Reserve Board"

# month/day/year becomes year-month-day
str_replace(c("10/22/2026", "01/05/2026"), "(\\d{2})/(\\d{2})/(\\d{4})", "\\3-\\1-\\2")

#> [1] "2026-10-22" "2026-01-05"

• Replacement: rewriting what the pattern matches. str_replace() changes the first match, str_replace_all() every match.
• Backreference: \\1, \\2 and \\3 in the replacement stand for what groups 1, 2 and 3 matched. The US writes dates month/day/year (10/22/2026), so the replacement puts the year first: 2026-10-22.
• as.numeric("1,250") gives NA, which is why the commas go first.

Testing a pattern

R Console; needs stringr

should_match <- c("the federal funds rate", "longer-term interest rates")
should_skip  <- c("at a moderate pace", "to generate sustained improvement")
str_detect(should_match, "\\brate\\b")

#> [1]  TRUE FALSE

str_detect(should_skip, "\\brate\\b")

#> [1] FALSE FALSE

str_detect(should_match, "\\brates?\\b")      # ? makes the "s" optional

#> [1] TRUE TRUE

str_detect(should_skip, "\\brates?\\b")

#> [1] FALSE FALSE

str_detect(should_skip, "rate")               # without \\b, "moderate" and "generate" match

#> [1] TRUE TRUE

• Test a pattern on strings with a known answer first. Goal here: the word “rate” or “rates”, and not “moderate” or “generate”.
• Should-match and should-skip tests: short vectors of real strings the pattern must catch and must leave alone, rerun after every change.
• \\brate\\b misses “rates” (false negative); "rate" catches both skip strings (false positives); \\brates?\\b passes both.
• Copy test strings from the documents. When the corpus shows a case the pattern gets wrong, add that string to one of the two vectors.

Exercise 1

• Exercise
• Solution

# Exercise 1 (5 minutes). R Console; needs stringr. Answers to a survey
# question on yearly income:
income <- c("$52,000", "48500", "Income: $61,250.00", "N/A", "about 40k",
            "$1,000,000")
# 1. Validation: write ok_pat, a pattern for an answer that is only an
#    amount: an optional $, then digits and commas, then optional cents
#    (.00). Run str_detect(income, ok_pat). Alone, $ means the end of the
#    text, so write \\$ for a dollar sign.
# 2. Extraction: pull the number out of every answer with str_extract().
# 3. Replacement: drop the commas with str_replace_all(), then as.numeric().
# 4. Read the result next to income. Which answers give NA or a wrong number?
#
# Two answers to compare with the room: how many answers pass step 1, and
# what "about 40k" becomes.

R Console; needs stringr

income <- c("$52,000", "48500", "Income: $61,250.00", "N/A", "about 40k",
            "$1,000,000")
ok_pat <- "^\\$?[0-9,]+(\\.\\d{2})?$"
str_detect(income, ok_pat)

#> [1]  TRUE  TRUE FALSE FALSE FALSE  TRUE

n <- str_extract(income, "[0-9,.]+")
as.numeric(str_replace_all(n, ",", ""))

#> [1]   52000   48500   61250      NA      40 1000000

Three answers pass: “$52,000”, “48500” and “$1,000,000”. “about 40k” becomes 40, a thousand times too small, with no warning. It fails step 1, which is how you find it.

Uses only the functions of this part. The $ needs an escape because, on its own, $ is the end anchor. Without ^ and $, “Income: $61,250.00” passes step 1. “N/A” gives NA in step 2, which is the right answer for a missing value.

4 · Patterns on 171 Fed statements

• The Fed’s policy statements
• Finding a word in 171 statements
• False positives in a large text
• Splitting text into words
• Word counts and stop words
• Exercise 2
• A finance word list and net tone

Unzip 16-text-regex.zip (the 171 statements in fomc_statements.csv, plus the files for later slides) and open 16-text-regex.Rproj.

The Fed’s policy statements

• The Federal Reserve (“the Fed”) is the US central bank.
• Its Federal Open Market Committee (FOMC) sets the policy interest rate. It meets eight times a year and publishes a statement after each meeting. A few statements follow unscheduled meetings.
• The federal funds rate is the rate banks charge each other for overnight loans. The Fed steers it.
• The target range is the band the FOMC announces for that rate. “4-3/4 to 5 percent” means 4.75 to 5. Before 16 December 2008 the FOMC announced a single number.
• The file fomc_statements.csv has one row per statement: 171 statements, 31 Jan 2006 to 16 Sep 2026, 165 of them after scheduled meetings.

Screenshot: federalreserve.gov, statement of 22 March 2023 (public domain).

This is the file the scraping session built. The meeting came 12 days after Silicon Valley Bank failed, which is why paragraph 2 opens “The U.S. banking system is sound and resilient.”

Finding a word in 171 statements

R Console, in the 16-text-regex folder; needs stringr; reads fomc_statements.csv (date, url, words, text, scheduled)

fomc <- read.csv("fomc_statements.csv")
dim(fomc)

#> [1] 171   5

c(inflation = sum(str_detect(fomc$text, "inflation")),
  Inflation = sum(str_detect(fomc$text, "Inflation")),
  either    = sum(str_detect(fomc$text, regex("inflation", ignore_case = TRUE))))

#> inflation Inflation    either 
#>       165        99       168

fomc$n_infl <- str_count(fomc$text, regex("inflation", ignore_case = TRUE))
summary(fomc$n_infl)

#>    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
#>   0.000   5.000   8.000   7.795  10.500  17.000

fomc$date[which(fomc$n_infl == 0)]   # which(): the positions where the condition is TRUE

#> [1] "2007-08-17" "2020-03-03" "2020-03-23"

• Literal pattern: a pattern with no metacharacters. Matching is case-sensitive: 165 statements contain “inflation”, 99 “Inflation”, 168 either.
• str_detect() gives TRUE or FALSE per statement, and sum() counts the TRUEs. str_count() gives the number of matches per statement, a new column n_infl (0 to 17, median 8).
• The three statements without “inflation” all followed unscheduled meetings: 17 Aug 2007, and 3 and 23 Mar 2020, as the coronavirus spread.

Paragraphs inside text are separated by a blank line (“”). The three statements with no “inflation” are the unscheduled 17 Aug 2007 and 23 Mar 2020 and the emergency cut of 3 Mar 2020.

False positives in a large text

R Console; needs fomc

sum(str_count(fomc$text, "rate"))

#> [1] 982

sum(str_count(fomc$text, "\\brate\\b"))

#> [1] 705

caught <- unlist(str_extract_all(fomc$text, "[a-z]*rate[a-z]*"))   # every word containing "rate"
head(sort(table(caught), decreasing = TRUE), 6)

#> caught
#>       rate   moderate      rates moderately  moderated   generate 
#>        705        144         58         42         21          3

• The literal “rate” matches 982 times, the whole word 705 times. The other 277 are 58 “rates”, which you may want (\\brates?\\b keeps them), and 219 inside other words, such as “moderate” (144).
• A false positive is text the pattern catches but should skip. In a short test vector you see one at a glance. In 171 statements, a table of everything the pattern caught shows them.
• unlist() puts every statement’s matches in one vector, so table() can count them.

Splitting text into words

R Console; needs stringr and fomc

i <- which(fomc$date == "2023-03-22")                         # the row of the 22 Mar 2023 statement
fomc$year <- as.numeric(str_sub(fomc$date, 1, 4))             # the year, used in Exercise 2
tokens <- str_extract_all(str_to_lower(fomc$text), "[a-z]+")  # every run of letters
head(tokens[[i]], 12)

#>  [1] "recent"     "indicators" "point"      "to"         "modest"    
#>  [6] "growth"     "in"         "spending"   "and"        "production"
#> [11] "job"        "gains"

length(tokens[[i]])

#> [1] 364

• Token: one unit of text after splitting, here a word in lower case. Extracting every run of letters does the split: spaces, punctuation and numbers are left out.
• str_to_lower() comes first so “Inflation” and “inflation” count as one word. tokens is a list with one vector of words per statement, and [[i]] picks one.
• The apostrophe is not a letter, so “Committee’s” becomes two tokens, “committee” and “s”.

The file’s words column says 365 for this statement. It counts words differently, which is why the slides say tokens.

Word counts and stop words

R Console; needs tokens and the stopwords package

all_words <- unlist(tokens)
length(all_words)

#> [1] 73839

head(sort(table(all_words), decreasing = TRUE), 8)

#> all_words
#>       the       and        to        of committee        in inflation      will 
#>      5320      2854      2576      2382      1706      1534      1330       950

stop_words <- stopwords::stopwords("en")     # pkg::fun(): a package's function without library()
kept <- all_words[!all_words %in% stop_words & nchar(all_words) > 1]   # !x %in% y: x is not in y
head(sort(table(kept), decreasing = TRUE), 8)

#> kept
#> committee inflation  economic      rate   federal    policy    market   percent 
#>      1706      1330       842       705       686       649       642       619

• Stop words: very common words dropped before counting (“the”, “of”, “will”). The package’s default English list (Snowball) has 175.
• Term frequency: a word’s count divided by all tokens. “inflation” is 1,330 of 73,839, or 1.8 percent.
• nchar() counts characters. nchar(all_words) > 1 drops 1,793 more one-letter tokens: the s of “Committee’s” and “U.S.”, and middle initials such as the H in Jerome H. Powell (each statement ends by naming the members who voted).
• The stop list also drops “not” and “no”, which reverse a sentence, and “against”, which is on the negative word list used for tone.

Exercise 2

• Exercise
• Solution

# Exercise 2 (5 minutes). R Console; needs fomc (with its year column) and
# tokens from the previous slides.
# Which words appear in the statements for the first time in 2022?
#
# 1. Put the words of the 2022 statements in one vector:
#      w22 <- unlist(tokens[fomc$year == 2022])
# 2. Put the words of every statement before 2022 in a second vector
#    called before.
# 3. Keep the 2022 words that are not in the vector before (use %in% and !).
# 4. Count them with table() and sort them, largest first.
#
# Two answers to compare with the room: the most frequent new word, and the
# new words that are people's names.

R Console; needs fomc and tokens

w22    <- unlist(tokens[fomc$year == 2022])
before <- unlist(tokens[fomc$year < 2022])
new    <- w22[!w22 %in% before]
head(sort(table(new), decreasing = TRUE), 16)

#> new
#>       war    russia   ukraine attentive  invasion    issued      cook  creating 
#>         8         7         7         6         6         6         5         5 
#>         d jefferson      lisa         n    philip  weighing      barr   collins 
#>         5         5         5         5         5         5         4         4

“war” (8), then “russia” and “ukraine” (7 each) and “invasion” (6): from 16 Mar 2022 the statements say “The invasion of Ukraine by Russia is causing tremendous human and economic hardship.” The names belong to four new voting members of the committee (Lisa D. Cook, Philip N. Jefferson, Michael S. Barr, Susan M. Collins); d and n are their middle initials.

Each statement lists the members who voted. Lisa D. Cook and Philip N. Jefferson appear from 15 Jun 2022, Michael S. Barr and Susan M. Collins from 27 Jul 2022; “d” and “n” are middle initials. Of the 2,886 words in the 2022 statements, 105 are new, 27 different words. The exercise needs no stop words, so a student whose stopwords install failed can still do it.

A finance word list

R Console; reads lm_words.csv

lm_words <- read.csv("lm_words.csv")         # word, list: "positive" or "negative"; lower case
table(lm_words$list)

#> 
#> negative positive 
#>     2345      347

pos <- lm_words$word[lm_words$list == "positive"]
neg <- lm_words$word[lm_words$list == "negative"]
c("liability", "tax", "cost") %in% neg      # three words a general-purpose list calls negative

#> [1] FALSE FALSE FALSE

• Word list (dictionary): a fixed list of words tied to a category. Counting the words on a positive and a negative list gives a statement’s tone.
• Loughran & McDonald (JF, 2011) built the lists for 10-Ks, the yearly report every listed US company files with the Securities and Exchange Commission (the US regulator of stock markets).
• Finding. In 10-Ks from 1994 to 2008, a general-purpose list (the Harvard General Inquirer, built for psychology and sociology) flagged many words as negative. Almost three-fourths of them, such as tax, cost and liability, are not negative in financial text.
• Takeaway: a word list built for one kind of text misreads another. Shapiro & Wilson (REStud, 2022) used these lists on the Fed’s meeting transcripts, and the next slides apply them to the statements.

Free for use in academic research; commercial licences from the authors (sraf.nd.edu).

Net tone of each statement

Net tone: positive words minus negative words, per 100 tokens.

R Console; needs fomc, tokens, pos and neg

net_tone <- function(w) 100 * (sum(w %in% pos) - sum(w %in% neg)) / length(w)
j <- which(fomc$date == "2008-12-16")
w <- tokens[[j]]
w[w %in% neg]

#> [1] "deteriorated" "declined"     "strained"     "weakened"     "diminished"  
#> [6] "declines"     "weaker"       "weak"

w[w %in% pos]

#> [1] "stability"     "exceptionally"

net_tone(w)

#> [1] -1.488834

fomc$tone <- sapply(tokens, net_tone)        # sapply(): the function on each statement's words
fomc[c(which.min(fomc$tone), which.max(fomc$tone)), c("date", "tone")]   # lowest, highest

#>           date      tone
#> 24  2008-09-16 -2.727273
#> 130 2021-07-28  2.057613

On 16 Dec 2008, during the financial crisis, the committee cut its target to a range of 0 to 1/4 percent; the statement has 403 tokens. The two positive words come from set phrases: “price stability” and “exceptionally low”.

The lowest, 16 Sep 2008, came the day after Lehman Brothers filed for bankruptcy. Its 220 tokens contain 1 positive and 7 negative words.

Drawing net tone over time

R Console; needs fomc with its tone column

par(mar = c(2.5, 4.5, 0.8, 0.5))               # margins in lines: bottom, left, top, right
plot(as.Date(fomc$date), fomc$tone, type = "h", ylim = c(-3.4, 2.8),
     xlab = "", ylab = "Net tone per 100 tokens", las = 1)    # "h": one bar per statement
abline(h = 0, col = "grey60")
lab <- which(fomc$date %in% c("2008-09-16", "2020-03-03", "2021-07-28"))
text(as.Date(fomc$date[lab]), fomc$tone[lab], c("16 Sep 2008", "3 Mar 2020", "28 Jul 2021"),
     pos = c(1, 3, 3), cex = 1, xpd = NA)

• type = "h" draws one vertical bar from 0 to each statement’s tone.
• text() labels three statements: the lowest (16 Sep 2008), the highest (28 Jul 2021) and 3 Mar 2020, a rate cut between scheduled meetings.
• pos = 1 puts a label below its bar and pos = 3 above it. xpd = NA lets a label run outside the box.

The plot is on the next slide.

Net tone of 171 statements, 2006–2026

Net tone per 100 tokens (Loughran-McDonald lists), one bar per statement: 103 above zero, 54 below, 14 at zero. Lowest: 16 Sep 2008, the day after the US bank Lehman Brothers failed. Highest: 28 Jul 2021. 3 Mar 2020, a rate cut between scheduled meetings as the coronavirus spread, still ranks 8th most positive.

5 · A chatbot writes the extraction code

The task: pull the target range for the federal funds rate (lower and upper limit, in percent) out of each of the 171 statements, with code a chatbot wrote.

• The prompt and the reply
• Reading its pattern piece by piece
• The reply’s tests on 171 statements
• Comparing it with FRED, the Fed’s daily series of the target
• Where the reply’s code fails
• Two hyphens that look the same
• Fixing the code

The prompt and the reply

I have a data frame in R called fomc with a column text that holds the full text of every FOMC statement since 2006, one statement per row. Each statement says what the Fed decided about its policy rate, for example: “the Committee decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent.” Write R code with stringr that extracts the target range from each statement into two numeric columns, lower and upper (here 4.75 and 5).

chat_reply.R in the zip: steps 1 to 3 of the reply's code, unchanged; the lines above them define num and to_num()

# 1. Isolate the part of the statement describing the decision on the funds rate
seg <- str_extract(
  fomc$text,
  regex("decided.*?federal funds rate.*?percent\\b", ignore_case = TRUE, dotall = TRUE)
)
# 2. Pull out "to/at/of X [to Y] percent"
pat <- str_c("\\b(?:to|at|of)\\s+(", num, ")(?:\\s+to\\s+(", num, "))?\\s+percent\\b")
m   <- str_match(seg, pat)
# 3. Convert to numeric
first  <- to_num(m[, 2])
second <- to_num(m[, 3])
fomc$lower <- first
fomc$upper <- ifelse(is.na(second), first, second)

The reply: “For your example, "...to 4-3/4 to 5 percent." gives lower = 4.75 and upper = 5.” The prompt shows one example sentence and gives no other statements to test the code on. What follows reads num, step 1 and step 2 in turn. The full reply is on the session page.

Reply from Claude (claude-sonnet-5-5) through Anthropic’s API, 20 October 2026, to the prompt shown. The same prompt in the chat at claude.ai can return different code.

This is the chatbot the course recommended. Students will get different code if they paste the prompt; the steps that follow apply to any reply.

Reading the number pattern

R Console, in the 16-text-regex folder; needs stringr and fomc

source("chat_reply.R")      # runs the reply's code, unchanged
str_view(c("5", "1/4", "4-3/4", "0.25"), num)

#> [1] │ <5>
#> [2] │ <1/4>
#> [3] │ <4-3/4>
#> [4] │ <0.25>

to_num(c("4-3/4", "1/4", "5", "0"))

#> [1] 4.75 0.25 5.00 0.00

• num: the reply’s pattern for one number as the Fed writes it (“5”, “1/4”, “4-3/4”). Step 2 uses it twice, for the lower and the upper limit.
• source() runs every line of a script file. chat_reply.R defines num, to_num(), seg and pat, and adds lower and upper to fomc.
• Known-answer test: inputs whose right output you know before running. In str_view() all four strings match in full, and to_num() returns 4.75, 0.25, 5 and 0, as expected.
• to_num() (in chat_reply.R) splits “4-3/4” into 4, 3 and 4 and returns 4 + 3/4.

Greedy and lazy matching

R Console; needs fomc and seg from chat_reply.R

i <- which(fomc$date == "2023-03-22")
greedy <- str_extract(fomc$text[i], regex("decided.*percent", dotall = TRUE))
lazy   <- str_extract(fomc$text[i], regex("decided.*?percent", dotall = TRUE))
c(nchar(greedy), nchar(lazy))       # nchar() counts characters

#> [1] 926  82

lazy

#> [1] "decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent"

identical(lazy, seg[i])             # the reply's step 1 gives the same text

#> [1] TRUE

• Step 1 of the reply cuts the decision sentence out of each statement with lazy .*?.
• Greedy .* takes as many characters as it can. Here greedy runs 926 characters, to the last “percent” in the statement (“…returning inflation to its 2 percent”).
• Lazy .*? takes as few as it can.
• dotall = TRUE lets . match a line break (“\n”). Paragraphs in text are separated by line breaks, so with dotall a match can run from one paragraph into the next.
• Why the reply stops at the first “percent” after “federal funds rate”: “This keeps it from picking up other numbers in the statement, such as the 2 percent inflation goal, ‘25 basis points’, or the discount rate.” A basis point is 0.01 percentage point; the discount rate is the rate the Fed charges banks that borrow from it.

Reading the range pattern

R Console; needs seg, pat and i

str_match(seg[i], pat)              # the reply's step 2

#>      [,1]                    [,2]    [,3]
#> [1,] "to 4-3/4 to 5 percent" "4-3/4" "5"

• str_c() joins strings end to end (base R paste0()). pat pastes in num, the previous slide’s number pattern, once per limit.
• On 22 March 2023, group 1 is “4-3/4” (the lower limit) and group 2 is “5” (the upper). to_num() turns them into 4.75 and 5.
• The second group is optional. A single target such as “at 5-1/4 percent”, the form used before December 2008, also matches, and the reply then sets upper equal to lower.

The reply’s tests on 171 statements

R Console; needs fomc after source("chat_reply.R")

sum(is.na(fomc$upper))                         # the reply's test 1 prints these rows; here we count them

#> [1] 32

summary(fomc[, c("lower", "upper")])           # the reply's test 2

#>      lower           upper      
#>  Min.   :0.000   Min.   :0.250  
#>  1st Qu.:0.000   1st Qu.:0.250  
#>  Median :1.000   Median :1.250  
#>  Mean   :1.939   Mean   :2.128  
#>  3rd Qu.:4.250   3rd Qu.:4.500  
#>  Max.   :5.250   Max.   :5.500  
#>  NAs    :32      NAs    :32

sum(fomc$lower > fomc$upper, na.rm = TRUE)     # the reply's test 3: lower <= upper

#> [1] 0

• The reply suggests three tests: print the NA rows (fomc[is.na(fomc$upper), "text"]), run summary(), and verify lower <= upper.
• It expects NA only for “a few unscheduled statements about liquidity facilities”. There are 32, and reading them shows wordings the pattern misses.
• summary() shows values from 0 to 5.5 percent, all plausible for a policy rate, and no row has lower above upper.
• The tests cannot say whether the other 139 values are right.

Comparing with FRED

R Console; needs fomc after the reply's code; reads fred_target_range.csv

fred <- read.csv("fred_target_range.csv")                # date, lower, upper: one row per day
k <- match(as.Date(fomc$date) + 1, as.Date(fred$date))    # + 1: the day after each statement
fomc$fred_lower <- fred$lower[k]
fomc$fred_upper <- fred$upper[k]
fomc$agree <- fomc$lower == fomc$fred_lower & fomc$upper == fomc$fred_upper
table(fomc$agree, useNA = "ifany")

#> 
#> FALSE  TRUE  <NA> 
#>     9   130    32

fomc$date[which(!fomc$agree)]

#> [1] "2014-03-19" "2014-04-30" "2014-06-18" "2014-07-30" "2014-09-17"
#> [6] "2014-10-29" "2019-12-11" "2020-01-29" "2022-06-15"

• FRED, the free data service of the Federal Reserve Bank of St. Louis, gives the target for every day: DFEDTAR, the single target to 15 Dec 2008 (copied into both columns), then DFEDTARL and DFEDTARU, the lower and upper limits.
• match() finds the position of each statement’s next day among FRED’s dates. Adding 1 to a date gives the next day. Since 2017 a new range takes effect the day after the statement, so the next day is the first day FRED shows it.
• agree is TRUE for 130 statements (both limits equal FRED’s), FALSE for 9 and NA for 32 (no number found). which() skips NA, so only the nine wrong statements print. In all nine, lower equals upper, so the reply’s lower <= upper test passes on every one.

Since March 2017 a new range takes effect, and appears in FRED, the day after the statement (30 changes). From 2006 to 2016 FRED shows it from the statement date (16 changes). On the statement date itself 106 agree.

Drawing the comparison with FRED

R Console; needs fred, and fomc with the reply's lower and upper and the FRED columns

d <- as.Date(fomc$date) + 1                    # the day after each statement
ok <- fomc$agree %in% TRUE                     # %in% TRUE: NA counts as not TRUE
bad <- fomc$agree %in% FALSE
par(mar = c(2.5, 4.5, 0.5, 0.5))               # margins in lines: bottom, left, top, right
plot(as.Date(fred$date), fred$upper, type = "s", col = "grey60", ylim = c(0, 6),
     xlab = "", ylab = "Target, percent", las = 1)    # "s": steps, flat between decisions
lines(as.Date(fred$date), fred$lower, type = "s", col = "grey60")
points(d[ok], fomc$upper[ok], pch = 16, cex = 0.7)
points(d[bad], fomc$lower[bad], pch = 4, cex = 1.4, lwd = 2, col = "firebrick")
segments(d[bad], fomc$fred_lower[bad], d[bad], fomc$lower[bad], col = "firebrick", lwd = 2)
rug(d[is.na(fomc$agree)], col = "darkorange", lwd = 2, ticksize = 0.04)   # a tick for each NA
label_x <- as.Date(c("2010-07-01", "2013-06-01", "2015-08-01", "2021-03-01", "2018-10-01", "2025-10-01"))
label_y <- c(0.8, 1.05, 1.65, 1.0, 0.55, 1.0)  # labels placed by hand at (date, percent)
text(label_x, label_y, cex = 0.9,
     c("\"will maintain\"", "2014: the match\nruns too far", "\"reaffirmed\"",
       "look-alike\nhyphen", "look-alike\nhyphen", "look-alike\nhyphen"),
     col = c("darkorange", "firebrick", "darkorange", "firebrick", "darkorange", "darkorange"))
legend("top", ncol = 2, bty = "n", cex = 0.85,
       legend = c("FRED: lower and upper", paste0("equal (", sum(ok), ")"),
                  paste0("wrong (", sum(bad), "): lower = upper"),
                  paste0("NA (", sum(is.na(fomc$agree)), ")")),
       col = c("grey60", "black", "firebrick", "darkorange"),
       lty = c(1, NA, 1, 1), lwd = c(1, NA, 2, 2), pch = c(NA, 16, 4, NA))

The plot is on the next slide. Grey steps: FRED’s lower and upper limits. Black dots: statements where the reply’s values equal FRED’s. Red crosses: wrong values, with a red line down to FRED’s lower limit. Orange ticks along the bottom: statements with NA.

Extracted range and FRED, 2006–2026

171 statements, each plotted on the day after it. Sources: federalreserve.gov, FRED. Each cause is listed on the next slide.

Where the reply’s code fails

What the statement says	Count	Dates	Code returns
“will maintain the target range … at 0 to 1/4 percent”: no “decided”, which step 1 needs	13	18 Mar 2009 to 21 Sep 2010	NA
“reaffirmed its view that the current 0 to 1/4 percent target range …”	8	17 Dec 2014 to 28 Oct 2015	NA
the range only in the instruction to the New York Fed (a regional Fed bank)	1	23 Mar 2020	NA
no target stated	1	17 Aug 2007	NA (correct)
a look-alike hyphen inside the upper number (“3‑3/4”; next slide)	9	13 Dec 2017 to 29 Apr 2026	NA
a look-alike hyphen inside the lower number (“1‑1/2”; next slide)	3	11 Dec 2019, 29 Jan 2020, 15 Jun 2022	1.75 to 1.75; FRED: 1.5 to 1.75
“decided” is first about asset purchases, so the match runs on (up to 2,354 characters, across paragraphs) to the first “to 1/4 percent”	6	19 Mar to 29 Oct 2014	0.25 to 0.25; FRED: 0 to 0.25

The 2014 segments run 2,330 to 2,354 characters with two paragraph breaks, except 29 Oct 2014 (851 characters). In these statements “federal funds rate” follows the range, so the first “to X percent” in the stretch is “to 1/4 percent”. The reply’s explanation quotes “by 1/2 percentage point, to 1 to 1-1/4 percent” as a phrase it handles; that is 3 Mar 2020, written with the look-alike hyphen, so that row is NA.

Two hyphens that look the same

R Console; needs fomc, seg and num from chat_reply.R

j <- which(fomc$date == "2025-12-10")
str_view(seg[j], num)

#> [1] │ decided to lower the target range for the federal funds rate by <1/4> percentage point to <3-1/2> to <3>‑<3/4> percent

utf8ToInt(str_extract(seg[j], "3.3/4"))     # . matches either hyphen; utf8ToInt(): each code point

#> [1]   51 8209   51   47   52

utf8ToInt("3-3/4")                           # typed with the hyphen key

#> [1] 51 45 51 47 52

sum(str_detect(fomc$text, "\u2011"))         # statements with character 8209

#> [1] 34

In the str_view() output, “3-1/2” is one match but “3‑3/4” is two, <3> and <3/4>: num stops at the look-alike hyphen, so the row is NA.

• Code point: the number a computer stores for each character. Unicode is the standard list of them.
• Character 8209: a non-breaking hyphen. Web pages use it to keep “3‑3/4” on one line. In R, "\u2011" writes it: 2011 is 8209 in hexadecimal (base 16), the form Unicode lists use.
• The - in a pattern matches only character 45.

The Fed’s pages use character 8209 fifty times, in 34 statements from July 2015 on, and in 12 of them it sits inside a number.

Fixing the reply’s code

R Console; needs fomc, num, to_num() and the FRED columns

# 1. the look-alike hyphen (character 8209) becomes "-"
fomc$text <- str_replace_all(fomc$text, "\u2011", "-")
# 2. stop at the period; 3. three verbs, no "federal funds rate"
seg <- str_extract(fomc$text, regex("(?:decided|maintain|reaffirmed)[^.]*?percent\\b",
                                    ignore_case = TRUE))
# 3. no "to/at/of" before the number
m <- str_match(seg, str_c("(", num, ")(?:\\s+to\\s+(", num, "))?\\s+percent\\b"))
fomc$lower <- to_num(m[, 2])
fomc$upper <- ifelse(is.na(m[, 3]), fomc$lower, to_num(m[, 3]))
fomc$agree <- fomc$lower == fomc$fred_lower & fomc$upper == fomc$fred_upper
table(fomc$agree, useNA = "ifany")

#> 
#> TRUE <NA> 
#>  170    1

• Change 1: every look-alike hyphen (character 8209) becomes “-”. This fixes the 12 hyphen rows.
• Change 2: [^.]*?, as few characters as possible that are not a period, replaces .*?, and dotall goes. Inside brackets the dot is a plain period, so the match ends with its sentence. The six 2014 rows now give NA in place of a wrong number.
• Change 3: step 1 accepts “maintain” and “reaffirmed” and drops “federal funds rate”, which can come after the number (“the current 0 to 1/4 percent target range for the federal funds rate”). Step 2 drops “to|at|of”, since “current 0 to 1/4” has none.

Results after each fix

Each row adds one change to the rows above. Counted against FRED on the day after each statement:

Code	NA	Wrong	Equal
The reply as given	32	9	130
1. Replace the look-alike hyphen with “-”	23	6	142
2. Keep the match in one sentence	36	0	135
3. Accept “maintain” and “reaffirmed”; take the range wherever it sits in the sentence	1	0	170

• Change 2 turns the six 2014 values into NA. It does the same to seven statements from Nov 2010 to Jan 2014, where “decided” is about asset purchases and the range sits in a later sentence. The old match reached it only by crossing sentences.
• The last NA is 17 Aug 2007, which states no target.
• To ask the chatbot for a fix, paste the failing sentences with FRED’s numbers for each. They are known-answer tests.

The seven: 3 Nov 2010, 14 Dec 2010, 26 Jan 2011, 15 Mar 2011, 27 Apr 2011, 18 Dec 2013 and 29 Jan 2014. The code for the two middle rows is on the page.

6 · When to use regex

• When to use regex
• What net tone misses
• Practice

When to use regex

Regex works well when

• the text follows a format: IDs, dates, amounts, codes
• the wording repeats, as in the Fed’s sentence that states the target range
• you can list the variants and test each one
• you need the same answer every time the code runs
• the corpus is large: a pattern runs on millions of documents in seconds, for free

Use a parser or a model when

• the meaning depends on context or on negation: the Fed’s “might not be strong enough” (13 Sep 2012) scores as positive
• the same idea is written in more ways than you can list
• the text is a web page (HTML) or data from an API (JSON), where parts sit inside other parts
• the variable is a judgment call, such as whether a sentence signals that the Fed will raise its rate

Parser: a function that reads a structured format into its parts, such as jsonlite::fromJSON() for JSON or rvest::read_html() for a web page. Model: a language model, the kind of program behind a chatbot, asked to read each text and return a label, or a statistical classifier trained on texts that people labeled by hand.

A pattern gives NA or a wrong number, with no error message, when the wording or a character changes. A chatbot can write the pattern. You still read it, test it on strings from your documents, and compare its output with a second source such as FRED.

What net tone misses

The target range has a second source, FRED, and 170 of 171 values now equal it. Net tone has no second source, so its errors show only when you read the sentences it scored. The finance word list scores each word on its own:

R Console; needs fomc

s <- str_extract(fomc$text[fomc$date == "2012-09-13"], "economic growth might not[^.]*")
str_view(s, "\\b(not|strong|improvement)\\b")

#> [1] │ economic growth might <not> be <strong> enough to generate sustained <improvement> in labor market conditions

• Negation. 13 Sep 2012 (also 24 Oct and 12 Dec 2012): “strong” and “improvement” count as positive. Each word is scored on its own, so the “not” before them changes nothing.
• Context. 25 Apr 2012, “the unemployment rate has declined but remains elevated”: “unemployment” and “declined” count as negative, although a falling unemployment rate is good news.
• The event behind the words. 3 Mar 2020, a cut in the target range between scheduled meetings as the coronavirus spread, is the 8th most positive of 171 statements. “strong”, “achieving” and “stability” count as positive. “coronavirus” and “risks” are on neither list.

The full 13 Sep 2012 sentence opens “The Committee is concerned that”, and “concerned” is on the negative list, so the sentence nets +1. Shapiro and Wilson ignored words after “not”, ran their estimates without “unemployment”, and filtered phrases such as “price stability”; their estimates barely moved (working paper version).

Practice

Ten exercises on the session page, with solutions behind a click. Run them after the slides’ code, which builds fomc, tokens, pos and neg from the files in the 16-text-regex folder. None needs an internet connection.

• 1–4 Patterns and numbers: the opening phrase by year; rate moves stated in basis points (1 basis point = 0.01 percentage point), and which come from members who voted against; the vote, such as “12 – 0”, in the 2026 statements; validating the web address (url) of each statement
• 5 Base R: the three fixes to the chatbot’s code, rewritten with gsub(), regmatches() and regexec()
• 6–7 Word counts: “inflation” as a share of tokens by year; the three word lists of Baker, Bloom & Davis (QJE, 2016) on the statements
• 8–9 Net tone: recomputed without unemployment, against, stability and exceptionally; the word after each “not”
• 10 Characters: every character with a code point above 127, such as the dash in “12 – 0”