16 · Text I: regex

Thursday, October 22, 2026

Slides · Live script · Source · the project

How to turn text into variables with patterns and word counts. The session starts with what a regular expression is and its metacharacters, then the three everyday uses (validating a format, extracting a value, replacing text) on short examples, then the same tools on a large text: the Federal Reserve’s policy statements since 2006. Two variables come out of them: the target range for the policy rate, pulled out of each statement with a regular expression and compared with the Fed’s own daily series, and the tone of each statement, counted with a finance word list. A chatbot writes the extraction code here, as it often will in your own work, so the session teaches enough of the syntax to read a pattern piece by piece and to test it on the whole corpus.

The project. Install the packages once, in the R Console: install.packages(c("stringr", "stopwords")). Download 16-text-regex.zip, unzip it, and open 16-text-regex.Rproj, so that R starts in the project folder. It holds:

• fomc_statements.csv: every policy statement of the Federal Open Market Committee from January 2006 to September 2026, one row each (date, url, words, text, scheduled).
• fred_target_range.csv: the Fed’s target for the policy rate on every day, from FRED.
• lm_words.csv: the positive and negative word lists of Loughran and McDonald.
• chat_prompt.txt, chat_reply.md and chat_reply.R: a prompt to a chatbot, its full reply, and the reply’s code, unchanged.
• SOURCE.txt: each source, its address, its terms of use and the date it was collected.

Objectives

By the end of this session you can:

• Say what a regular expression is and read its metacharacters.
• Write a pattern with character classes, quantifiers, ^, $ and \\b.
• Validate a format with an anchored pattern and str_detect().
• Extract numbers from text with str_extract(), capture groups and str_match().
• Clean and reformat text with str_replace_all() and backreferences.
• Give the base R name of each stringr function used.
• Rerun a test on strings copied from the documents after every change to a pattern.
• Find a word in many documents with str_detect().
• Count a word in each document with str_count().
• Split text into words with str_extract_all().
• Count words with table() after dropping stop words.
• Score each statement’s net tone with a finance word list.
• Read a chatbot’s pattern piece by piece, including (?: ), lazy .*? and the options of regex().
• Compare an extracted variable with FRED’s daily series row by row.
• Find a look-alike character by its code point with utf8ToInt().
• Replace a character in every document with str_replace_all().
• Say when a regular expression is the right tool, and when a parser or a model is.

Plan for today

1. Text as data in economics
   Three papers · the terms
2. Regex basics
   What a regular expression is · metacharacters · character classes · quantifiers · anchors · groups
3. Validation, extraction and replacement
   The functions · checking a format · pulling out numbers · cleaning text · testing a pattern · Exercise 1
4. Patterns on 171 Fed statements
   Finding a word · false positives · word counts · Exercise 2 · net tone
5. A chatbot writes the extraction code
   Reading its pattern · comparing it with the Fed’s daily series · a look-alike hyphen · fixing the code
6. When to use regex

From text to variables

The Federal Open Market Committee decided today to establish a target range for the federal funds rate of 0 to 1/4 percent.

Since the Committee’s last meeting, labor market conditions have deteriorated, and the available data indicate that consumer spending, business investment, and industrial production have declined. Financial markets remain quite strained and credit conditions tight. Overall, the outlook for economic activity has weakened further.

• A pattern finds 0 to 1/4 percent → lower = 0 and upper = 0.25 (percent).
• A count of words on a finance word list: the statement has 403 words. 8 are negative (the 4 in red are in these two paragraphs) and 2 are positive.
• Net tone = 100 × (positive − negative) / words = 100 × (2 − 8) / 403 = −1.49.
• Run on all 171 statements from 2006 to 2026, each method gives a column you can plot and compare with other data.

The Federal Reserve (the Fed) is the US central bank. Its policy committee, the Federal Open Market Committee (FOMC), sets a target range for the federal funds rate (the interest rate banks charge each other on overnight loans). It publishes a statement after each meeting.

Paragraphs 1 and 2 of the FOMC statement of 16 December 2008, during the financial crisis.

1 · Text as data in economics

Three papers that count words

Paper	Text	What they counted	Finding
Baker, Bloom & Davis (QJE, 2016)	articles in 10 US newspapers, each month	share of articles with a word from each of three lists: economy, policy, uncertainty (an uncertainty index)	the index rises near close elections, wars, 9/11, the failure of Lehman Brothers and the 2011 debt-ceiling fight
Shapiro & Wilson (REStud, 2022)	transcripts of Fed policy meetings, 2000–2011	negative and positive words from a finance word list	the committee aimed at about 1.5 percent inflation, below the 2 percent usually assumed
Djourelova (AER, 2023)	more than 2,000 US news outlets	the phrase “illegal immigrant”, before and after the Associated Press told its reporters to stop using it in April 2013	readers of papers that relied on AP stories became less supportive of restrictive immigration policy

Lehman Brothers: a large US investment bank that failed in September 2008. Debt ceiling: the legal limit on US federal borrowing. Associated Press (AP): a US news agency whose stories many local papers reprint.

Takeaway: each paper turns text into a variable by counting words or a phrase. In the first two, people read a random sample by hand to confirm the count measures what the authors claim. Details and figures are on the session page.

Baker, Bloom & Davis: a keyword index from newspapers

• Baker, Bloom & Davis (QJE, 2016) counted, each month, the articles in 10 large US newspapers that contain a word from each of three lists. The lists also take variants such as “uncertainties” and “the Fed”. (Congress is the US legislature; the White House is the President’s office.)
• Teams of University of Chicago students read about 12,000 randomly drawn articles and marked whether each discussed economic policy uncertainty. Their verdicts chose the policy words and tested the count.
• Keyword index: the share of documents in each period that contain chosen words.
• Finding. The index rises near close presidential elections, the Gulf Wars (1991 and 2003, both in Iraq), 9/11, the failure of Lehman Brothers (a large US investment bank, September 2008), and the 2011 dispute in Congress over the legal limit on federal borrowing.
• Takeaway: a widely used index is a count of documents matching three word lists, and people reading a sample by hand gave it credibility.

The index, monthly, January 1985 to September 2026: Baker, Bloom & Davis, policyuncertainty.com.

Shapiro & Wilson: the tone of Fed meetings

• Transcript: a lightly edited word-for-word record of an FOMC meeting, released after five years.
• Word list (dictionary): a fixed list of words tied to a category, such as negative words.
• Shapiro & Wilson (REStud, 2022) counted the negative and positive words on the lists of Loughran & McDonald (JF, 2011) in what each speaker said at each meeting. 14 research assistants scored 182 randomly drawn texts, and the word score agreed with them far better than a general-purpose list did.
• They related negative tone to the staff’s inflation forecast at each meeting. Negativity was lowest near 1.5 percent inflation.
• Finding. Over 2000–2011 the committee aimed at about 1.5 percent inflation, below the 2 percent usually assumed. The Fed first announced a 2 percent goal in January 2012.
• Takeaway: a word count on Fed text estimated the inflation goal the committee worked to before it announced one, once people’s scores on a random sample backed the count.

Djourelova: one phrase in the news

• The Associated Press (AP) is a US news agency. Many local papers reprint its stories, called AP copy. Its style guide, the wording rules its reporters follow, dropped “illegal immigrant” in April 2013.
• Djourelova (AER, 2023) counted the phrase in more than 2,000 US print and online outlets. In outlets that relied on AP copy, articles using it fell by 28% compared with other outlets.
• She linked answers to a large US election survey to the AP reliance of the newspapers sold in each respondent’s county.
• Finding. Readers exposed to the change through local media supported restrictive immigration policy less.
• Takeaway: an exact phrase count works when the wording itself is what you study.

Terms for text as data

Corpus: the 171 Fed statements, 2006–2026. Example document: the statement of 22 March 2023 (target range 4-3/4 to 5 percent).

2 · Regex basics

What a regular expression is

• A regular expression (regex) is a pattern, written as a string, that describes a set of texts. A function reads the pattern and finds the stretches of text that fit it, called matches.
• Most characters match themselves: the pattern "rate" finds the letters r, a, t, e in that order. These are literals.
• A few characters have a special meaning, such as ?, which makes the piece before it optional. These are metacharacters.
• The same syntax works in most programming languages and text editors.

R Console; needs stringr

# install.packages(c("stringr", "stopwords"))   # once
library(stringr)               # text functions; every name starts with str_
words <- c("color", "colour", "colr", "Colour")
str_detect(words, "colou?r")   # TRUE if the string contains a match; ? makes the u optional

#> [1]  TRUE  TRUE FALSE FALSE

str_view(words, "colou?r")     # the strings that match, each match in < >

#> [1] │ <color>
#> [2] │ <colour>

str_detect(words, regex("colou?r", ignore_case = TRUE))   # upper and lower case match alike

#> [1]  TRUE  TRUE FALSE  TRUE

Matching is case-sensitive, so “Colour” does not match until ignore_case = TRUE inside regex() turns that off.

Metacharacters

Symbol	Means	Pattern	Example match
.	any one character	r.te	rate, rite
[ ]	one character from the set	[0-9]	4
[^ ]	one character not in the set	[^0-9]	the - and / in 4-3/4
?	the piece before, zero or one time	rates?	rate, rates
*	the piece before, zero or more times	10*	1, 10, 100
+	the piece before, one or more times	[0-9]+	2026
{n}	the piece before, exactly n times	[0-9]{4}	2026
^ $	the start, the end of the text	^The	“The” at the start
|	the piece on either side	raise|lower	raise, lower
( )	a group, whose match R can return alone	([0-9]+)%	4% (group: 4)
\\	the next character is plain (in R, typed twice)	\\.	a period

Shortcuts: \\d a digit, \\s a space or line break, \\w a letter, digit or underscore, \\b the edge of a word. In R each backslash is written twice: you type "\\d" and the pattern receives \d. A piece is one character, a class in brackets, or a group.

Character classes [ ] and \\d

R Console; needs stringr

# a sentence from the Fed's policy statement of 22 March 2023
x <- "the Committee decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent."
str_view(x, "\\d")

#> [1] │ the Committee decided to raise the target range for the federal funds rate to <4>-<3>/<4> to <5> percent.

str_view(x, "[0-9/-]")

#> [1] │ the Committee decided to raise the target range for the federal funds rate to <4><-><3></><4> to <5> percent.

str_view(x, "[^a-z ]")      # ^ inside brackets: any character NOT in the set

#> [1] │ the <C>ommittee decided to raise the target range for the federal funds rate to <4><-><3></><4> to <5> percent<.>

• Character class: characters in brackets. It matches any one of them.
• Inside brackets, 0-9 means any digit and a-z any lowercase letter. A hyphen placed last, as in [0-9/-], is a plain hyphen.
• [^ ] matches any one character NOT in the set. The set in [^a-z ] is the lowercase letters and the space, so it catches the capital C, the digits, the - and the /, and the final period.
• \\d: one digit, the same as [0-9].
• The Fed writes rates as a whole number and a fraction: “4-3/4 percent” is 4.75 percent.

The dot and the escape

R Console; needs stringr and x

str_view(c("rate", "rite", "r8te", "rte"), "r.te")   # . : any one character

#> [1] │ <rate>
#> [2] │ <rite>
#> [3] │ <r8te>

str_view(x, "\\.")           # \\. : a period

#> [1] │ the Committee decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent<.>

c(str_count(x, "."), str_count(x, "\\."))

#> [1] 97  1

• The dot .: any one character.
• The dot needs exactly one character, so “rte” does not match. str_count() counts the matches in each string.
• Escape: a backslash that turns a metacharacter back into a plain character: \\. a period, \\$ a dollar sign, \\? a question mark, \\( a parenthesis.
• x has 97 characters. "." matches each of them, and "\\." only the period.

Quantifiers

R Console; needs x

str_extract_all(x, "[0-9/-]+")            # + : one or more of the piece before

#> [[1]]
#> [1] "4-3/4" "5"

str_extract(x, "\\d+-\\d+/\\d+")           # a whole number, a hyphen, a fraction

#> [1] "4-3/4"

str_detect(c("1/4 percentage point", "1/2 percentage points"), "percentage points?")

#> [1] TRUE TRUE

str_extract(c("year 2026", "26"), "\\d{4}")   # {4} : exactly four digits

#> [1] "2026" NA

• Quantifier: a mark after a piece (one character, a class in brackets, or a group) saying how many times it may repeat: + one or more, * zero or more, ? zero or one, {4} exactly four.
• str_extract() returns the first match in each string, or NA. str_extract_all() returns every match, as a list with one element per string (the [[1]]).

Anchors and word boundaries

R Console; needs stringr

s <- c("rate", "the rate rose", "moderate", "rates")
str_detect(s, "rate")             # the literal also matches inside other words

#> [1] TRUE TRUE TRUE TRUE

str_detect(s, "\\brate\\b")       # \\b : the edge of a word

#> [1]  TRUE  TRUE FALSE FALSE

str_detect(s, "^rate$")           # ^ : start of the text; $ : end of the text

#> [1]  TRUE FALSE FALSE FALSE

• Anchor: ^ pins the pattern to the start of the text and $ to the end. With both, the whole string must match. Inside brackets, ^ means NOT.
• Word boundary \\b: the edge between a letter, digit or underscore and anything else: a space, a punctuation mark, or the start or end of the text.
• False positive: text the pattern catches but should skip, such as “moderate” when you want the word “rate”. False negative: text it misses but should catch, such as “rates” under \\brate\\b.

Capture groups ( )

R Console; needs stringr

rates <- c("4-3/4", "1-1/2", "5")             # rates as the Fed writes them
m <- str_match(rates, "(\\d+)-(\\d+)/(\\d+)")  # whole number, numerator, denominator
m

#>      [,1]    [,2] [,3] [,4]
#> [1,] "4-3/4" "4"  "3"  "4" 
#> [2,] "1-1/2" "1"  "1"  "2" 
#> [3,] NA      NA   NA   NA

as.numeric(m[, 2]) + as.numeric(m[, 3]) / as.numeric(m[, 4])   # 4 + 3/4 = 4.75

#> [1] 4.75 1.50   NA

• Capture group: a part of the pattern inside ( ). str_match() returns the whole match in column 1, then one column per group, in the order the groups open.
• The groups split “4-3/4” into 4, 3 and 4, so the last line can compute 4 + 3/4 = 4.75.
• “5” has no fraction, so the pattern finds no match and the whole row is NA.

Alternatives |

R Console; needs stringr

# phrases from the Fed statements of 8 Aug 2006, 22 Mar 2023 and 16 Dec 2008
phrases <- c("keep its target for the federal funds rate at 5-1/4 percent",
             "raise the target range for the federal funds rate to 4-3/4 to 5 percent",
             "establish a target range for the federal funds rate of 0 to 1/4 percent")
str_match(phrases, "(?:at|to|of) ([0-9/-]+) percent")

#>      [,1]               [,2]   
#> [1,] "at 5-1/4 percent" "5-1/4"
#> [2,] "to 5 percent"     "5"    
#> [3,] "to 1/4 percent"   "1/4"

• at|to|of matches at, to or of. (?: ) groups the three words without adding a column, so the number lands in column 2.
• Rows 2 and 3 return only the upper end of the range:
  • The pattern needs a space and “percent” right after the number.
  • After “to 4-3/4” comes “to 5”, so the search moves on and matches “to 5 percent”.
  • Getting both ends takes a second number group.

3 · Validation, extraction and replacement

The functions for each task

Task and result	stringr	base R
find, validate TRUE or FALSE	str_detect(x, p)	grepl(p, x)
count a number per string	str_count(x, p)	lengths(regmatches(x, gregexpr(p, x)))
extract first match, or NA	str_extract(x, p)	regmatches(x, regexpr(p, x))
groups match, then groups	str_match(x, p)	regmatches(x, regexec(p, x))
extract all a list of matches	str_extract_all(x, p)	regmatches(x, gregexpr(p, x))
replace the new strings	str_replace_all(x, p, r)	gsub(p, r, x)

Result: what the stringr function returns. x is the text, p the pattern and r the replacement. Base R puts the pattern first, and its extract functions drop strings with no match. Chatbots write either style.

Where base R and stringr behave differently:

stringr	base R	What differs
str_extract(x, p)	regmatches(x, regexpr(p, x))	base R drops strings with no match, so the result is shorter than x
str_match(x, p)	regmatches(x, regexec(p, x))	a list, not a matrix; an optional group that did not match is "", not NA
str_split(x, p)	strsplit(x, p)	strsplit() drops a final empty string
str_c(...)	paste0(...)	paste0() writes NA as “NA”; str_c() returns NA
regex(p, ignore_case = TRUE)	ignore.case = TRUE

Engine: the code that runs a pattern. Base R has two, and perl = TRUE picks the second.

The other functions used today:

stringr	base R
str_split(x, p)	strsplit(x, p)
str_sub(x, 1, 4)	substr(x, 1, 4)
str_to_lower(x)	tolower(x)

On all 171 statements, the grepl() and gregexpr() counts equal the stringr results with or without perl = TRUE. The two engines differ on the dot: the default engine’s . matches a line break, which stringr’s does only with dotall = TRUE; with perl = TRUE, . stops at a line break unless the pattern starts with (?s). So the reply’s first step, run through regexpr(), gives 23 NA with the default engine (the same as stringr with dotall) and 36 with perl = TRUE. perl = TRUE also adds lookarounds such as (?=...). The three changes written in base R give 170 of 171 with either engine (Practice 5).

Validating a format

R Console; needs stringr

fips <- c("06037", "6037", "36109", "3610A", " 36109")   # US county codes
str_detect(fips, "^\\d{5}$")         # exactly five digits, nothing before or after

#> [1]  TRUE FALSE  TRUE FALSE FALSE

str_detect(fips, "\\d{5}")           # without the anchors, " 36109" passes

#> [1]  TRUE FALSE  TRUE FALSE  TRUE

fips[!str_detect(fips, "^\\d{5}$")]  # the values that fail

#> [1] "6037"   "3610A"  " 36109"

dates <- c("2026-10-22", "10/22/2026", "2026-1-5")
str_detect(dates, "^\\d{4}-\\d{2}-\\d{2}$")   # year-month-day, two digits each

#> [1]  TRUE FALSE FALSE

• Validation: checking that every value has the format it should, before you merge or analyze the data.
• A FIPS code is the US government’s five-digit code for each US county: two digits for the state, three for the county. Read as a number, “06037” (Los Angeles County, California) becomes 6037, and a merge on it fails.

Extracting numbers

R Console; needs stringr

prices <- c("$1,250", "price: $99.99", "free", "$52,000")
str_extract(prices, "[0-9,.]+")             # digits, commas and periods

#> [1] "1,250"  "99.99"  NA       "52,000"

ages <- c("<1", "1-5", "5-10", "15+")        # age groups in a mortality table
str_match(ages, "(\\d+)(?:-(\\d+))?")         # group 1: the first number; group 2: the second

#>      [,1]   [,2] [,3]
#> [1,] "1"    "1"  NA  
#> [2,] "1-5"  "1"  "5" 
#> [3,] "5-10" "5"  "10"
#> [4,] "15"   "15" NA

• Extraction: keeping only the part of each string that matches.
• str_extract() returns the match as text.
• str_match() returns one column per group, so “5-10” splits into “5” and “10”. (?:-(\\d+))? makes the hyphen and the second number optional together; when they are missing, group 2 is NA.
• The pattern returns 1 for “<1” and 15 for “15+”. You code by hand that “<1” starts at 0 and that “15+” has no upper end.

Replacing and cleaning text

R Console; needs prices

amount <- str_extract(prices, "[0-9,.]+")       # "free" has no digits, so NA
as.numeric(str_replace_all(amount, ",", ""))    # drop the commas, then convert

#> [1]  1250.00    99.99       NA 52000.00

str_squish("  Federal   Reserve \n Board ")     # \n: a line break; result: one space between words

#> [1] "Federal Reserve Board"

# month/day/year becomes year-month-day
str_replace(c("10/22/2026", "01/05/2026"), "(\\d{2})/(\\d{2})/(\\d{4})", "\\3-\\1-\\2")

#> [1] "2026-10-22" "2026-01-05"

• Replacement: rewriting what the pattern matches. str_replace() changes the first match, str_replace_all() every match.
• Backreference: \\1, \\2 and \\3 in the replacement stand for what groups 1, 2 and 3 matched. The US writes dates month/day/year (10/22/2026), so the replacement puts the year first: 2026-10-22.
• as.numeric("1,250") gives NA, which is why the commas go first.

Testing a pattern

R Console; needs stringr

should_match <- c("the federal funds rate", "longer-term interest rates")
should_skip  <- c("at a moderate pace", "to generate sustained improvement")
str_detect(should_match, "\\brate\\b")

#> [1]  TRUE FALSE

str_detect(should_skip, "\\brate\\b")

#> [1] FALSE FALSE

str_detect(should_match, "\\brates?\\b")      # ? makes the "s" optional

#> [1] TRUE TRUE

str_detect(should_skip, "\\brates?\\b")

#> [1] FALSE FALSE

str_detect(should_skip, "rate")               # without \\b, "moderate" and "generate" match

#> [1] TRUE TRUE

• Test a pattern on strings with a known answer first. Goal here: the word “rate” or “rates”, and not “moderate” or “generate”.
• Should-match and should-skip tests: short vectors of real strings the pattern must catch and must leave alone, rerun after every change.
• \\brate\\b misses “rates” (false negative); "rate" catches both skip strings (false positives); \\brates?\\b passes both.
• Copy test strings from the documents. When the corpus shows a case the pattern gets wrong, add that string to one of the two vectors.

Exercise 1

Exercise

# Exercise 1 (5 minutes). R Console; needs stringr. Answers to a survey
# question on yearly income:
income <- c("$52,000", "48500", "Income: $61,250.00", "N/A", "about 40k",
            "$1,000,000")
# 1. Validation: write ok_pat, a pattern for an answer that is only an
#    amount: an optional $, then digits and commas, then optional cents
#    (.00). Run str_detect(income, ok_pat). Alone, $ means the end of the
#    text, so write \\$ for a dollar sign.
# 2. Extraction: pull the number out of every answer with str_extract().
# 3. Replacement: drop the commas with str_replace_all(), then as.numeric().
# 4. Read the result next to income. Which answers give NA or a wrong number?
#
# Two answers to compare with the room: how many answers pass step 1, and
# what "about 40k" becomes.

Solution

R Console; needs stringr

income <- c("$52,000", "48500", "Income: $61,250.00", "N/A", "about 40k",
            "$1,000,000")
ok_pat <- "^\\$?[0-9,]+(\\.\\d{2})?$"
str_detect(income, ok_pat)

#> [1]  TRUE  TRUE FALSE FALSE FALSE  TRUE

n <- str_extract(income, "[0-9,.]+")
as.numeric(str_replace_all(n, ",", ""))

#> [1]   52000   48500   61250      NA      40 1000000

4 · Patterns on 171 Fed statements

The Fed’s policy statements

• The Federal Reserve (“the Fed”) is the US central bank.
• Its Federal Open Market Committee (FOMC) sets the policy interest rate. It meets eight times a year and publishes a statement after each meeting. A few statements follow unscheduled meetings.
• The federal funds rate is the rate banks charge each other for overnight loans. The Fed steers it.
• The target range is the band the FOMC announces for that rate. “4-3/4 to 5 percent” means 4.75 to 5. Before 16 December 2008 the FOMC announced a single number.
• The file fomc_statements.csv has one row per statement: 171 statements, 31 Jan 2006 to 16 Sep 2026, 165 of them after scheduled meetings.

Screenshot: federalreserve.gov, statement of 22 March 2023 (public domain).

Finding a word in 171 statements

R Console, in the 16-text-regex folder; needs stringr; reads fomc_statements.csv (date, url, words, text, scheduled)

fomc <- read.csv("fomc_statements.csv")
dim(fomc)

#> [1] 171   5

c(inflation = sum(str_detect(fomc$text, "inflation")),
  Inflation = sum(str_detect(fomc$text, "Inflation")),
  either    = sum(str_detect(fomc$text, regex("inflation", ignore_case = TRUE))))

#> inflation Inflation    either 
#>       165        99       168

fomc$n_infl <- str_count(fomc$text, regex("inflation", ignore_case = TRUE))
summary(fomc$n_infl)

#>    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
#>   0.000   5.000   8.000   7.795  10.500  17.000

fomc$date[which(fomc$n_infl == 0)]   # which(): the positions where the condition is TRUE

#> [1] "2007-08-17" "2020-03-03" "2020-03-23"

• Literal pattern: a pattern with no metacharacters. Matching is case-sensitive: 165 statements contain “inflation”, 99 “Inflation”, 168 either.
• str_detect() gives TRUE or FALSE per statement, and sum() counts the TRUEs. str_count() gives the number of matches per statement, a new column n_infl (0 to 17, median 8).
• The three statements without “inflation” all followed unscheduled meetings: 17 Aug 2007, and 3 and 23 Mar 2020, as the coronavirus spread.

False positives in a large text

R Console; needs fomc

sum(str_count(fomc$text, "rate"))

#> [1] 982

sum(str_count(fomc$text, "\\brate\\b"))

#> [1] 705

caught <- unlist(str_extract_all(fomc$text, "[a-z]*rate[a-z]*"))   # every word containing "rate"
head(sort(table(caught), decreasing = TRUE), 6)

#> caught
#>       rate   moderate      rates moderately  moderated   generate 
#>        705        144         58         42         21          3

• The literal “rate” matches 982 times, the whole word 705 times. The other 277 are 58 “rates”, which you may want (\\brates?\\b keeps them), and 219 inside other words, such as “moderate” (144).
• A false positive is text the pattern catches but should skip. In a short test vector you see one at a glance. In 171 statements, a table of everything the pattern caught shows them.
• unlist() puts every statement’s matches in one vector, so table() can count them.

Splitting text into words

R Console; needs stringr and fomc

i <- which(fomc$date == "2023-03-22")                         # the row of the 22 Mar 2023 statement
fomc$year <- as.numeric(str_sub(fomc$date, 1, 4))             # the year, used in Exercise 2
tokens <- str_extract_all(str_to_lower(fomc$text), "[a-z]+")  # every run of letters
head(tokens[[i]], 12)

#>  [1] "recent"     "indicators" "point"      "to"         "modest"    
#>  [6] "growth"     "in"         "spending"   "and"        "production"
#> [11] "job"        "gains"

length(tokens[[i]])

#> [1] 364

• Token: one unit of text after splitting, here a word in lower case. Extracting every run of letters does the split: spaces, punctuation and numbers are left out.
• str_to_lower() comes first so “Inflation” and “inflation” count as one word. tokens is a list with one vector of words per statement, and [[i]] picks one.
• The apostrophe is not a letter, so “Committee’s” becomes two tokens, “committee” and “s”.

Word counts and stop words

R Console; needs tokens and the stopwords package

all_words <- unlist(tokens)
length(all_words)

#> [1] 73839

head(sort(table(all_words), decreasing = TRUE), 8)

#> all_words
#>       the       and        to        of committee        in inflation      will 
#>      5320      2854      2576      2382      1706      1534      1330       950

stop_words <- stopwords::stopwords("en")     # pkg::fun(): a package's function without library()
kept <- all_words[!all_words %in% stop_words & nchar(all_words) > 1]   # !x %in% y: x is not in y
head(sort(table(kept), decreasing = TRUE), 8)

#> kept
#> committee inflation  economic      rate   federal    policy    market   percent 
#>      1706      1330       842       705       686       649       642       619

• Stop words: very common words dropped before counting (“the”, “of”, “will”). The package’s default English list (Snowball) has 175.
• Term frequency: a word’s count divided by all tokens. “inflation” is 1,330 of 73,839, or 1.8 percent.
• nchar() counts characters. nchar(all_words) > 1 drops 1,793 more one-letter tokens: the s of “Committee’s” and “U.S.”, and middle initials such as the H in Jerome H. Powell (each statement ends by naming the members who voted).
• The stop list also drops “not” and “no”, which reverse a sentence, and “against”, which is on the negative word list used for tone.

Exercise 2

Exercise

# Exercise 2 (5 minutes). R Console; needs fomc (with its year column) and
# tokens from the previous slides.
# Which words appear in the statements for the first time in 2022?
#
# 1. Put the words of the 2022 statements in one vector:
#      w22 <- unlist(tokens[fomc$year == 2022])
# 2. Put the words of every statement before 2022 in a second vector
#    called before.
# 3. Keep the 2022 words that are not in the vector before (use %in% and !).
# 4. Count them with table() and sort them, largest first.
#
# Two answers to compare with the room: the most frequent new word, and the
# new words that are people's names.

Solution

R Console; needs fomc and tokens

w22    <- unlist(tokens[fomc$year == 2022])
before <- unlist(tokens[fomc$year < 2022])
new    <- w22[!w22 %in% before]
head(sort(table(new), decreasing = TRUE), 16)

#> new
#>       war    russia   ukraine attentive  invasion    issued      cook  creating 
#>         8         7         7         6         6         6         5         5 
#>         d jefferson      lisa         n    philip  weighing      barr   collins 
#>         5         5         5         5         5         5         4         4

A finance word list

R Console; reads lm_words.csv

lm_words <- read.csv("lm_words.csv")         # word, list: "positive" or "negative"; lower case
table(lm_words$list)

#> 
#> negative positive 
#>     2345      347

pos <- lm_words$word[lm_words$list == "positive"]
neg <- lm_words$word[lm_words$list == "negative"]
c("liability", "tax", "cost") %in% neg      # three words a general-purpose list calls negative

#> [1] FALSE FALSE FALSE

• Word list (dictionary): a fixed list of words tied to a category. Counting the words on a positive and a negative list gives a statement’s tone.
• Loughran & McDonald (JF, 2011) built the lists for 10-Ks, the yearly report every listed US company files with the Securities and Exchange Commission (the US regulator of stock markets).
• Finding. In 10-Ks from 1994 to 2008, a general-purpose list (the Harvard General Inquirer, built for psychology and sociology) flagged many words as negative. Almost three-fourths of them, such as tax, cost and liability, are not negative in financial text.
• Takeaway: a word list built for one kind of text misreads another. Shapiro & Wilson (REStud, 2022) used these lists on the Fed’s meeting transcripts, and the next slides apply them to the statements.

Free for use in academic research; commercial licences from the authors (sraf.nd.edu).

Net tone of each statement

Net tone: positive words minus negative words, per 100 tokens.

R Console; needs fomc, tokens, pos and neg

net_tone <- function(w) 100 * (sum(w %in% pos) - sum(w %in% neg)) / length(w)
j <- which(fomc$date == "2008-12-16")
w <- tokens[[j]]
w[w %in% neg]

#> [1] "deteriorated" "declined"     "strained"     "weakened"     "diminished"  
#> [6] "declines"     "weaker"       "weak"

w[w %in% pos]

#> [1] "stability"     "exceptionally"

net_tone(w)

#> [1] -1.488834

fomc$tone <- sapply(tokens, net_tone)        # sapply(): the function on each statement's words
fomc[c(which.min(fomc$tone), which.max(fomc$tone)), c("date", "tone")]   # lowest, highest

#>           date      tone
#> 24  2008-09-16 -2.727273
#> 130 2021-07-28  2.057613

On 16 Dec 2008, during the financial crisis, the committee cut its target to a range of 0 to 1/4 percent; the statement has 403 tokens. The two positive words come from set phrases: “price stability” and “exceptionally low”.

Drawing net tone over time

R Console; needs fomc with its tone column

par(mar = c(2.5, 4.5, 0.8, 0.5))               # margins in lines: bottom, left, top, right
plot(as.Date(fomc$date), fomc$tone, type = "h", ylim = c(-3.4, 2.8),
     xlab = "", ylab = "Net tone per 100 tokens", las = 1)    # "h": one bar per statement
abline(h = 0, col = "grey60")
lab <- which(fomc$date %in% c("2008-09-16", "2020-03-03", "2021-07-28"))
text(as.Date(fomc$date[lab]), fomc$tone[lab], c("16 Sep 2008", "3 Mar 2020", "28 Jul 2021"),
     pos = c(1, 3, 3), cex = 1, xpd = NA)

• type = "h" draws one vertical bar from 0 to each statement’s tone.
• text() labels three statements: the lowest (16 Sep 2008), the highest (28 Jul 2021) and 3 Mar 2020, a rate cut between scheduled meetings.
• pos = 1 puts a label below its bar and pos = 3 above it. xpd = NA lets a label run outside the box.

5 · A chatbot writes the extraction code

The prompt and the reply

I have a data frame in R called fomc with a column text that holds the full text of every FOMC statement since 2006, one statement per row. Each statement says what the Fed decided about its policy rate, for example: “the Committee decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent.” Write R code with stringr that extracts the target range from each statement into two numeric columns, lower and upper (here 4.75 and 5).

chat_reply.R in the zip: steps 1 to 3 of the reply's code, unchanged; the lines above them define num and to_num()

# 1. Isolate the part of the statement describing the decision on the funds rate
seg <- str_extract(
  fomc$text,
  regex("decided.*?federal funds rate.*?percent\\b", ignore_case = TRUE, dotall = TRUE)
)
# 2. Pull out "to/at/of X [to Y] percent"
pat <- str_c("\\b(?:to|at|of)\\s+(", num, ")(?:\\s+to\\s+(", num, "))?\\s+percent\\b")
m   <- str_match(seg, pat)
# 3. Convert to numeric
first  <- to_num(m[, 2])
second <- to_num(m[, 3])
fomc$lower <- first
fomc$upper <- ifelse(is.na(second), first, second)

The reply: “For your example, "...to 4-3/4 to 5 percent." gives lower = 4.75 and upper = 5.” The prompt shows one example sentence and gives no other statements to test the code on. What follows reads num, step 1 and step 2 in turn. The full reply is on the session page.

Reply from Claude (claude-sonnet-5-5) through Anthropic’s API, 20 October 2026, to the prompt shown. The same prompt in the chat at claude.ai can return different code.

NoteThe full reply, as it came back

#> Warning in readLines("chat_reply.md", encoding = "UTF-8"): incomplete final
#> line found on 'chat_reply.md'

library(stringr)

# A number as it appears in statements: "5", "1/4", "4-3/4", or a decimal like "0.25"
num <- "\\d+(?:\\.\\d+)?(?:-\\d+/\\d+|/\\d+)?"

# Convert strings like "4-3/4", "1/4", "5" to numeric
to_num <- function(x) {
  m <- str_match(x, "^(?:(\\d+)-)?(\\d+(?:\\.\\d+)?)(?:/(\\d+))?$")
  whole <- as.numeric(m[, 2])   # part before the hyphen (if any)
  a     <- as.numeric(m[, 3])   # whole number, decimal, or numerator
  b     <- as.numeric(m[, 4])   # denominator (if any)
  ifelse(is.na(b), a, a / b) + ifelse(is.na(whole), 0, whole)
}

# 1. Isolate the part of the statement describing the decision on the funds rate
seg <- str_extract(
  fomc$text,
  regex("decided.*?federal funds rate.*?percent\\b", ignore_case = TRUE, dotall = TRUE)
)

# 2. Pull out "to/at/of X [to Y] percent"
pat <- str_c("\\b(?:to|at|of)\\s+(", num, ")(?:\\s+to\\s+(", num, "))?\\s+percent\\b")
m   <- str_match(seg, pat)

# 3. Convert to numeric
first  <- to_num(m[, 2])
second <- to_num(m[, 3])

fomc$lower <- first
fomc$upper <- ifelse(is.na(second), first, second)

For your example, "...to 4-3/4 to 5 percent." gives lower = 4.75 and upper = 5.

How it works - The first step grabs the text from “decided” through the first “percent” that follows “federal funds rate”. This keeps it from picking up other numbers in the statement, such as the 2 percent inflation goal, “25 basis points”, or the discount rate. - percent\\b means “percentage point” in phrases like “by 1/2 percentage point, to 1 to 1-1/4 percent” doesn’t end the match early. - to_num() handles whole numbers (5), fractions (1/4), and mixed numbers (4-3/4).

Cases to know about - Pre-December 2008 statements set a single target, such as “…to 5-1/4 percent”, not a range. The code sets lower and upper both to that value. If you’d rather have lower be NA there, use: r fomc$lower <- ifelse(is.na(second), NA_real_, first) - Statements with no rate decision (a few unscheduled statements about liquidity facilities, for example) return NA. - Quick check: run these to see what failed and to look at the extremes: r fomc[is.na(fomc$upper), "text"] summary(fomc[, c("lower", "upper")]) Also check that lower <= upper everywhere.

Reading the number pattern

R Console, in the 16-text-regex folder; needs stringr and fomc

source("chat_reply.R")      # runs the reply's code, unchanged
str_view(c("5", "1/4", "4-3/4", "0.25"), num)

#> [1] │ <5>
#> [2] │ <1/4>
#> [3] │ <4-3/4>
#> [4] │ <0.25>

to_num(c("4-3/4", "1/4", "5", "0"))

#> [1] 4.75 0.25 5.00 0.00

• num: the reply’s pattern for one number as the Fed writes it (“5”, “1/4”, “4-3/4”). Step 2 uses it twice, for the lower and the upper limit.
• source() runs every line of a script file. chat_reply.R defines num, to_num(), seg and pat, and adds lower and upper to fomc.
• Known-answer test: inputs whose right output you know before running. In str_view() all four strings match in full, and to_num() returns 4.75, 0.25, 5 and 0, as expected.
• to_num() (in chat_reply.R) splits “4-3/4” into 4, 3 and 4 and returns 4 + 3/4.

Greedy and lazy matching

R Console; needs fomc and seg from chat_reply.R

i <- which(fomc$date == "2023-03-22")
greedy <- str_extract(fomc$text[i], regex("decided.*percent", dotall = TRUE))
lazy   <- str_extract(fomc$text[i], regex("decided.*?percent", dotall = TRUE))
c(nchar(greedy), nchar(lazy))       # nchar() counts characters

#> [1] 926  82

lazy

#> [1] "decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent"

identical(lazy, seg[i])             # the reply's step 1 gives the same text

#> [1] TRUE

• Step 1 of the reply cuts the decision sentence out of each statement with lazy .*?.
• Greedy .* takes as many characters as it can. Here greedy runs 926 characters, to the last “percent” in the statement (“…returning inflation to its 2 percent”).
• Lazy .*? takes as few as it can.
• dotall = TRUE lets . match a line break (“\n”). Paragraphs in text are separated by line breaks, so with dotall a match can run from one paragraph into the next.
• Why the reply stops at the first “percent” after “federal funds rate”: “This keeps it from picking up other numbers in the statement, such as the 2 percent inflation goal, ‘25 basis points’, or the discount rate.” A basis point is 0.01 percentage point; the discount rate is the rate the Fed charges banks that borrow from it.

Reading the range pattern

R Console; needs seg, pat and i

str_match(seg[i], pat)              # the reply's step 2

#>      [,1]                    [,2]    [,3]
#> [1,] "to 4-3/4 to 5 percent" "4-3/4" "5"

• str_c() joins strings end to end (base R paste0()). pat pastes in num, the previous slide’s number pattern, once per limit.
• On 22 March 2023, group 1 is “4-3/4” (the lower limit) and group 2 is “5” (the upper). to_num() turns them into 4.75 and 5.
• The second group is optional. A single target such as “at 5-1/4 percent”, the form used before December 2008, also matches, and the reply then sets upper equal to lower.

The reply’s tests on 171 statements

R Console; needs fomc after source("chat_reply.R")

sum(is.na(fomc$upper))                         # the reply's test 1 prints these rows; here we count them

#> [1] 32

summary(fomc[, c("lower", "upper")])           # the reply's test 2

#>      lower           upper      
#>  Min.   :0.000   Min.   :0.250  
#>  1st Qu.:0.000   1st Qu.:0.250  
#>  Median :1.000   Median :1.250  
#>  Mean   :1.939   Mean   :2.128  
#>  3rd Qu.:4.250   3rd Qu.:4.500  
#>  Max.   :5.250   Max.   :5.500  
#>  NAs    :32      NAs    :32

sum(fomc$lower > fomc$upper, na.rm = TRUE)     # the reply's test 3: lower <= upper

#> [1] 0

• The reply suggests three tests: print the NA rows (fomc[is.na(fomc$upper), "text"]), run summary(), and verify lower <= upper.
• It expects NA only for “a few unscheduled statements about liquidity facilities”. There are 32, and reading them shows wordings the pattern misses.
• summary() shows values from 0 to 5.5 percent, all plausible for a policy rate, and no row has lower above upper.
• The tests cannot say whether the other 139 values are right.

Comparing with FRED

R Console; needs fomc after the reply's code; reads fred_target_range.csv

fred <- read.csv("fred_target_range.csv")                # date, lower, upper: one row per day
k <- match(as.Date(fomc$date) + 1, as.Date(fred$date))    # + 1: the day after each statement
fomc$fred_lower <- fred$lower[k]
fomc$fred_upper <- fred$upper[k]
fomc$agree <- fomc$lower == fomc$fred_lower & fomc$upper == fomc$fred_upper
table(fomc$agree, useNA = "ifany")

#> 
#> FALSE  TRUE  <NA> 
#>     9   130    32

fomc$date[which(!fomc$agree)]

#> [1] "2014-03-19" "2014-04-30" "2014-06-18" "2014-07-30" "2014-09-17"
#> [6] "2014-10-29" "2019-12-11" "2020-01-29" "2022-06-15"

• FRED, the free data service of the Federal Reserve Bank of St. Louis, gives the target for every day: DFEDTAR, the single target to 15 Dec 2008 (copied into both columns), then DFEDTARL and DFEDTARU, the lower and upper limits.
• match() finds the position of each statement’s next day among FRED’s dates. Adding 1 to a date gives the next day. Since 2017 a new range takes effect the day after the statement, so the next day is the first day FRED shows it.
• agree is TRUE for 130 statements (both limits equal FRED’s), FALSE for 9 and NA for 32 (no number found). which() skips NA, so only the nine wrong statements print. In all nine, lower equals upper, so the reply’s lower <= upper test passes on every one.

Drawing the comparison with FRED

R Console; needs fred, and fomc with the reply's lower and upper and the FRED columns

d <- as.Date(fomc$date) + 1                    # the day after each statement
ok <- fomc$agree %in% TRUE                     # %in% TRUE: NA counts as not TRUE
bad <- fomc$agree %in% FALSE
par(mar = c(2.5, 4.5, 0.5, 0.5))               # margins in lines: bottom, left, top, right
plot(as.Date(fred$date), fred$upper, type = "s", col = "grey60", ylim = c(0, 6),
     xlab = "", ylab = "Target, percent", las = 1)    # "s": steps, flat between decisions
lines(as.Date(fred$date), fred$lower, type = "s", col = "grey60")
points(d[ok], fomc$upper[ok], pch = 16, cex = 0.7)
points(d[bad], fomc$lower[bad], pch = 4, cex = 1.4, lwd = 2, col = "firebrick")
segments(d[bad], fomc$fred_lower[bad], d[bad], fomc$lower[bad], col = "firebrick", lwd = 2)
rug(d[is.na(fomc$agree)], col = "darkorange", lwd = 2, ticksize = 0.04)   # a tick for each NA
label_x <- as.Date(c("2010-07-01", "2013-06-01", "2015-08-01", "2021-03-01", "2018-10-01", "2025-10-01"))
label_y <- c(0.8, 1.05, 1.65, 1.0, 0.55, 1.0)  # labels placed by hand at (date, percent)
text(label_x, label_y, cex = 0.9,
     c("\"will maintain\"", "2014: the match\nruns too far", "\"reaffirmed\"",
       "look-alike\nhyphen", "look-alike\nhyphen", "look-alike\nhyphen"),
     col = c("darkorange", "firebrick", "darkorange", "firebrick", "darkorange", "darkorange"))
legend("top", ncol = 2, bty = "n", cex = 0.85,
       legend = c("FRED: lower and upper", paste0("equal (", sum(ok), ")"),
                  paste0("wrong (", sum(bad), "): lower = upper"),
                  paste0("NA (", sum(is.na(fomc$agree)), ")")),
       col = c("grey60", "black", "firebrick", "darkorange"),
       lty = c(1, NA, 1, 1), lwd = c(1, NA, 2, 2), pch = c(NA, 16, 4, NA))

Where the reply’s code fails

What the statement says	Count	Dates	Code returns
“will maintain the target range … at 0 to 1/4 percent”: no “decided”, which step 1 needs	13	18 Mar 2009 to 21 Sep 2010	NA
“reaffirmed its view that the current 0 to 1/4 percent target range …”	8	17 Dec 2014 to 28 Oct 2015	NA
the range only in the instruction to the New York Fed (a regional Fed bank)	1	23 Mar 2020	NA
no target stated	1	17 Aug 2007	NA (correct)
a look-alike hyphen inside the upper number (“3‑3/4”; next slide)	9	13 Dec 2017 to 29 Apr 2026	NA
a look-alike hyphen inside the lower number (“1‑1/2”; next slide)	3	11 Dec 2019, 29 Jan 2020, 15 Jun 2022	1.75 to 1.75; FRED: 1.5 to 1.75
“decided” is first about asset purchases, so the match runs on (up to 2,354 characters, across paragraphs) to the first “to 1/4 percent”	6	19 Mar to 29 Oct 2014	0.25 to 0.25; FRED: 0 to 0.25

Two hyphens that look the same

R Console; needs fomc, seg and num from chat_reply.R

j <- which(fomc$date == "2025-12-10")
str_view(seg[j], num)

#> [1] │ decided to lower the target range for the federal funds rate by <1/4> percentage point to <3-1/2> to <3>‑<3/4> percent

utf8ToInt(str_extract(seg[j], "3.3/4"))     # . matches either hyphen; utf8ToInt(): each code point

#> [1]   51 8209   51   47   52

utf8ToInt("3-3/4")                           # typed with the hyphen key

#> [1] 51 45 51 47 52

sum(str_detect(fomc$text, "\u2011"))         # statements with character 8209

#> [1] 34

In the str_view() output, “3-1/2” is one match but “3‑3/4” is two, <3> and <3/4>: num stops at the look-alike hyphen, so the row is NA.

• Code point: the number a computer stores for each character. Unicode is the standard list of them.
• Character 8209: a non-breaking hyphen. Web pages use it to keep “3‑3/4” on one line. In R, "\u2011" writes it: 2011 is 8209 in hexadecimal (base 16), the form Unicode lists use.
• The - in a pattern matches only character 45.

Characters and encodings. A file stores each character as bytes, and an encoding is the rule that turns bytes back into characters. UTF-8 is the rule the statements file uses and the default in current R. Read with the wrong rule, the same bytes give different characters: reading the file with read.csv("fomc_statements.csv", fileEncoding = "CP1252") turns “Today’s” into “Today’s”. The fix is to read the file again with the right rule (encoding = "UTF-8"), which gives the text back exactly; converting the damaged text back with iconv() does not restore every character.

Fixing the reply’s code

R Console; needs fomc, num, to_num() and the FRED columns

# 1. the look-alike hyphen (character 8209) becomes "-"
fomc$text <- str_replace_all(fomc$text, "\u2011", "-")
# 2. stop at the period; 3. three verbs, no "federal funds rate"
seg <- str_extract(fomc$text, regex("(?:decided|maintain|reaffirmed)[^.]*?percent\\b",
                                    ignore_case = TRUE))
# 3. no "to/at/of" before the number
m <- str_match(seg, str_c("(", num, ")(?:\\s+to\\s+(", num, "))?\\s+percent\\b"))
fomc$lower <- to_num(m[, 2])
fomc$upper <- ifelse(is.na(m[, 3]), fomc$lower, to_num(m[, 3]))
fomc$agree <- fomc$lower == fomc$fred_lower & fomc$upper == fomc$fred_upper
table(fomc$agree, useNA = "ifany")

#> 
#> TRUE <NA> 
#>  170    1

• Change 1: every look-alike hyphen (character 8209) becomes “-”. This fixes the 12 hyphen rows.
• Change 2: [^.]*?, as few characters as possible that are not a period, replaces .*?, and dotall goes. Inside brackets the dot is a plain period, so the match ends with its sentence. The six 2014 rows now give NA in place of a wrong number.
• Change 3: step 1 accepts “maintain” and “reaffirmed” and drops “federal funds rate”, which can come after the number (“the current 0 to 1/4 percent target range for the federal funds rate”). Step 2 drops “to|at|of”, since “current 0 to 1/4” has none.

Results after each fix

Each row adds one change to the rows above. Counted against FRED on the day after each statement:

Code	NA	Wrong	Equal
The reply as given	32	9	130
1. Replace the look-alike hyphen with “-”	23	6	142
2. Keep the match in one sentence	36	0	135
3. Accept “maintain” and “reaffirmed”; take the range wherever it sits in the sentence	1	0	170

• Change 2 turns the six 2014 values into NA. It does the same to seven statements from Nov 2010 to Jan 2014, where “decided” is about asset purchases and the range sits in a later sentence. The old match reached it only by crossing sentences.
• The last NA is 17 Aug 2007, which states no target.
• To ask the chatbot for a fix, paste the failing sentences with FRED’s numbers for each. They are known-answer tests.

The two middle rows of the table, run on a fresh copy of the reply’s output:

R Console, in the 16-text-regex folder; needs stringr; reads the files again

f <- read.csv("fomc_statements.csv")
f$text <- str_replace_all(f$text, "\u2011", "-")
k2 <- match(as.Date(f$date) + 1, as.Date(fred$date))
count_rows <- function(lo, up) {
  ag <- lo == fred$lower[k2] & up == fred$upper[k2]
  c(missing = sum(is.na(ag)), wrong = sum(ag %in% FALSE), equal = sum(ag %in% TRUE))
}
s1 <- str_extract(f$text, regex("decided.*?federal funds rate.*?percent\\b",
                                ignore_case = TRUE, dotall = TRUE))   # after change 1
m1 <- str_match(s1, pat)
count_rows(to_num(m1[, 2]), ifelse(is.na(m1[, 3]), to_num(m1[, 2]), to_num(m1[, 3])))

#> missing   wrong   equal 
#>      23       6     142

s2 <- str_extract(f$text, regex("decided[^.]*?federal funds rate[^.]*?percent\\b",
                                ignore_case = TRUE))                  # after change 2
m2 <- str_match(s2, pat)
count_rows(to_num(m2[, 2]), ifelse(is.na(m2[, 3]), to_num(m2[, 2]), to_num(m2[, 3])))

#> missing   wrong   equal 
#>      36       0     135

6 · When to use regex

When to use regex

Regex works well when

• the text follows a format: IDs, dates, amounts, codes
• the wording repeats, as in the Fed’s sentence that states the target range
• you can list the variants and test each one
• you need the same answer every time the code runs
• the corpus is large: a pattern runs on millions of documents in seconds, for free

Use a parser or a model when

• the meaning depends on context or on negation: the Fed’s “might not be strong enough” (13 Sep 2012) scores as positive
• the same idea is written in more ways than you can list
• the text is a web page (HTML) or data from an API (JSON), where parts sit inside other parts
• the variable is a judgment call, such as whether a sentence signals that the Fed will raise its rate

Parser: a function that reads a structured format into its parts, such as jsonlite::fromJSON() for JSON or rvest::read_html() for a web page. Model: a language model, the kind of program behind a chatbot, asked to read each text and return a label, or a statistical classifier trained on texts that people labeled by hand.

A pattern gives NA or a wrong number, with no error message, when the wording or a character changes. A chatbot can write the pattern. You still read it, test it on strings from your documents, and compare its output with a second source such as FRED.

What net tone misses

The target range has a second source, FRED, and 170 of 171 values now equal it. Net tone has no second source, so its errors show only when you read the sentences it scored. The finance word list scores each word on its own:

R Console; needs fomc

s <- str_extract(fomc$text[fomc$date == "2012-09-13"], "economic growth might not[^.]*")
str_view(s, "\\b(not|strong|improvement)\\b")

#> [1] │ economic growth might <not> be <strong> enough to generate sustained <improvement> in labor market conditions

• Negation. 13 Sep 2012 (also 24 Oct and 12 Dec 2012): “strong” and “improvement” count as positive. Each word is scored on its own, so the “not” before them changes nothing.
• Context. 25 Apr 2012, “the unemployment rate has declined but remains elevated”: “unemployment” and “declined” count as negative, although a falling unemployment rate is good news.
• The event behind the words. 3 Mar 2020, a cut in the target range between scheduled meetings as the coronavirus spread, is the 8th most positive of 171 statements. “strong”, “achieving” and “stability” count as positive. “coronavirus” and “risks” are on neither list.

Words the finance list misreads in Fed statements.

Word	List	Times	In a Fed statement
unemployment	negative	154	a statistic: “unemployment rate” (141)
against	negative	86	the vote: “Voting against” (75)
impede	negative	46	one set phrase about risks to the committee’s goals
declined	negative	45	good news in “the unemployment rate declined” (18 statements)
stability	positive	223	the goal: “price stability” (218)
exceptionally	positive	56	always “exceptionally low”, about the rate

Dropping unemployment, against, stability and exceptionally leaves the lowest statement unchanged and makes 12 Jun 2024 the highest (Practice 8).

Practice

Task 7 uses the word lists of Baker, Bloom & Davis (QJE, 2016).

# Practice. Run in the 16-text-regex folder with stringr loaded, after the
# slides' code has built fomc (with the three changes), tokens, pos, neg
# and the FRED columns.

# Patterns and numbers
#  1. Label each statement by its opening with ^: "Information received",
#     "Recent indicators", or "other". Count them by year with table().
#  2. Run str_match(fomc$text, "(\\d+) basis points") and table() column 2.
#     Read each matching sentence. Which matches are not the committee's
#     decision? Change the pattern so only decisions match.
#  3. The 2026 statements give the vote ("by a 12 – 0 vote"). Extract the
#     two numbers as numeric columns. Why does "by a (\\d+) - (\\d+) vote"
#     fail? Use utf8ToInt() on the character between the numbers.
#  4. Validate the url column: every address should look like
#     https://www.federalreserve.gov/newsevents/pressreleases/monetaryYYYYMMDDa.htm
#     Write the pattern with ^, \d{8} and $. Which addresses fail, and why?
# Base R
#  5. Rewrite the three changes with gsub(), regmatches() with regexpr(),
#     and regexec(). Do the same 170 statements equal FRED?
# Word counts
#  6. For each statement, compute the share of its tokens that are
#     "inflation". Average it by year with tapply(share, fomc$year, mean),
#     which applies mean() within each year, and plot it. Which year is
#     highest, which lowest?
#  7. Apply the core words of the three lists of Baker, Bloom & Davis
#     (QJE, 2016) to the statements with \\b and |. How many statements
#     match each list, and all three, and in which years?
# Tone
#  8. Recompute net tone after removing "unemployment" and "against" from
#     neg, and "stability" and "exceptionally" from pos. Which statements
#     are now lowest and highest? Where does 3 Mar 2020 land?
#  9. List the word after every "not" with str_match_all(), which returns
#     str_match()'s matrix for every match in each statement. How many are
#     on the positive list, and how many on the negative list?
# Characters
# 10. List every character with a code point above 127, with its code
#     point, its count, and the statements it appears in. Which would you
#     replace before matching?

NoteSolutions

R Console; needs fomc, tokens, pos, neg, num, to_num() and fred

# 1. Openings by year
op <- ifelse(str_detect(fomc$text, "^Information received"), "Information received",
      ifelse(str_detect(fomc$text, "^Recent indicators"), "Recent indicators", "other"))
table(op)

#> op
#> Information received                other    Recent indicators 
#>                   88                   60                   23

table(fomc$year, op)[c("2009", "2019", "2020", "2022", "2026"), ]

#>       op
#>        Information received other Recent indicators
#>   2009                    7     1                 0
#>   2019                    8     0                 0
#>   2020                    1     9                 0
#>   2022                    0     4                 4
#>   2026                    0     5                 1

# 2. Basis points: 20 matches, 7 of them in dissents
bp <- str_match(fomc$text, "(\\d+) basis points")
table(bp[, 2])

#> 
#> 25 50 75 
#> 14  4  2

dec <- str_match(fomc$text, "decided[^.]*?(\\d+) basis points")
table(dec[, 2])

#> 
#> 25 50 75 
#>  7  4  2

# 3. The vote: the character between the numbers is an en dash, 8211
vt <- str_match(fomc$text, "by a (\\d+) (.) (\\d+) vote")
vt[!is.na(vt[, 1]), ]

#>      [,1]               [,2] [,3] [,4]
#> [1,] "by a 12 – 0 vote" "12" "–"  "0" 
#> [2,] "by a 9 – 3 vote"  "9"  "–"  "3" 
#> [3,] "by a 12 – 0 vote" "12" "–"  "0"

utf8ToInt(vt[!is.na(vt[, 1]), 3][1])

#> [1] 8211

# 4. Statement addresses: three end in "b.htm", a second release that day
url_ok <- str_detect(fomc$url,
  "^https://www\\.federalreserve\\.gov/newsevents/pressreleases/monetary\\d{8}a\\.htm$")
fomc$url[!url_ok]

#> [1] "https://www.federalreserve.gov/newsevents/pressreleases/monetary20070817b.htm"
#> [2] "https://www.federalreserve.gov/newsevents/pressreleases/monetary20080122b.htm"
#> [3] "https://www.federalreserve.gov/newsevents/pressreleases/monetary20081216b.htm"

# 5. The three changes in base R
tx  <- gsub("\u2011", "-", fomc$text)
sg  <- regmatches(tx, regexpr("(?:decided|maintain|reaffirmed)[^.]*?percent\\b", tx,
                              ignore.case = TRUE, perl = TRUE))
has <- grepl("(?:decided|maintain|reaffirmed)[^.]*?percent\\b", tx,
             ignore.case = TRUE, perl = TRUE)
seg_b <- rep(NA_character_, length(tx))
seg_b[has] <- sg
mb <- regmatches(seg_b, regexec(paste0("(", num, ")(?:\\s+to\\s+(", num, "))?\\s+percent\\b"),
                                seg_b, perl = TRUE))
lo <- sapply(mb, function(v) if (length(v)) to_num(v[2]) else NA)
hi <- sapply(mb, function(v) if (length(v) && v[3] != "") to_num(v[3]) else NA)
hi <- ifelse(is.na(hi), lo, hi)
table(lo == fomc$fred_lower & hi == fomc$fred_upper, useNA = "ifany")

#> 
#> TRUE <NA> 
#>  170    1

# 6. "inflation" as a share of tokens, by year
share <- sapply(tokens, function(w) mean(w == "inflation"))
by_year <- tapply(share, fomc$year, mean)
round(100 * by_year[c(which.max(by_year), which.min(by_year))], 2)

#> 2018 2009 
#> 2.82 0.49

plot(as.numeric(names(by_year)), 100 * by_year, type = "b", xlab = "",
     ylab = "\"inflation\", percent of tokens", las = 1)
# 7. The three word lists of Baker, Bloom & Davis
E <- str_detect(fomc$text, regex("\\beconom(y|ic)\\b", ignore_case = TRUE))
U <- str_detect(fomc$text, regex("\\buncertain(ty)?\\b", ignore_case = TRUE))
P <- str_detect(fomc$text, "\\b(Congress|deficit|Federal Reserve|legislation|regulation|White House)\\b")
c(economy = sum(E), uncertainty = sum(U), policy = sum(P), all_three = sum(E & U & P))

#>     economy uncertainty      policy   all_three 
#>         171          41          53           8

table(fomc$year[E & U & P])

#> 
#> 2007 2008 2022 2026 
#>    2    2    1    3

# 8. Tone without the four misread words
pos2 <- setdiff(pos, c("stability", "exceptionally"))
neg2 <- setdiff(neg, c("unemployment", "against"))
tone2 <- sapply(tokens, function(w) 100 * (sum(w %in% pos2) - sum(w %in% neg2)) / length(w))
round(cor(fomc$tone, tone2), 2)

#> [1] 0.92

fomc$date[c(which.min(tone2), which.max(tone2))]

#> [1] "2008-09-16" "2024-06-12"

match(which(fomc$date == "2020-03-03"), order(-tone2))     # its rank, most positive first

#> [1] 43

# 9. The word after "not"
nm <- str_match_all(str_to_lower(fomc$text), "\\bnot ([a-z]+)")
after_not <- unlist(lapply(nm, function(x) x[, 2]))
length(after_not)

#> [1] 38

c(positive = sum(after_not %in% pos), negative = sum(after_not %in% neg))

#> positive negative 
#>        4        1

# 10. Characters above code point 127
ch <- unlist(strsplit(read.csv("fomc_statements.csv")$text, ""))
odd <- ch[sapply(ch, utf8ToInt) > 127]
sort(table(sapply(odd, utf8ToInt)), decreasing = TRUE)

#> 
#> 8209 8217 8211 8212 8195 
#>   50   25    3    2    1

Quick reference

Pattern pieces: . any character · \\. a period · \\d a digit · \\s a space or line break · [ ] one of a set · [^ ] one not in the set · + one or more · * zero or more · ? zero or one · {n} exactly n · .*? as few as possible · ^ start · $ end · \\b a word boundary · ( ) a capture group · (?: ) a group with no column · | either side · regex(p, ignore_case = TRUE, dotall = TRUE).

Functions: str_detect, str_count, str_view, str_extract, str_extract_all, str_match, str_replace_all, str_sub, str_c, str_to_lower, regex, utf8ToInt, match, table, sort, unlist, which, is.na, %in%, sapply, which.min, which.max.

The word list. The shipped file keeps the words whose Positive or Negative flag in the master dictionary is a year above 0, because a negative year marks a word removed from the list. It leaves out 17 removed words, so it has 347 positive and 2,345 negative words; Shapiro and Wilson used an earlier version. The master dictionary has five more lists (uncertainty, litigious, strong modal, weak modal, constraining).

Sources. Baker, Bloom & Davis (QJE, 2016); Shapiro & Wilson (REStud, 2022), “Taking the Fed at its Word: A New Approach to Estimating Central Bank Objectives using Text Analysis” (method details from FRBSF Working Paper 2019-02); Djourelova (AER, 2023); Loughran & McDonald (JF, 2011) and their master dictionary. FRED, Federal Reserve Bank of St. Louis: DFEDTAR, DFEDTARL, DFEDTARU. Board of Governors of the Federal Reserve System, federalreserve.gov (public domain; disclaimer).

The statements were collected from federalreserve.gov in October 2026, FRED’s series were downloaded the same month, and the word list is the authors’ March 2026 update.

Links: slides · stringr · Syllabus