AEM 6850

Empirical Methods for Applied Economists

Prof. Ariel Ortiz-Bobea

15 · Channels II: scraping

Tuesday, October 20, 2026

Plan for today

1. When to scrape
   Where to look first · Zillow · three papers
2. The ways to collect a site
   What to ask an agent · CSS selectors · dynamic pages · six routes
3. What to take into account
   Terms · robots.txt · personal data · user agent · fetch_page()
4. AI tools, in R or Python
   A brief · the demonstration starts · R or Python
5. Verifying what a scrape returns
   Two other lists · a seeded sample · an agent’s log
6. The demonstration’s results
   Marathon finish times · the R file against the Python file
7. A scrape someone else can re-run
   The project folder · the Wayback Machine · Wikipedia

The install line is on the next slide; ask everyone to run it now. Part 1 needs no R. The zip link on this slide and on the Part 2 divider is the same file as on the session page. The demonstration starts at the beginning of Part 4 and is read in Part 6: two terminal windows, ready before class (run sheet in the speaker notes of the brief slide).

1 · When to scrape

• Three ways to get data from a website
• The Fed’s policy statements, the example in today’s code
• Where to look before writing a scraper
• Where researchers get Zillow’s housing data
• Three papers built on scraped data

During Part 1, in the R Console: install.packages(c("rvest", "httr2", "jsonlite")). Before Part 2: unzip 15-scraping.zip and open 15-scraping.Rproj.

Three ways to get data from a website

• Web scraping: a program reads pages built for people and copies chosen pieces into a table.
• Feed or API: a service that answers a program with data in a fixed structure.
• HTML: the text a web page is written in.

Scrape when the first two ways do not give what you need.

The feed holds 4,648 releases in the saved copy; your number will be larger. The interest-rate page (H.15) offers its data files from its own page.

Today’s code and exercises collect the Fed’s policy statements

The statement of 15 June 2022, as the site shows it.

The Federal Reserve is the US central bank. Its Federal Open Market Committee (FOMC) sets the policy interest rate at eight scheduled meetings a year and publishes a written statement after each one.

The question: what has the committee said after each meeting since 2006?

The text of the statements exists only on these pages. The code in Parts 2 and 3 and both exercises work toward one file with every statement since 2006, fomc_statements.csv.

Where to look before writing a scraper

Look in this order. A scraper is the last step.

Look for	What it holds for the Fed’s policy statements
1. A dataset someone already built (a replication package, ICPSR, a vendor)	a paper’s files stop at the end of its sample; this session needs every statement to the latest one
2. A file to download on the site	interest rates as CSV files on the H.15 page; no statement text
3. An API or feed	json/ne-press.json: every release’s date, title and address; no text
4. A licence or an agreement with the publisher	not needed: the Board’s pages are public domain
5. The pages themselves	the only place the text exists, so scrape

With an agent. Before you brief a scrape, ask an assistant where else the data might be: a replication package, an archive, a download, an API. Then confirm each lead on the site yourself.

Where researchers get Zillow’s housing data

Zillow lists homes for sale and for rent across the US. Its listing pages are a case for not writing a scraper.

• The pages are closed to programs. The terms forbid “automated queries (including screen and database scraping, spiders, robots, crawlers …)”. The robots.txt disallows /homes/. One polite request from R met a press-and-hold human check.
• The listings carry personal data: agents’ names and phone numbers, photos of people’s homes.
• The data come by the routes on the last slide.
  • A download: Zillow Research’s CSV files of home values, rents and inventory.
  • A dataset someone built: Zillow’s transaction records (ZTRAX), through ICPSR.
  • A licence: listings from an MLS (the shared database real-estate agents list homes in) or from a data vendor.

The request was to a page robots.txt allows (the research data page) and came back “Access to this page has been denied” (October 2026). Part 3 explains terms, robots.txt and the human check.

If a student raises the law: terms bind most clearly once you sign in. A US court found that scraping public pages while logged out did not breach Meta’s terms (Meta Platforms v. Bright Data, N.D. Cal., 2024), and the hiQ judgment turned on fake accounts. If a student raises the rental-housing study coming up: a study like that goes through the IRB and the university’s counsel before any collection.

Three papers built on scraped data

Paper	What a program collected	What they found
Cavallo (AER, 2017)	online prices at 56 retailers in 10 countries, matched to the shelf prices workers scanned in the stores	the two prices were identical about 72 percent of the time
Kuhn & Shen (QJE, 2013)	1,051,706 ads from a Chinese job board, saved by a crawler	ads asking for one sex were common, and much rarer in jobs that need more skill
Christensen & Timmins (QJE, 2023)	rental listings in five US metro areas; then inquiries from names that signal race	discrimination cost African American renters the equivalent of 4.4 percent of annual income

In each, what the authors needed existed only on pages built for people.

2 · The ways to collect a site

• What you ask an AI agent for, and what you verify
• The Fed’s meeting calendar
• HTML, the browser inspector, CSS selectors
• read_html(), html_elements(), html_text2(), html_attr()
• Exercise 1
• Dynamic pages, and the file they load
• Six routes to every page on a site

Before this part: unzip 15-scraping.zip and open 15-scraping.Rproj.

What you ask an AI agent for, and what you verify

You will usually scrape with an AI coding agent: it writes the fetch, the selectors and the loop. Parts 2 and 3 show each piece so that you can ask for it and verify it. No function needs memorizing.

Ask for, in the brief	Verify, in what comes back
The unit, and the rule that decides what counts	its count against a second list: the site’s own, a feed, a stated total
The start pages and the route to every page	each selector, run on a saved page, against what the browser shows
A user agent with your address, one request a second, each page saved once, a log	each rule in its code; its report against its log
What to do at a 403, a CAPTCHA or a login page	the status codes in its log
The fields to keep, and no personal data	its printed output and its file
The numbers to print when it is done	those numbers against the browser, the site’s totals, a paper

Each slide that follows ends with what its piece means when an agent does the work.

The Fed’s meeting calendar

• The FOMC (the Fed’s policy committee) issues a statement on the last day of each meeting.
• One page, federalreserve.gov/monetarypolicy/fomccalendars.htm, lists 2021 to 2027: one panel (box) per year, one row per meeting.
• Each row links the statement (PDF and HTML), an implementation note (the technical instructions that go with the decision), the press conference and the minutes (the written record of the meeting).
• Each year from 2006 to 2020 has its own page. Each statement has its own page.

With an agent. Name these start pages in the brief: the calendar for 2021 on, and the year-page address for 2006 to 2020. Together they list every meeting.

Unscheduled calls, and votes taken in writing (notation votes), happen in crises; the 2025 panel’s last row is one, “August 22 (notation vote)”.

Reading a page’s HTML with the browser inspector

• Element: one labelled piece of the page. Its tag is the label in angle brackets (div, a); a class names elements of one kind (fomc-meeting); an id names one element only (article); an attribute is a named value in a tag, such as a link’s address, href.
• Inspect (right-click a meeting date on the live page) shows the HTML behind what you point at. View Source shows the file the server sent, which is what R reads.

With an agent. Inspect the element yourself before you accept an agent’s selector, and use these words to tell it which element you mean. A chat assistant given the element’s HTML can suggest a selector built on its class.

Picking out elements with CSS selectors

CSS selector: a short pattern that names elements. The table shows the kinds used today and how many elements each picks out of the saved calendar.

Selector	Picks	Count
p	every paragraph (by tag)	50
.fomc-meeting	every element with class fomc-meeting	57
#article	the one element with id article	1
#article p	paragraphs inside that element	4
a[href*='pressreleases/monetary']	links whose address contains this text	100
a[href$='a.htm']	links whose address ends in a.htm	53

.fomc-meeting matches the class fomc-meeting exactly, not fomc-meeting__month. On a statement page, #article p returns the statement’s own paragraphs (Exercise 2).

With an agent. Ask it to print each selector it uses and how many elements it matched. Run the same selector on the saved page and compare the count with what the browser shows.

Reading the saved calendar with read_html()

R Console, in the 15-scraping folder; needs rvest; reads cache/fomccalendars.htm; no network

# install.packages(c("rvest", "httr2", "jsonlite"))   # once
library(rvest)
cal    <- read_html("cache/fomccalendars.htm")       # the calendar as saved
panels <- html_elements(cal, ".panel-default")       # one panel per year
length(panels)

#> [1] 7

titles <- html_text2(html_element(panels, "h4"))     # the first h4 (a heading) inside each panel
titles

#> [1] "2026 FOMC Meetings" "2025 FOMC Meetings" "2024 FOMC Meetings"
#> [4] "2023 FOMC Meetings" "2022 FOMC Meetings" "2021 FOMC Meetings"
#> [7] "2027 FOMC Meetings"

The code reads the saved calendar, keeps one panel per year and prints each panel’s heading, in the page’s order (2026 first). html_elements() keeps every match; html_element() keeps the first match inside each element.

With an agent. Ask it to save each page before parsing it. You can then run its selectors on the same file it read.

The 2024 meeting rows as text

R Console; needs panels and titles from the previous step

p2024 <- panels[titles == "2024 FOMC Meetings"]
rows  <- html_elements(p2024, ".fomc-meeting")
length(rows)

#> [1] 8

data.frame(month = html_text2(html_element(rows, ".fomc-meeting__month")),
           dates = html_text2(html_element(rows, ".fomc-meeting__date")))

#>       month  dates
#> 1   January  30-31
#> 2     March 19-20*
#> 3   Apr/May   30-1
#> 4      June 11-12*
#> 5      July  30-31
#> 6 September 17-18*
#> 7  November    6-7
#> 8  December 17-18*

The eight rows are the eight scheduled meetings of a year. The asterisk marks a meeting that also publishes the members’ economic projections (the page’s footnote: “Meeting associated with a Summary of Economic Projections”).

With an agent. Ask it to print a few parsed rows. Read them against the page in the browser before you trust the rest.

Counting the calendar’s statement links three ways

R Console; needs cal

# both selectors start with .fomc-meeting: 6 more a.htm links sit outside the rows
a1 <- html_elements(cal, ".fomc-meeting a[href*='pressreleases/monetary']")  # address contains this
a2 <- html_elements(cal, ".fomc-meeting a[href$='a.htm']")                   # address ends in a.htm
c(contains = length(a1), ends_a = length(a2),
  text_HTML = sum(html_text2(a1) == "HTML"))     # sum() counts the TRUEs

#>  contains    ends_a text_HTML 
#>       100        47        46

• The calendar holds 46 statements: 57 rows, minus 10 meetings not yet held by the day the page was saved, minus one row that is not a meeting.
• contains, 100: every press-release link in the rows (46 statements, 46 implementation notes, 6 longer-run-goals statements, 2 balance-sheet plans).
• ends_a, 47: one more than the statements. Exercise 1 finds the row that holds it.
• text_HTML, 46: the links named HTML are the statements.
• html_attr(x, "href") returns a link’s address. Each row links its statement, its implementation note and more, so a rule must pick the statements out.

With an agent. It chooses one of these rules without telling you, and they give 47 and 46. Ask it to state its rule in its report, and compare its count per year with the meetings on the calendar.

Exercise 1

• Exercise
• Solution

# Exercise 1 (5 minutes). R Console, in the 15-scraping folder; needs rvest.
# No network. If you do not have cal, panels and titles yet, run these three:
#   cal    <- read_html("cache/fomccalendars.htm")
#   panels <- html_elements(cal, ".panel-default")
#   titles <- html_text2(html_element(panels, "h4"))
#
# 1. Keep the panel titled "2025 FOMC Meetings".
# 2. Select its .fomc-meeting rows and count them with length().
# 3. Print each row's month (.fomc-meeting__month) and dates
#    (.fomc-meeting__date) in one data frame.
# 4. Find the row that is not a meeting.
#
# Two answers to compare with the room: the number of rows, and
# which row is not a meeting.

R Console; needs panels and titles

p2025 <- panels[titles == "2025 FOMC Meetings"]
rows  <- html_elements(p2025, ".fomc-meeting")
length(rows)

#> [1] 9

data.frame(month = html_text2(html_element(rows, ".fomc-meeting__month")),
           dates = html_text2(html_element(rows, ".fomc-meeting__date")))

#>       month              dates
#> 1   January              28-29
#> 2     March             18-19*
#> 3       May                6-7
#> 4      June             17-18*
#> 5      July              29-30
#> 6    August 22 (notation vote)
#> 7 September             16-17*
#> 8   October              28-29
#> 9  December              9-10*

Nine rows. Row 6, “August 22 (notation vote)”, is a decision taken in writing without a meeting. Its release approved updates to the committee’s Statement on Longer-Run Goals, so it is not a policy statement; its link ends in a.htm, which is why the count before was 47 and not 46.

Uses only the functions of the last four slides. This is the one year where a count of rows and a count of policy statements differ. Debrief by call-out: who got nine?

Dynamic pages, and the file they load

The Network tab on the Fed’s press-release list page: the page’s script requests json/ne-press.json.

• Static page: the HTML the server sends is the page you see.
• Dynamic page: the server sends a layout and scripts, and the scripts fetch the content after the page loads. read_html() runs no scripts, so it gets the empty layout.
• The test: View Source and search for a title you can see. The Fed’s press-release list: 85 KB of HTML, “FOMC statement” zero times.
• Three ways in
  • request the file the page loads, found in the Network tab
  • Copy as cURL on that request; curl_translate() turns it into R
  • a headless browser (one with no window): read_html_live()

With an agent. An agent that reads a dynamic page with read_html() gets the empty layout and may conclude there is no data. Give it the address of the file the page loads, from your Network tab.

The feed’s address is assembled by the page’s script, so it is not in the HTML or in the main script file; the Network tab (filtered to Fetch/XHR) is where it shows. In R: sum(grepl(“FOMC statement”, readLines(“cache/pressreleases.htm”))) returns 0.

Six routes to every page on a site

Route	Example	When it fits	Load on the server
The site’s feed or API	ne-press.json, 4,648 releases in one request	the site offers one	one request
An index page with links	the calendar: 57 rows, 100 release links	a list page exists	one page
The sitemap	bls.gov/sitemap.xml, a sitemap index	robots.txt names one	one request per sitemap file
Pagination	one year page per year; ?page=2 elsewhere	the address carries the page number	one request per page
A crawler that follows every link	the job board in Kuhn & Shen (QJE, 2013)	no list exists	the heaviest
Bought data	a vendor’s crawl of job postings, in Hershbein & Kahn (AER, 2018)	the licence fits your use	none

With an agent. Name the route and the total you expect in the brief. Ask for a crawler only when no list exists: it puts the most load on the server.

The BLS index opens with sitemap-home.xml, advisory/sitemap-advisory.xml, air/sitemap-air.xml, and answers 403 without a contact address. The Internet Archive’s listing also holds pages the site has since removed (Part 7). Whatever the route, write down what you believe the full list is and compare your count with it.

3 · What to take into account

• Terms of use, copyright, robots.txt, personal data
• Identifying yourself with a user agent
• What the server’s reply means
• CAPTCHAs and login walls
• fetch_page(): pause, retry, save once, log
• Exercise 2

Terms of use and copyright

• Terms of use: the rules a site sets for visitors; using the site means accepting them. Look for footer links named Terms, Legal or Disclaimer. Zillow’s forbid “automated queries (including screen and database scraping …)”.
• Copyright: the owner’s right to control copies. The Board puts its own material in the public domain (above): anyone may copy it; cite the Board.
• Facts (prices, dates, vote counts) are outside US copyright (Feist v. Rural, 1991). The wording of a text is covered, unless it is public domain or licensed for copying.

With an agent. Read the terms yourself before you write the brief, and tell the agent what to keep: the facts, or the full text when it is public domain or licensed. Its requests carry the user agent you give it, so they go out under your name.

robots.txt lists the paths a site asks programs to skip

robots.txt: a plain-text file at a site’s main address (/robots.txt) that tells programs which paths to stay out of. The server (the computer that holds the pages) does not enforce it. A polite scraper follows it.

• federalreserve.gov/robots.txt does not exist. The server answers with status code 404, not found (the status code is the three-digit number at the top of every reply). The standard for robots.txt (RFC 9309) treats a 404 as no restriction.
• zillow.com/robots.txt, 328 lines; the lines that matter:

zillow.com/robots.txt as read on 18 October 2026, 6 of its 328 lines; the comments after a line are my notes

# Access to and use of Zillow.com is governed by our Terms of Use. See https://www.zillow.com/corp/Terms.htm
User-agent: *                   # line 3: every program
Allow: /homes/for_sale/$        # one of 64 Allow lines: the listing-index pages
Disallow: /homes/               # line 50, one of 212 Disallow lines: the listings
Crawl-delay: 1                  # line 302, in a block for one named crawler
Sitemap: https://www.zillow.com/xml/indexes/us/static.xml.gz   # 15 Sitemap lines

• User-agent: the name a program sends with each request. A block (the lines that follow) applies to the programs named; * means every program.
• Disallow: paths to skip (a path is the part of an address after the site name).
• Allow: exceptions to a Disallow ($ means the address ends there).
• Crawl-delay: seconds to wait between requests; not in the standard, but some sites write it.
• Sitemap: where the site lists the pages it wants found (Part 2).

With an agent. Put the paths to skip and any delay in the brief. Some robots.txt files name AI companies’ crawlers; your agent’s requests carry the user agent from your brief.

The file forbids the listing pages a researcher would want; Part 1’s Zillow slide drew on it. Eleven crawlers are named in later blocks: Amazonbot, for one, is disallowed everywhere except guides and blog pages. The three social-media crawlers at lines 261 to 265 form their own group with two Allow lines; the four at lines 304 to 309 are fully disallowed. The Fed’s sitemap.xml answers 404 as well. Measured 18 October 2026.

Personal data on the pages you scrape

From a Fed press-conference transcript (a PDF on federalreserve.gov): the reporter’s name and outlet.

• Personal data: information that identifies a living person: a name, a photo, an address, an email, an IP address.
• On the Fed’s site. The statements name the voting members. They act in public roles, so naming them raises no privacy issue. The press-conference transcripts name each reporter and outlet (above).
• For research.
  • Collect only the fields you need.
  • Research on identifiable people goes to the IRB (the university board that approves research on people) before you collect.
  • The EU’s data-protection law (GDPR) covers personal data on public pages too, and counts an IP address.

With an agent. Name the fields to keep and forbid printing the rest, then read its report: both marathon agents in Part 6 printed a runner’s name while studying the page.

The Wikipedia request later today leaves the editor’s name out. GDPR: article 4(1) defines personal data; recital 30 names IP addresses and cookie identifiers.

Identifying yourself with a user agent

• User agent: the name a program sends with every request. Put who you are and how to reach you in it. A robots.txt User-agent line matches this name.

• Three requests to bls.gov (the US Bureau of Labor Statistics) with httr2, the R package that sends requests; one request per row.

User agent sent	Status code
no user agent set (httr2 sends its own name)	403 Forbidden, an “Access Denied” page
AEM 6850 class exercise (a name, no email)	403 Forbidden
AEM 6850 class exercise; <your netid>@cornell.edu	200 OK, the page

R Console; needs internet and httr2; two requests; run after class (Practice 6), not in the room

library(httr2)
req <- request("https://www.bls.gov/cpi/")
req <- req_error(req, is_error = function(resp) FALSE)   # a 403 comes back as a reply, not an R error
resp_status(req_perform(req))                            # httr2's default user agent
#> [1] 403
req <- req_user_agent(req, "AEM 6850 class exercise; <your netid>@cornell.edu")
resp_status(req_perform(req))
#> [1] 200

With an agent. Put this user agent, with your address, in the brief, and find it in the agent’s code.

The 403 page’s own words: “Automated retrieval programs (commonly called robots or bots) can cause delays and interfere with other customers’ timely access to information. Therefore, bot activity that doesn’t conform to BLS usage policy is prohibited.” Wikimedia’s policy asks for a contact address as well (Part 7). The policy page itself answers 403 to a user agent without an email. Measured 18 October 2026.

What the server’s reply means, and what to do

Status code	Meaning	What to do	Seen in this session
200	it worked	go on; save the page	all 268 requests that built cache/
301, 302	the address moved	httr2 follows it; save the final address	46 of them, old addresses on the 2006 to 2010 year pages
403	forbidden	stop; read the site’s policy; identify yourself	bls.gov, with no email in the user agent
404	no such page	record it; do not retry	federalreserve.gov/robots.txt
429	too many requests	wait, then retry with a longer pause each time	none, at one request a second
500 and above	the server failed	retry later	none

With an agent. Ask it to log every status code, to stop and report at a 403, a CAPTCHA or a login page, and to test each 200 for a phrase the real page contains: a 200 can be a consent or an error page.

The prep scrape’s 268 requests and the agent’s 191, at one per second, all returned 200. Keep every request’s status in a log; the log is how you know which pages you still owe. The redirects were the old /press/monetary/ addresses on the 2006 to 2010 year pages.

What to do at a CAPTCHA and at a login wall

With an agent. Tell it to stop at a CAPTCHA or a login wall and report. If it offers a way around either, such as spoofing its address or rotating proxies, decline: sites ban scrapers that hide who is asking.

fetch_page(): one polite request, saved once and logged

R Console, in the 15-scraping folder; needs httr2; defines fetch_page() and fetches nothing yet

# the same function ships as fetch_page.R: source("fetch_page.R")
library(httr2)
fetch_page <- function(url, file, ua) {
  if (file.exists(file)) return("already saved")          # never fetch a page twice
  req  <- request(url)
  req  <- req_user_agent(req, ua)                         # who you are and how to reach you
  req  <- req_retry(req, max_tries = 4)                   # a 429 or 503: wait (Retry-After if given), try again
  req  <- req_error(req, is_error = function(resp) FALSE) # a 403 or 404 comes back as a reply, not an R error
  resp <- req_perform(req)
  if (resp_status(resp) == 200) writeBin(resp_body_raw(resp), file)   # the bytes exactly as sent
  write(paste(url, file, format(Sys.time()), resp_status(resp), sep = ","),
        "mine/fetch_log.csv", append = TRUE)              # one log row per request, failures included
  Sys.sleep(1)                                            # one request a second
  resp_status(resp)
}

source("fetch_page.R") defines it and creates mine/ with its log, mine/fetch_log.csv. The cache is the folder of saved pages that a rerun reads instead of asking the server again; writeBin() keeps the raw bytes as sent.

With an agent. Its fetch function should hold the same five pieces: the user agent, the retry, the file saved once, the log row, the pause. Find each in its code, and ask for the cache and the log as files. The R marathon agent fetched one page twice after an encoding error and said so; the Fed agent reported four 429 replies its own log does not contain (Part 5).

Exercise 2

Fetch one statement page yourself with fetch_page(), then read its paragraphs with read_html() and #article p (a statement page keeps its text in the same #article box as the calendar). The page is the statement of 15 June 2022.

• Exercise
• Solution

# Exercise 2 (5 minutes). R Console, in the 15-scraping folder; needs
# fetch_page(): run source("fetch_page.R") if it is not defined yet. One
# live request. Step 2's two lines are long: copy them from fetch_page.R.
#
# 1. Put your own address in the user agent:
#    ua <- "AEM 6850 student; <your netid>@cornell.edu"
# 2. Fetch the 15 June 2022 statement into mine/:
#    fetch_page("https://www.federalreserve.gov/newsevents/pressreleases/monetary20220615a.htm",
#               "mine/monetary20220615a.htm", ua)
#    No network? file.copy("cache/monetary20220615a.htm", "mine/"), then the same line.
# 3. Read the saved file with read_html() and select "#article p".
# 4. Count the paragraphs with length(); print them with html_text2().
#
# Two answers to compare with the room: the number of paragraphs, and
# which paragraph names the member who voted against the decision.

R Console; needs fetch_page() and ua. Output recorded when the exercise was tested; the shipped copy gives the same eight paragraphs

fetch_page("https://www.federalreserve.gov/newsevents/pressreleases/monetary20220615a.htm",
           "mine/monetary20220615a.htm", ua)
#> [1] 200
s  <- read_html("mine/monetary20220615a.htm")
ps <- html_text2(html_elements(s, "#article p"))
length(ps)
#> [1] 8
substr(ps, 1, 60)                       # substr(): the first 60 characters of each paragraph
#> [1] "June 15, 2022"
#> [2] "For release at 2:00 p.m. EDT"
#> [3] "Overall economic activity appears to have picked up after ed"
#> [4] "The invasion of Ukraine by Russia is causing tremendous huma"
#> [5] "The Committee seeks to achieve maximum employment and inflat"
#> [6] "In assessing the appropriate stance of monetary policy, the "
#> [7] "Voting for the monetary policy action were Jerome H. Powell,"
#> [8] "Implementation Note issued June 15, 2022"

Eight paragraphs. Three are not statement text: the date line, “For release at 2:00 p.m. EDT”, and the last one, a link to the implementation note. The vote is paragraph 7: Esther L. George voted against, preferring an increase of 0.5 percentage point.

That statement raised the federal funds rate (the overnight rate between banks) to 1.5 to 1.75 percent. Debrief by call-out: run step 2 again and it returns “already saved” at once; read.csv(“mine/fetch_log.csv”) shows one row, status 200; open the page in the browser: the same paragraphs, with the site’s header and the note link around them. One request per laptop leaves the room, each with a user agent and a one-second pause; the runs that built cache/ and the agent’s run saw no 429 at one per second, and req_retry() handles one if it comes. Older statements carry other extras: the January 2006 statement opens with “For immediate release” and ends with the Board’s discount-rate action.

4 · AI tools, in R or Python

• The brief to a coding agent
• The demonstration: marathon finish times, set up and started
• R or Python for a scrape
• An AI browser agent

The brief to the coding agent

Coding agent: an AI tool that writes and runs code on your computer from written instructions. Before class, one was run on the brief below, in an empty folder. Permission to read, write and run files was given before it started, so it never stopped to ask.

Terminal, in an empty folder holding brief.txt

$ ls
brief.txt
$ cat brief.txt
Collect every FOMC policy statement (the Federal Open Market Committee's statement on its
policy decision, issued after a scheduled meeting or an unscheduled call) from January 2006
to the latest one, from the Federal Reserve's website.

Start from the meeting calendar at https://www.federalreserve.gov/monetarypolicy/fomccalendars.htm
(2021 onward) and the one page per year at
https://www.federalreserve.gov/monetarypolicy/fomchistoricalYYYY.htm for 2006 to 2020.

Rules: send a user agent that identifies the request as "AEM 6850 class demo; <address>";
at most one request per second; save every page you fetch in cache/ and never fetch the same
page twice; if the server answers 429, wait and retry.

Write fomc_statements.csv with one row per statement and the columns date (YYYY-MM-DD), url
(the address after redirects), words, text (the statement's paragraphs only), scheduled
(TRUE after a scheduled meeting, FALSE after an unscheduled call). Use R. Print the number
of statements per year when you are done.

A brief names the unit and how to tell it apart, where to start, the rules (user agent, pace, cache, retries), the output columns and the numbers to print. This one never says which releases do not count; Part 5 shows what that cost.

The pitfalls the brief leaves out: implementation notes, longer-run-goals statements, the notation vote, the three b addresses, the liquidity and swap-line releases.

The demonstration: do marathon runners finish just under round times?

• The question. Allen, Dechow, Pope & Wu (Management Science, 2017) found more finishers just under 3:00, 3:30 and 4:00 than just over, in 9,789,093 marathon finish times. Runners treat round times as goals.
• The pages. The Wineglass Marathon in Corning, New York, an hour from Ithaca. Its timing company posts each year’s results as one plain page: one line per finisher, with the net time (start line to finish line).
• Allowed. The site has no robots.txt (404) and no terms of use. The lines carry names, ages and towns: keep place, sex, age group and time.

leonetiming.com/2023/Roads/Wineglass/WGMarathonNet.htm, the column headings and one line; name, age and town masked here

                                       PLACES IN               NET    NET
PLACE NAME                    AGE  SEX  &      AGEGROUP        TIME   PACE  CITY ...
---------------------------------------------------------------------------------
  665 ··························   F# 264   41/129  F30-34   3:59:00  9:08  ······

The page is fixed-width text inside one <pre> element: a selector finds the block, and each line is cut by position or by a pattern.

The brief the two agents get

brief.txt, alone in an empty folder; one folder's copy ends 'Use R.', the other's 'Use Python.'

Collect the finish times of every finisher of the Wineglass Marathon (Corning, New York) for
each year whose results Leone Timing posts as a web page. Start from the year's page at
https://www.leonetiming.com/YEAR/Roads/Wineglass/ for YEAR from 2018 to 2025; some years have
no page (there was no race in 2020). On each year's page, follow the link to the marathon
results by net time (not the half marathon, not the relay).

Rules: send a user agent that identifies the request as "AEM 6850 class demo; <address>"; at
most one request per second; save every page you fetch in cache/ and never fetch the same page
twice; if the server answers 429, wait and retry. The pages list runners' names, ages and
towns: keep only place, sex, age group and net time, and never print or save names, even
while you study a page's layout.

Write wineglass.csv with one row per finisher and the columns year, place, sex, age_group,
net_time (h:mm:ss) and net_minutes. Draw wineglass.png: the number of finishers in each
one-minute bin of net time from 2:30 to 6:30, all years together, with vertical lines at 3:00,
3:30, 4:00 and 4:30. Draw wineglass_years.png: the number of finishers and the median net
time, by year.

When you are done, print: for each year, the number of rows you kept and the last place
number on that year's page; the number of finishers at 3:57, 3:58, 3:59, 4:00, 4:01 and 4:02
(minutes, all years); and the median net time by year.

Use R.

Both agents start now, side by side, with permission to read, write and run files given up front. Part 6 reads what they produce. In prep each took between 2 and 3 minutes.

Run sheet. Before class: two terminal windows, side by side, in two empty folders, wineglass-r and wineglass-py, each holding only its brief.txt (your address in place of <address>; the Python copy ends “Use Python, with the Python at ~/demo-py/bin/python”). In each window type, at this slide:

claude –allowedTools “Bash,Read,Write,Edit,Glob,Grep” “$(cat brief.txt)”

Then go on with Part 4; glance back at the windows on the next slide. Prep runs (19 October 2026, both started together): R 2 min 49 s and 2 min 22 s; Python 2 min 11 s and 2 min 15 s; 15 requests each; the two files identical row for row, 11,466 finishers. If the site is down: copy the saved pages (aem6850-agent-runs/final-r/cache) into each folder’s cache/ and the agents read them; the slides in Part 6 carry the prep run.

R or Python for a scrape

Step	R	Python
Send a request with a user agent, a pause and retries	httr2	requests
Pick elements out of HTML with CSS selectors	rvest	BeautifulSoup, lxml
Read a page whose scripts must run first	read_html_live() (with chromote)	playwright, selenium
Crawl many thousands of pages, pausing and resuming	a loop over a cached fetch	scrapy

• Both send the same requests and read the same HTML. For tens or hundreds of pages, scrape in the language you analyse in.
• Python has more ready-made tools for very large crawls and for driving a browser.
• A coding agent writes either. The brief, the saved pages and the comparisons decide whether the file is right; Part 6 compares the two agents’ files.

An AI browser agent reads the calendar page

AI browser agent: a program that opens pages in your own browser and reads them for an assistant.

• Use it for
  • a page you have never seen, to find which element holds the date
  • a handful of pages
  • pages behind your own login, where the terms allow
• Why a loop collects every statement instead
  • the agent takes seconds to a minute per page; the loop takes about a second
  • the agent keeps no copy of the page, so there is nothing to cite or re-run
  • the site’s terms apply to it as to any program

Drawn from a run of a browser agent in a desktop assistant’s browser pane, October 2026: the agent’s steps and its answer.

Compare its answer with Exercise 1: nine rows, 22 August.

5 · Verifying what a scrape returns

• What the coding agent’s run on the Fed’s site did
• Three lists of one site: its file, the feed, the statements file
• Three statements drawn with a seed
• Its report against its own log

What the run did

From the brief alone, the agent collected 175 statements in 8 min 13 s.

• 191 pages fetched once each: the calendar, the 15 year pages and 175 statements.
• 175 rows, 31 January 2006 to 16 September 2026; 10 marked unscheduled (scheduled = FALSE).
• Its closing message names two doubts, cut from the crop above. Two unscheduled rows are liquidity releases (announcements of emergency lending). One address carries a wrong date (June 2007).
• It also reports four 429 replies. Its own log says otherwise (the last slide of this part).

Run 1, captured in October 2026; the tool’s screens change between versions, the steps do not.

The slide carries run 1, captured in prep. If you started a run on this brief before class, read its final table against this slide. The middle of its message, the table of statements per year, is what the next slide’s code reproduces. Its rule: a statement is the link named “Statement” under each heading on a year page, or the “HTML” link after “Statement:” on the calendar. Process: 37 tool calls, 179 lines of R in two files (fetch.R, a cached fetcher with the user agent, a one-second spacing and a 429 wait; collect_statements.R); it chose dplyr, stringr and readr beside rvest and httr2, packages the course does not use. A second run the same evening took 7 min 28 s and gave the same 175 dates. If the live run is shown, its own numbers replace these.

Three lists of one site disagree on what a statement is

Did the agent get the same statements as the statements file (ours, 171 rows, built from cache/ by the rule in SOURCE.txt)?

R Console, in the 15-scraping folder; reads agent_run/fomc_statements.csv (the agent's run 1) and fomc_statements.csv (ours)

agent <- read.csv("agent_run/fomc_statements.csv")
ours  <- read.csv("fomc_statements.csv")
c(agent = nrow(agent), ours = nrow(ours))

#> agent  ours 
#>   175   171

yr <- data.frame(year = 2006:2026)
yr$agent <- as.vector(table(substr(agent$date, 1, 4)))   # rows per year; substr(): characters 1 to 4
yr$ours  <- as.vector(table(substr(ours$date,  1, 4)))
print(yr[yr$agent != yr$ours, ], row.names = FALSE)      # the years that differ

#>  year agent ours
#>  2007    10    9
#>  2008    11   10
#>  2010     9    8
#>  2019     9    8

• The agent’s rule, every link the site labels Statement: 175. The four extra dates (above) announce lending programmes or the mechanics of a decision, not the committee’s decision on the policy rate.
• The feed: 179 titles contain “FOMC statement”; five are in neither file.
• The statements file’s rule: the title says “FOMC statement” and the release announces the rate decision: 171.

Write the rule down before you scrape, and reconcile every difference by hand.

Three statements drawn with a seed

The counts match on most years; does the text match? Seed: a starting number that makes a random draw repeat on every computer, so the room draws the same three rows and reads them against the statements file.

R Console; needs agent and ours, read on the counts slide

set.seed(6850)
pick <- sample(nrow(agent), 3)
agent[pick, c("date", "words", "scheduled")]

#>           date words scheduled
#> 134 2021-07-28   488      TRUE
#> 55  2012-01-25   418      TRUE
#> 28  2008-10-29   286      TRUE

# match(): the row of ours with the same date
agent$text[pick] == ours$text[match(agent$date[pick], ours$date)]

#> [1] TRUE TRUE TRUE

# the last row, so the scrape ran to the end
agent[nrow(agent), c("date", "words")]

#>           date words
#> 175 2026-09-16   131

The screenshot is the 29 October 2008 page, the third row drawn. In both files its text starts at the first paragraph; the page header and the link to the implementation note are left out. The three TRUEs say the agent’s text and the statements file’s text are identical on those three dates.

The latest statement opens with its vote (“approved the following statement for release by a 12 – 0 vote”); older ones end with a voting paragraph. Both files date the June 2007 statement 28 June from the page’s own date line, where the address says 0618: the address carries a typo. The January 2012 row is the policy statement (20120125a), not the longer-run-goals statement released the same day (20120125c). On the 171 shared dates the two files’ word counts differ on two rows, by 3 and 8 words, from different rules on headings and links inside a paragraph.

The agent’s 429 count, read from its own log

The agent said four 429 replies came. The log it wrote records every request. The claim, from its closing message: “The server returned 429 four times, and each request was retried and then succeeded.”

R Console, in the 15-scraping folder; reads agent_run/fetch_log.txt

fetch_log <- readLines("agent_run/fetch_log.txt")
c(lines = length(fetch_log), fetched = sum(startsWith(fetch_log, "fetched")))

#>   lines fetched 
#>     172     172

sub("fetched https://www.federalreserve.gov", "", grep("429", fetch_log, value = TRUE))   # the lines that contain 429

#> [1] "/newsevents/press/monetary/20090429a.htm"       
#> [2] "/newsevents/pressreleases/monetary20150429a.htm"
#> [3] "/newsevents/pressreleases/monetary20200429a.htm"
#> [4] "/newsevents/pressreleases/monetary20260429a.htm"

The agent counted the lines that contain 429 with the shell command grep -c 429 fetch_log.txt and got 4. Those four lines are fetches of statements dated 29 April: each address holds 0429. Every line in the log is a successful fetch. No 429 reply came.

Takeaway: a number in an agent’s summary is a claim about its files. Read the files before you repeat it.

The log holds 172 of the 191 fetches. The agent fetched the 16 list pages and three statements before it started the log; all 191 pages are in its cache/.

6 · The demonstration’s results

• The two agents’ figures
• The R file against the Python file
• The step that reads a line, in R and in Python

Switch to the two terminal windows and read each agent’s closing table aloud: rows kept against last place per year, the counts at 3:57 to 4:02. Then the next three slides, which carry the prep run.

Finish times bunch just under 3:30 and 4:00

The R agent’s figure

The Python agent’s figure

• 11,466 finishers in 2018, 2019 and 2021 to 2025, from 15 pages.
• 143 finished in the minute starting 3:58, 86 in the minute starting 3:59, and 62 in the minute starting 4:00. The paper’s pattern holds in one small race.

The prep run of 19 October 2026; the live run’s tables replace these numbers if they differ.

The R file and the Python file agree row for row

R Console, in the 15-scraping folder; reads the two agents' files in wineglass/ (prep run, 19 October 2026)

r  <- read.csv("wineglass/r/wineglass.csv", na.strings = c("", "NA"))       # blank sex reads as NA
py <- read.csv("wineglass/python/wineglass.csv", na.strings = c("", "NA"))
cols <- c("year", "place", "sex", "age_group", "net_time")
c(r = nrow(r), python = nrow(py))

#>      r python 
#>  11466  11466

identical(r[cols], py[cols])                                              # same rows, same values

#> [1] TRUE

rbind(rows = table(r$year), last_place = tapply(r$place, r$year, max))   # one column per year

#>            2018 2019 2021 2022 2023 2024 2025
#> rows       1674 1598 1389 1371 1647 1816 1971
#> last_place 1674 1598 1389 1371 1647 1816 1971

• Each year’s row count equals the last place on its page, in both files.
• Each agent’s report named its own slip. The R agent fetched one page twice after a text-encoding error; the Python agent printed one runner’s first name, “Chip”, which it had taken for a column heading.
• Prep times, the two agents started together: R 2 min 22 s, Python 2 min 15 s.

Finding the results link, in R and in Python

wineglass/r/wineglass.R (excerpt)

idx  <- fetch(base)                 # the year's page
a    <- html_elements(read_html(idx$body), "a")
txt  <- html_text(a)
href <- html_attr(a, "href")
i    <- which(grepl("Net", txt) &
              grepl("Marathon", href) &
              !grepl("Half|Relay", href,
                     ignore.case = TRUE))
stopifnot(length(i) == 1)
pg   <- fetch(paste0(base, href[i]))

wineglass/python/explore_results.py (excerpt)

status, html = get(url)             # the year's page
soup = BeautifulSoup(html, "html.parser")
for a in soup.find_all("a", href=True):
    href = a["href"]
    name = href.rsplit("/", 1)[-1].lower()
    if ("net" in a.get_text().lower()
            and "marathon" in name
            and "half" not in name):
        return urljoin(url, href)

• The same rule in both: the link whose text says Net and whose address says Marathon, not Half. The R agent also stops if it finds more than one.
• Both then cut each line of the <pre> text with a regular expression and stop at the 2019 page’s separate wheelchair table, whose places restart at 1. The whole scripts are in wineglass/.

7 · A scrape someone else can re-run

• The project folder: fetch once, build from the saved pages
• Past versions of a page from the Wayback Machine
• One Wikipedia article’s edit history

The project folder: fetch once, build from the saved pages

15-scraping/, the folder in 15-scraping.zip (trimmed)

reference_scrape.R    sent the 268 requests that built cache/, on 18 October 2026
cache/                every page as the server sent it, raw bytes
  fetch_log.csv       one row per request: address, final address, time, status, bytes
build_statements.R    reads cache/ only and writes fomc_statements.csv
fomc_statements.csv   171 rows, one per policy statement
SOURCE.txt            the addresses, dates, user agent, pace and counting rule
fetch_page.R          the polite fetch from Part 3; your fetches go to mine/
wineglass/            the demonstration: the brief, each agent's scripts, file and figures

• build_statements.R reruns with no network and returns the same 171 rows. A new fetch would differ: the site adds a statement after each meeting, and its pages change design (next slide).
• A replication package includes the scraping code even when the pages cannot ship: “All programs used to generate the analysis data from raw data must be included, even if the raw data cannot be provided” (AEA data and code policy).
• The Fed’s pages ship here because they are public domain. The marathon pages carry names, so wineglass/ holds the code and the files without names, and no saved pages.

With an agent. Ask for two scripts: one that fetches into cache/, and one that builds the file from cache/ with no network.

The Fed’s calendar in 2015, from the Wayback Machine

Wayback Machine: the Internet Archive’s copies of web pages since 1996, each one a snapshot; 19 yearly ones of the calendar.

R Console, in the 15-scraping folder; needs rvest; reads cache/fomccalendars_20150102.htm (live address on the page)

snap <- read_html("cache/fomccalendars_20150102.htm")   # the page as captured on 2 January 2015
length(html_elements(snap, ".panel-default"))            # today's selector finds nothing

#> [1] 0

html_text2(html_elements(snap, "th#year"))               # th#year: a th element whose id is year

#> [1] "2014 FOMC Meetings" "2013 FOMC Meetings" "2012 FOMC Meetings"
#> [4] "2011 FOMC Meetings" "2010 FOMC Meetings" "2009 FOMC Meetings"
#> [7] "2015 FOMC Meetings"

sum(html_text2(html_elements(snap, "a")) == "Statement")   # links named Statement on the 2015 page

#> [1] 50

In January 2015 the page was built from tables: .panel-default matches nothing, th#year holds the years, and the 50 links named Statement still work. A redesign changes the selectors.

With an agent. Ask it to record each snapshot’s timestamp and the layout it found: today’s selector finds nothing on the 2015 page.

One Wikipedia article’s edit history

• Revision: one saved edit, with its time, its size and the editor’s note.
• Wikipedia’s robots.txt closes the history pages to programs (Disallow: /w/). Its API returns the same history to a program that sends a contact address: four replies of up to 500 revisions.
• 1,613 revisions; the article’s information page states 1,613 edits.
• Zhang & Zhu (AER, 2011): when mainland China blocked Chinese Wikipedia in 2005, editors outside China cut their contributions by 42.8 percent.
• Greenstein & Zhu (AER P&P, 2012): the political slant of about 28,000 articles on US politics, across every revision.

With an agent. Name the fields to request, without the editor’s name, and the total to match: the information page’s 1,613.

The request left out the user field: it names people, and anonymous editors appear by IP address, which counts as personal data. The busiest months: November 2020 (203 edits), November 2017 (96, his nomination as chair) and January 2026 (85). Information page read 19 October 2026.

Practice

Ten exercises on the session page, with solutions behind a click, all in the 15-scraping folder. Task 6 sends two requests to bls.gov: after class, not in the room.

• 1–3 Selectors on the saved calendar: press-conference links per year; the minutes’ release dates; the meetings marked with an asterisk
• 4–5 The year pages: sort the 2008 and 2020 headings into meetings, calls and notation votes; find which calls issued a statement
• 6 Identifying yourself: the BLS request with and without a contact address
• 7 Whether you may scrape: the terms, robots.txt and sitemap of a site from your own research
• 8–9 Past versions: the Wayback captures of a page you use; the edit history of a Wikipedia page you choose
• 10 The statements file: words per statement since 2006 as a plot, and the medians by year