AEM 6850
Empirical Methods for Applied Economists
Prof. Ariel Ortiz-Bobea
Tuesday, October 27, 2026
LLM: a large language model, a program that writes a reply to written instructions. Running example: the policy statements of the Fed (the US central bank) and sentences from its meetings. All model answers were saved before class, so no account or password is needed. In the R Console, now: install.packages("jsonlite") to read them. Then download and unzip 17-text-llms.zip from this session’s page on the course website and open 17-text-llms.Rproj. That page also holds the full code and eleven practice exercises, with solutions.
Today’s running example is the text the Fed publishes. The FOMC, the rate committee of the US central bank (the Fed), publishes a statement after each meeting. A statement sets a target range for the main interest rate and hints which way rates go next.
Yellow: the target range the statement sets. Blue: a hint about future rates (“firming”: raising them). Board of Governors of the Federal Reserve System; public domain.
A language model reads the Fed’s statement of 22 March 2023 and returns a number and a judgement.
| The target range | The stance of a sentence | |
|---|---|---|
| The model reads | one whole statement | one sentence at a time |
| Example | “…the Committee decided to raise the target range for the federal funds rate to 4-3/4 to 5 percent” | “The Committee anticipates that some additional policy firming may be appropriate…” |
| The model returns | lower = 4.75, upper = 5 | one label (here hawkish) |
| Checked against | FRED, the Fed’s own daily record of the range | labels two people made by hand (stance has no official record) |
FRED: the free data service of the Federal Reserve Bank of St. Louis. stance: which way a sentence leans, toward higher rates (hawkish), lower rates (dovish), or neither (neutral).
What can a language model do with text, and when is it the right tool? Economists turn text into variables; a model is one of three ways to do it.
Each use turns text into a variable. Classifying picks a label from a list; extracting copies pieces out of the text; matching decides whether two names mean the same firm. edit distance: the number of characters to change to turn one name into the other. The examples, each explained on the next slides: a takeover headline, a misspelled survey answer, a stock analyst’s report, a firm name written two ways.
Classify: Lopez-Lira & Tang (JFE, 2026); Bursztyn et al. (AER, 2025), example from Mellon et al. (2024). Extract: Li et al. (RFS, 2026). Match: Dell (JEL, 2025); Allen & Hatfield (2025).
| “Cigna Calls Off Humana Pursuit, Plans Big Stock Buyback” | For Humana’s stock |
|---|---|
| A language model | bad news: a buyer willing to pay above market price walked away |
| A data vendor’s sentiment score, −1 to +1 | +0.65, good news |
Cigna, a US health insurer, had been trying to buy Humana. Headline from December 2023; example from Lopez-Lira & Tang, working-paper version, pp. 15–16. JFE: Journal of Financial Economics.
Extracting copies named pieces out of a document into columns, one field per column. A label picked from a list is classification.
| Field | Copied from an analyst report on Macy’s, March 2012 |
|---|---|
| cause | “focused selling culture” |
| effect | “double-digit earnings growth through 2015” |
Analyst report: a stock analyst’s report on a company, here by JP Morgan. Review of Financial Studies; example from the authors’ working-paper version.
| Name in the second list | The same firm as “Mellat Insurance Company”? | Edit distance |
|---|---|---|
| “Mellat Ins. Co.” | yes | 11 |
| “Mellot Insurance Corp.” | no, a different firm | 5 |
Edit distance calls the different firm the closer match. A model reads “Ins.” as “Insurance” and “Co.” as “Company”.
Name pair from Allen & Hatfield (Federal Reserve working paper, 2025), on firms in the US sanctions list. Edit distances from R’s adist().
Three tools turn text into a variable. The task decides which one:
| A pattern (regular expression) | A language model | A person | |
|---|---|---|---|
| Suits | a fixed format: dates, codes | meaning that changes with the wording | few items, or the reference labels |
| Cost | free | per token, or free on your own computer | about 15 seconds a sentence |
| Speed on today’s data | 171 Fed statements in under a second | the same 171 in 18 seconds | 2,480 Fed sentences in about 10 hours |
| Same answer every run | yes | not guaranteed | not guaranteed either |
| Checked against | example strings, or an outside record | an outside record, or a hand-labelled sample | a second person |
Start from the task: if the wording never changes, write a pattern; if meaning matters, use a model and check it; if the items are few or the labels are the reference, read them yourself. regular expression: a written search pattern that matches text by its form.
The choice, tested on one task. From each of the Fed’s 171 policy statements (2006–2026), read the target range for the federal funds rate, the Fed’s main interest rate, once with a pattern and once with a model, and compare each with FRED, the Fed’s own daily record of the range (from the Federal Reserve Bank of St. Louis).
| A regular expression | A language model | |
|---|---|---|
| Right, against FRED | 145 of the 145 it read | 170 of 171 |
| No answer | 26: the 25 statements of 2006–2008 that set one number (“at 5-1/4 percent”), and 17 Aug 2007, which states no target | 1: 17 Aug 2007, left empty as its instructions said |
| Time and cost | under a second, free | 18 seconds, about 47 cents |
The pattern would need a second pattern for the single-number years; the model read both wordings. When the wording is fixed, the free pattern does the job. The code for both, step by step, is on the session page.
A language model reads text you supply. Before you rely on its labels:
| Headline (only the firm’s name differs) | Model’s answer |
|---|---|
| “Facebook not planning to extend European privacy law globally” | good for the stock |
| “Blahblahblah not planning to extend European privacy law globally” | bad for the stock |
Bills, stock returns and the 5% fix: Ludwig, Mullainathan & Rambachan (Annual Review of Economics, 2026). Headline pair: Glasserman & Lin (Journal of Financial Data Science, 2024).
With your neighbour, no computer (4 minutes). For each task: a regular expression (a written search pattern), a language model or a person? For each model answer, what would you check it against?
1–3 a regular expression · 4–8 a language model · 9–10 a person
Patterns for fixed formats (1–3). A model where meaning matters, checked on a hand-labelled sample (4–8; task 7 can mix a pattern for the fixed formats with a model for the rest). A person for one high-stakes item (9) and for the reference labels (10). Task 6 is checked against a few hundred pairs matched by hand.
What is a label, and when do you make labels by hand: to train a model, or to check how often it is wrong? A model’s answers are a measurement, and you need something to compare them with.
A label is the category assigned to one item. Data in which every item carries one are labelled data. (The Fed: the US central bank.)
| Item | Label | Assigned by |
|---|---|---|
| Fed sentence: “And inflation is well above target.” | hawkish (higher interest rates ahead; the other labels: dovish, neutral) | two researchers (a hand label) |
| the same sentence | hawkish | a language model |
| Survey answer, misspelled: “Immergration” | immigration | a survey coder |
A model’s labels have an unknown error rate. Compare them with a reference:
About 200 random items pin the error rate within about ±7 percentage points (two standard errors of a share near one half); all 2,480 Fed sentences within ±2. Once checked, the model labels other text, which nobody labels by hand.
Hand labels have two jobs. Thousands train (fine-tune) a model for your own categories; a few hundred random ones check a model’s output. A large model used as it is needs only the check, and an outside record can replace even that.
| Text | Audio | Images | |
|---|---|---|---|
| Train | Moreno-Medina et al. (QJE, 2025) trained a model on about 200,000 news stories labelled by topic, to sort TV news into topics | clips transcribed by hand train a speech-to-text model for one accent or subject | photos labelled by workers paid per photo on online platforms |
| Check | two people label Fed sentences hawkish (higher rates), dovish (lower) or neutral: how often the model agrees | a hand transcript of a short clip: the share of words the model gets wrong | random photos labelled by hand: how often the model agrees |
| No hand labels | the Fed’s interest-rate target range read by the model, compared with FRED, the Fed’s own daily record | the model’s list of who spoke when, compared with the official transcript | fire damage read from photos of homes, compared with inspection records |
The Fed: the US central bank. QJE: Quarterly Journal of Economics.
How the 2,480 sentences from documents of the Fed (the US central bank) in the Trillion Dollar Words dataset of Shah, Paturi & Chava (ACL, 2023) got their labels: hawkish (pointing to higher interest rates), dovish (lower) or neutral. Two of the dataset’s authors each labelled the sentences on their own with the codebook, the written rules. They agreed on 90.7% and settled the rest with the codebook, adding a rule for each new kind of case. The published dataset holds only the settled labels: the reference labels, which a model is judged against.
R Console
#> [1] TRUE FALSE TRUE#> [1] 0.6666667Agreement before resolving disagreements: 89.0% for sentences from meeting minutes, 95.0% for press conferences, 91.1% for speeches, 90.7% in all (Shah, Paturi & Chava, ACL, 2023, Table 9).
Can a closed model, one that runs on the provider’s computers and bills each call, label the 2,480 Fed sentences, and how would we know?
No account with the provider is needed: the saved replies and the data are in the session’s zip file.
R Console, in the 17-text-llms folder (unzipped from 17-text-llms.zip); reads tdw_sentences.csv
#> [1] 2480 6#>
#> dovish hawkish neutral
#> 650 606 1224The reference for checking the model: a published dataset, Trillion Dollar Words, with sentences from the minutes (meeting records), press conferences and speeches of the Fed (the US central bank). Two researchers (annotators) labelled each one on their own and settled the rest together; the file holds the settled label as 0, 1, 2, turned into words above:
Trillion Dollar Words: Shah, Paturi & Chava (ACL, 2023); ACL: the main conference in computational linguistics.
The two people labelled each sentence by a codebook: the written definition of each label, with rules for the cases in between. Condensed to the box below, it is also what the model is told, so the model and the people follow the same rules:
codebook.txt, in the project folder; sent to the model with every sentence
You label one sentence from a document of the US Federal Reserve: FOMC
meeting minutes, the chair's press conferences, or speeches by Fed officials.
dovish: the sentence points to future easing of monetary policy (lower rates).
hawkish: the sentence points to future tightening of monetary policy (higher rates).
neutral: mixed signals, no change in policy, or not about the policy stance.
Rules for the cases in between:
- Inflation falling, unemployment rising, or growth projected low: dovish.
- Inflation rising, unemployment falling, growth projected high, or output above potential: hawkish.
- Unemployment or growth unchanged, maintained or sustained: neutral.
- The dollar rising in value: dovish. The dollar falling in value: hawkish.
- Oil, energy or house prices falling: dovish. Rising: hawkish.
- The Fed expecting low inflation: dovish. High inflation: hawkish.
- A foreign country's own economy or trade policy: neutral.
- "Accommodative", or a focus on "maximum employment": dovish.
- "Mixed", "moderate" or "reaffirmed": neutral.
Give exactly one label.FOMC: the Fed’s rate-setting committee. potential output: what the economy can produce with full use of its workers and machines. accommodative: keeping rates low. A stronger dollar makes imports cheaper and lowers inflation, so it is dovish. Condensed once from the annotators’ guide (Shah, Paturi & Chava, ACL, 2023), before any reply, and never edited after.
Run by the instructor before class: model_setup.R (part); needs ellmer and an API key in ~/.Renviron
library(ellmer) # install.packages("ellmer") once; calls language models from R
MODEL <- "<model id>" # the full model id from the provider's list, written once
CODEBOOK <- paste(readLines("codebook.txt"), collapse = "\n") # the box on the previous slide
stance_type <- type_object(stance = type_enum(c("dovish", "hawkish", "neutral"))) # the only answers allowed
chat <- chat_anthropic(system_prompt = CODEBOOK, model = MODEL, cache = "none")
chat$chat_structured("And inflation is well above target.", type = stance_type) # one sentence in, one label outchat_anthropic()), weights unpublished, each call billed through the API, the web address R sends requests to. The API key is a password that bills your account, kept in ~/.Renviron, never in a script.type_enum() limits the reply to the three labels (enum: a fixed list of allowed answers); cache = "none": no discount on the resent instructions, so the bill later is the full price. The run before class: this call once per sentence, 2,480 times.Each call’s reply was saved as one file before anything was done with it. The file for “And inflation is well above target.” (sentence 904, hawkish by hand):
R Console, in the 17-text-llms folder; needs jsonlite; reads one saved file; no API key
library(jsonlite) # reads JSON, the text format replies arrive in
r <- fromJSON("replies/run1/904.json") # one of the 2,480 saved files
r$answer$stance # the label the model gave
#> [1] "hawkish"
r$reply$model # the model id the provider reports
#> [1] "claude-sonnet-5-5"
r$reply$content$text # the reply's text, as the model wrote it
#> [1] "{\"stance\":\"hawkish\"}"
r$reply$stop_reason # why the model stopped writing
#> [1] "end_turn"
unlist(r$reply$usage[c("input_tokens", "output_tokens")]) # tokens billed: sent, written back
#> input_tokens output_tokens
#> 647 14raw reply: the reply exactly as the provider sent it. token: a word or a piece of a word, the unit providers bill by. stop reason: end_turn means the model finished its answer. The session’s results are read from these 2,480 files.
save_replies.R (part), inside the loop over items j; the whole function is in the project and on the session page
if (file.exists(f)) next # f: this item's file; saved already: skip
chat <- chat_anthropic(system_prompt = prompt, model = MODEL, cache = "none") # a fresh conversation per item
ans <- chat$chat_structured(texts[j], type = type) # the call: one label back
rec <- list(id = ids[j], answer = ans, model = MODEL,
reply = chat$last_turn()@json) # the reply as the provider sent it (JSON)A fresh chat for each item, because a chat resends its whole conversation and earlier sentences would sway later ones. An interrupted run resumes where it stopped, and a rerun makes no call.
Now the live part: five sentences drawn at random with a seed (set.seed(6850), the same five for everyone). Label each hawkish (higher interest rates ahead), dovish (lower) or neutral before the model does, because a label you have seen pulls yours towards it.
moved in sympathy: moved together. NIPA: the US national accounts. several: several Committee members. the Committee: the group in the Fed that sets interest rates. policy accommodation: keeping rates low. resource utilization: how fully workers and factories are in use.
Run by the instructor, live: demo.R, part 1; needs ellmer (calls models from R), the API key and model_setup.R
source("model_setup.R") # loads ellmer; MODEL, CODEBOOK, stance_type
tdw <- read.csv("tdw_sentences.csv")
set.seed(6850)
pick <- sample(nrow(tdw), 5) # the five sentences you labelled
chat <- chat_anthropic(system_prompt = CODEBOOK, model = MODEL, cache = "none")
five <- as.list(tdw$sentence[pick])
live1 <- parallel_chat_structured(chat, five, type = stance_type, on_error = "stop") # stop at the first error
live2 <- parallel_chat_structured(chat, five, type = stance_type, on_error = "stop")
data.frame(annotators = c("dovish", "hawkish", "neutral")[tdw$label[pick] + 1], # the file codes labels 0, 1, 2
live1 = live1$stance, live2 = live2$stance)
identical(live1$stance, live2$stance)The five you just labelled, sent from this laptop to the closed model twice, now, with the codebook as instructions (CODEBOOK) and the three labels as the only answers allowed (stance_type). Watch whether the two runs agree with each other and with your labels; identical() prints TRUE when the runs match. parallel_chat_structured() sends the five at once, each as a fresh conversation; annotators is the two people’s label. If the call fails, the next slide prints the saved run’s labels for the same five.
R Console; needs tdw (the 2,480 Fed sentences); reads the closed model's saved labels; no key
labels <- read.csv("replies/labels_run1.csv") # the 2,480 saved replies, collected: id, stance, tokens
tdw$model <- labels$stance[match(tdw$id, labels$id)] # the model's label next to the people's
set.seed(6850); pick <- sample(nrow(tdw), 5) # the five you labelled: a seed fixes a random draw
data.frame(annotators = tdw$annotators[pick], model = tdw$model[pick])#> annotators model
#> 1 hawkish neutral
#> 2 dovish dovish
#> 3 neutral hawkish
#> 4 hawkish hawkish
#> 5 hawkish hawkish#> [1] 0.6935484Before class the closed model labelled all 2,480 sentences following the codebook, one call each; labels_run1.csv collects the saved replies (stance: the label), and reading it needs no key. On your five it matches the two people on 3; on all 2,480, on 69.4%, its agreement rate with the hand labels, the same measure as between two people. The two people agreed with each other on 90.7%, so the model falls well short of a careful person. Whether that rate is good enough depends on the use; the measured error is what you report.
R Console; needs tdw with columns annotators (hand labels) and model (the model's labels)
#> model
#> annotators dovish hawkish neutral
#> dovish 0.68 0.07 0.25
#> hawkish 0.10 0.57 0.33
#> neutral 0.16 0.08 0.76#> dovish hawkish neutral
#> 444 347 929#> [1] 0.6935484A cheaper rule than a model is a word count. The Loughran & McDonald (Journal of Finance, 2011) word lists sort words into positive and negative; they were built to score the tone of company annual reports, not policy stance. The rule borrows them: net tone is positive minus negative words per 100 words, and good news points to higher rates, so positive means hawkish, negative dovish, zero neutral. Score the rule as the model was scored (4 minutes, R Console, in the 17-text-llms folder):
R Console; needs tdw (the 2,480 hand-labelled sentences); reads lm_words.csv
dict <- read.csv("lm_words.csv") # Loughran-McDonald word lists
pos <- dict$word[dict$list == "positive"]
neg <- dict$word[dict$list == "negative"]
w <- strsplit(tolower(tdw$sentence), "[^a-z]+") # the words of each sentence
tdw$tone <- sapply(w, function(x) 100 * (sum(x %in% pos) - sum(x %in% neg)) / length(x))
tdw$rule <- ifelse(tdw$tone < 0, "dovish", ifelse(tdw$tone > 0, "hawkish", "neutral"))
# 1. agreement rate: mean(tdw$rule == tdw$annotators)
# 2. confusion table: tab_r <- table(annotators = tdw$annotators, word_rule = tdw$rule)
# then round(prop.table(tab_r, 1), 2)#> [1] 0.4387097#> word_rule
#> annotators dovish hawkish neutral
#> dovish 0.49 0.16 0.34
#> hawkish 0.34 0.25 0.41
#> neutral 0.30 0.20 0.50The rule agrees on 43.9%, below the base rate; the model reached 69.4%. It calls 41% of hawkish sentences neutral and 34% dovish, because the lists score good or bad news, not the direction of policy:
R Console; needs labels (the model's saved labels and tokens); PRICE: the provider's list, dated
#> input_tokens output_tokens
#> 1723541 50003#> [1] 3.947112#> hours dollars
#> 20.66667 413.33333An open-weight model is a file you download once and run on your own computer, with no bill per call. Can it do the closed model’s job?
Each lab publishes a family in several sizes; Qwen is the one run on this laptop next. Parameters: the numbers inside the model (its weights), in billions. Stored at about 5 bits each (8 to a byte), as today’s files are, they take about 0.6 GB of memory per billion, which decides what can run the model.
| Family (maker) | Billions of parameters | Memory needed, and what has it |
|---|---|---|
| Gemma (Google) | 2 to 31 | 1 to 20 GB: a laptop |
| gpt-oss (OpenAI) | 21 and 117 | 13 GB: a laptop; 65 GB: one data-centre GPU |
| Qwen (Alibaba) | 0.8 to 397 | 0.5 to 240 GB: a laptop, up to a GPU server |
| Llama (Meta) | 1 to 405 | 1 to 245 GB: the same |
| DeepSeek | 284 and 1,600 | 170 GB and 1 TB: a GPU server |
| Kimi (Moonshot) | 1,000 and 2,800 | 0.6 and 1.7 TB: one or two GPU servers |
| Claude, GPT, Gemini | closed, not published | the maker’s data centre, reached through the API |
Sizes as the makers publish them, October 2026. GPU server: eight data-centre graphics processors (GPUs) of 80 GB or more, rented by the hour. The largest open models are mixtures of experts: all parameters sit in memory, but each token uses a fraction of them (49 billion of DeepSeek’s 1,600), so they run faster than their size suggests. The makers of the closed models publish no sizes; the one widely cited figure, 1.8 trillion for a 2023 GPT model, came from a chip maker’s 2024 keynote.
On this laptop, in the Terminal: install the model server (here Ollama) once, download (pull) one small model file, list what is installed, and send one sentence through it. No account and no key.
Terminal, on the instructor's laptop; captured 26 Oct 2026
$ brew install ollama # once; or the installer from the server's website
$ ollama serve # starts the server; keep this tab open (the installer app does it for you)
$ ollama pull qwen3.5:4b-q4_K_M # one model file, downloaded once
pulling manifest
pulling e56a899b1540: 100% ▕██████████████████▏ 675 MB
pulling a9d0a3e1d8c9: 100% ▕██████████████████▏ 2.6 GB
pulling 788a4d94bc3f: 100% ▕██████████████████▏ 177 B
verifying sha256 digest
writing manifest
success
$ ollama list
NAME SIZE
qwen3.5:27b-q4_K_M 17 GB
qwen3.5:4b-q4_K_M 3.3 GB
qwen3.5:9b-q4_K_M 6.6 GB
$ ollama run qwen3.5:4b-q4_K_M --think=false "Label this sentence from the US Federal Reserve \
as hawkish (pointing to higher interest rates), dovish (lower rates) or neutral. \
Answer with one word. Sentence: And inflation is well above target."
HawkishThe file’s name says its size in billions of weights and how the weights are compressed to fit in memory. --think=false turns off the reasoning text some models write before answering.
Run by the instructor, live: demo.R, part 2; needs ellmer and the model server
LOCAL_MODEL <- "<model file name>" # the open model's file, set once
chat_local <- chat_ollama(model = LOCAL_MODEL, system_prompt = CODEBOOK,
params = params(temperature = 0, seed = 6850,
reasoning_effort = "none"))
sapply(tdw$sentence[pick], USE.NAMES = FALSE,
function(s) chat_local$clone()$chat_structured(s, type = stance_type)$stance)R Console; needs tdw (the 2,480 sentences) and pick (your five); reads the open model's saved labels; no key
#> annotators closed open
#> 1 hawkish neutral neutral
#> 2 dovish dovish dovish
#> 3 neutral hawkish hawkish
#> 4 hawkish hawkish neutral
#> 5 hawkish hawkish hawkishchat_ollama() replaces chat_anthropic(); CODEBOOK, stance_type and the sentences are the same. The open model accepts a temperature of 0 and a seed, so a run repeats exactly; reasoning_effort = "none" turns off thinking (2 seconds a sentence instead of 23 to 126). Live now: your five sentences. In the table, the same run saved before class, annotators is the two people’s hand label, closed the closed model’s, open the open model’s; the closed model matches the people on 3 of 5, the open model on 2.
R Console; needs tdw (the 2,480 sentences), tok and PRICE (the cost slide); reads the open models' saved labels
score <- function(stance, seconds, dollars)
c(agreement = mean(stance == tdw$annotators, na.rm = TRUE),
sec_each = median(seconds, na.rm = TRUE),
hours_all = sum(seconds, na.rm = TRUE) / 3600, dollars = dollars)
open <- sapply(c("4b", "9b", "27b"), function(s) { # billions of weights
d <- read.csv(paste0("replies/local_", s, ".csv")) # id, stance, seconds
d <- d[match(tdw$id, d$id), ]
score(d$stance, d$seconds, 0)
})
closed <- c(agreement = mean(tdw$model == tdw$annotators),
sec_each = 2.0, hours_all = 10.5 / 60, # 2 s a call, 8 at a time (run log)
dollars = sum(tok * PRICE) / 1e6) # tok: the tokens billed, cost slide
round(cbind(open, closed), 3)#> 4b 9b 27b closed
#> agreement 0.644 0.622 0.696 0.694
#> sec_each 1.680 2.760 7.730 2.000
#> hours_all 1.146 1.876 5.444 0.175
#> dollars 0.000 0.000 0.000 3.947Three sizes of one open-weight model, run on this laptop (64 GB), and the closed model each labelled all 2,480 hand-labelled Fed sentences (hawkish, dovish or neutral). The 27-billion model agrees with the hand labels as often as the closed model, free per call but over 5 hours to its 10 minutes; the 9-billion does worse than the 4-billion. So yes, an open model can do the closed model’s job here, at the cost of time, once each candidate is scored against the hand labels. Always “neutral” would score 0.494; the two people agreed on 0.907.
To run an open-weight model, its whole file must sit in memory (a Mac’s shared memory, or a PC’s graphics-card memory), so memory decides which models fit. Memory bandwidth, in gigabytes a second (GB/s), decides how fast the model writes, because each new token reads the whole file once.
| Machine | Memory | Bandwidth | Largest model that fits | Speed, this laptop = 1 |
|---|---|---|---|---|
| Typical laptop | 16 GB | about 100 GB/s | about 9 billion weights | about ¼ |
| This laptop | 64 GB | 400 GB/s | about 35 billion | 1 (measured) |
| Desktop graphics card | 32 GB | 1,792 GB/s | about 27 billion | about 4.5 |
| Rented data-centre GPU | 80 GB and up | over 3,000 GB/s | 70 billion and up | about 8 |
GPU: a graphics card, the processor these models run on; its memory is the model’s alone, while a laptop shares its memory with everything else running. Speed: tokens written per second, estimated from bandwidth except this laptop’s. The machines, their specifications and prices: session page.
The last step: a coding agent, an AI tool such as Claude Code, writes and runs code from a request in plain words, this pipeline included. A request of this kind produced this project’s run_local.R:
In R with the ellmer package (it calls language models from R), write
run_local.R: label every sentence in tdw_sentences.csv with the open model
that the model server on this computer runs, with codebook.txt as the
system prompt (the rules sent with every sentence) and the three-label
answer format from model_setup.R. Save id, label and seconds to a CSV,
skipping ids already saved. Before the full run, show me the labels of 20
sentences next to their hand labels.A test on 20 sentences shows that the pipeline works; the open model then labels all 2,480 for nothing; the hand-labelled sample judges the result. run_local.R and its output, replies/local_4b.csv, are in the project folder you unzipped.
The checked closed model (the 27-billion open model would do as well) labelled the 2,540 sentences of the 171 Fed policy statements, which nobody labelled by hand. Each bar is one statement:
Stance (top) moves with the Fed’s rate (red). Net tone (bottom), the word count, does not: it measures good or bad news, a different thing.
Stance: hawkish minus dovish share of the closed model’s labels on each statement’s sentences. Red: the Fed’s target rate (upper limit, FRED). Net tone: positive minus negative words per 100. Grey: Sep 2008 (Lehman Brothers fails), Mar 2020 (pandemic rate cut), Mar 2022 (first rise after it).
Eleven exercises on the session page, with solutions behind a click, on the files in the 17-text-llms folder. Tasks 1–9 need no key; task 10 needs ellmer and your own key; task 11 an open model on your own computer.
AEM 6850 · Session 17
