17 · Text II — LLMs

Thursday, Oct 22, 2026

Materials for this session are not published yet. They appear here before class.

Objectives

By the end of this session you can:

  • Call an LLM from R with schema-constrained output.
  • Write prompts that state the failure mode.
  • Cache model responses so a pipeline re-runs deterministically.
  • Estimate what a batch job will cost before running it.
  • Spot-verify a sample against the source.

What we cover

  • What this unlocks: classifying millions of documents; structured extraction at a scale hand-coding cannot touch.
  • ellmer; schemas that force the output shape you need.
  • Temperature, consistency, and cost.
  • Hallucination as the native failure mode.
  • The five duties of a pipeline with a model in it: pin the exact model version in code; set temperature to zero; cache every response to disk and commit the cache; store the raw response, not just the label; document model and prompt in the README. If you cannot do all five, it does not go in a paper.
  • What the cache buys, honestly: your results re-run byte-for-byte from the committed responses; the live API is the weaker link, and the cache keeps it out of the analysis.
  • The gold set, and how to design one. A small subset you label by hand so it can grade the machine. The design questions are the whole thing: how big, drawn how, from what frame. Draw it with sample() and a seed so the draw itself re-runs, and label it before you look at the model’s answers.
  • Agreement rates, and when they mislead. A 95% rate on a category that is 95% of the data tells you nothing.
  • Confusion tables: not how often the model is wrong, but which categories it mixes up. The pattern usually points at the prompt.
  • Error analysis: sort the mistakes into kinds, then pick the fix. Re-prompt, re-tool, or hand-code the residual. Guessing which one without sorting first is how people waste a week.
  • In class: hand-label twenty items yourself, compare with the model, report the agreement rate and the confusion table.

Verification habit. Spot-verify a sample of the model’s answers against the source, and report the agreement rate you got.