AEM 6991

MPS Capstone Project I

Prof. Ariel Ortiz-Bobea

5 · Using AI in research

Wednesday, September 23, 2026

Motivation

• So far we’ve focused on coming up with questions and data.
• This week we turn to tools, and specifically to using AI for research:
  1. Understanding the existing literature.
  2. Writing up your findings.
  3. Coding your analysis.

Ninety seconds. First person. Four weeks on questions and data, and this week is the thing you will be driving both of them with.

Say the policy out loud, because it is not on the slide and because other sections of this course restrict these tools and this one does the opposite on purpose. The deliverable here is a research proposal, which is the genre where unverified output does the most damage. So the skill worth teaching is checking, and checking is what the rubric grades.

Name the three and say the hour has one section for each. Say which is riskiest and why: the literature, because a fabricated reference costs the tool nothing to produce and you will not notice it unless you go and open the paper. Korinek’s review for economists rates six task families and literature search is the only one he puts at the bottom, so this is not my opinion.

One line that meeting 10 is how to write, and today is what a tool does to a draft. Do not preview the risks; they arrive in their own blocks.

Outline

1. How these tools work
2. Different AI tool modalities
3. AI for literature reviews
4. AI for writing
5. AI for coding
6. Disclosure

Thirty seconds. Read the six and give the shape: two short sections on what these tools are and how you reach them, then one section for each place AI enters a project, then what applies to everything you hand in.

1. How these tools work

Thirty seconds. Say what the section covers: what the model actually does, what is wired around it, and the one consequence that generates every risk in the rest of the hour.

Keep this section moving. It is the only part of the hour that is not about their own project, and it earns its place only by making the rest predictable.

It predicts the next token

• The model predicts the next piece of text, then the next.
• Token: the unit it predicts. A word, part of a word, or a symbol.
• Nothing is looked up along the way.

Two minutes. Plain and quick, and resist going further into the machinery than this. Nobody in this room needs to know what a transformer is.

Give the examples out loud: the is one token, while rare words and long numbers split into several, which is a small reason numbers get garbled.

The last bullet is the load-bearing one. There is no database behind the window and no moment where it consults anything. When it writes a reference, it is writing the kind of text that a reference is, in the same way it writes the rest of the sentence.

Then move on. The next two slides are the consequences.

A model on its own can only write text

• Context window: everything the model can see at once.
• Long threads push early instructions out of the window.
• Harness: the program around the model, which controls what it sees and does.
• Tools: what the harness connects, such as search, files and code.
• Without tools, the model is a brain with no eyes or hands.
• It can only write from memory, which is where most invented citations come from.

Two minutes. This is the slide that makes the rest of the hour make sense, and the analogy is the part they will remember.

Take the three bold words in order. The context window is the model’s whole world, and nothing outside it exists. The harness is the program you are actually typing into, which loads files, calls tools and decides what the model sees. The tools are the eyes and the hands.

The second bullet answers something every one of them has experienced and none has been told the reason for. You set up a conversation carefully, and forty exchanges later it is ignoring half of what you said. It did not forget. Those instructions are no longer in front of it. So: one task per conversation, and paste the material again rather than pointing back at it.

The last bullet points ahead to the literature section. A model asked for a reference with no tools attached has nothing to consult, so it writes reference-shaped text. That is the only thing it can do.

It guesses when it does not know

• Asked something it cannot answer, it often guesses.
• The guess reads exactly like a correct answer.
• Training and scoring reward a guess over “I don’t know.”
• OpenAI researchers: “Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty.”

A theory paper. Its examples use DeepSeek-V3, GPT-4o and Llama.

Kalai, Nachum, Vempala and Zhang (arXiv, 2025), Why language models hallucinate

Ninety seconds. The two slides before this did the setup, so this one is short.

The exam-question analogy is theirs, not mine, and it carries the slide. Every risk in the rest of the hour follows from this one. A model that says “I do not know” scores zero. A model that guesses scores above zero some of the time. Train on that and you get a system that always answers. Their own line is that models are optimised to be good test-takers.

Say the consequence plainly: this comes from how the models are trained and scored. So the question for the rest of the hour is whether you will be able to tell.

VERIFY: arXiv 2509.04664, abstract, quotation exact.

2. Different AI tool modalities

Thirty seconds. Say what the section covers: the three ways you reach the same model, what changes as the tool gets more access to your work, what you may put into it, and the record you keep of what it did.

Three ways to reach the same model

	What it can do	A good use
Chat claude.ai	Sees only what you paste or attach. Cannot run your code or touch your machine. With Code execution on, runs its own Python on an attached file, on Anthropic’s servers.	“What are the top journals in environmental economics?” “Is this idea a good fit for JAERE, JEEM or AJAE?”
Cowork desktop app, Pro plan	Reads and edits the files in a folder you connect. In Manual mode, asks before each change. Can write code, but runs it only in its own sandbox. Not built for coding.	“Here are some PDFs. Search them for tables, to see if this is worth exploring as a project.”
Claude Code terminal	The most powerful modality. Edits files and runs commands on your machine, asking first by default.	Setting up a full research project that is reproducible, with every change to the code and the paper tracked.

Two minutes. Same model underneath all three. What differs is access: what the tool can see and what it can do. Claude Code is the most powerful because it can do the most on your machine. The model inside it is no smarter, which is the part students get wrong.

Chat is what most of them will use, so spend most of the time there, and be exact about code. Chat cannot run your script or touch your machine. With Code execution and file creation switched on under Settings → Capabilities (off by default on Free, Pro and Max; on by default for Team and Enterprise), it runs its own Python on an attached file in a sandbox on Anthropic’s servers. ChatGPT does the same by default. The number that comes back is not a result you can rerun, which is why the coding section asks for the code. Its good uses are questions about the field whose answers you can judge yourself, such as which journals publish this work and whether an idea fits one.

Cowork needs the desktop app and a Pro plan; in Manual mode it asks before each change, so leave it there. Anthropic’s help centre says it runs code in an isolated virtual machine that sees only the folder you connect, so it cannot run your analysis in your own setup. It is good at work on a pile of documents: go through a folder of PDFs, pull out the tables, and say whether a project is worth pursuing.

Claude Code works in your own folders with your own software. Its good use is the whole research project, set up so that it is reproducible, with every change to the code and the paper tracked in git. Git comes two slides on.

More access means more risk

• Chat cannot change your files. You copy out what you use.
• Cowork and Claude Code can overwrite a week of work.
• Use the least access the job needs.
• Before connecting a folder, know how to get it back.

One minute. The rule is the third bullet and everything else is the argument for it.

The copy-out step in chat is a real safety feature. You are the thing that moves the output into your work, so you see it once on the way past. Point a tool at a folder and that step is gone.

The last bullet sets up the git slide after next, so do not explain it here.

What you may put into these tools

• Nothing at moderate risk or above in a public AI tool. Licensed data counts.
• If you cannot upload your file, write the code on an invented sample with the same columns, then run it yourself.
• ChatGPT free: three file uploads a day.
• Claude: switch on Code execution and file creation under Settings → Capabilities.
• Cornell: “You are accountable for your work, regardless of the tools you use to produce it.”

IT@Cornell, AI guidelines · Claude Help Center, code execution

Ninety seconds. One line of callback to meeting 4 and no more; they have had the full treatment.

The second bullet is a commitment I made to this room last week. Two or three teams will have data they may not upload. The invented-sample route is how they work with it: the tool sees the column names and made-up rows, and the real file stays on their laptop.

The Claude settings path was checked against Anthropic’s help page, dated 6 August 2026. VERIFY ON THE MORNING: ChatGPT’s free-tier upload limit changes often; re-read OpenAI’s File Uploads FAQ before the meeting.

A record of what changed

• Git: a saved history of every change to your project.
• GitHub: a website that keeps that history online, so a team shares one copy.
• With git you can see what a tool changed, and undo it.
• Commit your folder before connecting it to an assistant.
• A chat transcript is not a record. The same prompt can give different answers.
• Step by step, in AEM 6850: version control with git.

Ninety seconds. Two audiences on one slide, and the second one is most of the room.

Git keeps the history on your laptop. GitHub keeps a copy online, which is how a team works from one version instead of emailing files. For the teams that will point Cowork or Claude Code at a folder, the first four bullets are the whole safety net: you can see every line the tool changed, and you can undo it. The concrete version is git init and a first commit before the folder is connected. AEM 6850 meeting 7 is the step-by-step version, and the page is public.

For everyone else, bullet five is the point, and it is about evidence. A screenshot of a chat is not a method section. Nothing in a transcript can be re-run.

Close with the test the whole course uses: could a teammate reproduce your number on a different laptop next week? Say it and let it sit.

3. AI for literature reviews

Thirty seconds. Say what the section covers: what the literature section of a proposal is for, what these tools do well in a search, the ways a reference list goes wrong, and the check.

Why review the literature?

• Your proposal has to say why the study is new and why it matters.
• Saying that clearly is framing.
• You can only frame a question well if you know the literature.
• The review explains and justifies the existence of your study.
• A tool cannot tell you what it failed to find.

One minute, and it is the argument the rest of the block rests on.

Put the title to the room and take one answer in the chat. The answer to land on: a proposal has to say why the study is new and why it matters, and saying that clearly is framing. You cannot say what is new without knowing what exists, so the literature is what the framing is built from. A review written for its own sake, one summary per paper, does none of that work, and it is what most first drafts look like. Framing gets its own slide at the end of the block; here it only needs a name.

The commonest way an MPS proposal fails on this section is a claim of novelty that a reader can puncture in thirty seconds, because they know the paper you missed.

The last bullet is the one to say slowly. A tool reports what it found. Nothing in the answer tells you the size of what it did not find, and there is no prompt that fixes that. Every slide after this is a version of that point.

References from memory are often wrong

• 8 free chatbots (February 2025): 26.5% of references fully correct.
• Economics: over 30% of GPT-3.5’s citations did not exist. GPT-4 was barely better.
• Korinek (JEL, 2023) asked GPT-4 for his own paper: real title, invented journal.
• Elicit, Perplexity, Ai2 ScholarQA and Consensus cited 5 to 18 of 21 retracted papers without saying so.

The 8 chatbots, free versions: GPT-4o-mini, Claude 3.5 Sonnet, Copilot (GPT-4), DeepSeek-V3, Gemini 2.0 Flash, Grok-3, Mistral Large (Le Chat), Perplexity Sonar.

Cabezas-Clavijo and Sidorenko-Bautista (JDIS, 2026) · Buchanan, Hill and Shapoval (American Economist, 2024) · Korinek (JEL, 2023) · MIT Technology Review, 2025

Ninety seconds, and no more. This is the whole of the failure evidence; the rest of the section is what to do about it.

Say the mechanism in one line: with no search attached, the model writes text shaped like a reference. The 26.5 percent is current and about the free tools they will use. Tested 7 to 9 February 2025: ChatGPT (GPT-4o-mini), Claude (3.5 Sonnet), Copilot (GPT-4), DeepSeek (V3), Gemini (2.0 Flash), Grok (Grok-3), Le Chat (Mistral Large), Perplexity (Sonar). Grok and DeepSeek produced no false references; Copilot, Perplexity and Claude produced the most. Perplexity searches the web and was still among the worst.

The economics number is Buchanan, Hill and Shapoval, GPT-3.5 and GPT-4 in 2023, stated verbatim in the abstract and cited by Korinek; the full text is paywalled and unread. Korinek’s example is GPT-4, December 2023 JEL: his paper with Zach Bethune, close to 200 citations, returned as Journal of Monetary Economics 117, 1-13, which is invented.

The retraction bullet: Elicit 5, Perplexity 11, Ai2 ScholarQA 17, Consensus 18, reported September 2025; Consensus was down to 5 by the time the article ran. People did this long before AI: only 5.4 percent of citations to retracted papers mention the retraction (Hsiao and Schneider, QSS, 2021).

If asked whether it is getting better: the Lancet audit of 2.5 million biomedical papers found fabricated citations rose more than twelvefold from 2023 to early 2026, so it is reaching print faster, not slower.

One line back to the New York lawyers from meeting 4, no more.

What an anchor paper is

• Anchor paper: a published paper close to your question that you have read.
• It asks your question in another setting, or a close question in yours.
• Or it uses the data or the method you plan to use.
• Start with two or three, from your advisor, a recent review or a first search.
• Anchors change as you find closer papers.

Ninety seconds. Define it plainly, because the whole method rests on it.

An anchor is the paper you would cite in the first paragraph of your literature section: the closest existing answer to your question. You must have read it, not just its abstract, because everything you find next hangs off it, and a bad anchor spreads.

Two or three is enough to start. Say the last bullet slowly. Anchors are a working set: if a later round turns up a paper closer to your question, it becomes an anchor and the search restarts from it.

How to find the papers

• Keep a log: one row per paper, where it came from, and whether you opened it.

Two minutes. This is the method, and it starts from papers you trust.

Step 1: two or three anchor papers, as defined on the previous slide. Read their introductions.

Step 2: collect related leads from each anchor. Its reference list, the papers that cite it (Google Scholar’s Cited by), and whatever the model suggests when you give it your anchors. Everything at this step is a lead, not a reference.

Step 3: check each lead. Find it in EconLit or Google Scholar, open it, read the abstract, and keep or drop it.

Step 4: update the anchors. If a kept paper is closer to your question than one of your anchors, it replaces it, and the next round starts from it.

Stop when a round adds nothing new. Systematic reviewers call that saturation, and it is the only honest stopping rule.

The log is what makes this concrete: one row per paper, with where it came from (a reference list, a citing paper, the model), whether you found it in a database, whether you opened it, and keep or drop. It is also most of your disclosure, because it records what the tool suggested and what you checked.

The next slides take steps 2 and 3 one at a time.

Follow the citations of your anchors

• Go back through each anchor’s reference list.
• Go forward to the papers that cite it.
• On 250 computer-science papers, a search that followed citation chains recovered over 80% of each paper’s references.
• Keyword search alone recovered under 20%.
• Tools: Google Scholar Cited by, Semantic Scholar, Connected Papers.

Pipeline built on GPT-OSS-120B, with Qwen3-Embedding-8B for ranking. Not yet peer reviewed.

Sahu, Charlin and Pal (arXiv, 2026)

Ninety seconds. This is step two, and it needs no AI at all. Backward means the reference list. Forward means the papers that cite it, which Google Scholar’s Cited by and Semantic Scholar show you.

The numbers come from one 2026 preprint on 250 computer-science papers, not yet peer reviewed, comparing plain keyword search with a pipeline that followed bibliographic links. Say that plainly. The size of the gap is the point, and it matches what research librarians have taught for decades.

Use memory for leads

• Give it your anchors. Ask for related papers, authors, search terms and journals.
• Treat every answer as a lead to look up.
• GPT-4 is “often able to provide standard references that are frequently cited in the literature.”
• ChatGPT got less reliable as questions got more specific.

Korinek (JEL, 2023) · Buchanan, Hill and Shapoval (American Economist, 2024)

Ninety seconds. This is the other half of step two. Give the model your anchors and ask what is around them: related papers, the names that recur, the words each field uses for the same idea, and which journals to look in. Search terms matter most, because economists, epidemiologists and marketers use different words for the same thing, and students are worst at this part.

Korinek, in the JEL article: GPT-4 “is often able to provide standard references that are frequently cited in the literature”. That is the famous end of the literature. Buchanan, Hill and Shapoval: “the reliability of the model decreases as the prompts become more specific”. The niche, recent papers that decide whether a question is new are exactly where it fails.

On naming journals in the prompt: it steers the search usefully, but it also makes the prompt more specific, and in Korinek’s own example the journal was the part it invented. So journal names in the answer are leads too. Every lead goes into a database before it goes anywhere else.

Tools improve fast. The evidence here is on 2023 to 2025 models, and nobody has measured today’s frontier models on this. Say so if asked.

Search a database and let the tool screen

• Look up each lead yourself, in EconLit or Google Scholar.
• Give the tool the results to sort and screen.
• Screening a known set, Elicit kept 91% of the relevant papers.
• Across 19 studies, AI screeners wrongly dropped a median 28%.
• Check what it drops.

Mazzali tested Elicit’s systematic-review workflow; no model version is named. Clark’s 19 studies tested mostly ChatGPT (GPT-3.5, GPT-4, GPT-4o), with some on Claude 2, Bing AI and Perplexity.

Mazzali et al. (BMC Med Res Methodol, 2026) · Clark et al. (Res Synth Methods, 2025)

Ninety seconds. The split the evidence draws is between finding and sorting. Used to search, generative AI missed a median 91 percent of the relevant studies (Clark et al., 19 comparative studies). Given a set to screen, it does much better.

The 91 percent kept is Mazzali et al., May 2026, an umbrella review on air pollution and acute lower respiratory infections, using Elicit’s systematic-review workflow; no Elicit version or model is named. It also over-included: it flagged 70 papers where the reviewers kept 33. The same study ran Elicit’s own search against a Boolean search: 500 papers against 324, and only 26 in both. A tool’s search is a different set, not a bigger one, which is why the search stays yours.

The 28 percent is the median of wrongly excluded studies across Clark’s 19 studies. That is the number that makes the last bullet the rule. EconLit is the field’s own index and Cornell subscribes to it through the library; none of the AI tools lists it as a source.

Write your own framing

• Framing: what you ask, why it matters, and what is new compared with your anchors.
• Tools are still weak at this. They produce the framing of an average paper.
• Write yours first, in two or three sentences.
• Then have the tool test it against your anchor papers:

Here is my framing and the introductions of my anchor papers. Quote each paper’s statement of its own contribution. Then list where my framing overlaps with theirs, and what a referee would say is not new.

• Revise it yourself. Do not ask for a new version.

Two minutes. Framing is the part of a proposal a reader judges first, so define it plainly. It is how you present your question: what you ask, why it matters, and what is new compared with the closest papers. It usually lives in the first paragraph of the introduction and the last paragraph of the literature section.

Tools are weak here. Asked for a framing, they produce the framing of an average paper in the area, because that is what they have seen most, and they overstate what is new, which is the overgeneralisation problem the writing section comes back to. This is my judgement from reading AI-drafted introductions; I have no measurement for it, so say so.

Once yours is written, the tool is useful as a tester. The prompt makes it do two things you can check: quote each anchor’s own contribution statement, which you can find in the paper, and list where yours overlaps. The referee line is the hostile reader. If you cannot say in one sentence how your framing differs from each anchor, it is not finished.

This links back to meeting 3, on finding the question.

Avoiding hallucinated references

• Paste the DOI after doi.org/. Not found is a strong warning sign.
• If it resolves, check the authors, year and title.
• No DOI: search the exact title in Google Scholar.
• Real papers get miscited too: 43% of GPT-3.5’s real citations had errors.
• Open the paper and find the passage you are citing.
• Cornell Library: “do check whether the references are real by looking up the source material.”

Walters and Wilder tested ChatGPT with GPT-3.5 and GPT-4, in 2023.

Walters and Wilder (Sci Rep, 2023) · Cornell University Library

Ninety seconds. This is the block’s check and it is the rule in the syllabus, so say it as a rule.

The 43 percent is the reason the routine does not stop at DOI resolution. A reference list can pass every existence test and still misattribute a finding, and misattribution is what a reader of your proposal will actually catch.

Close on the course rule: every reference in every submission is one you have opened. Then ask how you would know a reference was wrong. You opened it.

VERIFY: the 43 percent and the 24 percent for GPT-4 in Walters and Wilder.

4. AI for writing

Thirty seconds. Say the scope again in one line, because it is the section students will most want to over-read: meeting 10 is how to write, and this section is what a tool does to a draft.

What a tool cannot be responsible for

• The rules start when a document goes out under your name.
• A tool cannot be an author: “they cannot take responsibility for the submitted work.”
• Nature: polishing language is fine. Extensive writing help must be disclosed.
• Generating your reasoning without saying so is not allowed.
• “Human accountability is non-transferable.”

COPE, Authorship and AI tools · Nature Portfolio, AI policy

Ninety seconds. This is the block’s frame and it decides everything after it.

The first bullet is the practical version. Nobody cares what you type to sort out your own thinking. The rules start when the document goes to somebody who will act on it.

Nature’s three tiers are worth naming because they are the most useful thing any publisher has published on this. Polishing is green. Extensive writing support is amber and needs disclosure. Generating the reasoning is red.

Do not spend time on authorship as such; no student here is putting a chatbot on a paper. The transferable half is that responsibility does not move, which is what the last bullet says.

You write the outline

• The structure carries the argument. You write it.
• One bullet per paragraph, in two parts:
  • the topic sentence, in full
  • what the paragraph discusses
• Do not ask a tool to propose the structure.
• You would not know why it picked that one.

Two minutes. This is the method, and it is the one thing in the writing block I want them doing this week.

Say why the topic sentence is written out in full rather than summarised. A summary of a paragraph you have not written yet is a hope. A topic sentence is a commitment, and if you cannot write it, the paragraph has no point and should not be in the outline.

The last bullet is the reason for all of it, and it is about defending your own work. A structure you accepted is one you cannot argue for in a proposal defence, because you do not know what it was chosen over.

Say the size out loud: a background section is six to ten bullets, and it fits on one page.

Ask it to audit your plan

• Give it your outline. Ask what is wrong with it.
• Which paragraph makes two claims?
• Which bullet does not follow from the one before?
• Which claim has no evidence named?
• Where does the order bury the point?
• Ask for a list of problems, not a rewrite.

Two minutes. Step one is the outline. Step two is the audit, and step two is where a tool earns its place.

The four questions are the ones worth asking, and they are all structural. None of them asks whether the writing is good.

The last bullet is the general rule for this whole block, so say it slowly. A list leaves the outline in your hands and you decide each change. A rewrite gives you fluent text you did not write and have not checked, and it quietly launders a muddle you were about to notice.

Say plainly that this is permitted here and has to be disclosed, because several of them will have met a rule elsewhere that treats critique as off-limits.

Outlining a published paper

• This is where these tools are strongest.
• Give it a paper in your area.
• Ask what each paragraph of the introduction claims, and what job it does.
• You get the paper’s outline, in the same form as yours.
• Three papers in your field give you a model.
• The paper sits beside the answer, so checking is easy.

Two minutes. This is the permission slide for the block, and it goes here rather than at the front because it only makes sense once they know what an outline is.

The tool is reading here, so almost nothing can go wrong. The paper is open next to the reply and any claim about paragraph four can be checked by looking at paragraph four.

Say what they get out of it. Nobody in this room has written an economics introduction. Three unpacked introductions in their own field is a better model than any rule I could give them from the front, and it is twenty minutes of work.

It makes the claim bigger than the result

• DeepSeek, ChatGPT-4o and LLaMA 3.3 70B overgeneralised in 26–73% of summaries.
• Claude models were the most faithful to the original papers.
• Across all ten models, about 5 times as often as expert summaries.
• Three moves to look for in your draft:
  • a sample becomes people in general
  • past tense becomes present tense
  • a result becomes a recommendation
• Asking for accuracy made it about twice as likely.

The ten models: GPT-3.5 Turbo, GPT-4 Turbo, ChatGPT-4o, ChatGPT-4.5, LLaMA 2 70B, LLaMA 3.3 70B, Claude 2, Claude 3.5 Sonnet, Claude 3.7 Sonnet, DeepSeek. 4,900 summaries.

Peters and Chin-Yee (R Soc Open Sci, 2025)

Ninety seconds. The three moves are the usable part, because a student can run their eye down a paragraph and find them.

The last bullet closes the escape route, and it surprises people. Adding an accuracy instruction made overgeneralisation worse, so you cannot instruct your way out of it. The defence is reading your own sentences against your own result.

Tie it back: this is the second of the two failure modes the syllabus grades against, the plausible-but-wrong claim.

VERIFY: doi 10.1098/rsos.241776, the 26 to 73 percent range, and the accuracy-prompt finding.

Write the paragraphs yourself

• Step two: turn each bullet of your outline into a paragraph.
• The question, what is new and the design are yours to write.
• Then ask the tool what is wrong with each paragraph.
• One job is safe to hand over: the link between two of your paragraphs.

Write one sentence that links these two paragraphs. Add no new claim.

Two minutes. This is step two, and the tool does not write it. The parts that carry a proposal, which are the question, what is new and the design, are where the thinking happens, and a reader can tell when it was not done. Each bullet of the outline becomes a paragraph in your own words.

The tool’s job at paragraph level is the same as at outline level: ask what is wrong. Which sentence makes a claim the paragraph does not support? Where does it drift from the topic sentence? Ask for a list, and fix the wording yourself.

The transition is the one piece that is safe to delegate, because it adds no claim: the tool is given both ends and asked for the join.

CAPTURE NEEDED: run the transition prompt on two paragraphs from a real draft and put the reply on the slide. Ask the room to predict whether it will slip in a claim of its own.

The next slide is what happens when students lean on the tool anyway.

Loose text and the time it costs

• Leaning on the tool gives long, smooth text that says little.
• Cutting it back takes longer than writing it.
• You lose the practice. Editing is how writing is learned.
• Readers learn its habits and stop trusting the text.

Ninety seconds. This is the over-reliance slide and it closes the block’s risks.

Say it plainly first: try to avoid having AI write portions of your paper, and if you do, you will have to edit it heavily.

Bullet three is the one that matters to them and it is worth saying in the first person. I write for a living and the editing is where I learned to. If the tool drafts and you only tidy, you will be a worse writer in two years than you are now, and writing is most of what a research career is.

The first three bullets are judgement, not measurement; say so if asked.

If asked for evidence on the last bullet: Kobak et al. (Sci Adv, 2025) counted word frequencies in 15 million PubMed abstracts and estimated at least 13.5 percent from 2024 were processed by a model, with delves 28 times as common as before. No model was tested. It describes prose in aggregate and is not a detector.

Numbers come from your table

• While drafting, leave placeholders for results, such as [Table 2, row 3].
• No number goes in from memory, yours or the tool’s.
• When the analysis is done, fill each placeholder from the table.
• Then have the tool audit every number against the table:

List every number in this section. For each one, quote the table cell it came from. Mark any number you cannot find as “NOT IN TABLE”. Do not rewrite anything.

• Fix what it flags yourself.

Anthropic, Reduce hallucinations: verify with citations

Two minutes. This is the check for the writing section, and it has two halves.

While drafting, you leave a placeholder wherever a result will go, pointing at the table cell it will come from. No number enters the text from memory, yours or the tool’s, and you can write the argument before the analysis is finished. That is how most published papers are written: the text is built around the tables, not the other way round.

When the analysis is done, you fill the placeholders from the table. Then the tool earns its place: it lists every number in the section with the cell it came from, and flags anything it cannot find. The prompt follows Anthropic’s own “verify with citations” technique: find the supporting source for each claim, and mark what has none. “Do not rewrite anything” keeps the fixes in your hands.

Say what the audit cannot do. It checks your transcription against the table. If the table itself is wrong, the audit will confirm the wrong number without complaint, which is what the coding section is about.

The last bullet is the one they will skip. Re-prompting hands you a fresh paragraph that has not been audited, and the check resets to zero.

Then ask how you would know a sentence was wrong. You found the cell.

5. AI for coding

Thirty seconds. This section is four habits, distilled from AEM 6850’s two meetings on AI: plan first and work in small steps, commit before each task, explore the data before you analyze it, and build checks into every step. Before the third, one slide on how to acquire data.

Plan first and work in small steps

• Decide the question, the sample and the method yourself.
• Ask for a plan before any code, and read it.
• One task per conversation. Start a new one for the next task.
• A task is small enough when you can predict one number of its answer.

Adapted from AEM 6850, meetings 8 and 9

Two minutes. This is the general lesson of AEM 6850’s two meetings on AI, and it matters more than any prompt.

The design is yours: the question, the sample, the weights, the method. Let the tool choose and it picks something plausible, and plausible is not the same as right. That is the second failure mode the syllabus grades against. Ask it to describe its approach before writing any code, and fix misunderstandings there, where they are cheapest.

Small steps, for two reasons. Long threads push early instructions out of the context window, and a small task is one you can check. The test from 6850: if you cannot name one number the answer must contain and work it out yourself, the task is too big. Split it.

Commit before each task

• Commit your work before you ask the tool to change anything.
• Afterwards, look at what changed, line by line.
• Trust the changes you can see over the tool’s summary of them.
• Undo anything you did not ask for.

Step by step: AEM 6850, meeting 7 on version control with git

Ninety seconds. This is the git slide from the start of the hour, in practice. Every task starts from a saved state and ends with a list of exactly what changed. That list is the diff, and it is the honest account of what the tool did; its own summary is a claim like any other.

For teams working only in chat, the same habit is saving each version of a script under its own name before you change it.

How to acquire data

• Download if you can. It is boring and usually fastest.
• An API: a key, a rate limit, stable fields, a record of your request.
• Scrape last. Pages change, rows vanish silently, terms often forbid it.
• Every route has a wait, from days to months. Register now.
• The full list, with waiting times, is on the data sources page.

Where to get data

Ninety seconds, and resist reading a catalogue of sources from the front. The page does that better and it has the waits attached, which a slide cannot.

The one thing worth saying out loud is the timing, and that getting it wrong is the commonest way an MPS capstone dies. A library database is available now. Dewey takes days. A data-use agreement takes months and may fail. A team that discovers in November which row it is in has no project.

For teams with no file yet, the named route, the wait and a plan B are worth writing down now. Once the file arrives, the next slide is the first thing to do with it.

Explore the data before you analyze it

• Spend time with the file before any analysis.
• What is one row? What are the units? How is a missing value written?
• Plot it: maps for spatial data, line plots for time series.
• The plot shows the structure and the patterns in the data.
• It is also a check. A bad merge shows up as a hole or a jump.
• You cannot spot a wrong answer about a file you never looked at.

Ninety seconds. These are the questions to ask of any file, including the one your team is betting on.

What one row is decides every count they will later quote, and it is the question teams get wrong most often.

Then plot it, before any model. Spatial data go on a map, time series on a line plot, one line per unit when there are few. The aim is to see the structure: which units, which periods, where the data are dense and where they are thin. Whether every unit appears in every period is the question nobody asks, a plot answers it at a glance, and it is how a year-on-year comparison ends up comparing nine months against twelve.

Looking is also the cheapest quality check there is. A merge that dropped a state shows up as a hole in the map. A column that switched from thousands to units shows up as a cliff in the line. Ask of every plot whether you can spot something wrong.

The last bullet is the argument for all of it. Checking needs something to check against, and time with the raw file is what gives you that.

The worst errors raise no warning

• Code that fails is easy to fix. Code that runs and is wrong is the danger.
• The usual shapes: a merge that drops or duplicates rows, missing values handled silently, a unit mismatch.
• Reinhart and Rogoff (AER P&P, 2010)
  • Reported slower growth once public debt passes 90% of GDP.
  • A spreadsheet formula left out 5 of their 20 countries.
• Public Health England (October 2020)
  • Moved COVID-19 test results through an old Excel format capped at 65,536 rows.
  • 15,841 positive cases went unreported for a week, with no error.

Reinhart and Rogoff (AER P&P, 2010) · Herndon, Ash and Pollin (CJE, 2014) · PHE statement, 4 October 2020

Two minutes. The first two bullets are AEM 6850’s: the danger is code that runs.

Reinhart and Rogoff’s 90 percent threshold was quoted by policymakers arguing for austerity after 2010, which is why the error mattered. Corrected, the headline growth figure for high-debt countries went from minus 0.1 percent to 2.2 percent. Be fair about it: the spreadsheet error was only 0.3 points of that gap, and undisclosed choices about years and weights made the rest (Herndon, Ash and Pollin). Public Health England: an old Excel format stopped at 65,536 rows and said nothing, and about 48,000 close contacts were not traced in time.

Neither of these is a statistics error. Both were invisible in a spreadsheet and would have been visible in code, which is why the next slide is about checks.

Build checks into every step

• Ask for code you can rerun, not just an answer.
• Before you run it, write down the number you expect.
• A confident “no problems” is not a check. Ask for the code and the count.
• Check big claims in a fresh conversation.
• When a check fails, fix that one thing.

Adapted from AEM 6850, meetings 8 and 9 · Ferman and Finamor (arXiv, 2025)

Two minutes. Five habits, and each one is general.

Code you can rerun is AEM 6850’s rule that the research artifact is always code. A number in a chat, even one computed in the tool’s sandbox, cannot be rerun next week by anyone else. OpenAI’s own help page says to review the generated code, outputs and assumptions before relying on the result.

Write down the number you expect first. Ferman and Finamor ran a randomised experiment inside real recruitment for research jobs: people were almost 20 percent more likely to find a planted coding error when it produced an unexpected result. So the errors that confirm your expectations are the ones that survive. Better still, write the expected count into the script as a line it must pass, so the check travels with the code.

The confident “no” is the commonest way a check goes wrong: someone asks whether there are duplicated rows and takes the answer on trust. A fresh conversation that has seen none of the work is the reviewer; a model checking its own work is biased toward what it just wrote.

Fix one thing at a time. Asking for a whole new version throws away the reading you already did.

Five checks you run on any file

1. Row counts across a join. More rows: a duplicate key. Fewer: dropped rows.
2. Totals against a published figure.
3. Check a few raw records by eye.
4. Orders of magnitude. Is it 58 or 58,000?
5. Obtain the same result using an independent route.

Two and a half minutes. These are five checks worth running on any file, so give them in these words and let the room write them down.

Check five turns on the word independent. Two routes over the same bad file can agree and both be wrong. Double programming in clinical trials has exactly this blind spot, and the reviews of it say plainly that it validates outputs rather than inputs or logic. The strongest second route uses a different source, a different person, or both.

Then the framing that makes the list learnable: each check is the answer to “how would you know it was wrong” for one shape of fault. You pick the check by the shape you are worried about, which is why a team that finds a fault and cannot name the check has not learned the thing.

6. Disclosure

Thirty seconds. Say that the last two slides apply to everything handed in, wherever AI entered the work.

Disclosure is required

• Every deliverable in this course carries a disclosure block.
• Leaving it out is an academic integrity violation.
• Cornell’s Code counts AI help “not explicitly permitted by the instructor” as unauthorized assistance.
• The AER: AI use “must be briefly described during the submission process”.

Syllabus, AI policy · Cornell Code of Academic Integrity, 2026 · American Economic Review, editorial policy

One minute. The first two bullets are the syllabus rule. The third is why the second holds: this course permits AI with a disclosure block, so use without one is aid the instructor did not permit.

Cornell’s new Code took effect on August 24 and names artificial intelligence twice. The second is under fabricated sources: citing “false citations generated by artificial intelligence or other tools” is a violation, and students are responsible for every source they cite “regardless of how those sources were generated or obtained”. That is section 3’s rule with the university’s name on it, so say it.

The AER line is there so nobody hears this as a classroom rule. The journal asks for the same description at submission, and says authors are “solely accountable for, and must thoroughly fact-check, outputs created with the help of AI software”. Both texts were checked against the live pages on 23 September 2026.

What you write down about what you used

Three lines on every deliverable:

Tools. Claude Sonnet 5 (claude.ai) and ChatGPT 5.
Used for. Drafting the cleaning script. Auditing the outline of section 2.
Checked. Row counts before and after the join. Every reference opened.

• Too vague: AI, help with the code, it worked.
• Naming a mistake the tool made counts in your favour.

Ninety seconds. Show a filled-in block rather than an empty template, because a template teaches nothing and every disclosure I get back otherwise says “AI, for help with the code, and it worked”.

The three failing answers in the first bullet are the marking criteria stated as examples, so read them.

While this slide is up, answer the question they actually want answered, and answer it once. I do not run AI detectors. OpenAI withdrew its own because it caught 26 percent of AI text and falsely flagged 9 percent of human text. Seven commercial detectors falsely flagged 61 percent of essays by writers whose first language is not English, against 5 percent for US school essays. Vanderbilt turned Turnitin’s detector off because a 1 percent false positive rate across their submissions is about 750 students accused wrongly. The standard here is that you disclose and you verify.

Then close the hour by saying what comes after the break: a live demo of how to use AI to get a research project started. Next week’s two deliverables, a two-page project description and the team’s working agreement, will be posted on Canvas.