AEM 6991

MPS Capstone Project I

Prof. Ariel Ortiz-Bobea

2 · Theory and causality

Wednesday, September 2, 2026

Theory

The board is asked to act on a 40% gap

A grocery chain. Half its customers have installed the loyalty app. The analytics team reports to the board:

$140
a month, customers on the app

$100
a month, customers not on the app

“App customers spend 40% more. Sign the other half up and they will too.”

The board is asked to fund the sign-up campaign. Does it? Thumb up or down.

Two minutes. Thumbs held at the chest, up or down, so nobody is exposed. Count it out loud so the vote is on the record.

Take two reasons from the NO side. Do not resolve it. Do not name the failure. This claim comes back at “The counterfactual is those customers without the app”, twenty-five minutes from now, and the whole of block 3 is the answer to it.

The two numbers are true. The sentence after them is the leap. Do not say that yet either.

Most rooms split. The ones who say no usually cannot say why yet, which is exactly the gap the hour fills.

A theory makes a prediction you could be wrong about

• A theory is a structured framework of ideas used to explain facts, behaviours or phenomena.
• It starts from assumptions and ends in predictions.
• Four checks:

1. It says why, not only what.
2. It states its assumptions.
3. It predicts what you would see if it is right.
4. It says what you would see if it is wrong.

In a meeting you say: what has to be true for this to work, and what would prove me wrong?

Ninety seconds. Do not embellish the definition.

The code-switching line at the bottom recurs on about eight slides today. Say it out loud every time. The point of the device is that they leave able to say these things to somebody who has never heard the word confounder.

A prediction is not a theory

• A prediction says what will happen. You judge it by how often it is right. It can be right for no reason.
• A theory says why it happens. It names its assumptions, so it also says what changes if you change something.
• Only a theory tells you what to do.

Two minutes, and this is the block’s hinge.

Say it plainly: a model that fits beautifully can be about nothing. Fit is not evidence of mechanism.

This is correlation and causation wearing a machine learning hat, and they will meet it as a job. Meeting 3 warns that the prediction archetype dies when a team quietly starts making causal recommendations off a predictive model. Meetings 6 and 7 come back to it. Plant it here.

A prediction with no theory behind it fails without warning

• Google predicted flu doctor visits from search terms.
• For years it tracked the CDC count well.
• In February 2013 it predicted over twice the CDC count.
• Searching changed, not illness. The flu was in the news.

Schematic after Lazer et al. (2014), The Parable of Google Flu, Science 343, 1203–1205. Shapes, not data.

Ninety seconds. The point is not that big data is bad.

Why it went up, because somebody will ask. Three reasons, all in Lazer and co-authors. First, the 2012-13 season was severe and widely reported, so people who were not ill searched for flu terms; search volume followed the news, not the clinics. Second, Google changed its own search engine underneath the model: suggested searches and related-term prompts steered people toward flu queries, so the same level of illness produced more flu searches than before. Third, the model was a fit of 50 million search terms to about 1,150 CDC data points, so it had picked up terms that merely happened to move with flu season. It had no theory of why a search meant an illness, so it could not tell “people are sick” from “people are reading about flu”. It overestimated in 100 of 108 weeks from August 2011.

The point is that nobody could say why it worked, so nobody could say when it would stop. A model with no mechanism gives you no warning. You find out in production.

Lazer and co-authors argue the errors were largely avoidable, and part of the cause was that Google’s own search algorithm kept changing underneath the model. The thing being measured moved and the model could not know.

The consulting version, and worth saying out loud because it is where most of this room is heading: the deck says the intervention lifted revenue 14% at the pilot site. The client asks whether it will work in their market. Without a theory there is no honest answer. A number without a theory is a fact about one place at one time.

Which of these is a theory?

1	Sales are higher in December
2	Customers who contact support twice a month are three times more likely to leave
3	Customers leave when the cost of switching drops below the annoyance of staying
4	Firms should invest in sustainability
5	Firms maximise profit, and whatever they did was the profit-maximising thing

Hold up 1, 2, 3 or 4 fingers. 1 theory · 2 description · 3 prediction · 4 value judgment

Three minutes including the reveal.

Give them thirty seconds, then take all five at once on fingers. You are watching for the spread, not calling on anyone.

Two and three are the exercise. Both are about churn. Only one tells you what to do.

Five is the discussion. It looks the most like economics and it is the only one that cannot be wrong.

Only one of the five is a theory

1	Sales are higher in December	description
2	Customers who contact support twice a month are three times more likely to leave	prediction
3	Customers leave when the cost of switching drops below the annoyance of staying	theory
4	Firms should invest in sustainability	value judgment
5	Firms maximise profit, and whatever they did was the profit-maximising thing	cannot be wrong

Two and three are the same subject. Only three tells you what to change.

Ask which of two and three you would take to the executive who has to decide something. Two says who will leave. Three says what to do about it, and it names two levers: switching cost, and annoyance.

Point at four and say you are coming back to it in twenty minutes. It is the seed for the block on what data cannot settle.

Five: ask what result would be inconsistent with it. There is none. That is the tell.

Run the four checks from the definition slide out loud. One and two fail “says why”. Four makes no prediction about what you would see. Five cannot be wrong. Only three passes all four.

Two theories predict the same thing for different reasons

Human capital

• School makes you more productive.
• The degree raises your wage because you learned something.

Signalling

• School reveals who was already productive.
• The degree raises your wage because it separates you.

Both predict that graduates earn more. They cannot both be the reason.

Take thirty seconds. What would you look at to tell them apart?

Spence (1973), Job Market Signaling, Quarterly Journal of Economics 87(3), 355–374

Two minutes. This is the better of the two examples, because agreeing on the prediction is what makes the comparison visible.

The policy stakes are opposite. If human capital is right, more schooling makes people more productive and subsidising it grows the pie. If signalling is right, a year of school mostly reshuffles who gets which job.

Signalling predicts a jump at the diploma, not a slope

• Human capital: the return is smooth. Each year of school adds learning, so each year adds wage.
• Signalling: the return jumps at the certificate. The final year is worth more, because that is the year you get the paper.

Compare someone who finished with someone who did three and a half years and left. Same schooling, different paper.

The “sheepskin effect”. Hungerford and Solon (1987), Review of Economics and Statistics 69(1), 175–177, and a literature after it, find substantial premia at the diploma year

Two minutes, and this is the payoff of block 1.

Say the general lesson out loud, because it is the thing to remember: a theory earns its keep by telling you which comparison settles the argument. That is the same job the identification section of their proposal does in December.

Be honest about the state of the evidence. Diploma-year premia are robustly found, and they are evidence that some of the return is signalling. They do not settle the split, and almost nobody thinks it is all one or all the other.

What data can settle

Data can measure, rule out a sign, and choose between theories

Measure a magnitude	how big, with what uncertainty
Rule out a sign	it is not positive, whatever else it is
Choose between two theories	when they predict different things about the same comparison

Ninety seconds. The third is the one they under-rate and it is the one that ties this block back to Spence.

Say the shape: you do not test a theory, you test it against another one. A comparison that both theories predict identically is not evidence, however clean the estimate.

When you test with data, you test four things at once

• Your theory says the app raises spending. You test it. The number says no.

1. The theory. The app does not raise spending.
2. The measure. You counted in-store spend and missed online.
3. The sample. You looked at new customers only.
4. The analysis. You fitted a straight line to something that curves.

• Any one of the four produces that “no”. The number does not say which.

In a meeting you say: the number disagrees with us, and we do not yet know whether the idea is wrong or the data is.

Two minutes. Walk the four one at a time on the app example, because the abstract version of this slide did not land.

The practical version: when the result comes back with the wrong sign, the first move is not to abandon the theory. It is to work out which of the four is the weakest link, and usually it is the measure.

Can data settle this?

1	Does the loyalty programme increase spending
2	Is the minimum wage too high
3	Did the ad campaign pay for itself
4	Will this model still work next year
5	Should we prioritise growth over margin

Hold up 1, 2 or 3 fingers. 1 yes · 2 no · 3 only with an assumption you have to defend

Two and a half minutes with the reveal. Fingers, three categories.

Watch for the room putting 3 in “yes”. It is the interesting one: paying for itself needs a counterfactual, so it is only answerable with an assumption about what would have happened without the campaign.

Two of the five are not empirical questions

1	Does the app increase spending	yes, with a design
2	Is the minimum wage too high	no. “Too high” is a judgment
3	Did the ad campaign pay off	only with an assumption
4	Will the model work next year	only with an assumption
5	Growth over margin	no. That is what you want

Data can price a trade-off. It cannot tell you the trade is worth making.

Two is the one to dwell on. Data can tell you what a minimum wage costs in jobs and what it gains in incomes. Whether that trade is worth making is not in the data, and no sample size changes it.

Four is worth thirty seconds: it is the Google Flu slide again. Whether the model still works next year depends on whether the mechanism holds, and you cannot check that without a mechanism.

In December you show the comparison, not prove the theory

• You are not proving your theory in the proposal.
• You are showing which comparison your design makes.
• And what would have to be true for that comparison to mean what you claim.

That paragraph is the identification section. It is the hardest page in the document and it is worth more than the results.

One minute. Say it in September so November is not a surprise.

The failure mode: teams write the identification section as a description of the method. What it has to be is the assumption, named, plus what would break it.

Causality

Every customer has two outcomes, and you see one of them

Every unit has two outcomes, the thing you watch: here, spending. One with the treatment, the thing you change: here, the app. And one without.

Customer	On the app	Not on the app
Ana	$152	?
Ben	?	$98
Chen	$131	?
Dee	?	$104

Holland (1986), Statistics and Causal Inference, JASA 81(396), 945–960

Two minutes. Put the table up and say nothing for a moment. The column of question marks is the whole block.

Holland calls it the fundamental problem of causal inference. The causal effect for Ana is the difference between her two numbers, and one of them does not exist.

Causal inference is a missing data problem

Not a statistics problem.

• No estimator recovers a number that was never observed.
• Every method in this course is a different argument about how to fill that column in.
• So every comparison is a stand-in for a world you cannot see. The only question is whether the stand-in was any good.

Forty seconds. This reframing does more work than anything else in the lecture.

Students arrive believing causal inference is a harder kind of statistics. It is not. It is an argument about what to substitute for the missing column, and the statistics comes after you have won it.

The counterfactual is those customers without the app

“App customers spend 40% more. Sign the other half up and they will too.”

• The counterfactual is what those same customers would have spent without the app.
• It is not what the other customers spent.
• The other customers are the stand-in. Were they ever like the app customers?

Who signs up for a loyalty app? Say it out loud.

Three minutes, and it is the payoff for the opening vote. Remind them they voted.

Ask the room, open, no names: who signs up for a loyalty app? They say it themselves in about four seconds. People who already shop here a lot.

That is the whole failure, and it came from them rather than from you. The 40% is mostly who chose to join.

Say what would settle it: randomise the invitation, or find people who joined for a reason unrelated to how much they shop. Both are designs they will meet later.

Then answer the vote, because they cast one. The two numbers are real. The decision is not supported by them. On this evidence the board does not fund it, and the people who voted no were right without yet being able to say why.

What is the right comparison?

A store gets a refit. Sales rise 12% the following quarter.

1	The same store, the quarter before
2	Every other store in the chain, same quarter
3	Stores that were also due a refit but have not had it yet

Hold up 1, 2 or 3 fingers. Which stand-in is best?

Two and a half minutes including the reveal.

Expect a split between 1 and 3. Two is usually rejected quickly, which is the right instinct.

The best stand-in still needs an assumption

1	Same store, quarter before	ignores anything else that changed
2	Every other store	refitted stores were picked, not drawn
3	Stores due a refit, not yet done	closest, if the queue order is arbitrary

Three is the best available and it is still an assumption. You are claiming the queue order has nothing to do with sales.

Two minutes. The lesson is that there is no comparison that needs no assumption, only comparisons whose assumption you can state and defend.

Three is the shape of a design they meet in October. Do not name it.

Ask two questions of every number

Compared to what?

Would those two groups have looked the same anyway?

Every number in every deck, for the rest of your working life.

Forty seconds. Say it slowly. If they keep two things from today, these are the two.

A causal graph is a picture of what causes what

1. Causal graph, or DAG. Circles are things you could measure. An arrow means “causes”. Example: heavy shopper → spend.
2. Treatment, marked X. The thing you would change. Example: putting a customer on the app.
3. Outcome, marked Y. The thing you would watch. Example: monthly spending.
4. Z. A third thing that touches both. Where its arrows point is the whole question.
5. The red arrow is the effect you want to measure. Grey arrows are everything else.

Ninety seconds. Read it once, slowly. Every picture today uses only these parts: three circles, arrows, one red.

Say the fixed layout: X bottom left, Y bottom right, Z on top, on every graph today. The names change, the positions do not.

Five words for how a comparison breaks

1. Confounder: causes both treatment and outcome. Example: heavy shoppers join the app and spend more.
2. Collider: caused by both. Example: being hired is caused by the test and by the interview.
3. Mediator: on the path from treatment to outcome. Example: a price cut brings people in, and they buy.
4. Back door: a route from treatment to outcome that is not the effect. Example: app ← heavy shopper → spend.
5. Condition on, or control for: compare within a variable’s values. Example: heavy shoppers with heavy shoppers.

Ninety seconds. Definition, then example, for each. The next three slides draw words 1, 2 and 3, and the three claims after the rule test them, one each, in the same order.

Say that “control for” and “condition on” are the same act, because they will hear both and think they are two things.

A confounder causes both and opens a back door

• A confounder causes both the treatment and the outcome. Here: heavy shoppers join the app, and heavy shoppers spend more.
• That opens a back door. Compare heavy shoppers with heavy shoppers: that is conditioning on it, and it closes the door.

In a meeting you say: the two groups were already different before anything happened.

Two minutes. This is the one they already half know, so move.

Note the fixed layout out loud, once: X bottom left, Y bottom right, Z on top, on every graph today. Only the arrows move. Say it now and the next three slides read themselves.

Among the hired, a link appears that is not there

• Being hired is caused by the test and by the interview. That makes it a collider.
• Either was enough to get hired. So among the hired, a weak interview comes with a strong test.
• Across all applicants the two are unrelated. Looking only at the hired created the link.

Three minutes. This is the hardest concept of the day, so slow down.

Define, then the example, then the picture. Either a strong test or a strong interview was enough to get hired. So among the hired, someone with a weak interview must have had a strong test, or they would not be here. The trade-off is manufactured by looking only inside the box. In the applicant pool the two are unrelated.

Point at the arrows: they point INTO the top node here. On the last graph they pointed out of it. Same three positions.

The test of whether this landed is Good to Great, after the rule.

Selecting on the outcome is conditioning on a collider

• Among the people we hired
• Among our customers
• Of the startups that made it
• Looking at the funds that survived

• Selecting on the outcome is conditioning on a collider.
• It is the most common broken claim in business analytics, and it never looks broken.

In a meeting you say: you only looked at the winners, so you invented a trade-off.

Ninety seconds, and this is the transferable slide.

It is the same structure in medicine, where among low birth weight infants maternal smoking looks protective, and among people who already have a disease obesity sometimes looks protective. Same shape, different field. Mention it in one sentence so they see it is not a business quirk.

Control for the middle step and the effect disappears

• A mediator is the middle step from treatment to outcome. Here: a price cut brings people in, and they buy.
• Control for foot traffic and the price cut has nothing left to do. The effect you came for is gone.

In a meeting you say: you controlled away the thing you were looking for.

Two minutes.

The arrows here go X to Z to Y. That is the whole effect, running through Z. Block the middle and there is nothing left to find.

The tell in the wild: somebody adds a control, the effect disappears, and they report the disappearance as good news.

One rule decides what to condition on

Close the back doors. Leave the front door alone.

The back door is the route through the confounder: close it. The front door is the effect itself, through the mediator: leave it alone. And do not select your sample on the result.

One minute. Nothing to add. Let the picture do it.

If they photograph one slide today it should be this one or the dictionary.

Every control is a claim about the graph

• Controlling for a variable means comparing within its values. Conditioning on it is the same act.
• Every variable you control for is a claim about the graph. It is not a safety measure.
• A regression with every control you could find is a graph nobody drew.

If the coefficient moves when a control goes in, that is neither good news nor bad news until you can say which of the three that variable is.

Two minutes, and this is the most portable thing in the lecture.

The Berkeley case is the cleanest illustration if you have time. In 1973 Berkeley admitted 44% of male applicants and 35% of female applicants. Control for department and the gap mostly vanishes, because women applied in larger numbers to departments that admitted fewer people.

Now the question that makes the point: is department a confounder or a mediator? If applicants chose departments for reasons unconnected to the admissions process, it is a confounder and you should condition. If they were steered there, it is on the path and you have just controlled away part of what you were measuring. Same variable, same data, opposite handling, and only the causal question tells you which.

Bickel, Hammel and O’Connell (1975), Science 187, 398. Link is on the manual page.

What you hear, and what it is

What you actually hear	What it is
Firms that do X grow faster	confounder
Students who eat breakfast score higher	confounder
Among the people we hired, the good testers interviewed badly	collider
Nine of the top ten founders dropped out	collider, by survivorship
Once we control for X the gap disappears	mediator
More police, more crime	reverse causality
We sent the team to our worst stores and they improved	regression to the mean
Reported cybercrime doubled	the measure changed

Three are graph structures. Reverse causality is an arrow the wrong way round. The last two cannot be drawn.

Two minutes. This is the slide to photograph and you should say so.

Read the left column out loud in the voice somebody would actually use. The right column is the translation, and going back and forth between the two is the skill.

Do not work these. They are a reference. Four of the eight get worked in the next block.

Three claims, worked

Firms that hire consultants grow faster, so hire consultants

“Firms that hire consultants grow faster than firms that don’t. So hire consultants.”

A consulting firm’s own marketing deck

Hold up 1, 2, 3 or 4 fingers. 1 confounder · 2 collider · 3 mediator · 4 not a graph problem

Two minutes with the vote and the count. The first of three, one per structure, in the order they were defined.

Take one reason from the floor, no names, then reveal.

Growing firms could afford consultants

• Firms already growing could afford consultants. Growth caused the hiring.
• So the comparison is between growing firms and the rest. The back door is open.

In a meeting you say: the two groups were already different before anything happened.

Two minutes. This is the confounder, the structure nearly every project in the room will meet.

The fix they can name: compare firms that were growing at the same rate before the hire. Say the honest caveat: you can compare within past growth and still not be sure the door is closed, because being on a growth path is not the only reason a firm hires consultants. That is the unmeasured confounder, and October is about closing doors you cannot measure.

Good to Great picks eleven winners and asks what they share

Eleven companies beat the market by at least three times over fifteen years. The book finds what they have in common and recommends it.

The best-selling management book of its generation

Hold up 1, 2, 3 or 4 fingers. 1 confounder · 2 collider · 3 mediator · 4 not a graph problem

One minute on the vote. Some of them have read it.

Be fair to the book before you break it. It is careful, it has comparison companies, and the research effort was real. The problem is structural and no amount of care inside the design fixes it.

The sample was chosen on the outcome

• The eleven were chosen because they became great.
• Whatever else also caused that is now correlated with the practices, inside the sample.

Circuit City filed for bankruptcy in 2008. Fannie Mae was placed in conservatorship the same year.

Two minutes. Same structure as the hiring puzzle, arriving from somewhere completely different. Say that out loud, it is why this example is here.

The missing comparison is the firms that did all the same things and did not become great. They are not in the book, because the book selected on the outcome.

The two failures are not the argument, and do not oversell them. Plenty of good companies fail. The argument is the structure. The failures are just the reason people noticed.

HR controls for salary band and the gap disappears

“Once we control for salary band, the gender gap in promotion disappears. So there is no bias in promotion.”

An HR review, closing the matter

Hold up 1, 2, 3 or 4 fingers. 1 confounder · 2 collider · 3 mediator · 4 not a graph problem

One minute on the vote. Expect confounder from a good fraction, which is the mistake worth having.

The bias works through the salary band

• If the bias works by placing women in lower bands, the band is how the effect travels.
• Control for it and you have controlled away the thing you were looking for.

The disappearance is not the finding. It is the mechanism.

Two and a half minutes. Must land.

Say the honest caveat: if bands were assigned before anything the firm did, and for reasons outside it, the band could be a confounder instead. Which it is depends on the causal question, not on the data. This is “Every control is a claim about the graph” again, and it is why that slide comes first.

The right analysis is the total effect, gender to promotion, with nothing on the path conditioned on.

Every design ahead closes a back door

• Each one closes a back door in a different way.
• You will draw each of them as a graph before you hear its name.
• Next week you generate questions. The two questions you ask of every number are the filter that kills the bad ones.

Thirty seconds. Do not list the designs. They cannot hold names for methods they have not met, and the sentence is the whole point.

After the break, your own project

Sit at your numbered table	your team is on the sheet
Ground rules, one page, signed	while you still get along
Your project’s causal graph	on paper, by hand
Every team reports	two and a half minutes, hard stop

Both sheets go in before you leave.

One minute. Say that the timer is real and that it is nine teams.

Say what the report is: your question, and the graph. Nothing else.

Three questions, three minutes

On your own device. This one tests the plumbing and does not count.

Six across the term. Lowest one dropped. Each closes session A.

Three minutes. Poll Everywhere, items in _quizzes/s02-causality-import.txt.

Say two things before you start. This one is a test of the plumbing and it does not count. And the policy: six across the semester, lowest one dropped, each closing session A.

If you told the room “lowest two dropped” on August 26, correct it now. The slide behind you said one.