7 · Model failures
Wednesday, October 7, 2026
What you should be able to do after this meeting. Take a number apart and name the failure that produced it. Say which way it pushes the estimate, and whether more data would fix it. Then say what your own controls would have to close, and what they cannot.
Where to read more
Remix ch. 5 on unconfoundedness. Adams ch. 14 on reliability and validity.
Session A · 2:00–3:15
Four kinds of failure. Bias moves the estimate. False precision leaves the estimate where it is and shrinks the error bar around it. Low power leaves the error bar correct and too wide for the question. External validity asks whether a correct estimate holds for the population the decision concerns. Bias first, and there are five. Selection and sorting: the retention email went to the customers already most likely to renew, so the lift is who was picked, not what was sent. Omitted variables, with a rule for the direction — the omitted thing’s effect on the outcome, times its correlation with the treatment. Two signs multiplied. You can usually work out which way you are wrong before you have the data. Measurement error next. Classical error in the variable you care about pulls the estimate toward zero, always: self-reported spend, survey income, recalled hours. Non-classical error does not. People round, they under-report, and top-coded income errs one way only. Then it can go anywhere. Fourth, endogeneity and reverse causality. More police, more crime. The firm raised price in the quarter demand was strongest. Fifth, selection into the sample, the cousin of selection into treatment: trainees who found no job stop answering the follow-up survey, and only the firms that survived appear in the data. And then the question I will ask of every one of them: does more data fix it? No. A biased estimate with a million rows is a more precisely wrong number. Then the fix everybody reaches for, and its ceiling. Conditioning on observables, drawn on the graph — which back doors a set of controls closes, and the two controls that make things worse. One sits on the path from cause to effect. One opens a path by being conditioned on. Then matching, which is the same claim made by comparison rather than by regression, and it needs somebody to compare to. If no untreated store looks like your treated stores, there is nothing there. Then the ceiling. LaLonde put observational estimates with controls up against a randomised benchmark, and they did not reproduce it. ‘I added controls’ is a claim about the graph, not a technique. So state the claim out loud, and report how far your coefficient moved when the controls went in. Then the quieter family, which leaves the estimate untouched. Heteroskedasticity, and dependence ignored over time, within groups or across space: five hundred shoppers in ten stores is closer to ten observations than five hundred, and Bertrand, Duflo and Mullainathan found placebo policies in state panels rejecting at 45% instead of 5%. Multiple testing: twenty subgroups, one significant, one slide. Specification search: Silberzahn gave 29 teams one dataset and one question, and got estimates from 0.89 to 2.93. None of the six moves the number. All six make you sure of it. Then power. Last week’s checkout test could detect a lift of about 1.3 points, and median power across economics is 18% or less. The significant results of an underpowered study overstate the effect. With clustered data, power depends mostly on the number of clusters, and strata built on baseline outcomes raise it at little cost. Last, external validity. Opower’s first ten experiments overstated its energy savings in the next 101 sites. Name the population your decision concerns, and check that your sample covers it.
Session B · 3:30–4:30
Every session B opens with a ten-minute team check-in: what you did since last week, what is stuck, and who is doing what next. Written down, and handed in with that session’s work.
Ten minutes of team check-in, then the results-critique cards again — the same eight analysts, dealt fresh, and this time each one has come back with a fix. They added controls. Fifteen minutes to rule on it. Name the failure in the morning’s vocabulary. Say which way it pushes the estimate. Say whether more data would fix it. Then rule on the fix itself: does the control close a back door, sit on the path from cause to effect, or open a door by being conditioned on? Then one team is drawn at random to defend at the front while the room attacks, and we draw again as far as the clock allows. Nobody knows in advance whether they are up. Points for attacks that land, and you have to name what is wrong, not just dislike it. Card 7 is the one to watch. The advertising model with an R² of 0.94, now with month and competitor spend added, and still uninterpretable — the budget was set from the sales forecast, so the arrow runs backwards, and no control fixes an arrow that runs backwards. Then your own project. One threat, the one your design actually has to survive. Write down which way it pushes your estimate, whether more data would fix it, the variable that would close it, and whether you can obtain that variable before January. If you cannot, write the sentence you will put in your limitations section instead. One page, per team, handed in before you leave. You will be answering that threat in November.
Your team’s output is submitted before you leave.
Practice questions
These 10 questions practice the ideas of this lecture. Click an option to check it. A wrong pick lets you try again, and the right one links to the slide that covers it. The quiz on this lecture asks different questions on the same ideas.
1. A credit union reminds a random half of its retired members to save. Their balances end a precisely estimated 3% above the other half's. It now plans to remind its young members. Which failure threatens using the 3% estimate for that decision?
2. A hospital gives a discharge checklist to a random half of 2,000 patients and regresses each patient's length of stay on the checklist. Clerks record stays with random errors unrelated to the checklist. What would 4,000 patients do to the estimator?
3. A rental firm has six years of rents for 300 buildings. In year 4, some owners chose to add a fitness room, mostly in desirable, higher-rent neighborhoods. Neighborhood quality did not change. How should the firm estimate the room's effect on rent?
4. A toy maker assigns shelf spots by lottery and scores each spot's visibility in an audit. A quarter of the score's variance is random error. It regresses each store's units sold on the score, and the slope converges to 3. What is a point's true effect?
5. A state agency picks which manufacturers get export coaching. It regresses export growth on coaching across all of them, applicants or not, controlling for size, sector and past exports. Which way of picking makes that regression free of selection bias?
6. Truck fleets choose whether to install fuel-saving software. The vendor regresses each fleet's fuel cost on installing it, and the coefficient implies 15% lower costs. It says only that it controlled for everything. What should a reviewer ask for?
7. An insurer randomly picks regions for a discount and regresses each town's claim rate on the discount. Towns in a region share weather. The estimate is −2.0, with standard errors of 0.5 by default and 1.5 clustered by region. What should it conclude?
8. A paint maker randomly assigns drying times to batches and regresses reject rate on drying time. The scatter around the fitted line is small at typical times and large at extreme ones. How does the default standard error compare with the correct one?
9. A mobile carrier gives a new data plan to a random half of 600 customers and regresses each customer's monthly data use over 2 years on the plan. A customer's use runs high or low for several months at a time. Which standard error should it report?
10. A car-rental firm plans a test in which a random half of customers get a prepay discount. The estimator of the effect on the no-show rate will have a standard error of 1.5 percentage points. Does a two-sided 5% test have 80% power for a 3-point drop?