Regression and natural experiments

Regression analysis finds the line, or with several variables the plane, that best predicts an outcome from other variables; its slope measures how the outcome changes with each one. A natural experiment is an event that changes conditions for one group but not for a comparable group, and the difference-in-differences estimate of its effect is the change in the treated group’s outcome minus the change in the comparison group’s.

From theory to a testable prediction

Theory delivers a sign; evidence measures a size. Human capital theory predicts that schooling raises earnings (schooling as investment). To test it, collect data on many people or occupations, each an observation of years of schooling $S$ and the wage $w$, and estimate

$$\log w_i = \alpha + \beta S_i + u_i,$$

where $\alpha$ is the intercept, $\beta$ the slope and $u_i$ the error term: everything other than schooling that affects observation $i$‘s wage. The prediction is $\beta > 0$; the regression estimates $\beta$.

Fitting a line by least squares

For any candidate line, the residual of an observation is its vertical distance from the line: actual minus predicted, $\hat u_i = \log w_i - (\hat\alpha + \hat\beta S_i)$. Ordinary least squares (OLS) picks the $\hat\alpha$ and $\hat\beta$ that minimize the sum of squared residuals, $\sum_i \hat u_i^2$.

The residuals are squared for three reasons: positive and negative misses would otherwise cancel, so a bad line could score zero; large misses are penalized more than small ones; and the minimization has a unique solution with a formula,

$$\hat\beta = \frac{\sum_i (S_i - \bar S)(\log w_i - \overline{\log w})}{\sum_i (S_i - \bar S)^2}, \qquad \hat\alpha = \overline{\log w} - \hat\beta \bar S .$$

The second equation says the least-squares line always passes through the point of means $(\bar S, \overline{\log w})$.

Slope of your line

0.050

Height of your line at 13.35 years

3.14

Your line: intercept

2.473

Your line: slope

0.050

Sum of squared residuals

0.196

Smallest possible sum (least squares)

0.079

Open in Graph Lab

Figure 1. Ten occupations (made-up data) plotted by average years of schooling and average log hourly wage, with residuals as dashed purple segments. The least-squares line, log wage = 1.803 + 0.100 × schooling, passes through the point of means (13.35, 3.14) and has the smallest possible sum of squared residuals, 0.079. A line through the same point with slope 0.05 has a sum of 0.196.

Reading a log-wage regression

When the dependent variable is in logs, a coefficient is a semi-elasticity: multiplied by 100, it is the approximate percent change in the wage when the variable rises by one unit. In Figure 1, $\hat\beta = 0.100$, so one more year of schooling is associated with wages about 10 percent higher. The exact change is $100(e^{\hat\beta} - 1)$ percent, here 10.5 percent; the approximation is close for coefficients below about 0.1 and understates larger ones.

Multiple regression adds variables. In $\log w = \alpha + \beta_1 S + \beta_2 X + u$, the coefficient $\beta_1$ measures the effect of schooling holding $X$ constant, that is, comparing people with the same $X$. A typical output, with made-up numbers and standard errors in parentheses:

$$\widehat{\log w} = \underset{(0.060)}{1.420} + \underset{(0.004)}{0.082}\,\text{schooling} + \underset{(0.006)}{0.031}\,\text{experience}, \qquad R^2 = 0.24 .$$

Among workers with the same experience, a year of schooling is associated with wages about 8.2 percent higher.

How precise is an estimate?

• The standard error measures how much an estimate would vary across samples. A rough 95 percent confidence interval is the estimate plus or minus two standard errors: $0.082 \pm 0.008$.
• The t-statistic is $t = \hat\beta / \text{se}(\hat\beta)$. If $|t|$ exceeds about 2, the estimate is statistically significant: the hypothesis $\beta = 0$ is rejected at roughly the 5 percent level. Here $t = 0.082 / 0.004 = 20.5$.
• $R^2$ is the fraction of the variation in the dependent variable explained by the regressors, between 0 and 1. Individual wages depend on much that data do not record, so low values are common in wage regressions and say nothing about whether $\hat\beta$ is right.

Statistical significance is not economic importance: a large sample can make a trivial effect significant.

Correlation is not causation

The error term contains every omitted influence on wages. If one of them is correlated with schooling, $\hat\beta$ picks up its effect too: omitted variable bias. People with more ability tend to get more schooling and would earn more at any level of schooling, so a regression without ability overstates the return to schooling. The same problem appears as selection bias when people choose a treatment, such as a training program, because they differ in ways that also affect the outcome (evaluating training programs).

The fix is a source of variation unrelated to those omitted factors: a randomized experiment, or a natural experiment.

Difference in differences

A natural experiment is a policy change, rule or event that affects a treated group but not a similar comparison group (or control group), for reasons unrelated to the outcome. With outcome means for each group before and after the change,

$$\hat\delta = (\bar y_{T,\text{after}} - \bar y_{T,\text{before}}) - (\bar y_{C,\text{after}} - \bar y_{C,\text{before}}).$$

Each simpler comparison is biased. The treated group’s before-after change mixes the effect with everything else that changed over time, such as the business cycle. The after-only gap between the groups mixes the effect with differences that existed before. The comparison group’s change estimates the common trend, and subtracting it removes both biases.

The estimate rests on one assumption, parallel trends: without the treatment, the treated group’s outcome would have changed by the same amount as the comparison group’s. The assumption cannot be checked after the treatment, but parallel movement in the years before it makes it more credible. It fails when the groups were already diverging, or when the treated group was selected for a temporary shock, as with trainees whose earnings dip just before they enrol.

Build-up: 1 data, 2 counterfactual, 3 estimate

3

Change in New Jersey (treated)

0.59

Change in Pennsylvania (comparison)

−2.16

Difference in differences

2.75

Open in Graph Lab

Figure 2. Average full-time-equivalent employment per fast-food restaurant in New Jersey, whose minimum wage rose in April 1992, and eastern Pennsylvania, whose did not. Had New Jersey followed Pennsylvania's fall of 2.16, it would have ended at 18.28 (dashed); it ended at 21.03, so the difference-in-differences estimate is +2.75 employees per restaurant.

The same estimate comes from the regression $y = a + b\,T + c\,\text{After} + \delta\,(T \times \text{After}) + u$, where $T = 1$ for the treated group and $\text{After} = 1$ in the later period: $\delta$ is the difference in differences, and its standard error says how precise it is.

Key idea

Difference in differences subtracts the comparison group’s change from the treated group’s change. It removes both the common trend and any fixed gap between the groups, and it is valid only if the two groups would have moved in parallel without the treatment.

Worked example

On April 1, 1992, New Jersey raised its minimum wage from $4.25 to $5.05 an hour while neighbouring Pennsylvania did not. David Card and Alan Krueger surveyed more than 400 fast-food restaurants in both states before and after the increase.¹ Full-time-equivalent (FTE) employment counts full-time workers and managers plus half of part-time workers.

Average FTE employment per restaurant	Before (Feb.–Mar. 1992)	After (Nov.–Dec. 1992)	Change
New Jersey (treated)	20.44	21.03	+0.59
Pennsylvania (comparison)	23.33	21.17	−2.16
New Jersey minus Pennsylvania	−2.89	−0.14	+2.75

1. First difference within each state. New Jersey: $21.03 - 20.44 = 0.59$. Pennsylvania: $21.17 - 23.33 = -2.16$.

2. Second difference. $\hat\delta = 0.59 - (-2.16) = 2.75$ FTE employees per restaurant. The same number comes from differencing the bottom row: $-0.14 - (-2.89) = 2.75$.

3. Compare with the naive estimates. New Jersey’s before-after change, $+0.59$, ignores that employment fell in the comparison state. The after-only gap, $-0.14$, ignores that New Jersey restaurants were smaller to begin with.

4. Interpret. Relative to Pennsylvania, employment in New Jersey rose by 2.75 FTE per restaurant, about 13 percent of its starting level. The competitive model predicts a fall; the result is evidence against a large job loss from this increase, if the two states shared a common trend.

The result was contested. Re-estimating with payroll records from 230 restaurants, David Neumark and William Wascher found a 4.6 percent decline in New Jersey employment relative to Pennsylvania; Card and Krueger replied in the same issue of the journal.² The debate turned on data quality, not on the method: difference in differences is only as good as the measured outcomes and the parallel-trends assumption (minimum wage).

Evidence: natural experiments in this course

Question	Treated group	Comparison group	Finding
Does a daily overtime premium reduce long days? (overtime regulation)	California men, covered by daily overtime pay from 1980	Men in other states; California women, already covered	Daily overtime among California men fell substantially; the price elasticity of demand for overtime hours is at least 0.7 in absolute value3
How do disasters affect local labour markets? (shocks)	Florida counties hit by hurricanes	Counties not hit	Earnings per worker rise by as much as 4 percent in the quarter after a direct hit and move the opposite way in neighbouring counties4
Do hospitals have monopsony power over nurses? (monopsony)	Veterans Affairs hospitals where the Nurse Pay Act of 1990 raised nurses’ wages to local market levels	Other hospitals in the same local markets, and VA hospitals with smaller raises	Labour supply to an individual hospital is inelastic, about 0.1 in the short run5
Is the GED a signal? (signalling)	Low-scoring test-takers in states with lenient passing scores, who receive the GED	Test-takers with the same scores in strict states, who do not	The credential raises the earnings of young white dropouts by 10 to 19 percent6
Does job training pay? (evaluating training programs)	Trainees	Randomized controls, or non-participants	Non-experimental estimates often fail to reproduce experimental ones7

For schooling itself, Canadian census data show the raw pattern that regression starts from.

Median employment income by highest level of education, Canada

Median employment income in 2020, full-year full-time workers aged 25 to 54 (2021 Census)

No certificate, diploma or degree$44,800

High school diploma$54,000

Apprenticeship or trades certificate$60,800

College or CEGEP diploma$61,600

University certificate below bachelor's$63,600

Bachelor's degree or higher$81,000

Median employment income in 2020 of full-year, full-time workers aged 25 to 54 rose from $44,800 with no certificate to $54,000 with a high school diploma and $81,000 with a bachelor's degree or higher.Source: Statistics Canada, Table 98-10-0597-01, Employment income statistics by industry sectors, highest level of education, immigrant status and period of immigration, work activity during the reference year, age and gender (2021 Census of Population). Retrieved 2026-10-23.

Show the data and notes

The Labour Force Survey publishes no annual table of wages by education, so this uses the 2021 Census. Restricting to people who worked 49 or more weeks in 2020, mostly full time (30+ hours a week), holds hours roughly constant, so the gaps approximate differences in pay rates. Employment income = wages, salaries and commissions plus net self-employment income; both genders. Pandemic benefits such as CERB are government transfers and are not included. This table does not split degrees above the bachelor's level.

Category	Median employment income
No certificate, diploma or degree	$44,800
High school diploma	$54,000
Apprenticeship or trades certificate	$60,800
College or CEGEP diploma	$61,600
University certificate below bachelor's	$63,600
Bachelor's degree or higher	$81,000

The log gap between a bachelor’s degree and a high school diploma is $\ln(81{,}000/54{,}000) = 0.405$. Spread over the roughly four extra years of school, that is about 0.10, or 10 percent per year. It is a raw comparison: graduates differ from non-graduates in ability, family background and field, so it is an upper bound on the causal return only if those omitted factors raise both schooling and earnings (returns to schooling).

Common mistakes

Traps on exams

• Reading a log-wage coefficient as dollars. A coefficient of 0.08 on schooling means about 8 percent per year, not 8 cents.
• Treating significance as importance, or $R^2$ as validity. A significant effect can be tiny; a low $R^2$ does not bias the slope; omitted variables do.
• Differencing in the wrong order or dropping the comparison group. Take each group’s change first, then subtract the comparison group’s change from the treated group’s.
• Calling any before-after comparison a natural experiment. Without a comparison group that shares the trend, a before-after change measures the trend plus the effect.

On the exam

For a difference-in-differences question, draw the 2×2 table with a change column and a difference row, compute both differences, and state the parallel-trends assumption in one sentence. For a graph, plot both groups at the two dates and draw the treated group’s counterfactual parallel to the comparison group’s line; the vertical gap at the end is the estimate.

Check yourself

1.

A regression of the log hourly wage on years of schooling gives a schooling coefficient of 0.07 with a standard error of 0.01. What does the coefficient mean?

1One more year of schooling is associated with an hourly wage about 7 percent higher2One more year of schooling is associated with an hourly wage 7 cents higher3One more year of schooling is associated with an hourly wage 0.07 percent higher4Schooling explains 7 percent of the variation in log wages

Check answer

2.

A province raises its minimum wage; a neighbouring province does not. Average employment per restaurant:

	Before	After
Province that raised its minimum wage	18.0	17.2
Neighbouring province	21.0	21.4

What is the difference-in-differences estimate of the effect of the minimum wage on employment per restaurant?

Your answer (employees per restaurant)

Check answer

3.

What assumption makes a difference-in-differences estimate a valid estimate of a causal effect?

1Without the treatment, the treated group's outcome would have changed by the same amount as the comparison group's2The treated and comparison groups had the same outcome level before the treatment3The comparison group's outcome did not change over the period4The treatment was assigned to the group with the worst outcome before the treatment

Check answer

Practise more questions on this topic

Sources

1. Card, D. and Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772 to 793. Means computed from the authors’ public data set, njmin. ↩

2. Neumark, D. and Wascher, W. (2000). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania: Comment. American Economic Review, 90(5), 1362 to 1396; the 4.6 percent estimate is from the working paper version, NBER working paper 5224. Card, D. and Krueger, A. B. (2000). Reply. American Economic Review, 90(5), 1397 to 1420. ↩

3. Hamermesh, D. S. and Trejo, S. J. (2000). The demand for hours of labor: Direct evidence from California. Review of Economics and Statistics, 82(1), 38 to 47. NBER working paper 5973. ↩

4. Belasen, A. R. and Polachek, S. W. (2009). How disasters affect local labor markets: The effects of hurricanes in Florida. Journal of Human Resources, 44(1), 251 to 276. IZA discussion paper 2976. ↩

5. Staiger, D. O., Spetz, J. and Phibbs, C. S. (2010). Is there monopsony in the labor market? Evidence from a natural experiment. Journal of Labor Economics, 28(2), 211 to 236. NBER working paper 7258. ↩

6. Tyler, J. H., Murnane, R. J. and Willett, J. B. (2000). Estimating the labor market signaling value of the GED. Quarterly Journal of Economics, 115(2), 431 to 468. doi:10.1162/003355300554818. ↩

7. LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604 to 620. Ashenfelter, O. (1978). Estimating the effect of training programs on earnings. Review of Economics and Statistics, 60(1), 47 to 57. ↩