Evaluating government training programs

The effect of a training program is the difference between what participants earn after it and what the same people would have earned without it. The second number is never observed, and the obvious substitutes for it are biased: participants differ from non-participants (selection bias), and their earnings fall just before they enrol and then recover on their own (the Ashenfelter dip). Random assignment removes both biases for the people who apply; a difference-in-differences against a comparison group that shares the dip can come close.

Why governments fund training

Human capital theory predicts that training raises productivity and earnings. Governments pay for some of it because:

• Firms underinvest in general training, because skills other employers value leave when the worker quits.
• The workers who need training most can least afford it: a laid-off worker cannot easily pay tuition or live on a trainee’s wage.
• Taxpayers share the return: a worker who finds a job draws less Employment Insurance and social assistance and pays more tax.

In Canada, the Labour Market Development Agreements transfer Employment Insurance funds to the provinces for training and employment programs: $2.14 billion a year at the time of the federal government’s 2017 evaluation of Alberta’s programs.¹ Alberta’s training and employment services, for people who are unemployed, underemployed or on Employment Insurance, include occupational training with work experience, employer-driven worksite training and bridging programs for skilled immigrants. The federally funded Canada-Alberta Productivity Grant pays 75 percent of an employer’s eligible training costs, up to $5,000 per employed trainee a year; the employer pays the rest.²

The evaluation problem

Call participation the treatment. Participants form the treatment group; the people they are compared with form the control group (or comparison group). For a participant,

$$\text{effect} = w_{\text{with training}} - w_{\text{without training}}.$$

Only the first term is observed. The second, the counterfactual, must be estimated from other people’s earnings, and every evaluation method is a different way of estimating it. The two simplest are both biased:

1. Before and after: participants’ earnings after the program minus their own earnings before. This credits the program with everything else that changed, such as the business cycle and general wage growth.
2. Participants versus non-participants: participants’ earnings after the program minus non-participants’ earnings. This credits the program with every difference between the groups.

The second problem is selection bias: people choose whether to enrol, and the choice is related to their earnings prospects. The bias can go either way. Programs target workers with weak prospects, who would earn less than non-participants with or without training, which makes training look harmful. Applicants may also be the most motivated people in their situation, which makes training look better than it is.

Key idea

A training effect is a comparison with a counterfactual: what the participants themselves would have earned without the program. An estimate is only as good as the comparison group that stands in for that counterfactual.

The Ashenfelter dip

The Ashenfelter dip is the fall in trainees’ earnings in the period just before they enter a program. Orley Ashenfelter documented it for workers who entered US federal training under the Manpower Development and Training Act in 1964, and it has appeared in program after program since.³⁴

The dip follows from how people reach training: programs admit people with low recent earnings or recent unemployment, and people seek training after a layoff, an illness or a family crisis. Many such shocks are temporary, so earnings drift back toward their usual level whether or not the person trains: regression to the mean. A before-and-after comparison that starts at the bottom of the dip counts that recovery as a program effect.

Size of the dip before entry

$5.0k

True effect of training

$2.0k

True effect

$2.0k

Before-after, year −1 to +2

$8.5k

Difference-in-differences from year −1

$7.0k

Difference-in-differences from year −4

$2.0k

After-only gap in year +2

−$10.0k

Open in Graph Lab

Figure 1. Participants earn less than the comparison group in every year and dip in the year before entering the program. Only a difference-in-differences measured from a year before the dip recovers the true effect.

With a dip of $5,000 and a true effect of $2,000, the four estimates in Figure 1 are:

• Before-after, year −1 to year +2: $8,500, which is $1,500 of normal wage growth, $5,000 of recovery from the dip and $2,000 of training effect.
• Difference-in-differences from year −1: $7,000. Subtracting the comparison group’s $1,500 of growth removes the common trend but not the dip, because the comparison group had no dip.
• Difference-in-differences from year −4, before the dip: $2,000, the true effect.
• After-only, year +2: −$10,000, the permanent gap between the groups plus the effect.

The last difference-in-differences works only because, apart from a constant gap, the dip is the sole difference between the groups. Real data are rarely that clean. For trainees in 1976 US federal programs, Ashenfelter and David Card obtained estimates from about $200 to $2,000 depending on how they modelled who enrols, and concluded that randomized trials were needed to settle the question.⁴

Randomized experiments

In a randomized experiment, eligible applicants are assigned by lottery to a treatment group that receives training or a control group that does not. The groups are then alike on average in everything, observed or not, including motivation and the size of their dip, so the difference in mean earnings after the program is an unbiased estimate of the average effect.

Experiments have four limits:

• External validity. Both groups come from applicants, so the estimate is the effect on people who want the program and expect to gain from it. It says little about non-applicants or about making participation compulsory.
• Substitution and dropout. Controls can find similar training elsewhere and some of the treatment group quit, so the experiment measures the effect of being offered the program relative to the alternatives available, not training versus no training.⁵
• Averages only. Without further assumptions an experiment identifies the mean effect but not how the gains are distributed across participants.⁶
• Cost, delay and fairness. Experiments are expensive, take years and withhold help from the control group.

The National Supported Work Demonstration

The National Supported Work Demonstration (NSW) was a temporary employment program run by the Manpower Demonstration Research Corporation at ten US sites in the mid-1970s. It enrolled women on welfare (Aid to Families with Dependent Children, AFDC), former drug addicts, former offenders and young high-school dropouts. Qualified applicants were randomly assigned. The treatment group was guaranteed a job for 9 to 18 months, working in crews of three to five that met regularly with a counsellor, at wages below those of regular jobs; afterwards they had to find regular employment. The control group received nothing from the program. For AFDC women the program cost about $9,100 per participant in 1982 dollars.⁷

Treatments' gain, 1975 to 1979

$3,775

Controls' gain, 1975 to 1979

$2,942

Difference-in-differences

$833

Simple 1979 difference

$851

Open in Graph Lab

Figure 2. Mean annual earnings of AFDC women randomly assigned to the NSW treatment and control groups, in 1982 US dollars. Both groups' earnings rose sharply from 1975 to 1979; the treatment group's rose $833 more, the difference-in-differences estimate. The 1977 peak is pay from the supported-work jobs.

The control group’s earnings rose from $877 to $3,819 with no program at all. A before-and-after comparison for the treatment group would have credited the program with $3,775, more than four times the true effect. The difference-in-differences subtracts the control group’s change from the treatment group’s change:

$$\text{DiD} = (\$4{,}670 - \$895) - (\$3{,}819 - \$877) = \$3{,}775 - \$2{,}942 = \$833 .$$

For the men, the same calculation over 1975 to 1978 gives $(5{,}976 - 3{,}066) - (5{,}090 - 3{,}027) = 847$ dollars, close to their $886 difference in 1978 earnings.⁷ Random assignment made the groups’ 1975 earnings nearly identical, so the post-program difference and the difference-in-differences agree; the second also removes any chance difference at the start. Expressed in 1998 dollars, as in some textbook tables, the AFDC estimate is about $1,407: US consumer prices rose by a factor of about 1.69 between 1982 and 1998, and $833 \times 1.69 \approx 1{,}407$.

Rate of return. If the $833 gain is permanent, the $9,100 cost earns $833 / 9{,}100 \approx 9$ percent a year. Most of that cost is wages paid to participants; counted as a transfer rather than a social cost, they drop out and the cost per AFDC participant falls to $3,100.⁷

The lesson for non-experimental methods. Robert LaLonde set the NSW control group aside and asked whether standard econometric methods, applied to comparison groups drawn from national surveys, reproduced the experimental answer. For the men, a survey sample of male household heads gave a post-program difference of −$15,578 and a difference-in-differences of +$425, against the experiment’s +$886; across comparison groups and methods, estimates ranged from large losses to gains above $1,700.⁷

Worked example

A province randomly assigns applicants to a welding course. Mean annual earnings:

	Year before applying	Second year after the course
Treatment group	$14,000	$22,500
Control group	$14,200	$20,700
Non-applicants of the same age	$30,000	$31,500

1. Before-after for the treatment group: $22{,}500 - 14{,}000 = 8{,}500$ dollars. This includes the recovery from the dip that brought these people to the program.

2. Change for the control group: $20{,}700 - 14{,}200 = 6{,}500$ dollars. The controls had the same dip and recovered without the course.

3. Difference-in-differences: $8{,}500 - 6{,}500 = 2{,}000$ dollars a year, the estimated effect of the course.

4. After-only comparison with controls: $22{,}500 - 20{,}700 = 1{,}800$ dollars. Under random assignment this is also unbiased; it differs from $2,000 only because the groups started $200 apart by chance.

5. The wrong comparison group: with non-applicants, $8{,}500 - (31{,}500 - 30{,}000) = 7{,}000$ dollars. Non-applicants had no dip, so the estimate is inflated by $5,000.

Exercise: a smaller recovery

The control group’s earnings rise from $14,200 to $16,700 and the treatment group’s from $14,000 to $19,000. The difference-in-differences is $(19{,}000 - 14{,}000) - (16{,}700 - 14{,}200) = 5{,}000 - 2{,}500 = 2{,}500$ dollars. Before-after reports $5,000, double the effect.

Evidence from Canadian programs

Canadian programs are usually evaluated without experiments. Employment and Social Development Canada’s 2017 evaluation of Alberta’s Employment Insurance-funded programs matched participants with non-participants on up to 75 characteristics measured over the five years before participation, then compared the two groups’ earnings changes: a difference-in-differences on a matched comparison group. For EI claimants who started classroom training (Skills Development) between 2002 and 2005, the earnings gain rose from $887 in the first year after training to $5,680 in the fifth, about $17,000 over five years, and the probability of employment was 8 to 11 percentage points higher. The average cost per participant was $5,081. The evaluators note that matching on observed characteristics cannot rule out differences the data do not capture.¹

Slowly building gains are typical: a meta-analysis of more than 200 evaluations of active labour market programs found average effects near zero in the short run that become more positive two to three years after a program ends, and larger for programs that build human capital.⁸ A one-year evaluation understates the return to training.

Common mistakes

Traps on exams

• Reading a before-after rise as the program effect. Participants’ earnings recover from the dip anyway. The comparison group must share the dip.
• Thinking an experiment randomizes the whole population. It randomizes applicants, so the estimate applies to applicants: the external validity limit.
• Mis-ordering the difference-in-differences. Take each group’s change first, then subtract: $(T_{\text{after}} - T_{\text{before}}) - (C_{\text{after}} - C_{\text{before}})$.
• Assuming selection bias always flatters the program. When programs enrol people with poor prospects, a naive comparison makes a useful program look harmful.

On the exam

Define the Ashenfelter dip in three parts: participants’ earnings fall just before they enter a program; the fall is largely temporary, so earnings recover anyway (regression to the mean); comparisons that start from the dip credit the program with that recovery. Sketch participants below the comparison group, with the dip marked in the year before entry.

Check yourself

1.

A program randomly assigns half of its eligible applicants to training and half to a control group. What does random assignment achieve?

1The two groups are alike on average in motivation, past earnings and other traits, so their later earnings difference is an unbiased estimate of the effect on applicants2The estimate applies to every worker in the province, including people who would never apply3Control group members cannot obtain similar training anywhere else4The estimate shows how the gains from training are distributed among participants

Check answer

2.

In the National Supported Work Demonstration, AFDC women in the treatment group earned $895 in 1975 and $4,670 in 1979; the control group earned $877 and $3,819 (1982 dollars). What is the difference-in-differences estimate of the program's effect on annual earnings, in dollars?

Your answer (dollars a year)

Check answer

3.

A randomized experiment finds that a voluntary training program raised applicants' earnings by $2,000 a year. A minister proposes making the program compulsory for every unemployed worker and budgets for the same $2,000 gain per person. What is the main problem?

1The estimate applies to applicants, who wanted the training and expected to gain from it; the effect on people who would not have applied may be different2The estimate is biased by selection, because applicants chose to apply3Experiments always overstate effects because of the Ashenfelter dip4The control group's earnings were affected by the program

Check answer

Practise more questions on this topic

Sources

1. Employment and Social Development Canada (2017). Evaluation of the Canada-Alberta Labour Market Development Agreement: Synthesis report. Skills Development results are for participants who were active EI claimants when they started, 2002 to 2005, in current dollars. ↩ ↩²

2. Government of Alberta, Training and employment services and Canada-Alberta Productivity Grant, retrieved October 2026. The 75 percent share applies since October 8, 2025. ↩

3. Ashenfelter, O. (1978). Estimating the effect of training programs on earnings. Review of Economics and Statistics, 60(1), 47 to 57. ↩

4. Ashenfelter, O. and Card, D. (1985). Using the longitudinal structure of earnings to estimate the effect of training programs. Review of Economics and Statistics, 67(4), 648 to 660. The paper reviews the pre-program dip across several programs and studies trainees in the 1976 Comprehensive Employment and Training Act programs. ↩ ↩²

5. Heckman, J., Hohmann, N., Smith, J. and Khoo, M. (2000). Substitution and dropout bias in social experiments: A study of an influential social experiment. Quarterly Journal of Economics, 115(2), 651 to 694. ↩

6. Heckman, J. J. and Smith, J. A. (1995). Assessing the case for social experiments. Journal of Economic Perspectives, 9(2), 85 to 110. ↩

7. LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604 to 620. Program description and costs from section I; earnings from tables 2 and 3 (1982 dollars); comparison-group estimates from table 5. ↩ ↩² ↩³ ↩⁴

8. Card, D., Kluve, J. and Weber, A. (2018). What works? A meta analysis of recent active labor market program evaluations. Journal of the European Economic Association, 16(3), 894 to 931. NBER working paper 21431. ↩