Counterfactual impact evaluation of training policies: comparison between alternative control groups

The quality and reliability of net impact evaluation is strongly affected by the choice of the counterfactual group. In the case of randomized sample selection (ex-ante experimental attribution to the treated and control groups) the differences between the two groups are likely to be mainly due to the treatment. On the other hand, when applying quasi-experimental methods (ex-post selection of a control group among the non-treated units) it is very difficult to assess whether the observed differences derive from the treatment itself or rather from the selection method. Different non-experimental control groups face different sources of selection bias (Heckman, Lalonde, and Smith, 1999), but it is a priori not clear what choice would minimize it. Hence, in counterfactual analysis the main source of problems is linked to selection bias, which occurs whenever the decision on the treatment is not independent from observable and non-observable individual characteristics. A second source of problems concerns the availability of information, since not-treated units are not included in monitoring data-bases. In this sense, the availability and integration of administrative data-bases from different sources may open new frontiers to quasi-experimental methods. Nevertheless, in this new perspective two problems arise. Firstly, the technical use of administrative data-bases, designed for accountability purposes different than evaluation analysis, so requiring a long pre-processing . The second one concerns the choice among different counterfactual groups, mainly unexplored. In fact, to our knowledge no study compares different solutions implemented on the same policy. Then, this paper investigates the differences in net impact assessment ascribable to the use of different counterfactual groups. The evaluation exercise is performed on Piedmont training policies. The placement effects of regional training policies will be estimated using both a control sample selected from drop-outs (no-shows), and a control group appropriately extracted from the lists of unemployed individuals. We will run the same model to estimate the impact of the treatment, controlling for individual characteristics and comparing the results, with particular attention to selection bias. Although only a comparison of the different techniques to the experimental benchmark could provide ultimate indications, some useful insights can be gained on the potential sources and directions of distortion due to different counterfactuals.