The quality and reliability of results of net impact evaluation is strongly affected by the choice of the control counterfactual group to be used to estimate the effect of the policy. In the case of randomized sample selection (ex-ante experimental attribution to the treated group and control groups) the differences between the two groups are likely to be mainly connected due to the treatment effect. On the other hand, when applying quasi-experimental methods (ex-post selection of a control group of among the non-treated units) it is very difficult to assess whether the observed differences derive from the treatment itself or rather from the selection method or by measurement features. Different non-experimental control groups face different sources of selection bias (Heckman, Lalonde, and Smith, 1999), but it is a priori not clear what choice would minimize biasit. Hence, in counterfactual analysis the main source of problems is linked to selection bias, which occurs whenever the decision on the treatment (i.e. to be in the main group or in the control one) is not independent from observable and non-observable individual variablescharacteristics. A second source of problems is linked to concerns the availability of information, since not-treated units are not included in monitoring data-bases. In this sense, the availability and integration of administrative data-bases from different sources may open new frontiers to quasi-experimental methods, since they offer wider data-sets of micro-data, where each unit is described by multiple variables, and it is then possible to select a control group of units which are very similar to the treated ones. Nevertheless, in this new perspective two new problems arise. The first one is the technical employability use of administrative data-bases, which are born for accountability purposes different than evaluation analysis, and for this reason require requiring a long pre-processing whose length and result is difficult to assess previously. Also the reliability of these source is difficult to assess, due to the absence of any alternative source.The second one concerns the choice between among different types of control counterfactual groups, mainly unexplored. which is not easy becauseIn fact, up to now we have found to our knowledge no study compares in which different solutions have been implemented on the same policy and the results systematically compared. Then, In this paper we propose then to apply investigates the differences in net impact assessment ascribable to the use of counterfactual groups. extracted from different sources. The evaluation exercise is performed on Piedmont training policyies. The placement effects of regional training policies has been estimated using a control group sample selected fromdrop-outs (no-shows), individuals enrolling for a training course, and so perfectly eligible for treatment) , and the employment status has been analysed both through a survey and through information extracted by the COB (compulsory communications) database. The same model - to estimate the impact of individual characteristics and of the treatment, controlling for individual characteristics and comparing the results, with particular attention to the problem of selection bias - has been run and some useful insights on the potential sources and directions of distortion of the due to different csources arise.
Counterfactual impact evaluation of training policies: Comparison between control groups from alternative sources
2014
Abstract
The quality and reliability of results of net impact evaluation is strongly affected by the choice of the control counterfactual group to be used to estimate the effect of the policy. In the case of randomized sample selection (ex-ante experimental attribution to the treated group and control groups) the differences between the two groups are likely to be mainly connected due to the treatment effect. On the other hand, when applying quasi-experimental methods (ex-post selection of a control group of among the non-treated units) it is very difficult to assess whether the observed differences derive from the treatment itself or rather from the selection method or by measurement features. Different non-experimental control groups face different sources of selection bias (Heckman, Lalonde, and Smith, 1999), but it is a priori not clear what choice would minimize biasit. Hence, in counterfactual analysis the main source of problems is linked to selection bias, which occurs whenever the decision on the treatment (i.e. to be in the main group or in the control one) is not independent from observable and non-observable individual variablescharacteristics. A second source of problems is linked to concerns the availability of information, since not-treated units are not included in monitoring data-bases. In this sense, the availability and integration of administrative data-bases from different sources may open new frontiers to quasi-experimental methods, since they offer wider data-sets of micro-data, where each unit is described by multiple variables, and it is then possible to select a control group of units which are very similar to the treated ones. Nevertheless, in this new perspective two new problems arise. The first one is the technical employability use of administrative data-bases, which are born for accountability purposes different than evaluation analysis, and for this reason require requiring a long pre-processing whose length and result is difficult to assess previously. Also the reliability of these source is difficult to assess, due to the absence of any alternative source.The second one concerns the choice between among different types of control counterfactual groups, mainly unexplored. which is not easy becauseIn fact, up to now we have found to our knowledge no study compares in which different solutions have been implemented on the same policy and the results systematically compared. Then, In this paper we propose then to apply investigates the differences in net impact assessment ascribable to the use of counterfactual groups. extracted from different sources. The evaluation exercise is performed on Piedmont training policyies. The placement effects of regional training policies has been estimated using a control group sample selected fromdrop-outs (no-shows), individuals enrolling for a training course, and so perfectly eligible for treatment) , and the employment status has been analysed both through a survey and through information extracted by the COB (compulsory communications) database. The same model - to estimate the impact of individual characteristics and of the treatment, controlling for individual characteristics and comparing the results, with particular attention to the problem of selection bias - has been run and some useful insights on the potential sources and directions of distortion of the due to different csources arise.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
