Identification of Effective Methods to Support Research and Evaluation

Using Administrative Data for Randomized Evaluations
Workforce researchers can benefit from guidance on accessing administrative data to conduct impact studies. This resource details how to meet data compliance requirements and establish data use agreements with agencies involved in the data collection process. It focuses on the following topics: 1) standardized processes for accessing administrative data, 2) the ethical and legal framework surrounding the use of administrative data for randomized evaluations, and 3) common challenges in using administrative data. The authors conclude that using administrative data may be less expensive and logistically easier than collecting new data and may be more accurate than surveys in measuring characteristics that are complex or difficult for subjects to remember.

Six Rules of Thumb for Determining Sample Size and Statistical Power
An evaluation’s ability to detect a meaningful impact of a program is determined by the evaluation’s sample size and statistical power. This resource includes guidelines and diagrams to explain to policymakers and practitioners the relationship between sample size and statistical power to better support researchers conducting random assignment studies. Key findings include the following: 1) A larger sample increases the statistical power of the evaluation. Larger samples are more likely to be representative of the original population and are more likely to capture impacts that would occur in the population; and 2) If the effect size of a program is small, the evaluation needs a larger sample to achieve a given level of power. The effect size of an intervention is the magnitude of the impact of the intervention on a particular outcome of interest.

Studying the Use of Research Evidence: A Review of Methods
What are the most effective measures and methods to assess the Use of Research Evidence (URE)? This resource is organized into three sections to address that question: 1) The first section provides a set of core methodological issues and ideas central to the study of URE and highlights key issues that apply to that use; 2) The second is organized around specific methodological approaches; and 3) The final section identifies several key opportunities for moving the field forward. Key recommendations include developing a set of methodological templates to further support theoretical connections among studies and employing common methods to establish stronger grounds for generalization.

• Last Updated:
• Created:
• Posted by: Justin Venneri
• Posted in: Evaluation and Research Hub