Evaluating STEM Scouts: The Design of a Comprehensive Evaluation Plan and Feasibility Study
Funder: National Science Foundation
Collaborators: American Institutes for Research
Informal STEM education needs high quality program evaluation. Research is particularly needed on the relationship between STEM outcomes and positive youth development (PYD)/socio-emotional learning (SEL). This pilot and feasibility project involved a collaboration between experts in STEM education, out-of-school time programs (OST), PYD, SEL, evaluation, and program development. STEM Scouts helps youth grow in character and develop skills using experiential activities and interaction with STEM professionals. This project expanded the implementation of the Systems Evaluation Protocol (SEP), an evidence-based approach to evaluation planning. The goals were to: 1) develop a theory of change for STEM Scouts highlighting the relationship between STEM outcomes and PYD/SEL outcomes; 2) pilot enhancements to the SEP (System Mapping, Ecosystem Modeling, and Model Validation); and, 3) determine the feasibility of conducting a national STEM Scouts study. STEM Scouts leadership and project researchers worked through the SEP to generate a stakeholder map, logic model, and pathway model (PM). Five STEM Scouts Labs across the country participated in focus groups where they completed ecosystem maps to identify the system in which the Lab exists (e.g., stakeholders and decision-makers), reviewed and revised their system maps, identified key outcomes and connections in the PM, and discussed how the PM reflects their STEM Scouts experiences.
Boys Scouts of America National Character Initiative Phase 2 – Opening Up the Black Box: Uncovering the Role of Adults in Youth Character Development
Funder: S.D. Bechtel, Jr. Foundation
Collaborators: American Institutes for Research, Boy Scouts of America
The goal of the Boy Scouts of America Building Evidence in Scouting Together (BSA BEST) study was to understand how adult experiences and training impact youth character outcomes. The overarching research questions guiding this national-scale study were:
- What combination(s) of training and experience lead to the most effective (in terms of instilling character outcomes in youth) adult volunteers in Scouts BSA?
- How can BSA’s adult trainings and systems be enhanced to strengthen adult practice in support of youth character development?
- What variations exist relative to adult training implementation in Scouts BSA?
Diversity, equity, and inclusion related research questions were:
- How are adult leaders addressing diversity and inclusion in their work with youth, and through such efforts, potentially enhancing youth character development?
- What are youth experiences related to diversity and inclusion in BSA?
- How does diversity come up, if at all, in Wood Badge and Scoutmaster Position Specific adult training?
The third wave of data collection was conducted just when the COVID-19 pandemic hit the United States (March/April 2020). Therefore, we were able to add additional research questions related to youth mental health during the pandemic. The research questions were:
- Are youth with greater sense of purpose less likely to experience anxiety and depressive symptoms during COVID-19?
- In what ways do youths’ descriptions of their own prosocial behavior align with the relation between purpose and mental health?
The BSA BEST Study consists of four components with an equal emphasis on each: 1) quantitative surveys, 2) qualitative interviews, 3) observations, and 4) data use. We administered surveys to Scoutmasters, Assistant Scoutmasters, and Scouts across the country at 3 time points. A total of 9667 youth and 4362 adults in 1339 troops participated in at least one wave of survey data collection. We also conducted telephone interviews at 3 time points with about 100 adult leaders and 100 Scouts across the country. We observed Wood Badge trainings in 9 states, including 17 site visits where we were able to observe 326 training participants and 168 trainers. We also observed 4 Scoutmaster Position Specific trainings.
Perhaps most importantly, we focused on data use from the outset. Our primary goal has been to provide research results that are meaningful, actionable, and useful to our partners at BSA. Over the entire course of the study, a data use team met which consisted of representatives from BSA, American Institutes for Research, and Montclair State University. These meetings were punctuated by two data use retreats. The first retreat focused on findings related to training and occurred in-person in Fall 2019. The second set of retreats focused on findings related to diversity, equity, and inclusion and occurred virtually in Summer 2020.