- Funder: National Science Foundation
- Collaborators: American Institutes for Research
Informal STEM education needs high-quality program evaluation. Research is particularly needed on the relationship between STEM outcomes and positive youth development (PYD)/socio-emotional learning (SEL). This pilot and feasibility project involved a collaboration between experts in STEM education, out-of-school time programs (OST), PYD, SEL, evaluation, and program development. STEM Scouts helps youth grow in character and develop skills using experiential activities and interaction with STEM professionals. This project expanded the implementation of the Systems Evaluation Protocol (SEP), an evidence-based approach to evaluation planning. The goals were to: 1) develop a theory of change for STEM Scouts highlighting the relationship between STEM outcomes and PYD/SEL outcomes; 2) pilot enhancements to the SEP (System Mapping, Ecosystem Modeling, and Model Validation); and, 3) determine the feasibility of conducting a national STEM Scouts study. STEM Scouts leadership and project researchers worked through the SEP to generate a stakeholder map, logic model, and pathway model (PM). Five STEM Scouts Labs across the country participated in focus groups where they completed ecosystem maps to identify the system in which the Lab exists (e.g., stakeholders and decision-makers), reviewed and revised their system maps, identified key outcomes and connections in the PM, and discussed how the PM reflects their STEM Scouts experiences.
- Funder: S.D. Bechtel, Jr. Foundation
- Collaborators: American Institutes for Research, Boy Scouts of America
The goal of the Boy Scouts of America Building Evidence in Scouting Together (BSA BEST) study was to understand how adult experiences and training impact youth character outcomes. The overarching research questions guiding this national-scale study were:
- What combination(s) of training and experience lead to the most effective (in terms of instilling character outcomes in youth) adult volunteers in Scouts BSA?
- How can BSA’s adult trainings and systems be enhanced to strengthen adult practice in support of youth character development?
- What variations exist relative to adult training implementation in Scouts BSA?
Diversity, equity, and inclusion related research questions were:
- How are adult leaders addressing diversity and inclusion in their work with youth, and through such efforts, potentially enhancing youth character development?
- What are youth experiences related to diversity and inclusion in BSA?
- How does diversity come up, if at all, in Wood Badge and Scoutmaster Position Specific adult training?
The third wave of data collection was conducted just when the COVID-19 pandemic hit the United States (March/April 2020). Therefore, we were able to add additional research questions related to youth mental health during the pandemic. The research questions were:
- Are youth with greater sense of purpose less likely to experience anxiety and depressive symptoms during COVID-19?
- In what ways do youths’ descriptions of their own prosocial behavior align with the relation between purpose and mental health?
The BSA BEST Study consists of four components with an equal emphasis on each: 1) quantitative surveys, 2) qualitative interviews, 3) observations, and 4) data use. We administered surveys to Scoutmasters, Assistant Scoutmasters, and Scouts across the country at 3 time points. A total of 9667 youth and 4362 adults in 1339 troops participated in at least one wave of survey data collection. We also conducted telephone interviews at 3 time points with about 100 adult leaders and 100 Scouts across the country. We observed Wood Badge trainings in 9 states, including 17 site visits where we were able to observe 326 training participants and 168 trainers. We also observed 4 Scoutmaster Position Specific trainings.
Perhaps most importantly, we focused on data use from the outset. Our primary goal has been to provide research results that are meaningful, actionable, and useful to our partners at BSA. Over the entire course of the study, a data use team met which consisted of representatives from BSA, American Institutes for Research, and Montclair State University. These meetings were punctuated by two data use retreats. The first retreat focused on findings related to training and occurred in-person in Fall 2019. The second set of retreats focused on findings related to diversity, equity, and inclusion and occurred virtually in Summer 2020.
The PACE Project is an innovative approach to evaluation capacity building that offers unique professional development opportunities for staff of youth character development programs as well as evaluators.
For program staff, PACE is designed to immediately increase the capacity to evaluate, improve, defend and seek funding for programs. For current and future evaluation professionals, PACE offers a unique opportunity to develop expertise in evaluation capacity building and innovative evaluation techniques including systems evaluation and person-centered analyses.
The PACE approach is innovative in that it seeks to move beyond evaluation skill-building activities for program staff to also address the more foundational conditions of curiosity, motivation, and careful thinking that drive evaluation work forward. In addition, this Project is organized around a true partnership between evaluators and program staff, in which ownership of the evaluation process is shared, and both sets of professionals bring expertise to the project. Our hope is that this approach will result in sustained relationships between evaluators and youth character development program staff that will remain in place long after the Project concludes, and produce high quality youth character development program and evaluation work for years to come. For more information about the Project, visit the Detailed Project Description.
The PACE Project will support
- 16 youth character development programs, each with a $2,500 award (plus an additional $2,000 for those who continue on to evaluation planning), plus travel expenses
- 8 current or future evaluation professionals, each with a $6,000 award (plus an additional $2,500 per evaluation plan for those who continue on to evaluation planning with 1 or 2 partner programs), plus travel expenses
As a result of participating in the Project, programs and their staff will have:
- Enhanced evaluation capacity
- A well-developed theory of change
- A Program and Evaluation Profile (PEP) that can be used to communicate about the program and its evaluation with stakeholders
- Customized “talking points” that allow staff from participating programs to promote themselves as character/youth development programs and advocate for their approach to evaluation
- An Evaluative Thinking site plan, designed to incrementally improve the organizational culture and communication around evaluation
- An established relationship with an evaluation professional
- Access to a network of participating programs and evaluators
Those programs that choose to develop a complete evaluation plan and implement that plan in partnership with their partner evaluator will also have a formal and detailed evaluation plan.
Evaluators will have:
- Increased knowledge and understanding of Evolutionary Evaluation, the Systems Evaluation Protocol, Evaluative Thinking, Person-Centered Analysis, and Character Virtue Development programming
- Experience facilitating Evolutionary Evaluation, the Systems Evaluation Protocol, and Evaluative Thinking activities in an evaluation capacity building context
- Experience partnering with 2 programs to develop: theories of change and Program and Evaluation Profiles (PEPs)
- Experience with evaluation planning in partnership with up to 2 youth character programs (if mutually desired)
- A PACE Project credential from Montclair State University
The PACE Project is organized around a partnership between participating programs and evaluators that includes (for each cohort of programs):
- 2 workshops
- Several webinars
- Quarterly phone meetings
- A culminating conference
- Ongoing communication with partner program(s)/evaluator and facilitators