Systems Evaluation

DSSERL is currently collaborating with the Cornell Office for Research on Evaluation (CORE) on a National Science Foundation funded project that aims to develop and test a novel approach to program evaluation and planning that integrates theoretical and methodological thinking and tools from systems science (William Trochim, PI, Jennifer Brown Urban, co-PI). A systems evaluation approach considers the complex factors that are inherent in the larger system in which a program is embedded. In addition, our systems evaluation approach focuses on enhancing the internal evaluation capacity of programs and organizations, particularly those operating in informal STEM (science, technology, engineering, mathematics) education contexts. Our research is focused on assessing whether our approach to systems evaluation is associated with high quality evaluation plans and logic/pathway models and an increase in evaluation capacity for practitioners working in multiple STEM education systems.

A major initiative of this research project is the development of the Systems Evaluation Protocol (SEP) which is a guide to our approach to evaluation planning, implementation, and utilization. The SEP is a useful tool for anyone engaged in program evaluation and planning and can be accessed on the CORE website.

While the Systems Evaluation Protocol can be implemented as a manual process that does not depend on any specific technology platform, it is designed so that it can be enhanced throughout by using a system developed in our current research called the Netway, a Web 2.0 application consistent with second-generation web-based communities and hosted services such as social-networking sites, wikis, and blogs. For more information about the Netway see the CORE website.

Related Publications:

Urban, J.B., Burgermaster, M., Archibald, T., & Byrne, A. (In Press). Relationships between quantitative measures of evaluation plan and program model quality and a qualitative measure of participant perceptions of an evaluation capacity building approach. Journal of Mixed Methods Research. DOI: 10.1177/1558689813516388

Urban, J.B., Hargraves, M., & Trochim, W.M. (2014). Evolutionary evaluation: Implications for evaluators, researchers, practitioners, funders and the evidence-based program mandate. Evaluation and Program Planning, 45, 127-139. http://dx.doi.org/10.1016/j.evalprogplan.2014.03.011

Trochim, W., Urban, J.B., Hargraves, M., Hebbard, C., Buckley, J., Archibald, T., Johnson, M., & Burgermaster, M. (2012). The guide to the systems evaluation protocol. Ithaca, NY: Cornell Digital Print Services.

Urban, J.B. & Trochim, W.M. (2009). The role of evaluation in research-practice integration: Working toward the “golden spike”. American Journal of Evaluation, 30(4), 538-553.