Collective Evalatuion: Network + Tools
Networked evaluation is never quite as simple as the first two paths indicate; combinations of paths are common! For example, some collaboration might pre-exist across a network, and subsequent use of a common tool simultaneously accomplishes networked evaluation and increased connection and collaborative activity across the network.
Example: Oregon Outdoor Schools
Oregon Outdoor School (ODS) is located in the United States, and is a Statewide outdoor formal educational program, primarily for 5th and 6th grade students. Outdoor School is a hands-on week of experiential science education in the field, with a more than 60-year history. Originally, the reach of the program was not consistent throughout the State, but nonetheless, the originally active sites have a long history of familiarity and collaboration, so the network and sense of community already existed to a large degree. At this time, some networked evaluation existed in the program, including a formal summative evaluation, a logic model style evaluation, and an overall comprehensive analysis of any evaluation activities over a four-year period. These activities appear to have been primarily conducted to help programs clarify their goals, and seldom to have resulted in evidence-based programming updates.
New funding from the State legislature allowed expansion Statewide in the 2017-2018 academic year, and by the end of the 2018-2019 school year 84% of eligible 5th or 6th grade students had attended. With this expansion, evaluation needed to be established that would use a common measurement system, and that would enable programs to track their outcomes, looking at change over time within their individual programs, among various programs, and overall statewide trends. Evaluation started with a small project strategically funded, adapting and using the EE21 outcome measure (a survey that assesses a broad range of EE outcomes: Environmental Education for the 21st Century, EE21). Upon completion of the pilot study, the evaluative process/system was reconsidered, making efforts to center cultural responsiveness and to increase stakeholder participation. Methods and thinking were updated to be more culturally responsive, which included modifying our surveys and administration scripts, focusing our data analysis and engaging in Equity, Diversity and Inclusion training. The evaluation was implemented statewide at 39 outdoor schools in 2020 (Braun 2020).
All outdoor school programs receive private and anonymized final reports with data for their specific program in relation to the statewide sample. Subsequently, outdoor school providers are encouraged to attend an evidence-based workshop where they are supported in understanding evaluation results for their specific program, for all ODS programs statewide, and in the context of the field of EE. Each program then translates these learnings into discrete action steps to improve their respective programs, typically related to culturally responsive instruction and pedagogy.
One result of the evaluation activities was that analysis of findings at the statewide scale revealed assets and inequities related to access and ability, gender and racial identity. These findings strongly informed the nature of professional development opportunities stakeholders engaged with, or provided. Equity specific analyses of findings from the outcome-based evaluation led to expansion of evaluation in two particular areas. Process level evaluation tools and trainings were developed, and targeted inquiry/trainings were created - both of which centered on equity.
Extending to collective evaluation
The Oregon Outdoor School has a number of initiatives that extend their evaluative learning to the field of environmental education.
- Suite of self-evaluation tools and learning resources available free online
- Instructional Resources Self-evaluation Tool (Backe, Braun and O’Neil, 2021) and learning resources
- Culturally Responsive Self-Evaluation Tool (Brooks, Braun, Backe and Jones, 2020) and learning resources
- Special Education and Accessibility Self-Evaluation Tool (Brooks, Braun, Backe, Arbuckle and Jones, 2021) and learning resources
- Dr. Spirit Brooks from Oregon State University and Dr. Leilani Sabzalian from the University of Oregon created a free online class on Critical Orientations: Indigenous Perspectives in Outdoor Education. The course is designed to surface some of the common assumptions and misinformation about Indigenous people and cultures, particularly as they relate to outdoor education. While the course centers the 5th and 6th grade Outdoor School Program in Oregon, it is applicable to all levels of outdoor education and can be applied outside of Oregon.