Instructional Resource Self-Evaluation Tool

Background

This tool is part of a suite of self-evaluation tools developed by the Oregon State University Extension Service Outdoor Schools program to evaluate 5th- and 6th-grade outdoor school programs in the state of Oregon. While the tools are designed for outdoor education programs as part of public schools (or formal education), the creators of the tools hope they can be applicable to other formal, non-formal or informal contexts. The three self-evaluation tools include the following: 

• Instructional Resource Self-Evaluation Tool (IRSET) 
• Cultural Responsiveness Self-Evaluation Tool (CRSET)
• Special Education and Accessibility Self-Evaluation Tool (SEASET)

Format

Self-evaluation rubrics. The tool is presented as a set of three rubrics, one for each of three themes: 

1. Context and Settings for Learning
2. Content and Integration
3. Instruction and Pedagogy

Each rubric has between four and seven characteristics and definitions that allow the evaluating stakeholders to qualify each characteristic as absent, emerging, and/or highly effective. An additional column provides space for you to record evidence or specific examples for the qualification(s) that you chose. A Curriculum Report and Improvement Plan template is provided to summarize results across all three rubrics, and to seed team reflection and discussion for improvement and change.

For example, under the content and integration theme, one characteristic is whether the content is interdisciplinary. The text suggests that “absent” be chosen if the content is “unidimensional and pertains to only one discipline,” “emerging” if the content “addresses multiple disciplines, but does not integrate them…,” and “highly effective” if the content “examines issues, questions, perspectives, or topics across multiple disciplines in a cohesive manner…[and] draws students’ attention to...the ways they are integrating skills/content from multiple disciplines” (p. 11).

Audience

This tool can be used in any type of program with instructional resources or curriculum to evaluate.

When and how to use the tool

This self-evaluation tool can be used at any time, but ideal timing would be at a point in your programming schedule where you have the ability to give ample time to going through this evaluation and reflection process. For example, you might use this tool in the fall/winter before planning begins for summer programming.

Five steps are suggested:

1. Select an evaluation team. This can include any program staff, community members, participants, and any other stakeholders. 
2. Determine the scope of this self evaluation: what curriculum or instructional resources do you wish to evaluate, and who will be involved? Evaluating the entirety of a curriculum is recommended, rather than a single lesson or subset of it.
3. Individuals on the team read each characteristic and its definition (the far left column of each table). Estimate quality of alignment with that characteristic, circling “absent” “emerging” or “highly effective.” In the final column, provide evidence or example(s) for your choice. Ideally the entire evaluation team reviews each resource to enable all stakeholders to provide an important voice in the decision process. 
4. Gather the evaluation team to discuss the strengths and weaknesses of the resource evaluated. 
5. Aggregate results using the Curriculum Report and Improvement Plan (see the “How to analyze” selection below more detail on this step). 

Repeat steps 1-5 for all instructional tools you wish to evaluate.

How to analyze

Use the “Curriculum Report and Improvement Plan” to review the self-evaluation.

• Summarize how all individual team members rated each characteristic using the blank template “Curriculum Report and Improvement Plan,” which allows you to see which characteristics are absent, emerging, and/or highly effective across the whole curriculum.
• Discuss as a team the trends you see in the qualifications. Is your team agreeing on the quality of various characteristics? Are there differences in how some characteristics are qualified? Let disagreements be an opportunity for discussion and learning! Allow team members the opportunity to discuss why they qualified a characteristic as they did.
• Discuss priorities for improvement; you likely can’t do it all, all at once! You might target characteristics that are consistently absent across resources, or ones for activities, concepts or skills that are central to your program. The tool suggests 4 prompting questions to add in this discussion around assets, challenges, targets, and other considerations.

What to do next

After identifying what characteristics will be targeted for improvement, move forward in making an action plan. Use the “Curriculum Improvement Plan” to identify actions you will take to make changes and improve your program.

How to see if this tool would work with your program

This tool works well if you have existing programs and curriculum that the program wishes to systematically evaluate for improvement. Allowing sufficient time in your program schedule to complete this review so that improvements can be piloted, is key to success.

Tool tips

• The tool was developed for use in the Outdoor Schools Program in Oregon, and some of the characteristics will not apply to other programs or locations. For example, one of the content and integration characteristics asks whether content meets the mandated legislative guidelines of the Oregon Senate Bill that funds the program. Consider if your program needs to meet state standards or legislative guidelines and how you may adapt this characteristic to fit your program.
• Don’t forget that activities without explicit learning objectives, such as games or meals, work towards social-emotional and/or academic benefits, and should be included.
• Do take time to consider each characteristic, even if at first glance it seems less or not relevant. Pay particular attention to those that are frequently absent, but recognize that each resource is likely to have some characteristics that are absent. 
• Evaluate both resources and activities that are written down and those that are unwritten or only partially written. The tool provides recommendations for all these levels of documentation.
• Consider and collaboratively attend to social and power dynamics among the evaluation team, so that all voices are included, welcomed, and heard.
• This is a qualitative tool, and results are inherently subjective. Embrace this opportunity to surface and discuss perspectives, assumptions, and biases among team members.
• More tips on using this tool are available in the extensive tool guide created by Oregon State University Extension service.