What are some important next steps you can take to develop strong program evaluation practices?
Mentoring programs should not be discouraged if they cannot measure their long-term impact right away. There are many ways to build evaluation capacity over time, including partnering with an external evaluator and seeking out technical assistance.
Key concepts we’ll explore in this chapter:
- Tips for evaluation design
- Tips for evaluation implementation
-
tips for designing your evaluation Cast a realistic “net”: Realistically, a program like Super Scholars may not be able to administer pre- and post-program surveys for all 13 of the outcomes identified in the logic model. For now, it has identified a handful of outcome measures that will be realistic to administer. Programs may want to tightly focus their evaluation efforts on the outcomes that they are likely to achieve based on program activities and areas of emphasis. This can save time, energy, and money when it comes to conducting evaluation activities. It is also likely to increase the chances of finding meaningful positive impacts.
Use a comparison group if you can (strive for the most rigorous design you can): Experimental designs may not be realistic for all programs, especially as a starting point, but it is something to work toward.
Use validated instruments: Homemade surveys might seem harmless, but you always run the risk that they don’t actually collect the exact information you want or that youth may find the questions confusing and give wildly varied responses. Emphasize using tools that have been used in other research and tested for validity (they capture what they intend to accurately) and reliability (they produce consistent results again and again with different groups of youth). As mentioned throughout this module, you can find many validated instruments for measuring your program outcomes from the National Mentoring Resource Center.
-
tips for implementing your evaluation Follow up with ALL participants: Part of assessing the value of a program is determining whether it can actually deliver effective and meaningful services to all the young people who it sets out to serve. A truly accurate picture of what a program has achieved requires that evaluations include outcome data from the youth who quit, left, or otherwise could not meet the ideal level of participation.
Allow enough time for the “magic of mentoring” to actually happen: Always allow enough time for targeted youth outcomes to realistically be influenced by program participation. Although shorter-term mentoring models have shown some ability to make an impact on youth, looking for big changes early in a mentoring relationship will usually not be realistic. The result of such premature measurement can be that a program seems like it’s less effective than it may actually be.
Be mindful about how you collect data: Try to administer evaluation instruments consistently. Don’t let some youth take them home while others have only 10 minutes in class to complete them. Other factors to consider include giving youth privacy if they are sharing personal information and assuring respondents that you will not share their answers externally. There are many other things to keep in mind, such as getting consent from parents/guardians for data collection, that are discussed further in the Key Evaluation Considerations section of the Measurement Guidance Toolkit.
-
getting professional help and technical assistance 
Working with an external evaluatorWorking with an evaluator can:
- Reduce the burden on internal evaluation staff members
- Yield more honest and objective feedback
- Add credibility to your results
When preparing to engage an evaluator, it’s important to outline clear roles. Developing a shared memorandum of understanding (MOU) can help formalize the relationship. Check out these tips for developing an MOU with an external evaluator.
Getting Technical Assistance
The National Mentoring Resource Center offers technical assistance to mentoring programs, including assistance with evaluation planning. To get started, you can request no-cost help for your youth-mentoring program online. -
identifying next steps (activity) Congratulations!
This marks the end of the training module content. As a final step in this training, consider your organization’s evaluation capacity, identify concrete next steps you could take to strengthen your evaluation practices and review additional resources.
1. Identify next steps: Using the Identifying Next Steps worksheet, identify next steps your program can take to build (or strengthen) your evaluation practices in the following areas:
- Building a logic model
- Planning a process evaluation
- Measuring relationship quality
- Measuring outcomes
2. Review additional resources: Additional Reading and Resources handout