How do you evaluate initiatives that are trying to address complex social problems? When multiple partners and strategies are involved, how do you determine which efforts were the most effective? At The Improve Group, we’ve been exploring one extremely promising approach that can be used to answer those questions. It’s called outcome harvesting.
Recently we partnered with the Minnesota Evaluation Studies Institute, the Humphrey School of Public Affairs, and our Advisory Board to host a workshop examining the potential benefits of outcome harvesting and how evaluators, program managers, and policy makers can apply it to their work.
After being introduced by Michael Quinn Patton, Ricardo Wilson-Grau described outcome harvesting as a method best used in complex situations, such as when there are multiple contributing partners or interrelated issues. In these situations, when the pathway between effort and result is unclear, outcome harvesting can help to answer the following questions:
- What happened?
- Who did it (or contributed to it)?
- How do we know this? Is there corroborating evidence?
- Why is this important? What do we do with what we found out?
Two aspects of outcome harvesting resonated most for me. First, outcome harvesting can be used even when there aren’t clearly identified measures of success at the beginning of an initiative. For example, we considered how to evaluate a pilot initiative matching a community member with peers as they faced an illness. Initiatives in their infancy, like pilot projects, have broad goals that are likely to change significantly. Outcome harvesting can be used with staff, partners, and possibly even participants to explore initial results, gather any available data to substantiate those outcomes, and help leaders identify key strategies for successful program development.
Second, outcome harvesting gains rigor from its validation process. In the outcome harvesting process, knowledgeable people test identified outcomes and consider whether they are relevant, valid, and supported by evidence. For example, we are thinking through strategies to evaluate a large, statewide initiative designed to change health systems. Significant amounts of data will be available from medical records, claims, and reimbursements. In this case, we can use a multi-step process to harvest outcomes: we can start with insights from those very close to the initiative, then do quantitative analysis to identify key outcomes, and then test those outcomes with experts, partners, and community members. That testing will help ensure that the outcomes are truly valid and meaningful to the State.
I realized during our workshop that this method demands a different mindset from me as an evaluator. In a typical evaluation, we ask some standard questions to determine what data needs to be gathered and what indicators or benchmarks should be used. In the outcome harvesting process, the evaluator takes a completely different approach, working with data and ideas generated by those closest to an initiative—its staff and volunteers.
Outcome harvesting requires a very different division of responsibilities than we typically have when conducting an evaluation. But it can also provide some important benefits to program managers and policymakers who are trying to capture the early outcomes of extremely complex projects. Going forward, our team will continue to explore how we can leverage this approach to help our clients better understand and strengthen their programs.