 |
|
 |
Featured Client: City House - Perspectives on measuring outcomes from an emerging, innovative nonprofit City House, founded in 2000, provides services that have long been thought to be crucially important to the success of other forms of assistance. They help people draw upon their spirituality as a source of strength and hope. They travel with people, listening, offering and asking. They work to open up new perspectives and insights and point out possibilities along the way. Although it may be cited as a factor in the success of programs, spirituality is rarely independently addressed, and very little has been published about the outcomes specific to spirituality. When Tom Allen of City House read about Internal Learning Systems (ILS) in the October 2006 issue of Improve Groove, he thought the approach might be a good fit for both exploring what outcomes might be attributed to spirituality, and for measuring those outcomes with people participating in City House services. City House staff, volunteers and service provider partners spent time with IG Project Manager, Becky Stewart brainstorming what outcomes they had observed in program participants and how they relate to spirituality. The result was a logic model that highlighted the different levels of experience with spirituality. The logic model is in two parts, intended to depict two spirals. The first focuses on internal change – indicators of deepening spiritual experience; the second depicts external indicators and behaviors which may be supported by the internal change. The final evaluation plan contains two tools which draw on some of the values and ideas of the ILS system. One of the most basic similarities is that, as in the ILS system, they both use pictures, so the instruments are not as dependent on participants having higher literacy levels. The tools also meet the complicated demands of City House programming. At some sites, City House volunteers may only meet with participants a limited number of times, so a collage tool was developed to gain insight into the spiritual change experienced by these participants without requiring too much time. The second tool allows for ongoing tracking of change. The process for both tools encourages participants to reflect on what they see in the tools, in this way following the participatory values of the ILS system. After pilot testing, these instruments were revised and continue to be used and tested. In the past, volunteer spiritual companions had often asked about best practices of other volunteers and about how to better understand the populations they were encountering, but often did not understand. This evaluation system is beginning to help answer those questions. Click on each evaluation model for link to full page.
|
|
Timing is everything: Choosing the right time to evaluate your program
by Liz Radel
As evaluators, one of the key things to consider in creating an evaluation plan is deciding when to measure outcomes. To showcase a program’s successes, we measure outcomes at the time when the program has the biggest anticipated impact. For example, we would not measure smoking cessation two days into a four-month long smoking cessation program. While we can often use common sense and experience, it still remains a challenge to determine when a program will have the biggest impact. In his 1990 book, Design Sensitivity, Mark Lipsey outlines four models that are helpful in understanding how a program’s effect, or impact, may change over time: 1) Immediate effect, no decay; 2) Delayed effect; 3) Immediate effect, rapid decay and 4) Early effect, slow decay. To visualize the first model, imagine that a group of smokers were invited to a seminar aimed to convince them to stop smoking. Attendees were given $10,000 to quit smoking; to keep the money, they could never smoke again. After the seminar, every participant quit smoking and never touched a cigarette again. In this example, participants quit smoking (desired response), immediately after the seminar (immediate effect) and never smoked again for the rest of their lives (no decay in the desired response). This example is far-fetched because there are few programs that function this way in real life. Many programs have an effect that will decay over time, meaning that the desired response does not last forever. In the second model, Delayed Effect, the desired response is achieved long after the program ends. For example, if the desired response is smoking cessation, participants would most frequently stop smoking months after the program ends. In third the model, Immediate Effect, Rapid Decay, the desired response peaks as the program ends and quickly fades over time. Again using smoking cessation as an example, most participants would stop smoking at the end of the program, but gradually participants would begin smoking again in the months after the program. The final model that Lipsey presents is Early Effect, Slow Decay, in which the desired response occurs soon after the program begins and is sustained over time. This model is the ideal situation; using our example, most participants would stop smoking soon after the program began and remain non-smokers over time. The key message for organizations is that program exposure and program impact are rarely a one-to-one relationship. Some programs have a high impact early on, while others have the biggest impact long after the program has concluded. While there is no “magic answer” to determine when a program would have the greatest impact, organizations can reflect on their knowledge of the program and expertise in the field to help decide the best timing to showcase the program’s successes. Evaluators can supply additional information to help an organization decide such as best practices in evaluation and experiences with other, similar programs. Together, organizations and evaluators can combine their knowledge to best determine when to evaluate a program. |
|
| Grants and Upcoming Requests for Proposals by Susan Murphy Many groups would like to conduct research, analyze data, assess outcomes or share best practices about their programs and initiatives but do not have funds for it in their budget. The Improve Group often looks for funding opportunities to partner with organizations on program research and evaluation. The following federal opportunity in educational evaluation has a good lead time for planning and proposal preparation. Evaluation of State and Local Education Programs and Policies Research Grants is a federal research project from the Institute of Education Sciences that funds rigorous evaluation of initiatives that are either well-established or ready to be implemented. Applicants are asked to propose a thorough research project that will contribute to the Institute’s research in evaluation of education programs and policies implemented by state or local agencies. Research needs to contribute to improved academic achievement for all, especially those who are hindered by conditions related to poverty, race, English proficiency, disability or family circumstance. A notice to apply is due by July 10, 2008 and final applications through Grants.gov are due October 2, 2008. If you have an interest in assessing an initiative that you are ready to launch or have already launched, the Improve Group can be a partner in applying for this evaluation opportunity. We have extensive experience in conducting research and evaluation projects in educational settings for new initiatives and long-established programs. For more information about the Grant, click here. The Institute’s website also offers webinar sessions that introduce and give an overview about research topics and the application process. Please contact Susan Murphy at susanm@theimprovegroup.com if you are interested in working together on this worthwhile research.
|
|
When deciding on that “perfect time” to conduct an evaluation, an organization must balance many factors, such as available funds, reporting deadlines and availability of staff and/or an outside agency to complete the evaluation. No matter when the evaluation process begins, a well-thought out evaluation plan will result in the most useful findings for an organization.