 |
|
 |
Issue 21: February 2009 How Do I Measure What Can't Be Measured? Organizations, agencies and school districts have long been collecting data that is tangible and easily measured: number of clients receiving services, number of students graduating, number of classes held, etc. In recent years, however, as private and public funders have turned their focus to program outcomes, program providers face the complicated task of documenting what at first glance seems not so easily measured. When first designing an evaluation, organizations contend with defining measures for changes in knowledge, behavior and attitude. For example, we’ve helped clients determine how to measure changes in their participants’ feelings of safety, self-esteem or satisfaction, and ability to lead a healthy life. These phenomena are referred to as intangibles. Non-profits, government agencies, school districts and corporations alike face the challenge of measuring intangibles, and while there is still some debate on how to define intangibles, there is a reliable technique to measuring them: Define your organization’s intangible goal by breaking it down into manageable and measurable parts. As an evaluator of school, youth and community programs, the task of measuring intangibles arises frequently in my projects. The first rule of thumb I apply when an intangible rears its head is that any broad intangible goal is measurable by the ways you define it. In other words, you must break down that goal into smaller easily manageable indicators or measures of success. For example, if your program goal is to increase the self-esteem in adolescent girls, you must first define what “self-esteem” is. What are the components of self-esteem? How does one act when one has high self-esteem? What does one need to gain or eliminate? This process of identifying indicators of success can be accomplished in a brainstorming process among program staff, ideally during the program planning phase. With the answers to these questions, you are able to create indicators or measures of success such as:
The individual indicators are not meant to answer the overall question of whether girls’ self-esteem has improved, rather all the indicators taken together as a whole provide nuance and a comprehensive picture of what aspects of self-esteem have improved. The intangible goal has now become the sum of its (measurable) parts. Now that you have these measurable indicators identified, the next steps are to ensure that these indicators align back to your program outputs or services, and to identify how you are going to use the data collected on these indicators. Is your program designed to help adolescent girls develop positive and feasible plans for their future? If not, you may not want an indicator related to this. Once you have a list of feasible, measurable indicators to measure your overall program goal (in this case: increasing self-esteem in adolescent girls) designing your data collection instruments will be a seamless process. You can create standard pre/post test surveys, post/retrospective pre-test surveys, or focus group or interview protocols with items that directly align to your indicators. Once your data is collected and analyzed, you can then feel confident that you intangible goal has been effectively measured. You also will have a good working definition of your goal, that you can use when engaging partners, funders, staff and participants. This process of breaking down goals into measurable utilizable parts is highly valuable to any program. It can facilitate successful program planning, evaluation and decision making. If you are interested in learning more about measuring intangibles, the following are some informational websites: http://www.missiondrivenmarketing.com http://www.value-networks.com/guides_and_tools.htm
|
|
Record-Keeping - Minimizing the Burden As programs and organizations work to meet the demands of informing their board, engaging stakeholders and partners, answering funder questions and managing their work, there is usually no shortage of record-keeping. Records may be in computer notes, spreadsheets, databases or paper format…and occasionally in staff brains. Different staff often keep different kinds of records depending on their responsibilities within an organization or program. When an evaluator shows up on the scene, the temptation for a consultant might be to request that the organization or program begin tracking 100 new items specific to the evaluation. Despite the allure of all this data (perhaps only to evaluators), this kind of request would result in staff either spending their time record-keeping instead of pursuing their program goals, or in data not being tracked at all since the demands are so unrealistic. The Improve Group has several approaches that help to balance the need for data to inform an evaluation, with the burden that record-keeping can put on staff. The Improve Group’s work with the NetWork for Better Futures (the NetWork) provides some recent examples of these approaches. The Improve Group is serving as an independent evaluator, hired by the Minnesota Department of Corrections, to evaluate the NetWork’s demonstration project that aims to break new ground in helping men released from incarceration successfully re-integrate into the community. The NetWork, based in north Minneapolis, uses a holistic, community-based approach to help a target group of offenders who are historically at high risk for re-offending. The NetWork project approach has five core elements: housing, behavioral health, health care, short term employment and family/community re-engagement. The Improve Group recently completed our first annual report and submitted this to the Legislature and Commissioners of Employment and Economic Development, Housing Finance, Human Services and Corrections. With such a broad range of impacts in the NetWork’s holistic approach, the list of relevant indicators was long – ranging from health, housing, employment, recidivism to community participation. The approaches the Improve Group is using to manage the data collection burden on the NetWork are:
The NetWork still had significant responsibility for keeping some program records, and providing feedback on and contacts for the Improve Group’s data collection management. But, the Improve Group endeavored to keep the items that the NetWork was requested to track in its program records to those that are core to its program and work. Thus, the NetWork should be able to use this same data for reporting to other funders as well as managing its own work. The opportunities and challenges in collecting data are unique to every client as every organization and situation is different. While the Improve Group will continue to learn more with every client about how best to manage data collection in their particular situation, the overriding principles we adhere to are to devise strategies that result in valuable evaluation feedback while minimizing the burden on organization and program staff.
|
|
|
Sabbatical: A Break or change from a Normal Routine (as of employment) I have been evaluating, planning and researching for organizations for nearly a decade. Every September that rolls around I get the itch to enroll in another degree program. While I learn constantly through my practice, I get very little opportunity to immerse myself in the latest research and reflect on what I've learned. With the arrival of this fall, I decided to take matters into my own hands: a seven-week sabbatical timed to coincide with what is typically our quietest time of year. My carefully organized reading pile stretched about... two feet high. While I made a pretty significant dent, there is still much more I want to learn! However, I think some of the things I read will be of interest to our readers: ● The best laid plans of mice and men often go awry. The first articles I read focused on planning for businesses and organizations. One key area of consensus among all of the authors: plans can only take you so far. Organizations need to be clear about their core strengths,competencies and values because, when the unexpected happens, that clarity will help you rapidly regroup and respond. During my sabbatical we were also wrapping up some planning efforts with clients; these clients were also regrouping in light of the rapidly changing economy, and focusing on those strengths and values helped during that process. I wish I could say that all my plans during my sabbatical worked as intended (leave the office every day at 11, read for 3 hours, write every day), but of course they didn't! I was able to draw on my main purpose (refresh, learn, reflect) to help bring me back to center. ● Literature on evaluation and strategic planning contains a lot of observations and results from experts, but little in the way of reflections by organizations that are doing planning or being evaluated. That is a hole I hope to help fill the coming year. I'm going to be surveying our clients about their experiences with us, and hope to achieve a few more targeted interviews as well. I'll share those stories and lessons learned over the year. ● It seems everyone is interested in social networking. The astonishing statistic I heard while on sabbatical was that 35 percent of all adults in the U.S. have Facebook accounts. I have not been able to verify the fact, but I have been able to find nearly everyone I sought on Facebook (you can even "friend" the Improve Group!) We think social networking has tremendous potential for evaluation and planning. If you can get a group together to let them know about an upcoming survey, build consensus about key program goals, or simply provide a user-friendly way for them to say what they think an organizations' strengths and weaknesses are, all on-line, you've engaged people and potentially saved money, too. We've used Facebook and MySpace to recruit survey participants on behalf of the Minnesota State Arts Board and United Jewish Fund and Council, and have set up a Yahoo Group for county case managers for our waiver review project to share tools and resources across counties. We'll write more in the coming months about these tools. So, there you have it; a few ideas and tidbits. It's nice to be back in the office, but I plan to continue to make time to read. If you have any recommendations - send them our way!
|
|
