Issue 14 : July 2008

Building Confidence: The evaluator's #1 job
by Rebecca Stewart

Did we make a difference? Although many organizations can comfortably demonstrate how well their programs are liked, or how many people they served, it is not so easy to determine how a life changed thanks to the service. Even when relevant data is available, it isn’t easy to say how robust the data is – how much it represents a true change.

Trusting your data

How much you can trust your data is measured through statistical confidence. You probably have seen examples of statistical confidence, such as:

• We can say with 95% confidence participants in our program will have between $2,000 and $11,000 increase in income
• Treatment A was effective in about a third of the population (95% CI)

When thinking about confidence, you are balancing what is typical in a group of people with what happens to an individual. Individuals may experience a wide variety of outcomes. A confidence interval defines a small range of outcomes experienced by most people in the population under study. In the diagram below, the majority of people (95% of people) experienced an income change between $2,000 and $11,000, with a mean change of $6,500. So, you could say that you can be 95% confident that the average income change was between $2,000 and $11,000, or that there was an average income increase of $6,500, +/- $4,500.

Continuing with this example, if the organization recorded every person’s income both before and after the program, and every one of those people provided their income data, then the organization can feel 100% accurate about the typical increase or decrease in income after program participation. However, in most studies, data is not available for every single person in a program.

Two of the main factors affecting how much confidence an organization can have about its  service impact are related to data collection: (1)sample size and (2) response rate.

• Sample size refers to the size of the group which receives a survey. This is a subset of the whole population that the organization has served.
• Response rate is how many of those who received a survey return it.

Nearly all of our clients ask us about sample sizes and response rates that would ensure the accuracy of its survey results.  Although the answer is complicated, there are some principles we use to guide recommendations. In most studies, the goal is to achieve a confidence level of 95%. This means that the organization can be 95% sure that the results of its study represent the results they would get if they had received a response from every single person in their program. The response rate needed for a desired confidence interval can be affected by a number of factors about the program and its participants. The organization will need to work on achieving a higher response rate in the following situations:

• The population has a lot of variance. For instance, in the example about income, participants might have two income earners in their household or one; have varying levels of education; be employed in different industries that are affected by the economy in varying ways; or have varying health and family situations that effect income. .
• The organization wants to have analysis about different subgroups. In our income example, the organization may want to know how different age groups, parents, youth, seniors or immigrants effected by the program.
• The population under study is small (i.e. 100 or fewer people).
• The organization wants to ensure great accuracy. Any high-stakes studies (i.e., those of medical treatments) require great accuracy in pinpointing exact results (such as an extended life expectancy of 3 years vs. 4 years). However, determining whether a person had a $3,000 or a $3,200 increase in income may not be as important.

Ok, so after laying out all the confounding factors to confound you, here is a helpful chart, provided by Fred Van Bennekom, Dr. B.A., Principal Great Brook Consulting.

What helps to achieve a good response rate? The Improve Group experience shows higher response rates are supported when:

• A known population/participant group that has a strong relationship with an organization and its staff.
• Accurate contact information is available for participants.

• A program or larger authority (i.e. school) provides access to the population and a set time and place to complete a survey.
• Active consent is not required (active consent requires a step that may result in losing some participants).
• The survey length is limited to those items most critical for analysis of program impact.
• Respondents can easily return the survey (i.e. web-surveys for those with internet access, or a survey administrator collect surveys after group administration).
• Surveying is timed to capture impact (close to the end of program participation) but not at a time when the population is distracted with many other demands on their time (the last day of school).
• Provide an incentive.

Finally, how should the results be interpreted if the response rate was less than the desired level? The Improve Group has the following advice:

• It is always possible to gain insights and learn about a program, even if the response rate is not great. Although the results cannot be generalized to the broader population, because you do not know how that population differs from respondents, the results can tell you about the group of people that did respond.
• Share the confidence level that the response rate can support – maybe the data do not support a 95% confidence level, but do support an 80% confidence level which may be sufficient.
• If the response rate was better from some sub-populations, the results can be generalized to conclusions about that sub-group.
• If responses show limited variability, the required response rate for a high confidence level might be lower. So, if many of the responses are the same, the organization can have some confidence that this does represent how the whole population would respond.

One concern about lower response rates is that respondents might represent only those who had very positive or negative experiences, or that access to the survey may have been limited to a set of respondents with particular characteristics. An organization can review the data and the data collection process to get some insights into whether this is happening. If the respondents express both positive and negative views, the organization can be more reassured that a spectrum of these perspectives is represented. If respondent characteristics on a survey show some diversity, the organization can also have confidence that the respondents are representative of the broader population. In addition, a review of the data collection process may reveal a problem in the collection of surveys that has nothing to do with who decided to respond to the survey (such as a box of surveys being lost – our worst nightmare!). This situation would certainly suggest that there was something wrong in the data collection process, but not necessarily a bias in the data.

One of the important rights of passage from school student to future member of the job force is obtaining a summer job or special internship that introduces you to the world of work. Achieve!Minneapolis, a nonprofit organization bringing community resources to assist Minneapolis schools, is involved in giving students access to a wide variety of business internships. Their STEP-UP Summer Jobs Program was developed in 2002 to provide summer internships and job readiness training for Minneapolis school students ages 16 to 21. Last summer, this highly successful initiative engaged 131 employers who hired over 600 Minneapolis youth.

One of the ideas behind this program is to expose youth to a variety of careers, but the exposure is a two-way street. There is an injection of new life that comes to an office when interns are on the job, as we learned with our first interns in 2006.  A different energy and a different outlook infuses the workplace, which is a healthy boost to the whole staff. This summer, we are employing two new STEP-UP interns who are currently working hard at data entry on one of our education projects. Most of the data they are working with is from surveys of students their own age but in a different district; they are helping us to understand the youth point of view in some of the responses.

STEP-UP job applicants are matched to their jobs based on their career interests and particular skills. Employment lasts from 6 to 10 weeks and interns are expected to work between 20 and 40 hours per week. More information on this program is available on the STEP-UP website at: http://www.achieveminneapolis.org/aboutUs/stepUpContacts.html.