At the Improve Group, we work with surveys and survey data nearly every day. Surveys are a great method for reaching a large audience in a cost-effective manner, but they also have their drawbacks. The people you’re interested in reaching are not always motivated to take the survey and return it back to you. In our work with the Performing Arts Workshop, we have achieved some high response rates using an incentive to motivate teachers and parents to return surveys and consent forms.

In order for students to participate in our Performing Arts Workshop evaluation, their parents must first give consent. Last year, we found that many parents had not returned consent forms. We didn’t know if these parents truly did not want their child to participate in the evaluation or if they simply forgot to return the form. This year, we wanted to decrease the number of parents who forgot to return forms, so we enlisted the help of the teachers and offered an incentive for their help.

We consulted with teachers that were involved in last year’s evaluation and asked them suggestions for an incentive for their classrooms that would help motivate parents to return forms. Based on their advice, we are offering teachers with high response rates a $100 gift card to be used in the classroom for healthy snacks or other classroom supplies. This incentive has been highly successful so far; two weeks before the surveys are due back to us, we’ve already had three packets returned with 100% response rates.

We believe the incentive works so well because it motivates parents to return their forms and it also motivates teachers to follow up with parents that do not return forms. The incentive is given for offering the consent form, so parents who choose not to participate in the evaluation are not “counted against” the classroom total. We just ask that teachers let us know that the parents declined to participate in the study.

In offering an incentive, our goal was to ensure that each parent knew about the evaluation and was able to make an informed choice about whether their child would participate. Thanks to the help of the teachers, we have achieved this goal. We’ve found that the right incentive can do wonders for motivating people to proactively participate in evaluation.

Posted by admin at 08:51 AM | Permalink | Comments (0) | TrackBacks (0)

Chicago Opera Theater's Opera for All program brings professional performers into K-12 Chicago classrooms for a rigorous opera experience tied to the Theater's premier performance of the season. This year, Chicago Opera Theater is debuting a new opera called A Flowering Tree. Each class learned about the opera itself, as well as musical concepts, culture and the arts and the history and context of opera as an art form. Each class has a culminating performance -- with student responsibilities including writing, performing, directing, and set and costume design.

One high school class recently performed Pirates of Penzance, one of Leah's favorite operas as a kid. We've had the pleasure of working with the Opera for All program to design an evaluation that helped to better define the program's goals and outcomes.

Posted by admin at 11:43 AM | Permalink | Comments (0) | TrackBacks (0)

Last week I attended the Minnesota Evaluation Studies Institute (MESI) Evaluation around the world and in your backyard: Strategies, Tips, & Techniques 2008 Conference. This was the first time I was able to attend the entire conference and learn more about evaluation from the best in the field. Speakers included well respected practitioners such as Michael Q. Patton, Ross Conner, Jennifer Greene and Richard Krueger, to name a few.

I was particularly struck by lessons from Professor Ross Conner’s opening presentation on Evaluation Around the World. He spoke to us about several countries around the world and how they used and valued evaluation. For example: Africa felt evaluation could “bring the sun in on what was happening in their country.” In the Middle East evaluation was used in a context of supporting diversity and inclusiveness. For Romania, the field of evaluation was aimed at accountability and transparency.

Another lecturer of note was Michael Quinn Patton, former president of the American Evaluation Association. In his response to Ross Conner’s talk he reflected on the use of evaluation as “a form of discourse and deliberation in democracy.” He noted the secrecy that has crept into our society and how evaluation can be a “fundamental means of connecting to a free flow of information.”

There was a great international mix of people attending the event with a large contingent from Norway. What was most inspiring about the sessions was the theme of a global, multi-cultural approach in how evaluations are formed. A lot of our conversation turned to ethics and how information is used with regard to a civil society.

Minnesota is well-known for its strength in the advancement of evaluation as an important organizational tool. Thank you to MESI for helping evaluators stay current and responsible in their practice by hosting informative conferences like this.

Posted by admin at 11:33 AM | Permalink | Comments (0) | TrackBacks (0)

Here is a tale for anyone that has been surprised by evaluation results:

As an undergraduate student at American University, I got my first taste of “formal” evaluation by conducting an evaluation of a peer health program on campus. In its “natural” setting, the program was presented to a small group of college students; over milk and cookies, presenters would provide educational information to college students about health issues they faced. In the name of science, I recruited participants in the residence halls (bribing them with candy) and randomly assigned my participants to “treatment” (program) or “comparison” (no program) groups. After an arduous semester involving the institutional review board, coordinating program showings and tracking down college students to fill out my confidential surveys, I had finally finished.

I analyzed my results and was amazed to find that NOTHING HAPPENED! The students showed no change in attitudes or behaviors. They didn’t even show an increase in knowledge! It was hard for me to believe that a program I cared so greatly about did not work.

I have since learned my findings were more complex than I first thought. There are two main explanations for null results in evaluation: 1) the program really did not have any impact on the intended outcomes or 2) the program could have had impact on the intended outcomes, but my evaluation design and instruments did not capture these findings.

It is quite possible that my program did not have an impact on students’ knowledge, attitude or behaviors around health issues. It was one hour-long program in a busy semester for students. Another possibility is that the program did have impact on the intended outcomes, but they faded over time. Perhaps if I had administered the surveys immediately after the program, I would have seen improvements in attitudes or knowledge.

The goal of my project was to learn about research design, and in that sense, the project was a success. I also learned anticipate when an evaluation may have the most impact and design the evaluation accordingly. Please watch my Research Tidbits column for more information about evaluation timing. If you have any similar evaluation stories to share, I’d love to hear them!

Posted by Guest author at 10:56 AM | Permalink | Comments (0) | TrackBacks (0)

by Leah Goldstein Moses

In our evaluation and strategic planning work, we are increasingly emphasizing "authentic" data when developing a design with our clients. What I mean by authentic data is that as often as possible, we ask people directly how they have experienced a service and changed as a result. When the people experiencing a service are young kids, or have a different language or cultural reference from the person who is doing the study, it is vitally important to check to make sure you are interpreting results correctly.

A recent study by the University of Minnesota shows the importance of authentic data in developing programs. Researchers at the medical school found that when they asked children directly about their experiences, they had very different responses then their parents.

Additional resources: Students Are Stakeholders, Too! Including Every Voice in Authentic High School Reform. Edie Holcomb.

Posted by admin at 11:09 AM | Permalink | Comments (0) | TrackBacks (0)

In junior high school, did you ever feel like you were alone in this world, that no one really understood you? Well, welcome to the world of evaluation where being an “outsider” can be a good thing!

Outside, or external, evaluators have the ability to examine programs with a fresh perspective. In some cases, external evaluators bring more objectivity to the evaluation process as they have not developed expectations about a program’s success prior to the evaluation. External evaluators inherently have limited experience with the specific program to be evaluated. Time and time again, Improve Group staff members say that learning about the programs we evaluate is one of the best parts of our job. For some tips on when an external evaluator might be right for your organization, click here.

To effectively evaluate programs as an outsider, we must learn about the program from stakeholders, or people who have a vested interest in a program’s success. (Click here for the EPA’s glossary of the evaluation terms I discuss in this article and many, many more). Each program has many stakeholders; these can include program participants and line staff, funders, the Board of Directors or supervisory staff, government agencies and the general public.

Evaluators work with stakeholders to understand the program and develop the evaluation plan. Each group of stakeholders has different evaluation needs; sometimes the needs of one group conflict with the needs of another. Most often, it is not possible to collect all data requested due to time and budgetary constraints. Evaluators prioritize which data will be collected by identifying a primary audience and secondary audiences.

The primary audience can be any group of stakeholders. However, the funder is often the primary audience for programs that depend on outside funding. When the evaluation needs of the primary audience have been identified, evaluators must negotiate with the secondary audiences to determine which additional indicators are most important to measure. Effectively balancing stakeholder needs can be the key to a successful evaluation. Ideally, each stakeholder group will have a say in which data are collected and how the data will be used.

Note: Some of the information presented here came from an excellent presentation at the 2007 American Evaluation Conference by Kimberly Taylor of Schwab Rehabilitation Hospital. Ms. Taylor is an internal evaluator that beautifully uses and teaches the principles of evaluating like an “outsider.”

Posted by Guest author at 03:40 PM | Permalink | Comments (0) | TrackBacks (0)

On Dec. 12, 2007 a new executive order initiated a review of nearly every federal agency to determine:

1. If the agency targets were suitably outcome-oriented; and
2. If measures are appropriate for each target

The executive order classifies targets in three categories: (1) Long-term Measures: Program outcomes that fulfill the program’s purpose; (2) Annual Measures: Implementation of plans and efforts to achieve long-term and strategic goals; and (3) Efficiency Measures: Efforts to provide the most benefits (outcomes and outputs) for the taxpayer dollar spent.

The timeline of this review is very aggressive; each agency will be reviewed by February 22, with revised measures developed and adopted by June 30. What is interesting is the focus on reliability, validity and quality (see quote below) -- which align closely with the Guiding Principles for Evaluators.

From the Order:

The Government’s ability to determine a program’s effectiveness, and to direct attention to genuinely desired outcomes, is largely dependent upon the quality of the programs’ performance and efficiency goals, i.e., their measures and targets. PART goals aren’t always as outcome-oriented as they can be. We should continue to make PART program goals more outcome-oriented and aggressive as well as ensure measures are characterized correctly in PARTWeb.

Posted by admin at 01:05 PM | Permalink | Comments (0) | TrackBacks (0)

Last week, Marian and I attended the American Evaluation Association’s annual conference in Baltimore. The theme of the conference was “learning to evaluate…evaluating to learn.” Throughout the four-day conference, we were charged to learn new skills and reflect on the many ways that evaluators learn from our clients in the course of our work together. Looking back, I’m amazed at all that I had gained in a few short days.

With over 500 sessions to choose from throughout the four-day conference, we were certainly spoiled for choice. I really enjoyed attending skill building and demonstration sessions where I learned practical applications in the field. I was trained on an alternative to the traditional literature review, how to succinctly tell a program’s story and how to meet the diverse needs of multiple stakeholders. Watch my monthly blog column, research tidbits, in the coming months for brief overviews of these topics.

A definite highlight of my trip was spending time with our client, Girl Scouts of the USA (GSUSA) staff. Dr. James Riedel from GSUSA (with some assistance from Marian) presented retrospective pre-test data from the PAVE project in a multi-paper session on alternatives to randomized control trials. It was so exciting to see how the work we had done together in the PAVE project will benefit other evaluators in the field.

Of course, no American Evaluation Association conference would be complete without an opportunity for attendees to evaluate the conference. Attendees were asked to report “key insights” we gained throughout the week. For me, my key insight occurred in a session about evaluation theory. The presenters discussed how evaluators have as much to learn from clients’ expertise as they do from ours. This is precisely why I enjoy evaluation so much; we’re always learning from our clients. I look forward to reflecting on a year’s worth of learning in the field at the 2008 American Evaluation Association conference in Denver!

Posted by Guest author at 10:46 AM | Permalink | Comments (0) | TrackBacks (0)

Two recent studies have found conflicting conclusions on whether the No Child Left Behind Act (NCLB) is pressuring educators in struggling schools to focus on students who fall just below the passing threshold on state tests (i.e. bubble kids) at the expense of students at the high and low end of the achievement spectrum.

One study, conducted by University of Chicago economists Derek Neal and Diane Whitmore Schanzenbach, focused on the 421,000 Chicago school district. The study examined two time periods during which the school system was shifting to a testing-and-accountability system that increased pressure on educators to raise test scores. The time periods examined were the introduction of NCLB in 2002 and similar district level reforms in 1996.

In both periods, the study showed that the reforms generated the largest increases in reading and math scores among students in the middle of the pack. However, the lowest-performing students generated no score improvement and high-performing students achieved mixed evidence of score improvement. According to the Chicago economists, these results suggest that the choice of the proficiency standard in these accountability systems determines the amount of time that teachers devote to students of different ability levels (i.e. ‘education triage’).

Conversely, a study conducted by Vanderbilt University researcher Matthew G. Springer examined three years of test-score data on 300,000 students in an unnamed Western state. The study divided the students into 20 groups based on their test scores.

Springer found that schools identified as having fallen short of their performance goals succeeded in raising achievement for the entire range of students at risk of failing, without sacrificing the academic progress of the most gifted students. Springer concluded that the students did not show evidence that educational triage is taking place.

Since both NCLB and research of its effects on student achievement are relatively new, I think more research needs to be conducted to indicate conclusive trends of what may be going on in schools nationwide. Specifically, studies across different sized school districts with different racial, ethnic, socioeconomic and disability subgroups of students in different regions of the country.

Do you have anything to add from your experience working in education? I invite comments on this subject.

Posted by Guest author at 09:11 AM | Permalink | Comments (0) | TrackBacks (0)

Over the past year we have worked with Girl Scouts USA to evaluate the Uniquely ME! program, a program designed in 2002 to address low self-esteem, which is increasingly recognized as a problem facing young girls.

In our evaluation, we are focusing on the key components of girls' ability to "Discover, Connect, Take Action". Using a retrospective pre-test/post test model, we learned that the program has helped girls to take risks and try new things, confidently find their talents and find ways to make their lives better.

This initiative has been funded in large part by the Dove Self-esteem Fund, which recently prepared a short video called Onslaught demonstrating why this work is so necessary.

On a personal note, the importance of this work was made startlingly relevant to me a week ago when my five-year-old daughter informed me that I could buy a cream that would make my skin look as beautiful as hers. Needless to say, I was a little taken aback by the statement and her preoccupation with physical appearance! The troops in our area don't start until age 6, but we'll be looking forward to some of the lessons available from Girl Scouts.

Posted by admin at 04:33 PM | Permalink | Comments (0) | TrackBacks (0)

Evaluation deals with public information and involves the engagement of many different communities. When our work requires we access these communities , we have the responsibility to be professional, use valid methods and share current information. To assist in this effort the American Evaluation Association developed the Guiding Principles for Evaluators to provide an ethical framework within which evaluators should provide services to the public. They are also meant to foster discussion about promising practices and encourage continuing development in the profession.

The Guiding Principles for Evaluators are:

 Systematic Inquiry
 Competence
 Integrity/Honesty
 Respect for People
 Responsibilities for General and Public Welfare
(See Leah’s August 20th blog for more details on each principle).

At the Improve Group we have discussed these principles and endeavor to use them in a proactive way as we conduct our business. They keep us focused on veracity in our research and maintaining a transparent and open process with our clients.

We would encourage everyone who has had, or is planning to invite evaluators to work with their organization to read the detailed explanation of each of the principles and the context in which they were established. They can help you think about what to expect when working with an evaluator and understand why they make some of the recommendations they do.

Posted by admin at 02:44 PM | Permalink | Comments (0) | TrackBacks (0)

The American Evaluation Association (AEA) recently re-worked the Guiding Principles for Evaluators. These principles are the ethical foundation of our work. The AEA rightly notes that these principles are meant to inform but cannot account for every possible situation and that occasionally the principles come into conflict with each other.

We recently used the AEA guiding principles at a staff meeting to talk about how they guide our work. We examined many recent projects, including our work with the Minnesota Department of Human Services, the Northwest Minnesota Council of Collaboratives, and the Girl Scouts.

The principles follow, along with some example questions that evaluators can ask when using the principles to inform their work:

Systematic Inquiry: How are methods of participant selection important to the credibility of an evaluation? What about methods of data collection?

Competence: How can you decide what dimensions of competence are relevant for an evaluation? What is cultural competence and how will you know its presence?

Integrity/Honesty: How might the idea of integrity or honesty affect different stages of the evaluation?

Respect for People: What are major ways of showing respect for people in an evaluation context?

Responsibilities for General and Public Welfare: Are there limits to an evaluator’s responsibilities for general and public welfare?

Posted by admin at 03:45 PM | Permalink | Comments (0) | TrackBacks (0)

A while ago I blogged about qualitative and quantitative data, and promised a Dilbert comic when I received permission to use it. Well, here it is:

DILBERT: © Scott Adams/Dist. by United Feature Syndicate, Inc.

Posted by admin at 12:54 PM | Permalink | Comments (0) | TrackBacks (0)

From our Participant Primer series, you'll learn about what it is like to be a participant in interviews, focus groups, observations, or program/administrative data collection.

A common practice at the Improve Group is to also include participants in the design of our research and evaluation work. We like to hear from community members, people who attend programs, and staff:

1) what they hope to learn from an evaluation
2) what is the best method for collecting data
3) any logistical issues we should be aware of in collecting data
4) how they would like to receive information or reports about the evaluation and its results

Recently, I traveled to Mo Ranch near Hunt, Texas to meet with Hispanic Scholarship Fund staff and Peer Counselors (Peer Counselors are college students who are matched with high school students for mentoring and to share information about college). The purpose of the meeting was to ask the group of 19 young adults about their evaluation questions and what they hope to gain by being Peer Counselors. We had a great discussion that will help us as we develop surveys and focus groups for the evaluation.

In my drive from San Antonio to Hunt, TX, I was startled to come across Stonehenge (see below).

A quick Internet search led me to the following amusing articles in Roadside America and Texas Twisted.

Posted by admin at 01:12 PM | Permalink | Comments (0) | TrackBacks (0)

Are you the kind of person who loves a good story and can instantly think about how that experience might relate to other people? Or do you really enjoy seeing a chart or a table and thinking about how all the different numbers explain an issue?

Stories, videos, pictures, recordings all can provide valuable information -- and evaluators usually lump them together in the category "qualitative data". Charts, tables, percentages are equally valuable and usually lumped together as "quantitative data".

We usually use both types of data when helping our clients understand the outcomes or results of a program. While both presents interesting information, using them together provides us with a greater depth of understanding about what happened when someone tried a new program.

Yesterday someone asked me "How do you get someone to see the value of qualitative data?" and this was probably the tenth time I've been asked this question. Some people are just going to want hard numbers and that quantitative data is the best way to get at the truth of a situation. I don't dispute that numbers are great, but you can't guarantee that everyone is going to respond to them.

One way to make qualitative data more responsive to a number-phile is to present both types of information side-by-side. For example, you may have a chart on the left-hand side of a page, with relevant quotes on the right-hand side. (By the way, when I was checking to make sure "phile" was the right suffix to use, I found a number of great resources: the Columbia Guide to Standard American English, the suffix entry at Wikipedia, and the Vocabulary Workshop for people preparing for an American University with its extensive discussion on suffixes. There was also a nice list at http://www.michigan-proficiency-exams.com/suffix-list.html, but it didn't include "phile")

Another way to present qualitative data to those who prefer numbers is to convert stories to numbers. For example, you might do a focus group or two and learn that several people have enjoyed a specific social experiences through a program or organization. You could then design a survey for more people to find out if they similarly enjoyed the social aspect of a program, using some sort of scale. We used this approach in our work with United Jewish Fund and Council, in which we first explored some issues through focus groups and then conducted a larger community survey.

There is a related Dilbert cartoon; I am waiting for permission to publish it.

Posted by admin at 02:01 PM | Permalink | Comments (0) | TrackBacks (0)

Improve Group staff are devoted to using evaluation to improve outcomes, so conducting a test presentation of our new workshop series seemed natural. On June 14th, 2007 we gave a pilot run of How Do I Know That: Creating tools to understand program outcomes and Inside Knowledge, Outside Perspective: Evaluating your organization form within. The series is aimed at those who want to begin or expand their evaluation knowledge and feel empowered to use it in their organization.

Presenter Leah Goldstein Moses taught participants how to develop evaluations that fit smoothly with program activities while maintaining validity and accuracy. Presenter Marian Kimball Eichinger gave insight into how to improve the evaluation work within organizations and how to select the right tools to get legitimate results.

Attending the pilot was Cassaundra Adler - Financial Educator with Lutheran Social Services, Trisha Lee Cook – Member Services Associate for the Minnesota Council on Foundations and Nathan Salzl – Evaluation Specialist for the Greater Twin Cities United Way 211 Call Center. Also attending was Improve Group consultant Jules Goldstein (who is a member of the Board of United Jewish Fund and Council) and staff member Susan Murphy.

Our thanks to Cassaundra, Trisha Lee and Nathan who were a perfect blend of non-profit experience, foundation background and small and large organization perspectives. The attendees gave honest and valuable feedback on what were the most useful aspects of the training, what could be added or deleted to better serve the needs of the target audience and how to enhance specific exercises and teaching materials.

The pilot group gave a unanimous thumbs-up to the new series, saying:

• The workshops helped me figure out where to start with my organization's evaluation. I felt overwhelmed before and unsure whether I was representing my own opinions or had solid facts; now I know how to collect, analyze and present accurate data.
• I gained a great vocabulary for talking with funders, program managers and staff; the tools provided during the workshop can easily be adapted for our programs.
• I am excited to use the information for my program; I like learning and continually improving my program and was never sure how to ask questions to get honest and fair responses. I am going to review our surveys to make sure they are gathering the information I need.
• The workshops were very fun, very informative and I feel lucky to be invited.

Space in the July 19th sessions is available; to download a registration form click here.

Posted by admin at 11:33 AM | Permalink | Comments (0) | TrackBacks (0)

While participating in a recent training on human subject protection in research, I was struck by one topic that I would not have normally thought of as a thorny ethical dilemma. In our experience, incentives are usually determined by a balance of the resources an organization can give with what we (the researcher and the organization together) believe will encourage people to participate.

As we have described in a recent newsletter article (http://www.theimprovegroup.com/newsletter/Jan2007webnews.pdf), it is difficult to determine fair compensation if you try to pay people for their time, so we rarely think of incentives in that way. Instead, we often think of an incentive as a way to make a social compact (when provided upfront) and/or a way to help overcome a final lingering barrier to participation, whether that be braving a rainy evening to attend a focus group or spending 15 more minutes on the computer to fill out a web survey at the end of a long day.

However, a main tenet of human subject protection is to protect the voluntary nature of participation in a research project. The National Cancer Institute says “There are no clear rules or standards for payment other than a general prohibition against coercion or the exercise of undue influence. There is no agreement about whether it is right to pay research subjects.” (http://cme.cancer.gov/c01/b04_03.htm )

This issue is particularly sticky for those researchers whose studies may pose a risk to research subjects, such as medical studies, wherein the incentive could theoretically coerce someone into taking on an ill-understood risk for an immediate pay-off. Much of the research we do poses much risk, but often does incur some inconvenience or time commitment. But, we still need to be aware of the ethics surrounding the incentives we offer. We continue to think of them as something that helps a willing participant overcome hesitation about the inconvenience or time commitment of being involved in research. Moreover, I have to wonder about the quality of information we would get from an UNwilling participant whose only motivation is the incentive and not the desire to give their perspective on an issue or feedback to improve a program.

Posted by Guest author at 11:56 AM | Permalink | Comments (0) | TrackBacks (0)

Many of our clients have done previous strategic planning or evaluation projects, and are familiar with external consultants. Their previous experience sometimes lead them to expect to be somewhat removed from the project, with the consultant completing tasks independently.

At the Improve Group, we tend to take a more developmental role to strategic planning and evaluation -- forming a "learning community" partnership with our clients. We also encourage our clients to engage others - program participants or students, elected officials, people with common interests, etc. -- in their process.

For some organizations, a somewhat formal public relations process can be helpful. For example, in our work with Austin (MN) public schools, we engaged students, teachers, parents, administrators and community members to establish a vision and set goals for their community. The district supported this process through press releases and frequent communication. A recent article in the Rochester Post-Bulletin describing the results so far of the effort is one example; previous articles invited people to complete a survey and participate in community meetings. The result was a participation rate higher than we've seen in comparable projects.

Doing a google search for "strategic planning press release" or "evaluation press release" turns up mostly the findings or results of a project. What has made our work with Austin so unique is that they engaged their local media throughout the process. Their goals in doing so were to:

(1) Increase participation in the strategic planning process
(2) Ensure that everyone who chose to do so had an opportunity to learn about the process
(3) Keep people informed as the process evolves
(4) Ensure that no one is surprised with the results because there have been summaries at multiple stages

The press may not always be a good tool. In Austin, the District had a good sense about what sources of information people used and how to access each of them. They included web- and radio-announcements in their strategy. In some communities we work with, there is uncertainty about which media outlets people access the most, or media are so saturated that it is cost-prohibitive to reach more than a small fraction of the population. In those cases, it is important to be targeted with your approach, using media outlets to access populations that would otherwise not participate.

Posted by admin at 11:30 AM | Permalink | Comments (0) | TrackBacks (0)

As members of the American Evaluation Association we were recently asked to submit nominations for annual awards, including the Marcia Guttentag award for evaluators in their first five years in the profession. After reviewing the criteria, we knew Becky exemplified the criteria:

* Promise will have been demonstrated in early notable and substantial accomplishments.
* Contributions may include evidence of early involvement in professional activities and successful leadership.
* Contributions should have extended beyond the accomplishment of one or two evaluations of quality.

As part of our nomination of Becky, we got to ask some of our clients for recommendations. It was a delight to read their kind words:

Though Ms. Stewart began evaluating our project in January, her impact on our
evaluation processes has been profound. She helped us clarify our project objectives and improve the
alignment of our evaluation instruments with the newly refined objectives... Becky’s commitment to developing shared understandings of our project goals and measures has meant that each facilitator’s point of view and understanding has been engaged - Byron Richard, Perpich Center for Arts Education

Rebecca's biggest strength is reflecting the principle of responsibilties for the general and public welfare. Not only does she take into account the diversity of interests and values related to the general and public welfare in Northwestern Minnesota, she very much leads by example. She continually focuses on sound evaluation methods, a solid partnership approach and providing information and results that help us inform our partnership and improve our overall service delivery. - Colleen MacRae, Project Director, Stephen-Argyle Safe Schools Initiative

We are all rooting for you, Becky!

Posted by admin at 03:35 PM | Permalink | Comments (0) | TrackBacks (0)

A few weeks ago I wrote about evaluating programs designed to solve problems. In the blog I discussed logic models and how they can be used to understand a program, its activities and intended outcomes.

Shortly after writing the blog, the U.S. congress began exploring how to define outcomes for the war in Iraq. Setting aside opinions about the war, I found the debate fascinating, particularly because I had just been musing about logic models and how they help me think through problems. Could a logic model be used to define goals, activities and outcomes for the war? Could developing a logic model help two seemingly incompatible points of view come to agreement about the next steps for the war? Is a progress report a good way of measuring the success of the war?

Although clearly foreign policy in general and wars in particular are complicated by the unknown responses and actions of other players, I couldn’t help but begin diagramming my questions into a logic model.

I guess I am not immune to seeing things through the lens of my discipline!

I mentioned our work with Safe Schools/Healthy Students grantees in Spring Lake Park, Northwestern Minnesota, and a cross-site research project for most of Minnesota's grantees before. In the recently released RFP for new grants, applicants are required to submit a logic model of their program. Although competition this year is expected to be very strong (there will likely be close to 500 applicants for approximately 20-30 funded grants), a strong logic model can help guide the rest of the grant narrative. This approach worked for Performing Arts Workshop when applying for their U.S. Department of Education Arts in Education Model Development and Dissemination grant; they were able to include input from schools, artists and program staff in the goals, outcomes and measures for their ARISE project. Please feel free to contact us to talk about developing a logic model for a grant application.

Posted by admin at 04:34 PM | Permalink | Comments (0) | TrackBacks (0)

Our organizations, programs and services do not thrive in a vacuum; most collaborate with, or serve, other people and organizations. These other stakeholders can include clients, patients, board members, member organizations, or programs and organizations that contribute to the delivery of our goals. When conducting research or evaluation to inform decision-making, organization staff input is very valuable, but does not represent the whole picture; often, community stakeholders can give powerful insights on an organization’s effectiveness.

A good example of incorporating community insights in a project is the current strategic planning efforts being carried out by the Austin, Minnesota school district. Superintendent Candace Raskin approached the Improve Group about the district’s desire to develop a strategic planning process that would set a well-informed course for Austin schools for the next 3 to 5 years.

From the onset, the school district understood the value of making this process inclusive and soliciting the opinions and suggestions of not only the teachers, board and staff, but also parents, students, community organizations and other institutions providing education or serving youth in the community. Their opinions and suggestions help identify priorities and guide district administration as they make their decisions for the future.

Please click on this link to a recent article from the Post Bulletin of Rochester, MN to read a brief account of the current status of the Austin strategic planning process.
Survey: Austin residents are satisfied with schools' academics.

Posted by admin at 04:12 PM | Permalink | Comments (0) | TrackBacks (0)

When reading the newspaper or browsing the internet I tend to be drawn to articles in which someone is developing a solution to a widespread (or at least widely talked about) problem.

Recent stories that come to mind focused on faith-based financial education as a way to help individuals increase financial discipline, the recent announcement by the Broad and Gates foundations of a major investment in increasing the American dialog for education (and ultimately to ensure more students go on to postsecondary education) and the large investment by the Robert Wood Johnson Foundation to fight childhood obesity.

These stories interest me because I am fascinated with how people define problems or needs and then develop strategies to solve them. In our work with clients we often use an approach called logic modeling. (You can find somewhat academic background articles about logic models here and here). When we first suggest developing a logic model to a client, we try to emphasize that it isn't an intimidating process, although it can take considerable time and effort to get it right. We describe logic models as a simple, visual way to describe (1) what you hope to accomplish, (2) who or what is expected to change, (3) through what efforts and (4) why the effort is important. They can take many forms; we recently created a bulls eye-shaped logic model for City House that started out as a diagram of a person; another logic model of collaboration (see a blog about our presentation to Minnesota Superintendents) took a much more traditional flow-chart format.

Logic models get tweaked periodically as the environment and programs evolved. For example, in our Safe Schools/Healthy Students evaluation work with Spring Lake Park District 16 over a 4-year period, we revisited the logic model twice, with the final version most strongly representing the unique characteristics of the districts in the context of the federal program.

When I hear of new initiatives, abbreviated logic models are the easiest way for me to understand what is happening. Sometimes before I realize it, I have created a mental diagram of the changes a program is expecting and through what initiatives. I am not sure if this is common to other people who do evaluation work, but is the primary frame for the way I look at private and public efforts to improve the public good.

Posted by admin at 09:14 AM | Permalink | Comments (0) | TrackBacks (0)

State and Federal requirements to establish and maintain charter schools contain numerous requirements to plan to assess student achievement and report on outcomes. One of the six statutory purposes for charter schools from the Application to Create a Charter School in Minnesota and apply for a Federal Charter Schools Program Planning Grant is directly related to evaluation:

"Require the measurement of learning outcomes and create different and innovative forms of measuring outcomes.” *

Evaluation also plays another major role after a charter has been granted:

“I assure that the school will provide annually to the U.S. Secretary of Education and the Minnesota Department of Education such information as may be required to determine if the charter school is making satisfactory progress toward achieving the funded activities. This includes participation in any federal or state funded charter school evaluations or studies and compliance with all requests for information.
I assure that the school will cooperate with the U.S. Secretary of Education and the Minnesota Department of Education in evaluating the school’s program.”**

The Improve Group has proven experience providing educational planning, evaluation and reporting to charter schools, their sponsors and the Minnesota Department of education. We have worked directly with the charter school system where we supplied assistance that included:

• designing an evaluation plan and all measurement tools
• conducting an in-depth evaluation of a sample of charter schools
• evaluating data for all charter schools and comparing it to similar data for other public schools

We have also produced valid, compelling program reports that meet State and Federal requirements and can be used to start conversations with staff and Board members about school goals and accomplishments. If your organization is applying to form a new charter school or requesting on-going approval and you are interested in forming a relationship with an evaluator, please contact Susan Murphy at susanm@theimprovegroup.com to set up an appointment to discuss how the Improve Group can help.

For more information on the Minnesota Charter school system go to:
http://education.state.mn.us/MDE/Academic_Excellence/School_Choice/Public_School_Choice/Charter_Schools/index.html

*Charter School Application Narrative, Pg. 7, Section III. A.
*Charter School Application, Appendix 4, Assurances Statement, Pg. 20, Program Evaluation section.

Posted by admin at 03:42 PM | Permalink | Comments (0) | TrackBacks (0)

An article I read in the NY Times about the federally funded Study of Early Child Care and Youth Development (SECCYD) stated that keeping a preschooler in child day care a year or more increased the chances that the child would become more disruptive in class. I found even more interesting the study’s finding that this result held up regardless of the child’s sex or family income, and regardless of the quality of the day care center.

What intrigued me most about the article is the criticism that the researchers were unable to use a research method called “random assignment” in their study. Random assignment is a research method where a study’s participants are pooled into two groups, an experimental group and a control group. The experimental group receives the treatment that the study is measuring (in this case child day care). The control group receives no treatment (no child day care). The theory is any differences between the groups are due to treatment alone. Random assignment is widely used in studies with numerous participants and currently is widely used in federally funded studies.

After reading about this criticism and through my knowledge of random assignment-based research, a couple of points came to mind:

• Due to ethical issues such as denying children who otherwise qualify for day care services to create a control group, testing the effects of specific treatments on a certain population is very difficult to do because it may adversely affect children’s development process.

• Random assignment generally does not accurately measure the effects of cultural factors between the experiment and control groups in the results of the study.

• In long-term studies such as SECCYD, it is difficult to control for unanticipated changes in the conditions between the two groups, which potentially weakens the validity of the differences when the two groups are compared.

In short, while random assignment may seem like a straightforward way to do research, especially with large groups of participants, one must be aware of potential weaknesses of this method.

Posted by Guest author at 03:32 PM | Permalink | Comments (0) | TrackBacks (0)

Our current evaluation of Performing Arts Workshop's Artists-in-Schools program involves researching how the effects of the program differ for students in general and special education. A concern of ours is how to conduct our research in a way that identifies and accommodates the needs of students in special education without adding to the stigma of being in special education. According to the National Institute of Health, research involving children demands a particularly high level of care and consideration by investigators since they are not considered able to make informed choices independently, therefore exposure of children to more than minimal risks must be weighed carefully. Although we have conducted research and evaluation projects with children before, we wanted to be exceptionally careful working with students in special education, since they are even more vulnerable as research subjects. One of our particular concerns was that students in special education would be stigmatized in the classroom by completing surveys that were adapted to their needs or by having an adult assist them with their surveys. However, education specialists in the San Francisco Unified School District informed us that students in special education are already stigmatized in the classroom by having additional help and so our surveys would not significantly stigmatize them further. We have adapted our research methods for this project to minimize the risks to all student research subjects, but particularly those in special education:

• The student survey was adapted to gather appropriate information from students in general and special education on the same form.
• There is no text on any surveys completed by students or teachers that indicates our research is specifically about special education – this not only minimizes stigmatization risks for students, but also ensures that teachers are not actively comparing their general and special education students on their surveys.
• Classroom teachers go over the entire student survey as their classes complete it; special education students receive additional help at this time as needed.
• Teachers and parents do not identify their students as being in special education; instead the Improve Group tracks status by collecting behavior and education goals from students’ Individualized Education Plans.

Posted by Guest author at 10:10 AM | Permalink | Comments (0) | TrackBacks (0)

When conducting an evaluation, we are required to minimize the risks to participants; however we sometimes struggle to identify what those risks may be and how they relate to any potential benefits. Particularly when the study involves issues like the arts, academic achievement or interest in extra-curricular activities, the risks may seem very remote or negligible.

However, professional ethics demands that we fairly weight these risks and be honest with participants about what the risks may be. A training from the National Cancer Institute helped us to define these risks recently. Risks to individuals are classified as physical, psychological, social, legal, and economic. In the process of determining what constitutes a risk, only those risks that may result from the research, as distinguished from those associated with therapies participants would undergo even if not engaged in research, should be considered.

In our work, the most apparent risks are psychological or social. We must be careful that the participants in our evaluation studies do not suffer any social stigma from their participation. Also, we must be careful when wording questions and designing evaluation studies that the process is not traumatic in any way for participants.

Posted by admin at 02:41 PM | Permalink | Comments (0) | TrackBacks (0)

Collaboration is receiving increasing visibility in recent years; funders are requiring collaboration as a way to ensure services are not duplicated and address complex problems while local community agencies see collaboration as a way to broaden their reach.

Last fall we concluded fifteen months of original research into collaboration in five Minnesota communities. These communities were recipients of the federal Safe Schools/Healthy Students grant, and had school district, law enforcement and mental health partners. We had the opportunity to present lessons learned at the Minnesota Association of School Administrator's conference on Friday, March 15. Chief among our findings, and one that participants most related to, is that collaboration may take longer than "going it alone", but there is much greater potential for systemic change when working with a broad base of support. Similarly, it is important that a structure is well-defined, as are a broad vision and goals, but that individual partners have the opportunity to shape the specific steps.

See a .pdf version of our presentation here.

Posted by admin at 10:39 AM | Permalink | Comments (0) | TrackBacks (0)

Several of the arts educators we work with have mentioned over the years that No Child Left Behind (NCLB) was the first time the arts were recognized as one of five core subject areas (including in a recent interview of Perpich Center for Arts Education's Mike Hiatt). However, in a review of NCLB documents from the U.S. Department of Education, very few list core subject areas, and when they do, there are more than five core areas listed. The documents that refer to core academic ares indicate subjects in which teachers must be qualified, and these include English, reading or language arts, mathematics, science, foreign languages, civics and government, economics, arts, history, and geography.

Furthermore, some documents from the U.S. Department of Education suggest that the artse are viewed more as an enhancement than as a core subject. These include a letter from the Acting Assistant Secretary Ronald Tomalis, in which all arts activities not considered to be "the music" or "the arts" class are considered enhancements. This allows flexibility for schools in providing the arts (including bringing in artists as paraprofessionals into the classroom, the model used by another of our clients, Performing Arts Workshop), but also leads to challenges in truly integrating the arts as a core academic area.

A final weakness is that of the core subjects, only reading and math have assessment requirements, with science to be added in the coming year. States are required to test students in reading and math, and also subject themselves to peer review around reading and math implementation.

Posted by admin at 11:56 AM | Permalink | Comments (0) | TrackBacks (0)

Because we are on an email list, we have received several notices over the last few months that the U.S. Departments of Education, Health and Human Services and Justice will soon be opening a new Safe Schools/Healthy Students funding opportunity.

Some districts are already planning their applications. In the Richmond, Indiana area, five school districts within one county began a joint effort to seek the funding on Tuesday, March 6.

The grant application materials are expected to be available April 6 and close May 31. We worked with two grantees funded in different years; they had slightly different program requirements between the two funding cycles. We are looking forward to seeing what changes are in store for the next cycle and hoping to work with districts again to help evaluate funded programs.

Posted by admin at 09:51 PM | Permalink | Comments (0) | TrackBacks (0)

We thought we’d share news about an interesting analysis method we recently used…

Performing construct analysis is a method you can use when analyzing survey data. First you must build “constructs” from among your survey questions, or items. Each construct may refer directly back to an overarching goal of the program you are evaluating, such as “Improve school climate.” You simply select all the survey items used to measure improvements in school climate to create your construct. Next, to perform analysis on your constructs, the survey items within your construct must be “collapsed” or summed to create a new variable.

The guiding principal behind this method is that, essentially, any one item can not tell you the whole story on complex topics or concepts. By using multiple items to assess your goal, you place less pressure on any single survey item to answer your research question.

The Improve Group recently used this technique doing survey analysis for a client. This research project had a few characteristics that were especially conducive to this type of analysis: post/retrospective pre-test survey data; a survey asking girls about specific topics they had learned in the program; and items that specifically aligned with overall client goals.

In our case, we created two groups for each construct: “After” and “Before.” We then performed t-tests to determine whether there was statistical significance in the difference between after and before for each construct as a whole.

We had to keep in mind that the final difference scores (between “Before” and “After”) represent the mean of individual items. You may want to look at scores of individual items within a construct for consistency. If one or two items within a construct had much less or much greater change than the other items, it may skew the total mean score for the overall construct.

We would like to thank James Riedel at GSUSA for asking us to perform this type of analysis. In addition, we would like to thank Sanford Weisberg of the University of Minnesota’s Department of Statistics and Robert Eichinger of Cornferry International and Loeminger Enterprises who kindly and graciously provided us with advice and assistance on this statistical project.

Posted by Guest author at 01:11 PM | Permalink | Comments (0) | TrackBacks (0)