 |
|
 |
Issue 17: October 2008 Client Update: Chicago Opera Theater Client Spotlight: Chicago Opera Theater In fall 2007, Kara Kane of Chicago Opera Theater (COT) contacted the Improve Group for help documenting evidence of learning in its Opera for All education program. She wanted to explore how the program helped students understand different cultures, express themselves in new ways, make connections between their lives and art forms and demystify opera. Because Chicago Opera Theater had a very small evaluation budget, Kara Kane and Leah Goldstein Moses developed a coaching model in which Kara identified key questions and potential sources of information, and Leah made suggestions about collecting data, interpreting results, and using findings for program development. Leah recently talked with Kara about Opera for All and how the evaluation is supporting its work. LGM: What aspects of the evaluation were most useful? KK: There were two things that proved particularly useful in this collaboration; First was hearing about Improve Group’s work with other organizations asking artists, classroom teachers and students to share information. It helped us frame our evaluations and also gave us confidence in our approach focused on creating meaningful evaluations without overburdening everyone. Second, we were able to develop some new ways of talking with our artists about evidence of learning, and how this is tied to their curriculum and lesson plans. I had a tough time finding the language to tangibly map COT’s program goals to our program design in a way that made sense to our artists – we are now treating our goals more like a logic model that guides our program. LGM: What were some of the things you learned in your evaluation work? KK: We learned that our biggest outcomes were demystifying opera, building understanding of other cultures, and helping students express themselves in new ways. Our year’s curriculum was centered on our production of A Flowering Tree, which is a story from South India. In future years, we might not have as strong of an emphasis on culture, and find that other outcomes take precedence. However, I learned how to more aptly identify our core goals, and then drape a couple key evaluation questions around measuring these goals. LGM: What do you hope to do next? KK: By working with the Improve Group we developed a strong evaluation system that I feel comfortable implementing. The data that we get will be used to begin discussions with our artists and school partners during annual program reflection periods to see what is working, what we could tweak to better achieve our goals, and what new opportunities we could leverage to enhance our program.
LGM: What were some of the challenges involved with developing this evaluation? How did you overcome them? KK: Probably the biggest challenge was in helping our artistic staff to link their efforts with outcomes. In the arts, it is pretty easy to hear if someone is hitting a note, or to see if someone understands where they should be on a stage. What is difficult is thinking about how those relate to our broader goals; what does that “evidence” represent? We spent a lot of time talking about the activities and lessons about our program and how they relate to goals. Our artistic staff is very enthusiastic this year about organizing their full year’s worth of lessons so they build skills related to our goals. LGM: What are you looking forward to in the next few years? KK: Through this process of evaluation we’re identifying ways to codify the programs and processes, make it more standard across schools, and also enhance the program to make it richer and more interesting not only for the students but also for the artists and teachers. Each year the opera focus changes, but we are looking to identify reusable components so we’re not reinventing the program every year. As a result, we’ll be able to better articulate our program so we can expand the Opera for All program to serve more students.
|
|
Featured Article: Pictorial Evaluation Methods In November, Research Managers Deborah Mattila and Rebecca Stewart will give a presentation about the use of pictures in evaluation instruments at the annual American Evaluation Association Conference in Denver, Colorado. The presentation will draw on ongoing and recent work with two Improve Group clients: City House (Minneapolis, MN) and the Performing Arts Workshop (San Francisco, CA). For Performing Arts Workshop, students drew pictures on their surveys to depict themselves performing with their friends. These drawings were analyzed to find out what students consider as performing arts and how they feel while performing. For City House, participants coded their feelings using certain icons or pictures in diaries or in collages (the Image Grouping Method© of evaluation developed by the Improve Group). Analysis of these instruments provides the opportunity to understand the importance of internal or spiritual change to participants, track what internal or external changes that participants experience over time, and to look for any correlations between internal (spiritual) and external (behavioral) changes. The Improve Group selected pictorial methods for these projects for both similar and different reasons. Both projects sought to provide an alternative, nonverbal method for the special populations participating in the clients’ programs to express themselves in the evaluation process. The Performing Arts Workshop program works with 3rd through 5th grade students in Special Education classrooms and settings. Students who participate in the program and evaluation have varying cognitive ability and expression skills. It is important to the evaluation for most students to contribute to at least one part of a student survey. A student survey was developed with check-one, write-in and pictorial questions. Improve Group evaluators consulted with Performing Arts Workshop staff and Special Education educators to design the survey with utmost accessibility to a wide range of students.
For City House, the organization’s participants are all adults that have varying levels of literacy skills. The pictorial instruments also brought other benefits to the evaluation. They were colorful and different from the normal “survey” that many of the organization’s participants are asked to complete by other organizations. The pictorial instruments were designed to attract participants’ interest in the tools and the evaluation. In addition, the hope was that, if the tools were more accessible and engaging, participants may be more likely to take an active role in the evaluation, providing their own observations about changes they may be experiencing. Indeed, observations during the pilot testing of instruments showed participants talking and sharing with one another about the pictures and their feelings during the Image Grouping exercise. Finally, by using a non-traditional layout, the Improve Group also hoped to discourage participants from selecting what they anticipate to be the “right” answers to questions; City House had observed that participants sometimes have done this out of a desire to please service providers in traditional survey instruments.
|
|
|
New to the Improve Group Team by Younkyeong Nam As an intern at the Improve Group, I am so happy to have such a good opportunity to learn about evaluation research. Through my career in secondary science and as a researcher, it was a long journey for me to come here. After I finished my first semester of a doctoral program in South Korea, I was still not sure what I could accomplish in my field of interest - earth science education. I had a strong belief that earth science is not an isolated discipline but an integrated science that teaches students to view the natural world holistically. With a vague notion of finding a theoretical framework to support my ideas and scholars who share similar beliefs about earth science, I decided to study abroad. It was not an easy decision to give up the regular income as a public school teacher, and was made more difficult as, a second language user and as a mother of an 18-month old baby. However, four years ago, with my husband’s support, I started my first long journey to a strange land (North America) to find the way to communicate my beliefs about earth science education to the world. Throughout my studies in the University of Minnesota doctoral program, helping science teachers to see the Earth as a holistic system has become my life goal as a scholar. During the doctoral program, I have been working as an evaluator of a teacher professional program of “Earth Science System for K-9 Teachers” for three years. In this program, I have been helping to design and conduct two evaluation studies that give me an opportunity to interview urban teachers and observe their classrooms. Through the process of the evaluation, I found that my evaluation study could improve the teacher development program and help ensure it delivered relevant earth science knowledge to in-service teachers. Therefore, I wanted to learn more about evaluation, collecting data that is more valid, and analysis that is more reliable. My desire to learn about evaluation leads me here. Even though I may have quite a different background than other Improve Group members, I believe we have the same enthusiasm for improving and supporting programs and organizations through evaluation. I am happy to be here with the best evaluation team and am looking forward to working with them.
|
|
Like many of the other presenters at the conference, Deborah and Rebecca will describe their experience and observations in using pictorial instruments, and then engage in a discussion with other evaluators about the promises and pitfalls of using this methodology in evaluation. They will discuss the use of pictures in evaluation and other fields such as Art Therapy. For instance, Rebecca drew heavily upon the 