AEA2014: Evaluation Ideas that are Ready to Retire

By Teresa McCoyAssistant Director of Evaluation and Assessment, University of Maryland Extension

Did you ever have a pair of old shoes or jeans that you just could not bear to part with—no matter how tattered or worn out? I know that I have and bet you have, too. At the 2014 American Evaluation Association annual conference in Denver, I attended a session given by Michael Quinn Patton entitled, “Lack-of-vision evaluation ideas that should be retired to realize visionary evaluation for a sustainable, equitable future” where he presented his suggestions for evaluation ideas that we should discard (like those old shoes or jeans).

If you have heard Michael speak, you know he generates thought-provoking engagement with his audience and this session title certainly sparked my interest. I wasn’t disappointed. In his new edition of Qualitative Research Evaluation Methods: Integrating Theory and Practice (4th edition) (due out in November and coming in at length of over 800 pages), he discusses the 10 outdated evaluation ideas and/or approaches that he thinks should be retired, including such classics as anecdotal, gold standard, best practices, and site visits.  You will have to buy the book to see all of his explanations about the practices that are outdated.

After presenting his ideas, Michael asked the audience for further nominations. There were lots of suggestions! I had my own: cost-benefit. I think that is a term borrowed from the business world that does not translate to the not-for-profit or government sectors. For example, how I do calculate the costs of not knowing how to prepare healthy meals for children versus the benefits? Is there a way we can calculate the costs of a bridge, such as the Chesapeake Bay Bridge in Maryland, versus the benefits? We know the costs to build the Bay Bridge and to maintain it each year, but what about the benefits? When I drive over the Bay Bridge, I am amazed at the beauty of the Chesapeake Bay. I benefit by being able to see my Extension colleagues on the Shore within one-to-two hours. However, what is that benefit actually worth?

With costs and benefits, the question has to be raised about costs and benefits to whom? The State of Maryland benefits from the tolls I pay each time I cross the bridge. Yet, the State of Maryland incurs a cost in highway and bridge maintenance, air pollution, and disruptions to the ecology of the Bay because I drive across to the Eastern Shore.

Another nomination from an evaluation expert in the room was the idea of logic models. This person suggested we move away from the term logic model to that of program map.  I agree. The best way to clear a room of Extension people is to say, “I’m here from the Evaluation Department to teach you about logic models.” I often start trainings with that line and always get a laugh. In my practice, I have moved away from teaching logic models to teaching program theory and program maps. I advise people to forget about the logic model form and use the tool that works best for them to figure out what outcomes their program is designed to accomplish. I showed an image of a tree (roots, trunk, limbs, leaves) logic model to a group and a woman said, “OK, I understand that now. I wish someone would have told me this earlier.”

I would like to hear from other Extension evaluators what old ideas and approaches you think we should leave behind like those old shoes and jeans. Perhaps this discussion could help us move our practices and our profession ahead in the next few years.

Upcoming Series: Reflections on the 2014 American Evaluation Association Conference

Hello! I am Tom Archibald, Assistant Professor and Extension Specialist with the Department of Agricultural, Leadership, and Community Education at Virginia Tech; I have the distinct honor of serving as the new coordinator for this blog.

My goal is to continue providing excellent, relevant evaluation information and insights to you via this blog, just as it has done in the past. Please post comments to suggest topics that you would like to see covered here. There are expert evaluators, from Extension and from elsewhere in the broader field of evaluation, who would love to share their thoughts on whatever evaluation topic you find most relevant to you.

To begin, I will be bring you a series of reflections from Extension evaluators who recently returned from the annual conference of the American Evaluation Association (AEA). The AEA conference is a wonderful opportunity to learn about and reflect on evaluation. My hope is that, by reading these contributors thoughts on and lessons learned from the AEA conference, the benefits of this excellent learning opportunity can be shared more widely.

In this series, there will be new posts every two or three days, providing you something of a virtual visit to the EAE conference. You will hear from:

  • Teresa McCoy (Assistant Director of Evaluation and Assessment, University of Maryland Extension) on a session with Michael Quinn Patton on evaluation ideas that should be retired
  • Brigitte Scott (Evaluation and Research Specialist for the Military Families Learning Network) on being a new evaluator and a new participant in AEA, and on the notion of being an “internal” or “external” evaluator
  • Pennie Crinion (Director of Program Planning and Evaluation, University of Illinois Extension) on the benefits of joining AEA, and on this year’s conference theme, “Visionary Evaluation for a Sustainable, Equitable Future”
  • Ben Silliman (Extension Specialist and Professor of Youth, Family, and Community Sciences at North Carolina State University) on “right-sized” evaluation

I hope you enjoy and learn from these perspectives on the annual evaluation conference! Remember, please post comments to let us know what topics you’d like to see covered here in the future.