By Teresa McCoy, Assistant Director of Evaluation and Assessment, University of Maryland Extension
Did you ever have a pair of old shoes or jeans that you just could not bear to part with—no matter how tattered or worn out? I know that I have and bet you have, too. At the 2014 American Evaluation Association annual conference in Denver, I attended a session given by Michael Quinn Patton entitled, “Lack-of-vision evaluation ideas that should be retired to realize visionary evaluation for a sustainable, equitable future” where he presented his suggestions for evaluation ideas that we should discard (like those old shoes or jeans).
If you have heard Michael speak, you know he generates thought-provoking engagement with his audience and this session title certainly sparked my interest. I wasn’t disappointed. In his new edition of Qualitative Research Evaluation Methods: Integrating Theory and Practice (4th edition) (due out in November and coming in at length of over 800 pages), he discusses the 10 outdated evaluation ideas and/or approaches that he thinks should be retired, including such classics as anecdotal, gold standard, best practices, and site visits. You will have to buy the book to see all of his explanations about the practices that are outdated.
After presenting his ideas, Michael asked the audience for further nominations. There were lots of suggestions! I had my own: cost-benefit. I think that is a term borrowed from the business world that does not translate to the not-for-profit or government sectors. For example, how I do calculate the costs of not knowing how to prepare healthy meals for children versus the benefits? Is there a way we can calculate the costs of a bridge, such as the Chesapeake Bay Bridge in Maryland, versus the benefits? We know the costs to build the Bay Bridge and to maintain it each year, but what about the benefits? When I drive over the Bay Bridge, I am amazed at the beauty of the Chesapeake Bay. I benefit by being able to see my Extension colleagues on the Shore within one-to-two hours. However, what is that benefit actually worth?
With costs and benefits, the question has to be raised about costs and benefits to whom? The State of Maryland benefits from the tolls I pay each time I cross the bridge. Yet, the State of Maryland incurs a cost in highway and bridge maintenance, air pollution, and disruptions to the ecology of the Bay because I drive across to the Eastern Shore.
Another nomination from an evaluation expert in the room was the idea of logic models. This person suggested we move away from the term logic model to that of program map. I agree. The best way to clear a room of Extension people is to say, “I’m here from the Evaluation Department to teach you about logic models.” I often start trainings with that line and always get a laugh. In my practice, I have moved away from teaching logic models to teaching program theory and program maps. I advise people to forget about the logic model form and use the tool that works best for them to figure out what outcomes their program is designed to accomplish. I showed an image of a tree (roots, trunk, limbs, leaves) logic model to a group and a woman said, “OK, I understand that now. I wish someone would have told me this earlier.”
I would like to hear from other Extension evaluators what old ideas and approaches you think we should leave behind like those old shoes and jeans. Perhaps this discussion could help us move our practices and our profession ahead in the next few years.