Tending to the Forest and the Trees in Survey Design

Michael W. Duttweiler
Assistant Director for Program Development and Accountability
Cornell Cooperative Extension
mwd1@cornell.edu

Monica Hargraves
Manager of Evaluation for Extension and Outreach
Cornell Office for Research on Evaluation
mjh51@cornell.edu

With many helpful resources available for designing survey instruments one might think that instrument design would be among the easiest parts of evaluation planning.  Yet even seasoned educators and evaluators routinely are surprised or disappointed by the data their carefully considered instruments yield.  There must be more to it.  What we present in this month’s set of four blog posts is a combination of steps we have found effective in sharpening data gathering.  There is nothing particularly novel in our approach; each of the steps likely will be familiar to you.  Rather, it is the disciplined application of the steps that we have found to improve the value of information gathered.  These posts are meant to share our experience and invite your observations and additional suggestions.

Each post will describe one of four phases in the design and revision process.  In general terms, they are:

  1. Developing a Precise Evaluation Purpose Statement and Evaluation Questions
  2. Identifying and Refining Survey Questions
  3. Applying Golden Rules for Instrument Design
  4. Testing, Monitoring and Revising

 

With that introduction, we first address the critical roles and value of developing an evaluation purpose statement and associated evaluation questions. The essence of phase 1 is that you must know precisely what questions you are trying to answer before developing the questions you need to ask.

In our parlance, an Evaluation Purpose Statement is a one-paragraph description of your evaluation effort.  It describes what is and is not being evaluated and the goal or purposes of the evaluation.  It sets boundaries by including a description of the program elements and time frame being considered, which audiences are being addressed, and which goals or objectives are of most interest. Equally important, it identifies major elements of the program which are not being assessed.

Example: The purpose of this evaluation is to assess the effectiveness of the 2012 Master Forest Owners Workshop in supporting and prompting volunteers to extend their knowledge to other forest owners in their local communities.  A secondary purpose is to provide documentation and assessment information for use by persons considering replicating the model with other forest owner groups.  Considerations include program structure and processes, curricular choices, and short-term impact assessment.  Other means of supporting forest management volunteers such as our newsletter and quarterly conference calls will not be assessed.  

In a collaborative evaluation setting (aren’t most?), it is essential that all key partners agree to the purpose statement.  Also important is to assure that “nice to know” purposes don’t creep in – the statement should convey your essential intent in conducting the evaluation. Having the agreed-upon Purpose Statement written down is very helpful later on, for keeping the evaluation on target.

With the purpose statement in hand, you can move on to identifying evaluation questions.  The core Evaluation Questions (EQs) are the small number of essential questions that must be answered in order to meet your evaluation purposes. Continuing the Example above:

EQ1) To what extent did the forest management volunteers attending the 2012 Master Forest Owners Workshop extend knowledge gained at the workshop in their communities?

EQ2) How well did the workshop prepare the volunteers for this role? 

EQ3) What critical information will be needed by those looking to replicate the workshop?

The purpose statement and core evaluation questions are the springboards for the evaluation planning process, allowing you to consider who has the required information, how it might be collected, and what methods and instruments will be employed.  Both should be revisited and refined throughout the design process.

Our next post explores the importance of refining your initial evaluation questions.

 

This article (Tending to the Forest and the Trees in Survey Design) was originally published Friday, October 5, 2012 on the Evaluation Community of Practice blog, a part of eXtension.   This work is licensed under a Creative Commons Attribution 3.0 Unported License.