Survey Design: Golden Rules of Survey Development

Monica Hargraves
Manager of Evaluation for Extension and Outreach
Cornell Office for Research on Evaluation
mjh51@cornell.edu

Ok, we are FINALLY going to talk about designing surveys.  Just to be clear: the principles discussed here also apply to other types of measurement such as focus group protocols and interview questions. They are relevant whether you are designing an instrument from scratch, or adapting an existing instrument.

There are many good resources for instrument development.  For good overviews of surveys and NON-SURVEY options, see Unit 5 of University of Wisconsin Extension’s “Building Capacity in Evaluating Outcomes”.

Inspiration for the “Golden Rules” presented here comes from various professional sources, but also from personal frustration with the mixed quality and sheer number of surveys we encounter these days. Car dealerships, grocery stores, hotels – everyone asks for feedback these days. Survey fatigue is real, and requires us to be even more mindful in our work.

With lots of technical guidance available, it can be useful to have a short and easier-to-remember list to start from. Here are my boiled-down Golden Rules, with elaboration below:

Respect your respondent

Mind your “EQs” (evaluation questions)

Look ahead (to data management and analysis)

Pilot Test!

 Respect your respondent

  • Use clear, well-worded questions without jargon
  • Avoid double-barreled questions
  • Indicate what type of response you are looking for (if you needs answers in years, say so)
  • Make sure response options cover all possibilities (and anticipate diversity in participants’ potential responses!)
  • Be sensitive to whether the information you’re asking for is readily at hand, or will take time to look up
  • DON’T ask anything you don’t need to
  • Ask first, thank in advance, thank at the end
  • Explain how you will handle and use their input
  • Give them someone to contact
  • Be culturally thoughtful, and sensitive about what could be sensitive
  • Go through the IRB (Human Subjects Review)!

These pointers are not just matters of courtesy – falling short will affect the completeness and quality of your data.

Mind your “EQs” (evaluation questions)

  • Match each survey question to one or more EQs. If some don’t match, revise or delete
  • Assemble all the survey questions associated with each EQ and make sure you will be getting all the info yo
  • u need to answer the EQ
  • Make sure survey items are phrased in a way that will work for your EQ. (Beware of Y/N questions!)

These pointers are to help ensure you’ll get the data you need. Yes/No survey questions can be valuable, but might not work well if you are trying to assess something that might have changed incrementally.  Consider using “To what extent did you …” instead of “Did you …”, because the former might capture small changes that your program did achieve, which would have been lost if respondents were only able to say yes or no.

Look ahead (to data management and analysis)

  • What kind of data will you have?
  • What form will the answers be in, and will you be able to add/average/group/test them as needed?
  • Do you want an odd or even number of categories in a scaled response question?
  • Are you putting open-ended and closed-ended questions to their best use?
  • Do the response categories match the question?
  • Do multiple choice options cover the information you will need?
  • Will you be able to defend your results against claims of “bias” or “leading questions”?

It really pays to “think forward” when you’ve drafted your survey, to make sure that you’ll be able to use the data.

The final Golden Rule, “Pilot Test!”, is the subject of next week’s blog.

Here are two versions of a “Checklist for Newly Developed Surveys” that may be helpful for refining a newly-developed survey. (If prompted for login for either file, just click cancel and the file should appear.)

Microsoft Word Version with Protected Fields for Data Entry (DOCX)

Adobe Acrobat Version (PDF)