Survey Question Composition 101

Follow these basic guidelines and your survey items should yield useful data.

Quantitative (Numeric) Data

1. Working from the outline that you created (see #3 above under "What do you want to know?"), break down each issue or detail into a single coherent, concrete thought or concept. Look for items in your outline that might be dealing with two or more different ideas or concepts. Split these up into separate lines or bullet points. Look for items that are vague or abstract and consider how they might be made more concrete.
2. Decide what kind of rating labels you want to work with - for example, you might be measuring "level of satisfaction" (5-point scale with responses ranging from "very satisfied" to "very dissatisfied") or you might be measuring "level of agreement" (responses ranging from "strongly agree" to "strongly disagree"). Often, it does not matter so much which labels you choose. What matters is that you use them consistently in the survey and phrase all items work with your chosen labels. It is OK have different rating labels for different sections of your survey, but keep changes to a minimum and be sure you keep the transitions clear in your own mind.
3. Keeping your rating labels in mind, turn each line or bullet point in your outline into a survey item. Do this by re-phrasing each line in the form of a question or statement that works with the rating labels.
4. At this point, you have your initial set of survey items, but your work is not done yet. Chances are, your survey is much too long at this point. If your survey is too long, people will not take the time to fill it out. It is now time to start looking for ways to shorten the survey. At the same time, you do not want to drop items that will get you the information you need. The balance between shortening the survey and making sure you get the data you need can be a tricky one. Ultimately, you must make a series of judgement calls with regard to these issues, but stick to this basic rule for each and every item - ask yourself what exactly what kind of information each and every question will give you that you would not have if you get rid of the question. If you cannot come up with a concrete answer, you should drop the item. Generally, you should not keep items in your survey simply because you think they might come in handy at some point.
5. After you have finished shortening the survey, look for ways to make each item more precise. Chances are, many of your items are suffering in this area for any of the following reasons.
   • Complex wording or structure - compound sentences, complex vocabulary words, unusual terminology, idiomatic expressions - all these things will confuse some people and make your data less precise. Use simple sentences and vocabulary appropriate to your audience. This is especially important if any portion of your respondents might not be native English speakers (or native speakers of the language your survey is written in).
   • Vague or overly general questions - Are any of your questions so broad that they will not give you specific, actionalable information? "Overall satisfaction" type questions often fall into this category. If you ask yourself what you will learn from the responses, positive or negative, from each question, you will know whether they are too vague.
   • Items that could be misinterpreted - For each survey item, ask youself whether there are any ways in which it might be misinterpreted. Many words have different meanings to different people. References to imprecise concepts such as time and distance are subjective.
   • Questions with a "right" answer - It is an easy and common mistake to write items that have a socially "correct" or desirable answer. Also be sure you do not lead your respondents to answer in a particular way by making them think you "want" them to provide a certain response. Your items need to be presented neutrally.
   • Double-Barreled Items - These are items that ask about more than one thing. If you see the word "and" or "or" in your item, chances are it is double-barreled. Data from items like this are useless because you have no way of knowing what part of the question each respondent was thinking of when responding.

1. If possible, consider conducting a pilot survey or gather a focus group to identify potential areas in which qualitative data might be most beneficial. A pilot survey might simply be your complete survey sent out to 10% of the respondent group. Based on the responses you receive, you might notice some problem areas where more information from respondents would be quite useful.
2. When composing qualitative items, the same rules apply as above, but you need to also be sure the questions will provoke respondents to provide insightful information. The worst kinds of qualitative items are those that ask a "yes/no" question or those that invite a specific (and often brief) answer. If you give respondents an easy out like this, they will most often take it rather than take it upon themselves to provide more a more detailed or involved response than you asked for.
3. The flip-side of the "yes/no" item is also a problem. If you are too broad in your qualitative items, respondents will not provide the kind of specific information that would be most useful to you. Finding the right balance between these two extremes is the best approach. Do not be afraid to guide respondents with some basic triggers to get their thoughts going, but do not be too specific with these triggers either.
4. The best test to see whether a qualitative item is well written is to answer the question yourself - read the question as if you were a survey respondent and consider whether it allows you provide a one or two word response. Consider also whether the question is too vague or general. If it falls somewhere in between and invites you to give the kind of response that you are looking for, chances are the item will work for your survey.
5. Don't overdo it. You will get more thorough responses from a few well-written qualitative items than you will from many such items, regardless of how well-written they are. It takes time for respondents to answer qualitative items and they will quickly grow tired of doing so and skip over such items or worse yet, not complete the survey at all.

---

360 Degree Feedback System

Employee Satisfaction Survey

info@CustomInsight.com       750 Arrowhead Dr.       Carson City, NV 89706        650-577-9604
Company Info   |   privacy policy