Interpreting Survey Data

After you have collected an adequate number of responses to your survey, it's finally time to take a look at the data to see what they say. Gaining an accurate understanding of your survey results is the final step in the survey process (excluding any action you might take based on those results).

What's your n?

Before digging too far into your results, it's very important to be sure you have an adequate number of responses. Whether you are looking at results from "all respondents" or just a demographic slice, you need to be sure there are enough respondents to make the results statistically meaningful. How many respondents do you need? There is no hard and fast rule, but more is better. Our article on Random Sampling talks about sample size and statistical accuracy. In practical terms, realize that if you have a smaller number of respondents, you need much stronger results in order to draw conclusions from the numbers. If you have just 10 respondents and they all said "strongly disagree" then you can probably trust that, but if just 7 out of 10 said "strongly disagree" then you might want to collect more data to be sure there is a trend there. On the other hand, if you have 1000 respondents and 700 of them said "strongly disagree" you can be pretty sure that this result is meaningful.

This concept is especially important to remember as you look at results for different demographic subgroups. Pay attention to cases where your n is low and realize that the conclusions you draw from the results for these smaller groups are less certain than for larger groups.


Who's your n?

If you included demographic questions in your survey, then you may have a pretty good idea of who your respondents are, but keep in mind the different ways in which the respondents might not represent all of the people in your "population". For example, if you are surveying customers and your survey is internet-based, you are not likely to get much feedback from people who don't use computers. Every situation is unique and you should spend a bit of time thinking about the kinds of people who might be under represented in your survey results.


Garbage In - Garbage Out

Your results will only be as good as the questions that you asked people. If your questions were poorly worded, you are probably going to find that your data are not very useful. Two common examples of this to be aware of:

1. The "easy" question - if a question is too softly worded or has an obvious answer, you will find that almost all respondents answered with the same response. This means your question has not effectly distinguished what it was intended to distinguish.
2. The "confusing" question - there are so many forms of confusing question, but they generally all yield similar response patterns - you will see an unusually high number of "unable to rate" responses as well a larger than average spread of responses in the frequency distribution. What you are seeing in this is that people just did not understand the question or that different people interpreted the question differently.

1. Overall Average Scores - high or low? This is the obvious first place to start. Very high or very low scores mean either that you are doing really well or really poorly in an area - or they might mean that the question is poorly worded.
2. Relative Scores - how do the scores on each item compare to the scores on similar items in your survey?
3. Standard Deviations and Frequency Distributions - A low standard deviation means people generally had a higher level of agreement in how they respondened. Higher standard deviations mean less agreement. The frequency distribution will help you get a better idea of what is happening here. One pattern in particular to look for is a bi-modal distribution where there are clusters of responses on both the high and low ends of the response spectrum. These items might show up as having an overall average score, thus looking unremarkable from that perspective, but the bi-modal distribution might mean that there are two different demographic groups who had very different responses.
4. Look at the results for the different demographic subgroups, especially focusing on the items where you had interesting things happening in the frequency distributions.
5. If you are serious about understanding your numeric data, you should also perform some more advanced statistical analyses. In particular, a correlation matrix will often reveal where different questions in your survey have relationships to one another. If you are not familiar with these types of statistical analysis, you should work with somebody who understands how to apply them and interpret the results. (see also - Measuring Importance in Satisfaction Surveys)

1. Start by reading through all the comments. Get a feeling for what people are saying.
2. Now go back and categorize the comments into different areas. The categories you put them into are up to you, but after having read through all the comments, you should have an idea of where to begin. Do your best to categories all the comments, but don't be too concerned if you have a handful left over at the end which don't fit in any category.
3. Now look at each category separately. How many unique comments are in each? How detailed are those comments? How strongly are they stated? At this point, you should be able to identify which categories are more important and which are less important. It's not an exact process, but patterns almost always emerge if you have enough response data to work with. If you find that you have several categories which seem to be equally important, that's fine too.
4. Now, if your survey included demographic questions, look at the different subgroups to see if any relationships emerge between demographic groups and categories of comments. This might seem like a time consuming process, but the outcome will be worth the effort.

---

360 Degree Feedback System

Employee Satisfaction Survey

info@CustomInsight.com       750 Arrowhead Dr.       Carson City, NV 89706        650-577-9604
Company Info   |   privacy policy