Interpreting Survey Results

After you have collected an adequate number of responses to your survey, it's finally time to take a look at the data to see what they say. Gaining an accurate understanding of your survey results is the final step in the survey process (excluding any action you might take based on those results).

These tips are designed for employee satisfaction surveys or employee engagement surveys, but the same principles can be used for other types of surveys as well.


What's your n?

Before digging too far into your results, it's important to be sure you have an adequate number of responses. Whether you are looking at results from "all respondents" or just a demographic slice, you need to be sure enough employees responded to make the survey results statistically meaningful. How many respondents do you need? There is no hard and fast rule, but more is better. Our article on random sampling talks about sample size and statistical accuracy. In practical terms, realize that if you have a smaller number of respondents, you need much stronger results in order to draw conclusions from the numbers. For example, if you have just 10 respondents and they all said "strongly disagree" then you can probably trust that, but if just 7 out of 10 said "strongly disagree" then you might want to collect more data to be sure there is a trend there. On the other hand, if you have 1000 respondents and 700 of them said "strongly disagree" you can be pretty sure that this result is meaningful. This concept is especially important to remember as you look at results for different employee demographic groups. Pay attention to cases where your n is low and realize that the conclusions you draw from the results for these smaller groups of employees are less certain than for larger groups. Who's your n? If you included demographic questions in your survey, then you may have a pretty good idea of who your respondents are, but keep in mind ways in which the respondents might not represent all of the people in your "population". For example, if you are surveying employees and your survey is conducted online, you are not likely to get much feedback from employees who don't use a computer. Every situation is unique and you should spend a bit of time thinking about the kinds of people who might be under represented in your survey results.

What percentage of employees completed the survey? In addition to the actual number of responses, it is also important to look at the response rate. Employee satisfaction surveys typically have response rates in the 70% - 80% range (or higher). If your response rate was less than 65% - 70%, this could be an indication of problems with the survey process itself (e.g. poor communication), or it could indicate a problem with employee attitudes and levels of engagement. Company culture can have an impact on employee survey response rates. For example, if there is a low level of trust among employees, they may not feel comfortable sharing their opinions in a survey. If employee surveys were conducted in the past, but results were not shared and/or no action was taken to correct problems, employees may feel that completing the survey is a waste of their time.

Garbage In - Garbage Out

Your results will only be as good as the questions that you asked people. If your questions were poorly worded, you are probably going to find that your data are not very useful. Two common examples of this to be aware of:

1. The "easy" question - if a question is too softly worded or has an obvious answer, you will find that almost all respondents answered with the same response. This means your question has not effectively distinguished what it was intended to distinguish.
2. The "confusing" question - there are many forms of confusing questions, but they all generally yield similar response patterns - you will see an unusually high number of "unable to rate" responses as well as a larger than average spread of responses in the frequency distribution. What you are seeing in these patterns is that people just did not understand the question or that different people interpreted the question differently.

1. If your employee survey questions are benchmarked, focus on the benchmark scores. Don't ignore the raw (average) scores, but the benchmark scores will give you the best indication of how good or bad your results are. (more on benchmark scores)
2. Overall Average Scores (or benchmark scores if available) - high or low? This is the obvious first place to start. Very high or very low scores mean either that you are doing really well or really poorly in an area - or they might mean that the survey question is poorly worded.
3. Relative Scores - how do the scores on each item compare to the scores on similarly structured items in your survey?
4. Standard Deviations and Frequency Distributions - a low standard deviation means employees generally had a higher level of agreement in how they responded. Higher standard deviations mean less agreement. The frequency distribution will help you get a better idea of what is happening here. One pattern in particular to look for is a bi-modal distribution where there are clusters of responses on both the high and low ends of the response spectrum. These items might show up as having a moderate overall average score, thus looking unremarkable from that perspective, but the bi-modal distribution might mean that there are two different demographic subgroups who had very different responses.
5. Look at the results for the different demographic subgroups, especially focusing on the items where you had interesting things happening in the frequency distributions.
6. If you are serious about understanding your numeric data, you should also perform some more advanced statistical analyses. In particular, a correlation matrix will often reveal where different questions in your survey have relationships to one another. If you are not familiar with these types of statistical analysis, you should work with somebody who understands how to use them. (see also - Identifying Employee Engagement Priorities)

1. Start by reading through all the comments. Get a feeling for what people are saying.
2. Now go back and categorize the comments into different areas. The categories you put them into are up to you to determine, but having read through all the comments, you should have an idea of where to begin. Don't be concerned if there are some comments left over that don't fit into any category.
3. Now look at each category separately. How many unique comments are in each? How detailed are those comments? How strongly are they stated? At this point, you should be able to identify which categories are more important and which are less important. It's not an exact process, but patterns almost always emerge if you have enough response data to work with. If you find that you have several categories which seem to be equally important, that's fine too.
4. Refer back to the numeric results to identify any survey questions that address the same topics as the comments. Look for links between the numeric results and the comments. If you identified a theme in the comments (positive or negative), is this supported by the numeric results?
5. If your survey included demographic questions, look at the different subgroups to see if any relationships emerge between demographic groups and categories of comments. This can be a time consuming process, but the outcome is often worth the effort.
6. You can also work from the numeric results down to the comments - for example, identify a group of employees that gave low scores in some area, and see if those low scores are reflected in the comments.