Dangerous Interpretations: Improper Surveys Methods = Invalid Results
by Dr. Jodie Monger

Having access to customer feedback makes you feel empowered, even invincible. Decisions are made based on survey results - as they should be. You are doing all the right things by asking for customer feedback, interpreting it, and managing to it. You can identify the correct causes for aspects of your results. You make a change and see the expected increase in caller satisfaction scores. Life is good.

How, then, do you explain survey results that are disconnected from the reality within your contact center? Survey results that do not assist you in uncovering reasons for the customer perception are most likely contaminated with bias. Biased survey results are very dangerous - very, very dangerous - because you are making operational and personnel decisions with flawed information.

Imagine taking a road trip this weekend with your only guide being a map that was produced in 1952. You may arrive at your destination but it could be mostly luck. While you did finally arrive, you certainly didn't get there in the most efficient and effective manner. For all you knew with this guide, the route was absolutely correct. Walking around with biased survey results may lead you down the wrong path, at great expense, and you will do it with utmost confidence which makes the error one that you will spend much time justifying the actions you took.

Why might your customer feedback results be biased?

  • The questions you ask. Too often survey questions are leading, worded incorrectly or contain multiple concepts. If your survey design is poor, it is better to not measure. In this case, no roadmap is better than a defective one.

  • The methodology you are using, specifically follow up telephone surveys or mail surveys. With the technology in call centers, there is no reason to utilize mail surveys to measure caller satisfaction. To reduce bias, you need to close the gap between the service experience and evaluation of that experience. If the gap is too large, the customer does not remember accurately and the CSRs will discount the feedback as "it must belong to someone else". Follow up surveys also have a gap, although not always as large, but the cost to implement such a program leads to the next problem on list.

  • Small sample size. You need to be able to generalize your sample to the entire call population, so it cannot be too small. The budgetary constraint of the follow up telephone surveys often times creates small sample sizes. You can collect more data for less by using a Real-Time Customer Feedback approach.

  • Incorrect statistical analysis or not enough analysis with only a focus on total scores. Looking only at the average score for each question is not enough. In order to manage to the information, you need to know what is driving caller satisfaction, how different types/levels of customers perceive service, how your evaluations track with operational metrics. There are many opportunities to produce the information you as a manager need to make the critical decisions.

There is nothing more powerful to represent the customers' perception of your service delivery than a continuous, sound piece of caller satisfaction research. When you use an effective methodology where callers immediately evaluate the service, ask succinct behaviorally anchored and non-leading questions, have an adequate sample size so your results can be generalized to the entire call population and produce actionable results, then LIFE IS GOOD !!

Dr. Jodie Monger, PhD is the President of Customer Relationship Metrics, LC (www.Metrics.net). Prior to joining Metrics, she was the founding Associate Director of Purdue University's Center for Customer-Driven Quality. Her expertise is working with Fortune 1000 companies to help them quantify the Voice of their Customer.

Many more articles in Performance Improvement in The CEO Refresher Archives


Copyright 2003 by Dr. Jodie Monger. All rights reserved.

Current Issue - Archives - CEO Links - News - Conferences - Recommended Reading