Conducting an Accurate Net Promoter Score Survey

Net Promoter Scores are only as accurate as your survey methods.  If you have been tracking your Net Promoter Score (NPS) but your scores do not provide any valid and actionable insights, flawed survey methods may be at fault. Because only one question is asked, NPS surveys are sometimes conducted with less care and diligence than is ideal. If you are frustrated with limited applicability of NPS survey results, check your methodology for these three common Net Promoter Score survey problems.

Lifting the Curtain

When employees conducting your NPS survey are evaluated based on survey results, they may explain the survey instrument too thoroughly to customers: “If you don’t rate me a nine or ten on this scale, I’ll fail and I could lose my job.” Faced with a statement like the foregoing, no customer can be expected to provide a truthful answer.

To avoid lifting the curtain when conducting an NPS survey, ensure that employees collecting data are not also being evaluated according to the Net Promoter Score results produced. This may mean hiring someone to survey recent customers.

Oversimplification

How can one oversimplify a survey that only has one question? Surprisingly, many companies taint their NPS survey results by trying to further simplify an already immensely concise tool. If a customer has purchased multiple products or services from your organization, inaccurate results may be seen when asking that customer to answer a Net Promoter Score survey about the organization.

Overarching opinion is important; you know you have done well when a customer unhappy with one product still self-identifies as a Promoter for your company in its entirety.  However, more accurate results may be seen by augmenting your NPS survey with questions referring to specific products and services.

Failure to Comprehensively Evaluate Results

Imagine that you’ve hired an independent contractor to conduct a Net Promoter Score survey over a period of two weeks. At the end of this time frame, you’re provided with a report showing that your company’s NPS is well above your desired target, showing substantial improvement over your last survey. Delighted, you pay your contractor and assume you are on the right track. But shortly after you receive these encouraging results, business drops and you see several negative Web reviews of your products. What happened?

Even if a company’s NPS remains strong overall, a defect in product or service quality may create Detractors. It’s not enough to simply review your overall score. If pockets of Detractors exist, seek a common factor in their experiences. Did they purchase the same product or service? Did the same employee help them? Did they come at a time of day when your employees are slow to provide service? If you can discover a factor common to your Detractors, you  can reach out to all individuals sharing that trait and offer a solution.

How do you provide NPS evaluation without allowing conscious or unconscious bias to sway results?

1 comments
John
John

It’s worrying that many marketers seem happy to accept conventional wisdoms like NPS without question or any hard evidence that it’s true. Perhaps it’s no surprise therefore to see marketers and marketing industry publications sometimes regurgitate the latest consultancy fads and anecdotes with little sign of prior critical review.The claims made by Reichheld regarding Net Promoter’s linkage to growth and customers’ future loyalty behaviors are false.And so, for the first half of the last decade at least, Frederick Reichheld’s Net Promoter Score (NPS) – first published in the highly respected Harvard Business Review (HBR) in December 2003 under the confident title The One Number You Need to Grow (and subsequently in his bookThe Ultimate Question) – was trumpeted far and wide as the single most reliable indicator of company growth.The NPS metric was derived from survey responses to a simple question, which today my telco also asked me to answer: “How likely are you to recommend us to a friend?” Respondents who provide a rating of 9–10 on a 10-point scale are classified as “promoters”; respondents who provide a rating of 6 or lower are classified as “detractors.” A company’s NPS is calculated by subtracting the proportion of detractors from the proportion of promoters. If 50 percent of respondents answer 9-10 and 20 percent answer 6 or less, then the NPS is 30.Basically, Reichheld’s premise was that if companies can measure and manage the customer relationship, they then have a simple, easy to understand measure of customer satisfaction that links directly to the economics of growth. The rates of growth attributed to changes in Net Promoter Scores by some authors were significant – that Net Promoter leaders outgrow their competitors in most industries by an average of 2.5 times, and that a 12-point increase in Net Promoter Scores leads to a doubling of a company’s growth rate on average. Wow!Its initial rise to stardom was meteoric and was closely tied to the burgeoning word-of-mouth marketing industry, which seized on NPS as a way to quantify positive customer buzz. Some of the world’s leading businesses also adopted NPS, including General Electric and American Express, and American CEOs regularly reported Net Promoter data to analysts. Even Harvard Business Online surveyed its customers using only two Net Promoter questions. And it also had it’s own New Zealand-based apostles including consultants like Carlson Marketing, the Strategic Resource Development Group, Team Factors Limited, and Dr Ian Brooks (the consultant, not the university academic) who claimed to be New Zealand’s leading business advisor.One simple metric that says it all. The marketer’s Holy Grail… or so it clearly seemed to many.Far too many as it happens because, despite the hype, there was no independent research that quantified the relationship between Net Promoter Scores and firm revenue growth as claimed by Reichheld.Well that was the case until research by Keiningham et al appeared in the Journal of Marketing in July 2007 (which you can download for free from http://www.atypon-link.com/AMA/toc/jmkg/71/3 ). Keiningham and colleagues examined Reichheld’s Net Promoter research and findings, and the validity of his claim that Net Promoter is “the single most reliable indicator of a company’s ability to grow”. Their conclusions:No it’s not
Reichheld’s work appears seriously flawed
There is very strong evidence it suffers from research bias.
Keiningham’s team even found for two of the industries cited in The Ultimate Question(airlines and personal computers), the American Customer Satisfaction Index (ACSI)—which the HBR article specifically repudiates—actually outperformed NPS as an indicator of growth. Reichheld’s claim that NPS was the best growth indicator was proven false.“The most upsetting aspect of our research for me and my co-authors was the very strong evidence we uncovered of research bias in Reichheld’s reported research,” Tim Keiningham said, “Given that Reichheld’s research was published in the Harvard Business Review, it is extremely troubling and totally unacceptable.”Of even greater concern, when the authors analysed the HBR results and the appendices at the back of The Ultimate Question they concluded, “Reichheld specifically selected data sets to demonstrate the linkage between Net Promoter and growth.”Talk about the Emperor’s New Clothes!It’s clear HBR published the Reichheld NPS paper and many companies have adopted the Net Promoter metric, on the misplaced belief that sound science underpinned Reichheld’s claim of NPS being superior to other metrics.Keiningham’s paper clearly demonstrates this was not the case. It also highlights the gap that exists between the work of marketing academics (which is based on sound theory, analytical rigour and a scientific approach) and the writings of some marketing practitioners and consultants (who often place greater emphasis on simple inter-relationships without understanding the complex reasons why those relationships might exist).