The Market Research Hall Of Shame

MSN Money recently published its Customer Service Hall of Shame, which found banks like Bank of America, Citibank, and Wells Fargo in the “Bottom 10”.

As a way to generate publicity, this is an infallible approach. Survey some consumers, ask them to rate a bunch of big companies on customer service, and blare the results all over the press and the Internet heralding the worst customer service providers in the country. Great job, MSN Money!

But as serious market research, this is garbage. How did MSN Money come up with its list? By first asking consumers to nominate firms to the potential list of bad service providers. So right off the bat, when asking consumers to rate those firms in the follow up survey, two things are guaranteed:

  • Only large firms will make the list, since the sample size for smaller or regionally-focused firms won’t reach critical mass. Is it true that only large firms deliver bad service? No.
  • More importantly, there’s no way to determine the difference in ratings between the “worst” firms (those included in the survey) and the “best” firms. Who those firms are, of course, is anybody’s guess since they weren’t nominated or included in the survey.

How can anybody evaluate the context of the results?

One of the firms on the dishonorable mention list is Macy’s. Yet only 10% of the customers surveyed rated its customer service as “poor” and, in fact, 60% rated it “good” or “excellent”! So tell us, MSN Money — how does this compare to the very best customer service providers? You don’t know. [By the way, of the six firms on the dishonorable mention list, five of them were rated as “good” or “excellent” by more than half of their customers].

What this is is nothing more than a shameless hatchet piece designed to embarrass a bunch of big companies and generate publicity for the firm releasing the survey. If this happens to your firm, you should:

1) Publicly ignore the study. By issuing public responses to the study, firms like BofA and Comcast wrongly bestow legitimacy to it. And with all due respect to BofA and its customer service executive (who the response came out from), the BofA response was, in my opinion, weak and confusing. While it would be my inclination to blast the methodology of the study (as I’m doing here), a wise firm would simply ignore it.

2) Privately strike back. If I were the top market research exec at any of the firms on the list, I would immediately send out a memo to my market research providers letting them know that if any of them participated in any of these “designed-to-be-a-hatchet-job” studies, they would be banned from bidding on any jobs in my firm for three years.

The firms on the list represent 16 of the largest firms in the country, and presumably, all of them spend a fair penny on market research. Do you think Zogby would have been so quick to take this job if it thought it would be shut out from working with these firms — and who knows how many others? I’m not saying that market research firms should clear studies with other clients. But you couldn’t convince me in a million years that no one at Zogby knew how the results of this study were going to be used.

So congratulations MSN Money — you’re the first inductee in my Market Research Hall of Shame.

Technorati Tags: , , ,

[Personal note to John Dawson: Is there someone you’d like to nominate?]


3 thoughts on “The Market Research Hall Of Shame

  1. Hi Ron,

    Great article again. You’re right that this is a pet topic of mine. The article i wrote singled out the Newspaper Marketing Agency for their efforts in justifying spending on that medium but there are many other potential nominees. I think this is going to be a great article to keep open and updating as new claims arise. However there is one point to remember – how many times did someone commissioning marketing research ask for the considered opinion of the agency in question rather than the answer they wanted? In my experience, few organisations are in the position to commission research with large enough sample sizes to get fantastic statistical accuracy so they bend the rules (sometimes with and sometimes without the knowledge of the client). Once a researcher releases their findings, it’s not really their responsibility to manage how that research is used. In the case of the NMA, I assume that those agencies involved in writing up studies pointed out a series of caveats (much like lawyers) which were ignored for PR purposes.

    Is it poor research or misleading reporting – I think it’s often the latter.


  2. Thanks for the comment, John. What really irked me about the “study” was partly the methodology (only rating a selected group of companies versus the broader set), but more so the motivation.

    It’s “caveat emptor” as far as I’m concerned when it comes to interpreting the results of surveys commissioned by firms that show support for the products or services they sell.

    But MSN Money’s effort was nothing but a hatchet job. And here they are claiming that they’ve found the firms with the worst customer service? BS.

    What I’m really hoping people will pick up on is my second bullet point about striking back. Zogby lent the study credibility because of its reputation. But what if the 25 largest firms in the US told their research providers they’re fired if they take on BS studies like this? Then the next time MSN Money or somebody else wanted to generate this BS PR, they’d have to rely on a no-name research outfit that wouldn’t bestow instant credibility on the effort.

  3. My guess is that none of the top 25 firms would want others to think that they had ever engaged in poor research in the past. Maybe what’s required here is some sort of independent certification rather like that which exists when auditors sign off on accounts – “we certify …”

Comments are closed.