It’s been a dry spell for my subway series: I haven’t seen many good ads to riff on for a while now.  This morning, though, I saw an ad with this claim: “The internet’s top-rated sheets delivered right to your door.”  Aggregating customer reviews and ratings for purposes of a claim like this requires some serious analysis. Fortunately, NAD has engaged in that analysis on several occasions and its decisions provide useful guidance for marketers looking to harness the customer reviews and ratings it has received to craft a claim.  

NAD’s recent case involving claims by tax preparation software provider, TaxSlayer LLC, is a case in point.  TaxSlayer claimed to be “#1 Rated on Trustpilot,” a review site. Intuit, maker of TurboTax, challenged the claim.

First, NAD examined what messages were conveyed by the claim.  It determined that the claim not only conveyed the literal (truthful) message that Trustpilot rated TaxSlayer #1 in the tax preparation software category, but also the implied claim that the rating was “based on a reliable and representative survey of consumers using products across the entire tax preparation software category.”  Moreover, NAD determined, although the reviews on Trustpilot represent customers’ satisfaction with their own products and not product comparisons, the claim “Rated #1” was an inherently comparative claim.  NAD noted that “Trustpilot’s rating collects the reviews for each tax prep software company and creates a score that represents Trustpilot’s determination of the comparative satisfaction of one company’s consumers versus the consumer satisfaction of other companies, an analysis which is essentially comparative.”

Next, NAD examined the advertiser’s support for these claims. It noted that to support a claim that a product is Rated #1 within a broad product category, an advertiser should, as a general rule, compare itself to at least 85% of the relevant marketplace.  Thus, to support a #1 rated claim based on customer reviews, the reviews should have covered 85% of the relevant market.  Moreover, the customers surveyed should be representative of the broad base of customers who use the product.   

Using these standards, NAD determined that TaxSlayer did not adequately substantiate its Rated #1 on Trustpilot claim. First, the population of online reviews that created the basis for Trustpilot’s score was not representative of tax preparation software consumers because Trustpilot collects reviews from consumers that use programs from companies with which it has a business relationship at a much higher rate than those with which it does not. NAD noted that, because of this, “while TaxSlayer may be confident that its own reviews represent the opinions of its own consumers, it cannot say the same about the companies against which it is being compared.”  

Indeed, because of how Trustpilot collects reviews, the market leader (which does not have a business relationship with Trustpilot) had only received 15 reviews on the site, as compared to the over 2,500 TaxSlayer reviews.  Moreover, Trustpilot did not adequately qualify its claim to make it clear to consumers that the ranking was based on this lopsided information.  NAD noted that while the claim itself says that the claim is limited to companies in the tax prep software category on Trustpilot’s website, it does not inform them that this ranking is not based on a representative sample of tax prep software consumers. The reviews also did not cover the requisite 85% of providers, as required for an unqualified #1 claim.

NAD had other issues about the reliability of Trustpilot’s data.  It noted that, to be reliable support for an advertising claim, only those reviews that are associated with verified purchases should be counted.  Trustpilot, however, had no means to verify that the reviewers were legitimate consumers of the companies that they were reviewing – except those with which Trustpilot had a business relationship.  Moreover, Trustpilot had no way of preventing consumers from submitting multiple reviews. (By contrast, see this case where the advertiser could substantiate that the reviews upon which it relied for its "over 10,000 5 star reviews" claim were verified and not double-counted.) And, finally, NAD was troubled by the fact that there was insufficient information in the record about how Trustpilot actually created its rankings based on customer review data because other metrics were considered using Trustpilot’s proprietary “Trustscore”.

Accordingly, NAD recommended that TaxSlayer discontinue using the claim #1 Rated on Trustpilot.

Case Report #6286

An addendum:  Since I published this post, NAD issued another decision addressing (among other things) the use of customer reviews, specifically the claim "4.4/5 stars."  NAD recommended that the advertiser discontinue the claim or modify it to clearly and conspicuously disclose that it is based on reviews from consumers who had received free or discounted products.  The fact that individual reviews disclosed that reviewers received a discount or free product was, in NAD's view, insufficient because "reasonable consumers may skip reviewing each of the hundreds of reviews on the...website and assume that the aggregate claim, “4.4/5 stars” represents the sum total of all the reviews."  Case Report #6290.