In a recent case, NAD reviewed the claims of Fitness Cubed, Inc. for its Cubii Seated Elliptical Trainer, an under-desk elliptical device, at the behest of challenger Actegy Health Inc. Cubii made several types of claims about the health benefits of its product in a series of tv commercials. The spots included claims, presented in the form of user testimonials and statements from lab-coated personnel, that the using the device helps with post-surgical physical rehab, improves circulation and quality of life, makes users more active, reduces pain, and more. To support its claims in the challenge, Cubii offered the results of two online user surveys and two small pilot studies, a certification from the Mayo Clinic and letters from healthcare experts. 

Not surprisingly, NAD began its discussion by reiterating its standard for health claims: such claims should be supported by competent and reliable scientific evidence, which means “methodologically sound testing with sufficient controls to properly determine whether a purported benefit is a result of the product’s efficacy rather than a placebo effect.”  Specifically, that means that “[t]he study’s objectives should be described clearly, and the methodology must be appropriate for obtaining the objectives posed by the study. The study’s duration should be sufficient to detect an effect on the outcome and the sample size should be large enough to provide sufficient statistical power, with the study population representative of the target population to which the claim is targeted.”  A well-designed clinical trial is “double-blind, randomized, and appropriately-controlled, demonstrating that the treatment group experienced statistically significant difference to the 95% confidence level as compared to the control group.”  And, finally, “the results must translate into meaningful benefits for consumers that relate directly to the performance attributes promised by the advertising.”

Applying these standards, NAD found that the advertiser lacked adequate support for its claims. The clinical studies and certification were too small and/or too little information was provided about them, and the expert letters were inconclusive. Most interesting to me, though, was NAD’s analysis of the two large scale user surveys Cubii conducted and relied upon for its claims. One was a survey of over 1000 Cubii users who responded to questions about perceived changes in pain, activity levels, mood, sleep, range-of-motion, use of balance aids, and reliance on pain medication. The respondents were first asked questions about their health conditions, followed by questions about those conditions since using Cubii. 

The second was a survey of over 1600 Cubii customers. In addition to asking customers about their usage habits, the survey also asked respondents how much they agreed or disagreed with various statements about using Cubii, like whether it made them feel stronger and helped them reach their wellness goals.  Although in both surveys users reported improvements on the various metrices surveyed, NAD rejected both as substantiation for the advertiser’s health claims because both “relied purely on respondents to self-report on subjective and imprecise outcomes.”  Indeed, NAD found, “[s]elf-reporting surveys can be unreliable, especially when the questions require respondents to report on conditions that are very personal and difficult to objectively measure, like pain, mobility and quality of life.”  Rather, an appropriate study would have controlled for variables that may affect outcomes, like lifestyle, diet, medications, etc., to ensure that the product itself is responsible for the claimed benefits, and not the other variables. Thus, NAD concluded that the evidence provided was not sufficiently reliable to support the advertiser’s health-related claims and recommended that all the challenged claims be discontinued.

This is interesting to me for a couple of reasons.  First, self-reported data is commonly used as substantiation, in home-use tests and central location tests. Indeed, some industries regularly collect information about their users’ experience with products and how they perceive the benefits and then use that data in claims: for example, cosmetic companies often conduct studies about what their users perceive about the benefits of the products and design claims around those self-reported perceptions (albeit, often paired with clinical data).  Further, subjective experience – like pain –  seems like it would an appropriate metric for self-reported data.  Who knows better how much pain you’re experiencing than you?  But what NAD's Decision underscores is that even if users’ subjective experience is relevant, the means for obtaining it must still follow rigorous protocols, such as blinding in order to mitigate the effect of purchase bias (liking any product you spent money on), and controls to ensure that other variables (like diet, medications, etc) are not responsible for the claimed outcomes.

Further, if you’re making impactful health benefit claims, you will probably not be able to rely on self-reported data alone, no matter how much you’re collecting.  Here, NAD called the results of the two studies “anecdotal,” notwithstanding the fact that the anecdotes came from over 2000 users.  That’s because benefits such as increased mobility and shortening the time in a rehab facility after surgery are ones that can be studied clinically and objectively: users’ perceptions of how well the product is working for them may be an interesting data point for a company, but are not likely to be dispositive as to the actual benefit provided by the product unless the study accounts for other factors that could impact those claimed benefits.

Large consumer use studies are costly to undertake, and if the results are good, marketing teams will really want to use the results in claims. But, typically, studies conducted for claims substantiation purposes  -- not to mention an actual clinical study -- have far more guardrails than one designed for market research purposes.  And those guardrails are really going to matter when a company wants to say that its product will help address a serious health issue.

NAD/CARU Case Reports, Case #7145 (March 2023)