Not All Scientific Evidence is Created Equal
For health tech companies, establishing credibility is one of the most important building blocks of success. When a company is forming their team, this usually involves recruiting and hiring people who have impressive credentials. When a company is marketing their new product or service, it often means using phrases like “science-based” and “evidence-based.” These phrases convey trustworthiness and expertise — two ingredients that are absolutely essential in order to attract new users.
The phrase “science-based” is an effective marketing tool because it closes the gap in people’s minds between “I believe in science” and “I believe in what this company is offering.” After all, if there’s objective scientific evidence to support a company’s claims, then (the thinking goes) it must be reasonable to believe those claims. But what do these ubiquitous phrases mean, exactly? When a product or service is said to be “evidence-based,” is the company referring to high-quality evidence or low-quality evidence?
The health tech industry is still in its infancy, but the need for a standardized way to assess the quality of scientific evidence is nothing new. The GRADE approach was developed by the Grades of Recommendation, Assessment, Development and Evaluation Working Group (GRADE Working Group) for this purpose. Over 100 organizations, including the World Health Organization and Cochrane, use the GRADE system to assess the certainty, or quality, of evidence. In short, it’s a three-step process.
First, an initial level of certainty is assigned based on study design. In general, randomized trials are considered to have high certainty, while non-randomized studies (including observational studies) are considered to have low certainty. Once the initial level of certainty has been established, the second step is to consider raising or lowering the level of certainty based on other factors. For example, it may be lowered if there is a risk of bias or possible publication bias. On the other hand, it may be raised if there is a large effect or dose response. Ultimately, once all the necessary factors have been taken into account, a final level of certainty rating is given: high, moderate, low, or very low.
When considering a medical intervention (whether that intervention is a medication, device, procedure, or some other product or service), the quality of the evidence supporting that intervention is critical information. High-quality evidence is reassuring because it indicates there’s a higher degree of certainty that the benefits outweigh the risks. But the converse is also true. When a medical intervention is based on low-quality evidence, there’s inherently much less certainty that the benefits outweigh the risks. This may impact whether or not one decides to proceed with the intervention after all.
The direct-to-consumer business model is becoming more prevalent among health tech companies, and therefore, it’s becoming increasingly important for laypeople to understand that not all scientific evidence is created equal. Companies have no qualms about using phrases like “science-based” and “evidence-based” as shortcuts to establish credibility, but the quality of that evidence must always be taken into account.