Original Article

Unravelling the truth: Examining the evidence for health-related claims made by naturopathic influencers on social media – a retrospective analysis

Introduction

Health misinformation on social media is a serious public health concern. The rapid, unregulated dissemination of false health-claims can result in misinformed individuals who make decisions for themselves and their families that can have deleterious consequences.¹ Inaccurate health information can cause harm by influencing the decisions people make about their health, and alternative therapies that replace evidence-based health can lead to adverse consequences.²

Social media platforms have emerged as popular sources of health-related information for the general public.³ Here, people are exposed to self-management health advice, support and self-tracking of personal health and fitness with other community members.⁴ With few legal regulations governing the distribution of poor-quality information, social media provides a dynamic forum for circulating health misinformation.⁵

One of the ways in which social media is being used is to discuss health information in relation to complementary and alternative medicine (CAM), including naturopathy.⁶ Although the training and regulation of naturopathic practitioners vary worldwide, naturopathic clinical education in many countries include treatment methods such as lifestyle-oriented self-care, dietary nutrition, homeopathy, herbal medicine, and over-the-counter medicines.⁷ Qualified clinicians, including naturopaths, should be providing evidence-informed advice to their patients and be careful in declaring conflicts of interest, and this should extend to online spaces.⁸ To date, little has been reported in the literature regarding health-related claims made on social media, especially on Instagram.⁵ Previous studies exploring health related information on Instagram specifically have focussed on HPV,⁹ vaping,¹⁰ COVID-19,¹¹ and Zika virus.¹² However, the validity of health-related claims made by naturopathic influencers on Instagram remains unknown. Previous research has highlighted concerns about low quality health information on social media and its impact on health outcomes.¹³ As the use of social media to seek health information grows, so do the concerns over the validity of online health information.¹⁴

The aim of this study was to measure how often naturopathic influencers make evidence-informed recommendations on Instagram, and to examine associations between the level of evidence available or presented, and user engagement. The study objectives were to: (1) determine the proportion of health-related claims posted by naturopathic influencers that are supported by readily available high-quality evidence or include a link to sources of evidence; and (2) identify whether evidence-based health claims generate a higher or lower number of Instagram Likes and Comments relative to their number of Followers.

Materials and Methods

Study design and cohort selection

A retrospective observational study using quantitative content analysis on health-related claims made by naturopathic influencers on Instagram was conducted. Social media accounts are dynamic: new accounts are frequently created, and their number of followers changes continuously. An interactive sampling approach was therefore used to obtain a sample of Instagram accounts managed by naturopathic influencers. Initially, a seeding set of Instagram accounts from a public website that provides resources to consumers interested in naturopathic and alternative treatments was used as a starting point.¹⁵ From this list of Instagram accounts, the ‘suggested for you’ function on Instagram was used as a snowballing tool to identify additional contemporary Instagram accounts that met our inclusion criteria, as of January 2020. Instagram accounts were included in this study if they met the following criteria: (a) accounts held by individuals claiming to be a naturopathic doctor in the ‘Bio’ section of their Instagram account, and (b) accounts with at least 30 000 followers. We chose to only include accounts where qualifications were specified because the naturopathic influencers were more likely to be viewed as a trusted source of information by the public, and we expected these individuals to provide health advice that is supported by evidence. A cut-off of 30 000 followers was chosen. This is a level considered sufficient to constitute a celebrity endorsement for medicines by the UK Advertising Standards Authority.¹⁶

Data variable extraction and definition

Health-related claims extracted from Instagram posts were converted to a clinical question using the PICO (population, intervention, comparison, outcome) framework to quantify the number of health-related claims present within each post.¹⁷ For example, where an Instagram post contained a claim that ‘beetroot improves liver function’, this was converted to P = general population, I = beetroot, C = placebo, O = improves liver function. The number of public Likes and Comments for each Instagram post containing a health-related claim was also recorded.

The number of health claims formulated for each Instagram post was dependent on the number of interventions and outcomes mentioned in the post. For each health claim, we searched PubMed using Medical Subject Headings (MeSH) and non-MeSH terms. The search results were filtered to show only human studies and studies published before the Instagram post. It was beyond the scope of this study to conduct a systematic review for each health claim, so the first 10 articles ranked by relevance on PubMed were retrieved and reviewed for whether they produced a conclusion that aligned with the claim to serve as a proxy for relevant literature. Those articles were then assigned a level of evidence using the National Health and Medical Research Council (NHMRC) levels of evidence for intervention studies (Supplementary file 1).¹⁸ Studies were considered to refute the claim where reported findings contradicted the health claim made on the Instagram post.

Data collection commenced in January 2020, covering a six-month retrospective period to account for potential seasonal effects in the topics of Instagram posts. To be included in this study, Instagram posts must have been dated between 1 June 2019 and 31 December 2019, with posts containing a health claim promoting a health intervention. Video posts and those that related to a risk-based exposure, rather than an intervention, were excluded (e.g., ‘bisphenol A (BPA) is associated with lower fertility and increased risk of miscarriage’). To reduce the risk of temporal bias in relation to the number of Followers, Likes and Comments captured during data collection, data extraction for each naturopathic influencer was performed for the same month before moving on to the subsequent month.

The data variables of interest were:

1. The highest level of evidence to support or refute health-related claims posted on Instagram by naturopathic influencers, graded using the NHMRC levels of evidence for intervention studies.¹⁸

2. Number of health-related posts that contained a link to sources of evidence. A link to source(s) of evidence is provided if the naturopathic influencer mentioned the name of the study, author(s) of the study or provided a direct web link to the published study.

3. Level of evidentiary support: sources of evidence provided or identified in PubMed were evaluated using the NHMRC levels of evidence.¹⁸

4. The number of public Likes and Comments for each Instagram post containing a health-related claim.

Statistical analysis

Descriptive summative statistics were used to present whether evidence was provided by naturopathic influencers (binary: no; yes) and the evidence available from the 10 most relevant articles on PubMed for health claims posted by each individual naturopathic influencer (three-level: no evidence identified; supporting evidence identified; refuting evidence identified) and for different health claim categories (). To analyse the relationship between health-related Instagram posts and the number of Likes and Comments, the level of evidentiary support for posts, for which evidence was found, were divided into five categories: high, moderately high, moderate, low, and none. Level of evidentiary support for post was assigned, as follows: high = 76-100% of claims within post with evidence; moderately high = 51-75% of claims within post with evidence; moderate = 26-50% of claims within post with evidence; low = 0-25% of claims within post with evidence, none = no evidence.

Close

Figure 1. Levels of evidence assigned for each health-related claim and Instagram post

Linear regression was computed to measure the association between health-related posts and the number of Likes, and Comments. No evidence and refuting evidence were collapsed into one category as ‘no evidentiary support’ due to the small number of claims identified with refuting evidence. The Likes and Comments were entered into the model as dependent variables while evidentiary support for claim (no; yes), highest evidence level identified and level of evidence provided by naturopathic influencer (high, moderately high, moderate, low, and none, based on NHMRC levels of evidence), link to evidence provided by naturopathic influencer (no; yes), and health claim category were entered as independent variables, number of Followers was entered as a control variable. Collinearity was assessed to ensure the value of tolerance was less than 1.0 for each analysis. Residual plots were also performed to ensure a linear relationship between the logit of the outcome and each predictor variable. Statistical significance was set at α = 0.05. Statistical analysis was performed using IBM SPSS Statistics V.25.0.¹⁹

Results

A total of nine Instagram accounts met the inclusion criteria of ≥ 30 000 followers. Of these nine accounts, one was excluded from data extraction due to posting only video content. From the remaining eight accounts, a total of 494 health claims were extracted for data analysis. Over half (51.0%) of the 494 health claims had no evidence available (Table 1). Regarding the claims with evidence available, 10.9% (n = 54/494) were systematic reviews of randomised controlled trials (Level I), 19.8% (n = 98/494) were randomised controlled trials (Level II), 0.2% (n = 1/494) were pseudorandomised controlled trials (Level III-1), 10.5% (n = 52/494) were comparative studies with concurrent controls (Level III-2), 5.5% (n = 27/494) were comparative studies without concurrent controls (Level III-3) and 2.0% (n = 10/494) were based on case studies (Level IV).

From the total number of health claims extracted, 6.9% (n = 34/494) of health claims provided Instagram users with a link to evidence to support the claim made (Table 2). Three out of the eight naturopathic influencers provided no evidence to support any of the health claims they made on Instagram. When evidence was provided to support the claims made, it was most commonly in the form of a randomised controlled trial (Table 3).

Claims related to ‘women’s health’ (19.0%, n = 94/494) and ‘hair, nail and skin’ (15.0%, n = 74/494) were the two most identified health claim categories (Table 4). However, more than half of the claims had no evidence for either of these health claim categories. Claims related to mental health had the highest proportion of posts with NHMRC Level I (n = 13/50, 26%) and Level II evidence (n = 14/50, 28%). Claims related to ‘oral health’ (0% with evidence, n = 0/1), ‘medications’ (0% with evidence, n = 0/5) and ‘eye health’ (6.2% with evidence, n = 1/16) had the lowest proportion of claims with evidence available.

Posts with low and moderate levels of evidentiary support had the highest median number of Likes (). Posts with a moderate level of evidentiary support had the greatest distribution in the number of Likes. A condensed spread of Likes is seen for post with no evidence (). Posts with a high level of evidentiary support had the lowest median number of Comments (). For posts with none, low, moderate, and moderately high level of evidentiary support, the median number of Comments increased with increasing levels of evidentiary support ().

Close

Figure 2. Box plot showing level of evidentiary support for Instagram post and the number of likes.Level of evidentiary support for post: high = 76-100% of claims within post with evidence; moderately high = 51-75% of claims within post with evidence; moderate = 26-50% of claims within post with evidence; low = 0-25% of claims within post with evidence, none = no evidence. Circles and asterisks represent outliers

Close

Figure 3. Box plot showing level of evidentiary support for Instagram post and the number of comments. Level of evidentiary support for post: high = 76-100% of claims within post with evidence; moderately high = 51-75% of claims within post with evidence; moderate = 26-50% of claims within post with evidence; low = 0-25% of claims within post with evidence, none = no evidence. Circles and asterisks represent outliers

After controlling for the number of followers, posts with links to evidence had fewer Likes (β = -0.1, P = 0.02) and fewer Comments (β = -0.2, P = 0.01), compared to posts without links to evidence (Table 5). Health claims categorised under ‘general health’ with evidentiary support were associated with a higher number of Likes (β = -0.4, P < 0.001) and Comments (β = 0.3, P = 0.03), which included issues related to anti-aging, increasing in-vivo collagen production and reducing in vivo inflammatory responses. Health claims with evidentiary support for topics related to ‘cold, flu and immunity’ were associated with approximately 225 more Comments than claims within the same category without evidentiary support (β = 0.4, P < 0.001).

Discussion

We found approximately half of the health claims posted on Instagram by naturopathic influencers with 30 000 or more followers were unsupported by evidence. Of those with evidence clearly available, approximately 10% of health claims were underpinned by high quality evidence such as systematic reviews of randomised controlled trials. Health claims were rarely accompanied by a link to supporting evidence. Posts that had evidence available received fewer Likes and Comments, whether the evidence was linked or if it was identified via a literature search.

Little attention has been directed towards the credibility of health information available on social media despite evidence showing that social media use can influence health behaviours, decision-making, and risk perceptions.²⁰ Many health claims on social media promote health remedies that are untested, ineffective, or unsafe, which may introduce unnecessary risks of harm.²¹ The results may be partially explained by the relatively small number of clinical trials undertaken for CAM, which are complicated by barriers such as lack of access to funding and lack of accessibility to well-trained scientists to conduct high-quality clinical trials.²² Although social media guidance exists for registered health practitioners in Australia, for example²³; engagement in controversial topics on social media has been linked to an increase in public interest and discussion, generating higher amounts of traffic to the social media profile.²⁴

Lower engagement with claims supported by evidence or provided a link to supporting evidence signals a potential disinterest in evidence-based medicine by those Instagram users following naturopathic influencers. This may be explained by limited knowledge and understanding of the scientific process, issues with health literacy and digital health literacy,²⁵ or may reflect a lack of trust or disengagement with established institutions and those with recognised expertise.²⁶ Disinterest towards evidence-based medicine also is thought to be driven by a preference for lived experience over evidence from published research outlined in academic journal articles.²⁷ To encourage the effective communication of evidence-based health information, the use of storytelling can improve engagement with evidence-based information, while increasing the chances for this information to be absorbed and retained.²⁸

Studies on other social media platforms have produced results that converge with our findings. A study investigating health information related to sun protection and skin cancer prevention on YouTube concluded that videos on tanning beds and sunscreen contained inaccurate information 40% and 20% of the time, respectively.²⁹ A study exploring the dissemination of public health information on the Zika virus via Facebook found that misleading posts were far more popular than posts sharing accurate, relevant information.³⁰ Additionally, other healthcare professionals, including physicians and dieticians, have also shared a considerable number of medical information on Twitter that were later concluded to be false based on expert review.⁸ In contrast to the health-related claims topics posted by naturopathic influencers in our study, a systematic review investigating the spread of health-related misinformation on social media generally found misinformation largely related to vaccines and communicable diseases, and to a lesser extent, chronic non-communicable diseases, such as cancer.²⁷

A key limitation was relying on the 10 most relevant retrieved articles in PubMed, rather than conducting an extensive search for evidence, when not provided alongside the claim. It is possible we did not identify existing evidence to support some of the claims; however, it would not have been feasible to construct a comprehensive search strategy and screen articles for every health claim made. Our search strategy served as a proxy measure for the evidence to support the claim and it is unlikely that the 10 most relevant articles on PubMed would systematically fail to identify existing supporting evidence.

The risk of subjectivity in the extraction of interventional health claims from Instagram posts and conversion into a PICO is another limitation. The subjectivity of data extraction was minimised using standards which included extracting specific terms used by the naturopathic influencer to perform database searches on PubMed. If not explicitly stated, the intended target audience of the health claims were made using educated guesses based on the intervention and outcome promoted in the claim. For the claim to be considered supported by evidence, study participants must match the intended audience of the health claim on Instagram(e.g., postmenopausal women, athletes). When the study with the highest level of evidence were inconclusive due to reasons such as inadequate cohort size, conflicting results for different cohort (e.g., male vs. females), heterogeneity between studies and poor quality of studies, it was concluded that the evidence did not support the health claim. When there was considerable uncertainty related to the health intervention or outcome, it was excluded from the study.

Our study captured a sample of naturopathic influencers, using an iterative approach. The population of potentially eligible Instagram accounts could not be known or specified at each stage of the sampling process. Accounts were only considered if they met our inclusion criteria. Verification and replication of the study sample is unlikely to be feasible given the dynamic nature of Instagram accounts. Therefore, repetition of this process may result in a slightly different sample. However, we do not believe this limits the representativeness and external validity of the sample identified.

Claims were only extracted if they related to health interventions, and we excluded any claims related to a health risk exposure. Other limitations related to the sample size; including video posts or expanding the scope of influencers that could be included in the study may have revealed additional or alternative findings related to evidence or non-evidence-based posts on Instagram. It is not possible to determine the nature of the relationship between Instagram accounts providing evidence-based health claims and number of Instagram Likes and Comments from this study. The relationship may be due to the lack of interest in evidence-based information, the number of Followers the account already has, Followers viewing the claim as a form of advertisement, or other unknown reasons.

Clinical implications and future recommendations

The lack of evidence to support online health claims made on social media platforms such as Instagram is an important and understudied issue. Social media platforms allow inappropriate content to be reported, and this may be a useful way for social media platforms to partner with trusted authorities to identify and flag or downrank health claims that are unsubstantiated and may cause harm. Due to the highly variable quality of online health information, consumers may not be well-equipped with the skills to discern the quality of health information. There is potential for the use of automatic credibility appraisal tools to identify communities at higher risk of exposure to low-credibility health information on the internet.³¹ Given the influence of social media in healthcare, clinicians and researchers should focus on identifying the most effective ways of disseminating evidence-based health information on social media, populations most at risk of health misinformation and interventions to reduce the spread of health misinformation on social media.

Conclusion

This study is one of the first to look at the evidence available to support health-related claims on Instagram. Our findings highlight that around half of posts from popular naturopathic influencers with health claims that are supported by high-quality evidence. The dissemination and amplification of health misinformation, especially those that are counter to the policies and activities of public health organisations, can lead to harmful behaviours.

Author Contributions

Conceptualisation: Andrea L Smith, Adam G. Dunn, Mitchell Sarkies.

Methodology: Van Nguyen, Andrea L Smith, Louise A. Ellis, Adam G. Dunn, Mitchell Sarkies.

Investigation: Van Nguyen.

Formal analysis: Van Nguyen, Luke Testa, Louise A. Ellis.

Visualization: Van Nguyen, Luke Testa.

Writing–original draft preparation: Van Nguyen.

Writing–Review & Editing: Luke Testa, Andrea L Smith, Louise A. Ellis, Adam G. Dunn, Jeffrey Braithwaite, Mitchell Sarkies.

Supervision: Andrea L Smith, Mitchell Sarkies.

Funding

No funding declared.

Ethical Approval

Ethical review was not required for this study as it did not involve human subjects.

Competing Interests

The authors have declared that there are no financial or non-financial competing interests

Supplementary Files

Supplementary file 1. NHMRC levels of evidence for intervention studies.

References

1. Citizens versus the internet: confronting digital challenges with cognitive tools. Psychol Sci Public Interest 2020; 21(3):103-56. doi: 10.1177/1529100620946707 [Crossref]
2. Complementary medicine, refusal of conventional cancer therapy, and survival among patients with curable cancers. JAMA Oncol 2018; 4(10):1375-81. doi: 10.1001/jamaoncol.2018.2487 [Crossref]
3. Addressing health-related misinformation on social media. JAMA 2018; 320(23):2417-8. doi: 10.1001/jama.2018.16865 [Crossref]
4. Social media-enabled healthcare: a conceptual model of social media affordances, online social support, and health behaviors and outcomes. Technol Forecast Soc Change 2021; 166:120574. doi: 10.1016/j.techfore.2021.120574 [Crossref]
5. Prevalence of health misinformation on social media: systematic review. J Med Internet Res 2021; 23(1):e17187. doi: 10.2196/17187 [Crossref]
6. Exploring the combined effects of social media use and medical skepticism tendency on recourse to complementary and alternative medicine. J Altern Complement Med 2021; 27(8):710-2. doi: 10.1089/acm.2020.0514 [Crossref]
7. Overview of international naturopathic practice and patient characteristics: results from a cross-sectional study in 14 countries. BMC Complement Med Ther 2020; 20(1):59. doi: 10.1186/s12906-020-2851-7 [Crossref]
8. Are health-related tweets evidence based? Review and analysis of health-related tweets on Twitter. J Med Internet Res 2015; 17(10):e246. doi: 10.2196/jmir.4898 [Crossref]
9. A content analysis of HPV related posts on Instagram. Hum Vaccin Immunother 2019; 15(7-8):1476-8. doi: 10.1080/21645515.2018.1560774 [Crossref]
10. Vaping on Instagram: cloud chasing, hand checks and product placement. Tob Control 2016; 26(5):575-8. doi: 10.1136/tobaccocontrol-2016-053052 [Crossref]
11. COVID-19 on Instagram: a content analysis of selected accounts. Health Policy Technol 2021; 10(1):165-73. doi: 10.1016/j.hlpt.2020.10.016 [Crossref]
12. Public sentiment and discourse about Zika virus on Instagram. Public Health 2017; 150:170-5. doi: 10.1016/j.puhe.2017.07.015 [Crossref]
13. Social media use for health purposes: systematic review. J Med Internet Res 2021; 23(5):e17917. doi: 10.2196/17917 [Crossref]
14. Health information seeking behaviors on social media during the COVID-19 pandemic among American social networking site users: survey study. J Med Internet Res 2021; 23(6):e29802. doi: 10.2196/29802 [Crossref]
15. Top 50 Naturopathic Doctors to Follow on Instagram (Unranked). Available from: https://naturalpath.net/natural-news/top-50-naturopathic-doctors-you-should-be-following-on-instagram/. Accessed October 20, 2020.
16. ASA Ruling on Sanofi UK in association with This Mama Life. Available from: https://www.asa.org.uk/rulings/sanofi-uk-A19-557609.html. Accessed May 16, 2022.
17. Enhancing your practice through evidence-based decision making: PICO, learning how to ask good questions. J Evid Based Dent Pract 2001; 1(2):136-41. doi: 10.1016/S1532-3382(01)70024-3 [Crossref]
18. National Institute of Clinical Studies. Appendix F: Levels of Evidence and Recommendation Grading. Available from: https://www.nhmrc.gov.au/sites/default/files/images/appendix-f-levels-of-evidence.pdf. Accessed October 20, 2020.
19. SPSS Statistics. Available from: https://www.ibm.com/products/spss-statistics. Accessed January 17, 2023.
20. Social media, e-health, and medical ethics. Hastings Cent Rep 2019; 49(1):24-33. doi: 10.1002/hast.975 [Crossref]
21. Protecting consumers from fraudulent health claims: A taxonomy of psychological drivers, interventions, barriers, and treatments. Soc Sci Med 2020; 259:112790. doi: 10.1016/j.socscimed.2020.112790 [Crossref]
22. Barriers to the conduct and application of research in complementary and alternative medicine: a systematic review. BMC Complement Altern Med 2017; 17(1):166. doi: 10.1186/s12906-017-1660-0 [Crossref]
23. Social Media: How to Meet Your Obligations Under the National Law. Available from: https://www.medicalboard.gov.au/Codes-Guidelines-Policies/Social-media-guidance.aspx. Accessed June 3, 2022.
24. Garimella K, De Francisci Morales G, Gionis A, Mathioudakis M. The effect of collective attention on controversial debates on social media. In: Proceedings of the 2017 ACM on Web Science Conference; New York, USA. p. 43-52. 10.1145/3091478.3091486
25. Exploring the role of health literacy in the evaluation of online health information: insights from a mixed-methods study. Patient Educ Couns 2016; 99(6):1017-25. doi: 10.1016/j.pec.2016.01.007 [Crossref]
26. The spread of true and false news online. Science 2018; 359(6380):1146-51. doi: 10.1126/science.aap9559 [Crossref]
27. Systematic literature review on the spread of health-related misinformation on social media. Soc Sci Med 2019; 240:112552. doi: 10.1016/j.socscimed.2019.112552 [Crossref]
28. Rethinking communication: integrating storytelling for increased stakeholder engagement in environmental evidence synthesis. Environ Evid 2018; 7(1):6. doi: 10.1186/s13750-018-0116-4 [Crossref]
29. YouTube as a source of health information: analysis of sun protection and skin cancer prevention related issues. Dermatol Online J 2017; 23(1):13030/qt91401264. doi: 10.5070/d3231033669 [Crossref]
30. Zika virus pandemic-analysis of Facebook as a social media health information platform. Am J Infect Control 2017; 45(3):301-2. doi: 10.1016/j.ajic.2016.08.022 [Crossref]
31. Automatically appraising the credibility of vaccine-related web pages shared on social media: a Twitter surveillance study. J Med Internet Res 2019; 21(11):e14007. doi: 10.2196/14007 [Crossref]