Researchers from Torino, Italy, utilized Chat Generative Pretrained Transformer (ChatGPT), one of the most widely used chatbots, developed by OpenAI, to generate nutrition advice using prompts mimicking typical user questions to investigate the accuracy of advice provided.ย
Results showed that ChatGPT offered mostly clear advice, however, when the chatbot handled overlapping conditions, limitations emerged, resulting in some contradictory or inappropriate advice, the authors noted in the Journal ‘Nutrients’.โ
โAlthough ChatGPT exhibited a reasonable accuracy in providing general dietary advice for NCDs, its efficacy decreased in complex situations necessitating customized strategies,โ they concluded.
ChatGPT reliabilityโ
NCDs account forย 74% of global deathsโ, with cardiovascular diseases (CVD) being a major contributor: According toย global estimatesโ, there are approximately 43.8 million cases of type 2 diabetes (T2DM), 18.5 million cases of hypertension and 1.2 billion cases of non-alcoholic fatty liver disease (NAFLD) worldwide.
Lifestyle and dietary factors play a crucial role in NCDs, and onlineย health information searchesโย are on the rise.
ChatGPT, which uses machine learning algorithms to generate responses to text-based queries, mimicking human-like conversation,ย is gaining popularityโย as a source of personalized and immediate medical advice.
There is growing application of artificial intelligence (AI) in the domain of patient education and support, and anย increasing interest among healthcare professionalsโย to explore the tool’s potential to enhance patient care, diagnosis and treatment.
However, there is limited literature on the use of ChatGPT for improving nutrition knowledge in patients with different chronic NCDs.
One recent study explored the potential ofย ChatGPT in delivering nutritional informationโย and concluded that it could not replace the expertise of a registered dietitian, particularly when addressing complex medical conditions, while other research found that theย chatbot performed on par with human dietitiansโย in delivering accurate answers.
Therefore, the objective of the study was to compare the nutritional information supplied by ChatGPT (version 3.5) regarding different NCDs requiring dietary advice with the nutritional recommendations from international guidelines.
Assessing nutritional guidanceย โ
During November 2023, the researchers assessed the appropriateness of dietary advice given by ChatGPT through the formulation of a set of prompts for various conditions: Dyslipidemia (hypercholesterolemia and hypertriglyceridemia), arterial hypertension, type 2 diabetes mellitus (T2DM), obesity, NAFLD, chronic kidney disease (CKD) and sarcopenia.
Questions were designed by a panel of experts (medical doctors and registered dietitians) using language and sentence structure that replicated how patients might inquire with a healthcare professional.
Three prompts were tested for each medical condition: โCould you provide guidance on planning an optimal diet to manage [disease]?โ; โWhat are the dietary recommendations for [disease]?โ; and โI have [disease], what should I eat?โ.
The dietary advice given by ChatGPT was compared with nutrition guidelines, and then the chatbotโs capacity to manage a complex case was investigated by introducing more intricate scenarios involving the coexistence of multiple conditions.
The expert panel assessed and designated ChatGPTโs responses as โappropriateโ if the content aligned with the guidelines, โinappropriateโ if in contradiction of the guidelines, โnot supportedโ if not confirmed in the guidelines, and โnot fully matchedโ if the responses did not fully fulfill the guideline recommendations.
ChatGPT advice that was not specifically focused but promoted an overall healthy diet was classified as โgeneral adviceโ. Any dietary guideline recommendation that was not present in the ChatGPT response was labeled as โmissingโ.
Results showed that overall, ChatGPT offered clear advice, with the appropriateness of responses ranging from 55.5% (sarcopenia) to 73.3% (NAFLD).
Including both categories of โappropriateโ and โgeneral adviceโ, the overall accuracy of nutritional recommendations reached 100% for sarcopenia.
However, two recommendations contradicted guidelines: one conflicting with mealtime and snack guidelines from theย European Association for the Study of Obesityโ; another advising supplements such as vitamin E, omega-3 fatty acids and antioxidants in NAFLD, while the European Society for Clinical Nutrition and Metabolism (ESPEN) assert that these supplementsย cannot be endorsedโย while further data on efficacy is pending.
A single suggestion for T2DM was found to be โunsupportedโ, while many recommendations for various NCDs were deemed to be โnot fully matchedโ to the guidelines despite not directly contradicting them.
Additionally, the researchers found that limitations emerged when the chatbot handled overlapping conditions, resulting in some contradictory or inappropriate advice.
“Our findings indicated that the chatbot was unable to combine different recommendations, which resulted in contradictory or inappropriate advice that could have confused users,” the authorsย concluded.
The authors also noted that the use of ChatGPT in nutrition education could be limited by the patientโs digital literacy, and while younger individuals may readily embrace technology, older patients may not be as willing to do so and might encounter difficulties understanding the platform.
Journal: Nutrientsโ
doi:ย https://doi.org/10.3390/nu16040469โ
โIs ChatGPT an Effective Tool for Providing Dietary Advice?โ
Authors: Ponzo, V. Et al.