(51) Evaluation of ChatGPT Responses to Common Puberty Questions

Background: ChatGPT, a large language model based chatbot launched in 2022, is one of the fastest-growing artificial intelligence (AI) applications in history. It has shown tremendous potential across many industries, however its accuracy and reliability must be verified prior to endorsement, especially in academia and in education. A proposed use in clinical medicine is for patient education. Adolescents as a group may be reluctant to discuss puberty and menstruation with their caregivers or physicians and may rely on online sources such as ChatGPT. Some studies reported that up to 60% of adolescents in the United States have used the application, but the ability for ChatGPT to provide reliable and evidenced based answers to common questions regarding puberty is unknown.

Methods: ChatGPT Version 3.5 was queried on 10 common questions regarding female puberty. References for the given answer were requested. Qualitative analysis was then performed and responses were graded as “acceptable” or “unacceptable.” Validated resources such as American College of Obstetricians and Gynecologists (ACOG) publications were used as comparisons. Consideration was taken regarding whether answers were age appropriate for the demographic studied. The study was designed to grade cited resources as “verified”, “broken”, “irrelevant”, “non-existent” or “no references”.

Results: Out of the questions queried, 60% of the responses were graded as acceptable (6/10) and the remaining responses were unacceptable (4/10). All of the unacceptable answers were incomplete. Of note, ChatGPT was not able to provide verifiable references for any of the questions.

Conclusions: Responses provided by ChatGPT were well written and accurate however 40% of responses were unacceptable and 100% lacked references. As patients look to the internet to synthesize the plethora of medical information available on the internet, it is essential for AI generated information to be both accurate and verifiable. While ChatGPT has great potential to provide easily accessible health information to adolescents, further study and development is needed before it can be endorsed for this purpose.