Feed: Engadget is a web magazine with obsessive daily coverage of everything new in gadgets and consumer electronics
Feed Link: https://www.engadget.com/
---

Title: GPT-4 performed close to the level of expert doctors in eye
assessments

Date: Thu, 18 Apr 2024 13:15:17 +0000
Link: https://www.engadget.com/gpt-4-performed-clos...

As learning language models (LLMs) continue to advance, so do questions about
how they can benefit society in areas such as the medical field. A recent
study from the University of Cambridge's School of Clinical Medicine found
that OpenAI's GPT-4 performed nearly as well in an ophthalmology assessment
as experts in the field, the Financial Times first reported.

In the study, published in PLOS Digital Health, researchers tested the LLM,
its predecessor GPT-3.5, Google's PaLM 2 and Meta's LLaMA with 87 multiple
choice questions. Five expert ophthalmologists, three trainee
ophthalmologists and two unspecialized junior doctors received the same mock
exam. The questions came from a textbook for trialing trainees on everything
from light sensitivity to lesions. The contents aren't publicly available, so
the researchers believe LLMs couldn't have been trained on them previously.
ChatGPT, equipped with GPT-4 or GPT-3.5, was given three chances to answer
definitively or its response was marked as null.

GPT-4 scored higher than the trainees and junior doctors, getting 60 of the
87 questions right. While this was significantly higher than the junior
doctors' average of 37 correct answers, it just beat out the three trainees'
average of 59.7. While one expert ophthalmologist only answered 56 questions
accurately, the five had an average score of 66.4 right answers, beating the
machine. PaLM 2 scored a 49, and GPT-3.5 scored a 42. LLaMa scored the lowest
at 28, falling below the junior doctors. Notably, these trials occurred in
mid-2023.

While these results have potential benefits, there are also quite a few risks
and concerns. Researchers noted that the study offered a limited number of
questions, especially in certain categories, meaning the actual results might
be varied. LLMs also have a tendency to "hallucinate" or make things up.
That's one thing if its an irrelevant fact but claiming there's a cataract or
cancer is another story. As is the case in many instances of LLM use, the
systems also lack nuance, creating further opportunities for inaccuracy.

This article originally appeared on Engadget at https://www.engadget.com/gpt-
4-performed-close-to-the-level-of-expert-doctors-in-eye-assessments-
131517436.html?src=rss

---
VRSS v2.1.180528