|
Researchers from the Jamieson group, in collaboration with the UT Southwestern Simulation Center, are using advanced artificial intelligence tools to analyze, grade, and provide swift evaluations for medical students’ Objective Structured Clinical Examination (OSCE), a standard approach to measuring student competence through a live-action, simulated patient encounter with an actor.
In the study, the researchers collected notes from over 1,000 UT Southwestern medical students’ OSCE exams between 2019 and 2023, yielding 10,000 exam notes. Typically, these notes are assessed by a specially trained standard patient evaluator. The researchers tested whether an advanced large language model, Chat GTP-4, could do the same rubric-based grading without requiring any prior training data or labels, a technique known as "zero-shot". They dub this approach “Rubrics to Prompts”.
Using secure model access via UT Southwestern’s Azure cloud, the researchers fed exam notes into the AI and requested performance assessments. They compared the GPT scores with the human evaluators and found that zero-shot GPT-4 achieved 89.7% agreement compared to standard patient evaluators and a correlation of 0.86 with total score for the Fall 2022 OSCE prior to the inaugural deployment.
The AI evaluation process has transformed OSCE feedback since the fall 2023 student cohort, for whom AI replaced more than 91% of the human grading efforts and delivered their results within days, instead of the typical weeks required to complete evaluations.
UT Southwestern is uniquely equipped to develop tools like this. Our Simulation Center is one of the largest in the nation and gathers rich sets of data from video and audio recordings and encounter notes.
“As an innovative school, it’s important we develop the skills on campus to work deftly with AI so it can complement our projects,” said Andrew Jamieson, Ph.D., Assistant Professor in the Lyda Hill Department of Bioinformatics. “If we are not ahead, we’re behind.”
Moving forward, the Jamieson group and Simulation Center will work on how to use AI to deliver narrative feedback to students, beyond a simple score. Additionally, they are prototyping systems that can analyze recorded videos of the students' exam. Overall, such efforts are leading to more efficient, standardized, and impactful performance evaluation processes for our learners.
The Jamieson Lab is now seeking opportunities to leverage their expertise gained through this research to assist in AI implementation projects across campus.
Contact andrew.jamieson@utsouthwestern.edu
| 
