(386) Highly Reliable Data Collection by a Large Language Model in Fetal Ultrasound Reports

To develop a reliable and efficient data collection process for unstructured fetal ultrasound reports involving single- and higher-order gestations. At our institution, documentation of middle cerebral artery (MCA) Doppler often includes more than one value and range per fetus, contributing to variability in report structure and complexity. We evaluated the performance of an off-the-shelf Large Language Model (LLM) integrated into our Clinical Outcomes Data Archive (CODA) platform, aiming to reduce human effort while maintaining high data fidelity. 

Study Design:

A subset of fetal ultrasound reports was processed using an LLM in CODA. For each fetus, we abstracted whether an MCA Doppler was performed and extracted the peak systolic velocity (PSV), and up to two multiple-of-the-median (MoM) values when a range was reported. All outputs were reviewed by a human validator. Discrepancies were recorded, and agreement between the LLM and final validated values were calculated for each variable. 

Results:

Of 494 fetal ultrasound reports processed using an LLM in CODA, the LLM correctly identified 349 reports that had at least one MCA Doppler study. These reports included 630 fetus-level evaluations, yielding 1,249 datapoints for analysis. Among 244 studies ultimately deemed not performed, the LLM incorrectly labeled 10 as performed, though it accurately marked all three study values as missing in these cases. All three target values for the MCA Doppler parameters performed reached >99% accuracy. 

Conclusion:

An off-the-shelf LLM integrated into the CODA platform demonstrated high reliability in abstracting structured data from unstructured fetal ultrasound reports. The model achieved >99% accuracy for all extracted MCA Doppler values and maintained perfect performance in marking missing data. These findings support the feasibility of LLM-assisted abstraction to reduce manual workload while preserving data quality in complex clinical documentation, including higher-order gestations.