October 9, 2025Open Access

Automating Evaluation of LLM-generated Responses to Patient Questions about Rare Diseases

View Full Paper

Ask AI

Bookmark

LLM-based evaluation showed stronger alignment (ρ=0.758) with physician scores compared to traditional NLP metrics.
The reference-guided evaluation using GPT-4 achieved the highest correlation with physician-assigned scores at 0.758.
Traditional NLP metrics provided weak correlations (ρ=0.240-0.403) with physician scores, indicating limitations in assessing response accuracy.
Reference-guided scoring methods demonstrate a scalable approach for evaluating LLM-generated responses about rare diseases.

Authors

MZMin Zhao IOInez Y. OhWashington University in St. Louis AGAditi Gupta

Discussion

No takes yet. Share an insight, caveat, or question.

Implication

Automated evaluation outperforms traditional NLP metrics in assessing chatbot responses, suggesting better alignment with expert judgment for rare disease inquiries.

Cite This Study

Zhao et al. (2025) studied this question.

synapsesocial.com/papers/68e70da790569dd607ee59fc https://doi.org/10.1101/2025.10.06.25337181

View Full Paper

Ask AI

Bookmark

Implication

Automated evaluation outperforms traditional NLP metrics in assessing chatbot responses, suggesting better alignment with expert judgment for rare disease inquiries.

Automating Evaluation of LLM-generated Responses to Patient Questions about Rare Diseases

Key Points

Discussion

Implication

Cite This Study

Automating Evaluation of LLM-generated Responses to Patient Questions about Rare Diseases

Key Points

Discussion

Implication

Cite This Study

Also Consider

Also Consider