September 5, 2025JMIRx MedOpen Access

Rapidly Benchmarking Large Language Models for Diagnosing Comorbid Patients: Comparative Study Leveraging the LLM-as-a-Judge Method

View Full Paper

Ask AI

Bookmark

Authors

Discussion

Comparative study assesses diagnostic abilities of 21 large language models, highlighting LLM-as-a-judge method and retrieval-augmented generation.

Gemini 2.5 achieved the highest diagnostic hit rate of 97.4%, outperforming other large language models.
The analysis involved 18 LLMs on 1000 MIMIC-IV hospital admissions, using prompts and temperature settings.
Retrieval-augmented generation improved the hit rate of GPT-4o 05-13 by an average of 0.8%, indicating better diagnostic accuracy.
Significant model performance variation was noted across different prompts, suggesting the need for diverse datasets.

Sarvari et al. (2025) studied this question.

View Full Paper

Ask AI

Bookmark

September 5, 2025JMIRx MedOpen Access

View Full Paper

Ask AI

Bookmark

Authors

Comparative study assesses diagnostic abilities of 21 large language models, highlighting LLM-as-a-judge method and retrieval-augmented generation.

Gemini 2.5 achieved the highest diagnostic hit rate of 97.4%, outperforming other large language models.
The analysis involved 18 LLMs on 1000 MIMIC-IV hospital admissions, using prompts and temperature settings.
Retrieval-augmented generation improved the hit rate of GPT-4o 05-13 by an average of 0.8%, indicating better diagnostic accuracy.
Significant model performance variation was noted across different prompts, suggesting the need for diverse datasets.

Sarvari et al. (2025) studied this question.

View Full Paper

Ask AI

Bookmark