Performance of Large Language Models in Answering Juvenile Systemic Lupus Erythematosus Questions: A Blinded Expert-based Comparative Study
Journal of Clinical Rheumatology, 2026 (SCI-Expanded, Scopus)
- Yayın Türü: Makale / Tam Makale
- Basım Tarihi: 2026
- Doi Numarası: 10.1097/rhu.0000000000002411
- Dergi Adı: Journal of Clinical Rheumatology
- Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, CINAHL, EMBASE, MEDLINE
- Anahtar Kelimeler: artificial intelligence, large language models, pediatrics, rheumatology, systemic lupus erythematosus
- İstanbul Üniversitesi-Cerrahpaşa Adresli: Evet
Özet
Background: – Large language models are increasingly used to obtain medical information, but their performance in juvenile systemic lupus erythematosus remains insufficiently characterized. Objective: – To compare expert-rated response quality and clinical relevance across 4 artificial intelligence (AI) models using standardized juvenile systemic lupus erythematosus–related questions. Design: – Cross-sectional expert-based comparative study. Setting: – Freely accessible consumer-facing web interfaces evaluated on December 30, 2025. Participants: – Ten pediatric rheumatology experts. Intervention: – Not applicable. Main Outcome Measures: – Twenty standardized questions generated 80 AI responses, rated on a 5-point Likert Scale. Model performance was compared using the Friedman test with Kendall W and Bonferroni-corrected post hoc analyses; interrater agreement was assessed using intraclass correlation coefficients (ICCs). Results: – DeepSeek achieved the highest median rating [5 (1 to 5)], followed by Gemini [4 (2 to 5)], ChatGPT [4 (3 to 5)], and Copilot [4 (3 to 5)]. Overall expert-rated performance differed significantly among models [Friedman χ2(3) = 63.553, p < 0.001; Kendall W = 0.106]. DeepSeek outperformed ChatGPT, Gemini, and Copilot (all p < 0.001), while Gemini performed better than ChatGPT (p = 0.007) and Copilot (p < 0.001). Interrater agreement was highest for DeepSeek (ICC =0 .734) and lowest for Copilot (ICC = 0.043). Trust scores remained unchanged before and after evaluation [4 (3 to 5) vs. 4 (3 to 5), p=0.414]. Limitation: – Single-time-point, single-response testing did not assess within-model reproducibility, and the question set was not formally validated. Conclusion: – AI models differed in expert-rated performance and interrater agreement; outputs should be interpreted cautiously and with specialist oversight.