Cited 0 times in Scipus Cited Count

Implementation of the COVID-19 vulnerability index across an international network of health care data sets: Collaborative external validation study

Authors
Reps, JM | Kim, C | Williams, RD | Markus, AF | Yang, C | Duarte-Salles, T | Falconer, T | Jonnagaddala, J | Williams, A | Fernández-Bertolín, S | DuVall, SL | Kostka, K | Rao, G | Shoaibi, A | Ostropolets, A | Spotnitz, ME | Zhang, L | Casajust, P | Steyerberg, EW | Nyberg, F | Kaas-Hansen, BS | Choi, YH  | Morales, D | Liaw, ST | Fernandes Abrahão, MT | Areia, C | Matheny, ME | Lynch, KE | Aragón, M | Park, RW  | Hripcsak, G | Reich, CG | Suchard, MA | You, SC | Ryan, PB | Prieto-Alhambra, D | Rijnbeek, PR
Citation
JMIR medical informatics, 9(4). : e21547-e21547, 2021
Journal Title
JMIR medical informatics
ISSN
2291-9694
Abstract
Background: SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who require hospitalization from those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision-making during the pandemic. However, the model is at high risk of bias according to the "prediction model risk of bias assessment" criteria, and it has not been externally validated. Objective: The aim of this study was to externally validate the C-19 index across a range of health care settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases. Methods: We followed the Observational Health Data Sciences and Informatics (OHDSI) framework for external validation to assess the reliability of the C-19 index. We evaluated the model on two different target populations, 41,381 patients who presented with SARS-CoV-2 at an outpatient or emergency department visit and 9,429,285 patients who presented with influenza or related symptoms during an outpatient or emergency department visit, to predict their risk of hospitalization with pneumonia during the following 0-30 days. In total, we validated the model across a network of 14 databases spanning the United States, Europe, Australia, and Asia. Results: The internal validation performance of the C-19 index had a C statistic of 0.73, and the calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data, the model obtained C statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US, and South Korean data sets, respectively. The calibration was poor, with the model underestimating risk. When validated on 12 data sets containing influenza patients across the OHDSI network, the C statistics ranged between 0.40 and 0.68. Conclusions: Our results show that the discriminative performance of the C-19 index model is low for influenza cohorts and even worse among patients with COVID-19 in the United States, Spain, and South Korea. These results suggest that C-19 should not be used to aid decision-making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.
Keywords

DOI
10.2196/21547
PMID
33661754
Appears in Collections:
Journal Papers > School of Medicine / Graduate School of Medicine > Infectious Diseases
Journal Papers > School of Medicine / Graduate School of Medicine > Biomedical Informatics
Ajou Authors
박, 래웅  |  최, 영화
Full Text Link
Files in This Item:
There are no files associated with this item.
Export

qrcode

해당 아이템을 이메일로 공유하기 원하시면 인증을 거치시기 바랍니다.

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Browse