Prediction models for the coronavirus (COVID-19) are being developed worldwide: which people are more at risk of getting it, which patients with complaints have it and which characteristics and test results determine the course of the disease? A study into 31 models underlines the need to share more data between them. “Most prediction models are not yet scientifically reliable enough to base medical decisions on.”
Researchers Laure Wynants of Maastricht University and Maarten van Smeden of the UMC Utrecht are worried. Together with a group of international researchers, they assessed all prediction models thus far available for the early detection and course of COVID-19. They conclude that some models contain important information for healthcare professionals, but most models are as yet based on insufficient scientific evidence. Their publication has been published in the leading medical science journal British Medical Journal*.
Reliable prediction models
“Reliable prediction models are desperately needed," says Maarten. “General practitioners and specialists in hospitals are now using different prediction models under high pressure. Which people are at greater risk of contracting COVID-19? Which patients with disease symptoms actually have a COVID-19 infection? And what is the expected course of the disease in patients with COVID-19? In concrete terms, you can think of a model that estimates the chance that a 35-year-old man with shortness of breath actually has a COVID-19 infection. Sometimes it takes too long before you, as a doctor, get a test result and you want to be sure sooner. In such a situation, prediction models come into the picture, but they are not yet sufficiently accurate to base a medical decision on.”
National healthcare systems
These types of prediction models are not only important to the individual patient but to the care for patients as a whole. Laure says, “COVID-19 is an acute threat to global public health, with numbers of infections and deaths rising daily. Since the outbreak at the end of last year, the pandemic has threatened to overload virtually all national healthcare systems.” For this reason, early detection and prediction of the course of the disease are essential for the proper prevention, diagnosis and therapy of patients. The more targeted the use, the more efficient the care as a whole. Laure says, “For example, you want to prevent patients ending up in hospital unnecessarily, but also their being wrongly sent home and having to be admitted later on.”
The researchers examined 31 prediction models from 27 international studies, most of which (25) from China. The data for these studies was collected between December 8, 2019 and March 15, 2020. Many of these studies have serious shortcomings. “Non-representative control patients, for example," says Maarten, "and limited data sets.” The latter may be a consequence of the rush to come up with a prediction model. In view of the emergency situation, you can hardly expect otherwise, he thinks, but this does not alter the fact that the result is scientifically irresponsible. “There was a study that examined how long patients stayed in hospital. When the study was completed after fourteen days, the patients who were still hospitalized were removed from the study.” Logical consequence: the model based on this study underestimates how long COVID-19 patients can stay in hospital.
Call upon developers
The research by Maarten and colleagues did reveal a number of predictive factors that are important for care and care policy. Laure says, “Age is such a factor, as are gender and certain lab values such as C-reactive protein and lactate dehydrogenase.” Nevertheless, the prediction models are on the whole based on insufficient scientific evidence. In order to achieve an improvement quickly, more data is needed to begin with. Therefore, the researchers call upon developers of prediction models to immediately share data publicly: “Share COVID-19 patient data, because only then can we develop, test and apply reliable and widely useable prediction models in day-to-day practice. An online place coordinated by an organization such as the WHO is desperately needed, the researchers believe.
The researchers continue the current research as a so-called living review, with biweekly updates in the British Medical Journal. The aim is to continuously provide healthcare professionals and policymakers with up-to-date information on the quality of COVID-19 prediction models. They will add new available models to their study and critically evaluate them as they update the study. They review not only the official publications of predictive models in scientific journals, but especially studies that are only available on the internet in so-called 'pre-print' archives, because the models from these pre-prints are often already available for use by healthcare professionals.
* The publication in the British Medical Journal has been provided by a consortium of international researchers, led by Laure Wynants of the Care and Public Health Research Institute of Maastricht University and Maarten van Smeden of the Julius Center of the UMC Utrecht.