Comparing the precision of the pSOFA and SIRS scores in predicting sepsis-related deaths among hospitalized children: a multi-center retrospective cohort study

Chun Zhao, Mei-yun Xin, Jing Li, Jin-fang Zhao, Yu-juan Wang, Wei Wang, Qian Gao, Jie Chen, Qi-wei Wang, You-peng Jin,

1Department of Pediatric Intensive Care Unit, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan 250021, China

2Department of Pediatrics, Affiliated Hospital of Jining Medical University, Jining 272129, China

3Department of Pediatrics, Qingdao Women and Children Hospital, Qingdao 266034, China

4Department of Pediatric Intensive Care Unit, Shandong Provincial Hospital, Cheeloo College of Medicine, Shandong University, Jinan 250021, China

Corresponding Author: You-peng Jin, Email: jinyp79@sina.cn

KEYWORDS: Simple-pSOFA; pSOFA; Children; Mortality; Infection; SIRS

INTRODUCTION

Children are highly susceptible to infections. Unfortunately, severe infections are associated with sepsis, multiple organ failure, and even death. Several studies have revealed an increase in sepsis prevalence amongst children.In fact, the year 2010 saw 64.0% (4.879 million) sepsis-related deaths among all childrelated (≤5 years) deaths.Moreover, children who survive sepsis often have a reduced quality of life.Based on a European investigation, 31% of pediatric patients with a history of sepsis developed disabilities.Likewise, in the SPROUT study, 17% of children who survived sepsis were found to develop at least one disability.Thus, it is imperative to detect sepsis early to enhance prognosis by off ering timely and appropriate infection therapy.

The historical sepsis descriptions, namely, the first and second definitions, employed the controversial SIRS criteria.The third and latest sepsis characterization in 2016, however, included the presence of both infection and multi-organ failure, as evidenced by the Sequential Organ Failure Assessment (SOFA) score. The SOFA score was previously validated as an independent risk factor of adult mortality associated with confirmed or suspected sepsis.But, the SOFA score was not normalized against age, thereby making it unsuitable for children.

At present, three independent pediatric SOFA (pSOFA) scores are available.In this research, we used the pSOFA scoring system described by Matics and Sanchez-Pinto in 2017,as it is the most comparable to the adult SOFA score.However, its capability in predicting in-hospital mortality needs further investigation. Moreover, this scoring methodology includes numerous test indicators, making this process arduous and costly. As such, the use of pSOFA in developing nations is limited.

Here, we aimed to compare the eff ectiveness of the pSOFA in estimating sepsis-related pediatric mortality in PICUs with SIRS and explore a convenient, economical and effective scoring method to predict the risk of sepsis-related death.

METHODS

Study design and population

PICUs belonging to three independent tertiary care hospitals within the Shandong Province joined the study. Due to the study design, formal consent was not required. The study protocol conformed to the ethical guidelines of the 1975 Declaration of Helsinki was approved by the local ethics committee of each participating hospital, namely, the medical research ethics review board of Shandong Provincial Hospital Affiliated to Shandong First Medical University (No. 2019142), the medical research ethics review board of Affiliated Hospital of Jining Medical University (No. 2020C019) and the medical research ethics review board of Qingdao Women and Children Hospital (No. QFELL-KY-202028).

We retrospectively analyzed the electronic health records of all PICU-admitted suspected or confirmed septic children between January 2011 and December 2019. Charts were reviewed by the hospital residents, and an overview of the medical data, pSOFA and SIRS scores, along with the evaluation of research eligibility was done by two senior physicians (Dr. YP Jin and Dr. C Zhao).

Study population eligibility was based on the following criteria: (1) age ＞28 days and ＜18 years, (2) PICU admission with confirmed or suspected infection, (3) adequate clinical information, (4) PICU length of stay (LOS) more than 24 h. Patients with deficiencies in clinical information or those transferred to other hospitals were eliminated. The flowchart of patient recruitment is showed in Figure 1.

We collected the following information from the first 24 h of PICU admission: age, sex, body weight, primary infection location, co-morbidities, and the highest and lowest cardio-respiratory, neurologic, hepatic, renal and hematological organ functional data points. Then, the SIRS and pSOFA scores were calculated. At the same time, we recorded the expenditure and time required to complete these scoring systems.

Evaluations of the scoring systems

Infection was assessed with a culture test, tissue stain, pathogen-specific polymerase chain reaction (PCR) examination, clinical examination, imaging, or laboratory test.

Scoring algorithms were employed to compute the 24 h-pSOFA and SIRS scores.Incomplete data were given a zero value. Patients without a prior history of organ dysfunction were given a baseline pSOFA score of zero; otherwise, an acute rise in the pSOFA score from up to 48 h before the infection to 24 h after the infection and who received antimicrobial therapy in the PICU was recorded.Moreover, the mechanical ventilation (MV) duration, PICU duration of stay, and sepsis-related deaths were monitored and documented.

Development of the spSOFA score

The simplified pSOFA (spSOFA) was developed by pSOFA through Youden index (YI). Three sub-categories with relatively good predictability were screened out from the six sub-categories of pSOFA scoring. The three sub-categories, namely, the coagulation based on platelet count, the age-dependent cardiovascular variables, and the Glasgow Coma Scale criteria for the neurologic subscore, were kept identical to the original score. We also recorded the cost and time required to complete the new scoring systems.

Statistical analysis

Data are expressed as either percentage (%), mean ± standard deviation, or median (interquartile range, IQR). The study population was assessed with descriptive analyses. Comparisons of continuous variables between groups were analyzed using Student’s t-test or Wilcoxon Mann-Whitney U-test, depending on the Shapiro-Wilk normality test-concluded data distribution. Comparisons of rates were conducted using Chi-square test.

We also performed univariate and multivariate analyses to establish risk factors for in-hospital deaths. Following the screening of risk factors, an area under the relative operating characteristic curve (AUROC) analysis was employed to calculate the predictive accuracy of each scoring system. The diagnostic value of the AUROC was stratified as follows: 0.90-1.00, excellent; 0.80-0.90, good; 0.70-0.80, fair; 0.60-0.70, poor; and 0.50-0.60, failure.

The diagnostic capacity of the pSOFA and SIRS scores was further evaluated with the Delong’s test (Medcalc software, Belgium). Additionally, sensitivity and specificity values were obtained from the AUROC. Next, an optimal threshold was selected based on the maximum YI.

SPSS Version 23.0 (IBM, USA) was employed for all data analyses, except for the Delong’s test. The level of significance was set as P-value ＜0.05.

RESULTS

Patient demographics

A total of 1,543 patients, aged between ＞28 days and ＜18 years, hospitalized due to diagnosed or yetto-be-diagnosed infection, between January 2011 and December 2019 were included in this study. Among them, 451 participants were eliminated: 69 due to hospital transfer and 382 due to missing medical information. Ultimately, 1,092 patients were eligible for this study (Figure 1). Among them, 669 (61.3%) were males, with a median age of 7.0 months (IQR 2.9-19.2 months). The respiratory tract was the most affected (57.2%), with subsequent infections in the bloodstream (13.5%) and cerebrum (12.1%). The total sepsis-related deaths at the PICU were 23.4% (Table 1). From the perspective of changes, the risk of PICU deaths elevated with the rise of the 24 h-pSOFA and SIRS score (Figure 2).

Figure 1. Flowchart of selection of patients with infection.

Comparing the demographic and clinical data between patients who survived and those who expired

No discernible differences were observed in the median age, gender, and surgical need of patients who survived sepsis versus those who did not. Non-survived patients had a higher prevalence of digestive tract and bloodstream infections, relative to those who survived (16.5% vs. 9.0%, P＜0.01; 17.6% vs. 12.2%, P＜0.05, respectively). In contrast, those who survived had a higher prevalence of respiratory tract infection than those who did not (60.0% vs. 48.2%, P=0.001). The median 24 h-pSOFA score for non-survived patients was markedly elevated, compared to sepsis survivors (6.0 vs. 3.0, P＜0.001). Although SIRS scores between the two groups were statistically diff erent, the mean values were similar that they were not clinically significant.

Among the infected children who expired, 94.1% underwent MV and 20.0% received continuous blood purification (CBP) therapy. This is in stark contrast to the low number of MV and CBP therapy among children who survived sepsis (P＜0.001). In addition, the non-survived patients required longer MV duration and shorter PICU stays than patients who survived sepsis (P＜0.001) (Table 1).

Figure 2. In-hospital mortality rate based on the SIRS criteria and the maximal pSOFA score.

Furthermore, we conducted univariate analysis and screened out statistically significant variables as candidate for multivariate analysis (Supplementar y Tables 1 and 2). Multivariate analysis demonstrated that an elevated 24 h-pSOFA value, along with bloodstream infections and MV requirement increased the sepsisrelated PICU mortality (Table 2).

Comparing SIRS versus pSOFA

The 24 h-pSOFA had far greater predictive power (AUROC=0.740, 95% CI: 0.706-0.774) in estimating sepsis-related PICU deaths than SIRS (AUROC=0.559, 95% CI: 0.519-0.600). Moreover, based on the max YI analysis, the best threshold for mortality estimation of the 24 h-pSOFA was ≥5, whereas for SIRS, it was ≥3. To conduct our sensitivity and specifciity analysis, we adjusted 3 thresholds for the 24 h-pSOFA scores, namely ≥2, 4 and 5, and 2 thresholds for the SIRS scores, namely, ≥2 and 3. Based on these values, the sensitivity was highest for 24 h-pSOFA≥2 (97.6%), while the specifciity was best for 24 h-pSOFA≥5 (69.2%) (Table 3).

Comparing spSOFA with SIRS and pSOFA

To conduct a deeper comparative investigation with SIRS, we independently computed the predictability of each pSOFA sub-category (Supplementary Table 1). The negative predictive value (NPV) for a neurologic score of ≥1 was 94.23%, whereas the positive predictive value (PPV) for a hepatic score of ≥4 was 99.6%. To further enhance the predictive power of the pSOFA score, coagulation, cardiovascular, and neurologic variables with relatively good predictability, as assessed by the YI, were employed for the spSOFA scoring system (Table 4).

The spSOFA AUROC (AUROC=0.751, 95% CI: 0.716-0.786) was markedly elevated, compared to the SIRS (AUROC=0.559, 95% CI: 0.519-0.600) (P=0.004) and 24 h-pSOFA (AUROC=0.740, 95% CI: 0.706-0.774) (P＜0.001) (Table 3, Figure 3). The sensitivity, specificity, NPV, and PPV of the spSOFA score were showed in table 3, using multiple thresholds. Among them, a threshold of 3 provided optimal parameter values at 65.9%, 74.6%, 87.8%, and 44.1%, respectively. Compared with 24 h-pSOFA≥5, the sensitivity of spSOFA was slightly less, but its specificity was better (Table 3).

Table 1. Demographic and clinical characteristics of survivors and non-survivors with infectious disease

Table 2. Multivariable logistic regression to analyze the risk factors of in-hospital mortality

Time and cost required to complete SIRS, 24 h-pSOFA and spSOFA scoring

To further elucidate the suitability of the 24 h-pSOFA, SIRS and spSOFA scoring systems in PICUs, we compared the expenditure and time associated with these scoring systems. We observed that it took significantly longer time to complete the 24 h-pSOFA scoring system (154 s) than to complete the SIRS scoring scale (27 s) and spSOFA scoring (85 s). In addition, the cost of 24 h-pSOFA was the highest (607 CNY), followed by spSOFA (51 CNY) and SIRS (37 CNY) (Supplementary Table 3).

DISCUSSION

Among the 1,092 patients examined, the most common infectious site was the lung, followed by the bloodstream and cerebrum. Moreover, in-hospital deaths reached as high as 23.4%. In addition, using multivariate analysis, we revealed that an elevated pSOFA value, MV requirement, and infected bloodstream were critical factors driving increased risk of in-hospital deaths. We analyzed the effectiveness of the pSOFA score and SIRS in estimating sepsis-related PICU deaths. We also assessed the feasibility of adopting the pSOFA score in three PICUs. Relative to SIRS, our multi-center cohort study showed that the 24 h-pSOFA exhibited superior prognostic precision in estimating in-hospital deaths. The disadvantage of 24 h-pSOFA is time-consuming and expensive, so we proposed spSOFA. Our study showed that the latter reduced the time/economic cost, and had a better predictive value for the mortality of children with sepsis.

Figure 3. AUROC curve of SIRS, 24 h-pSOFA and simple-pSOFA for in-hospital mortality in patients with suspected infection.

Table 3. The predictive value of SIRS, pSOFA and simple-pSOFA for in-hospital mortality of infected children

Table 4. Simple-pediatric sequential organ failure assessment score

To date, SIRS was used for almost two decades to define sepsis in pediatric patients. However, the secondary definition of pediatric sepsis only included systemic inflammation,but without organ failure. A prior report suggested that SIRS was only met in ＞90% of febrile children who sought emergency care, and among them, only ＜5% received ICU admission.Although SIRS has remarkable sensitivity, it has a rather poor specificity toward sepsis. Moreover, Kaukonen et alreported SIRS to be inadequate in predicting ICU deaths, particularly, due to its exclusion of some infected patients, some with organ failure, and some who eventually died. Using the ROC curve analysis, we demonstrated that the sensitivity and specificity of SIRS in estimating sepsis-related PICU deaths were 74.1% and 32.6%, respectively. In a previous study involving children with infected bloodstreams, SIRS predicted the 30-d mortality, based on bacteremia analysis, to be 1%. However, when organ failure was included in the investigation, the 30-d mortality rose to 17%.It is crucial to define “infection” and “sepsis” properly. Unlike an infection, sepsis involves organ failure.In contrast to the SIRS scoring system, the SOFA scoring system off er more precision in predicting ICU mortality.Moreover, the comprehensiveness of the third and latest definition of sepsis, involves the inclusion of multi-organ failures, automatically raising SOFA scoring by ≥ 2 points.The 24 h-pSOFA score, in our analysis, were much better in predicting the death of hospitalized infected children, compared to SIRS.

Based on the YI, we adjusted the best 24 h-pSOFA threshold value to predict in-hospital deaths at ≥5, which is diff erent from the previous study, where the threshold was set at ＞8 in critically ill children.Moreover, the pSOFA computation provided the maximum pSOFA score throughout the hospital stay, which they deemed better than the 24 h-pSOFA score in predicting the death of hospitalized infected children.However, pSOFA in the whole hospitalization process is difficult to achieve for most PICU in Shandong Province. We, therefore, ascertained that the 24 h-pSOFA score might be more suitable for pediatric patients in China.

Although the 24 h-pSOFA score is much more effective in predicting in-hospital mortality than SIRS, SIRS is primarily used in emergency departments in developing countries due to its relatively low cost and less time consumption. Ideally, the optimal scoring system should include convenience and rapid turnout, high sensitivity and specificity, and low medical cost. Hence, we combined both pSOFA and SIRS scoring techniques to develop a novel method termed spSOFA. In doing so, we were able to take full advantage and eliminate the disadvantages of both prior scoring systems while enhancing the mortality prediction within the hospitalized infected children population.

To score the spSOFA, the Glasgow coma score, mean arterial pressure and platelet count are needed. It is important to note that the spSOFA score does not alter the detection item requirement. Among the three scoring systems mentioned in this paper, the spSOFA score exhibited the largest AUROC, indicating that it might be the optimal in detecting the risk of sepsis-related PICU deaths. The optimal threshold of spSOFA was ≥3 in our study, and its sensitivity was slightly lower than that of 24 h-pSOFA≥5, but its specificity was better. So spSOFA not only reduces cost, but also offers satisfactory estimations of sepsis-related PICU deaths.

Limitations

Our work had certain limitations. Firstly, we collected data from three independent PICUs. However, non-PICU patients were not studied. To examine the accuracy of the pSOFA score, all settings and populations should be included. Secondly, we only analyzed the pSOFA score from the first 24 h of hospital admission. Computing both the SIRS and pSOFA scores need the highest and lowest recorded values obtained during the hospitalization. However, we used the information from the first 24 h, and this led to a selection bias. Thirdly, our work was restricted by its retrospective design, which may have introduced both selection and recall bias. Finally, our sample size was relatively small, particularly during the screening of the spSOFA score. Hence, enough data were not available to draw objective conclusions for some parameters. It is, therefore, crucial to perform a study with a large sample population to validate our results.

CONCLUSIONS

We demonstrated that the pSOFA score was better than SIRS in estimating infected patient in-hospital mortality. However, its application is restricted due to its lengthy and expensive process, particularly in low- and middle-income regions. Our goal is to aid emergency departments and primary hospitals in the low- and middle-income nations in the early detection of infected pediatric patients at a high risk of death, without imposing a financial burden on their families. So we propose a new scoring system, spSOFA, which is both quick and inexpensive, with relatively good predictive value in estimating in-hospital sepsis-related deaths.

None.

The study was approved by the local ethics committee of each participating hospital.

All authors declare that they have no conflicts of interest with the contents of this article.

CZ and MYX contributed equally to this work. All authors take responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.

All the supplementary files in this paper are available at http://wjem.com.cn.