Validation of different pediatric triage systems in the emergency department

Kanokwan Aeimchanbanjong, Uthen Pandee

Department of Pediatrics, Faculty of Medicine Ramathibodi Hospital, Mahidol University, Bangkok, Thailand

Validation of different pediatric triage systems in the emergency department

Kanokwan Aeimchanbanjong, Uthen Pandee

Department of Pediatrics, Faculty of Medicine Ramathibodi Hospital, Mahidol University, Bangkok, Thailand

BACKGROUND: Triage system in children seems to be more challenging compared to adults because of their different response to physiological and psychosocial stressors. This study aimed to determine the best triage system in the pediatric emergency department.

METHODS: This was a prospective observational study. This study was divided into two phases.The fi rst phase determined the inter-rater reliability of fi ve triage systems: Manchester Triage System(MTS), Emergency Severity Index (ESI) version 4, Pediatric Canadian Triage and Acuity Scale (CTAS),Australasian Triage Scale (ATS), and Ramathibodi Triage System (RTS) by triage nurses and pediatric residents. In the second phase, to analyze the validity of each triage system, patients were categorized as two groups, i.e., high acuity patients (triage level 1, 2) and low acuity patients (triage level 3, 4, and 5). Then we compared the triage acuity with actual admission.

RESULTS: In phase I, RTS illustrated almost perfect inter-rater reliability with kappa of 1.0(P＜0.01). ESI and CTAS illustrated good inter-rater reliability with kappa of 0.8–0.9 (P＜0.01).Meanwhile, ATS and MTS illustrated moderate to good inter-rater reliability with kappa of 0.5–0.7(P＜0.01). In phase II, we included 1 041 participants with average age of 4.7±4.2 years, of which 55%were male and 45% were female. In addition 32% of the participants had underlying diseases, and 123(11.8%) patients were admitted. We found that ESI illustrated the most appropriate predicting ability for admission with sensitivity of 52%, specif i city of 81%, and AUC 0.78 (95%CI 0.74–0.81).

CONCLUSION: RTS illustrated almost perfect inter-rater reliability. Meanwhile, ESI and CTAS illustrated good inter-rater reliability. Finally, ESI illustrated the appropriate validity for triage system.

Triage; Pediatric; Emergency department

INTRODUCTION

Overcrowded emergency department (ED) leads to a delay in treatment of critically ill patients, increasing the rate of morbidity and mortality.[1]Therefore, the triage system in emergency care is an important tool to prioritize seriously ill children.[2]Triage system in children seems to be more challenging compared to adults because of their different response to physiological and psychosocial stressors.[3,4]There are four international triage systems:[5]Manchester Triage System (MTS), Emergency Severity Index (ESI) version 4, Pediatric Canadian Triage and Acuity Scale (CTAS), and Australasian Triage Scale(ATS), have been using for triage in children at ED.

However, Faculty of Medicine Ramathibodi Hospital,Bangkok, Thailand, has invented and used its own triage system, which is called "Ramathibodi Triage System (RTS)".Up till now, no study has compared four international pediatric triage systems in emergency department in the same group population. Moreover, Thailand has no standard pediatric triage system. As a result, this leads to our objective to find the best triage system in predicting patient acuity.Nevertheless, there is no gold standard for decision of urgency of treatment. Therefore, our study used admission to imply severity of illness and urgency of treatment. The objective of this study were to assess and compare the reliability of 5 triage systems (CTAS, ESI, MTS, ATS and RTS), as well as its validity for predicting hospital admission in a prospective cohort of pediatric patients at emergency department.

METHODS

Study design

This study was conducted as prospective, single-center,and observational study to determine the reliability and validity of CTAS, ESI, MTS, ATS and RTS. The medical ethics committee of Mahidol University approved this study and informed consent was obtained from all the participants.

Study setting and population

This study was performed at Ramathibodi Hospital(tertiary care, university hospital with -hour ED services)with approximately 10 000 pediatric patients per annual.Ramathibodi Hospital has used RTS for more than a decade. Data were collected in all children aged under fifteen years old presenting to the ED from the period of April to July 2015 at Ramathibodi Hospital, Mahidol University. All pediatric triage nurses and pediatric residents in the study have been formally trained how to apply all triage systems. In this study, patients presented with trauma and incomplete records were excluded.

Triage system

Each triage system used chief complaint and physiological parameters such as body temperature, heart rate, respiratory rate, oxygen saturation, blood pressure,work of breathing, and capillary refill in prediction of illness's severity. Each triage system has five levels of severity. Level 1 triage is the most emergency treatment,whereas level 5 triage is the least emergency treatment.

Manchester Triage System (MTS)[6–11]

MTS was fi rst used in the England. It has 52 fl owcharts representing a chief complaint, of which 49 flowcharts are suitable for children. Based on the flowcharts, general discriminators such as life threatening conditions and conscious level are considered. Then selected discriminator identif i es an urgency level. Medical care should be delivered immediately for level 1, within 10 minutes for level 2, within 60 minutes for level 3, within 120 minutes for level 4 and within 240 minutes for level 5.

Emergency Severity Index version 4 (ESI)[12–13]

ESI was developed in the United States with fi ve levels.Patients requiring immediate life-saving interventions and must be seen immediately are level 1. Patients in high risk conditions, who are confused, lethargic, disoriented, having severe pain, distress and highly abnormal vital signs and should be seen within ten minutes, are level 2. Level 3 is for patients who are expected to require two or more resources, which can be diagnostics in term of laboratory investigations or electrocardiogram. Level 4 is for patient who are expected to require one resource, and level 5 if no resources are expected to be required.[2]The specific fl owchart for children with fever was added in the fourth version of ESI.[2]

Canadian Triage and Acuity Scale (CTAS)[14–19]

CTAS is based on a list of patients presenting complaints with fi rst- and second-order modifiers for specif i c conditions.Its principle operational objective determines the time for the patient's initial assessment by a physician.[3]Medical care should be delivered immediately for level 1, within 15 minutes for level 2, within 30 minutes for level 3, within 60 minutes for level 4 and within 120 minutes for level 5.[4]

Australasian Triage Scale (ATS)[20–22]

ATS can be divided into primary and secondary triage decisions. Primary triage decisions are based on the triage assessment, allocation of a triage category and patient deposition. Meanwhile secondary triage decisions are based on the initiation of nursing interventions for emergency care and promote patient comfort.[2]Medical care should be delivered immediately for level 1, within 10 minutes for level 2, within 30 minutes for level 3, within 60 minutes for level 4 and within 120 minutes for level 5.

Ramathibodi Triage System (RTS)[23]

RTS has five levels. It uses computer for patient's categorization, which are based on chief complaint and vital signs. Criteria of each level of triage were constructed by experts and specialists.[5]Medical care should be delivered immediately for level 1, within 30 minutes for level 2 and non-urgency for level 3, 4, 5.

Admission

Decision of admission were made by proxy of severity of illness and based on the decision of each pediatric resident in the department.

Study protocol

This study was conducted in two phases. The first phase of the study was to determine the inter-rater reliability between each triage. Two triage pediatric residents and two nurses were blindly asked to determine the triage level in each patient for the fi rst twenty patients.Inter-rater reliability between pediatric triage nurses and pediatric residents, among pediatric residents and between the nurses were measured using Kappa statistics.

After ensuring good inter-rater reliability (＞0.70),the second phase of the study was started. The fi ve triage systems were conducted by pediatric triage nurses and pediatric residents during each shift. To analyze validity of each triage system, patients were divided into high acuity (triage level 1, 2) and low acuity (triage level 3, 4, 5). Then the levels of acuity were compared with admission, which implied the severity of illness and urgency of treatment. Validity was measured using area under the receiver operating characteristics ROC curves.

Data analysis Sample size

Sample size was calculated from the probability of expected sensitivity of 0.63 in the previous study.[7]Therefore, our sample size of 920 patients was suff i cient to validate the five triage systems.

Statistical analysis

All data were collected and reviewed by both authors.All statistical analyses were performed using SPSS version 17.0 (IBM corporation, Armonk, New York).

RESULTS

In phase I, RTS illustrated perfect inter-rater reliability with kappa of 1 (P＜0.01). ESI and CTAS illustrated good inter-rater reliability with kappa of 0.8–0.9 (P＜0.01). ATS and RTS illustrated moderate to good inter-rater reliability with kappa of 0.5–0.7 (P＜0.01).These results were shown in Table 1.

Phase II illustrated that ESI was the most appropriate predictor of admission and urgency treatment. A total of 1 103 patients were enrolled during the study period.We excluded 62 patients due to incomplete record and exclusion criteria. Thus, 1 041 participants with average age of 4.7±4.2 years were included for validation analyses,of which 55% were male and 45% were female. In addition 32% of the participants had underlying diseases,and 123 (11.8%) patients were admitted. Diagnoses were categorized according to the systems and were summarized along with patient demographic data as illustrated in Table 2. The second phase determined the predicting ability of each triage system using area under the receiver operating characteristics curves (AUC), sensitivity and specif i city as shown in Table 3 and Figure 1.

Table 1. Inter-rater reliability with Kappa analysis

Table 2. Patient characteristics

Table 3. Sensitivity, specif i city and ROC curve of fi ve triage systems

Figure 1. ROC curves of fi ve triage systems.

We found that ESI illustrated the most appropriate predicting ability for admission, i.e., AUC 0.78 (95%CI 0.74–0.81), sensitivity of 52% and specificity of 81%.MTS illustrated AUC 0.7 (95%CI 0.66–0.74), sensitivity of 57% and specificity of 69%. CTAS illustrated AUC 0.64 (95%CI 0.59–0.70), sensitivity of 50% and specificity of 74%. ATS illustrated AUC 0.73 (95%CI 0.69–0.77), sensitivity of 13% and specificity of 94%.Finally, RTS illustrated AUC 0.66 (95%CI 0.60–0.70),sensitivity of 64% and specif i city of 64%.

DISCUSSION

Four triage systems are extensively used to triage children in emergency department across different part of the world. Several studies were performed to assess the reliability and validity of these systems in children. The best triage was the one that can accurately detect urgency of treatment, leading to decreases in overcrowded emergency department, morbidity and mortality.[24–25]

Study on reliability of pediatric triage systems using inter-rater agreement analyzed by Cohen's kappa.The interpretations of kappa (K) are: poor if K＜0.20,fair if 0.21＜K＜0.40, moderate if 0.40＜K＜0.60, good if 0.61＜K＜0.80 and very good if K＞0.80 (95% confidence interval).[26]Most of the studies using written case scenarios for testing triage reliability. The studies of ESI[12–13]seem to be the most reliability which has very good interrater agreement (kappa 0.84–1). MTS study from the Netherlands[8]has a good reliability, kappa 0.62–0.83.CTAS study[15]found that the reliability was moderate(kappa 0.51–0.61). The study of ATS from Australia[22]has poor to moderate reliability (kappa 0.21–0.4) respectively.The results of our study showed the same trend. The most reliability of different five triage systems are very good for RTS (kappa 1) and ESI (kappa 0.81–0.90), good for CTAS(kappa 0.72–0.82), moderate for MTS (kappa 0.56–0.72)and moderate for ATS (kappa 0.55–0.69) with better interrater reliability compared to previous study.[22]Computerized triage systems showed a better agreement compared to the triage without the computerized system, and resulted in better triage reliability.[27]This may explain our result of a perfect inter-rater reliability of RTS (kappa 1), which using computerized application for triage in the study compared to the other triage systems without computerized application.Therefore, a computerized application of triage system should be used for the reliability of triage.

Since the measurements used for validity of studies are different, a comparison between triage systems cannot be made on how they predict the "true" urgency. Different approaches are used to assess validity of triage. The outcome measurement such as ICU admission, hospitalization,resource uses, length of stay in ED or the cost of ED consultation are used.[13,15]However, from the previous studies and the design of the triage systems, some important points can be made. Study on outcome measurement by using admission rate found that both ESI[13]and CTAS[15]triage system can predict the admission and have a good correlation between urgency and admission. One study showed the other aspect of outcome measurement by comparison of three triage systems [Informally Structured Triage System (ISS), ESI and MTS] with the reference standard (vital signs, diagnosis, resource use, admission rate, and follow-up) and found that they were equally valid.However, the ESI showed the highest percentage of under triage whereas the ISS showed the lowest percentage of undertriage.[28]Another study compared the MTS with the reference standard, showing sensitivity of 63% (95%CI 59–66) and specif i city of 79% (95%CI 79–80).[8]Our study was different from other studies as we compared between each triage and admission (implied urgency treatment). However,the measurement of outcome was similar to previous studies.In particular, our study showed the sensitivity and specif i city of MTS of 57% and 69%, respectively. In overall, the results of our study showed that ESI expressed the best validity followed by ATS, MTS, RTS and CTAS based on outcome measurement by using triage urgency for prediction of admission respectively.

Limitations

Our study had some limitations which were quite similar to other studies.[25]Now there is no gold standard for decision of urgency of treatment. Therefore, our study used admission to imply severity of illness and urgency of treatment. The under or over triage categorized maybe the appropriate outcome measurement in the further study. The other limitation was that our study investigated only at a single pediatric hospital, and the results may not be generalizable to other institutions.

Area of future research

We think it would be important for future studies to investigate multiple pediatric hospitals as the results would be more generalized and reliable. Another important area would be searching for a gold standard for detecting severity of illness and urgency of treatment,and accurately determining the best triage system.

CONCLUSION

In conclusion, our study showed that RTS was the most appropriate triage system in term of reliability among five triage systems. In particular, RTS showed almost perfect inter-rater reliability with kappa of 1 (P＜0.01). In the aspect of triage validity, we found that ESI illustrated the most appropriate validity for triage system by express highest AUC to predicting ability for admission.

ACKNOWLEDGMENTS

The authors hereby showed gratitude to all triage nurses,pediatric residents at ED of Ramathibodi Hospital, Mahidol University, Thailand for data collections, and all patients for participating in this research.

Funding: None.

Ethical approval: The medical ethics committee of Mahidol University approved this study.

Conflicts of interest: The authors declare that there are no conf l icts of interest related to the publication of this paper.

Contributors: Aeimchanbanjong K proposed the study and wrote the fi rst draft. All authors read and approved the fi nal version of the paper.

1 Morris ZS, Boyle A, Beniuk K, Robinson S. Emergency department crowding: towards an agenda for evidence-based intervention. Emerg Med J. 2012;29(6):460–6.

2 Cain P, Waldrop RD, Jones J. Improved pediatric patient fl ow in a general emergency department by altering triage criteria. Acad Emerg Med. 1996;3(1):65–71.

3 Fernandes CMB, Tanabe P, Gilboy N, Johnson LA, McNair RS,Rosenau AM, et al. Five-level triage: a report from the ACEP/ENA fi ve-level triage task force. J Emerg Nurs. 2005;31(1):39–50.

4 Christ M, Grossmann F, Winter D, Bingisser R, Platz E.Modern triage in the emergency department. Dtsch Arztebl Int.2010;107(50):892–8.

5 van Veen M, Moll HA. Reliability and validity of triage systems in paediatric emergency care. Scand J Trauma Resusc Emerg Med. 2009;17:38.

6 Mackway-Jones K, Marsden J, Windle J. Emergency Triage,Manchester Triage Group. Second. Oxford: Blackwell Publishing Ltd; 2006.

7 Roukema J, Steyerberg EW, van Meurs A, Ruige M, van der Lei J, Moll HA. Validity of the Manchester Triage System in paediatric emergency care. Emerg Med J. 2006;23(12):906–10.

8 van Veen M, Steyerberg EW, Ruige M, van Meurs AH, Roukema J, van der Lei J, et al. Manchester triage system in paediatric emergency care: prospective observational study. BMJ.2008;337:a1501.

9 van der WI, van Baar ME, Schrijvers AJ. Reliability and validity of the Manchester Triage System in a general emergency department patient population in the Netherlands: results of a simulation study. Emerg Med J. 2008;25(7):431–4.

10 van Veen M, Teunen-van der Walle VF, Steyerberg EW,van Meurs AH, Ruige M, Strout TD, et al. Repeatability of the Manchester Triage System for children. Emerg Med J.2010;27(7):512–6.

11 van Veen M, Steyerberg EW, Van't Klooster M, Ruige M, van Meurs AH, van der Lei J, et al. The Manchester triage system:improvements for paediatric emergency care. Emerg Med J.2012;29(8):654–9.

12 Travers DA, Waller AE, Katznelson J, Agans R. Reliability and validity of the emergency severity index for pediatric triage.Acad Emerg Med. 2009;16(9):843–9.

13 Green NA, Durani Y, Brecher D, DePiero A, Loiselle J, Attia M.Emergency Severity Index version 4: a valid and reliable tool in pediatric emergency department triage. Pediatr Emerg Care.2012;28(8):753–7.

14 Warren DW, Jarvis A, LeBlanc L, Gravel J, CTAS National Working Group, Canadian Association of Emergency Physicians,et al. Revisions to the Canadian Triage and Acuity Scale paediatric guidelines (PaedCTAS). CJEM. 2008;10(3):224–43.

15 Gouin S, Gravel J, Amre DK, Bergeron S. Evaluation of the Paediatric Canadian Triage and Acuity Scale in a pediatric ED.Am J Emerg Med. 2005;23(3):243–7.

16 Gravel J, Gouin S, Goldman RD, Osmond MH, Fitzpatrick E, Boutis K, et al. The Canadian Triage and Acuity Scale for children: a prospective multicenter evaluation. Ann Emerg Med.2012;60(1):71–7.e3.

17 Gravel J, Fitzpatrick E, Gouin S, Millar K, Curtis S, Joubert G, et al. Performance of the Canadian Triage and Acuity Scale for children: a multicenter database study. Ann Emerg Med.2013;61(1):27–32.e3.

18 Bullard MJ, Chan T, Brayman C, Warren D, Musgrave E, Unger B, et al. Revisions to the Canadian Emergency Department Triage and Acuity Scale (CTAS) Guidelines. CJEM.2014;16(6):485–9.

19 Mirhaghi A, Heydari A, Mazlom R, Ebrahimi M. The Reliability of the Canadian Triage and Acuity Scale: Meta-analysis. N Am J Med Sci. 2015;7(7):299–305.

20 Ebrahimi M, Heydari A, Mazlom R, Mirhaghi A. The reliability of the Australasian Triage Scale: a meta-analysis. World J Emerg Med. 2015;6(2):94–9.

21 Allen AR, Spittal MJ, Nicolas C, Oakley E, Freed GL. Accuracy and interrater reliability of paediatric emergency department triage. Emerg Med Australas. 2015;27(5):447–52.

22 Sanjay J, Ankur V, Tamorish K. Bombings specif i c triage (Bost Tool) tool and its application by healthcare professionals. World J Emerg Med. 2015;6(4):289–92.

23 Sittichanbuncha Y, Rojsaengroeng R, Wongsakornpattana S. Nurse triage in an accident and emergency department Ramathibodi Hospital: an audit. J Accid Emerg Med. 1994;11(2):91–5.

24 Thomas DO. Special considerations for pediatric triage in the emergency department. Nurs Clin North Am. 2002;37(1):145–59, viii.

25 Maldonado T, Avner JR. Triage of the pediatric patient in the emergency department: are we all in agreement? Pediatrics.2004;114(2):356–60.

26 Viera AJ, Garrett JM. Understanding interobserver agreement:The kappa statistic. Fam Med. 2005;37(5):360–3.

27 Dong SL, Bullard MJ, Meurer DP, Colman I, Blitz S, Holroyd BR, et al. Emergency triage: comparing a novel computer triage program with standard triage. Acad Emerg Med. 2005;12(6):502–7.

28 Storm-Versloot MN, Ubbink DT, Kappelhof J, Luitse JS.Comparison of an informally structured triage system, the emergency severity index, and the manchester triage system to distinguish patient priority in the emergency department. Acad Emerg Med. 2011;18(8):822–9.

Accepted after revision May 6, 2017

Uthen Pandee, Email: uthenp@hotmail.com

World J Emerg Med 2017;8(3):223–227

10.5847/wjem.j.1920–8642.2017.03.010

December 18, 2016