Validity and reliability of Korean version Competency Assessment Tool-Mental Health

Hyun Mee Cho, Jeong Won Hn, Eun Joung Choi, Hyo Eun Jeong, Bo Rm Hong,Eun Yong Kim

a Department of Nursing Science, Kaya University, Gimhae 50830, Republic of Korea

bCollege of Nursing Science, Kyung Hee University, Seoul 02447, Republic of Korea

Abstract: Objective: To establish basic data for identifying the level of mental health care competency of general nurses through verification of the reliability and validity of Competency Assessment Tool-Mental Health in the development of a measurement tool for mental health care competency in Korea.Methods: This study was conducted on nurses working at general wards, excluding those working at the Department of Psychiatry,in five hospitals with 200 beds or more located in Korea. Content, construct, concurrent validity, and internal consistency of the measurement were confirmed.Results: As a result of the construct validity, the section on importance of skills and knowledge for mental health care had 21 items, whereas the section on benefits of additional education had 22 items. The internal consistency of measurement was confirmed as follows: Cronbach’s α = 0.96 for the section on importance and 0.96 for the section on benefits section.Conclusions: This study verified the high validity and reliability of the tool in assessing the mental health care competency of nurses, and it is believed to be significant as basic data for enhancing such competency.

Keywords: competency · mental health · nurse · reliability · validity

1. Introduction

According to a status report on mental diseases in Korea published by the Korean Ministry of Health and Welfare, approximately 4.7 million adults, or 11.9% of the total adult population in Korea, experience at least one mental health issue in a year, with a prevalence rate of 25.4% for mental diseases within their lifetime.1In addition, a number of people in Korea either directly or indirectly experience mental health issues. As of 2015, the number of patients who received inpatient mental health treatment reached approximately 81,000, whereas the number of outpatients was approximately 45,000/day. Although the number of patients in 2015 decreased slightly compared with 2014, it has shown a consistent upward trend over the last decade.2-4

Symptoms of mental health issues include auditory hallucination, delusion, depression, and asthenia, which lead to self-harm, suicide, drug addiction, and falling. In the United States, One out of eight inpatients in emergency ward is admitted to general ward due to suffering from mental health issues.5Moreover, when patients have mental health issues, they often struggle to realize quickly and make a judgment regarding their physical diseases or do not voluntarily seek medical assistance on time, leaving diseases untreated or missing out on the proper time for treatment. Many patients come to hospital or seek professional medical assistance only after their condition has deteriorated.6

Given such circumstances, nurses at general wards often encounter a situation where they need to care the patients with mental health issues.5Nurses are voicing difficulties in caring for such patients as they lack with appropriate knowledge and experience in mental health care. This problem can act as an obstacle for the effective communication between nurses and patients, thereby increasing at least twofold the probability of preventable physical adverse reactions.7

Meanwhile, mental health care is an essential nursing service for providing necessary health care to patients in need by identifying any actual or potential mental health issues and for maintaining and promoting their mental health.8To implement mental health care adequately, profound understanding and professional knowledge and skills are required, particularly those related to the growth and developmental tasks of human beings. The competency of nurses with relevant education, training, and experience is an important prerequisite to the provision of care for patients with mental health issues.7Nurses at the Department of Psychiatry consider themselves as a tool for nursing; they control the pace of their speech, vocal tone, and level of activity among other aspects, when conversing or interacting with patients to reduce and control stimulation, and ultimately ensure the well-being of patients and foster a therapeutic environment.9In addition, an accurate assessment of patients during nursing enables nurses to predict unconventional and at times destructive behaviors of patients with mental health issues and helps them to keep the patients safe, enhancing the efficiency of treatment.10For the well-being and recovery of patients and reduction in treatment cost, nurses at general wards need to develop competency in mental health care.7However, compared with nurses at the Department of Psychiatry, nurses at general wards have relatively less knowledge and interest as well as experience in mental health care.11,12Moreover, nurses at general wards focus more on the physical conditions than mental conditions of patients10which may lead to an extension of hospital stay, additional hospitalization cost, and increased risk of disease recurrence,7,13,14ultimately decreasing the treatment satisfaction level of patients. Such situation has given rise to the increasing need for interest in and education on mental health care by nurses at general wards.12

Nurses at general wards feel stress or burden from patients with mental health issues,5,15and it is reported that nurses experience “care with no effect,” “bias,” “avoidance of mental health care,” “difficulties in care owing to lack of knowledge or environmental restrictions,” and other problems when caring for patients with mental health issues. To address these problems, nurses need to enhance their competency in mental health care; it is important to provide regular education and training related to mental health care for nurses at general wards to that end.16In addition, in a study on the mental health care competency of nurses at general wards and patients with mental disorders conducted in Korea, 91.5% of incumbent nurses at general wards of hospitals call for refresher courses on mental health care.17

To develop educational programs and intervention methods on mental health care for general nurses, their actual competency in mental health care need to be assessed by identifying their skills and level of knowledge in mental health care. A number of studies conducted in Korea have used the Behavioral Health Care Competency (BHCC) tool.7Another measurement tool is the Competency Assessment Tool-Mental Health (CAT-MH).11There has been no development of similar tools in Korea.

To evaluate the mental health care competency of general nurses in Korea in a reasonable and systematic manner, there is a need for the development of mental health care tools that adequately consider Korean culture, the increasing demand for mental health care, and characteristics of the nursing practice environment. In this context, this study was conducted to provide essential information on the development of educational programs and intervention methods for mental health care by verifying the reliability and validity of CAT-MH and establishing basic data for identifying the level of mental health care competency of general nurses. The findings are expected to contribute to future development of an effective measurement tool of mental health care competency in Korea. The specific objectives of this study are as follows:

(1) To confirm the content validity of CAT-MH,

(2) To confirm the construct validity of CAT-MH,

(3) To confirm the concurrent validity of CAT-MH, and

(4) To confirm the internal consistency for reliability verification of CAT-MH.

2. Material and methods

2.1. Research design

This study aimed to establish basic data for identifying the level of mental health care competency of general nurses through verification of the reliability and validity of CAT-MH in the development of a measurement tool for mental health care competency in Korea.

2.2. Research subjects

This methodological study sought to test the validity and reliability of the translated Korean version of CAT-MH, a competency measurement tool for mental health.

This study was conducted on nurses working at general wards, excluding those working at the Department of Psychiatry, in five hospitals with 200 beds or more located in Busan metropolitan city, Gyeongsangnam-do in Korea. In this study, factor analysis was used as a primary analysis method. To conduct a factor analysis for validity analysis, the number of respondents is required to be at least 5-10 times greater than the number of the expected items in the questionnaire.18We calculated the sample size based on the guidelines given by Comrey and Lee,19which stipulated that a sample size of 200 participants or more is adequate for an exploratory factor analysis. Therefore, considering the 25 items expected in this study and a withdrawal rate of 10%, the total number of study participants was set at 275.

A total of 275 questionnaire copies were distributed, and 275 responses were received (collect rate: 100.0% and response rate: 98.9%). Excluding three insincere responses, a total of 272 responses were included in this study. All study procedures were conducted after obtaining approval from the Institutional Bioethics Committee of Kosin University. The study participants were recruited in cooperation with the nursing center of each hospital, and only participants who consented to participate in this study were selected. The participants were informed of the study purpose and that the collected data would be used solely for academic purposes. They were assured of the privacy of data used in validation, and of their anonymity throughout the analyses and statistical processing. Collection of data was conducted outside the working hours of the participants to ensure no disturbance to their work. The participants were informed of their discretion to withdraw from this study at any time and to request for the partial or complete deletion of their data. The person in charge of the collected data was the researcher of this study. The data were stored at a designated place; electronic records were stored and managed in the researcher's personal computer, which required a password to access for security. The data obtained from the surveys were not used for purposes other than academic research. Anonymity was maintained in the study results. The data used in this study will be stored for 3 years; electronic copies will be permanently deleted; and all other forms of documentation, printouts, letters, and records will be shredded or incinerated.

2.3. Data collection and analysis

All data collected throughout this study were analyzed using SPSS 20.0 for Windows (SPSS Korea Data Solution Inc.) and AMOS 21.0 (SPSS Korea Data Solution Inc.). The general characteristics of the participants were calculated using real numbers, and percentages, mean, and standard deviations were obtained. Content, construct, and concurrent validity were confirmed. The content validity index (CVI) was used to estimate the content validity. Exploratory and confirmatory factor analyses were conducted to test the construct validity. In addition, the BHCC for nonpsychiatric nurses and CAT-MH were used to confirm the Pearson's correlation coefficient, and Cronbach's α was used to confirm internal consistency.

3. Measurement

3.1. Behavioral Health Care Competency

This study used the translated Korean version of BHCC (2016) developed in the United States for nonpsychiatric nurses. The tool comprises the four subfactors: assessment, practice/intervention competency, recommendation of psychotropics, and resource adequacy. It is a 5-point scale assessment with 1 indicating “strongly disagree” and 5 indicating “strongly agree.” A higher score indicated greater BHCC of the nonpsychiatric nurse. In the study by Han and Woo,20the reliability of the tool was defined according to the following Cronbach's α values: 0.85 for assessment, 0.86 for intervention, 0.86 for recommendation of psychotropics, and 0.87 for resource adequacy. In this study, the Cronbach's α values for reliability were as follows: 0.91 for assessment, 0.90 for intervention, 0.78 for recommendation of psychotropics, and 0.78 for resource adequacy.

3.2. Competency Assessment Tool-Mental Health

The CAT-MH for assessing the mental health care competency of medical professionals was translated into Korean, amended, and complemented on the approval of the developers. The Korean version CATMH underwent validity and reliability tests before use. The tool comprised a total of 26 items (5-point scale) that were classified into sections based on importance and benefits. At the time of the development, the tool reliability was as follows: Cronbach's α = 0.93 for knowledge and Cronbach's α = 0.95 for benefits.

4. Preliminary study

Before proceeding with the study,21approval for use of the tools was obtained from the developers of the BHCC for nonpsychiatric nurses and CAT-MH via e-mail, in accordance with the guidelines of the World Health Organization. The primary translation of the tool from English to Korean was performed by a professor at Seoul National University with a doctorate from a nursing graduate school who is fluent in both Korean and English, and with experience of living in an English-speaking country for over 5 years and of working as a nurse. The translated Korean version was then reverse-translated from Korean to English by a professor at the same university who has been a professor of English literature for over 20 years, and with experience of living in the United States for over 10 years; the professor reviewed the sentence structure of the items and the similarity in meaning between the original and translated versions. Subsequently, a panel of experts checked the content validity of the tool. A total of 25 items were included in the tool for the use of this study, and one item with low content validity was excluded.

After obtaining approval from the Institutional Bioethics Committee of Kosin University (1040549-180611-SB-0062-01), a preliminary study was conducted from August 1 to 10, 2018 on 10 clinical nurses working at general wards to identify issues with the questionnaire items and other potential issues that may arise during the main study. The participants of the preliminary study were selected based on the same criteria as that of the tool and were asked to respond tothe Korean version CAT-MH in a self-reported method.The general characteristics of the participants were as follows: average age of 34.24 ± 2.41 years; possessing a bachelor's degree (six participants); average years of work experience of 9.01 ± 3.68 years; religious faith (four participants); and general nurse as position at work (six participants). No participant had difficulty in interpreting or reading the questionnaire, and 5-10 min was consumed for responding to the questionnaire.

5. Results

5.1. General characteristics of participants

The report showed that women accounted for the largest portion of gender with 93.1%, followed by 47% in their 20s, 38.4% in their 30s, 12.1% in their 40s, and 2.6% in their 50s. The number of unmarried people was 59.9%, compared with 39.7% of married people. Religion (70.7%) outnumbered absenteeism (29.3%), followed by bachelor's degrees (65.1%), professional college graduates (26.7%), and graduate schools (8.25%). The number of general nurses (70.3%) was the largest, followed by those who were charge nurses (24.1%) and those who were nurses (5.6%). The total clinical experience was 28% for ＜3 years, 39.2% for ＜9 years and ＞4 years, and 32.8% for ＞10 years. The total average year of clinical experience was 8.09 years. The working years at the current department had the most experience with ＜3 years (64.7%), followed by ＜9 years (31.5%) and ＞10 years (3.9%). The average working year at the current department was 3.44 years. There were 55.6% more nursing experience than there were none (45.4%) and 66.8% more psychiatric education experience (33.2%). Psychological nursing learning experience (84.1%) was higher than none (15.9%).

5.2. Content validity

CVI was measured to test the validity by two professors in nursing, one doctoral student in nursing with ＞18 years of experience as a psychiatric nurse, one doctoral student in nursing with ＞6 years of clinical experience, and two master's students in nursing with ＞5 years of clinical experience. The result showed that all items showed a CVI of 0.80, except for Item 26 that had a CVI of 0.8, with Scale-CVI (average) = 0.95 and Scale-CVI (universal average) = 0.85.

5.3. Construct validity

5.3.1. Exploratory factor analysis

A Kaiser-Meyer-Olkin test was conducted for an exploratory factor analysis, and the result was 0.95. A Bartlett's test of spherecity confirmed the adequacy of the correlation matrix for the factor analysis, and the result indicated the adequacy of the data for the analysis, χ2= 3,907.85 (p＜ 0.001). A principal component analysis was conducted for the factor analysis, and varimax orthogonal rotation was used for the rotation of factors. A factor analysis of the 25 items of CAT-MH showed that 4 items of the section on importance of skills and knowledge (Items 1, 2, 4, and 16) and 3 items (Items 3, 4, and 16) of the section on benefits of additional education had a communality of 0.40 and were removed. The eigenvalue of each section was 1, and the factor load was 0.40. As a result of the exploratory factor analysis, the section on importance of skills and knowledge for mental health care had 21 items, whereas the section on benefits of additional education had 22 items, see Tables 1 and 2.

Table 1. Factor analysis of the Competency Assessment Tool-Mental Health (the section on importance of skills and knowledge).

5.3.2. Confirmatory factor analysis

An additional confirmatory factor analysis was conducted using the confirmed items in the exploratory factor analysis. The goodness of fit of the model for the section on importance of skills and knowledge was checked: χ2= 530.07 (p＜ 0.001), df = 183, χ2/df = 2.89, adjusted goodness-of-fit index = 0.90, goodness of fit index = 0.97, comparative fit index = 0.98, root mean square error of approximation = 0.03, and normed fit index = 0.94. Thus, the model was adequate. In addition, the factor load was checked and the result showed that each item had a value of 0.50, confirming the significance of the 21 items as shown in Figure 1. Construct reliability was calculated as 0.98, and the average variance extracted was confirmed as 0.71. The goodness of fit of the model for the section on benefits of additional education was checked; χ2= 458.36 (p＜ 0.001), df = 208, χ2/df = 2.2, adjusted goodness-of-fit index = 0.92, goodness of fit index = 0.96, comparative fit index = 0.93, root mean square error of approximation = 0.02, and normed fit index = 0.89. Thus, the model was adequate. In addition, the factor load was checked and the result showed that each item had a value of 0.50, confirming the significance of the 22 items as shown in Figure 2. Construct reliability was calculated as 0.98, and the average variance extracted was 0.70.

Table 2. Factor analysis of the Competency Assessment Tool-Mental Health (the section on benefits of additional education).

5.4. Concurrent validity

To test the concurrent validity, the BHCC for nonpsychiatric nurses was used to identify the correlation between the tool and the Korean version CAT-MH in this study. The results showed the correlation coefficient between the tools as 0.13-0.68 (p＜ 0.001), as given in Table 3.

5.5. Reliability

The internal consistency of CAT-MH was confirmed as follows: Cronbach's α = 0.96 for the section on importance of skills and knowledge, and Cronbach's α = 0.96 for the section on benefits of additional education (Tables 1 and 2).

6. Discussion

This study aimed to provide basic data for the development of a mental health care competency measurement that is appropriate for the Korean clinical environment after translating the CAT-MH into Korean. We verified that the Korean version CAT-MH is a valid tool for evaluating the mental health care competency of Korean nurses. The internal consistency of CAT-MH was confirmed (given in Cronbach's α values): 0.96 for the section on importance of skills and knowledge and 0.93 for the section on benefits of additional education. Thus, the present results showed no significance differences from those given by Clasen et al.,11which showed Cronbach's α values of 0.96 and 0.95 for the sections on importance of skills and knowledge and on benefits of additional education, respectively.

In addition, the exploratory factor analysis showed that the data were adequate for the factor analysis. The factor analysis revealed that four items in the sections on importance of skills and knowledge (Item 1: “I treat study participants with respect and consider them as equal partners”; Item 2: “I include to the selfhelp group family members or other supporters in the process of patients' treatment”; Item 4: “I respond to culture, race, and gender issues that affect participants”; and Item 16: “I support patients”) and three items in the section on benefits of additional education (Item 3: “I'm aware of mental disorder symptoms and characteristics”; Item 4: “I respond to culture, race, and gender issues that affect patients”; and Item 16: “I support patients”) had a communality of 0.4 and were removed. This outcome indicates the cultural and background differences between the participants in the present study and those given by Clasen et al. The study by Clasen et al.11was conducted on mental health care medical personnel in Ohio, USA, whereas the present work was conducted in Korea on general nurses with no experience in psychiatric care. In particular, Item 4 was about culture, race, and gender issues. Given the difference in culture and working environment between the United States and Korea, there might have been disparities in the perspective of nurses toward treating patients with mental health issues, resulting in the difference between the two studies.17With regard to Items 1, 2, and 16, general nurses in Korea focus on assessing present symptoms and the rapid relief of such symptoms in clinical practice.10However, the primary causes of mental issues include the growth and personal environment of patients, involving emotional influence by family or particular incidents.7,8Therefore, it is considered important to recognize that the relief of symptoms through medication should not be the only focus and that patient's family members or supporters who might have influenced the mental health issues or recovery of the patients should be included in the treatment process of patients with mental health issues.

When comparing the confirmatory factor analysis with the exploratory factor analysis conducted in this study, the former proved a more important analysis process in testing the construct validity, confirming that the Korean version CAT-MH was an adequate model. Not only this result is evident that the tool adequately measures the nurses' mental health care competency but it also reconfirms the need for adequate education and training to enhance mental health care competency geared toward satisfying the care needs of patients on top of patients' physical issues. In this context, to enhance the mental health care competency of nurses, hospitals must provide education and training on concepts in mental health care and care practice for patients with mental health issues. To this end, the Korean version CAT-MH, in which reliability and validity were tested in this study, needs to be used to identify the mental health care competency of general nurses to provide necessary education and training.

To confirm the concurrent validity of the Korean version CAT-MH, its correlation with the BHCC for nonpsychiatric nurses was identified. The results showed that it had statistically significant correlation with the two measurement tools. Furthermore, this CAT-MH version can be assumed to be a valid tool for assessing the BHCC of nurses based on the correlation coefficients of 0.60-0.80 (indicating “high validity”) and 0.80-1.00 (indicating “very high validity”) obtained in the evaluation of validity through correlation coefficient.22In this context, the BHCC of nonpsychiatric nurses in Korea and actual shortcomings in their competency need to be identified. Subsequently, concrete measures to enhance the CAT-MH of clinical nurses can be formulated.

Table 3. Correlation among variable.

7. Conclusions

This study aimed to provide basic data for enhancing the mental health care competency of nonpsychiatric nurses by using and verifying the CAT-MH for nurses in Korea. This study also verified the high validity and reliability of the tool in assessing the mental health care competency of nurses and is believed to be significant as basic data for enhancing such competency. However, additional studies are needed on the questionnaire items for mental health care competency to assess whether they take into account the nursing environment in Korea. In addition, the predictability of the Korean version of CAT-MH needs to be verified.

Ethics approval

The study protocol was approved by the Institutional Bioethics Committee of Kosin University (No. 1040549-180611-SB-0062-01).

Conflicts of interest

All contributing authors declare no conflicts of interest.