Validity and Reliability of Metacognitive Rating Scale for Nursing Students

1. Background

Metacognition refers to the awareness, knowledge, and control ability of one's own cognition. If cognition is an intellectual activity for processing a certain task, metacognition refers to the activity of monitoring such cognitive activity [1,2]. In other words, metacognition refers to psychological structures, knowledge, events, and processes related to the control, modification, and interpretation of thinking itself [3]. This capacity allows them to plan, control, and check their learning and to determine how to apply previously acquired knowledge and experiences [1,4]. For nursing students, metacognitive competence is the ability to assess outcomes of their nursing actions based on their own cognition as health care providers, make composite plans, and evaluate the results [2]. Metacognition serves as an important variable in learning, and students who show higher levels show greater academic success as well [5].

Because nursing students invest years in learning to execute high-level nursing actions in health care practice, they can experience stress and anger from, for example, strict learning environments that can lead to negative emotional responses such as anxiety, depression, and lowered self-esteem [6-9]. It would clearly be advantageous, therefore, to improve the metacognitive competence of nursing students and assist them with their learning achievement.

The self-regulatory executive function (S-REF) model, which explains metacognition, recognizes that dysfunctional metacognitive beliefs occur based on the people’s type of response to negative thoughts and emotions, namely cognitive attentional syndrome (CAS) [10,11]. In the self-regulating executive function model, maladaptive beliefs are caused by a series of psychological processes called cognitive-attentional syndrome (CAS). CAS includes three main processes, each of which constructs extended thinking in response to negative thoughts [10]. Control, a way of coping with stress that is described as suppression of thinking, is closely related to aggression, and aggressive behavior can arouse anger [12,13]. Metacognition as the highest area of cognition can have impacts on both behavior and affect as well as on cognition [14], and worry and anxiety are associated with each other in the cognitive process [15].

Metacognition is a high-level psychological structure conceptualized as an ability to perceive and control one's own thinking process and beliefs [16], and we devised the metacognitive rating scale, MCRS to measure diverse facets of metacognition. The tools for measuring metacognition in previous studies are as follows. Klein’s [17] metacognitive questionnaire consisted of the subfactors of cognitive strategy, planning, and self-examination, and Schraw and Dennison’s [18] metacognitive awareness inventory consisted of cognition knowledge and cognition control. Lee Ji Hye’s [19] metacognitive measurement test consists of the subfactors of planning, inspection, and control. The Metacognitive Questionnaire (MCQ) developed by Cartwright-Hatton and Wells [15] is needed to measure the maladaptive metacognitive beliefs to grasp nursing student’s psychological aspects such as anxiety, anger and impulse control due to academic stress.

The MCQ was developed with 65 items to determine levels of metacognition loaded onto five factors: positive beliefs about worry, negative beliefs about the uncontrollability of thoughts and corresponding danger, cognitive confidence, beliefs about the need to control thoughts, and cognitive self-consciousness [15]. The same authors developed a 30-item short form of the questionnaire (MCQ-30) that consists of items with the highest loading among the subfactors [3]. Seol [20] translated the MCQ-30 for use in South Korea, and K-MCQ-30 has been actively used by researchers for general college students, but there has been little research among nursing students in South Korea.

Although the MCQ-30 is a simpler, but still multidimensional, measure of metacognition that is more efficient than the full MCQ, the factor analysis at the time of its development showed weak associations on one of the subfactors. Specifically, only three of the six items for the subfactor beliefs about the need to control thoughts were highly associated, and the other three (“It is bad to think certain thoughts”, “If I did not control a worrying thought and then it happened, it would be my fault”, “I will be punished for not controlling certain thoughts”) associations were weak [3]. In a study using K-MCQ-30, it was found that there were items with low correlation among the subfactors [21]. Therefore, more in-depth research on this topic was warranted, and we undertook this study. For the study, we revised items related to beliefs about the need to control thoughts with a focus on the adverse effects of thought control, so that the adverse effects could be associated with anger expressed as aggressive impulses [12,13]. Based on the MCQ [15], MCQ-30 [3], and K-MCQ-30 [20], we conducted this study to reliability and validity a metacognitive rating scale (MCRS) that we revised to include the concepts of anger and anxiety, emotions nursing students might experience during high-intensity academic work.

2. Purpose

We aimed to test the reliability and validity of the metacognitive rating scale (MCRS) revising and supplementing the MCQ-30.

1. Study Design

The authors designed this study to test a revised version of the MCQ-30, the MCRS, among a group of nursing students in South Korea. The MCRS contained added measurements of anger and anxiety, potentially common emotions among nursing students.

2. Research Procedure and Scale Adaptation Process

Before we conducted this study, we also obtained permission from authors of the MCQ and the MCQ-30, K-MCQ-30. We tested the revised scale, the MCRS, for reliability and performed confirmatory factor analysis (CFA) to confirm that the subfactors divided into components with the same concepts as the established scale.

Notices were posted on notice boards in the nursing departments at four colleges in North and South Gyeongsang Provinces, South Korea, seeking volunteers to participate in our study. The investigators obtained a signature from the participant on a written informed consent document. We also provided an explanation of the purpose and methods of this study in the preface of the questionnaire we distributed. We explained to the participants that they could discontinue participating and refuse to answer any more questions at any time. We also explained to participants that psychological counseling could be provided if any items on the cognitive scale triggered emotional difficulties, and the participants were instructed to contact the researcher immediately in that event.

3. Subjects and Data Collection

Before we began the data collection, we obtained approval from the institutional review board of Keimyung University (IRB No. 40525-202011-HR-069-03). Using G*Power 3.1.9, we calculated the number of study subjects as 600 based on significance of 0.05, effect size of 0.3, power of 0.80, and a previous study [21] of 662 college students based on the MCQ-30 factor structure. We distributed a total of 620 copies of the MCRS to male and female nursing students we received 603 completed copies, for a 97.3% return rate. We excluded 12 surveys that were incomplete or that did not clearly indicate the respondent’s written consent, leaving 591 completed surveys for analysis.

4. Measurement Tool

MCRS is a revised version of the MCQ developed by Cartwright-Hatton and Wells [15] and its short forms, the MCQ-30 [3] and K-MCQ-30 [20]. Therefore, we added anger and anxiety to the MCQ-30 to develop the MCRS, and we revised a total of seven items: “I easily get angry when I feel irritated”, “I always try to suppress my anger”, “It is my weakness that I cannot control my anger”, “I cannot control expressions of anger”, “Expressing anger is bad”, “I can express anger in words that are not aggressive”, and “When you become addicted to worry, even the slightest amount of stress lead to anxiety”.

The scale had a total of 30 items and consisted of a 5-point scale, and the degree of agreement with each item was rated on a Likert scale ranging from 1 point for “strongly disagree” to 5 points for “strongly agree”. The questionnaire items were divided into five subfactors: positive belief in worry (POS; six items), negative belief that worries are uncontrollable and dangerous (NEG; six items), cognitive confidence (CC; six items), need for control-personality tendency (NC; six items), and cognitive self-consciousness (CSC; six items). The Cronbach’s ⍺ for the reliability of the MCQ ranged from .72 to .89 at the time of its development and from .66 to .88 in this study.

5. Analysis

We used SPSS 26.0 software to process the survey data we collected. Specifically, we calculated descriptive statistics, two-sample t-test, and analysis of variance (ANOVA), estimated reliability with Cronbach’s ⍺, and performed CFA to check validity. We also calculated Pearson’s correlation coefficients for correlations among the five subfactors.

1. Demographic Characteristics

Of the 591 participants in this study, most were women (83.1% versus 16.9% men), most were high school graduates (90.7% compared with 9.3% college graduates or higher), and their mean age was 21.7 years.

2. Reliability and Validity Analysis

We performed confirmatory factor analysis to measure the reliability and validity of the MCRS and used principal component analysis to simplify relevant variables and characterize each factor; the number of factors was determined base on eigenvalue ≥1, and we used varimax rotation in the factor analysis. Table 1 presents the Cronbach’s ⍺ reliability findings for the MCRS.

Table 1.

Factor Structure and Reliability and Validity of MCRS (N=591)

For factor 1 (POS), Cronbach’s ⍺ was .77, which indicated reliability, whereas total cumulative variance was low (47.2%), which partially met criteria for validity. For factor 2, NEG, Cronbach’s ⍺= .88, eigenvalue ≥1, and total cumulative variance=62.7%, which indicated validity and reliability. For factor 3, CC, Cronbach’s ⍺= .86, which indicated reliability, and total cumulative variance=59.2%, which partially indicated validity. For factor 4, NC, Cronbach’s ⍺= .66, indicating reliability, and total cumulative variance=42.5%, which partially indicated validity. For factor 5, CSC, Cronbach’s ⍺=.74, indicating reliability, and total cumulative variance=44.1%, which partially indicated validity. CFA revealed that the first factor had high correlations in the order of items 19, 10, 7, 1, 28, and 23. In the second factor, items 9, 21, 11, 4, 2, and 15 showed high correlations in that order. Under factor 3, items 17, 26, 8, 29, 24, and 14 were highly correlated, and in factor 4, items 22, 20, 6, 25, and 13 items were highly correlated. However, we found no high correlations for 27 items. Under the fifth factor, items 16, 12, 30, 18, 3, and 5 correlated highly correlated in that order. In short, reliability was indicated for all five factors, POS, NEG, CG, NC, and CSC, but validity was only partially confirmed (Table 1).

3. Mean Difference Test

The mean difference test results for gender and education are presented in Table 2. We found no statistically significant differences among POS, NEG, CC, NC, or CSC by gender, although high school graduates scored statistically significantly higher for POS than did college graduates.

Table 2.

Mean Difference Test by Gender and Education (N=591)

4. Correlation Analysis among MCRS Subfactors

Table 3 presents the correlation analysis findings based on the composite scores for the five scale subfactors in MCRS is presented in. Correlations were positive among all the subfactors, with POS and NEG, NEG and CSC being most strongly correlated at r=.69 and POS and CC, CC and CSC being least correlated at r=.28.

Table 3.

Correlation Analysis of Subfactors (N=591)