A Systematic Review and Meta-Analysis of Studies on Psychiatric Nursing Simulation Program-Focused on Scenario

1. Research Design

This study aims to conduct a SR and meta-analysis with the primary objective of comprehensively examining the research findings related to mental health nursing simulation programs.

2. Inclusion and Exclusion Criteria for the Literature

This study was conducted following the guidelines outlined in the Cochrane handbook for SR of interventions 5.1.0 and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (https://prismastatement.org/PRISMAStatement/, 2020). After compiling a list of studies retrieved through the databases, the references were organized using the citation management program, EndNote X9. The specific literature inclusion criteria were: 1) Studies measuring the educational effectiveness of simulation programs in the field of psychiatric nursing, 2) Studies targeting nurses or nursing students, 3) Studies including randomized controlled trials, experimental designs, or quasi-experimental research designs. Exclusion criteria for literature were as follows: 1) Studies outside the field of psychiatric nursing, 2) Studies not utilizing simulation, 3) Studies not targeting nurses or nursing students, 4) Studies that were not randomized controlled trials, experimental designs, or quasi-experimental research, 5) Gray literature (proceedings, theses, dissertations, etc.), 6) Studies with selectively reported experimental results, 7) Studies not written in Korean or English.

3. Search Strategies

The flow of this study was described using the PRISMA flow chart, outlining the step-by-step process of literature selection. The initial search period was from January 1, 2000, to September 30, 2023. The databases utilized were EMBASE, OVID Medline & PubMed, Web of Science, CINAHL, and the Cochrane Library. The search primarily focused on title/abstract, and the main keywords used for the search were (Nurse OR nurse practitioner OR Nursing Students) AND (Augmented reality OR Virtual reality OR Simulation OR standardized patient) AND (psychiatry OR psychiatric OR mental health OR mental illness) AND (Nursing) AND (Education).

4. Data Extraction

A total of 2,571 studies were initially identified through the following databases: 449 from EMBASE, 1,309 from PubMed, 492 from CINAHL, 166 from Cochrane Library, and 155 from Web of Science. Among these, there were 826 duplicate records. Following the duplication process, 1,745 records were examined by their titles and abstracts to remove any literature that did not fit within the defined scope. Consequently, 1,570 records were excluded, and a review of full-text was conducted for 149 records. After the full-text review, 140 records were further excluded based on the following criteria: 8 were not related to mental health (psychiatric) nursing, 3 did not utilize simulation, 11 did not focus on nurses or nursing students, 102 were not experimental studies, 7 were gray literature, 2 were not reporting research results, 3 were no control group (single group study), and 4 were duplicates. In the end, a total of 35 records were included for systematic literature review, and among them, 20 records were subjected to meta-analysis.

5. Data Evaluation

For organizing the literature, relevant information pertaining to research design, country, sample size, type of simulation, contents of simulation, variables, and results was extracted and recorded in a coding table.

The data collection and selection process involved four researchers independently reviewing all studies included in the analysis. In cases where opinions differed, the researchers reviewed and discussed the literature together, following the selection and exclusion criteria until a consensus was reached. When assessing the quality of the literature, evaluations for randomized controlled trials utilized the revised Cochrane Risk of Bias 2.0 (RoB 2.0) tool [16]. For nonrandomized comparative studies, the evaluation was carried out using the Risk of Bias in Nonrandomized Studies of Interventions (ROBINS-I) tool [17], which examined eight items.

6. Data Analysis

All statistical analyses regarding the effect sizes of the specified program variables were performed using Comprehensive Meta-Analysis (CMA, V4) software version 4.0. The effect sizes were interpreted according to Hedges’ g criteria [18], classifying values of 0.2~0.5 as small, 0.5~0.8 as medium, and≥0.8 as large, while maintaining a significance level of 95%. Considering that meta-analyses entail consolidating findings from diverse studies to interpret the effect size, evaluating the uniformity or diversity of the research findings becomes pivotal. In this regard, we utilized the I-squared for assessment. The I-squared within the range of 0~24% indicate no heterogeneity, 25~49% signify low heterogeneity, 50~74% represent moderate heterogeneity, and values≥75% indicate high heterogeneity [19]. To investigate the possibility of publication bias, we employed both funnel plot analysis and the trim-and-fill method [20]. We suspected the presence of publication bias if the effect size, particularly the adjusted measure quantifying the intervention’s impact while accounting for potential publication bias using the trim-and-fill method, deviated by 10% or more compared to the previous estimate [20]. The adjusted effect size offers a more precise depiction of the genuine effect, accounting for any potentially absent studies. This approach entails the removal of asymmetric effect sizes from the initial funnel plot (trimming), followed by estimating missing studies presumed to have been excluded. This estimation relies on studies that fill in the gaps, aiming to attain symmetry around the newly calculated mean effect size [21].

1. General Characteristics of the Selected Studies

All the papers utilized for the SR were published in academic journals, summing up to a total of 35 articles. The publication year of the articles included in the SR was highest in 2017 with 8 articles (22.9%), followed by 2018 (14.3%) as the second highest. The years 2010, 2015, and 2022 had the lowest number of publications, each with only 1 article (2.9%). In terms of the nationality of the research papers, the USA led with 16 papers (45.7%). South Korea followed with 8 papers (22.9%), Australia with 4 papers (11.4%), the UK with 3 papers (8.6%), Türkiye with 2 papers (5.7%), and both Malaysia and Singapore had 1 paper each (2.9%). Regarding research design, Non-Randomized Controlled Trials (NRCT) were the most common at 22 articles (62.9%), followed by Mixed-Methods Studies (MMS) with 11 articles (31.4%). When it came to the participants of the studies, a vast majority of 28 papers (80.0%) were focused on Nursing Students (SN). Meanwhile, 4 papers (11.4%) exclusively targeted Registered Nurses (RN). There were also 3 papers (8.6%) that involved a mixed group comprising nurses and other professionals, such as psychiatrists, social workers, and occupational therapists. In relation to the type of simulation, the Standardized Patients (SP) simulation was the most prominent, represented in 24 studies (68.6%). Within this category, the High-Fidelity Patient (or Human) Simulation (HFPS) was the subject of 7 studies (20.0%), the Interprofessional Mental Health Simulation (IPMHS) was covered in 3 papers (8.6%), and the Video Module Simulation (VMS) was utilized in 3 studies (8.6%). Notably, out of the VMS studies, 2 (5.7%) implemented the Flipped Learning technique. Additionally, research on Voices Simulation (VS) encompassed 2 papers (5.7%), while Virtual Reality (VR) simulation was the focus of 3 papers (8.6%). Concerning the research environment, a majority of 28 studies (80.0%) were conducted offline, while 2 (5.7%) combined both online and offline methods, and 5 (14.3%) were solely online. Among the offline studies, the majority, 23 papers (65.7%), were based in school classrooms or simulation rooms. Furthermore, 4 studies (11.4%) used the simulation rooms of Professional Training Centers (PTC), and 1 study (2.9%) was conducted in a ward setting (Table 1).

Table 1.

General Characteristics of Selected Studies

2. Systematic Review of Studies on Psychiatric Nursing Simulation Program

In psychiatric nursing simulation scenarios, the most frequently measured variable was Knowledge, appearing in 18 out of 35 scenarios, making it the most common. Following Knowledge, Confidence was measured in 14 scenarios, and Attitude was the next most frequent, appearing in 7 scenarios. In studies on mental health nursing simulations, the most frequently utilized theoretical/conceptual framework was Jeffries’ Simulation Model, which was applied in 4 out of 35 studies. Following this, both self-efficacy Theory and Kolb’s theory of experiential learning were employed in 3 studies each. However, there were also 15 out of 35 studies that did not apply any theoretical/conceptual framework.

A total of 20 studies recorded statistics that could confirm effects. When standardizing the effect sizes of these studies to Hedges’s g (p), significant effects were revealed in several studies, including: Graves et al. (2020) (g=0.98, p=.040), Im & Jang (2019) (g=0.59, p=.017), Kameg et al. (2010) (g=0.64, p<.001), and Koetting & Freed (2017) (g=0.73, p<.001), among others. On the other hand, the following studies were found to have no statistically significant effects: Kunst et al. (2017) (g=0.19, p=.052), Lee & Jang (2021) (g=0.47, p=.074), Luebbert R et al. (2023) (g=0.11, p=.484), Soccio (2017) (g=0.09, p=.769), and Yang & Kang (2023) (g=0.35, p=.182), among others. These outcomes highlight the range of effectiveness across different studies, with some showing significant impacts and others indicating no significant effects. Detailed information can be found in Table 2.

Table 2.

Systematic Review of Studies on Psychiatric Nursing Simulation Programs

3. Analysis of the Psychiatric Nursing Simulation Scenario

The content analysis of the psychiatric nursing simulation scenario is shown in Table 3.

Table 3.

Analysis of the Psychiatric Nursing Simulation Program

In an analysis of psychiatric nursing simulation programs, the study by Evans et al. (2015) had the highest number of participants, totaling 256. This was followed by Ng et al. (2017) with 206 participants, and Luebbert R et al. (2023) with 178 participants. On the other hand, the study by Martinez (2017) had the fewest participants, with only 15. Lee & Choi (2023) had 20 participants, and Chambers et al. (2018) had 24 participants, making them among the studies with the lowest number of participants.

Among the types of simulators, SP was the most common, featured in 22 out of 35 papers, making it the most frequent. HFPS followed as the second most common, appearing in 8 papers. In studies using SP simulators, the most common approach involved hiring trained or professional actors for simulation training, accounting for 9 studies. There were 4 studies that utilized trained individuals or lay people as SP. However, there were also 5 studies that used SP patients but did not specify the type of SP involved.

The longest-running psychiatric nursing simulation program was conducted by Choi et al. (2016), which was operated over 5 weeks in two sessions. Similarly, the study by Lee et al. (2021) also ran a program over 2 weeks, spanning a total of 10 days, and divided into four sessions. On the other hand, the shortest psychiatric nursing simulation program was by Ng et al. (2017), which consisted of watching a 5-minute educational video. Next, the study by Ok et al. (2020) was the second shortest program, operating scenarios that lasted approximately 10~12 minutes.

Results of reviewing a total of 35 scenarios, scenarios that allow training in therapeutic communication skills during interactions with subjects showing significant psychiatric symptoms such as anxiety, depression, aggression, auditory hallucinations, delusions, and suicidal thoughts were the most common, with a total of 15 (42.9%). To train therapeutic communication skills, various psychiatric problems were covered, including psychological problems such as anxiety and depression, psychotic symptoms such as auditory hallucinations and delusions, and behavioral problems such as aggression and suicide attempts. Among them, there were two scenarios (5.7%) that applied both the nursing process and therapeutic communication to solve patients’ problems in emergency situations. Mental illnesses commonly observed in clinical settings include schizophrenia, bipolar disorder, depression, suicide, alcohol use disorder, and substance addiction. There were a total of 10 studies (28.6%) that evaluated the mental status of patients showing the main symptoms of these diseases and psychological problems such as anxiety and agitation and applied the nursing process to solve nursing problems. In these studies, SP mainly implemented the behavioral characteristics of patients, showing the main symptoms of each disease and the side effects of drugs, and applied the nursing process to solve the patients’ health problems. One was designed to evaluate patients and use therapeutic communication in emergencies. There were three scenarios (8.6%) dealing with suicide attempts, such as poisoning, hanging, and self-harm, impulsive behavior due to auditory hallucinations, and coping with emergencies due to alcohol withdrawal symptoms. This is a psychiatric emergency, and the ability to respond to it is crucial. It was a program to train. There was one scenario (2.9%) about the evaluation and response to a patient with complex psychiatric symptoms such as suicide and hallucinations along with physical problems such as fractures, diabetes, and respiratory problems that can be experienced in the emergency room. In addition, there were two studies (5.7%) that focused on auditory hallucination experiences that provided voices to understand the problem of auditory hallucinations and empathize with patients’ experiences. Four scenarios (11.4%) also dealt with collaboration between family members and experts in the field to treat mentally ill patients.

Out of a total of 35 studies, 16 had a control group. Among these 16 studies, 6 mentioned using conventional and traditional educational methods for the control group, including educational materials on depression. The remaining studies differentiated the educational methods between the experimental and control groups by using online lectures, video lectures, and other methods. Meanwhile, there were also 5 studies that did not specify what educational methods were provided to the control group, or if any were provided at all.

4. Risk of Bias

The quality assessment results of studies using ROB2 and ROBINS-I are as presented in Figure 1. The RCT studies analyzed by ROB2 totaled four, with the overall evaluation showing a low risk of bias in 75% of the cases and a moderate risk in 25%. However, some concerns were noted in specific domains, particularly in deviations from intended interventions. In contrast, the studies assessed by ROBINS-I numbered 31, making up the majority. The overall evaluation indicated varying levels of risk, with no studies showing a severe risk of bias. However, there were studies that exhibited serious bias in participant selection, and moderate risk was frequently observed in domains related to intended interventions. Overall, the assessment revealed that while many studies maintained a low or moderate risk of bias, there were recurring concerns in specific areas such as intended interventions and participant selection (Figure 1).

Fig. 1.

Risk of bias summary.

5. Results of a Meta-Analysis on Psychiatric Nursing Simulation Programs

The studies with the largest effect sizes were Martinez (2017) and Sarikoc et al. (2017) with Hedges’s g values of 1.47 (95% CI: 0.74~2.20; p<.001) and 1.15 (95% CI: 0.69~1.61; p<.001) respectively. On the other hand, no significant effects were observed in the studies by Choi et al. (2016), Kameg et al. (2021), Kunst et al. (2017), Lee et al. (2021), Lubbert et al. (2023), Soccio (2017), and Yang & Kang (2022). Overall, considering all the studies, the effect sizes (Hedges’s g) ranged from -0.16 to 1.47, with a pooled Hedges’s g value of 0.56 (95% CI: 0.35~0.78; p<.001), indicating a significant effect (Figure 2). The I-squared which measures the percentage of variation across studies that is due to heterogeneity rather than chance, was 86.80%. This high I-squared suggests substantial heterogeneity among the included studies. This indicates that the effect sizes vary significantly across studies, likely due to differences in study design, populations, interventions, and other factors. Such high heterogeneity warrants careful interpretation of the pooled effect size, as it reflects the variability in the effects of psychiatric nursing simulation programs across different contexts. The distribution of effect sizes, the confidence intervals, and the heterogeneity of the included studies are summarized in Figure 2.

Fig. 2.

Total effect size.

6. Funnel Plot and Trim-and-Fill for the Publication Bias of the Studies

In our assessment of 20 studies through a funnel plot, we examined the symmetry to evaluate the presence of any publication bias. The funnel plot exhibited a generally symmetrical distribution around the central axis, suggesting an absence of publication bias. The outcomes of the trim-and-fill analysis further corroborated this by showing zero studies required trimming. This implies that none of the studies were deemed to have a biased representation that necessitated adjustment or exclusion. Consequently, the effect sizes and confidence intervals maintained their initial estimates. From this, we deduce that any potential publication bias in the included studies exerted negligible influence on the overall results, leaving the original findings on effect sizes and confidence intervals intact.