Verification of the Validity and Reliability of Therapeutic Communication Responses Generated by Large Language Models (LLMs): A Comparative Study of Prompt-Engineering Strategies
Article information
Abstract
Purpose
This study evaluated the validity and reliability of therapeutic communication responses generated by GPT-5 by comparing different prompt-engineering strategies and identifying optimal methods for psychiatric nursing education.
Methods
A methodological design compared four strategies—Zero-shot, Few-shot, Chain-of-Thought (CoT), and Role-prompt—applied to ten psychiatric nursing communication questions. Fourteen experts with over ten years of combined clinical and educational experience rated 40 anonymized responses using a 4-point scale. Inter-rater reliability was assessed with Fleiss' κ and Krippendorff's ⍺ from 400 repeated outputs. Semantic consistency was examined using cosine similarity. Analyses were conducted in Python.
Results
Role-prompt showed the highest content validity (mean=3.54±0.39, S-CVI/Ave=0.95), followed by CoT and Zero-shot. Inter-rater agreement was slight (κ=0.07; ⍺=0.15). Semantic consistency was highest for CoT (0.76±0.12) and Role-prompt (0.75±0.11). A Friedman test indicated significant differences among strategies (x2(3)=11.88, p=.008).
Conclusion
Role-prompt yielded the most valid therapeutic responses, while CoT produced the most consistent outputs. These findings support the use of Role-prompt and CoT strategies to enhance the validity and reliability of therapy.
INTRODUCTION
Large Language Models (LLMs) are driving innovation across various specialized fields, including healthcare, based on the exponential advancement of natural language processing capabilities [1]. Particularly in the field of psychiatric nursing, LLMs demonstrate considerable potential in enhancing clinical decision support systems, facilitating education and testing, rapidly generating customized educational materials for patients and caregivers, and analyzing vast amounts of unstructured data for nursing research [2,3]. For instance, nurses can swiftly access the latest insights on complex clinical cases or drug interaction information via LLMs to inform nursing care planning [2]. They can also leverage LLMs to generate post-discharge educational materials tailored to the patient's comprehension level [4]. Despite these possibilities, several critical prerequisites—such as ensuring the clinical accuracy, emotional appropriateness, and safety of generated responses—must be met before applying LLMs to psychiatric nursing practice. Because psychiatric nursing involves highly vulnerable patient populations and emotionally sensitive interactions, failure to verify these prerequisites can directly compromise patient well-being [5,6]. Accordingly, ensuring the safety of LLM-assisted psychiatric care requires a clear understanding of what constitutes validity and reliability. In this study, validity refers to the degree to which LLM responses align with academic and clinical evidence, whereas reliability reflects the stability of responses across repeated generations. Consequently, rigorous methodological research is required to establish engineering approaches that ensure the accuracy, consistency, and safety of LLM-generated information.
Within this context, prompt engineering techniques aimed at improving the response quality of large language models have emerged as a key solution. Prompt engineering refers to the technique of carefully designing the questions or instructions—known as prompts—input into the model to optimally elicit the desired output [7]. Among the representative methods, the Zero-shot prompting, which relies solely on questions without additional information, is simple but heavily dependent on the model's pre-training data, leading to high unpredictability and limited consistency. In contrast, the Few-shot approach, which incorporates several examples (demonstrations) into the prompt to guide the expected format or content, can enhance consistency and accuracy compared to Zero-shot, though its performance is strongly influenced by the quality and representativeness of the examples provided [8]. More recently, the Chain-of-Thought (CoT) technique has gained attention for tasks requiring complex reasoning, as it prompts the model to articulate intermediate reasoning steps [9]. By making the reasoning process explicit, CoT improves explainability and facilitates validity verification, although it is not effective for all question types and may generate unnecessarily verbose outputs. Finally, the Role-prompt technique [9]—which assigns the model a specific role (e.g., "You are an advanced practice nurse in psychiatry with 20 years of experience")—has the potential to enhance domain-specific accuracy and tone. However, if the model has insufficient understanding of the assigned role, it may produce inaccurate or superficial responses. These general characteristics highlight the need to examine how different prompting approaches influence LLM performance in psychiatric communication contexts.
In this study, four prompt-engineering strategies were selected because they represent commonly used approaches that may differently influence the clinical quality of LLM outputs in psychiatric settings. Zero-shot prompting allows examination of the model's baseline capability without contextual scaffolding. Few-shot prompting enables evaluation of whether example-based guidance improves therapeutic communication responses. Chainof-Thought prompting provides insight into how explicit reasoning instructions affect consistency and explainability— key requirements in high-risk clinical settings. Roleprompting examines whether assigning a psychiatric nursing role can enhance domain-appropriate tone, empathy, and accuracy. These characteristics make the four strategies directly relevant to investigating how prompt design influences LLM performance in therapeutic communication tasks.
While the impact of various prompt engineering techniques on LLM performance has been discussed theoretically, empirical research systematically comparing and validating their effects on the validity and reliability of responses within the specific context of therapeutic communication in psychiatric nursing is still lacking. Existing studies have focused mainly on exploring the potential of prompt strategies across healthcare or mental health in general. For example, Filienko et al. [10] sought to enhance the response quality of LLMs by utilizing various prompt techniques (e.g., protocol-guided prompts) during the symptom assessment and goal-setting stages, which correspond to the initial phase of problem-solving therapy. They reported a trend toward greater empathy and consistency based on expert evaluations. However, this study was limited to confirming the case-specific effectiveness of certain prompt strategies rather than conducting a direct performance comparison among Zero-shot, Few-shot, CoT, and Role-prompt approaches.
Furthermore, Kermani et al. [11] compared fine-tuning, Retrieval-Augmented Generation (RAG), and prompt engineering strategies for emotion classification and mental health state detection tasks, suggesting that prompt-based approaches could serve as a practical alternative in terms of resource efficiency. However, this study primarily focused on classification accuracy and processing efficiency rather than the emotional appropriateness of LLM responses or the accuracy of clinical reasoning, and it did not thoroughly examine the clinical applicability of different prompt types.
Psychiatric nursing questions, such as "How can trust be established when a patient shows anxiety during the initial encounter?", require more than simple fact-checking. They demand a nuanced empathetic approach, ethical considerations, and a deep understanding of the latest clinical guidelines. Empirical data on the specific strengths and limitations of Zero-shot, Few-shot, CoT, and Roleprompt techniques for such complex and sensitive questions are critically lacking. Evaluating whether an LLM's response is valid and reliable necessitates empirical analyses using multidimensional evaluation metrics beyond simple accuracy rates, including clinical consistency, emotional appropriateness, alignment with current evidencebased practice, and user satisfaction [3]. This represents a crucial task for the responsible deployment of LLMs, particularly in highly human-centered and ethically sensitive domains such as psychiatric nursing.
In psychiatric nursing practice, therapeutic communication is a core competency that enables trust-building with patients, identification of their needs, and the implementation of appropriate nursing interventions. Beyond mere information exchange, it functions as a process that promotes behavioral change and recovery through emotional support and interactive engagement [12,13], and is regarded as a key determinant of a psychiatric nurse's professional expertise and holistic nursing performance. For this reason, psychiatric nursing education emphasizes systematic acquisition and clinical application of therapeutic communication skills.
The five domains underlying the questionnaire—Relationship Building, Information Exchange, Problem Solving, Verbal and Nonverbal Skills, and Patient-Centeredness— reflect essential components of therapeutic communication competence. Together, these domains capture the multifaceted nature of psychiatric nursing interactions, from establishing rapport and conveying information to facilitating behavioral change and respecting patient autonomy. Thus, they provide a comprehensive framework for evaluating how effectively LLMs support therapeutic communication tasks.
Accordingly, this study aims to empirically compare the validity and reliability of LLM responses generated by major prompt engineering techniques in therapeutic communication question-and-answer tasks with patients experiencing mental health issues. Through this analysis, the study seeks to provide practical and scholarly evidence for identifying optimal prompt strategies that can be applied in psychiatric nursing practice, education, and research.
1. Purpose
The purpose of this study is to establish an evidence base for the safe and ethical integration of LLMs in psychiatric nursing education, particularly for therapeutic communication competency training. To this end, the study investigates the content validity (assessed by expert ratings) and consistency (measured via semantic similarity analysis) of responses generated by GPT-5. Four distinct prompt strategies (Zero-shot, Few-shot, CoT, and Role-prompt) were applied to core psychiatric nursing questions related to therapeutic communication. Based on these findings, we aim to discuss specific strategies for the effective implementation of LLMs in psychiatric nursing education and to propose standard guidelines for LLM application within the field of psychiatric nursing in Korea.
METHODS
1. Study Design
This study employed a methodological design to compare and analyze the content validity and consistency of responses generated by GPT-5. Four distinct prompt strategies (Zero-shot, Few-shot, CoT, and Role-prompt) were applied to 10 core questions related to therapeutic communication competency in psychiatric nursing.
2. Participants
To evaluate the content validity and reliability of the LLM responses, this study recruited an expert rater panel comprising 14 psychiatric nursing specialists [14]. The inclusion criteria for these experts were: (1) possession of 10 or more years of combined clinical and educational experience in the field of psychiatric nursing, and (2) a clear understanding of the research purpose and provision of informed consent to participate. Prior to the rating process, participants were provided with an evaluation rubric (4-point Likert scale) and representative examples. To ensure blinding and minimize bias, the names and specific characteristics of the different prompt strategies were not disclosed. Furthermore, to mitigate evaluation bias, the detailed operational principles of the LLM were only minimally explained to the raters.
3. Data Collection Procedure
The first step of data collection was clinical question development. An initial pool of 20 questions was drafted based on the five attributes of therapeutic communication competency in psychiatric nursing (4 questions per attribute). These items were then sent to three experts with PhDs in psychiatric nursing for an item validity assessment using a 4-point scale (1=inappropriate, 4=very appropriate). Only items with an Item-level Content Validity Index (I-CVI) of 0.80 or higher were retained, and a final set of 10 questions was confirmed, ensuring representativeness across categories.
The second step was LLM response generation. This was conducted in the ChatGPT Plus environment (GPT-5 as of October 16, 2025; the API was not used, default set tings were employed, and no system prompt was set). The four prompts (Zero-shot, Few-shot, CoT, Role-prompt) were input for each question in separate new chat windows, securing a total of 40 primary responses. The chat window was reset each time, opening a new chat for each question-strategy combination to collect the 40 primary responses. This was to prevent the context of a previous conversation from affecting a new answer. The strategies were implemented as: Zero-shot (without prior examples), Few-shot (including 2 examples), CoT (using a reasoning-inducing prompt, but only the final answer was presented to the evaluators), and Role-prompt (instructing the role of a psychiatric nursing professor with 20 years of experience). The context length was not limited.
The third step was expert validity assessment. The 40 generated responses were anonymized (prompt types hidden) and randomly arranged using a Latin square design. The rating forms, along with the informed consent forms, were distributed via email to the 14 evaluators, who were asked to rate the responses using a 4-point Likert scale. Prompt strategy information was removed, and instructions were unified so that response length or format did not imply the strategy. Based on the methodology suggested by Gwet et al. [14], the minimum sample size was calculated. Setting the expected value of Fleiss' Kappa (κ) coefficient at 0.60 (substantial agreement), with a Type I error (⍺) of .05 and statistical power (1-β) of .80, the minimum required number of evaluators was calculated to be 13 [14]. To ensure a robust sample size and account for potential dropouts or incomplete responses, 14 experts were recruited. All 14 participants successfully completed the evaluation and were included in the final analysis.
The final step was the construction of a corpus for response reliability verification. The 40 prompts previously used were each executed 10 times in new chat windows, collecting a total of 400 additional responses. The number of repetitions (10) was determined based on the cosine similarity distribution obtained in a pilot study, ensuring the 95% confidence interval length for the mean value would be 0.05 or less.
4. Data Analysis Methods
Collected data were analyzed using Python, primarily utilizing the pandas, NumPy, SciPy, pingouin, scikit-learn, and stats models packages. The significance level was set at a=.05, and a random seed (random.seed=42) was fixed for all analyses to ensure reproducibility. The detailed analysis process is as follows.
1) Analysis of expert characteristics
The demographic and academic characteristics (sex, age, education level, education/research experience, practical experience, total experience) of the 14-member expert rater panel were summarized using descriptive statistics. For continuous variables, mean, standard deviation, minimum, and maximum values were presented, and the normality of the distribution was checked using the Shapiro-Wilk test. In cases of non-normal distribution, the median and interquartile range (IQR) were also reported. For categorical variables, frequencies and percentages were presented.
2) Calculation of content validity
Content validity during the question development phase was calculated based on the 4-point scale (1=inappropriate, 4=very appropriate) ratings from three psychiatric nursing PhDs, yielding the Item-level Content Validity Index (I-CVI) (Supplementary Table 1). The I-CVI was calculated as the proportion of experts rating an item as 3 or 4. Given the panel size of three experts, only items with an I-CVI=1.00 were adopted [14].
3) Content validity of responses by prompt strategy
To verify the content validity of the responses generated by each prompt strategy, an evaluation was conducted with 14 experts in the field. Each expert assessed the validity of responses generated by the four strategies (Zeroshot, Few-shot, CoT, Role-prompt) on a 4-point scale from 'not valid at all' (1 point) to 'very valid' (4 points). The mean, standard deviation, median, interquartile range, minimum, and maximum of the validity scores for each prompt strategy were calculated to ascertain the overall level of validity. Furthermore, to ensure the reliability of the content validity, the Item-level Content Validity Index (I-CVI) and the Scale-level Content Validity Index (S-CVI) were calculated. The I-CVI was calculated as the proportion of experts who responded 'valid'(3 points) or 'very valid'(4 points) for each item. The S-CVI was calculated in two ways for a comprehensive analysis: the average of all I-CVI values (S-CVI/Ave) and the proportion of items unanimously rated as 'valid' or higher by all experts (SCVI/UA) [15].
4) Inter-rater reliability
Inter-rater agreement was reported using Fleiss' k [16], which is suitable for multiple raters. The 95% confidence interval was estimated using bootstrapping (1,000 resamples, fixed seed). To reflect the ordinal nature of the scale, Krippendorff's a (ordinal distance) [17] was supplementarily calculated to verify the stability of the ratings. The interpretation criteria for Fleiss' k [16] and Krippendorff's a [18] followed: (0.00~0.20=slight, 0.21~0.40=fair, 0.41~0.60=moderate, 0.61~0.80=substantial, 0.81~1.00=almost perfect). Reliability indices were presented for both the entire response set and for subsets by strategy to examine variations in agreement according to strategy.
5) Analysis of semantic similarity by prompt strategy
Response consistency for each strategy was evaluated by the semantic similarity among 10 repeatedly generated responses from the identical prompt. Text embedding was performed using the sentence-transformers' multilingual model (paraphrase-multilingual-mpnet-base-v2), and response-level vectors were derived by applying mean pooling to sentence-level embeddings. Cosine similarity was calculated for all pairs of the 10 responses for each question-strategy combination. The mean and standard deviation were reported as primary indicators, with the median and IQR as supplementary indicators. Outlier exclusion was not performed in principle; however, in cases where system errors or significant deviations were suspected, a sensitivity analysis excluding outliers based on a predefined IQR criterion was additionally presented.
6) Comparison of response consistency (reliability)
Response consistency for each prompt strategy was assessed using the cosine similarity between response vectors generated from 10 repetitions of the same prompt. Response texts were embedded using a Sentence-BERT model [19]. The mean and standard deviation of the cosine similarities among the 10 responses were calculated for each question-strategy pair, and outliers were detected and excluded based on the IQR criterion (exceeding 1.5* IQR). Differences in consistency among strategies were compared using the Friedman test. If significant (a<.05), pairwise comparisons were performed using the Wilcoxon signed-rank test with Holm-Bonferroni correction. The effect size was reported using Kendall's W.
5. Ethical Considerations
This study was conducted after obtaining approval from the Institutional Review Board (GWNUIRB-2025-23) of the affiliated institution prior to research commencement. All participants were fully informed of the research purpose, procedures, potential risks and benefits, and their right to withdraw at any time, and written consent for voluntary participation was obtained. Information collected from evaluators was managed in a de-identified form, and only anonymized data were used for analysis. In the process of prompt and LLM response generation, the risk of personal information leakage was fundamentally blocked by using hypothetical information instead of inputting actual personal data. Given the nature of using a web-based LLM interface, which is subject to system updates or model drift, the data collection time point, model notation, and session management procedures were recorded and stored. Raw data (full prompts, original model responses, evaluation data) were encrypted, stored in secure storage, and prohibited from use for any purpose other than this research. Study results were reported only as aggregate statistics to ensure that no individual evaluator or case could be identified.
RESULTS
1. General Characteristics of Experts
A total of 14 people participated in the general characteristics' evaluation of the expert panel, all of whom were female, and their educational level was exclusively doctoral degrees (Table 1). The average age was 48.14±5.04 years, the median age was 49.0 years, and the range was 36.00~55.00 years. Education and research experience averaged 12.48±7.90 years, with a median of 11.08 years and a range of 0.67~26.58 years, and practice experience averaged 10.08±6.92 years (median 9.25 years, range 1.00 ~25.58 years). Total experience was an average of 22.56± 6.96 years, with a median of 21.67 years and a range of 10.25~34.00 years. The normality of the experts selected as study subjects was secured (p>.005).
Demographic and Professional Characteristics of the Expert Panel (N=14)
2. Content Validity of Responses by Prompt Strategy
The content validity rating results among prompt strategies are shown in Table 2. The mean rating score (M±SD) was highest for Role-prompt at 3.54±0.39, with a median (IQR) of 3.55 (0.53) and an observed range of 2.60~4.00. CoT showed a similar level at 3.49±0.46 (median 3.60, IQR 0.43, range 2.50~4.00), and Zero-shot was 3.49±0.41 (median 3.50, IQR 0.62, range 2.60~4.00). Few-shot was the lowest at 2.87±0.74 (median 2.94, IQR 0.95, range 1.70~ 4.00). In the content validity indices, Role-prompt's mean I-CVI and S-CVI/Ave were the highest at 0.95 and 0.95, respectively, and its proportion of universal agreement (S-CVI/UA) also surpassed other strategies at 0.50. Zeroshot's mean I-CVI was 0.94, S-CVI/Ave was 0.94, and S-CVI/UA was 0.30. CoT's mean I-CVI was 0.89, S-CVI/Ave was 0.89, and S-CVI/UA was 0.00. Few-shot's mean I-CVI and S-CVI/Ave were both 0.69, and its S-CVI/UA was 0.00. These results indicated that for the therapeutic communication task, the Role-prompt strategy secured the highest expert consensus, while Zero-shot and CoT showed a moderate level, and Few-shot demonstrated a relatively low level of content validity.
Content Validity of Generated Responses by Prompt Strategy (N=14)
3. Inter-Rater Reliability
Inter-rater agreement by prompt strategy was calculated using Fleiss's κ and Krippendorff's ⍺, and both indicators confirmed an overall very low level of agreement (Table 3). For the entire dataset, Fleiss's κ was 0.07 (95% CI: 0.05~0.22) and Krippendorff's ⍺ was 0.15 (95% CI: 0.07~0.35), corresponding to "slight agreement" in interpretation. By strategy, CoT showed the highest values with Fleiss's κ at 0.08 (0.02~0.29) and Krippendorff's ⍺ at 0.07 (-0.01~0.34), but this remained at the "slight agreement" level. Role-prompt also showed a similarly low level of reliability, with κ at 0.03 (-0.01~0.20) and ⍺ at 0.07 (0.02~0.23). Conversely, the reliability for Zero-shot and Few-shot was found to be lower than chance. Zero-shot's Fleiss's κ was -0.03 (-0.04~0.10) and Krippendorff's ⍺ was -0.01 (-0.03~0.14). Few-shot's κ was -0.04 (-0.05~0.08) and ⍺ was -0.03 (-0.05~0.06), with all classified as "poor agreement".
Inter-rater Reliability by Prompt Strategy (N=40)
4. Analysis of Semantic Similarity (Cosine Similarity) by Prompt Strategy
Among Prompt Strategies The semantic similarity of repeatedly generated responses among prompt strategies was highest for CoT, with a mean of 0.76±0.12, median 0.76, and IQR 0.08. Role-prompt followed closely with a mean of 0.75±0.11, median 0.75, and IQR 0.08. Zero-shot showed a moderate level of consistency with a mean of 0.67±0.11, median 0.68, and IQR 0.14, while Few-shot exhibited the lowest mean and largest variance, with a mean of 0.63±0.21, median 0.63, and IQR 0.11. The narrow IQR (IQR=0.08) commonly observed in both CoT and Roleprompt suggests that variability across the strategies was relatively small. In comparison, the wider IQR of Zeroshot and the large standard deviation of Few-shot demonstrate significant variability in response patterns during repeated executions of the same item (Table 4). According to Table 4, CoT and Role-prompt consistently produced reproducible semantic structures, whereas Few-shot was found to be unstable in terms of both mean level and variance indicators. The detailed item-by-strategy metrics in Supplementary Table 2 reproduce this trend at the item level, while also revealing the item-specificity of each strategy's strengths. In a majority of items, CoT and Roleprompt simultaneously showed high mean similarity and low variance. For example, on Q8, CoT was the best among the four strategies with a mean of 0.91±0.02 (IQR 0.04), and Role-prompt also maintained a high level at 0.86± 0.04 (IQR 0.05). On Q10 as well, CoT 0.89±0.02 (IQR 0.03) and Role-prompt 0.89±0.04 (IQR 0.07) were top performers, confirming the stable reproducibility of both strategies. Conversely, cases where the relative rankings were partially reversed depending on the item were also observed. On Q5, Zero-shot recorded the top score at 0.86, and Few-shot followed at 0.82. While the mean value itself appeared favorable, it is interpreted that Few-shot's overall mean and standard deviation were worsened by its large variability in other items. On Q7, Role-prompt showed the highest value at 0.77, followed by CoT (0.70) and Zero-shot (0.55). Few-shot was significantly lower at a 0.40 level, suggesting the existence of strategy sensitivity depending on item difficulty or the interpretation method of the instructions. Zero-shot generally showed a moderate-level mean, but its relatively wide IQR and range consistently indicated a pattern where the semantic structure of responses was less strategically controlled across repeated executions. In particular, the wide range between the minimum and maximum values for Zero-shot across several items reflected a characteristic where excellent attempts occurred sporadically, but a low bottom line pulled down the overall mean and confidence interval. Details are provided in Supplementary Table 2.
Analysis of Semantic Similarity (Cosine Similarity) by Prompt Strategy (N=10)
5. Comparison of Response Consistency by Prompt Strategy
A Friedman test was conducted to compare the effect of prompt strategies on the consistency (reliability) of answer generation (Table 5). The analysis results showed a statistically significant difference in cosine similarity scores among the four prompt strategies (Zero-shot, Fewshot, CoT, Role-prompt) (x2(3)=11.88, p=.008). The Ken-dall's W coefficient, indicating the effect size, was 0.099, suggesting that although the difference between strategies was statistically significant, its magnitude was small. To identify which specific strategy pairs contributed to the overall difference found in the Friedman test, a post-hoc analysis was conducted using the Wilcoxon signed-rank test with Holm-Bonferroni correction. The results confirmed that the CoT strategy showed statistically significantly higher consistency than the Few-shot strategy (p=.012). However, no statistically significant differences were found in the comparisons between any other strategy pairs (p>.05).
Comparison of Response Consistency by Prompt Strategy (N=10)
DISCUSSION
This study aimed to identify the optimal prompt-engineering strategy by comparatively analyzing how four principal techniques—Zero-shot, Few-shot, CoT, and Roleprompt— affect the validity and reliability of LLM responses in therapeutic-communication scenarios for individuals with mental health issues. This section discusses specific prompt-engineering strategies for LLMs and their performance in delivering therapeutic-communication training for individuals with mental health issues, based on the foregoing findings.
The findings suggest that strategies enhancing directive clarity and contextual scaffolding—namely Role-prompt and CoT—produce superior response quality for therapeutic-communication queries involving psychiatric populations, outperforming the minimally guided Zero-shot baseline. Notably, the Role-prompt strategy achieved the highest validity—and thus the highest overall response quality—in therapeutic-communication question scenarios for individuals with mental health issues. This aligns with prior studies that compared Role-prompt with other strategies in health-related queries [18,20], suggesting that assigning an expert role in questions concerning therapeutic communication with psychiatric populations likely enhanced both the accuracy and acceptability of responses. Furthermore, GPT-5 has been reported to exert even stronger effects on health-related questions than GPT-4, which itself demonstrated human-level performance on a range of professional and academic benchmarks [21,22]. Considering that therapeutic communication critically involves recognizing and reflecting patients' emotions and engaging in shared decision-making [23], the use of Role-prompt with GPT-5 likely functioned as an advantage— facilitating more accurate information provision and enabling the model to manifest the structured conversational flow of therapeutic communication learned from diverse pretraining sources. Therefore, when using LLMs such as GPT for learning therapeutic communication with individuals with mental health issues, prior instruction on prompting should be provided, as such training is expected to increase the likelihood of obtaining more accurate and reliable responses. In particular, for Role-prompt, the quality of responses can vary depending on how the role is specified. Moreover, because cases such as the "grandma exploit" demonstrate attempts to coerce inappropriate answers by assigning a particular persona, such misuse should be proactively prevented [22]. Accordingly, it is necessary to develop a concrete design for the Role-prompt tailored to the instructional topic at hand. Furthermore, by incorporating the degree of role specification into prompt design, it should be possible to implement learner-tailored instruction that accounts for learners' proficiency levels in therapeutic communication.
However, the inter-rater agreement across the four prompt strategies examined in this study was generally very low; although the CoT and Role-prompt strategies reached only slight agreement, overall concordance remained limited. Consistent with prior work indicating that the most informative evaluation in many natural-language processing tasks is human assessment of model outputs [2,24], we asked therapeutic-communication experts in psychiatric nursing—each with an average of more than 22 years of combined clinical and teaching experience—to rate the responses produced under each strategy. Nevertheless, when response length, ordering, and style vary by prompt strategy, raters may apply different implicit weights (e.g., personal preference, perceived accuracy, or fidelity), which can depress agreement; moreover, even for the same sample, within-rater reproducibility may differ across raters [24]. To mitigate these issues, it is advisable to pre-specify and train evaluators on the rating criteria with exemplars and to implement procedural controls prior to scoring.
Meanwhile, semantic similarity by prompt strategy—defined as the extent to which repeatedly generated answers under the same prompt convey similar meanings— was generally higher for the CoT and Role-prompt strategies in this study. This appears to be because CoT, in therapeutic-communication questions, standardizes a sequence of intermediate reasoning steps, thereby guiding responses to converge on similar core meanings [9,25,26]; in the case of Role-prompt, assigning a role such as "psychiatric advanced practice nurse" likely induced convergence of the essential content of responses to therapeutic-communication queries in a similar direction across generations [18].
Semantic similarity also varied by question type: for questions focused on rapport building (e.g., Q1), Roleprompt showed higher semantic similarity, whereas for problem-solving items (e.g., Q10), CoT yielded higher semantic similarity. This pattern suggests that the most suitable prompting strategy may differ by question type. For interaction-oriented questions in therapeutic communication— such as empathy, affect reflection, and relationship formation—Role-prompt may be preferable [23], as it structurally elicits role-consistent therapeutic actions, thereby increasing semantic similarity. Conversely, for questions requiring causal reasoning, stepwise explanation, or judgment, CoT may be more appropriate [9], because it imposes a standardized discourse skeleton via stepwise reasoning, which in turn promotes similarity in the core meaning of responses.
Accordingly, rather than relying on a single prompting approach, it is necessary to select the optimal prompt strategy based on the type of therapeutic-communication question; in some cases, a hybrid use of CoT and Roleprompt may be warranted.
This study also examined whether the within-strategy consistency of repeated responses varied when different prompt strategies were applied to the same task. The results showed differences among the four strategies, but the effect size (Kendall's W=0.099) was small; post-hoc analyses indicated that CoT demonstrated significantly higher consistency than Few-shot, while the other pairwise comparisons were not significant. This suggests that CoT—which structures the reasoning process for therapeutic-communication prompts—imposes step-by-step justification during response generation and thereby enhances semantic cohesion, helping to suppress variability across repeated generations more effectively than Fewshot, which is sensitive to exemplar quality/fit [25,26]. In the case of Few-shot, variability may increase due to overfitting to a small number of examples or subtle mismatches between exemplars and situational inputs; and in the case of Role-prompt, while it sets sentence tone and style according to the assigned role, it does not explicitly constrain the core reasoning structure, so its stabilizing effect may be more limited than that of CoT [25]. In learning therapeutic communication with individuals with mental health issues using LLMs, securing response consistency is essential to ensure the reliability of learning and assessment. In light of our findings, the CoT strategy appears to enhance the semantic consistency of repeated generations and, to contribute to the standardization of reasoning procedures. On this basis, it can be a viable option for educational contexts that require repeated practice of the same procedure (e.g., simulation-based training).
However, this study has several limitations. First, we applied only a single prompt-engineering strategy to obtain LLM response to therapeutic communication questions. Consequently, we were unable to examine the effects of combining prompt-engineering techniques. Given our results—namely, that the Role-prompt strategy demonstrated higher content validity and the CoT strategy showed higher consistency—it would be meaningful to compare the effects of using these two strategies in combination against their standalone use. Second, because we relied solely on the ChatGPT 5.1 model to examine responses by prompt-engineering strategy, we cannot guarantee that the same effects would be observed with other models, such as Gemini 2.5. Third, because prompts were entered in Korean, we cannot rule out the possibility that different effects would emerge if another language were used.
CONCLUSION
This study sought to identify the most appropriate prompt strategy for learning therapeutic communication with individuals with mental health issues. Based on expert ratings, the Role-prompt strategy achieved the highest content validity, and both CoT and Role-prompt showed high semantic similarity within strategies. Taken together, these findings indicate that employing Roleprompt and CoT in LLM-based training for therapeutic communication can yield responses with higher validity and reliability, underscoring the practical utility of these prompt strategies in educational settings.
The use of LLMs in medical and nursing education is becoming unavoidable. In this context, to enhance accuracy and practicality, formal curricula should include prior instruction on prompt design. Moreover, the effects of combining effective prompt-engineering techniques should be continually evaluated. Accordingly, future research should compare several configurations—such as singlestrategy use and various combinations—to determine their relative effectiveness
SUPPLEMENTARY MATERIAL
Development and Content Validity of Clinical Question Items
Detailed Semantic Consistency (Cosine Similarity) by Question and Strategy
Notes
Miran Jung has been a member of the editorial board since January 2024, but she had no role on the decision to publish this article. Except for that, no potential conflict of interest relevant to this article was reported.
AUTHOR CONTRIBUTIONS
Conceptualization and Methodology: Kim, SK & Kim, GM
Validation and Visualization: Kim SK
Writing-original draft: Kim SK, Kim, GM, Cha S & Jung, M
Writing-review & editing: Kim SK, Kim, GM, Cha S & Jung, M