qualitative research study article

Criteria for Good Qualitative Research: A Comprehensive Review

• Regular Article
• Open access
• Published: 18 September 2021
• Volume 31 , pages 679–689, ( 2022 )

Cite this article

You have full access to this open access article

• Drishti Yadav   ORCID: orcid.org/0000-0002-2974-0323 1  

97k Accesses

44 Citations

71 Altmetric

Explore all metrics

This review aims to synthesize a published set of evaluative criteria for good qualitative research. The aim is to shed light on existing standards for assessing the rigor of qualitative research encompassing a range of epistemological and ontological standpoints. Using a systematic search strategy, published journal articles that deliberate criteria for rigorous research were identified. Then, references of relevant articles were surveyed to find noteworthy, distinct, and well-defined pointers to good qualitative research. This review presents an investigative assessment of the pivotal features in qualitative research that can permit the readers to pass judgment on its quality and to condemn it as good research when objectively and adequately utilized. Overall, this review underlines the crux of qualitative research and accentuates the necessity to evaluate such research by the very tenets of its being. It also offers some prospects and recommendations to improve the quality of qualitative research. Based on the findings of this review, it is concluded that quality criteria are the aftereffect of socio-institutional procedures and existing paradigmatic conducts. Owing to the paradigmatic diversity of qualitative research, a single and specific set of quality criteria is neither feasible nor anticipated. Since qualitative research is not a cohesive discipline, researchers need to educate and familiarize themselves with applicable norms and decisive factors to evaluate qualitative research from within its theoretical and methodological framework of origin.

Similar content being viewed by others

Good Qualitative Research: Opening up the Debate

Beyond qualitative/quantitative structuralism: the positivist qualitative research and the paradigmatic disclaimer.

What is Qualitative in Research

Avoid common mistakes on your manuscript.

Introduction

“… It is important to regularly dialogue about what makes for good qualitative research” (Tracy, 2010 , p. 837)

To decide what represents good qualitative research is highly debatable. There are numerous methods that are contained within qualitative research and that are established on diverse philosophical perspectives. Bryman et al., ( 2008 , p. 262) suggest that “It is widely assumed that whereas quality criteria for quantitative research are well‐known and widely agreed, this is not the case for qualitative research.” Hence, the question “how to evaluate the quality of qualitative research” has been continuously debated. There are many areas of science and technology wherein these debates on the assessment of qualitative research have taken place. Examples include various areas of psychology: general psychology (Madill et al., 2000 ); counseling psychology (Morrow, 2005 ); and clinical psychology (Barker & Pistrang, 2005 ), and other disciplines of social sciences: social policy (Bryman et al., 2008 ); health research (Sparkes, 2001 ); business and management research (Johnson et al., 2006 ); information systems (Klein & Myers, 1999 ); and environmental studies (Reid & Gough, 2000 ). In the literature, these debates are enthused by the impression that the blanket application of criteria for good qualitative research developed around the positivist paradigm is improper. Such debates are based on the wide range of philosophical backgrounds within which qualitative research is conducted (e.g., Sandberg, 2000 ; Schwandt, 1996 ). The existence of methodological diversity led to the formulation of different sets of criteria applicable to qualitative research.

Among qualitative researchers, the dilemma of governing the measures to assess the quality of research is not a new phenomenon, especially when the virtuous triad of objectivity, reliability, and validity (Spencer et al., 2004 ) are not adequate. Occasionally, the criteria of quantitative research are used to evaluate qualitative research (Cohen & Crabtree, 2008 ; Lather, 2004 ). Indeed, Howe ( 2004 ) claims that the prevailing paradigm in educational research is scientifically based experimental research. Hypotheses and conjectures about the preeminence of quantitative research can weaken the worth and usefulness of qualitative research by neglecting the prominence of harmonizing match for purpose on research paradigm, the epistemological stance of the researcher, and the choice of methodology. Researchers have been reprimanded concerning this in “paradigmatic controversies, contradictions, and emerging confluences” (Lincoln & Guba, 2000 ).

In general, qualitative research tends to come from a very different paradigmatic stance and intrinsically demands distinctive and out-of-the-ordinary criteria for evaluating good research and varieties of research contributions that can be made. This review attempts to present a series of evaluative criteria for qualitative researchers, arguing that their choice of criteria needs to be compatible with the unique nature of the research in question (its methodology, aims, and assumptions). This review aims to assist researchers in identifying some of the indispensable features or markers of high-quality qualitative research. In a nutshell, the purpose of this systematic literature review is to analyze the existing knowledge on high-quality qualitative research and to verify the existence of research studies dealing with the critical assessment of qualitative research based on the concept of diverse paradigmatic stances. Contrary to the existing reviews, this review also suggests some critical directions to follow to improve the quality of qualitative research in different epistemological and ontological perspectives. This review is also intended to provide guidelines for the acceleration of future developments and dialogues among qualitative researchers in the context of assessing the qualitative research.

The rest of this review article is structured in the following fashion: Sect.  Methods describes the method followed for performing this review. Section Criteria for Evaluating Qualitative Studies provides a comprehensive description of the criteria for evaluating qualitative studies. This section is followed by a summary of the strategies to improve the quality of qualitative research in Sect.  Improving Quality: Strategies . Section  How to Assess the Quality of the Research Findings? provides details on how to assess the quality of the research findings. After that, some of the quality checklists (as tools to evaluate quality) are discussed in Sect.  Quality Checklists: Tools for Assessing the Quality . At last, the review ends with the concluding remarks presented in Sect.  Conclusions, Future Directions and Outlook . Some prospects in qualitative research for enhancing its quality and usefulness in the social and techno-scientific research community are also presented in Sect.  Conclusions, Future Directions and Outlook .

For this review, a comprehensive literature search was performed from many databases using generic search terms such as Qualitative Research , Criteria , etc . The following databases were chosen for the literature search based on the high number of results: IEEE Explore, ScienceDirect, PubMed, Google Scholar, and Web of Science. The following keywords (and their combinations using Boolean connectives OR/AND) were adopted for the literature search: qualitative research, criteria, quality, assessment, and validity. The synonyms for these keywords were collected and arranged in a logical structure (see Table 1 ). All publications in journals and conference proceedings later than 1950 till 2021 were considered for the search. Other articles extracted from the references of the papers identified in the electronic search were also included. A large number of publications on qualitative research were retrieved during the initial screening. Hence, to include the searches with the main focus on criteria for good qualitative research, an inclusion criterion was utilized in the search string.

From the selected databases, the search retrieved a total of 765 publications. Then, the duplicate records were removed. After that, based on the title and abstract, the remaining 426 publications were screened for their relevance by using the following inclusion and exclusion criteria (see Table 2 ). Publications focusing on evaluation criteria for good qualitative research were included, whereas those works which delivered theoretical concepts on qualitative research were excluded. Based on the screening and eligibility, 45 research articles were identified that offered explicit criteria for evaluating the quality of qualitative research and were found to be relevant to this review.

Figure  1 illustrates the complete review process in the form of PRISMA flow diagram. PRISMA, i.e., “preferred reporting items for systematic reviews and meta-analyses” is employed in systematic reviews to refine the quality of reporting.

PRISMA flow diagram illustrating the search and inclusion process. N represents the number of records

Criteria for Evaluating Qualitative Studies

Fundamental criteria: general research quality.

Various researchers have put forward criteria for evaluating qualitative research, which have been summarized in Table 3 . Also, the criteria outlined in Table 4 effectively deliver the various approaches to evaluate and assess the quality of qualitative work. The entries in Table 4 are based on Tracy’s “Eight big‐tent criteria for excellent qualitative research” (Tracy, 2010 ). Tracy argues that high-quality qualitative work should formulate criteria focusing on the worthiness, relevance, timeliness, significance, morality, and practicality of the research topic, and the ethical stance of the research itself. Researchers have also suggested a series of questions as guiding principles to assess the quality of a qualitative study (Mays & Pope, 2020 ). Nassaji ( 2020 ) argues that good qualitative research should be robust, well informed, and thoroughly documented.

Qualitative Research: Interpretive Paradigms

All qualitative researchers follow highly abstract principles which bring together beliefs about ontology, epistemology, and methodology. These beliefs govern how the researcher perceives and acts. The net, which encompasses the researcher’s epistemological, ontological, and methodological premises, is referred to as a paradigm, or an interpretive structure, a “Basic set of beliefs that guides action” (Guba, 1990 ). Four major interpretive paradigms structure the qualitative research: positivist and postpositivist, constructivist interpretive, critical (Marxist, emancipatory), and feminist poststructural. The complexity of these four abstract paradigms increases at the level of concrete, specific interpretive communities. Table 5 presents these paradigms and their assumptions, including their criteria for evaluating research, and the typical form that an interpretive or theoretical statement assumes in each paradigm. Moreover, for evaluating qualitative research, quantitative conceptualizations of reliability and validity are proven to be incompatible (Horsburgh, 2003 ). In addition, a series of questions have been put forward in the literature to assist a reviewer (who is proficient in qualitative methods) for meticulous assessment and endorsement of qualitative research (Morse, 2003 ). Hammersley ( 2007 ) also suggests that guiding principles for qualitative research are advantageous, but methodological pluralism should not be simply acknowledged for all qualitative approaches. Seale ( 1999 ) also points out the significance of methodological cognizance in research studies.

Table 5 reflects that criteria for assessing the quality of qualitative research are the aftermath of socio-institutional practices and existing paradigmatic standpoints. Owing to the paradigmatic diversity of qualitative research, a single set of quality criteria is neither possible nor desirable. Hence, the researchers must be reflexive about the criteria they use in the various roles they play within their research community.

Improving Quality: Strategies

Another critical question is “How can the qualitative researchers ensure that the abovementioned quality criteria can be met?” Lincoln and Guba ( 1986 ) delineated several strategies to intensify each criteria of trustworthiness. Other researchers (Merriam & Tisdell, 2016 ; Shenton, 2004 ) also presented such strategies. A brief description of these strategies is shown in Table 6 .

It is worth mentioning that generalizability is also an integral part of qualitative research (Hays & McKibben, 2021 ). In general, the guiding principle pertaining to generalizability speaks about inducing and comprehending knowledge to synthesize interpretive components of an underlying context. Table 7 summarizes the main metasynthesis steps required to ascertain generalizability in qualitative research.

Figure  2 reflects the crucial components of a conceptual framework and their contribution to decisions regarding research design, implementation, and applications of results to future thinking, study, and practice (Johnson et al., 2020 ). The synergy and interrelationship of these components signifies their role to different stances of a qualitative research study.

Essential elements of a conceptual framework

In a nutshell, to assess the rationale of a study, its conceptual framework and research question(s), quality criteria must take account of the following: lucid context for the problem statement in the introduction; well-articulated research problems and questions; precise conceptual framework; distinct research purpose; and clear presentation and investigation of the paradigms. These criteria would expedite the quality of qualitative research.

How to Assess the Quality of the Research Findings?

The inclusion of quotes or similar research data enhances the confirmability in the write-up of the findings. The use of expressions (for instance, “80% of all respondents agreed that” or “only one of the interviewees mentioned that”) may also quantify qualitative findings (Stenfors et al., 2020 ). On the other hand, the persuasive reason for “why this may not help in intensifying the research” has also been provided (Monrouxe & Rees, 2020 ). Further, the Discussion and Conclusion sections of an article also prove robust markers of high-quality qualitative research, as elucidated in Table 8 .

Quality Checklists: Tools for Assessing the Quality

Numerous checklists are available to speed up the assessment of the quality of qualitative research. However, if used uncritically and recklessly concerning the research context, these checklists may be counterproductive. I recommend that such lists and guiding principles may assist in pinpointing the markers of high-quality qualitative research. However, considering enormous variations in the authors’ theoretical and philosophical contexts, I would emphasize that high dependability on such checklists may say little about whether the findings can be applied in your setting. A combination of such checklists might be appropriate for novice researchers. Some of these checklists are listed below:

The most commonly used framework is Consolidated Criteria for Reporting Qualitative Research (COREQ) (Tong et al., 2007 ). This framework is recommended by some journals to be followed by the authors during article submission.

Standards for Reporting Qualitative Research (SRQR) is another checklist that has been created particularly for medical education (O’Brien et al., 2014 ).

Also, Tracy ( 2010 ) and Critical Appraisal Skills Programme (CASP, 2021 ) offer criteria for qualitative research relevant across methods and approaches.

Further, researchers have also outlined different criteria as hallmarks of high-quality qualitative research. For instance, the “Road Trip Checklist” (Epp & Otnes, 2021 ) provides a quick reference to specific questions to address different elements of high-quality qualitative research.

Conclusions, Future Directions, and Outlook

This work presents a broad review of the criteria for good qualitative research. In addition, this article presents an exploratory analysis of the essential elements in qualitative research that can enable the readers of qualitative work to judge it as good research when objectively and adequately utilized. In this review, some of the essential markers that indicate high-quality qualitative research have been highlighted. I scope them narrowly to achieve rigor in qualitative research and note that they do not completely cover the broader considerations necessary for high-quality research. This review points out that a universal and versatile one-size-fits-all guideline for evaluating the quality of qualitative research does not exist. In other words, this review also emphasizes the non-existence of a set of common guidelines among qualitative researchers. In unison, this review reinforces that each qualitative approach should be treated uniquely on account of its own distinctive features for different epistemological and disciplinary positions. Owing to the sensitivity of the worth of qualitative research towards the specific context and the type of paradigmatic stance, researchers should themselves analyze what approaches can be and must be tailored to ensemble the distinct characteristics of the phenomenon under investigation. Although this article does not assert to put forward a magic bullet and to provide a one-stop solution for dealing with dilemmas about how, why, or whether to evaluate the “goodness” of qualitative research, it offers a platform to assist the researchers in improving their qualitative studies. This work provides an assembly of concerns to reflect on, a series of questions to ask, and multiple sets of criteria to look at, when attempting to determine the quality of qualitative research. Overall, this review underlines the crux of qualitative research and accentuates the need to evaluate such research by the very tenets of its being. Bringing together the vital arguments and delineating the requirements that good qualitative research should satisfy, this review strives to equip the researchers as well as reviewers to make well-versed judgment about the worth and significance of the qualitative research under scrutiny. In a nutshell, a comprehensive portrayal of the research process (from the context of research to the research objectives, research questions and design, speculative foundations, and from approaches of collecting data to analyzing the results, to deriving inferences) frequently proliferates the quality of a qualitative research.

Prospects : A Road Ahead for Qualitative Research

Irrefutably, qualitative research is a vivacious and evolving discipline wherein different epistemological and disciplinary positions have their own characteristics and importance. In addition, not surprisingly, owing to the sprouting and varied features of qualitative research, no consensus has been pulled off till date. Researchers have reflected various concerns and proposed several recommendations for editors and reviewers on conducting reviews of critical qualitative research (Levitt et al., 2021 ; McGinley et al., 2021 ). Following are some prospects and a few recommendations put forward towards the maturation of qualitative research and its quality evaluation:

In general, most of the manuscript and grant reviewers are not qualitative experts. Hence, it is more likely that they would prefer to adopt a broad set of criteria. However, researchers and reviewers need to keep in mind that it is inappropriate to utilize the same approaches and conducts among all qualitative research. Therefore, future work needs to focus on educating researchers and reviewers about the criteria to evaluate qualitative research from within the suitable theoretical and methodological context.

There is an urgent need to refurbish and augment critical assessment of some well-known and widely accepted tools (including checklists such as COREQ, SRQR) to interrogate their applicability on different aspects (along with their epistemological ramifications).

Efforts should be made towards creating more space for creativity, experimentation, and a dialogue between the diverse traditions of qualitative research. This would potentially help to avoid the enforcement of one's own set of quality criteria on the work carried out by others.

Moreover, journal reviewers need to be aware of various methodological practices and philosophical debates.

It is pivotal to highlight the expressions and considerations of qualitative researchers and bring them into a more open and transparent dialogue about assessing qualitative research in techno-scientific, academic, sociocultural, and political rooms.

Frequent debates on the use of evaluative criteria are required to solve some potentially resolved issues (including the applicability of a single set of criteria in multi-disciplinary aspects). Such debates would not only benefit the group of qualitative researchers themselves, but primarily assist in augmenting the well-being and vivacity of the entire discipline.

To conclude, I speculate that the criteria, and my perspective, may transfer to other methods, approaches, and contexts. I hope that they spark dialog and debate – about criteria for excellent qualitative research and the underpinnings of the discipline more broadly – and, therefore, help improve the quality of a qualitative study. Further, I anticipate that this review will assist the researchers to contemplate on the quality of their own research, to substantiate research design and help the reviewers to review qualitative research for journals. On a final note, I pinpoint the need to formulate a framework (encompassing the prerequisites of a qualitative study) by the cohesive efforts of qualitative researchers of different disciplines with different theoretic-paradigmatic origins. I believe that tailoring such a framework (of guiding principles) paves the way for qualitative researchers to consolidate the status of qualitative research in the wide-ranging open science debate. Dialogue on this issue across different approaches is crucial for the impending prospects of socio-techno-educational research.

Amin, M. E. K., Nørgaard, L. S., Cavaco, A. M., Witry, M. J., Hillman, L., Cernasev, A., & Desselle, S. P. (2020). Establishing trustworthiness and authenticity in qualitative pharmacy research. Research in Social and Administrative Pharmacy, 16 (10), 1472–1482.

Article   Google Scholar  

Barker, C., & Pistrang, N. (2005). Quality criteria under methodological pluralism: Implications for conducting and evaluating research. American Journal of Community Psychology, 35 (3–4), 201–212.

Bryman, A., Becker, S., & Sempik, J. (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11 (4), 261–276.

Caelli, K., Ray, L., & Mill, J. (2003). ‘Clear as mud’: Toward greater clarity in generic qualitative research. International Journal of Qualitative Methods, 2 (2), 1–13.

CASP (2021). CASP checklists. Retrieved May 2021 from https://casp-uk.net/casp-tools-checklists/

Cohen, D. J., & Crabtree, B. F. (2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. The Annals of Family Medicine, 6 (4), 331–339.

Denzin, N. K., & Lincoln, Y. S. (2005). Introduction: The discipline and practice of qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The sage handbook of qualitative research (pp. 1–32). Sage Publications Ltd.

Google Scholar  

Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38 (3), 215–229.

Epp, A. M., & Otnes, C. C. (2021). High-quality qualitative research: Getting into gear. Journal of Service Research . https://doi.org/10.1177/1094670520961445

Guba, E. G. (1990). The paradigm dialog. In Alternative paradigms conference, mar, 1989, Indiana u, school of education, San Francisco, ca, us . Sage Publications, Inc.

Hammersley, M. (2007). The issue of quality in qualitative research. International Journal of Research and Method in Education, 30 (3), 287–305.

Haven, T. L., Errington, T. M., Gleditsch, K. S., van Grootel, L., Jacobs, A. M., Kern, F. G., & Mokkink, L. B. (2020). Preregistering qualitative research: A Delphi study. International Journal of Qualitative Methods, 19 , 1609406920976417.

Hays, D. G., & McKibben, W. B. (2021). Promoting rigorous research: Generalizability and qualitative research. Journal of Counseling and Development, 99 (2), 178–188.

Horsburgh, D. (2003). Evaluation of qualitative research. Journal of Clinical Nursing, 12 (2), 307–312.

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10 (1), 42–46.

Johnson, J. L., Adkins, D., & Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. American Journal of Pharmaceutical Education, 84 (1), 7120.

Johnson, P., Buehring, A., Cassell, C., & Symon, G. (2006). Evaluating qualitative management research: Towards a contingent criteriology. International Journal of Management Reviews, 8 (3), 131–156.

Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 23 (1), 67–93.

Lather, P. (2004). This is your father’s paradigm: Government intrusion and the case of qualitative research in education. Qualitative Inquiry, 10 (1), 15–34.

Levitt, H. M., Morrill, Z., Collins, K. M., & Rizo, J. L. (2021). The methodological integrity of critical qualitative research: Principles to support design and research review. Journal of Counseling Psychology, 68 (3), 357.

Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation, 1986 (30), 73–84.

Lincoln, Y. S., & Guba, E. G. (2000). Paradigmatic controversies, contradictions and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 163–188). Sage Publications.

Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology, 91 (1), 1–20.

Mays, N., & Pope, C. (2020). Quality in qualitative research. Qualitative Research in Health Care . https://doi.org/10.1002/9781119410867.ch15

McGinley, S., Wei, W., Zhang, L., & Zheng, Y. (2021). The state of qualitative research in hospitality: A 5-year review 2014 to 2019. Cornell Hospitality Quarterly, 62 (1), 8–20.

Merriam, S., & Tisdell, E. (2016). Qualitative research: A guide to design and implementation. San Francisco, US.

Meyer, M., & Dykes, J. (2019). Criteria for rigor in visualization design study. IEEE Transactions on Visualization and Computer Graphics, 26 (1), 87–97.

Monrouxe, L. V., & Rees, C. E. (2020). When I say… quantification in qualitative research. Medical Education, 54 (3), 186–187.

Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52 (2), 250.

Morse, J. M. (2003). A review committee’s guide for evaluating qualitative proposals. Qualitative Health Research, 13 (6), 833–851.

Nassaji, H. (2020). Good qualitative research. Language Teaching Research, 24 (4), 427–431.

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89 (9), 1245–1251.

O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19 , 1609406919899220.

Reid, A., & Gough, S. (2000). Guidelines for reporting and evaluating qualitative research: What are the alternatives? Environmental Education Research, 6 (1), 59–91.

Rocco, T. S. (2010). Criteria for evaluating qualitative studies. Human Resource Development International . https://doi.org/10.1080/13678868.2010.501959

Sandberg, J. (2000). Understanding human competence at work: An interpretative approach. Academy of Management Journal, 43 (1), 9–25.

Schwandt, T. A. (1996). Farewell to criteriology. Qualitative Inquiry, 2 (1), 58–72.

Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5 (4), 465–478.

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22 (2), 63–75.

Sparkes, A. C. (2001). Myth 94: Qualitative health researchers will agree about validity. Qualitative Health Research, 11 (4), 538–552.

Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2004). Quality in qualitative evaluation: A framework for assessing research evidence.

Stenfors, T., Kajamaa, A., & Bennett, D. (2020). How to assess the quality of qualitative research. The Clinical Teacher, 17 (6), 596–599.

Taylor, E. W., Beck, J., & Ainsworth, E. (2001). Publishing qualitative adult education research: A peer review perspective. Studies in the Education of Adults, 33 (2), 163–179.

Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19 (6), 349–357.

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16 (10), 837–851.

Download references

Open access funding provided by TU Wien (TUW).

Author information

Authors and affiliations.

Faculty of Informatics, Technische Universität Wien, 1040, Vienna, Austria

Drishti Yadav

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Drishti Yadav .

Ethics declarations

Conflict of interest.

The author declares no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Yadav, D. Criteria for Good Qualitative Research: A Comprehensive Review. Asia-Pacific Edu Res 31 , 679–689 (2022). https://doi.org/10.1007/s40299-021-00619-0

Download citation

Accepted : 28 August 2021

Published : 18 September 2021

Issue Date : December 2022

DOI : https://doi.org/10.1007/s40299-021-00619-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Evaluative criteria
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 27 May 2020

How to use and assess qualitative research methods

• Loraine Busetto   ORCID: orcid.org/0000-0002-9228-7875 1 ,
• Wolfgang Wick 1 , 2 &
• Christoph Gumbinger 1  

Neurological Research and Practice volume  2 , Article number:  14 ( 2020 ) Cite this article

768k Accesses

348 Citations

90 Altmetric

Metrics details

This paper aims to provide an overview of the use and assessment of qualitative research methods in the health sciences. Qualitative research can be defined as the study of the nature of phenomena and is especially appropriate for answering questions of why something is (not) observed, assessing complex multi-component interventions, and focussing on intervention improvement. The most common methods of data collection are document study, (non-) participant observations, semi-structured interviews and focus groups. For data analysis, field-notes and audio-recordings are transcribed into protocols and transcripts, and coded using qualitative data management software. Criteria such as checklists, reflexivity, sampling strategies, piloting, co-coding, member-checking and stakeholder involvement can be used to enhance and assess the quality of the research conducted. Using qualitative in addition to quantitative designs will equip us with better tools to address a greater range of research problems, and to fill in blind spots in current neurological research and practice.

The aim of this paper is to provide an overview of qualitative research methods, including hands-on information on how they can be used, reported and assessed. This article is intended for beginning qualitative researchers in the health sciences as well as experienced quantitative researchers who wish to broaden their understanding of qualitative research.

What is qualitative research?

Qualitative research is defined as “the study of the nature of phenomena”, including “their quality, different manifestations, the context in which they appear or the perspectives from which they can be perceived” , but excluding “their range, frequency and place in an objectively determined chain of cause and effect” [ 1 ]. This formal definition can be complemented with a more pragmatic rule of thumb: qualitative research generally includes data in form of words rather than numbers [ 2 ].

Why conduct qualitative research?

Because some research questions cannot be answered using (only) quantitative methods. For example, one Australian study addressed the issue of why patients from Aboriginal communities often present late or not at all to specialist services offered by tertiary care hospitals. Using qualitative interviews with patients and staff, it found one of the most significant access barriers to be transportation problems, including some towns and communities simply not having a bus service to the hospital [ 3 ]. A quantitative study could have measured the number of patients over time or even looked at possible explanatory factors – but only those previously known or suspected to be of relevance. To discover reasons for observed patterns, especially the invisible or surprising ones, qualitative designs are needed.

While qualitative research is common in other fields, it is still relatively underrepresented in health services research. The latter field is more traditionally rooted in the evidence-based-medicine paradigm, as seen in " research that involves testing the effectiveness of various strategies to achieve changes in clinical practice, preferably applying randomised controlled trial study designs (...) " [ 4 ]. This focus on quantitative research and specifically randomised controlled trials (RCT) is visible in the idea of a hierarchy of research evidence which assumes that some research designs are objectively better than others, and that choosing a "lesser" design is only acceptable when the better ones are not practically or ethically feasible [ 5 , 6 ]. Others, however, argue that an objective hierarchy does not exist, and that, instead, the research design and methods should be chosen to fit the specific research question at hand – "questions before methods" [ 2 , 7 , 8 , 9 ]. This means that even when an RCT is possible, some research problems require a different design that is better suited to addressing them. Arguing in JAMA, Berwick uses the example of rapid response teams in hospitals, which he describes as " a complex, multicomponent intervention – essentially a process of social change" susceptible to a range of different context factors including leadership or organisation history. According to him, "[in] such complex terrain, the RCT is an impoverished way to learn. Critics who use it as a truth standard in this context are incorrect" [ 8 ] . Instead of limiting oneself to RCTs, Berwick recommends embracing a wider range of methods , including qualitative ones, which for "these specific applications, (...) are not compromises in learning how to improve; they are superior" [ 8 ].

Research problems that can be approached particularly well using qualitative methods include assessing complex multi-component interventions or systems (of change), addressing questions beyond “what works”, towards “what works for whom when, how and why”, and focussing on intervention improvement rather than accreditation [ 7 , 9 , 10 , 11 , 12 ]. Using qualitative methods can also help shed light on the “softer” side of medical treatment. For example, while quantitative trials can measure the costs and benefits of neuro-oncological treatment in terms of survival rates or adverse effects, qualitative research can help provide a better understanding of patient or caregiver stress, visibility of illness or out-of-pocket expenses.

How to conduct qualitative research?

Given that qualitative research is characterised by flexibility, openness and responsivity to context, the steps of data collection and analysis are not as separate and consecutive as they tend to be in quantitative research [ 13 , 14 ]. As Fossey puts it : “sampling, data collection, analysis and interpretation are related to each other in a cyclical (iterative) manner, rather than following one after another in a stepwise approach” [ 15 ]. The researcher can make educated decisions with regard to the choice of method, how they are implemented, and to which and how many units they are applied [ 13 ]. As shown in Fig.  1 , this can involve several back-and-forth steps between data collection and analysis where new insights and experiences can lead to adaption and expansion of the original plan. Some insights may also necessitate a revision of the research question and/or the research design as a whole. The process ends when saturation is achieved, i.e. when no relevant new information can be found (see also below: sampling and saturation). For reasons of transparency, it is essential for all decisions as well as the underlying reasoning to be well-documented.

Iterative research process

While it is not always explicitly addressed, qualitative methods reflect a different underlying research paradigm than quantitative research (e.g. constructivism or interpretivism as opposed to positivism). The choice of methods can be based on the respective underlying substantive theory or theoretical framework used by the researcher [ 2 ].

Data collection

The methods of qualitative data collection most commonly used in health research are document study, observations, semi-structured interviews and focus groups [ 1 , 14 , 16 , 17 ].

Document study

Document study (also called document analysis) refers to the review by the researcher of written materials [ 14 ]. These can include personal and non-personal documents such as archives, annual reports, guidelines, policy documents, diaries or letters.

Observations

Observations are particularly useful to gain insights into a certain setting and actual behaviour – as opposed to reported behaviour or opinions [ 13 ]. Qualitative observations can be either participant or non-participant in nature. In participant observations, the observer is part of the observed setting, for example a nurse working in an intensive care unit [ 18 ]. In non-participant observations, the observer is “on the outside looking in”, i.e. present in but not part of the situation, trying not to influence the setting by their presence. Observations can be planned (e.g. for 3 h during the day or night shift) or ad hoc (e.g. as soon as a stroke patient arrives at the emergency room). During the observation, the observer takes notes on everything or certain pre-determined parts of what is happening around them, for example focusing on physician-patient interactions or communication between different professional groups. Written notes can be taken during or after the observations, depending on feasibility (which is usually lower during participant observations) and acceptability (e.g. when the observer is perceived to be judging the observed). Afterwards, these field notes are transcribed into observation protocols. If more than one observer was involved, field notes are taken independently, but notes can be consolidated into one protocol after discussions. Advantages of conducting observations include minimising the distance between the researcher and the researched, the potential discovery of topics that the researcher did not realise were relevant and gaining deeper insights into the real-world dimensions of the research problem at hand [ 18 ].

Semi-structured interviews

Hijmans & Kuyper describe qualitative interviews as “an exchange with an informal character, a conversation with a goal” [ 19 ]. Interviews are used to gain insights into a person’s subjective experiences, opinions and motivations – as opposed to facts or behaviours [ 13 ]. Interviews can be distinguished by the degree to which they are structured (i.e. a questionnaire), open (e.g. free conversation or autobiographical interviews) or semi-structured [ 2 , 13 ]. Semi-structured interviews are characterized by open-ended questions and the use of an interview guide (or topic guide/list) in which the broad areas of interest, sometimes including sub-questions, are defined [ 19 ]. The pre-defined topics in the interview guide can be derived from the literature, previous research or a preliminary method of data collection, e.g. document study or observations. The topic list is usually adapted and improved at the start of the data collection process as the interviewer learns more about the field [ 20 ]. Across interviews the focus on the different (blocks of) questions may differ and some questions may be skipped altogether (e.g. if the interviewee is not able or willing to answer the questions or for concerns about the total length of the interview) [ 20 ]. Qualitative interviews are usually not conducted in written format as it impedes on the interactive component of the method [ 20 ]. In comparison to written surveys, qualitative interviews have the advantage of being interactive and allowing for unexpected topics to emerge and to be taken up by the researcher. This can also help overcome a provider or researcher-centred bias often found in written surveys, which by nature, can only measure what is already known or expected to be of relevance to the researcher. Interviews can be audio- or video-taped; but sometimes it is only feasible or acceptable for the interviewer to take written notes [ 14 , 16 , 20 ].

Focus groups

Focus groups are group interviews to explore participants’ expertise and experiences, including explorations of how and why people behave in certain ways [ 1 ]. Focus groups usually consist of 6–8 people and are led by an experienced moderator following a topic guide or “script” [ 21 ]. They can involve an observer who takes note of the non-verbal aspects of the situation, possibly using an observation guide [ 21 ]. Depending on researchers’ and participants’ preferences, the discussions can be audio- or video-taped and transcribed afterwards [ 21 ]. Focus groups are useful for bringing together homogeneous (to a lesser extent heterogeneous) groups of participants with relevant expertise and experience on a given topic on which they can share detailed information [ 21 ]. Focus groups are a relatively easy, fast and inexpensive method to gain access to information on interactions in a given group, i.e. “the sharing and comparing” among participants [ 21 ]. Disadvantages include less control over the process and a lesser extent to which each individual may participate. Moreover, focus group moderators need experience, as do those tasked with the analysis of the resulting data. Focus groups can be less appropriate for discussing sensitive topics that participants might be reluctant to disclose in a group setting [ 13 ]. Moreover, attention must be paid to the emergence of “groupthink” as well as possible power dynamics within the group, e.g. when patients are awed or intimidated by health professionals.

Choosing the “right” method

As explained above, the school of thought underlying qualitative research assumes no objective hierarchy of evidence and methods. This means that each choice of single or combined methods has to be based on the research question that needs to be answered and a critical assessment with regard to whether or to what extent the chosen method can accomplish this – i.e. the “fit” between question and method [ 14 ]. It is necessary for these decisions to be documented when they are being made, and to be critically discussed when reporting methods and results.

Let us assume that our research aim is to examine the (clinical) processes around acute endovascular treatment (EVT), from the patient’s arrival at the emergency room to recanalization, with the aim to identify possible causes for delay and/or other causes for sub-optimal treatment outcome. As a first step, we could conduct a document study of the relevant standard operating procedures (SOPs) for this phase of care – are they up-to-date and in line with current guidelines? Do they contain any mistakes, irregularities or uncertainties that could cause delays or other problems? Regardless of the answers to these questions, the results have to be interpreted based on what they are: a written outline of what care processes in this hospital should look like. If we want to know what they actually look like in practice, we can conduct observations of the processes described in the SOPs. These results can (and should) be analysed in themselves, but also in comparison to the results of the document analysis, especially as regards relevant discrepancies. Do the SOPs outline specific tests for which no equipment can be observed or tasks to be performed by specialized nurses who are not present during the observation? It might also be possible that the written SOP is outdated, but the actual care provided is in line with current best practice. In order to find out why these discrepancies exist, it can be useful to conduct interviews. Are the physicians simply not aware of the SOPs (because their existence is limited to the hospital’s intranet) or do they actively disagree with them or does the infrastructure make it impossible to provide the care as described? Another rationale for adding interviews is that some situations (or all of their possible variations for different patient groups or the day, night or weekend shift) cannot practically or ethically be observed. In this case, it is possible to ask those involved to report on their actions – being aware that this is not the same as the actual observation. A senior physician’s or hospital manager’s description of certain situations might differ from a nurse’s or junior physician’s one, maybe because they intentionally misrepresent facts or maybe because different aspects of the process are visible or important to them. In some cases, it can also be relevant to consider to whom the interviewee is disclosing this information – someone they trust, someone they are otherwise not connected to, or someone they suspect or are aware of being in a potentially “dangerous” power relationship to them. Lastly, a focus group could be conducted with representatives of the relevant professional groups to explore how and why exactly they provide care around EVT. The discussion might reveal discrepancies (between SOPs and actual care or between different physicians) and motivations to the researchers as well as to the focus group members that they might not have been aware of themselves. For the focus group to deliver relevant information, attention has to be paid to its composition and conduct, for example, to make sure that all participants feel safe to disclose sensitive or potentially problematic information or that the discussion is not dominated by (senior) physicians only. The resulting combination of data collection methods is shown in Fig.  2 .

Possible combination of data collection methods

Attributions for icons: “Book” by Serhii Smirnov, “Interview” by Adrien Coquet, FR, “Magnifying Glass” by anggun, ID, “Business communication” by Vectors Market; all from the Noun Project

The combination of multiple data source as described for this example can be referred to as “triangulation”, in which multiple measurements are carried out from different angles to achieve a more comprehensive understanding of the phenomenon under study [ 22 , 23 ].

Data analysis

To analyse the data collected through observations, interviews and focus groups these need to be transcribed into protocols and transcripts (see Fig.  3 ). Interviews and focus groups can be transcribed verbatim , with or without annotations for behaviour (e.g. laughing, crying, pausing) and with or without phonetic transcription of dialects and filler words, depending on what is expected or known to be relevant for the analysis. In the next step, the protocols and transcripts are coded , that is, marked (or tagged, labelled) with one or more short descriptors of the content of a sentence or paragraph [ 2 , 15 , 23 ]. Jansen describes coding as “connecting the raw data with “theoretical” terms” [ 20 ]. In a more practical sense, coding makes raw data sortable. This makes it possible to extract and examine all segments describing, say, a tele-neurology consultation from multiple data sources (e.g. SOPs, emergency room observations, staff and patient interview). In a process of synthesis and abstraction, the codes are then grouped, summarised and/or categorised [ 15 , 20 ]. The end product of the coding or analysis process is a descriptive theory of the behavioural pattern under investigation [ 20 ]. The coding process is performed using qualitative data management software, the most common ones being InVivo, MaxQDA and Atlas.ti. It should be noted that these are data management tools which support the analysis performed by the researcher(s) [ 14 ].

From data collection to data analysis

Attributions for icons: see Fig. 2 , also “Speech to text” by Trevor Dsouza, “Field Notes” by Mike O’Brien, US, “Voice Record” by ProSymbols, US, “Inspection” by Made, AU, and “Cloud” by Graphic Tigers; all from the Noun Project

How to report qualitative research?

Protocols of qualitative research can be published separately and in advance of the study results. However, the aim is not the same as in RCT protocols, i.e. to pre-define and set in stone the research questions and primary or secondary endpoints. Rather, it is a way to describe the research methods in detail, which might not be possible in the results paper given journals’ word limits. Qualitative research papers are usually longer than their quantitative counterparts to allow for deep understanding and so-called “thick description”. In the methods section, the focus is on transparency of the methods used, including why, how and by whom they were implemented in the specific study setting, so as to enable a discussion of whether and how this may have influenced data collection, analysis and interpretation. The results section usually starts with a paragraph outlining the main findings, followed by more detailed descriptions of, for example, the commonalities, discrepancies or exceptions per category [ 20 ]. Here it is important to support main findings by relevant quotations, which may add information, context, emphasis or real-life examples [ 20 , 23 ]. It is subject to debate in the field whether it is relevant to state the exact number or percentage of respondents supporting a certain statement (e.g. “Five interviewees expressed negative feelings towards XYZ”) [ 21 ].

How to combine qualitative with quantitative research?

Qualitative methods can be combined with other methods in multi- or mixed methods designs, which “[employ] two or more different methods [ …] within the same study or research program rather than confining the research to one single method” [ 24 ]. Reasons for combining methods can be diverse, including triangulation for corroboration of findings, complementarity for illustration and clarification of results, expansion to extend the breadth and range of the study, explanation of (unexpected) results generated with one method with the help of another, or offsetting the weakness of one method with the strength of another [ 1 , 17 , 24 , 25 , 26 ]. The resulting designs can be classified according to when, why and how the different quantitative and/or qualitative data strands are combined. The three most common types of mixed method designs are the convergent parallel design , the explanatory sequential design and the exploratory sequential design. The designs with examples are shown in Fig.  4 .

Three common mixed methods designs

In the convergent parallel design, a qualitative study is conducted in parallel to and independently of a quantitative study, and the results of both studies are compared and combined at the stage of interpretation of results. Using the above example of EVT provision, this could entail setting up a quantitative EVT registry to measure process times and patient outcomes in parallel to conducting the qualitative research outlined above, and then comparing results. Amongst other things, this would make it possible to assess whether interview respondents’ subjective impressions of patients receiving good care match modified Rankin Scores at follow-up, or whether observed delays in care provision are exceptions or the rule when compared to door-to-needle times as documented in the registry. In the explanatory sequential design, a quantitative study is carried out first, followed by a qualitative study to help explain the results from the quantitative study. This would be an appropriate design if the registry alone had revealed relevant delays in door-to-needle times and the qualitative study would be used to understand where and why these occurred, and how they could be improved. In the exploratory design, the qualitative study is carried out first and its results help informing and building the quantitative study in the next step [ 26 ]. If the qualitative study around EVT provision had shown a high level of dissatisfaction among the staff members involved, a quantitative questionnaire investigating staff satisfaction could be set up in the next step, informed by the qualitative study on which topics dissatisfaction had been expressed. Amongst other things, the questionnaire design would make it possible to widen the reach of the research to more respondents from different (types of) hospitals, regions, countries or settings, and to conduct sub-group analyses for different professional groups.

How to assess qualitative research?

A variety of assessment criteria and lists have been developed for qualitative research, ranging in their focus and comprehensiveness [ 14 , 17 , 27 ]. However, none of these has been elevated to the “gold standard” in the field. In the following, we therefore focus on a set of commonly used assessment criteria that, from a practical standpoint, a researcher can look for when assessing a qualitative research report or paper.

Assessors should check the authors’ use of and adherence to the relevant reporting checklists (e.g. Standards for Reporting Qualitative Research (SRQR)) to make sure all items that are relevant for this type of research are addressed [ 23 , 28 ]. Discussions of quantitative measures in addition to or instead of these qualitative measures can be a sign of lower quality of the research (paper). Providing and adhering to a checklist for qualitative research contributes to an important quality criterion for qualitative research, namely transparency [ 15 , 17 , 23 ].

Reflexivity

While methodological transparency and complete reporting is relevant for all types of research, some additional criteria must be taken into account for qualitative research. This includes what is called reflexivity, i.e. sensitivity to the relationship between the researcher and the researched, including how contact was established and maintained, or the background and experience of the researcher(s) involved in data collection and analysis. Depending on the research question and population to be researched this can be limited to professional experience, but it may also include gender, age or ethnicity [ 17 , 27 ]. These details are relevant because in qualitative research, as opposed to quantitative research, the researcher as a person cannot be isolated from the research process [ 23 ]. It may influence the conversation when an interviewed patient speaks to an interviewer who is a physician, or when an interviewee is asked to discuss a gynaecological procedure with a male interviewer, and therefore the reader must be made aware of these details [ 19 ].

Sampling and saturation

The aim of qualitative sampling is for all variants of the objects of observation that are deemed relevant for the study to be present in the sample “ to see the issue and its meanings from as many angles as possible” [ 1 , 16 , 19 , 20 , 27 ] , and to ensure “information-richness [ 15 ]. An iterative sampling approach is advised, in which data collection (e.g. five interviews) is followed by data analysis, followed by more data collection to find variants that are lacking in the current sample. This process continues until no new (relevant) information can be found and further sampling becomes redundant – which is called saturation [ 1 , 15 ] . In other words: qualitative data collection finds its end point not a priori , but when the research team determines that saturation has been reached [ 29 , 30 ].

This is also the reason why most qualitative studies use deliberate instead of random sampling strategies. This is generally referred to as “ purposive sampling” , in which researchers pre-define which types of participants or cases they need to include so as to cover all variations that are expected to be of relevance, based on the literature, previous experience or theory (i.e. theoretical sampling) [ 14 , 20 ]. Other types of purposive sampling include (but are not limited to) maximum variation sampling, critical case sampling or extreme or deviant case sampling [ 2 ]. In the above EVT example, a purposive sample could include all relevant professional groups and/or all relevant stakeholders (patients, relatives) and/or all relevant times of observation (day, night and weekend shift).

Assessors of qualitative research should check whether the considerations underlying the sampling strategy were sound and whether or how researchers tried to adapt and improve their strategies in stepwise or cyclical approaches between data collection and analysis to achieve saturation [ 14 ].

Good qualitative research is iterative in nature, i.e. it goes back and forth between data collection and analysis, revising and improving the approach where necessary. One example of this are pilot interviews, where different aspects of the interview (especially the interview guide, but also, for example, the site of the interview or whether the interview can be audio-recorded) are tested with a small number of respondents, evaluated and revised [ 19 ]. In doing so, the interviewer learns which wording or types of questions work best, or which is the best length of an interview with patients who have trouble concentrating for an extended time. Of course, the same reasoning applies to observations or focus groups which can also be piloted.

Ideally, coding should be performed by at least two researchers, especially at the beginning of the coding process when a common approach must be defined, including the establishment of a useful coding list (or tree), and when a common meaning of individual codes must be established [ 23 ]. An initial sub-set or all transcripts can be coded independently by the coders and then compared and consolidated after regular discussions in the research team. This is to make sure that codes are applied consistently to the research data.

Member checking

Member checking, also called respondent validation , refers to the practice of checking back with study respondents to see if the research is in line with their views [ 14 , 27 ]. This can happen after data collection or analysis or when first results are available [ 23 ]. For example, interviewees can be provided with (summaries of) their transcripts and asked whether they believe this to be a complete representation of their views or whether they would like to clarify or elaborate on their responses [ 17 ]. Respondents’ feedback on these issues then becomes part of the data collection and analysis [ 27 ].

Stakeholder involvement

In those niches where qualitative approaches have been able to evolve and grow, a new trend has seen the inclusion of patients and their representatives not only as study participants (i.e. “members”, see above) but as consultants to and active participants in the broader research process [ 31 , 32 , 33 ]. The underlying assumption is that patients and other stakeholders hold unique perspectives and experiences that add value beyond their own single story, making the research more relevant and beneficial to researchers, study participants and (future) patients alike [ 34 , 35 ]. Using the example of patients on or nearing dialysis, a recent scoping review found that 80% of clinical research did not address the top 10 research priorities identified by patients and caregivers [ 32 , 36 ]. In this sense, the involvement of the relevant stakeholders, especially patients and relatives, is increasingly being seen as a quality indicator in and of itself.

How not to assess qualitative research

The above overview does not include certain items that are routine in assessments of quantitative research. What follows is a non-exhaustive, non-representative, experience-based list of the quantitative criteria often applied to the assessment of qualitative research, as well as an explanation of the limited usefulness of these endeavours.

Protocol adherence

Given the openness and flexibility of qualitative research, it should not be assessed by how well it adheres to pre-determined and fixed strategies – in other words: its rigidity. Instead, the assessor should look for signs of adaptation and refinement based on lessons learned from earlier steps in the research process.

Sample size

For the reasons explained above, qualitative research does not require specific sample sizes, nor does it require that the sample size be determined a priori [ 1 , 14 , 27 , 37 , 38 , 39 ]. Sample size can only be a useful quality indicator when related to the research purpose, the chosen methodology and the composition of the sample, i.e. who was included and why.

Randomisation

While some authors argue that randomisation can be used in qualitative research, this is not commonly the case, as neither its feasibility nor its necessity or usefulness has been convincingly established for qualitative research [ 13 , 27 ]. Relevant disadvantages include the negative impact of a too large sample size as well as the possibility (or probability) of selecting “ quiet, uncooperative or inarticulate individuals ” [ 17 ]. Qualitative studies do not use control groups, either.

Interrater reliability, variability and other “objectivity checks”

The concept of “interrater reliability” is sometimes used in qualitative research to assess to which extent the coding approach overlaps between the two co-coders. However, it is not clear what this measure tells us about the quality of the analysis [ 23 ]. This means that these scores can be included in qualitative research reports, preferably with some additional information on what the score means for the analysis, but it is not a requirement. Relatedly, it is not relevant for the quality or “objectivity” of qualitative research to separate those who recruited the study participants and collected and analysed the data. Experiences even show that it might be better to have the same person or team perform all of these tasks [ 20 ]. First, when researchers introduce themselves during recruitment this can enhance trust when the interview takes place days or weeks later with the same researcher. Second, when the audio-recording is transcribed for analysis, the researcher conducting the interviews will usually remember the interviewee and the specific interview situation during data analysis. This might be helpful in providing additional context information for interpretation of data, e.g. on whether something might have been meant as a joke [ 18 ].

Not being quantitative research

Being qualitative research instead of quantitative research should not be used as an assessment criterion if it is used irrespectively of the research problem at hand. Similarly, qualitative research should not be required to be combined with quantitative research per se – unless mixed methods research is judged as inherently better than single-method research. In this case, the same criterion should be applied for quantitative studies without a qualitative component.

The main take-away points of this paper are summarised in Table 1 . We aimed to show that, if conducted well, qualitative research can answer specific research questions that cannot to be adequately answered using (only) quantitative designs. Seeing qualitative and quantitative methods as equal will help us become more aware and critical of the “fit” between the research problem and our chosen methods: I can conduct an RCT to determine the reasons for transportation delays of acute stroke patients – but should I? It also provides us with a greater range of tools to tackle a greater range of research problems more appropriately and successfully, filling in the blind spots on one half of the methodological spectrum to better address the whole complexity of neurological research and practice.

Availability of data and materials

Not applicable.

Abbreviations

Endovascular treatment

Randomised Controlled Trial

Standard Operating Procedure

Standards for Reporting Qualitative Research

Philipsen, H., & Vernooij-Dassen, M. (2007). Kwalitatief onderzoek: nuttig, onmisbaar en uitdagend. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [Qualitative research: useful, indispensable and challenging. In: Qualitative research: Practical methods for medical practice (pp. 5–12). Houten: Bohn Stafleu van Loghum.

Chapter   Google Scholar  

Punch, K. F. (2013). Introduction to social research: Quantitative and qualitative approaches . London: Sage.

Kelly, J., Dwyer, J., Willis, E., & Pekarsky, B. (2014). Travelling to the city for hospital care: Access factors in country aboriginal patient journeys. Australian Journal of Rural Health, 22 (3), 109–113.

Article   Google Scholar  

Nilsen, P., Ståhl, C., Roback, K., & Cairney, P. (2013). Never the twain shall meet? - a comparison of implementation science and policy implementation research. Implementation Science, 8 (1), 1–12.

Howick J, Chalmers I, Glasziou, P., Greenhalgh, T., Heneghan, C., Liberati, A., Moschetti, I., Phillips, B., & Thornton, H. (2011). The 2011 Oxford CEBM evidence levels of evidence (introductory document) . Oxford Center for Evidence Based Medicine. https://www.cebm.net/2011/06/2011-oxford-cebm-levels-evidence-introductory-document/ .

Eakin, J. M. (2016). Educating critical qualitative health researchers in the land of the randomized controlled trial. Qualitative Inquiry, 22 (2), 107–118.

May, A., & Mathijssen, J. (2015). Alternatieven voor RCT bij de evaluatie van effectiviteit van interventies!? Eindrapportage. In Alternatives for RCTs in the evaluation of effectiveness of interventions!? Final report .

Google Scholar  

Berwick, D. M. (2008). The science of improvement. Journal of the American Medical Association, 299 (10), 1182–1184.

Article   CAS   Google Scholar  

Christ, T. W. (2014). Scientific-based research and randomized controlled trials, the “gold” standard? Alternative paradigms and mixed methodologies. Qualitative Inquiry, 20 (1), 72–80.

Lamont, T., Barber, N., Jd, P., Fulop, N., Garfield-Birkbeck, S., Lilford, R., Mear, L., Raine, R., & Fitzpatrick, R. (2016). New approaches to evaluating complex health and care systems. BMJ, 352:i154.

Drabble, S. J., & O’Cathain, A. (2015). Moving from Randomized Controlled Trials to Mixed Methods Intervention Evaluation. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry (pp. 406–425). London: Oxford University Press.

Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science : IS, 8 , 117.

Hak, T. (2007). Waarnemingsmethoden in kwalitatief onderzoek. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [Observation methods in qualitative research] (pp. 13–25). Houten: Bohn Stafleu van Loghum.

Russell, C. K., & Gregory, D. M. (2003). Evaluation of qualitative research studies. Evidence Based Nursing, 6 (2), 36–40.

Fossey, E., Harvey, C., McDermott, F., & Davidson, L. (2002). Understanding and evaluating qualitative research. Australian and New Zealand Journal of Psychiatry, 36 , 717–732.

Yanow, D. (2000). Conducting interpretive policy analysis (Vol. 47). Thousand Oaks: Sage University Papers Series on Qualitative Research Methods.

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22 , 63–75.

van der Geest, S. (2006). Participeren in ziekte en zorg: meer over kwalitatief onderzoek. Huisarts en Wetenschap, 49 (4), 283–287.

Hijmans, E., & Kuyper, M. (2007). Het halfopen interview als onderzoeksmethode. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [The half-open interview as research method (pp. 43–51). Houten: Bohn Stafleu van Loghum.

Jansen, H. (2007). Systematiek en toepassing van de kwalitatieve survey. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [Systematics and implementation of the qualitative survey (pp. 27–41). Houten: Bohn Stafleu van Loghum.

Pv, R., & Peremans, L. (2007). Exploreren met focusgroepgesprekken: de ‘stem’ van de groep onder de loep. In L. PLBJ & H. TCo (Eds.), Kwalitatief onderzoek: Praktische methoden voor de medische praktijk . [Exploring with focus group conversations: the “voice” of the group under the magnifying glass (pp. 53–64). Houten: Bohn Stafleu van Loghum.

Carter, N., Bryant-Lukosius, D., DiCenso, A., Blythe, J., & Neville, A. J. (2014). The use of triangulation in qualitative research. Oncology Nursing Forum, 41 (5), 545–547.

Boeije H: Analyseren in kwalitatief onderzoek: Denken en doen, [Analysis in qualitative research: Thinking and doing] vol. Den Haag Boom Lemma uitgevers; 2012.

Hunter, A., & Brewer, J. (2015). Designing Multimethod Research. In S. Hesse-Biber & R. B. Johnson (Eds.), The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry (pp. 185–205). London: Oxford University Press.

Archibald, M. M., Radil, A. I., Zhang, X., & Hanson, W. E. (2015). Current mixed methods practices in qualitative research: A content analysis of leading journals. International Journal of Qualitative Methods, 14 (2), 5–33.

Creswell, J. W., & Plano Clark, V. L. (2011). Choosing a Mixed Methods Design. In Designing and Conducting Mixed Methods Research . Thousand Oaks: SAGE Publications.

Mays, N., & Pope, C. (2000). Assessing quality in qualitative research. BMJ, 320 (7226), 50–52.

O'Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine : Journal of the Association of American Medical Colleges, 89 (9), 1245–1251.

Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018). Saturation in qualitative research: Exploring its conceptualization and operationalization. Quality and Quantity, 52 (4), 1893–1907.

Moser, A., & Korstjens, I. (2018). Series: Practical guidance to qualitative research. Part 3: Sampling, data collection and analysis. European Journal of General Practice, 24 (1), 9–18.

Marlett, N., Shklarov, S., Marshall, D., Santana, M. J., & Wasylak, T. (2015). Building new roles and relationships in research: A model of patient engagement research. Quality of Life Research : an international journal of quality of life aspects of treatment, care and rehabilitation, 24 (5), 1057–1067.

Demian, M. N., Lam, N. N., Mac-Way, F., Sapir-Pichhadze, R., & Fernandez, N. (2017). Opportunities for engaging patients in kidney research. Canadian Journal of Kidney Health and Disease, 4 , 2054358117703070–2054358117703070.

Noyes, J., McLaughlin, L., Morgan, K., Roberts, A., Stephens, M., Bourne, J., Houlston, M., Houlston, J., Thomas, S., Rhys, R. G., et al. (2019). Designing a co-productive study to overcome known methodological challenges in organ donation research with bereaved family members. Health Expectations . 22(4):824–35.

Piil, K., Jarden, M., & Pii, K. H. (2019). Research agenda for life-threatening cancer. European Journal Cancer Care (Engl), 28 (1), e12935.

Hofmann, D., Ibrahim, F., Rose, D., Scott, D. L., Cope, A., Wykes, T., & Lempp, H. (2015). Expectations of new treatment in rheumatoid arthritis: Developing a patient-generated questionnaire. Health Expectations : an international journal of public participation in health care and health policy, 18 (5), 995–1008.

Jun, M., Manns, B., Laupacis, A., Manns, L., Rehal, B., Crowe, S., & Hemmelgarn, B. R. (2015). Assessing the extent to which current clinical research is consistent with patient priorities: A scoping review using a case study in patients on or nearing dialysis. Canadian Journal of Kidney Health and Disease, 2 , 35.

Elsie Baker, S., & Edwards, R. (2012). How many qualitative interviews is enough? In National Centre for Research Methods Review Paper . National Centre for Research Methods. http://eprints.ncrm.ac.uk/2273/4/how_many_interviews.pdf .

Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing & Health, 18 (2), 179–183.

Sim, J., Saunders, B., Waterfield, J., & Kingstone, T. (2018). Can sample size in qualitative research be determined a priori? International Journal of Social Research Methodology, 21 (5), 619–634.

Download references

Acknowledgements

no external funding.

Author information

Authors and affiliations.

Department of Neurology, Heidelberg University Hospital, Im Neuenheimer Feld 400, 69120, Heidelberg, Germany

Loraine Busetto, Wolfgang Wick & Christoph Gumbinger

Clinical Cooperation Unit Neuro-Oncology, German Cancer Research Center, Heidelberg, Germany

Wolfgang Wick

You can also search for this author in PubMed   Google Scholar

Contributions

LB drafted the manuscript; WW and CG revised the manuscript; all authors approved the final versions.

Corresponding author

Correspondence to Loraine Busetto .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Busetto, L., Wick, W. & Gumbinger, C. How to use and assess qualitative research methods. Neurol. Res. Pract. 2 , 14 (2020). https://doi.org/10.1186/s42466-020-00059-z

Download citation

Received : 30 January 2020

Accepted : 22 April 2020

Published : 27 May 2020

DOI : https://doi.org/10.1186/s42466-020-00059-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Mixed methods
• Quality assessment

Neurological Research and Practice

ISSN: 2524-3489

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 05 October 2018

Interviews and focus groups in qualitative research: an update for the digital age

• P. Gill 1 &
• J. Baillie 2  

British Dental Journal volume  225 ,  pages 668–672 ( 2018 ) Cite this article

31k Accesses

61 Citations

20 Altmetric

Metrics details

Highlights that qualitative research is used increasingly in dentistry. Interviews and focus groups remain the most common qualitative methods of data collection.

Suggests the advent of digital technologies has transformed how qualitative research can now be undertaken.

Suggests interviews and focus groups can offer significant, meaningful insight into participants' experiences, beliefs and perspectives, which can help to inform developments in dental practice.

Qualitative research is used increasingly in dentistry, due to its potential to provide meaningful, in-depth insights into participants' experiences, perspectives, beliefs and behaviours. These insights can subsequently help to inform developments in dental practice and further related research. The most common methods of data collection used in qualitative research are interviews and focus groups. While these are primarily conducted face-to-face, the ongoing evolution of digital technologies, such as video chat and online forums, has further transformed these methods of data collection. This paper therefore discusses interviews and focus groups in detail, outlines how they can be used in practice, how digital technologies can further inform the data collection process, and what these methods can offer dentistry.

You have full access to this article via your institution.

Similar content being viewed by others

Interviews in the social sciences

Professionalism in dentistry: deconstructing common terminology

A review of technical and quality assessment considerations of audio-visual and web-conferencing focus groups in qualitative health research, introduction.

Traditionally, research in dentistry has primarily been quantitative in nature. 1 However, in recent years, there has been a growing interest in qualitative research within the profession, due to its potential to further inform developments in practice, policy, education and training. Consequently, in 2008, the British Dental Journal (BDJ) published a four paper qualitative research series, 2 , 3 , 4 , 5 to help increase awareness and understanding of this particular methodological approach.

Since the papers were originally published, two scoping reviews have demonstrated the ongoing proliferation in the use of qualitative research within the field of oral healthcare. 1 , 6 To date, the original four paper series continue to be well cited and two of the main papers remain widely accessed among the BDJ readership. 2 , 3 The potential value of well-conducted qualitative research to evidence-based practice is now also widely recognised by service providers, policy makers, funding bodies and those who commission, support and use healthcare research.

Besides increasing standalone use, qualitative methods are now also routinely incorporated into larger mixed method study designs, such as clinical trials, as they can offer additional, meaningful insights into complex problems that simply could not be provided by quantitative methods alone. Qualitative methods can also be used to further facilitate in-depth understanding of important aspects of clinical trial processes, such as recruitment. For example, Ellis et al . investigated why edentulous older patients, dissatisfied with conventional dentures, decline implant treatment, despite its established efficacy, and frequently refuse to participate in related randomised clinical trials, even when financial constraints are removed. 7 Through the use of focus groups in Canada and the UK, the authors found that fears of pain and potential complications, along with perceived embarrassment, exacerbated by age, are common reasons why older patients typically refuse dental implants. 7

The last decade has also seen further developments in qualitative research, due to the ongoing evolution of digital technologies. These developments have transformed how researchers can access and share information, communicate and collaborate, recruit and engage participants, collect and analyse data and disseminate and translate research findings. 8 Where appropriate, such technologies are therefore capable of extending and enhancing how qualitative research is undertaken. 9 For example, it is now possible to collect qualitative data via instant messaging, email or online/video chat, using appropriate online platforms.

These innovative approaches to research are therefore cost-effective, convenient, reduce geographical constraints and are often useful for accessing 'hard to reach' participants (for example, those who are immobile or socially isolated). 8 , 9 However, digital technologies are still relatively new and constantly evolving and therefore present a variety of pragmatic and methodological challenges. Furthermore, given their very nature, their use in many qualitative studies and/or with certain participant groups may be inappropriate and should therefore always be carefully considered. While it is beyond the scope of this paper to provide a detailed explication regarding the use of digital technologies in qualitative research, insight is provided into how such technologies can be used to facilitate the data collection process in interviews and focus groups.

In light of such developments, it is perhaps therefore timely to update the main paper 3 of the original BDJ series. As with the previous publications, this paper has been purposely written in an accessible style, to enhance readability, particularly for those who are new to qualitative research. While the focus remains on the most common qualitative methods of data collection – interviews and focus groups – appropriate revisions have been made to provide a novel perspective, and should therefore be helpful to those who would like to know more about qualitative research. This paper specifically focuses on undertaking qualitative research with adult participants only.

Overview of qualitative research

Qualitative research is an approach that focuses on people and their experiences, behaviours and opinions. 10 , 11 The qualitative researcher seeks to answer questions of 'how' and 'why', providing detailed insight and understanding, 11 which quantitative methods cannot reach. 12 Within qualitative research, there are distinct methodologies influencing how the researcher approaches the research question, data collection and data analysis. 13 For example, phenomenological studies focus on the lived experience of individuals, explored through their description of the phenomenon. Ethnographic studies explore the culture of a group and typically involve the use of multiple methods to uncover the issues. 14

While methodology is the 'thinking tool', the methods are the 'doing tools'; 13 the ways in which data are collected and analysed. There are multiple qualitative data collection methods, including interviews, focus groups, observations, documentary analysis, participant diaries, photography and videography. Two of the most commonly used qualitative methods are interviews and focus groups, which are explored in this article. The data generated through these methods can be analysed in one of many ways, according to the methodological approach chosen. A common approach is thematic data analysis, involving the identification of themes and subthemes across the data set. Further information on approaches to qualitative data analysis has been discussed elsewhere. 1

Qualitative research is an evolving and adaptable approach, used by different disciplines for different purposes. Traditionally, qualitative data, specifically interviews, focus groups and observations, have been collected face-to-face with participants. In more recent years, digital technologies have contributed to the ongoing evolution of qualitative research. Digital technologies offer researchers different ways of recruiting participants and collecting data, and offer participants opportunities to be involved in research that is not necessarily face-to-face.

Research interviews are a fundamental qualitative research method 15 and are utilised across methodological approaches. Interviews enable the researcher to learn in depth about the perspectives, experiences, beliefs and motivations of the participant. 3 , 16 Examples include, exploring patients' perspectives of fear/anxiety triggers in dental treatment, 17 patients' experiences of oral health and diabetes, 18 and dental students' motivations for their choice of career. 19

Interviews may be structured, semi-structured or unstructured, 3 according to the purpose of the study, with less structured interviews facilitating a more in depth and flexible interviewing approach. 20 Structured interviews are similar to verbal questionnaires and are used if the researcher requires clarification on a topic; however they produce less in-depth data about a participant's experience. 3 Unstructured interviews may be used when little is known about a topic and involves the researcher asking an opening question; 3 the participant then leads the discussion. 20 Semi-structured interviews are commonly used in healthcare research, enabling the researcher to ask predetermined questions, 20 while ensuring the participant discusses issues they feel are important.

Interviews can be undertaken face-to-face or using digital methods when the researcher and participant are in different locations. Audio-recording the interview, with the consent of the participant, is essential for all interviews regardless of the medium as it enables accurate transcription; the process of turning the audio file into a word-for-word transcript. This transcript is the data, which the researcher then analyses according to the chosen approach.

Types of interview

Qualitative studies often utilise one-to-one, face-to-face interviews with research participants. This involves arranging a mutually convenient time and place to meet the participant, signing a consent form and audio-recording the interview. However, digital technologies have expanded the potential for interviews in research, enabling individuals to participate in qualitative research regardless of location.

Telephone interviews can be a useful alternative to face-to-face interviews and are commonly used in qualitative research. They enable participants from different geographical areas to participate and may be less onerous for participants than meeting a researcher in person. 15 A qualitative study explored patients' perspectives of dental implants and utilised telephone interviews due to the quality of the data that could be yielded. 21 The researcher needs to consider how they will audio record the interview, which can be facilitated by purchasing a recorder that connects directly to the telephone. One potential disadvantage of telephone interviews is the inability of the interviewer and researcher to see each other. This is resolved using software for audio and video calls online – such as Skype – to conduct interviews with participants in qualitative studies. Advantages of this approach include being able to see the participant if video calls are used, enabling observation of non-verbal communication, and the software can be free to use. However, participants are required to have a device and internet connection, as well as being computer literate, potentially limiting who can participate in the study. One qualitative study explored the role of dental hygienists in reducing oral health disparities in Canada. 22 The researcher conducted interviews using Skype, which enabled dental hygienists from across Canada to be interviewed within the research budget, accommodating the participants' schedules. 22

A less commonly used approach to qualitative interviews is the use of social virtual worlds. A qualitative study accessed a social virtual world – Second Life – to explore the health literacy skills of individuals who use social virtual worlds to access health information. 23 The researcher created an avatar and interview room, and undertook interviews with participants using voice and text methods. 23 This approach to recruitment and data collection enables individuals from diverse geographical locations to participate, while remaining anonymous if they wish. Furthermore, for interviews conducted using text methods, transcription of the interview is not required as the researcher can save the written conversation with the participant, with the participant's consent. However, the researcher and participant need to be familiar with how the social virtual world works to engage in an interview this way.

Conducting an interview

Ensuring informed consent before any interview is a fundamental aspect of the research process. Participants in research must be afforded autonomy and respect; consent should be informed and voluntary. 24 Individuals should have the opportunity to read an information sheet about the study, ask questions, understand how their data will be stored and used, and know that they are free to withdraw at any point without reprisal. The qualitative researcher should take written consent before undertaking the interview. In a face-to-face interview, this is straightforward: the researcher and participant both sign copies of the consent form, keeping one each. However, this approach is less straightforward when the researcher and participant do not meet in person. A recent protocol paper outlined an approach for taking consent for telephone interviews, which involved: audio recording the participant agreeing to each point on the consent form; the researcher signing the consent form and keeping a copy; and posting a copy to the participant. 25 This process could be replicated in other interview studies using digital methods.

There are advantages and disadvantages of using face-to-face and digital methods for research interviews. Ultimately, for both approaches, the quality of the interview is determined by the researcher. 16 Appropriate training and preparation are thus required. Healthcare professionals can use their interpersonal communication skills when undertaking a research interview, particularly questioning, listening and conversing. 3 However, the purpose of an interview is to gain information about the study topic, 26 rather than offering help and advice. 3 The researcher therefore needs to listen attentively to participants, enabling them to describe their experience without interruption. 3 The use of active listening skills also help to facilitate the interview. 14 Spradley outlined elements and strategies for research interviews, 27 which are a useful guide for qualitative researchers:

Greeting and explaining the project/interview

Asking descriptive (broad), structural (explore response to descriptive) and contrast (difference between) questions

Asymmetry between the researcher and participant talking

Expressing interest and cultural ignorance

Repeating, restating and incorporating the participant's words when asking questions

Creating hypothetical situations

Asking friendly questions

Knowing when to leave.

For semi-structured interviews, a topic guide (also called an interview schedule) is used to guide the content of the interview – an example of a topic guide is outlined in Box 1 . The topic guide, usually based on the research questions, existing literature and, for healthcare professionals, their clinical experience, is developed by the research team. The topic guide should include open ended questions that elicit in-depth information, and offer participants the opportunity to talk about issues important to them. This is vital in qualitative research where the researcher is interested in exploring the experiences and perspectives of participants. It can be useful for qualitative researchers to pilot the topic guide with the first participants, 10 to ensure the questions are relevant and understandable, and amending the questions if required.

Regardless of the medium of interview, the researcher must consider the setting of the interview. For face-to-face interviews, this could be in the participant's home, in an office or another mutually convenient location. A quiet location is preferable to promote confidentiality, enable the researcher and participant to concentrate on the conversation, and to facilitate accurate audio-recording of the interview. For interviews using digital methods the same principles apply: a quiet, private space where the researcher and participant feel comfortable and confident to participate in an interview.

Box 1: Example of a topic guide

Study focus: Parents' experiences of brushing their child's (aged 0–5) teeth

1. Can you tell me about your experience of cleaning your child's teeth?

How old was your child when you started cleaning their teeth?

Why did you start cleaning their teeth at that point?

How often do you brush their teeth?

What do you use to brush their teeth and why?

2. Could you explain how you find cleaning your child's teeth?

Do you find anything difficult?

What makes cleaning their teeth easier for you?

3. How has your experience of cleaning your child's teeth changed over time?

Has it become easier or harder?

Have you changed how often and how you clean their teeth? If so, why?

4. Could you describe how your child finds having their teeth cleaned?

What do they enjoy about having their teeth cleaned?

Is there anything they find upsetting about having their teeth cleaned?

5. Where do you look for information/advice about cleaning your child's teeth?

What did your health visitor tell you about cleaning your child's teeth? (If anything)

What has the dentist told you about caring for your child's teeth? (If visited)

Have any family members given you advice about how to clean your child's teeth? If so, what did they tell you? Did you follow their advice?

6. Is there anything else you would like to discuss about this?

Focus groups

A focus group is a moderated group discussion on a pre-defined topic, for research purposes. 28 , 29 While not aligned to a particular qualitative methodology (for example, grounded theory or phenomenology) as such, focus groups are used increasingly in healthcare research, as they are useful for exploring collective perspectives, attitudes, behaviours and experiences. Consequently, they can yield rich, in-depth data and illuminate agreement and inconsistencies 28 within and, where appropriate, between groups. Examples include public perceptions of dental implants and subsequent impact on help-seeking and decision making, 30 and general dental practitioners' views on patient safety in dentistry. 31

Focus groups can be used alone or in conjunction with other methods, such as interviews or observations, and can therefore help to confirm, extend or enrich understanding and provide alternative insights. 28 The social interaction between participants often results in lively discussion and can therefore facilitate the collection of rich, meaningful data. However, they are complex to organise and manage, due to the number of participants, and may also be inappropriate for exploring particularly sensitive issues that many participants may feel uncomfortable about discussing in a group environment.

Focus groups are primarily undertaken face-to-face but can now also be undertaken online, using appropriate technologies such as email, bulletin boards, online research communities, chat rooms, discussion forums, social media and video conferencing. 32 Using such technologies, data collection can also be synchronous (for example, online discussions in 'real time') or, unlike traditional face-to-face focus groups, asynchronous (for example, online/email discussions in 'non-real time'). While many of the fundamental principles of focus group research are the same, regardless of how they are conducted, a number of subtle nuances are associated with the online medium. 32 Some of which are discussed further in the following sections.

Focus group considerations

Some key considerations associated with face-to-face focus groups are: how many participants are required; should participants within each group know each other (or not) and how many focus groups are needed within a single study? These issues are much debated and there is no definitive answer. However, the number of focus groups required will largely depend on the topic area, the depth and breadth of data needed, the desired level of participation required 29 and the necessity (or not) for data saturation.

The optimum group size is around six to eight participants (excluding researchers) but can work effectively with between three and 14 participants. 3 If the group is too small, it may limit discussion, but if it is too large, it may become disorganised and difficult to manage. It is, however, prudent to over-recruit for a focus group by approximately two to three participants, to allow for potential non-attenders. For many researchers, particularly novice researchers, group size may also be informed by pragmatic considerations, such as the type of study, resources available and moderator experience. 28 Similar size and mix considerations exist for online focus groups. Typically, synchronous online focus groups will have around three to eight participants but, as the discussion does not happen simultaneously, asynchronous groups may have as many as 10–30 participants. 33

The topic area and potential group interaction should guide group composition considerations. Pre-existing groups, where participants know each other (for example, work colleagues) may be easier to recruit, have shared experiences and may enjoy a familiarity, which facilitates discussion and/or the ability to challenge each other courteously. 3 However, if there is a potential power imbalance within the group or if existing group norms and hierarchies may adversely affect the ability of participants to speak freely, then 'stranger groups' (that is, where participants do not already know each other) may be more appropriate. 34 , 35

Focus group management

Face-to-face focus groups should normally be conducted by two researchers; a moderator and an observer. 28 The moderator facilitates group discussion, while the observer typically monitors group dynamics, behaviours, non-verbal cues, seating arrangements and speaking order, which is essential for transcription and analysis. The same principles of informed consent, as discussed in the interview section, also apply to focus groups, regardless of medium. However, the consent process for online discussions will probably be managed somewhat differently. For example, while an appropriate participant information leaflet (and consent form) would still be required, the process is likely to be managed electronically (for example, via email) and would need to specifically address issues relating to technology (for example, anonymity and use, storage and access to online data). 32

The venue in which a face to face focus group is conducted should be of a suitable size, private, quiet, free from distractions and in a collectively convenient location. It should also be conducted at a time appropriate for participants, 28 as this is likely to promote attendance. As with interviews, the same ethical considerations apply (as discussed earlier). However, online focus groups may present additional ethical challenges associated with issues such as informed consent, appropriate access and secure data storage. Further guidance can be found elsewhere. 8 , 32

Before the focus group commences, the researchers should establish rapport with participants, as this will help to put them at ease and result in a more meaningful discussion. Consequently, researchers should introduce themselves, provide further clarity about the study and how the process will work in practice and outline the 'ground rules'. Ground rules are designed to assist, not hinder, group discussion and typically include: 3 , 28 , 29

Discussions within the group are confidential to the group

Only one person can speak at a time

All participants should have sufficient opportunity to contribute

There should be no unnecessary interruptions while someone is speaking

Everyone can be expected to be listened to and their views respected

Challenging contrary opinions is appropriate, but ridiculing is not.

Moderating a focus group requires considered management and good interpersonal skills to help guide the discussion and, where appropriate, keep it sufficiently focused. Avoid, therefore, participating, leading, expressing personal opinions or correcting participants' knowledge 3 , 28 as this may bias the process. A relaxed, interested demeanour will also help participants to feel comfortable and promote candid discourse. Moderators should also prevent the discussion being dominated by any one person, ensure differences of opinions are discussed fairly and, if required, encourage reticent participants to contribute. 3 Asking open questions, reflecting on significant issues, inviting further debate, probing responses accordingly, and seeking further clarification, as and where appropriate, will help to obtain sufficient depth and insight into the topic area.

Moderating online focus groups requires comparable skills, particularly if the discussion is synchronous, as the discussion may be dominated by those who can type proficiently. 36 It is therefore important that sufficient time and respect is accorded to those who may not be able to type as quickly. Asynchronous discussions are usually less problematic in this respect, as interactions are less instant. However, moderating an asynchronous discussion presents additional challenges, particularly if participants are geographically dispersed, as they may be online at different times. Consequently, the moderator will not always be present and the discussion may therefore need to occur over several days, which can be difficult to manage and facilitate and invariably requires considerable flexibility. 32 It is also worth recognising that establishing rapport with participants via online medium is often more challenging than via face-to-face and may therefore require additional time, skills, effort and consideration.

As with research interviews, focus groups should be guided by an appropriate interview schedule, as discussed earlier in the paper. For example, the schedule will usually be informed by the review of the literature and study aims, and will merely provide a topic guide to help inform subsequent discussions. To provide a verbatim account of the discussion, focus groups must be recorded, using an audio-recorder with a good quality multi-directional microphone. While videotaping is possible, some participants may find it obtrusive, 3 which may adversely affect group dynamics. The use (or not) of a video recorder, should therefore be carefully considered.

At the end of the focus group, a few minutes should be spent rounding up and reflecting on the discussion. 28 Depending on the topic area, it is possible that some participants may have revealed deeply personal issues and may therefore require further help and support, such as a constructive debrief or possibly even referral on to a relevant third party. It is also possible that some participants may feel that the discussion did not adequately reflect their views and, consequently, may no longer wish to be associated with the study. 28 Such occurrences are likely to be uncommon, but should they arise, it is important to further discuss any concerns and, if appropriate, offer them the opportunity to withdraw (including any data relating to them) from the study. Immediately after the discussion, researchers should compile notes regarding thoughts and ideas about the focus group, which can assist with data analysis and, if appropriate, any further data collection.

Qualitative research is increasingly being utilised within dental research to explore the experiences, perspectives, motivations and beliefs of participants. The contributions of qualitative research to evidence-based practice are increasingly being recognised, both as standalone research and as part of larger mixed-method studies, including clinical trials. Interviews and focus groups remain commonly used data collection methods in qualitative research, and with the advent of digital technologies, their utilisation continues to evolve. However, digital methods of qualitative data collection present additional methodological, ethical and practical considerations, but also potentially offer considerable flexibility to participants and researchers. Consequently, regardless of format, qualitative methods have significant potential to inform important areas of dental practice, policy and further related research.

Gussy M, Dickson-Swift V, Adams J . A scoping review of qualitative research in peer-reviewed dental publications. Int J Dent Hygiene 2013; 11 : 174–179.

Article   Google Scholar  

Burnard P, Gill P, Stewart K, Treasure E, Chadwick B . Analysing and presenting qualitative data. Br Dent J 2008; 204 : 429–432.

Gill P, Stewart K, Treasure E, Chadwick B . Methods of data collection in qualitative research: interviews and focus groups. Br Dent J 2008; 204 : 291–295.

Gill P, Stewart K, Treasure E, Chadwick B . Conducting qualitative interviews with school children in dental research. Br Dent J 2008; 204 : 371–374.

Stewart K, Gill P, Chadwick B, Treasure E . Qualitative research in dentistry. Br Dent J 2008; 204 : 235–239.

Masood M, Thaliath E, Bower E, Newton J . An appraisal of the quality of published qualitative dental research. Community Dent Oral Epidemiol 2011; 39 : 193–203.

Ellis J, Levine A, Bedos C et al. Refusal of implant supported mandibular overdentures by elderly patients. Gerodontology 2011; 28 : 62–68.

Macfarlane S, Bucknall T . Digital Technologies in Research. In Gerrish K, Lathlean J (editors) The Research Process in Nursing . 7th edition. pp. 71–86. Oxford: Wiley Blackwell; 2015.

Google Scholar  

Lee R, Fielding N, Blank G . Online Research Methods in the Social Sciences: An Editorial Introduction. In Fielding N, Lee R, Blank G (editors) The Sage Handbook of Online Research Methods . pp. 3–16. London: Sage Publications; 2016.

Creswell J . Qualitative inquiry and research design: Choosing among five designs . Thousand Oaks, CA: Sage, 1998.

Guest G, Namey E, Mitchell M . Qualitative research: Defining and designing In Guest G, Namey E, Mitchell M (editors) Collecting Qualitative Data: A Field Manual For Applied Research . pp. 1–40. London: Sage Publications, 2013.

Chapter   Google Scholar  

Pope C, Mays N . Qualitative research: Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ 1995; 311 : 42–45.

Giddings L, Grant B . A Trojan Horse for positivism? A critique of mixed methods research. Adv Nurs Sci 2007; 30 : 52–60.

Hammersley M, Atkinson P . Ethnography: Principles in Practice . London: Routledge, 1995.

Oltmann S . Qualitative interviews: A methodological discussion of the interviewer and respondent contexts Forum Qualitative Sozialforschung/Forum: Qualitative Social Research. 2016; 17 : Art. 15.

Patton M . Qualitative Research and Evaluation Methods . Thousand Oaks, CA: Sage, 2002.

Wang M, Vinall-Collier K, Csikar J, Douglas G . A qualitative study of patients' views of techniques to reduce dental anxiety. J Dent 2017; 66 : 45–51.

Lindenmeyer A, Bowyer V, Roscoe J, Dale J, Sutcliffe P . Oral health awareness and care preferences in patients with diabetes: a qualitative study. Fam Pract 2013; 30 : 113–118.

Gallagher J, Clarke W, Wilson N . Understanding the motivation: a qualitative study of dental students' choice of professional career. Eur J Dent Educ 2008; 12 : 89–98.

Tod A . Interviewing. In Gerrish K, Lacey A (editors) The Research Process in Nursing . Oxford: Blackwell Publishing, 2006.

Grey E, Harcourt D, O'Sullivan D, Buchanan H, Kipatrick N . A qualitative study of patients' motivations and expectations for dental implants. Br Dent J 2013; 214 : 10.1038/sj.bdj.2012.1178.

Farmer J, Peressini S, Lawrence H . Exploring the role of the dental hygienist in reducing oral health disparities in Canada: A qualitative study. Int J Dent Hygiene 2017; 10.1111/idh.12276.

McElhinney E, Cheater F, Kidd L . Undertaking qualitative health research in social virtual worlds. J Adv Nurs 2013; 70 : 1267–1275.

Health Research Authority. UK Policy Framework for Health and Social Care Research. Available at https://www.hra.nhs.uk/planning-and-improving-research/policies-standards-legislation/uk-policy-framework-health-social-care-research/ (accessed September 2017).

Baillie J, Gill P, Courtenay P . Knowledge, understanding and experiences of peritonitis among patients, and their families, undertaking peritoneal dialysis: A mixed methods study protocol. J Adv Nurs 2017; 10.1111/jan.13400.

Kvale S . Interviews . Thousand Oaks (CA): Sage, 1996.

Spradley J . The Ethnographic Interview . New York: Holt, Rinehart and Winston, 1979.

Goodman C, Evans C . Focus Groups. In Gerrish K, Lathlean J (editors) The Research Process in Nursing . pp. 401–412. Oxford: Wiley Blackwell, 2015.

Shaha M, Wenzell J, Hill E . Planning and conducting focus group research with nurses. Nurse Res 2011; 18 : 77–87.

Wang G, Gao X, Edward C . Public perception of dental implants: a qualitative study. J Dent 2015; 43 : 798–805.

Bailey E . Contemporary views of dental practitioners' on patient safety. Br Dent J 2015; 219 : 535–540.

Abrams K, Gaiser T . Online Focus Groups. In Field N, Lee R, Blank G (editors) The Sage Handbook of Online Research Methods . pp. 435–450. London: Sage Publications, 2016.

Poynter R . The Handbook of Online and Social Media Research . West Sussex: John Wiley & Sons, 2010.

Kevern J, Webb C . Focus groups as a tool for critical social research in nurse education. Nurse Educ Today 2001; 21 : 323–333.

Kitzinger J, Barbour R . Introduction: The Challenge and Promise of Focus Groups. In Barbour R S K J (editor) Developing Focus Group Research . pp. 1–20. London: Sage Publications, 1999.

Krueger R, Casey M . Focus Groups: A Practical Guide for Applied Research. 4th ed. Thousand Oaks, California: SAGE; 2009.

Download references

Author information

Authors and affiliations.

Senior Lecturer (Adult Nursing), School of Healthcare Sciences, Cardiff University,

Lecturer (Adult Nursing) and RCBC Wales Postdoctoral Research Fellow, School of Healthcare Sciences, Cardiff University,

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to P. Gill .

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Gill, P., Baillie, J. Interviews and focus groups in qualitative research: an update for the digital age. Br Dent J 225 , 668–672 (2018). https://doi.org/10.1038/sj.bdj.2018.815

Download citation

Accepted : 02 July 2018

Published : 05 October 2018

Issue Date : 12 October 2018

DOI : https://doi.org/10.1038/sj.bdj.2018.815

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Assessment of women’s needs and wishes regarding interprofessional guidance on oral health in pregnancy – a qualitative study.

• Merle Ebinghaus
• Caroline Johanna Agricola
• Birgit-Christiane Zyriax

BMC Pregnancy and Childbirth (2024)

Translating brand reputation into equity from the stakeholder’s theory: an approach to value creation based on consumer’s perception & interactions

• Olukorede Adewole

International Journal of Corporate Social Responsibility (2024)

Perceptions and beliefs of community gatekeepers about genomic risk information in African cleft research

• Abimbola M. Oladayo
• Oluwakemi Odukoya
• Azeez Butali

BMC Public Health (2024)

Assessment of women’s needs, wishes and preferences regarding interprofessional guidance on nutrition in pregnancy – a qualitative study

‘baby mamas’ in urban ghana: an exploratory qualitative study on the factors influencing serial fathering among men in accra, ghana.

• Rosemond Akpene Hiadzi
• Jemima Akweley Agyeman
• Godwin Banafo Akrong

Reproductive Health (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed, qualitative study, affiliations.

• 1 University of Nebraska Medical Center
• 2 GDB Research and Statistical Consulting
• 3 GDB Research and Statistical Consulting/McLaren Macomb Hospital
• PMID: 29262162
• Bookshelf ID: NBK470395

Qualitative research is a type of research that explores and provides deeper insights into real-world problems. Instead of collecting numerical data points or intervening or introducing treatments just like in quantitative research, qualitative research helps generate hypothenar to further investigate and understand quantitative data. Qualitative research gathers participants' experiences, perceptions, and behavior. It answers the hows and whys instead of how many or how much. It could be structured as a standalone study, purely relying on qualitative data, or part of mixed-methods research that combines qualitative and quantitative data. This review introduces the readers to some basic concepts, definitions, terminology, and applications of qualitative research.

Qualitative research, at its core, asks open-ended questions whose answers are not easily put into numbers, such as "how" and "why." Due to the open-ended nature of the research questions, qualitative research design is often not linear like quantitative design. One of the strengths of qualitative research is its ability to explain processes and patterns of human behavior that can be difficult to quantify. Phenomena such as experiences, attitudes, and behaviors can be complex to capture accurately and quantitatively. In contrast, a qualitative approach allows participants themselves to explain how, why, or what they were thinking, feeling, and experiencing at a particular time or during an event of interest. Quantifying qualitative data certainly is possible, but at its core, qualitative data is looking for themes and patterns that can be difficult to quantify, and it is essential to ensure that the context and narrative of qualitative work are not lost by trying to quantify something that is not meant to be quantified.

However, while qualitative research is sometimes placed in opposition to quantitative research, where they are necessarily opposites and therefore "compete" against each other and the philosophical paradigms associated with each other, qualitative and quantitative work are neither necessarily opposites, nor are they incompatible. While qualitative and quantitative approaches are different, they are not necessarily opposites and certainly not mutually exclusive. For instance, qualitative research can help expand and deepen understanding of data or results obtained from quantitative analysis. For example, say a quantitative analysis has determined a correlation between length of stay and level of patient satisfaction, but why does this correlation exist? This dual-focus scenario shows one way in which qualitative and quantitative research could be integrated.

Copyright © 2024, StatPearls Publishing LLC.

PubMed Disclaimer

Conflict of interest statement

Disclosure: Steven Tenny declares no relevant financial relationships with ineligible companies.

Disclosure: Janelle Brannan declares no relevant financial relationships with ineligible companies.

Disclosure: Grace Brannan declares no relevant financial relationships with ineligible companies.

• Introduction
• Issues of Concern
• Clinical Significance
• Enhancing Healthcare Team Outcomes
• Review Questions

Similar articles

• Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
• Macromolecular crowding: chemistry and physics meet biology (Ascona, Switzerland, 10-14 June 2012). Foffi G, Pastore A, Piazza F, Temussi PA. Foffi G, et al. Phys Biol. 2013 Aug;10(4):040301. doi: 10.1088/1478-3975/10/4/040301. Epub 2013 Aug 2. Phys Biol. 2013. PMID: 23912807
• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Invited review: Qualitative research in dairy science-A narrative review. Ritter C, Koralesky KE, Saraceni J, Roche S, Vaarst M, Kelton D. Ritter C, et al. J Dairy Sci. 2023 Sep;106(9):5880-5895. doi: 10.3168/jds.2022-23125. Epub 2023 Jul 18. J Dairy Sci. 2023. PMID: 37474366 Review.
• Participation in environmental enhancement and conservation activities for health and well-being in adults: a review of quantitative and qualitative evidence. Husk K, Lovell R, Cooper C, Stahl-Timmins W, Garside R. Husk K, et al. Cochrane Database Syst Rev. 2016 May 21;2016(5):CD010351. doi: 10.1002/14651858.CD010351.pub2. Cochrane Database Syst Rev. 2016. PMID: 27207731 Free PMC article. Review.
• Moser A, Korstjens I. Series: Practical guidance to qualitative research. Part 1: Introduction. Eur J Gen Pract. 2017 Dec;23(1):271-273. - PMC - PubMed
• Cleland JA. The qualitative orientation in medical education research. Korean J Med Educ. 2017 Jun;29(2):61-71. - PMC - PubMed
• Foley G, Timonen V. Using Grounded Theory Method to Capture and Analyze Health Care Experiences. Health Serv Res. 2015 Aug;50(4):1195-210. - PMC - PubMed
• Devers KJ. How will we know "good" qualitative research when we see it? Beginning the dialogue in health services research. Health Serv Res. 1999 Dec;34(5 Pt 2):1153-88. - PMC - PubMed
• Huston P, Rowan M. Qualitative studies. Their role in medical research. Can Fam Physician. 1998 Nov;44:2453-8. - PMC - PubMed

Publication types

• Search in PubMed
• Search in MeSH
• Add to Search

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• NCBI Bookshelf

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• What Is Qualitative Research? | Methods & Examples

What Is Qualitative Research? | Methods & Examples

Published on June 19, 2020 by Pritha Bhandari . Revised on June 22, 2023.

Qualitative research involves collecting and analyzing non-numerical data (e.g., text, video, or audio) to understand concepts, opinions, or experiences. It can be used to gather in-depth insights into a problem or generate new ideas for research.

Qualitative research is the opposite of quantitative research , which involves collecting and analyzing numerical data for statistical analysis.

Qualitative research is commonly used in the humanities and social sciences, in subjects such as anthropology, sociology, education, health sciences, history, etc.

• How does social media shape body image in teenagers?
• How do children and adults interpret healthy eating in the UK?
• What factors influence employee retention in a large organization?
• How is anxiety experienced around the world?
• How can teachers integrate social issues into science curriculums?

Table of contents

Approaches to qualitative research, qualitative research methods, qualitative data analysis, advantages of qualitative research, disadvantages of qualitative research, other interesting articles, frequently asked questions about qualitative research.

Qualitative research is used to understand how people experience the world. While there are many approaches to qualitative research, they tend to be flexible and focus on retaining rich meaning when interpreting data.

Common approaches include grounded theory, ethnography , action research , phenomenological research, and narrative research. They share some similarities, but emphasize different aims and perspectives.

Note that qualitative research is at risk for certain research biases including the Hawthorne effect , observer bias , recall bias , and social desirability bias . While not always totally avoidable, awareness of potential biases as you collect and analyze your data can prevent them from impacting your work too much.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Each of the research approaches involve using one or more data collection methods . These are some of the most common qualitative methods:

• Observations: recording what you have seen, heard, or encountered in detailed field notes.
• Interviews:  personally asking people questions in one-on-one conversations.
• Focus groups: asking questions and generating discussion among a group of people.
• Surveys : distributing questionnaires with open-ended questions.
• Secondary research: collecting existing data in the form of texts, images, audio or video recordings, etc.
• You take field notes with observations and reflect on your own experiences of the company culture.
• You distribute open-ended surveys to employees across all the company’s offices by email to find out if the culture varies across locations.
• You conduct in-depth interviews with employees in your office to learn about their experiences and perspectives in greater detail.

Qualitative researchers often consider themselves “instruments” in research because all observations, interpretations and analyses are filtered through their own personal lens.

For this reason, when writing up your methodology for qualitative research, it’s important to reflect on your approach and to thoroughly explain the choices you made in collecting and analyzing the data.

Qualitative data can take the form of texts, photos, videos and audio. For example, you might be working with interview transcripts, survey responses, fieldnotes, or recordings from natural settings.

Most types of qualitative data analysis share the same five steps:

• Prepare and organize your data. This may mean transcribing interviews or typing up fieldnotes.
• Review and explore your data. Examine the data for patterns or repeated ideas that emerge.
• Develop a data coding system. Based on your initial ideas, establish a set of codes that you can apply to categorize your data.
• Assign codes to the data. For example, in qualitative survey analysis, this may mean going through each participant’s responses and tagging them with codes in a spreadsheet. As you go through your data, you can create new codes to add to your system if necessary.
• Identify recurring themes. Link codes together into cohesive, overarching themes.

There are several specific approaches to analyzing qualitative data. Although these methods share similar processes, they emphasize different concepts.

Qualitative research often tries to preserve the voice and perspective of participants and can be adjusted as new research questions arise. Qualitative research is good for:

• Flexibility

The data collection and analysis process can be adapted as new ideas or patterns emerge. They are not rigidly decided beforehand.

• Natural settings

Data collection occurs in real-world contexts or in naturalistic ways.

• Meaningful insights

Detailed descriptions of people’s experiences, feelings and perceptions can be used in designing, testing or improving systems or products.

• Generation of new ideas

Open-ended responses mean that researchers can uncover novel problems or opportunities that they wouldn’t have thought of otherwise.

Prevent plagiarism. Run a free check.

Researchers must consider practical and theoretical limitations in analyzing and interpreting their data. Qualitative research suffers from:

• Unreliability

The real-world setting often makes qualitative research unreliable because of uncontrolled factors that affect the data.

• Subjectivity

Due to the researcher’s primary role in analyzing and interpreting data, qualitative research cannot be replicated . The researcher decides what is important and what is irrelevant in data analysis, so interpretations of the same data can vary greatly.

• Limited generalizability

Small samples are often used to gather detailed data about specific contexts. Despite rigorous analysis procedures, it is difficult to draw generalizable conclusions because the data may be biased and unrepresentative of the wider population .

• Labor-intensive

Although software can be used to manage and record large amounts of text, data analysis often has to be checked or performed manually.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Chi square goodness of fit test
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

There are five common approaches to qualitative research :

• Grounded theory involves collecting data in order to develop new theories.
• Ethnography involves immersing yourself in a group or organization to understand its culture.
• Narrative research involves interpreting stories to understand how people make sense of their experiences and perceptions.
• Phenomenological research involves investigating phenomena through people’s lived experiences.
• Action research links theory and practice in several cycles to drive innovative changes.

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

There are various approaches to qualitative data analysis , but they all share five steps in common:

• Prepare and organize your data.
• Review and explore your data.
• Develop a data coding system.
• Assign codes to the data.
• Identify recurring themes.

The specifics of each step depend on the focus of the analysis. Some common approaches include textual analysis , thematic analysis , and discourse analysis .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2023, June 22). What Is Qualitative Research? | Methods & Examples. Scribbr. Retrieved August 12, 2024, from https://www.scribbr.com/methodology/qualitative-research/

Is this article helpful?

Pritha Bhandari

Other students also liked, qualitative vs. quantitative research | differences, examples & methods, how to do thematic analysis | step-by-step guide & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

• Qualitative studies
• Research design
• Get an email alert for Qualitative studies
• Get the RSS feed for Qualitative studies

Showing 1 - 13 of 1,198

View by: Cover Page List Articles

Sort by: Recent Popular

Feasibility of a self-management intervention to improve mobility in the community after stroke (SIMS): A mixed-methods pilot study

Ahmad Sahely, Carron Sintler, Andrew Soundy, Sheeba Rosewilliam

Experiences of smoking and tobacco use during pregnancy: A qualitative study protocol

Maria Agràs-Guàrdia, Sara Martínez-Torres,  [ ... ], Francisco Martín-Luján

Translation and interpreting teachers’ perceptions of dilemma and needs in their professional development

Yi Liu, Jianyu Liu

Patients’ experiences with musculoskeletal spinal pain: A qualitative systematic review protocol

Alaa El Chamaa, Katie Kowalski, Pulak Parikh, Alison Rushton

Brace compliance process in adolescents with spinal deformities: A qualitative study

Faezeh Ghorbani, Mohammad Kamali,  [ ... ], Taher Babaee

Integrated maternity care: A concept analysis

Evelien Cellissen, Ruben van Zelm,  [ ... ], Marianne Nieuwenhuijze

Professionals’ experiences of what affects health outcomes in the sick leave and rehabilitation process—A qualitative study from primary care level

Märit Löfgren, Karin Törnbom,  [ ... ], Dominique Hange

Pharmacists’ perception about efficacy, safety, and quality of dietary supplements that used for rheumatic disorders in the Iraqi pharmaceutical market

Ali Haidar Al-Hadi, Ehab Mudher Mikhael

Development of the ECLIPSE model of meaningful outcome domains following lower limb amputation and prosthetic rehabilitation, through systematic review and best fit framework synthesis

Chantel Ostler, Alex Dickinson, Cheryl Metcalf, Maggie Donovan-Hall

Safety climate in the operating room in the pre-pandemic and pandemic period of COVID-19: A mixed method study

Rosilene Alves Ferreira, Cintia Silva Fassarella,  [ ... ], Ricardo de Oliveira Meneses

Humor in parenting: Does it have a role?

Lucy Emery, Anne Libera, Erik Lehman, Benjamin H. Levi

Predictors, barriers, and facilitators to refugee women’s employment and economic inclusion: A mixed methods systematic review

Areej Al-Hamad, Yasin M. Yasin, Kateryna Metersky

Patient perceptions of the impact of inducible laryngeal obstruction on quality of life

Katherine M. McConville, Susan L. Thibeault

Connect with Us

• PLOS ONE on Twitter
• PLOS on Facebook

Sample articles

Qualitative psychology ®.

• Recommendations for Designing and Reviewing Qualitative Research in Psychology: Promoting Methodological Integrity (PDF, 166KB) February 2017 by Heidi M. Levitt, Sue L. Motulsky, Fredrick J. Wertz, Susan L. Morrow, and Joseph G. Ponterotto
• Racial Microaggression Experiences and Coping Strategies of Black Women in Corporate Leadership (PDF, 132KB) August 2015 by Aisha M. B. Holder, Margo A. Jackson, and Joseph G. Ponterotto
• The Lived Experience of Homeless Youth: A Narrative Approach (PDF, 109KB) February 2015 by Erin E. Toolis and Phillip L. Hammack
• Lifetime Activism, Marginality, and Psychology: Narratives of Lifelong Feminist Activists Committed to Social Change (PDF, 93KB) August 2014 by Anjali Dutt and Shelly Grabe
• Qualitative Inquiry in the History of Psychology (PDF, 82KB) February 2014 by Frederick J. Wertz

More About this Journal

• Qualitative Psychology
• Pricing and subscription info
• Special section proposal guidelines
• Special sections
• Call for articles: Methodological Retrospectives

Contact Journals

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Prev Med

Qualitative Methods in Health Care Research

Vishnu renjith.

School of Nursing and Midwifery, Royal College of Surgeons Ireland - Bahrain (RCSI Bahrain), Al Sayh Muharraq Governorate, Bahrain

Renjulal Yesodharan

1 Department of Mental Health Nursing, Manipal College of Nursing Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, India

Judith A. Noronha

2 Department of OBG Nursing, Manipal College of Nursing Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, India

Elissa Ladd

3 School of Nursing, MGH Institute of Health Professions, Boston, USA

Anice George

4 Department of Child Health Nursing, Manipal College of Nursing Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, India

Healthcare research is a systematic inquiry intended to generate robust evidence about important issues in the fields of medicine and healthcare. Qualitative research has ample possibilities within the arena of healthcare research. This article aims to inform healthcare professionals regarding qualitative research, its significance, and applicability in the field of healthcare. A wide variety of phenomena that cannot be explained using the quantitative approach can be explored and conveyed using a qualitative method. The major types of qualitative research designs are narrative research, phenomenological research, grounded theory research, ethnographic research, historical research, and case study research. The greatest strength of the qualitative research approach lies in the richness and depth of the healthcare exploration and description it makes. In health research, these methods are considered as the most humanistic and person-centered way of discovering and uncovering thoughts and actions of human beings.

Introduction

Healthcare research is a systematic inquiry intended to generate trustworthy evidence about issues in the field of medicine and healthcare. The three principal approaches to health research are the quantitative, the qualitative, and the mixed methods approach. The quantitative research method uses data, which are measures of values and counts and are often described using statistical methods which in turn aids the researcher to draw inferences. Qualitative research incorporates the recording, interpreting, and analyzing of non-numeric data with an attempt to uncover the deeper meanings of human experiences and behaviors. Mixed methods research, the third methodological approach, involves collection and analysis of both qualitative and quantitative information with an objective to solve different but related questions, or at times the same questions.[ 1 , 2 ]

In healthcare, qualitative research is widely used to understand patterns of health behaviors, describe lived experiences, develop behavioral theories, explore healthcare needs, and design interventions.[ 1 , 2 , 3 ] Because of its ample applications in healthcare, there has been a tremendous increase in the number of health research studies undertaken using qualitative methodology.[ 4 , 5 ] This article discusses qualitative research methods, their significance, and applicability in the arena of healthcare.

Qualitative Research

Diverse academic and non-academic disciplines utilize qualitative research as a method of inquiry to understand human behavior and experiences.[ 6 , 7 ] According to Munhall, “Qualitative research involves broadly stated questions about human experiences and realities, studied through sustained contact with the individual in their natural environments and producing rich, descriptive data that will help us to understand those individual's experiences.”[ 8 ]

Significance of Qualitative Research

The qualitative method of inquiry examines the 'how' and 'why' of decision making, rather than the 'when,' 'what,' and 'where.'[ 7 ] Unlike quantitative methods, the objective of qualitative inquiry is to explore, narrate, and explain the phenomena and make sense of the complex reality. Health interventions, explanatory health models, and medical-social theories could be developed as an outcome of qualitative research.[ 9 ] Understanding the richness and complexity of human behavior is the crux of qualitative research.

Differences between Quantitative and Qualitative Research

The quantitative and qualitative forms of inquiry vary based on their underlying objectives. They are in no way opposed to each other; instead, these two methods are like two sides of a coin. The critical differences between quantitative and qualitative research are summarized in Table 1 .[ 1 , 10 , 11 ]

Differences between quantitative and qualitative research

Qualitative Research Questions and Purpose Statements

Qualitative questions are exploratory and are open-ended. A well-formulated study question forms the basis for developing a protocol, guides the selection of design, and data collection methods. Qualitative research questions generally involve two parts, a central question and related subquestions. The central question is directed towards the primary phenomenon under study, whereas the subquestions explore the subareas of focus. It is advised not to have more than five to seven subquestions. A commonly used framework for designing a qualitative research question is the 'PCO framework' wherein, P stands for the population under study, C stands for the context of exploration, and O stands for the outcome/s of interest.[ 12 ] The PCO framework guides researchers in crafting a focused study question.

Example: In the question, “What are the experiences of mothers on parenting children with Thalassemia?”, the population is “mothers of children with Thalassemia,” the context is “parenting children with Thalassemia,” and the outcome of interest is “experiences.”

The purpose statement specifies the broad focus of the study, identifies the approach, and provides direction for the overall goal of the study. The major components of a purpose statement include the central phenomenon under investigation, the study design and the population of interest. Qualitative research does not require a-priori hypothesis.[ 13 , 14 , 15 ]

Example: Borimnejad et al . undertook a qualitative research on the lived experiences of women suffering from vitiligo. The purpose of this study was, “to explore lived experiences of women suffering from vitiligo using a hermeneutic phenomenological approach.” [ 16 ]

Review of the Literature

In quantitative research, the researchers do an extensive review of scientific literature prior to the commencement of the study. However, in qualitative research, only a minimal literature search is conducted at the beginning of the study. This is to ensure that the researcher is not influenced by the existing understanding of the phenomenon under the study. The minimal literature review will help the researchers to avoid the conceptual pollution of the phenomenon being studied. Nonetheless, an extensive review of the literature is conducted after data collection and analysis.[ 15 ]

Reflexivity

Reflexivity refers to critical self-appraisal about one's own biases, values, preferences, and preconceptions about the phenomenon under investigation. Maintaining a reflexive diary/journal is a widely recognized way to foster reflexivity. According to Creswell, “Reflexivity increases the credibility of the study by enhancing more neutral interpretations.”[ 7 ]

Types of Qualitative Research Designs

The qualitative research approach encompasses a wide array of research designs. The words such as types, traditions, designs, strategies of inquiry, varieties, and methods are used interchangeably. The major types of qualitative research designs are narrative research, phenomenological research, grounded theory research, ethnographic research, historical research, and case study research.[ 1 , 7 , 10 ]

Narrative research

Narrative research focuses on exploring the life of an individual and is ideally suited to tell the stories of individual experiences.[ 17 ] The purpose of narrative research is to utilize 'story telling' as a method in communicating an individual's experience to a larger audience.[ 18 ] The roots of narrative inquiry extend to humanities including anthropology, literature, psychology, education, history, and sociology. Narrative research encompasses the study of individual experiences and learning the significance of those experiences. The data collection procedures include mainly interviews, field notes, letters, photographs, diaries, and documents collected from one or more individuals. Data analysis involves the analysis of the stories or experiences through “re-storying of stories” and developing themes usually in chronological order of events. Rolls and Payne argued that narrative research is a valuable approach in health care research, to gain deeper insight into patient's experiences.[ 19 ]

Example: Karlsson et al . undertook a narrative inquiry to “explore how people with Alzheimer's disease present their life story.” Data were collected from nine participants. They were asked to describe about their life experiences from childhood to adulthood, then to current life and their views about the future life. [ 20 ]

Phenomenological research

Phenomenology is a philosophical tradition developed by German philosopher Edmond Husserl. His student Martin Heidegger did further developments in this methodology. It defines the 'essence' of individual's experiences regarding a certain phenomenon.[ 1 ] The methodology has its origin from philosophy, psychology, and education. The purpose of qualitative research is to understand the people's everyday life experiences and reduce it into the central meaning or the 'essence of the experience'.[ 21 , 22 ] The unit of analysis of phenomenology is the individuals who have had similar experiences of the phenomenon. Interviews with individuals are mainly considered for the data collection, though, documents and observations are also useful. Data analysis includes identification of significant meaning elements, textural description (what was experienced), structural description (how was it experienced), and description of 'essence' of experience.[ 1 , 7 , 21 ] The phenomenological approach is further divided into descriptive and interpretive phenomenology. Descriptive phenomenology focuses on the understanding of the essence of experiences and is best suited in situations that need to describe the lived phenomenon. Hermeneutic phenomenology or Interpretive phenomenology moves beyond the description to uncover the meanings that are not explicitly evident. The researcher tries to interpret the phenomenon, based on their judgment rather than just describing it.[ 7 , 21 , 22 , 23 , 24 ]

Example: A phenomenological study conducted by Cornelio et al . aimed at describing the lived experiences of mothers in parenting children with leukemia. Data from ten mothers were collected using in-depth semi-structured interviews and were analyzed using Husserl's method of phenomenology. Themes such as “pivotal moment in life”, “the experience of being with a seriously ill child”, “having to keep distance with the relatives”, “overcoming the financial and social commitments”, “responding to challenges”, “experience of faith as being key to survival”, “health concerns of the present and future”, and “optimism” were derived. The researchers reported the essence of the study as “chronic illness such as leukemia in children results in a negative impact on the child and on the mother.” [ 25 ]

Grounded Theory Research

Grounded theory has its base in sociology and propagated by two sociologists, Barney Glaser, and Anselm Strauss.[ 26 ] The primary purpose of grounded theory is to discover or generate theory in the context of the social process being studied. The major difference between grounded theory and other approaches lies in its emphasis on theory generation and development. The name grounded theory comes from its ability to induce a theory grounded in the reality of study participants.[ 7 , 27 ] Data collection in grounded theory research involves recording interviews from many individuals until data saturation. Constant comparative analysis, theoretical sampling, theoretical coding, and theoretical saturation are unique features of grounded theory research.[ 26 , 27 , 28 ] Data analysis includes analyzing data through 'open coding,' 'axial coding,' and 'selective coding.'[ 1 , 7 ] Open coding is the first level of abstraction, and it refers to the creation of a broad initial range of categories, axial coding is the procedure of understanding connections between the open codes, whereas selective coding relates to the process of connecting the axial codes to formulate a theory.[ 1 , 7 ] Results of the grounded theory analysis are supplemented with a visual representation of major constructs usually in the form of flow charts or framework diagrams. Quotations from the participants are used in a supportive capacity to substantiate the findings. Strauss and Corbin highlights that “the value of the grounded theory lies not only in its ability to generate a theory but also to ground that theory in the data.”[ 27 ]

Example: Williams et al . conducted a grounded theory research to explore the nature of relationship between the sense of self and the eating disorders. Data were collected form 11 women with a lifetime history of Anorexia Nervosa and were analyzed using the grounded theory methodology. Analysis led to the development of a theoretical framework on the nature of the relationship between the self and Anorexia Nervosa. [ 29 ]

Ethnographic research

Ethnography has its base in anthropology, where the anthropologists used it for understanding the culture-specific knowledge and behaviors. In health sciences research, ethnography focuses on narrating and interpreting the health behaviors of a culture-sharing group. 'Culture-sharing group' in an ethnography represents any 'group of people who share common meanings, customs or experiences.' In health research, it could be a group of physicians working in rural care, a group of medical students, or it could be a group of patients who receive home-based rehabilitation. To understand the cultural patterns, researchers primarily observe the individuals or group of individuals for a prolonged period of time.[ 1 , 7 , 30 ] The scope of ethnography can be broad or narrow depending on the aim. The study of more general cultural groups is termed as macro-ethnography, whereas micro-ethnography focuses on more narrowly defined cultures. Ethnography is usually conducted in a single setting. Ethnographers collect data using a variety of methods such as observation, interviews, audio-video records, and document reviews. A written report includes a detailed description of the culture sharing group with emic and etic perspectives. When the researcher reports the views of the participants it is called emic perspectives and when the researcher reports his or her views about the culture, the term is called etic.[ 7 ]

Example: The aim of the ethnographic study by LeBaron et al . was to explore the barriers to opioid availability and cancer pain management in India. The researchers collected data from fifty-nine participants using in-depth semi-structured interviews, participant observation, and document review. The researchers identified significant barriers by open coding and thematic analysis of the formal interview. [ 31 ]

Historical research

Historical research is the “systematic collection, critical evaluation, and interpretation of historical evidence”.[ 1 ] The purpose of historical research is to gain insights from the past and involves interpreting past events in the light of the present. The data for historical research are usually collected from primary and secondary sources. The primary source mainly includes diaries, first hand information, and writings. The secondary sources are textbooks, newspapers, second or third-hand accounts of historical events and medical/legal documents. The data gathered from these various sources are synthesized and reported as biographical narratives or developmental perspectives in chronological order. The ideas are interpreted in terms of the historical context and significance. The written report describes 'what happened', 'how it happened', 'why it happened', and its significance and implications to current clinical practice.[ 1 , 10 ]

Example: Lubold (2019) analyzed the breastfeeding trends in three countries (Sweden, Ireland, and the United States) using a historical qualitative method. Through analysis of historical data, the researcher found that strong family policies, adherence to international recommendations and adoption of baby-friendly hospital initiative could greatly enhance the breastfeeding rates. [ 32 ]

Case study research

Case study research focuses on the description and in-depth analysis of the case(s) or issues illustrated by the case(s). The design has its origin from psychology, law, and medicine. Case studies are best suited for the understanding of case(s), thus reducing the unit of analysis into studying an event, a program, an activity or an illness. Observations, one to one interviews, artifacts, and documents are used for collecting the data, and the analysis is done through the description of the case. From this, themes and cross-case themes are derived. A written case study report includes a detailed description of one or more cases.[ 7 , 10 ]

Example: Perceptions of poststroke sexuality in a woman of childbearing age was explored using a qualitative case study approach by Beal and Millenbrunch. Semi structured interview was conducted with a 36- year mother of two children with a history of Acute ischemic stroke. The data were analyzed using an inductive approach. The authors concluded that “stroke during childbearing years may affect a woman's perception of herself as a sexual being and her ability to carry out gender roles”. [ 33 ]

Sampling in Qualitative Research

Qualitative researchers widely use non-probability sampling techniques such as purposive sampling, convenience sampling, quota sampling, snowball sampling, homogeneous sampling, maximum variation sampling, extreme (deviant) case sampling, typical case sampling, and intensity sampling. The selection of a sampling technique depends on the nature and needs of the study.[ 34 , 35 , 36 , 37 , 38 , 39 , 40 ] The four widely used sampling techniques are convenience sampling, purposive sampling, snowball sampling, and intensity sampling.

Convenience sampling

It is otherwise called accidental sampling, where the researchers collect data from the subjects who are selected based on accessibility, geographical proximity, ease, speed, and or low cost.[ 34 ] Convenience sampling offers a significant benefit of convenience but often accompanies the issues of sample representation.

Purposive sampling

Purposive or purposeful sampling is a widely used sampling technique.[ 35 ] It involves identifying a population based on already established sampling criteria and then selecting subjects who fulfill that criteria to increase the credibility. However, choosing information-rich cases is the key to determine the power and logic of purposive sampling in a qualitative study.[ 1 ]

Snowball sampling

The method is also known as 'chain referral sampling' or 'network sampling.' The sampling starts by having a few initial participants, and the researcher relies on these early participants to identify additional study participants. It is best adopted when the researcher wishes to study the stigmatized group, or in cases, where findings of participants are likely to be difficult by ordinary means. Respondent ridden sampling is an improvised version of snowball sampling used to find out the participant from a hard-to-find or hard-to-study population.[ 37 , 38 ]

Intensity sampling

The process of identifying information-rich cases that manifest the phenomenon of interest is referred to as intensity sampling. It requires prior information, and considerable judgment about the phenomenon of interest and the researcher should do some preliminary investigations to determine the nature of the variation. Intensity sampling will be done once the researcher identifies the variation across the cases (extreme, average and intense) and picks the intense cases from them.[ 40 ]

Deciding the Sample Size

A-priori sample size calculation is not undertaken in the case of qualitative research. Researchers collect the data from as many participants as possible until they reach the point of data saturation. Data saturation or the point of redundancy is the stage where the researcher no longer sees or hears any new information. Data saturation gives the idea that the researcher has captured all possible information about the phenomenon of interest. Since no further information is being uncovered as redundancy is achieved, at this point the data collection can be stopped. The objective here is to get an overall picture of the chronicle of the phenomenon under the study rather than generalization.[ 1 , 7 , 41 ]

Data Collection in Qualitative Research

The various strategies used for data collection in qualitative research includes in-depth interviews (individual or group), focus group discussions (FGDs), participant observation, narrative life history, document analysis, audio materials, videos or video footage, text analysis, and simple observation. Among all these, the three popular methods are the FGDs, one to one in-depth interviews and the participant observation.

FGDs are useful in eliciting data from a group of individuals. They are normally built around a specific topic and are considered as the best approach to gather data on an entire range of responses to a topic.[ 42 Group size in an FGD ranges from 6 to 12. Depending upon the nature of participants, FGDs could be homogeneous or heterogeneous.[ 1 , 14 ] One to one in-depth interviews are best suited to obtain individuals' life histories, lived experiences, perceptions, and views, particularly while exporting topics of sensitive nature. In-depth interviews can be structured, unstructured, or semi-structured. However, semi-structured interviews are widely used in qualitative research. Participant observations are suitable for gathering data regarding naturally occurring behaviors.[ 1 ]

Data Analysis in Qualitative Research

Various strategies are employed by researchers to analyze data in qualitative research. Data analytic strategies differ according to the type of inquiry. A general content analysis approach is described herewith. Data analysis begins by transcription of the interview data. The researcher carefully reads data and gets a sense of the whole. Once the researcher is familiarized with the data, the researcher strives to identify small meaning units called the 'codes.' The codes are then grouped based on their shared concepts to form the primary categories. Based on the relationship between the primary categories, they are then clustered into secondary categories. The next step involves the identification of themes and interpretation to make meaning out of data. In the results section of the manuscript, the researcher describes the key findings/themes that emerged. The themes can be supported by participants' quotes. The analytical framework used should be explained in sufficient detail, and the analytic framework must be well referenced. The study findings are usually represented in a schematic form for better conceptualization.[ 1 , 7 ] Even though the overall analytical process remains the same across different qualitative designs, each design such as phenomenology, ethnography, and grounded theory has design specific analytical procedures, the details of which are out of the scope of this article.

Computer-Assisted Qualitative Data Analysis Software (CAQDAS)

Until recently, qualitative analysis was done either manually or with the help of a spreadsheet application. Currently, there are various software programs available which aid researchers to manage qualitative data. CAQDAS is basically data management tools and cannot analyze the qualitative data as it lacks the ability to think, reflect, and conceptualize. Nonetheless, CAQDAS helps researchers to manage, shape, and make sense of unstructured information. Open Code, MAXQDA, NVivo, Atlas.ti, and Hyper Research are some of the widely used qualitative data analysis software.[ 14 , 43 ]

Reporting Guidelines

Consolidated Criteria for Reporting Qualitative Research (COREQ) is the widely used reporting guideline for qualitative research. This 32-item checklist assists researchers in reporting all the major aspects related to the study. The three major domains of COREQ are the 'research team and reflexivity', 'study design', and 'analysis and findings'.[ 44 , 45 ]

Critical Appraisal of Qualitative Research

Various scales are available to critical appraisal of qualitative research. The widely used one is the Critical Appraisal Skills Program (CASP) Qualitative Checklist developed by CASP network, UK. This 10-item checklist evaluates the quality of the study under areas such as aims, methodology, research design, ethical considerations, data collection, data analysis, and findings.[ 46 ]

Ethical Issues in Qualitative Research

A qualitative study must be undertaken by grounding it in the principles of bioethics such as beneficence, non-maleficence, autonomy, and justice. Protecting the participants is of utmost importance, and the greatest care has to be taken while collecting data from a vulnerable research population. The researcher must respect individuals, families, and communities and must make sure that the participants are not identifiable by their quotations that the researchers include when publishing the data. Consent for audio/video recordings must be obtained. Approval to be in FGDs must be obtained from the participants. Researchers must ensure the confidentiality and anonymity of the transcripts/audio-video records/photographs/other data collected as a part of the study. The researchers must confirm their role as advocates and proceed in the best interest of all participants.[ 42 , 47 , 48 ]

Rigor in Qualitative Research

The demonstration of rigor or quality in the conduct of the study is essential for every research method. However, the criteria used to evaluate the rigor of quantitative studies are not be appropriate for qualitative methods. Lincoln and Guba (1985) first outlined the criteria for evaluating the qualitative research often referred to as “standards of trustworthiness of qualitative research”.[ 49 ] The four components of the criteria are credibility, transferability, dependability, and confirmability.

Credibility refers to confidence in the 'truth value' of the data and its interpretation. It is used to establish that the findings are true, credible and believable. Credibility is similar to the internal validity in quantitative research.[ 1 , 50 , 51 ] The second criterion to establish the trustworthiness of the qualitative research is transferability, Transferability refers to the degree to which the qualitative results are applicability to other settings, population or contexts. This is analogous to the external validity in quantitative research.[ 1 , 50 , 51 ] Lincoln and Guba recommend authors provide enough details so that the users will be able to evaluate the applicability of data in other contexts.[ 49 ] The criterion of dependability refers to the assumption of repeatability or replicability of the study findings and is similar to that of reliability in quantitative research. The dependability question is 'Whether the study findings be repeated of the study is replicated with the same (similar) cohort of participants, data coders, and context?'[ 1 , 50 , 51 ] Confirmability, the fourth criteria is analogous to the objectivity of the study and refers the degree to which the study findings could be confirmed or corroborated by others. To ensure confirmability the data should directly reflect the participants' experiences and not the bias, motivations, or imaginations of the inquirer.[ 1 , 50 , 51 ] Qualitative researchers should ensure that the study is conducted with enough rigor and should report the measures undertaken to enhance the trustworthiness of the study.

Conclusions

Qualitative research studies are being widely acknowledged and recognized in health care practice. This overview illustrates various qualitative methods and shows how these methods can be used to generate evidence that informs clinical practice. Qualitative research helps to understand the patterns of health behaviors, describe illness experiences, design health interventions, and develop healthcare theories. The ultimate strength of the qualitative research approach lies in the richness of the data and the descriptions and depth of exploration it makes. Hence, qualitative methods are considered as the most humanistic and person-centered way of discovering and uncovering thoughts and actions of human beings.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

• Open access
• Published: 14 August 2024

Qualitative studies involving users of clinical neurotechnology: a scoping review

• Georg Starke 1 , 2 ,
• Tugba Basaran Akmazoglu 3 ,
• Annalisa Colucci 4 ,
• Mareike Vermehren 4 ,
• Amanda van Beinum 5 ,
• Maria Buthut 4 ,
• Surjo R. Soekadar 4 ,
• Christoph Bublitz 7 ,
• Jennifer A. Chandler 6 &
• Marcello Ienca 1 , 2  

BMC Medical Ethics volume  25 , Article number:  89 ( 2024 ) Cite this article

Metrics details

The rise of a new generation of intelligent neuroprostheses, brain-computer interfaces (BCI) and adaptive closed-loop brain stimulation devices hastens the clinical deployment of neurotechnologies to treat neurological and neuropsychiatric disorders. However, it remains unclear how these nascent technologies may impact the subjective experience of their users. To inform this debate, it is crucial to have a solid understanding how more established current technologies already affect their users. In recent years, researchers have used qualitative research methods to explore the subjective experience of individuals who become users of clinical neurotechnology. Yet, a synthesis of these more recent findings focusing on qualitative methods is still lacking.

To address this gap in the literature, we systematically searched five databases for original research articles that investigated subjective experiences of persons using or receiving neuroprosthetics, BCIs or neuromodulation with qualitative interviews and raised normative questions.

36 research articles were included and analysed using qualitative content analysis. Our findings synthesise the current scientific literature and reveal a pronounced focus on usability and other technical aspects of user experience. In parallel, they highlight a relative neglect of considerations regarding agency, self-perception, personal identity and subjective experience.

Conclusions

Our synthesis of the existing qualitative literature on clinical neurotechnology highlights the need to expand the current methodological focus as to investigate also non-technical aspects of user experience. Given the critical role considerations of agency, self-perception and personal identity play in assessing the ethical and legal significance of these technologies, our findings reveal a critical gap in the existing literature. This review provides a comprehensive synthesis of the current qualitative research landscape on neurotechnology and the limitations thereof. These findings can inform researchers on how to study the subjective experience of neurotechnology users more holistically and build patient-centred neurotechnology.

Peer Review reports

Introduction

Due to a rapid expansion in public-private investment, market size and availability of Artificial Intelligence (AI) tools for functional optimization, the clinical advancement of novel neurotechnologies is accelerating its pace [ 1 ]. Bidirectional intelligent Brain-Computer interfaces (BCI) that aim at merging both read-out and write-in devices are in active development and are expanding in functional capabilities and commercial availability. [ 2 , 3 ]. Such BCIs that can decode and modulate neural activity through direct stimulation of brain tissue, promise additional avenues in the treatment of neurological diseases by adapting to the particularities of individual users’ brain. Potential applications are Parkinson’s disease [ 4 ] or epilepsy [ 5 ] as well as psychiatric disorders, such as major depressive disorder [ 6 ] or obsessive compulsive disorder [ 7 ]. Driven by these advances and in conjunction with progress in deep learning and generative AI software as well as higher-bandwidth hardware, clinical neurotechnology is likely to take an increasingly central role in the prevention, diagnosis and treatment of neuropsychiatric disorders.

In line with these scientific trends, the last decade has seen a consequent fast rise in the ethical attention devoted to neurotechnological systems that establish a direct connection with the human central nervous system [ 8 ], including neurostimulation devices. Yet, at times, neuroethical concerns may have outpaced real-life possibilities, particularly with view to the impact of neurotechnology on personality, identity, autonomy, authenticity, agency or self (PIAAAS) [ 9 ]. This points to the need for basing ethical assessments and personal decisions about deploying devices on solid empirical grounds. In particular, it is crucial to gain a comprehensive understanding of the lived experience of using neurotechnologies from the epistemically privileged first-person perspective of users – “what it is like” to use neurotechnologies. Its examination by empirical studies have added a vital contribution to the literature [ 10 ].

Yet, few reviews have attempted to synthesize the growing body of empirical studies on user experience with clinical neurotechnology. Burwell et al. [ 11 ] reviewed literature from biomedical ethics on BCIs up to 2016, identifying key ethical, legal and societal challenges, yet noting a lack of concrete ethical recommendations for implementation. Worries about a lack of attention to ethics in BCI studies have been further corroborated by two reviews by Specker Sullivan and Illes, reviewing BCI research published up until 2015. They critically assessed the rationales of BCI research studies [ 12 ] and found a remarkable absence of ethical language in published BCI research [ 13 ]. Taking a different focus, Kögel et al. [ 14 ] have provided a scoping review summarizing empirical studies investigating ethics of BCIs until 2017, with a strong focus on quantitative methods in the reviewed papers. Most recently, this list of reviews has been complemented by van Velthoven et al. [ 15 ], who review empirical and conceptual ethical literature on the use of visual neuroprostheses.

To the best of our knowledge, a specific review of qualitative research on the ethics of emerging neurotechnologies such as neuroprosthetics, BCIs and neuromodulation systems is outstanding. We believe that qualitative research involving actual or prospective neurotechnology users is particularly significant as it allows researchers to tap into the richness of first-person experiences as compared to standardized questionnaires without the option of free report. In the following, we synthesize published research on the subjective experience of using clinical neurotechnologies to enrich the ethical debate and provide guidance to developers and regulators.

On January 13, 2022 we conducted a search of relevant scientific literature across 5 databases, namely Pubmed (89 results), Scopus (178 results), Web of Science (79 results), PsycInfo (134 results) and IEEE Xplore (4 results). The search was performed for title, abstract and keywords, using a search string to identify articles employing qualitative methods that engaged with users of neurotechnology, and covered normative issues: [“qualitative” OR “interview” OR “focus group” OR “ethnography” OR “grounded theory” OR “discourse analysis” OR “interpretative phenomenological analysis” OR “thematic analysis”] AND [“user” OR “patient” OR “people” OR “person” OR “participant” OR “subject”] AND [“Brain-Computer” OR “BCI” OR “Brain-Machine” OR “neurostimulation” OR “neuromodulation” OR “TMS” OR “transcranial” OR “neuroprosthetic*” OR “neuroprosthesis” OR “DBS”] AND [“ethic*” OR “bioethic*” OR “normative” OR “value” OR “evaluation”].

Across databases, search syntax was adapted to reflect the respective logic of each library. Our search yielded a total of 484 articles. Of these, 133 duplicates were removed. 52 further results were marked as ineligible by automation tools, due to either not being written in English or not representing original research in a peer-reviewed journal. The remaining 299 were screened manually, with screening tasks being shared equally among the authors GS, TBA, AC, MV, CB, JC, and MI. Articles were included if they were written in English, published in a peer-reviewed journal, and reported original research of empirical qualitative findings among human users of a neurotechnological system that establishes a direct connection with the human central nervous system (including neurostimulation devices). Other types of articles such as perspectives, letters to the editor, or review articles were not included. Potential methods included individual interviews, focus groups, stakeholder consultations but excluded studies that did not use any direct verbal input from the users. Each abstract was screened individually by two reviewers. Unclear cases were resolved by discussion among reviewers. This process resulted in the exclusion of 247 articles, leaving 52 publications for inclusion into the final synthesis.

Full texts of these 52 articles were retrieved and assessed for eligibility. Again, this task was shared equally across the 7 authors who made independent recommendations whether an article was included for further analysis, and disagreement was resolved by discussion. 20 articles were excluded at this stage, due to not meeting the inclusion criteria. This resulted in a body of 32 articles plus 4 additional papers identified through citation chaining, as customary in scoping reviews.

In the data analysis phase, we compiled a descriptive summary of the findings and conducted a thematic analysis. When compiling the descriptive summary, we followed the recommendations by Arksey and O’Malley [ 16 ] and included comprehensive information beyond authors, year, and title of the study, extracting also study location, methodology, study population, type of neurotechnology, and more. For the thematic analysis, the full text was read and coded by the authors through annotations in pdf files, with papers evenly distributed among the group. Coding was based on a previously agreed coding structure of four thematic families, covering (1) subjective experience with BCIs, (2) aspects concerning usability and technology, (3) ethical questions, (4) impact on social relations, and a fifth miscellaneous category for future resolution. In accordance with the suggestions by Braun and Clarke [ 17 ], codes that were not clearly covered by the coding tree were grouped into a category “miscellaneous”, and after discussion used to develop new themes or subsumed under the existing thematic families. The results were compiled and unified by the first author and imported into the Atlas.ti software (version 22.2), with adaptations to the coding tree being discussed between first and last author.

In line with the framework suggested by Pham, Rajić [ 18 ], we adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) in conducting and presenting our results [ 19 ]. A flow diagram representing the entire process is depicted in Fig.  1 .

PRISMA flow diagram: search and screening strategy. Based on Page et al [ 19 ]

Descriptive findings

Our study included 36 papers reporting original qualitative research among users of BCIs, neuroprosthetics and neuromodulation. We found a pronounced increase in the number of publications employing qualitative methods in the investigation of such neurotechnology users over time, with the earliest study dating back to 2012. However, contrary to what one may expect as reflection of the growing number of neurotechnology users, we did not find an increase in the average sample size of participants enrolled in qualitative studies nor a correlation between year of publication and number of participants (see Fig.  2 ).

Average number of participants and number of publications over time

The included studies were exclusively conducted in Western countries, with 11 studies from the US, 9 from Australia and the remaining 16 distributed across Europe (UK: 6, Germany: 4, Sweden, Netherlands and Switzerland 2 each). The majority of studies investigated the effects of invasive neurotechnology in the form of Deep Brain Stimulators (DBS) (26/36), especially in patients with Parkinson’s Disease (PD) (19/36). Many papers also investigated users’ experiences with non-invasive EEG-based BCIs (7/36), whereas all other technologies such as TMS, ECT, FES, intracortical microelectrode arrays, or spinal cord stimulation were only covered by one or two papers each. Footnote 1 Due to the large focus on PD patients, other potential fields for clinical neurotechnological applications were much less present in the analysed research, with only 4 papers each investigating the effects of DBS on patients with major depressive disorder (4/36) or obsessive-compulsive disorder (OCD) (4/36). Across all technologies and patient groups, studies most frequently relied on semi-structured interviews with individual participants (28/36), with much fewer studies using focus groups (3/36) or other qualitative methods.

We found that a large number of papers (14/36) incorporated longitudinal aspects in their study design. With view to non-invasive BCIs, this comprised involving users in the development and testing of BCIs for acquired brain injury [ 20 , 21 ], assessing subjective reports across sessions for experimental BCI training [ 22 ], or having a 2-month follow-up interview for users of a BCI for pain management after spinal cord injury [ 23 ]. Studies of invasive devices often included interviews pre- and post-implantation, with a potential third follow-up. In studies with two interviews, the first interview after implantation took place a few weeks after implantation [ 24 , 25 ], after 3 months [ 26 ], after 9 months [ 27 , 28 ] or after a year [ 29 ]. In studies with 3 interviews, post-implantation interviews were either conducted after surgery and again after 3 months in a study on spinal cord stimulation [ 30 ] or, in the case of DBS for PD, after 3 and 6 months [ 31 , 32 ] or after 3–6 and 9–12 months respectively [ 33 ]. Table  1 provides a full overview over the included studies.

Thematic findings

Our findings from the thematic analysis can be grouped into four overlapping thematic families, namely (1) ethical challenges of neurotechnology use, (2) subjective experience with clinical neurotechnologies, (3) impact on social relations, and (4) usability and technological aspects. The raw data of our findings are accessible in the supplementary file.

Ethical concerns

With respect to users’ experiences of neurotechnology that touch on classical ethical topics, we found that autonomy played a central role in slightly more than half of all papers (20/36), yet in four different ways. Many papers noted the positive impact neurotechnology has on users’ autonomy. Users often perceive the technology as enabler of greater control over their own life, allowing them “to become who they wanted to be” [ 2 ], providing them with agency and greater independence, restoring their ability to help others, or allowing them to be more spontaneous in their everyday life [ 2 , 10 , 28 , 31 , 32 , 34 , 35 , 36 , 37 ]. Some studies reported how neurotechnology may impact users’ autonomy negatively, especially by making them more dependent on technological and medical support [ 25 , 28 , 35 , 38 , 39 ]. When balancing these positive and negative impacts, some users seem to prefer such dependency and to leave control over the devices to healthcare professionals, to ensure its safe and appropriate working [ 2 , 32 , 39 , 40 ]. Also related to autonomy were concerns about consent, especially with a view to the level of information patients received before the implantation of an invasive device, which was deemed inadequate by some patients [ 2 , 24 , 31 , 34 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ]. Several papers called to include patients during the technology design process [ 2 , 31 , 39 ]. In addition, questions of responsibility and accountability in case of malfunctioning were repeatedly named as key concern [ 10 , 25 , 37 , 38 , 45 , 47 ].

Concerns about beneficence and about harming patients also featured prominently in most of the analysed papers (24/36), yet with substantive differences on a more granular level. While symptom improvement and restorative changes were widely reported [ 2 , 10 , 23 , 26 , 29 , 31 , 33 , 34 , 35 , 38 , 39 , 40 , 43 , 44 , 46 ], some users reported experiencing physical or psychological side effects, such as postoperative complications, new worries – for instance about magnetic fields or about changing batteries –, stigma, or becoming more aware of their past suffering [ 23 , 25 , 26 , 28 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 42 , 46 , 48 , 49 ]. Less frequently we found concerns about patient-doctor-relationships [ 2 , 24 , 32 , 40 , 42 , 43 ], which seem to mediate the acceptance of clinical neurotechnologies but are also themselves impacted by technology use. For instance, while some research points to the importance of patients’ trust in healthcare professionals for the acceptance of neurotechnology [ 24 ], a personal narrative described a breakdown of patient-physician relationship following a distressful DBS implantation for treating PD [ 42 ].

Impact on subjective experiences

Since the subjective lived experiences of neurotechnology users commonly constituted the central element of the reviewed qualitative papers, we found a rich field of reports in the vast majority of paper (31/36), describing experiences that were perceived as positive, negative or neutral. Neurotechnology-induced behavioural changes [ 28 , 36 , 37 , 40 , 42 , 46 , 47 , 49 ], as well as changes in feelings [ 27 , 41 , 42 ], (self-) perception [ 10 , 23 , 34 , 36 , 40 , 41 , 42 , 44 , 48 , 50 ], personality [ 27 , 29 , 34 , 35 , 36 , 37 , 42 , 43 , 44 , 47 , 49 ], preferences [ 49 , 50 ] or thinking [ 10 , 41 ] were also reported, particularly in users receiving continuous, non-adaptive deep brain stimulation (DBS).

Behavioural changes often concerned desired outcomes such as fewer obsessive thoughts and compulsive behaviours after successful OCD treatment [ 49 ], acting with less impediment due to seizure predictions [ 36 ], or acting more boldly with more energy and increased confidence due to symptom improvement in PD [ 37 , 47 ]. Nevertheless, it was necessary for patients and for their environment to adapt and get used to new patterns of behaviour. Some patients also reported undesirable behavioural changes after subthalamic DBS implantation, “bordering on mania” [ 42 ], such as being excessively talkative [ 46 ] or shopping compulsions that were later described by the patient as “ridiculous” [ 28 ].

These outwardly observable changes were often related to psychological changes that users reported. Some DBS users experienced mood changes, ranging from elevated to depressed [ 27 , 41 , 42 , 44 ], while others reported changed preferences. Sometimes this affected what users valued as important in life [ 50 ], sometimes it related to very particular preferences, such as taste in music, with one patient attributing a transition from The Rolling Stones and The Beatles to Johnny Cash to their DBS implantation [ 49 ]. In patients treated for OCD or motor disorders, two studies also found positive impact on users’ thinking, whether by freeing them from obsessive thoughts [ 41 ] or improving their concentration skill [ 10 ]. In line with the large neuroethical debate on the subject, changes at times amounted to what neurotechnology users described as personality changes. Such changes included negative impacts such as being more irritable, anxious or less patient [ 34 , 35 ] or overly increased libido [ 49 ], neutral changes, such as (re-)taking an interest in politics or movies [ 49 ], and positive changes linked to improvement of psychiatric symptoms, such as being more easy-going and daring, being more expressive and assertive, or simply being more confident [ 35 , 49 ].

In line with the diversity of these changes, patients reported a vast spectrum of different attitudes towards and relations with the neurotechnology. Some users embraced the BCI explicitly as part of themselves [ 14 , 37 , 39 , 49 ] and described how “DBS becomes a part of who you are rather than changing you” [ 37 ]. Others felt estranged using the BCI [ 28 , 36 , 37 , 42 , 49 ] and even expressed desires to remove the alien device in forceful terms: “I hate it! I wish I could pull it out!” [ 37 ]. Aside from changes brought about by the device, the patients’ state before using neurotechnology and especially their relation to their illness seemed to play a crucial role [ 28 , 51 ]. An overview over the different thematic findings is provided in Fig.  3 .

Impact of clinical neurotechnology on subjective experience. The colours represent the valence of the impact, with orange dots representing negative, green dots representing positive, and blue dots representing ambivalent changes

The overwhelming majority of studies (23/36) reported improvements of the treated symptoms [ 2 , 26 , 28 , 31 , 33 , 34 , 35 , 37 , 40 , 41 , 42 , 43 , 46 , 47 , 48 , 49 , 50 , 52 ], making patients’ lives easier [ 48 , 49 ] or – as some put it – even saving their lives [ 34 , 45 , 48 ]. Patients felt that the neurotechnology allowed them an increase in activity [ 33 , 34 , 40 ] and a return to previous forms of behaviour [ 33 , 40 , 48 , 49 ], strengthening their sense of freedom and independence [ 2 , 10 , 22 , 33 , 34 , 35 , 36 , 40 , 43 , 49 , 50 , 53 ]. Emotionally, users reported feeling more daring [ 29 , 35 ], self-confident [ 28 , 35 , 36 , 37 , 44 ] or more stable [ 34 , 50 ] as well as feelings of hope or joy [ 10 , 22 , 35 , 50 ]. For better or worse, such changes were sometimes perceived as providing a “new start” [ 34 , 48 ] or even a “new identity” [ 34 , 41 , 42 , 49 ], while others perceived their changes as a reversion to their “former” [ 28 , 29 , 47 , 49 , 50 ] or their “real” self [ 36 , 42 , 49 ].

Among the negative subjective impacts of clinical neurotechnology mentioned in the literature (16/36), users commonly reported issues of estrangement, caused by self-perceived changes to behaviour, feelings, personality traits, or patients’ relation to their disease or disorder [ 28 , 36 , 37 , 42 , 49 ]. The negative impact differed largely depending on the type of neurotechnology used as well as on the disorders and symptoms treated with the technology. While ALS patients as users of non-invasive BCIs for spelling interfaces reported increased anxiety in interaction with the devices [ 53 ], PD patients with invasive DBS reported presurgical fears of pain and of the invasive procedure as well as fear of outward manipulation within their brain through the DBS implantation [ 40 , 43 , 54 ]. Frequently, it was not entirely clear whether adverse developments such as further cognitive decline were attributable to the implanted device or to the persisting disease and its natural trajectory [ 31 , 33 , 34 , 40 , 43 , 48 , 50 ]. However, occasionally very severe psychiatric consequences of treatment were reported, notably by one PD patient who experienced mania and depressive symptoms through DBS treatment, resulting in a suicide attempt [ 42 ]. For DBS patients with OCD, negative impacts seem more related to difficulties of adapting to the new situation [ 35 , 49 ], for instance to their suddenly increased libido as a side-effect of DBS use that may be perceived as “too much” [ 49 ], or to a perceived lack of preparation for their new (OCD-free) identity [ 41 ]. In two studies on patients with OCD, the sudden improvement of symptoms also led to moments of existential crisis, given that the symptoms had shaped a great part of their previous daily activities [ 41 , 49 ].

Impact on social relations

Using a neurotechnology not only impacts users but can also affect social relations with others (23/36), particularly primary caregivers. While some neurotechnologies such as non-invasive BCIs for communication may create additional workload for caregivers if the BCI needs to be set up, neurotechnologies can also reduce their burden by rendering patients more independent [ 10 , 34 , 40 , 53 ]. Beyond workload, neurotechnologies were also reported to enrich social relations by facilitating communication [ 10 , 34 , 53 ], though in some cases, they led to potential tension between informal caregivers and patients, e.g. due to personality changes [ 28 , 35 , 37 , 40 , 42 , 47 , 49 , 55 ] or if the device was blamed for a patient’s behaviour or suggested as a solution to interpersonal problems [ 2 ]. Whether positive or negative, family and social support were reportedly playing a vital role in the treatment [ 2 , 28 , 40 , 50 ].

Similarly important was support by clinicians [ 39 , 40 ] and the wish for support groups with fellow neurotechnology users [ 27 , 30 , 40 , 41 ]. Inclusion in research activities was also reported as a positive effect of (experimental) BCIs [ 10 , 38 ]. More importantly though, in a large number of studies, neurotechnology users reported positive effects on their social relations [ 2 , 29 , 35 , 43 , 46 , 48 , 50 ], with some users reporting an increased wish to help others [ 35 , 50 ]. A negative social consequence in public was perceived stigma [ 25 , 35 , 48 ], even though some patients chose to actively show their device in public, “to spread information and knowledge about this treatment” [ 39 ].

Usability concerns

Concerns with technical questions and usability issues comprising efficiency, effectiveness and satisfaction [ 52 ] were also raised by almost half of the research papers (17/36), yet differed greatly between neurotechnologies, owing to large differences in hardware (e.g., between EEG caps and implanted electrodes) and handling (e.g., between passive neurostimulation or training-intensive active BCIs). Across all applications, invasive as much as non-invasive, the most frequent concerns (8/36 each) related to hardware issues [ 2 , 22 , 23 , 38 , 39 , 46 , 52 , 53 ] as well as to the required fine-tuning of devices to find optimal settings, associated with time-burden for their users [ 20 , 23 , 27 , 32 , 39 , 46 , 50 , 56 ]. Similarly, the training of patients required for the successful use of non-invasive, active BCIs was reported as being perceived as cumbersome or complicated, providing a potential obstacle to their implementation in everyday contexts [ 38 , 52 ]. Several studies reported that the use of such active BCIs required considerable concentration, leading to fatigue after prolonged use [ 10 , 38 , 53 ]. Mediating factors to address such obstacles were the availability of technical support [ 33 , 53 ], general attitudes towards technology [ 53 ], ease of integrating the technology into everyday life [ 10 , 38 , 53 ] and realistic expectations regarding the neurotechnology’s effects [ 30 , 38 , 40 , 46 ].

The identified publications highlight that qualitative research through interviews and focus groups offers a useful way to gain access to the subjective experience of users of a diverse range of neurotechnologies. Such investigation of users’ privileged knowledge about novel devices in turn is crucial to improve future neurotechnological developments and align them with ethical considerations already at an early stage [ 57 ]. Here, we discuss our findings by comparing different clinical neurotechnologies, identify gaps in the literature and point to the limitations of our scoping review.

One finding of our scoping review is that qualitative research on neurotechnologies has so far primarily focused on users of DBS treated for PD. In part, this may reflect that DBS is an established, effective treatment for controlling motor symptoms in PD, improving patients’ quality of life, resulting in its wide-spread adoption in many different healthcare systems worldwide [ 58 , 59 , 60 , 61 ]. Still, it would be highly beneficial to extend qualitative research to different patient groups and other clinical neurotechnologies that directly target mental states or processes, where more pronounced effects of subjective experiences may be expected.

A potential obstacle to involving more neurotechnology users beyond PD patients treated with DBS is that, for many other technologies, users are still likely to receive their treatment as part of an experimental trial. Qualitative research with such patients may face the additional practical barrier of convincing the other researchers to facilitate access to their patients. Better communication across disciplines and research fields may facilitate such access, providing much-needed insights into user experiences of experimental neurotechnologies.

Some of the articles reviewed here already offer such perspectives, e.g. the ones investigating DBS used for major depressive disorder or OCD. Such research may also help to further clarify which differences in subjective outcome are owed to technology and which are owed to differences in the treated disorders. As different patient groups are likely to have different needs and views, further research is needed to explore those needs and views and develop implementation strategies designed to address them in a patient-tailored manner. Furthermore, different neurotechnologies (and applications thereof) are likely to impact the mind of their users in a different way. Therefore, future research should investigate whether the type and modality of stimulation exert differential impacts on the subjective experience of the end users.

Our findings reveal differential effects among patients using DBS for the treatment of PD and patients using DBS for the treatment of OCD, respectively. For example, some reported effects of invasive neurotechnology such as the induction of more assertive behaviour may be a reason for concern in PD [ 28 ], while being considered a successful treatment outcome in OCD [ 35 , 49 ]. More comparative research among DBS users treated for OCD or other neuropsychiatric disorders, such as depression, are needed [ 62 ] and may help to better understand which experiences are directly attributable to the stimulation of specific brain areas such as the subthalamic nucleus for PD and the nucleus accumbens for OCD, and which result from other factors, e.g., related to undergoing surgery or to different treatment settings in neurological and psychiatric care [ 63 , 64 ].

Research on such differences may also imply practical consequences. For instance, one may wonder whether different preparation stages and possibly different degrees of information for obtaining consent may be called for between invasive clinical neurotechnologies used in psychiatry and neurology—or whether, on the contrary, similarities in the use of neurotechnologies ultimately point towards ending the distinction between mental and neurological illnesses [ 63 ]. In either case, our findings highlight that psychological impacts of clinical neurotechnologies are complex and multi-faceted phenomena—mediated by many factors—calling for more qualitative research to better grasp the lived experiences of those using novel neurotechnologies.

Our scoping review identified several gaps in the literature related to research methodology, investigated topics and investigated neurotechnologies. First, while a large number of studies embrace a longitudinal approach to investigating users’ experiences, none of the included studies looked at impacts beyond a timeframe of one year. However, as is known from DBS studies in major depressive disorder, it is important to investigate and evaluate long-term effects of neurotechnologies such as DBS [ 6 ]. Future qualitative research should therefore address this gap. Connected to this are, second, research questions that have not yet been investigated in full, such as long-term impacts of clinical neurotechnologies on memory or belief continuity. Third, empirical findings on closed-loop neurotechnologies that integrate artificial intelligence are so far nascent [ 2 , 36 ]. As there are important conceptual and ethical questions that arise specifically from the integration of human and artificial intelligence, e.g. questions of control and responsibility, further qualitative research should be conducted on users of such devices.

Finally, our findings reveal a complex and multifaceted landscape of ethical considerations. While considerations regarding personal autonomy appear largely prevalent among users, the perceived or expected impacts of neurotechnology use on personal autonomy differ significantly. Some studies suggest that neurotechnology use may enhance personal autonomy by allowing users to be more autonomous and independent in their daily lives and even restore part of the autonomous control that was disrupted by their disorders. Other studies suggest that some neurotechnologies, especially neural implants relying on autonomous components, may diminish autonomy as they may override some users’ intentions. Sometimes this ambivalent effect is observed within the same study. This is consistent with previous theoretical reflections on this topic [ 65 ] and urges scientists to develop fine-grained and patient-centred models for assessing the impact of neurotechnology on personal autonomy. These models should distinguish on-target and off-target effects and elucidate which subcomponents of personal autonomy (e.g., volition, behavioural control, authenticity etc.) are impacted by the use of neurotechnology.

Our scoping review has several limitations. Owing to the nature of a scoping review and to our inclusion criteria, there may be relevant literature that we missed to identify and analyse. For instance, since we only included English publications, we may have missed relevant research published in other languages, which may explain why we only found qualitative studies conducted in Western countries. Furthermore, our narrow search strategy excluded other relevant research, for instance qualitative studies conducted with potential users of clinical neurotechnology or with caregivers. Yet, a scoping reviews can provide a useful tool to map existing literature [ 16 , 18 ], and given recent advances in technology and accompanying qualitative research, an update of earlier reviews such as the one by Kögel et al. [ 14 ], provides an important addition to the existing literature. By looking at qualitative studies only we further import general limitations of qualitative studies, such as a lack of generalizability and a dependency on the skills and experience of the involved researchers. More standardized instruments to complement the investigation of subjective experiences of neurotechnology users therefore seem highly desirable. Recent quantitative approaches such as online surveys assessing the subjective preferences of DBS users concerning the timing of implantation [ 66 ] or studies combining qualitative data with quantitative assessments [ 67 ] point in this direction. Additionally, experimental approaches to the monitoring and evaluation of the effects of neurotechnology on the user’s experience are currently absent. Therefore, future research should complement qualitative and quantitative user evaluations based on social science methods (e.g., interviews, focus groups and questionnaires) with experimental models.

The findings of our review emphasize the diversity of individual experiences with neurotechnology across individuals and different technologies. They underscore the need to conduct qualitative research among diverse groups at different time-points to better assess the impact of such technologies on their users, which are important to inform requirements of efficacy and safety for clinical neurotechnologies. In addition, qualitative research offers one way to implement user-centred ethical considerations into product development through user-centred design and to accompany the development of novel neurotechnologies with ethical considerations as they mature and become clinical standard.

Data availability

The availability of the full data supporting the findings of this study is subject to restrictions due to the copyright of the included papers. The quotes analysed during this study are included in this published article and its supplementary information files. Further data are available from the authors upon request.

As many publications included patients with different diagnoses or investigated the effects of different neurotechnologies, the numbers indicated here do not add up.

UNESCO. Unveiling the neurotechnology landscape: scientific advancements innovations and major trends. 2023.

Klein E, et al. Brain-computer interface-based control of closed-loop brain stimulation: attitudes and ethical considerations. Brain-Computer Interfaces. 2016;3(3):140–8.

Article   Google Scholar  

Kellmeyer P, et al. The effects of closed-loop medical devices on the autonomy and accountability of persons and systems. Camb Q Healthc Ethics. 2016;25(4):623–33.

Limousin P, Foltynie T. Long-term outcomes of deep brain stimulation in Parkinson disease. Nat Reviews Neurol. 2019;15(4):234–42.

Alkawadri R. Brain–computer interface (BCI) applications in mapping of epileptic brain networks based on intracranial-EEG: an update. Front NeuroSci. 2019;13:191.

Crowell AL, et al. Long-term outcomes of subcallosal cingulate deep brain stimulation for treatment-resistant depression. Am J Psychiatry. 2019;176(11):949–56.

Mar-Barrutia L, et al. Deep brain stimulation for obsessive-compulsive disorder: a systematic review of worldwide experience after 20 years. World J Psychiatry. 2021;11(9):659.

Clausen J, et al. Help, hope, and hype: ethical dimensions of neuroprosthetics. Science. 2017;356(6345):1338–9.

Gilbert F, Viaña JNM, Ineichen C. Deflating the DBS causes personality changes bubble. Neuroethics. 2021;14(1):1–17.

Kögel J, Jox RJ, Friedrich O. What is it like to use a BCI? - insights from an interview study with brain-computer interface users. BMC Med Ethics. 2020;21(1):2.

Burwell S, Sample M, Racine E. Ethical aspects of brain computer interfaces: a scoping review. BMC Med Ethics. 2017;18(1):1–11.

Sullivan LS, Illes J. Beyond ‘communication and control’: towards ethically complete rationales for brain-computer interface research. Brain-Computer Interfaces. 2016;3(3):156–63.

Specker Sullivan L, Illes J. Ethics in published brain–computer interface research. J Neural Eng. 2018;15(1):013001.

Kögel J, et al. Using brain-computer interfaces: a scoping review of studies employing social research methods. BMC Med Ethics. 2019;20(1):18.

van Velthoven E, et al. Ethical implications of visual neuroprostheses—a systematic review. J Neural Eng. 2022;19(2):026055.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77–101.

Pham MT, et al. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synthesis Methods. 2014;5(4):371–85.

Page MJ, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Reviews. 2021;10(1):1–11.

Mulvenna M, et al. Realistic expectations with brain computer interfaces. J Assist Technol. 2012;6(4):233–44.

Martin S, et al. A qualitative study adopting a user-centered approach to design and validate a brain computer interface for cognitive rehabilitation for people with brain injury. Assist Technol. 2018;30(5):233–41.

Kryger M, et al. Flight simulation using a brain-computer interface: a pilot, pilot study. Exp Neurol. 2017;287:473–8.

Al-Taleb M, et al. Home used, patient self-managed, brain-computer interface for the management of central neuropathic pain post spinal cord injury: usability study. J Neuroeng Rehabil. 2019;16(1):1–24.

Wexler A, et al. Ethical issues in intraoperative neuroscience research: assessing subjects’ recall of informed consent and motivations for participation. AJOB Empir Bioeth. 2022;13(1):57–66.

Goering S, Wexler A, Klein E. Trading vulnerabilities: living with Parkinson’s Disease before and after deep brain stimulation. Camb Q Healthc Ethics. 2021;30(4):623–30.

Maier F, et al. Patients’ expectations of deep brain stimulation, and subjective perceived outcome related to clinical measures in Parkinson’s disease: a mixed-method approach. J Neurol Neurosurg Psychiatry. 2013;84(11):1273–81.

Thomson CJ, Segrave RA, Carter A. Changes in Personality Associated with Deep Brain Stimulation: a qualitative evaluation of clinician perspectives. Neuroethics. 2021;14:109–24.

Thomson CJ, et al. He’s back so I’m not alone: the impact of deep brain stimulation on personality, self, and relationships in Parkinson’s disease. Qual Health Res. 2020;30(14):2217–33.

Lewis CJ, et al. Subjectively perceived personality and mood changes associated with subthalamic stimulation in patients with Parkinson’s disease. Psychol Med. 2015;45(1):73–85.

Ryan CG, et al. An exploration of the experiences and Educational needs of patients with failed back surgery syndrome receiving spinal cord stimulation. Neuromodulation. 2019;22(3):295–301.

Kubu CS, et al. Patients’ shifting goals for deep brain stimulation and informed consent. Neurology. 2018;91(5):e472–8.

Merner AR, et al. Changes in patients’ desired control of their deep brain stimulation and subjective Global Control over the Course of Deep Brain Stimulation. Front Hum Neurosci. 2021;15:642195.

Liddle J, et al. Impact of deep brain stimulation on people with Parkinson’s disease: a mixed methods feasibility study exploring lifespace and community outcomes. Hong Kong J Occup Ther. 2019;32(2):97–107.

Chacón Gámez YM, Brugger F, Biller-Andorno N. Parkinson’s Disease and Deep Brain Stimulation Have an Impact on My Life: A Multimodal Study on the Experiences of Patients and Family Caregivers. Int J Environ Res Public Health. 2021;18(18):9516.

de Haan S et al. Effects of deep brain stimulation on the lived experience of obsessive-compulsive disorder patients: in-depth interviews with 18 patients. PLoS One. 2015;10(8):e0135524.

Gilbert F, et al. Embodiment and estrangement: results from a first-in-Human Intelligent BCI Trial. Sci Eng Ethics. 2019;25(1):83–96.

Gilbert F, et al. I miss being me: phenomenological effects of deep brain stimulation. AJOB Neurosci. 2017;8(2):96–109.

Grübler G, et al. Psychosocial and ethical aspects in non-invasive EEG-based BCI research - A survey among BCI users and BCI professionals. Neuroethics. 2014;7(1):29–41.

Hariz G-M, Hamberg K. Perceptions of living with a device-based treatment: an account of patients treated with deep brain stimulation for Parkinson’s disease. Neuromodulation: Technol Neural Interface. 2014;17(3):272–8.

Liddle J, et al. Mapping the experiences and needs of deep brain stimulation for people with Parkinson’s disease and their family members. Brain Impairment. 2019;20(3):211–25.

Bosanac P, et al. Identity challenges and ‘burden of normality’ after DBS for severe OCD: a narrative case study. BMC Psychiatry. 2018;18(1):186.

Gilbert F, Viaña JN. A personal narrative on living and dealing with Psychiatric symptoms after DBS surgery. Narrat Inq Bioeth. 2018;8(1):67–77.

Cabrera LY, Kelly-Blake K, Sidiropoulos C. Perspectives on deep brain stimulation and its earlier use for parkinson’s disease: a qualitative study of US patients. Brain Sci. 2020;10(1).

Bluhm R, et al. They affect the person, but for Better or worse? Perceptions of Electroceutical interventions for Depression among psychiatrists, patients, and the Public. Qual Health Res. 2021;31(13):2542–53.

Sankary LR et al. Exit from Brain Device Research: A Modified Grounded Theory Study of Researcher Obligations and Participant Experiences. AJOB Neurosci. 2021;1–12.

Thomson CJ, et al. Nothing to lose, absolutely everything to Gain: patient and caregiver expectations and subjective outcomes of deep brain stimulation for treatment-resistant depression. Front Hum Neurosci. 2021;15:755276.

Mosley PE, et al. Woe betides anybody who tries to turn me down.’ A qualitative analysis of neuropsychiatric symptoms following subthalamic deep brain stimulation for Parkinson’s Disease. Neuroethics. 2021;14:47–63.

Hariz G-M, Limousin P, Hamberg K. DBS means everything-for some time. Patients’ perspectives on daily life with deep brain stimulation for Parkinson’s disease. J Parkinson’s Disease. 2016;6(2):335–47.

de Haan S, et al. Becoming more oneself? Changes in personality following DBS treatment for psychiatric disorders: experiences of OCD patients and general considerations. PLoS ONE. 2017;12(4):e0175748.

Shahmoon S, Smith JA, Jahanshahi M. The lived experiences of deep brain stimulation in parkinson’s disease: an interpretative phenomenological analysis. Parkinson’s Disease. 2019;2019(1):1937235.

Adamson AS, Welch HG. Machine learning and the Cancer-diagnosis problem - no gold Standard. N Engl J Med. 2019;381(24):2285–7.

Zulauf-Czaja A, et al. On the way home: a BCI-FES hand therapy self-managed by sub-acute SCI participants and their caregivers: a usability study. J Neuroeng Rehabil. 2021;18(1):1–18.

Blain-Moraes S, et al. Barriers to and mediators of brain-computer interface user acceptance: Focus group findings. Ergonomics. 2012;55(5):516–25.

LaHue SC, et al. Parkinson’s disease patient preference and experience with various methods of DBS lead placement. Parkinsonism Relat Disord. 2017;41:25–30.

Lewis CJ, et al. The impact of subthalamic deep brain stimulation on caregivers of Parkinson’s disease patients: an exploratory study. J Neurol. 2015;262(2):337–45.

Cabrera LY, et al. Beyond the cuckoo’s nest: patient and public attitudes about Psychiatric Electroceutical interventions. Psychiatr Q. 2021;92(4):1425–38.

Jongsma KR, Bredenoord AL. Ethics parallel research: an approach for (early) ethical guidance of biomedical innovation. BMC Med Ethics. 2020;21(1):1–9.

Lozano AM, et al. Deep brain stimulation: current challenges and future directions. Nat Reviews Neurol. 2019;15(3):148–60.

Schuepbach W, et al. Neurostimulation for Parkinson’s disease with early motor complications. N Engl J Med. 2013;368(7):610–22.

Follett KA, et al. Pallidal versus subthalamic deep-brain stimulation for Parkinson’s disease. N Engl J Med. 2010;362(22):2077–91.

Mahajan A, et al. Global variability in Deep Brain Stimulation practices for Parkinson’s Disease. Front Hum Neurosci. 2021;15:667035.

Bublitz C, Gilbert F, Soekadar SR. Concerns with the promotion of deep brain stimulation for obsessive-compulsive disorder. Nat Med. 2023.

White P, Rickards H, Zeman A. Time to end the distinction between mental and neurological illnesses. BMJ. 2012;344.

Martin JB. The integration of neurology, psychiatry, and neuroscience in the 21st century. Am J Psychiatry. 2002;159(5):695–704.

Ferretti A, Ienca M. Enhanced cognition, enhanced self? On neuroenhancement and subjectivity. J Cogn Enhancement. 2018;2(4):348–55.

Montemayor J, et al. Deep brain stimulation for Parkinson’s Disease: why earlier use makes Shared decision making important. Neuroethics. 2022;15(2):1–11.

Maier F, et al. Subjective perceived outcome of subthalamic deep brain stimulation in Parkinson’s disease one year after surgery. Parkinsonism Relat Disord. 2016;24:41–7.

Download references

Acknowledgements

GS would like to thank the attendees of the ERA-NET NEURON mid-term seminar (Madrid, January 2023) for kind and constructive feedback on an earlier draft.

This work was supported by the ERA-NET NEURON project HYBRIDMIND (SNSF 32NE30_199436; BMBF, 01GP2121A and -B), and in part by the European Research Council (ERC) under the project NGBMI (759370), the Federal Ministry of Research and Education (BMBF) under the projects SSMART (01DR21025A), NEO (13GW0483C), QHMI (03ZU1110DD), QSHIFT (01UX2211) and NeuroQ (13N16486), as well as the Einstein Foundation Berlin (A-2019-558).

Author information

Authors and affiliations.

Faculty of Medicine, Institute for History and Ethics of Medicine, Technical University of Munich, Munich, Germany

Georg Starke & Marcello Ienca

College of Humanities, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland

Faculty of Law, University of Ottawa, Ottawa, ON, Canada

Tugba Basaran Akmazoglu

Clinical Neurotechnology Laboratory, Department of Psychiatry and Neurosciences at the Charité Campus Mitte, Charité – Universitätsmedizin Berlin, Berlin, Germany

Annalisa Colucci, Mareike Vermehren, Maria Buthut & Surjo R. Soekadar

Centre for Health Law Policy and Ethics, University of Ottawa, Ottawa, ON, Canada

Amanda van Beinum

Bertram Loeb Research Chair, Faculty of Law, University of Ottawa, Ottawa, ON, Canada

Jennifer A. Chandler

Faculty of Law, Universität Hamburg, Hamburg, Germany

Christoph Bublitz

You can also search for this author in PubMed   Google Scholar

Contributions

GS, TBA, AC, MV, SS, CB, JC and MI contributed to the design and planning of the review, conducted the literature searches and organized and analyzed collected references. GS and MI wrote different sections of the article. All authors provided review of analysis results and suggested revisions for the write-up. All authors reviewed and approved the manuscript before submission.

Corresponding author

Correspondence to Georg Starke .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Starke, G., Akmazoglu, T.B., Colucci, A. et al. Qualitative studies involving users of clinical neurotechnology: a scoping review. BMC Med Ethics 25 , 89 (2024). https://doi.org/10.1186/s12910-024-01087-z

Download citation

Received : 23 January 2023

Accepted : 02 August 2024

Published : 14 August 2024

DOI : https://doi.org/10.1186/s12910-024-01087-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Neurotechnology
• Qualitative research
• Subjective experience
• Self-perception
• Patient-centred technology

BMC Medical Ethics

ISSN: 1472-6939

• General enquiries: [email protected]

• Open access
• Published: 12 August 2024

Healthcare staff experiences on the impact of COVID-19 on emergency departments: a qualitative study

• Ahmet Butun   ORCID: orcid.org/0000-0002-6856-9389 1  

BMC Health Services Research volume  24 , Article number:  921 ( 2024 ) Cite this article

1 Altmetric

Metrics details

The COVID-19 pandemic has had a major impact on the access and delivery of healthcare services, posing unprecedented challenges to healthcare staff worldwide. Frontline healthcare staff faced unique stressors and challenges that impact their well-being and patient care. This qualitative study aimed to explore the experiences and perspectives of frontline ED healthcare staff on emergency care services during the COVID-19 pandemic, providing valuable insights into the challenges, adaptations, and lessons learned in delivering emergency care.

This study utilized a qualitative approach. In-depth semi-structured interviews were conducted with 30 ED healthcare staff from three different hospitals located in Turkey between 15/03/2022 and 30/04/2022. Convenience sampling was used to recruit participants. The duration of the interviews ranged from 28 to 37 min. Data saturation was reached as no new information was gathered. The data were analyzed using the thematic analysis method. NVivo software was used to manage the data analysis process. Member check was carried out to ensure that the generated themes conformed to the participants’ views.

15 sub-themes under three themes emerged: (1) the impact of COVID-19 on emergency care services, including sub-themes of “introducing a COVID-19 unit in the ED”, “changes in the routine functioning of EDs”, “changes in the number of ED visits”, “quality of care”, “resources”, and “increased workload”; (2) the psychological effects of COVID-19 on ED healthcare staff, including sub-themes of “staying away from family”, “fear”, “society’s perspective on healthcare professionals”, “morale-staff burnout”, “psychological and emotional effects”, and “unable to receive sufficient support”; and (3) the difficulties faced by ED healthcare staff, including sub-themes of “difficult working conditions”, “community-based effects difficulties”, and “COVID-19 is an unknown situation”.

Staff burnout threatens the quality of patient care and staff retention, and therefore this should be addressed by ED directors and leaders. This study could inform appropriate stakeholders regarding lessons learned from COVID-19 to better manage future pandemics. Learning from such lived experiences and developing appropriate interventions to minimize the difficulties faced during COVID-19 would allow better management of future pandemics. This study calls for a reform to address the challenges faced by healthcare staff, improve the overall response to public health crises, and enhance the resilience of healthcare systems for future crises.

Peer Review reports

The COVID-19 pandemic has had a major impact on the access and delivery of healthcare services, posing unprecedented challenges to healthcare staff worldwide. The COVID-19 pandemic caused a high mortality rate and disruption to healthcare and social care around the world [ 1 , 2 ]. The pandemic has required the reconfiguration of health services to ensure the maintenance of essential health services [ 3 ]. The COVID-19 pandemic also caused delays in seeking care due to the fear of infection [ 4 ]. Delaying care may increase morbidity and mortality in non-COVID-19 patients [ 2 , 5 ]. The first COVID-19 case was identified on 11 March 2020 in Turkey. Around 35 million people were tested, around 2.9 million patients were diagnosed with COVID-19, around 30.000 patients were died within 1 year of starting of the pandemic, between 11 March 2020 and 11 March 2021 [ 6 ].

Emergency Departments (EDs) healthcare staff face an enormous mental burden and physical exertion when caring for patients potentially infected with COVID-19. Frontline healthcare staff face unique stressors and challenges that impact their well-being and patient care [ 7 , 8 ]. ED healthcare staff were deeply impacted by COVID-19, and they are at a high risk of burnout due to COVID-19 [ 9 , 10 ]. However, the COVID-19 pandemic has underscored the need for qualitative research to understand the experiences and perceptions of healthcare staff, as well as the use of healthcare services during this unprecedented health emergency [ 11 , 12 , 13 ]. Therefore, exploring the experiences of ED healthcare staff throughout the COVID-19 pandemic will provide a comprehensive understanding of their unique experiences and challenges.

In addition, studies showed that EDs in low-income and middle-income countries are likely to be impacted more significantly by the effects of the COVID-19 pandemic than those in high-income countries [ 9 , 14 , 15 ]. Thus, studying how the COVID-19 pandemic affects emergency care services in the context of a middle-income country (Turkey) is needed. This study focused on the impact of the COVID-19 pandemic on the general working conditions of the emergency care system, including ED services, ED healthcare staff, and ED patients in addition to its impact on general healthcare system. This qualitative study aims to explore the experiences and perspectives of frontline ED healthcare staff on emergency care services during the COVID-19 pandemic, providing valuable insights into the challenges, adaptations, and lessons learned in delivering emergency care.

Study design

This study utilized a qualitative exploratory descriptive approach as this allows to explore, describe, and a deep understanding of ED healthcare staff experiences during the COVID-19 pandemic.

Participants and settings

Participants were ED healthcare staff including ED nurses ( n  = 20), ED physicians ( n  = 8), and ED consultants ( n  = 2) from 3 different hospitals located in Turkey, namely, Mardin Training and Research Hospital ( n  = 21), Midyat Public Hospital ( n  = 6), and Kiziltepe Public Hospital ( n  = 3). Six healthcare staff refused to participate in this study. Mardin Training and Research Hospital is a public and tertiary hospital with a 700-bed capacity and around 150 ED healthcare staff. Midyat Public Hospital and Kiziltepe Public Hospital are public hospitals with 150-bed and 300-bed capacity, respectively. Convenience sampling was used to recruit participants. ED healthcare staff working in three hospitals were invited to participate in an interview. Those who accept to participate were included in the study. Participants characteristics were provided in Table  1 .

Procedure for the interviews

The interview guide (provided in Table  2 ) was developed by the researcher (A.B.) and piloted with 4 healthcare staff before commencing data collection. In-depth semi-structured interviews were conducted by researcher (A.B.), who had experience in conducting interviews and qualitative research, with 30 ED healthcare staff between 15/03/2022 and 30/04/2022. A quiet and comfortable private room at each hospital was arranged for the interviews. All interviews were audio-recorded and conducted in Turkish. The duration of the interviews ranged from 28 to 37 min. Data saturation was reached as no new information was gathered.

Ethical considerations

Ethical approval was obtained from the Mardin Artuklu University Non-Interventional Clinical Research Ethics Committee (Date: 08/03/2022, REF: E-76272411-900-47908). The participants were informed about the aim of the study, and verbal consent was obtained from all participants.

Data analysis and rigor

The data were analyzed using thematic analyses. A six-step thematic analysis developed by Braun and Clarke [ 16 ] was followed: (1) familiarizing with the data, (2) generating initial codes, (3) searching for themes, (4) reviewing themes, (5) defining and naming themes, and (6) producing the report. NVivo software was used to manage the data analysis process. The researcher (A.B.) generates the themes and sub-themes by following a six-step thematic analysis. The developed sub-themes and themes were reviewed and checked by a second qualitative researcher (Y.Y.) using NVivo software, and consensus was reached by discussion. In addition, member check was carried out to ensure that the generated themes conformed to the participants’ views of the topic and the process, and therefore minimized researcher bias. Following the data analysis process, the researcher (A.B.) contacted five participants by email to compare how the themes generated from the analysis related to their experience (member check). They reported a high level of congruence (around 95%) between the themes descriptions and their views, thus adding credibility to the results. All of these processes add to the rigor of the results and increase their credibility and trustworthiness.

The thematic analysis was concluded with 15 sub-themes under three themes. The three themes were the impact of COVID-19 on emergency care services, the psychological effects of COVID-19 on ED healthcare staff, and the difficulties faced by ED healthcare staff.

Impact of COVID-19 on emergency care services

This theme describes how COVID-19 changed the routine functioning of EDs, how ED visits were affected, how protective measures were taken, and how it affected the quality of care, resources used, and workload. Six sub-themes emerged: “introducing a COVID-19 unit in the ED”, “changes in the routine functioning of EDs”, “changes in the number of ED visits”, “quality of care”, “resources”, and “increased workload”.

Introducing a COVID-19 unit in the ED

A separate unit was introduced beside the ED to care for those with COVID-19 or suspected of COVID-19. Extra precautions against COVID-19 were taken in these units. Introducing these services might prevent contamination between those with and without COVID-19. In addition, the COVID-19 process delayed treatment in the ED. ED staff had to take precautions for all patients, which increased the time allocated for each patient and subsequently delayed treatment.

“The COVID-19 unit was introduced in the ED. Those with suspected COVID-19 were referred to the COVID-19 unit for examination. COVID-19 tests were also conducted here.” (Participant 16). “The ED has been divided into two units , the adult ED and the COVID-19 unit. The staff working in the COVID-19 unit paid attention to the use of masks , gowns , and visors , social distance , and cleaning rules while providing care to the patients.” (Participant 19). “The COVID-19 unit was introduced. In the past , we used to quickly perform the procedures of the incoming patients without masks and gloves , but because of the pandemic , we started to perform the procedures of the patients by taking precautions. Because we did not know whether the incoming patients had COVID-19 or not , we took the same precautions for each patient.” (Participant 5).

Changes in the routine functioning of EDs

The COVID-19 pandemic has changed the routine functioning of ED services. ED healthcare staff started to perceive all ED patients as suspected COVID-19 patients. Also, the ED healthcare staff had to take more precautions while examining the patients. They had to use protective equipment such as gloves, masks, aprons, and visors. In addition, healthcare staff also had to pay attention to social distancing and hygiene rules while providing care. All these extra precautions lead to fatigue and working under difficult working conditions, which could lead to high levels of stress and burnout. Working under such circumstances overwhelmed the ED healthcare staff during the COVID-19 pandemic.

“We approached each patient as if they had COVID-19 , without knowing whether the patients had COVID-19 or not. Also , it was very difficult to change the protective equipment each time.” (Participant 16). “We started working with protective equipment. We were working more distantly with the patients. We have started to pay more attention to hygiene rules. The workload of the ED has increased enormously.” (Participant 22).

The ED healthcare staff explained that they had difficulties dealing with COVID-19 as this was a new pandemic and it was an unknown process. ED healthcare staff start to use protective equipment and explain the importance of paying attention to prevention and precautionary methods to patients. In addition, ED healthcare staff had difficulties dealing with COVID-19 because of its high contagiousness.

“Since it was the first time , we had experienced such a process , we had a lot of difficulties managing the process. The disease was new , and people were unconscious. Patients began to be treated more distantly and more carefully. We started to use protective equipment and made a lot of effort to explain the methods of protection and precaution to the patients. The contagiousness of the disease made our job even more difficult.” (Participant 25). “We started using protective equipment to protect ourselves. Disinfectants were used. We pay attention to social distancing while examining patients. We changed the gloves frequently. I washed my hands often. Hand washing is essential. The risk of infection was very high. I started to use protective equipment regularly.” (Participant 6).

Changes in the number of ED visits

ED healthcare staff stated that the number of ED visits by those whose condition was non-urgent decreased. The decrease in the number of ED visits could be a result of fear of acquiring COVID-19 infection and a desire to reduce the pressure on the ED. However, ED healthcare staff reported that patients started to visit the EDs again after a few months from the start of the pandemic. Such decrease in the number of non-urgent ED visits was temporary only at peak incidences of COVID-19.

“At the beginning of the pandemic , non-urgent ED visits decreased. Patients did not come to the ED because of fear of COVID-19. However , they come to the ED now , it has become normal for the people , they do not care about COVID-19.” (Participant 10). “In the early days of COVID-19 , when there were restrictions , the number of patients with non-urgent conditions was quite low. When the restrictions were abolished , it started to increase again. The number of patients with non-urgent conditions increases when COVID-19 cases decrease , and the number of those with non-urgent conditions decreases when COVID-19 cases increase.” (Participant 5).

Quality of care

ED healthcare staff stated that the COVID-19 pandemic decreased the quality of care in the ED because of the limited time allocated for each patient. This is an important issue for patient safety. ED healthcare staff stated that they had limited time for patient examination and this could affect the quality of care provided to patients.

“There has been a decrease in the quality of treatment and care provided to patients. There has not been enough time for patients to be examined.” (Participant 11).

ED healthcare staff stated that they experienced a lack of resources and sometimes unavailability of resources during COVID-19. ED healthcare staff stated that they sometimes had to work without protective equipment while dealing with patients with COVID-19. Working without protective equipment increases the risk of COVID-19 transmission. In addition, some medications were out of stock, and they had to use other available medications while caring for patients.

“Protective equipment such as masks and gloves have decreased over time. Equipment began to be distributed in limited quantities per staff member. There were times when we had to work without protective equipment due to a lack of resources. This increased the risk of transmission of the disease.” (Participant 12).

Increased workload

The participants stated that the workload of the ED increased during COVID-19. Increased workload decreases staff performance and efficiency. In addition, some of the staff were infected with COVID-19 and therefore unable to work during that time. This caused an increased workload for those who are not yet infected. Healthcare staff reported that they experienced irregular working hours due to an unpredicted number of staff to work, and this led to an irregular and limited social life. ED healthcare staff had to deal with COVID-19 with a limited number of staff. ED healthcare staff could not get enough rest, which caused staff burnout.

“When we were infected with COVID-19 , we were away from the hospital and from our work for a week. Therefore , there was a lack of staff and an increase in the workload. The staff had to work a 24-hour shift because employees with COVID-19 could not come to work. We had to work overtime. In the past , we worked according to a certain plan; however , there was an irregular working plan during the COVID-19 process. The high contagiousness of COVID-19 caused disruptions.” (Participant 1). “Our working conditions have become very difficult. Our workload has increased considerably.” (Participant 10). “The workload of the ED has increased with the pandemic. We could not get enough rest. There were limited number of staff. Our working life is always busy , stressful , and exhausting. I had a harder time with the disease. My work routine has changed. There was an unnecessary workload , but there was not enough healthcare staff.” (Participant 26).

Psychological effects of COVID-19 on ED healthcare staff

This theme explains how the COVID-19 pandemic affects ED healthcare staff, including their social life, fears, psychological and emotional effects, and how they were supported during COVID-19. Six sub-themes emerged: “Staying away from family”, “fear”, “society’s perspective on healthcare professionals”, “morale-staff burnout”, “psychological and emotional effects” and “unable to receive enough support”.

Staying away from family

Almost all participants reported that they had experienced difficult times because they had to stay away from their families to reduce the risk of infection and keep them safe. Staying away from their families affects ED healthcare staff psychologically and increases their anxiety and stress levels.

“During the COVID-19 pandemic , most of the staff did not go home; they stayed away from their families because of the risk of infection. Some people stayed in hotels and other types of accommodation” (Participant 10). “The time we spend with our families has decreased. The high contagiousness of the disease and our fear of infecting our loved ones increased our anxiety and stress levels. We had to be very careful not to get the disease and infect our loved ones.” (Participant 11). “Being away from my family made me psychologically depressed.” (Participant 16).

The participants stated that they experienced fear regarding COVID-19. Such fear was regarding their own health and the potential consequences of contracting COVID-19, hospitalization and transmitting the virus to others, including colleagues, patients, and family members. Participants stated that the pandemic affected their mental well-being. Many healthcare staff faced considerable stress and anxiety during COVID-19.

“At the beginning of the pandemic , we were all psychologically feared. I was worried and afraid of how the disease would progress , how I would pass it if I infected , whether I would be hospitalized , and whether it would infect others if it infected me.” (Participant 16). “Psychologically , we were in a constant state of fear.” (Participant 24). “It affected me badly. I am a person who loves to live. I was afraid of dying. The pandemic process was very difficult and worn me out.” (Participant 26). “When there are deaths or negative situations among our colleagues , we are inevitably affected emotionally. We were affected emotionally. Also , I was afraid of infecting my family or any other person” (Participant 1).

Society’s perspective on healthcare professionals

Almost all healthcare staff stated that they were excluded from the members of the society because of being a healthcare staff. Being healthcare staff during the pandemic times means that they have a higher risk of transmission of the disease, and therefore, they were not welcomed in society. Healthcare staff felt excluded by members of society during the pandemic. Such exclusion by society affects the mental health and morale of healthcare staff.

“Society ran away from us during the pandemic.” (Participant 10). “In this process , everyone treated us as if we had COVID-19 because we were healthcare professionals. I felt so excluded.” (Participant 11). “Because I was a healthcare staff member , even my neighbours were not close to me , and they did not even want to use the same lift.” (Participant 13). “Society has become afraid of us.” (Participant 14).

Morale-staff burnout

Some ED healthcare staff members were psychologically affected. ED healthcare staff struggled with COVID-19 and experienced sleeplessness, stress, and exhaustion. Participants reported a high level of burnout related to COVID-19. It was found that COVID-19 caused an increased workload in the ED, which led to staff being exhausted, getting stressed, working without getting enough sleep, and experiencing burnout.

“In addition to the serious battle we fought physically , we were also fighting a great battle spiritually. We had many friends whose psychology was disturbed by this disease. As a result , we were constantly tensed and stressed. We experienced burnout in the process.” (Participant 25). “Due to the increasing number of cases , our workload in the ED has increased a lot. This made it very difficult for us; we were sleepless for days.” (Participant 27). “I cried for nights because of the difficulty of this process. It was an exhausting process. Understanding and tolerance are always expected from us , but we are never shown these things. We are human too , so we get burnout , angry , and tired.” (Participant 30). “Healthcare staff who contracted COVID-19 had to start work after a few days without being tested again.” (Participant 24). “This process has demoralized all of us. Our staff and friends have been infected with COVID-19. Some of the healthcare staff members died. We have also experienced such incidents. We are very saddened by these events.” (Participant 4).

Psychological and emotional effects

Participants expressed that the pandemic had a significant negative impact across multiple domains of life, including family, social, and work life. Participants reported that their social life was negatively affected. They experienced feelings of isolation or disconnection from friends, extended family, or social networks. Such social isolation and working under difficult circumstances overstressed ED healthcare staff and, affected them psychologically and emotionally. ED healthcare staff stated that such conditions increased their stress levels.

“Our family life , social life , and working life have been affected in a very negative way.” (Participant 25). “It was a very difficult process , and we were affected psychologically.” (Participant 28). “It wore me out psychologically. My social life was affected too much. I could not see my loved ones outside.” (Participant 8). “We had difficult days psychologically and physically” (Participant 12). “The pandemic has affected us negatively. We lost a chief physician who was a former colleague of mine. I was affected by this loss. When we thought about the risk of our family , relatives , and those with chronic diseases getting COVID-19 , we were burned out more.” (Participant 13).

Unable to receive sufficient support

Some ED healthcare staff reported that they did not receive sufficient support during the COVID-19 pandemic. Their motivation decreased due to unable to receive the required support. They experienced violence and were affected financially and spiritually. In addition, they experienced social isolation and a lack of social life. Accumulation of all these negative conditions affects ED healthcare staff psychologically and leads them to feel alone. The ED healthcare staff described that the absence of support and motivation contributed to their ability to cope with the challenges they faced.

“This process was difficult. There was no source of motivation. During this period , I was separated from my family and friends. I had difficulties both financially and spiritually. Cases of verbal and physical violence have increased. Health policies must be changed.” (Participant 30).

Difficulties faced by the ED healthcare staff

This theme describes the difficulties faced by ED healthcare staff during the COVID-19 pandemic. Three sub-themes emerged “difficult working conditions”, “community-based effect difficulties”, and “COVID-19 was an unknown situation”.

Difficult working conditions

ED healthcare staff worked under difficult working conditions during COVID-19. They had to work by using protective equipment all the time while working. In addition, ED staff stated that the risk of infection for them was high, and this caused them to be stressed, stay away from their families, and be isolated from society.

“We had a lot of trouble. We had to work with heavy protective equipment. The risk of infection was very high. As the number of cases increased , patient circulation increased , which affected us negatively.” (Participant 1). “We tried to be more careful while working. Healthcare staff are faced with difficulties due to a lack of protective equipment. We put in more effort. We sweated while working with protective equipment. There have also been times when we have put our health at risk.” (Participant 12). “It was very difficult to work with protective equipment; standing with them all day long left us drenched in sweat. It was a very difficult process , and it affected us psychologically.” (Participant 28).

Community-based effect difficulties

The ED healthcare staff stated that they faced significant difficulties due to non-compliance with safety standards by members of the community. ED healthcare staff reported members of the community did not adhere to recommended safety measures such as wearing masks, maintaining social distance, and practising proper hygiene. This non-compliance could cause serious risks for both healthcare staff and other patients because it increases the likelihood of virus transmission within healthcare settings.

“We had serious problems because the patients did not comply with the rules of masking , social distance , and hygiene.” (Participant 10). “When patients are referred to the COVID-19 unit , they do not go there to avoid testing. We started missing real emergency cases. Patients with high blood pressure , heart failure , and diabetes started to burden the emergency department because they did not use their medications or because their controls were delayed.” (Participant 10).

COVID-19 is an unknown situation

The participants stated that the COVID-19 pandemic was an unknown situation and, therefore, they did not know how to respond to this pandemic, including the symptoms of the disease, how to approach patients, what precautions should be taken, and how to treat or alleviate the disease.

“At the beginning of the pandemic , we did not know exactly what to do because the disease was new. Information was limited. Therefore , the emergency department could not be managed.” (Participant 2). “Since we were caught unprepared at the beginning of the pandemic , we had difficulties about what to do , how to take actions , how to make the patient use the protective equipment , and how to protect ourselves.” (Participant 18). “Because we had no information about the disease , we had difficulties in controlling the process , and we did not know how to approach the patients.” (Participant 22). “Since we have experienced such a process for the first time , we have had many difficulties in managing the process. The disease was new , and people were unconscious. All patients were approached with suspicion of COVID-19. Patients began to be treated more distantly and more carefully.” (Participant 25). “We did not know what the disease was like at the beginning of the pandemic and how we should approach patients. Therefore , the care and treatment provided to patients was inadequate.” (Participant 28).

This study found that COVID-19 negatively affects emergency care services in Turkey, including changes in the functioning of EDs and ED visits, decreased quality of care, increased use of resources, and workload. In line with the existing literature [ 2 , 17 , 18 , 19 , 20 ], this study revealed that the number of ED visits decreased at the beginning of the pandemic. The existing literature showed that the number of ED visits decreased by 65% [ 20 ], by 50% [ 21 ], and by 37% [ 22 ] during the first lockdown. Such a reduction in the number of ED visits could be a result of restrictions and the fear of being infected with COVID-19. The existing literature showed that patients were concerned about visiting the ED during the COVID-19 pandemic [ 23 ]. However, ED visits enormously increased after abolishing the restrictions as patients adapted their health-seeking behaviors throughout the pandemic.

The results of this study highlighted that ED healthcare staff experienced many difficulties during the COVID-19 pandemic, such as staying away from their family, fear, negative society perspective on healthcare staff, morale and staff burnout, psychological and emotional effects, and inability to receive enough support. This study found that ED healthcare staff had to stay away from their families during the COVID-19 pandemic because of the risk of infection to their families, which concurred with the existing literature [ 24 , 25 ]. This affects ED healthcare staff psychologically and increases their anxiety and stress levels. In addition, this study found that ED healthcare staff experienced distress and a high level of burnout during the COVID-19 pandemic, which concurs with some of the existing studies [ 8 , 26 , 27 , 28 , 29 , 30 ]. In line with some studies [ 26 , 31 , 32 ], this study found that experiencing burnout could be negatively associated with patient satisfaction, quality of care, staff morale and retention, and therefore a loss of workforce for the future.

In line with the results of this study, ED healthcare staff faced challenges such as increased workload and resource constraints during the COVID-19 pandemic [ 10 , 33 , 34 , 35 ]. ED healthcare staff face mental burdens and physical exertion when caring for patients in the ED during the COVID-19 pandemic. ED healthcare staff provide care under difficult circumstances with limited resources. In line with these results, some studies suggested that stress factors in relation to providing health care for patients with COVID-19 should be addressed [ 7 , 36 ].

Strengths and limitations

One of the strengths of this study is to include a large sample of ED healthcare staff working in three different hospitals with in-depth semi-structured interviewing resulting in a rich and detailed source of qualitative data for analysis. One of the limitations of this study could be the generalizability of the results due to the nature of qualitative research, which does not attempt to generalize the results to other populations. In addition, this study was conducted around 1 year later than the starting point of time of COVID-19. Therefore, participants may not recall all their experiences during COVID-19. The results may be transferrable to policymakers, ED directors, or other key stakeholders across Turkey and other countries with similar contexts.

Conclusion and recommendations

This study could provide a better understanding of how ED services and ED staff were affected by the COVID-19 pandemic, including decreased quality of care in the ED, increased workload, resource strains, psychological effects on ED staff, and related difficulties. Learning from such lived experiences and developing appropriate interventions to minimize the difficulties faced during COVID-19 would allow better management of future pandemics.

Staff burnout threatens the quality of patient care and staff retention, and therefore this should be addressed by ED directors and leaders. Supporting staff in dealing with difficulties such as psychological problems, fears, burnout and providing a safe working environment could contribute to staff well-being, a better workforce for the future, and staff retention. This study calls for a reform to address the challenges faced by healthcare staff, improve the overall response to public health crises such as the COVID-19 pandemic, and enhance the resilience of healthcare systems despite future crises. This study could inform appropriate stakeholders regarding lessons learned from COVID-19 to better manage future pandemics. Learning from such lived experiences and developing appropriate interventions to minimize the difficulties faced during COVID-19 would allow better management of future pandemics.

Data availability

No datasets were generated or analysed during the current study.

Reschen ME, et al. Impact of the COVID-19 pandemic on emergency department attendances and acute medical admissions. BMC Emerg Med. 2021;21(1):143.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Razimoghadam M, et al. Changes in emergency department visits and mortality during the COVID-19 pandemic: a retrospective analysis of 956 hospitals. Archives Public Health. 2024;82(1):5.

Article   Google Scholar  

Kleine-Bingham M, et al. Country Learning on maintaining quality essential Health services during COVID-19 in Timor-Leste: a qualitative analysis. BMJ Open; 2023.

Davis AL, et al. Caregiver-reported delay in presentation to pediatric emergency departments for fear of contracting COVID-19: a multi-national cross-sectional study. Can J Emerg Med. 2021;23(6):778–86.

Sahakyan S, et al. Factors associated with delay or avoidance of medical care during the COVID-19 pandemic in Armenia: results from a nationwide survey. BMC Health Serv Res. 2024;24(1):20.

Article   PubMed   PubMed Central   Google Scholar  

Ministry of Health Turkey. COVID-19 Information Platform. 2024. [Accessed: 19/07/2024]; https://covid19.saglik.gov.tr/TR-66935/genel-koronavirus-tablosu.html

Raabia S, et al. Caring in a Crisis: understanding the stressors and Uplifts for National Health Service Frontline Staff through the Lens of clinical psychologists. Stress and Health; 2022.

Ching SM, et al. Psychological distress among healthcare providers during COVID-19 in Asia: systematic review and meta-analysis. PLoS ONE. 2021;16(10):e0257983.

Mitchell R, Banks C. Emergency departments and the COVID-19 pandemic: making the most of limited resources. Emerg Med J. 2020;37(5):258–9.

Article   PubMed   Google Scholar  

Morgantini LA, et al. Factors contributing to healthcare professional burnout during the COVID-19 pandemic: a rapid turnaround global survey. PLoS ONE. 2020;15(9):e0238217.

Awaji NNA, Almudaiheem AA, Mortada EM. Assessment of caregivers’ perspectives regarding Speech-Language services in Saudi Arabia during COVID-19. PLoS ONE. 2021;16(6):e0253441.

Bugis BA. Patients self-reporting of utilizing Teledental services during the COVID-19 pandemic in Saudi Arabia. J Patient Experience, 2022.

Mishra S, et al. Impact of COVID-19 pandemic on safe abortion and Family Planning Services at a Tertiary Care women’s Hospital in Nepal. International Journal of Reproduction Contraception Obstetrics and Gynecology; 2021.

Bong CL, et al. The COVID-19 pandemic: effects on low- and Middle-Income Countries. Anesth Analg. 2020;131(1):86–92.

Article   CAS   PubMed   Google Scholar  

Gupta M, et al. The need for COVID-19 research in low- and middle-income countries. Global Health Res Policy. 2020;5(1):33.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77–101.

Moynihan R, et al. Impact of COVID-19 pandemic on Utilisation of Healthcare Services: a systematic review. BMJ Open; 2021.

Raman R, Madhusudan M. Impact of the COVID-19 pandemic on admissions to the Pediatric Emergency Department in a Tertiary Care Hospital. Indian J Pediatr. 2021;88(4):392–392.

Kyle D, et al. The wider implications of the COVID-19 pandemic: assessing the impact of accident and emergency use for frequent attenders. Int Emerg Nurs. 2021;56:100984.

Panovska-Griffiths J, et al. Exploring overcrowding trends in an inner city emergence department in the UK before and during COVID-19 epidemic. BMC Emerg Med. 2021;21(1):43.

Ghanbari V, et al. Emergency care utilization and patients’ outcome before and after COVID-19 national lockdown in Iran: a cross-sectional study. BMC Emerg Med. 2023;23(1):114.

Mitchell RD, et al. Impact of COVID-19 State of Emergency restrictions on presentations to two victorian emergency departments. Emerg Med Australasia. 2020;32(6):1027–33.

Buntine P, et al. Behavioural drivers influencing emergency department attendance in Victoria during the 2020 COVID-19 pandemic: a mixed methods investigation. Emerg Med Australasia. 2022;34(5):758–68.

Dempster P, et al. Australian emergency nurses’ experiences of working with personal protective equipment during the COVID-19 pandemic. A qualitative study. Australasian Emerg Care. 2024;27(1):63–70.

Asa GA, et al. The impact of working in COVID-19 hospital on Indonesian nurses’ mental health and wellbeing: a qualitative study. BMC Nurs. 2022;21(1):345.

Dixon E, Murphy M, Wynne R. A multidisciplinary, cross-sectional survey of burnout and wellbeing in emergency department staff during COVID-19. Australasian Emerg Care. 2022;25(3):247–52.

Horyniak D, et al. Pattern and characteristics of ecstasy and related drug (ERD) presentations at two hospital emergency departments, Melbourne, Australia, 2008–2010. Emerg Med J. 2014;31(4):317–22.

Chor WPD, et al. Burnout amongst emergency healthcare workers during the COVID-19 pandemic: a multi-center study. Am J Emerg Med. 2021;46:700.

Alanazi TNM, et al. Reported effects of the COVID-19 pandemic on the psychological status of emergency healthcare workers: a scoping review. Australasian Emerg Care. 2022;25(3):197–212.

Scott AM, et al. I was prepared to become infected as a frontline medical staff’: a survey of Australian emergency department staff experiences during COVID-19. Emerg Med Australasia. 2022;34(4):569–77.

Mercuri M, et al. Canadian emergency medicine physician burnout: a survey of Canadian emergency physicians during the second wave of the COVID-19 pandemic. Can J Emerg Med. 2022;24(3):288–92.

Mirzaei A, Mozaffari N, Habibi A, Soola. Occupational stress and its relationship with spiritual coping among emergency department nurses and emergency medical services staff. Int Emerg Nurs. 2022;62:101170.

Conlon C, et al. The impact of the COVID-19 pandemic on child health and the provision of Care in Paediatric Emergency departments: a qualitative study of frontline emergency care staff. BMC Health Serv Res. 2021;21(1):279.

Ahmadidarrehsima S, et al. Exploring the experiences of nurses caring for patients with COVID-19: a qualitative study in Iran. BMC Nurs. 2022;21(1):16.

Mohammadi F, et al. Management of COVID-19-related challenges faced by EMS personnel: a qualitative study. BMC Emerg Med. 2021;21(1):95.

Patterson JE, et al. Moral distress of medical family therapists and their physician colleagues during the transition to COVID-19. Journal of Marital and Family Therapy; 2021.

Download references

Acknowledgements

The author would like to thank Dr Yesim Yesil for her contribution to this study. The author would also like to acknowledge the participants and the hospital management teams for their help with participant recruitment.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and affiliations.

Faculty of Health Sciences, Mardin Artuklu University, Kampus yerleskesi, Artuklu/Mardin, Mardin, 47000, Turkey

Ahmet Butun

You can also search for this author in PubMed   Google Scholar

Contributions

A.B.: Conceptualization, Resources, Data curation, Software, Visualization, Methodology, Project administration, Formal analysis, Writing – original draft, Writing – review & editing.

Corresponding author

Correspondence to Ahmet Butun .

Ethics declarations

Ethics approval and consent to participate.

Ethical approval was obtained from Mardin Artuklu University Non-Interventional Clinical Research Ethics Committee (Date: 08/03/2022, REF: E-76272411-900-47908). The participants were informed about the aim of the study, and verbal consent was obtained from all participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Butun, A. Healthcare staff experiences on the impact of COVID-19 on emergency departments: a qualitative study. BMC Health Serv Res 24 , 921 (2024). https://doi.org/10.1186/s12913-024-11362-9

Download citation

Received : 27 June 2024

Accepted : 25 July 2024

Published : 12 August 2024

DOI : https://doi.org/10.1186/s12913-024-11362-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Emergency departments
• Emergency care services
• Healthcare staff
• Qualitative

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open access
• Published: 14 August 2024

Investigating the implementation challenges of the research doctoral program and providing related solutions: a qualitative study

• Alireza Koohpaei 1 ,
• Maryam Hoseini Abardeh 2 ,
• Shahnaz Sharifi 3 ,
• Majid Heydari 2 &
• Zeynab Foroughi 4  

BMC Medical Education volume  24 , Article number:  878 ( 2024 ) Cite this article

Metrics details

Doctoral programs have consistently garnered the attention of policymakers in medical education systems due to their significant impact on the socio-economic advancement of countries. Therefore, various doctoral programs have been implemented with diverse goals. In Iran, a research doctorate program, known as PhD by Research, was introduced primarily to engage in applied research related to healthcare needs. Nevertheless, the achievement of the program’s goals has been questioned. This study aimed to identify the implementation challenges of the Research Doctorate Program and its solutions in Iran.

This descriptive qualitative study followed the Standards for Reporting Qualitative Research: A Synthesis of Recommendations and was conducted in two steps. Firstly, the challenges of the Iranian Ph.D. by research program were identified through the perspectives of the program’s students and graduates. In the second step, relevant solutions to these challenges were determined by focus groups of key informant experts. The transcripts were analyzed using qualitative content analysis.

Five students and six graduates were interviewed in the first step and seven experts participated in the second one. The challenges and related solutions are explored in four main themes, including: (1) admission criteria, (2) program goals and expected outcomes, (3) curricula, and (4) financial and human resources. The study showed that various dimensions of the doctoral program are not aligned with each other and how to adapt the program in these dimensions.

The study revealed the importance of a systematic approach in defining various dimensions of doctoral programs according to program goals and provided specific solutions for defining a research doctorate program in the context of a low- and middle-income country.

Peer Review reports

Doctoral education plays a strategic role in national and regional economic, scientific, technological, and social development [ 1 ]. It lies at the heart of a university’s research capacity, which is also recognized as the primary source for research productivity and innovation in the global knowledge economy [ 2 ]. Hence, the significance of doctoral education captures the interest of policymakers at both international and national levels, as well as institutional leaders [ 3 , 4 ].

Over the past decades, doctoral education has witnessed a profound transformation [ 5 ] and takes various forms that can impact the quality and success of doctoral programs [ 6 ]. Doctoral programs offer students a study plan in their chosen field, which helps them gain a broad understanding of their discipline, develop expertise in the fundamental knowledge and methodologies, and acquire competencies to contribute to meaningful and practical scientific advancements [ 7 ]. Also, it prepares candidates for their various academic tasks [ 8 ].

Around the world, universities and medical education systems have established various types of doctoral programs tailored to their unique goals and requirements. Therefore, there is a wide range of doctoral programs. The most prevalent form of doctoral degree is the ‘Doctor of Philosophy’ or Ph.D., which signifies the recognition of students’ expertise in conducting research and contributing to generating novel knowledge [ 3 ]. In addition, the highest level of formal education belongs to the Doctor of Philosophy (Ph.D.) degree, because it equips individuals with the necessary knowledge and skills to push forward the boundaries of knowledge in a specific field [ 9 ]. Traditional Ph.D. programs typically center around dissertations. Additionally, there are also taught Ph.D. and Ph.D. by publication models, which respectively emphasize coursework and publications. Also, to enhance graduates’ preparation for the work environment, there are various types of work-based and professional doctoral programs [ 10 ]. The most important reasons for reforming traditional doctoral programs and creating diversity within them include: increasing the employment opportunities for graduates in the private sector [ 11 ], heightened focus on commercializing research outcomes [ 12 ], fostering competition and enhancing skills among graduates, facilitating a transition in career paths from academia to industry through collaborations between industry and universities [ 13 ], and aligning with market demands in the context of a competitive and dynamic knowledge-based economy [ 14 ].

Extensive research has been conducted on doctoral programs, resulting in a substantial amount of literature available. Some studies focused on students ‘experiences during the doctorate journey, because students go through an emotionally and intellectually demanding journey that encompasses a diverse range of both positive and negative experiences [ 15 ]. As well as, their live truly is a ‘constant juggling act’ and they may encounter different challenges and experiences that undergraduate may not come across [ 16 , 17 ]. From this perspective, Pyhältö and his et al. (2012) reported doctoral students’ problems which were related to supervision, the research community, domain specific, the general working process and resources [ 17 ]. Prendergast et al. studied the well-being of doctoral students [ 16 ].

Other studies are concentrated on the evaluation of doctoral programs. For example, Cross and Backhouse conducted a comprehensive investigation of the various limitations, obstacles, and possibilities within African doctoral education. They also proposed a framework for evaluating these programs which consisted of six elements including (1) expected outcomes, (2) candidates in context, (3) curriculum, (4) structures, (5) resources, and (6) funding, and partnership opportunities [ 18 ]. Meuleners et al. evaluated five aspects of the 82 life science doctoral programs in Germany, including (1) interdisciplinary, (2) the international orientation of these programs, (3) courses offered, (4) formal characteristics of supervision, and (5) examination regulations of the doctoral programs (6).

Assessment of research-doctorate programs have been conducted in different regions such as the United States [ 19 ] and Africa [ 20 ]. The University of Pennsylvania School of Nursing revised research-focused doctorate programs in October 2019. Some of the proposed changes involve enhancing the readiness of Ph.D. program graduates to connect research with practical applications, redesigning funding and support systems for students on an accelerated Ph.D. track, and developing ways to measure and evaluate the achievements of graduates [ 21 ].

In research-focused doctorate, it is crucial for doctoral students to gain a deep understanding of specific concepts in order to become independent researchers [ 22 ]. Studies in this area have demonstrated that traditional Ph.D. programs may not adequately provide graduates with the essential skills and knowledge they need [ 23 ]. To ensure the successful completion and achievement of doctoral graduates, it is important to consistently work towards developing doctoral programs that are adaptable to the learning needs of doctoral candidates and to overcome any barriers to desired outcomes [ 8 ].

In 2008, Iranian educational policymakers in the Ministry of Health and Medical Education (MoHME) made the decision to design a research-focused doctorate program (Ph.D. by research) to enhance the practicality of doctoral education and make a connection between doctoral education and job requirements. The purpose of this program was to educate candidates who can meet the needs of the country and expand the boundaries of knowledge by using advanced research methods and the latest research for problem solving [ 24 ]. This program consists of two parts, in the first part (M.Phil.), candidates learn research and technology theoretical and scientific skills, and in the second one, they should conduct a thesis and they are supported by a supervisory team which typically consists of two supervisors. The program was revised in 2013, 2014, and 2020. However, it appears that the program has not effectively achieved its intended goal. The evidence regarding the situation of graduates in the job market and their struggles in finding suitable employment confirms several obstacles within the program. Therefore, the aim of this study was to detect the implementation challenges of the Research Doctorate Programs from the students and graduates’ perspectives.

Materials and methods

This study was conducted according to the Standards for Reporting Qualitative Research: A Synthesis of Recommendations [ 25 ].

Study design

We applied a qualitative descriptive methodology to achieve an in-depth and rigorous description of the challenges of the research-focused doctorate program and relevant solutions. The study was conducted in two steps. Firstly, the challenges of the Iranian Ph.D. by research program were identified, and in the second step, relevant solutions to these challenges were determined.

Participant and sampling

Participants were selected based on their direct experience and knowledge of the Iranian Ph.D. by research program. Therefore, purposeful sampling was used to select participants, including students and graduates (P) from various fields in the doctoral program (first step). The purposeful sampling was of the maximum diversity type. This means that the students were selected from different fields so that the type of field does not lead to bias in available data. Also, information-rich experts were invited to participate in focus groups to propose solutions regarding the identified challenges (second step). In this step, experts (E) were selected from decision makers and policymakers in the doctorate program, medical education experts and researchers, professors and directors from academic institutions that conducted the program. In the first step, two participants were selected according to program records and the further participants were selected through snowball sampling technique. The interview guide and informed consent form were sent to potential research participants via email. If they agree, schedule the interview with them.

The inclusion criteria for the first step were students enrolled in a research doctorate program who were at least in their third year of study or had graduated from the program and had signed the informed consent form to participate in the research. The exclusion criteria included students who were below the third year of their study and those who did not wish to participate in the interview. For the second step, the inclusion criteria were decision-makers and policymakers in the doctorate program, medical education experts and researchers, faculty members, and directors from academic institutions who had been involved with the program for at least five years and had also signed the informed consent form to participate in the research. The exclusion criteria were experts who did not want to participate and did not have at least five years of experience with this program.

Data collection

For the first step, data collection was conducted through in-depth interviews with students and graduates (one in-depth interview with each participant). Data saturation determined the size of the study sample and the number of interviews. There are various models of saturation in qualitative studies. Saunders et al. identified four main saturation models including data saturation, a priori thematic saturation, Theoretical saturation and Inductive thematic saturation [ 26 ]. Data saturation implies on situation when data collection doesn’t provide any new data [ 27 , 28 ]. The interview guide was developed by conducting three pilot interviews. Transcripts of pilot interviews were included in the study analysis. The semi-structured interview was done face-to-face by MHA and ShSh and audio recorded with the participants’ permission. The interviews were transcribed verbatim from the audio recordings. The mean length of interviews was 45 min.

To addressing the identified challenges, we conducted semi-structured focus groups with experts. Data saturation was achieved by conducting five focus group sessions, each with an average of five participants. The team of facilitators included a discussion facilitator who motivated participants to engage in conversations with one another. The second one was responsible for taking notes and documenting the responses and memos. The third facilitator guided the focus group in answering the questions on the interview guide. Data was collected through audio recording and note-taking during the focus group sessions. The average duration of focus groups was 60 min. We have provided the study scripts in Supplementary files 1 & 2 .

Data analysis

The transcribed recorded in-depth and focus group interviews, as well as the notes of facilitators, were managed and organized using MAXQDA 20 software. The transcripts of in-depth interviews with students and graduates were analyzed conventionally. Accordingly, the transcripts were read word by word and key concept were highlighted where appropriate. In this step, three researchers independently analyzed the data, and the final codes, categories, and themes were discussed to achieve consensus. The analysis process includes repeatedly reading the transcripts, assigning meaning to each phrase, labeling the meaning units with codes, reviewing the codes, and organizing them into categories based on their similarities. Finally, the main themes are identified by interconnecting the categories.

In the second step, the focus group transcripts were analyzed using directed content analysis. In fact, the passages were coded using primary codes and categories from the first step.

Trustworthiness

This study describes the experience of conducting a doctoral program, including its challenges and solutions. Therefore, the study can provide guiding principles to consider when conducting any doctoral program. The credibility of study is confirmed by its adherence to the steps of the inductive content analysis method. Also, conformity was achieved by introducing the background of the researchers, who have various experiences and knowledge to analyze data from different perspectives. Additionally, the researchers confirmed the participants’ responses by transcribing the interviews and sharing the transcriptions with them. The interviewees confirmed that the transcripts contain their own words.

Description of participants

In the first step, out of the 15 individuals initially contacted, 11 agreed to participate and signed the consent form. Among the participants, five were actively enrolled in Ph.D. programs, while six had already graduated. Three participants self-identified as male (27%) and eight as female (73%). The backgrounds of the participants were illustrated in Table  1 . The shortest interview lasted 20 min, while the longest interview lasted 60 min. This phase was conducted from September 21, 2023, to December 10, 2024, at the research centers and their workplaces.

At the second stage, the invitation emails were sent to 10 experts and seven agreed to participate in this phase. The focus groups were conducted on January 2024, at the National Agency for Strategic Research in Medical Sciences Education.

Description of experts

Seven experts, including the program’s decision makers (2 participants, 28.5%), directors (2 participants, 28.5%), and medical education experts (3 participants, 43%) were emailed and recruited to discuss about the potential solutions in dealing with detected challenges (Table  2 ). Four experts were male (57%) and three as female (43%). The interview guide constitutes four main questions based on the detected challenges at the first step.

The authors concluded that data saturation had been achieved, indicating that additional interviews would not have resulted in new or distinctive findings.

The explored themes were related to: (1) unspecified admission criteria, (2) deviation from defined goals and expected outcomes, (3) ineffective curriculum to achieve program goals, (4) financial and human resources challenges. Detected themes, their classes and sub-classes are presented in Table  3 . As the focus groups were conducted based on the identified challenges in the first step, the solutions were categorized and presented within each theme as subcategories (Table  4 ).

Theme 1: unspecified admission criteria

Our analysis revealed some issues related to admission criteria, such as admission bias and special requirement.

1–1: admission bias

In many interviews selection based on supervisor ‘s preferences emerged: “ Since the acceptance (at the interview stage) is based on the supervisor’s opinion , the interest of the professors will play an important role in this process (P2). “Most centers choose candidates based on previous acquaintance with students. Personally , I was introduced to several centers based on my selection priorities , and later I found out that in the centers where I was not accepted , the accepted student had already been selected and the professor and student knew each other perfectly (P4)”.

1–2: special requirement

Our data illustrate that the specialized requirement of research institutes and the professional and occupational records of candidates in the specific field are not considered in admission process: “Most centers choose candidates based on previous acquaintance with students. Personally , I was introduced to several centers based on my selection priorities , and later I found out that in the centers where I was not accepted , the accepted student had already been selected and the professor and student knew each other perfectly (P4)”. “ In my opinion , that is better to admit candidates who have worked in the healthcare system for some time , they have known the problems of the system , and they can better solve system problems with their research projects (P6)”.

1–3: solutions

Adapting admission criteria based on program goals.

Experts emphasized the importance of redefining criteria for student admissions. According to their opinions, the criteria should be aligned with the institution’s mission and defined specific to program goal. In fact, students should be selected according to their potential to be a good fit for job in their expertise.

They reached a consensus on considering relevant work experience and published research in the field of study and alignment with the institution’s mission as effective criteria for achieving the objectives of the doctoral program. “ In fact , it is better that the students’ articles be related to the mission of the institution because it is effective in achieving the objective of conducting applied research and increasing the employability of the students (E1)”. “ The mission of the institution where the student is going to spend his/her education should be considered when choosing a student (E2)”.

Theme 2: deviation from defined goals and expected outcomes

This theme includes two classes (1) objectives unrelated to the program and (2) implementation barriers.

2–1: objectives unrelated to the program

This class includes two subclasses: 1) increase the ranks of the center,2) employment of graduates.

Candidates and graduates brought up how the goals and expected outcomes did change because the centers follow objectives which are not related to the goals and objectives of the program: “Many research centers accept Ph.D. students because they only want to increase the ranks of the center in the ranking systems , by implementing research projects that do not consider as the priority of the health system (P2)”. “ The goal of this initiative is to facilitate the employment of graduates in the job market , rather than solely focusing on training a few research doctoral students. (P7)”.

2–2: implementation barriers

This class is related to the providing working opportunities as an important goal of the program which are not reached because of various implementation barriers. Moreover, they acknowledge that the defined purposes and outcomes did not reach: “ No thought for recruitment after graduation. The decision makers should have thought about the working opportunities of the graduates , from the beginning (P5)”.

2–3: solutions

Clarifying students’ future duties and expectations during admission.

Regarding increasing commitment and adherence to the objectives of the institution and the field of study, it is also important for participants to be aware of the program goals, their duties, and the expectations placed on them during and after completing the program. “ At the beginning , we must clarify for the student what we want from her/him during the education , many times neither the student knows what we want from her nor we ourselves (E4)”.

Creating a robust control and evaluation system

Institutions should be continually monitored and evaluate regarding their adherence to the program goals. This requires the creation of a monitoring and evaluation system and the definition of indicators for successful performance in inputs, processes, and outputs. “ Research centers should admit students in a purposeful manner and their performance should be continuously evaluated and monitored by the Ministry of Health and Medical Education (E5)”.

Theme 3: ineffective curriculum to achieve program goals

This theme captured specific ideas and recommendations for the curriculum and includes two classes: (1) inefficient courses, (2) lack of priority setting.

3–1: inefficient courses

The non-applicable courses were emerged in this class. According to the results, the training methods and material of courses are not up to date and based on current relevant issues in field of studies: “ the lessons were not useful at all. We didn’t learn anything new in the general courses , which should have taught us about research , statistics , and epidemiology (P5)”.

3–2: lack of priority setting

Irrelevant lessons to fields priorities was proposed by the participants. Further, the thesis topics and research institutes’ priorities are not consistent: “ At least some theoretical courses should be customized for the scientific field of the student. All students pass shared courses in all research centers with different fields of activity (P1)”.

Curiously, most students suggested that the curriculum should be revised according to the candidates’ learning needs, current issues, and the competencies which they are required in their future jobs.

3–3: solutions

Aligning curriculums with program goals and structure.

Experts stated that the program structure and courses’ curriculums should be adjusted based on the fields of studies. “ Conducting need-based applied research requires students to have relevant professional skills and knowledge in their field of study (E3)”.

In addition, they believed that the program contents are needed to revise based on the program objectives. “ Currently , all students in different research centers study the same courses , while the needs of each center and field must be identified first , and then courses based on them should be defined (E6)”.

Theme 4: financial and human resources challenges

This theme consisted of two classes, (1) human resources problems and (2) financial issues.

4–1: human resources problems

Faculties are not able to prepare students for job market and conducting need-based researches. This might be due to the lack of sufficient faculty members in the educational system and their high workload which are stated by candidates. “ Supervisors need to dedicate more time to their students , but they are primarily focused on administrative tasks. (P1)”. In addition, faculty members have poor understanding of the program, have not sufficient practical experience in their field of expertise and they restrict candidates’ freedom of action. “ My supervisor did not have any learning program or research idea (P5)”. “ The supervisors turn the student into a task-fulfilling machine , and the student has no authority in any of the academic fields , including the courses and even the title of the thesis , and only says yes , sir! (P7)”. Many respondents mentioned unprepared faculty members as a challenge of the program. “ The professors themselves have not been well explained about the program and it seems that the professors are still not aware of the requirements of Ph.D. by research program (P3)”.

4–2: financial resources problems

Another aspect is the financial resources issues. Lack of financial support and failure in timely funding were defined as two subclasses.

Another aspect is the Lack of financial resources. This challenge is related to student perspective and suggestions about financial problems: “ Don’t talk about financial support! As much as the university gave a grant , I also spend additional cost for the thesis! (P5)”. “ Due to the high cost of the thesis , the payments were not made on time (P9)”. In addition, students noted the importance of timely funding in completion of their applied research: “ The professor admitted the student , then applied for a grant or research budget. It’s very late! (P5)”.

4–3: solutions

Providing additional supervisor with relevant practical experience.

Another important aspect of achieving the objective of conducting need-based applied research is to ensure that supervisors possess relevant practical experience and knowledge in the field of study. According to participants’ opinions, this achievement can be accomplished through collaboration between relevant academic institutions, health service providers, and product provision institutions in the introduction of supervisors. “ One important aspect to take into account in this program is the utilization of faculty members who have expertise in research and possess teaching relevant skills . (E4)”.

Clarity of duties and performance criteria

Lack of sufficient faculty members and their high workload necessitate managing them by standardizing and documenting their duties and clearly defining expectations. “ It is important to distribute students to supervisors based on their workload , such as assigning fewer students to professors with administrative responsibilities. (E5) ”.

Sustaining financial resources

Diversifying financial resources through collaboration with relevant public or private academic, health service, and product provision institutions was the main recommendation of experts to provide sustainable funding for the doctorate program. “ Faculty members should try to obtain national and international research grants such as World Health Organization grants (E7)”.

This study aimed to detect implementation challenges and relevant solutions of the research doctorate program in context of a low-middle income country from the perspectives of its beneficiary including students, graduates and key informants.

Based on the analysis of semi-structured interviews, four challenges were identifying, including unspecified admission criteria, deviation from defined goals and expected outcomes, ineffective curriculum to achieve program goals, financing and human resources.

Challenge 1: unspecified admission criteria

As Burford noted the doctoral admissions process is a subject of intense global discussion [ 29 ] and a wide range of admission criteria has been observed in doctoral programs which are encompass various aspects such as academic preparation, potentialities, attitudes, and competences [ 30 ]. Meanwhile, admission involves evaluative processes that are frequently unclear to those outside the system, but are considered routine by those within. In this regard professors play an important role as gatekeepers of the profession [ 31 ]. According to our findings, selection between applicants was based on supervisor ‘s preferences and previous acquaintance with applicants, and they were led to a decrease in the quality of research doctorate program. In addition, the lack of transparency in the terms and conditions for entering the program were reported by participants. These criteria should be clearly defined during the student recruitment process [ 32 , 33 ]. Therefore, admission criteria for research doctorate programs should be adjusted to ensure the admission of students with the necessary ability, motivation, and commitment to conduct problem-based research. It is essential to consider the diversity (geographical, racial, and ethnic) within the admitted groups.

In addition, having relevant work experience in the specialized field facilitates conducting applied research and enables teaching the course on a part-time basis. As well as ensuring the employability of students for related jobs is guaranteed [ 34 ].

Challenge 2: deviation from defined goals and expected outcomes

This issue emerged as the second challenge of the program. In Iran, the goal of establishing a research doctorate program is to maximize the benefits influenced by stakeholders and beneficiaries, including individuals, groups, parties, and institutions. Meanwhile, students and graduates of the program face some challenges as they are not trained according to the needs of research institutes. Additionally, they struggle to find suitable job positions and encounter issues related to academic-family integration which are consistence whit Rockinson-Szapkiw findings [ 35 ]. In general, the continuation of this process can lead to a lack of motivation among the beneficiaries of the research doctorate program, including professors and students. Urgent reforms should be implemented in this program. In accordance with our results, other researchers have also addressed this issue [ 8 , 36 , 37 ]. It is necessary to identify the potential success metrics of the doctoral program, collect information related to the results of each metric, and standardize them based on the reports provided by various higher education institutions [ 16 ].

Challenge 3: ineffective curriculum to achieve program goals

According to the results, students and graduates of research doctorate program in Iran are studying and working in ambiguous and ineffective conditions. The results of this research are in line with the results of studies by Anderson et al. [ 38 ], Keshmiri et al. [ 39 ], and Shin et al. [ 40 ], but there are differences in Iran. The main difference is that in research doctorate programs in Iran, special skills such as commercialization or other market skills are not included in the curriculum. There are no differences in terms of the designed and offered characteristics between research-oriented and education-oriented curriculums. Additionally, a significant aspect of the program is based on research. In fact, this program trains professional experts who are also researchers. Unlike the education-based doctorate, its goal is not to train researchers in a specific specialty. The various countries analyzed in this research follow two approaches: (1) Offering professional doctorate programs to managers, senior employees, and individuals with extensive experience, or (2) mandating a master’s degree, relevant work experience, and a concurrent affiliation with the relevant work environment [ 6 ].

As a result, the curriculum should primarily focus on new scientific topics, expanding current fields of knowledge, and the emergence of new fields that are influenced by economic, cultural, and technological conditions, as well as the provision of healthcare services and policies [ 41 ].

Challenge 4: financing and human resources

In relation to this problem, participants mentioned that they had various roles and responsibilities beyond those of a doctoral student, indicating that they are “more than just a doctoral student.”

They also expressed dissatisfaction with the low quality of student guidance programs and described mentorships as below average. In various countries, the standards of doctoral programs in medical sciences regarding mentoring activities are reviewed and presented in a consolidated format [ 42 ]. In this regard, the following principles are recommended: (1) Establish quality standards for student guidance activities (2). Create a guideline that supervisors and students can follow. Professors and students should be aware of the standards of student guidance activities. Additionally, providing incentives can enhance the productivity of the relationship between the supervisor and the students.

Students and candidates noted that their supervisors are busy and do not spend enough time on their duties as a supervisor. To address this issue, the following solutions are recommended based on expert feedback: (1) Establishing internal and external collaborations among various specialties and institutions, (2) Taking into account the professors’ workloads, (3) Sharing responsibilities and fostering participation, and (4) Providing flexibility in selecting supervisors.

Based on the study by Meuleners et al., it has been determined that assigning a single supervisor is usually not favourable for students. Instead, the use of a number of supervisors/mentors or a supervision team is recommended [ 6 ]. In this situation, it is possible to develop efficient projects based on the up-to-date needs of society. In Iran, although this possibility exists, the shortage of professors and various problems and challenges within academic groups prevent it. In the research- doctorate program, it is necessary for each student to have one or more senior researchers to guide, help, and support the student in developing their research skills. In fact, the vital role of authentic mentorship is to guide doctoral students through designing their career development plans, assisting in overcoming challenges in doctoral studies, and facilitating professional networking. This can lead to significant job opportunities not only during the doctoral program but also after graduation [ 43 ].

Financial resources also play a crucial role in the success of doctoral programs [ 15 ]. Based on our results, the limitation of financial resources for research doctorate education was another challenge. Therefore, it is recommended to develop a strategy on the best approach to ensure the resources required by the faculty. Utilizing the partnership method is an effective way to maximize resources through collaboration. Partnership is the process of collaborating with other institutions and individuals to achieve shared goals. Therefore, the partners share the same risks and benefits. The use of private financing programs can lead to increased initiatives in specialized doctoral education.

Based on our findings, it seems that in Iran, similar to East Asian countries, a hybrid system combining elements from the USA and European models has been utilized in designing research doctorate programs. This approach emphasizes both supervision and coursework components. On the one hand, this system reduces the level of creativity due to excessive supervision of students’ activities and emphasizes passing certain courses, thus limiting opportunities for defining problem-oriented projects. These conditions can be altered by transitioning to the European system and thoroughly evaluating the goals and anticipated results. Therefore, based on the results of this study, it is suggested to develop competency based curriculum or to reform the current program in order to solve its current problems. Future research is suggested to examine the practicality and effectiveness of the policy options proposed in the present study and prioritize them in terms of efficacy and effectiveness.

This study acknowledges a potential limitation in the alignment of proposed solutions with the actual challenges faced by students. While solutions are derived from experts’ interpretations of student-reported problems, there may be an inadvertent overlap of differing rationalities. This suggests a need for a more nuanced explanation of the contrasting perspectives between students and experts in the analysis. By analyzing the challenges raised by the students, the solutions proposed by experts, and reviewing similar studies in the discussion section, we aimed to elucidate this difference of opinion for the readers of the article.

This study proposes evidence-based solutions for a research doctorate program tailored to the specific context of Iran’s medical education system. Since the majority of researches on doctoral programs are grounded in Western perspectives on students, faculty, resources, and cultural contexts, this study has the potential to offer valuable insights and fresh perspectives.

The proposed framework is based on the outcome-based curriculum approach, which focuses on the essential competencies that students should achieve by the end of the program. The solutions consist of four main themes: admission criteria, goals and outcomes, curriculum, and resources, which aim to develop the technical and practical competencies of the students and graduates.

Research doctorate program graduates can play a vital role in improving the quality and performance of healthcare services by pursuing various career pathways and job categories that align with their skills and qualifications. However, to achieve this, they need to be supported by the MoHME, which should review and update the curriculum according to the program goals and international best practices. Additionally, redefining admission criteria, clarifying future duties, managing human and financial resources, and providing effective mentoring are essential. Moreover, graduates of research doctorate programs should collaborate with other health professionals, policymakers, and stakeholders to promote inter-professional collaboration and enhance integrated health system improvement.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Ministry of Health and Medical Education

Doctor of Philosophy

Sonesson A, Stenson L, Edgren G. Research and education form competing activity systems in externally funded doctoral education. Nordic J Stud Educational Policy. 2023:1–18.

Nerad M, Heggelund M. Toward a global PhD? Forces and forms in doctoral education worldwide. University of Washington; 2011.

Diogo S, Gonçalves A, Cardoso S, Carvalho T. Tales of doctoral students: motivations and expectations on the route to the unknown. Educ Sci. 2022;12(4):286.

Article   Google Scholar  

Zhuchkova S, Bekova S. Building a strong foundation: how pre-doctorate experience shapes doctoral student outcomes. PLoS ONE. 2023;18(9):e0291448.

Cardoso S, Santos S, Diogo S, Soares D, Carvalho T. The transformation of doctoral education: a systematic literature review. High Educ. 2022;84(4):885–908.

Meuleners JS, Boone WJ, Fischer MR, Neuhaus BJ, Eberle J, editors. Evaluation of structured doctoral training programs in German life sciences: how much do such programs address hurdles faced by doctoral candidates? Frontiers in Education. Frontiers; 2023.

Weaver TE, Lott S, McMullen P, Leaver CA, Zangaro G, Rosseter R. Research focused doctoral nursing education in the 21st century: curriculum, evaluation, and postdoctoral considerations. J Prof Nurs. 2023;44:38–53.

Craig W, Khan W, Rambharose S, Stassen W. The views and experiences of candidates and graduates from a South African emergency medicine doctoral programme. Afr J Emerg Med. 2023;13(2):78–85.

Nursing AAoCo. The research-focused doctoral program in nursing: pathways to excellence. 2020.

Gill TG, Hoppe U. The business professional doctorate as an informing channel: a survey and analysis. Int J Doctoral Stud. 2009;4(1):27–57.

Lee H-f, Miozzo M. How does working on university–industry collaborative projects affect science and engineering doctorates’ careers? Evidence from a UK research-based university. J Technol Transf. 2015;40:293–317.

Bienkowska D, Klofsten M. Creating entrepreneurial networks: academic entrepreneurship, mobility and collaboration during PhD education. High Educ. 2012;64:207–22.

Kitagawa F. Industrial doctorates: employer engagement in research and skills formation: LLAKES centre, Institute of Education; 2011.

Kehm BM. Doctoral education in Europe and North America: a comparative analysis. Wenner Gren Int Ser. 2006;83:67.

Google Scholar  

Corcelles M, Cano M, Liesa E, González-Ocampo G, Castelló M. Positive and negative experiences related to doctoral study conditions. High Educ Res Dev. 2019;38(5):922–39.

Prendergast A, Usher R, Hunt E. A constant juggling act—the daily life experiences and well-being of doctoral students. Educ Sci. 2023;13(9):916.

Pyhältö K, Toom A, Stubb J, Lonka K. Challenges of becoming a scholar: a study of doctoral students’ problems and well-being. International Scholarly Research Notices. 2012;2012.

Cross M, Backhouse J. Evaluating doctoral programmes in Africa: Context and practices. High Educ Policy. 2014;27:155–74.

Ostriker JP, Kuh CV, Voytuk JA. A data-based assessment of research-doctorate programs in the United States. National Academies; 2011.

Voytuk JA, Kuh CV, Ostriker JP, Council NR. Assessing research-doctorate programs: a methodology study. 2003.

Fairman JA, Giordano NA, McCauley K, Villarruel A. Invitational summit: re-envisioning research focused PHD programs of the future. J Prof Nurs. 2021;37(1):221–7.

Tyndall DE, Firnhaber GC, Kistler KB. An integrative review of threshold concepts in doctoral education: implications for PhD nursing programs. Nurse Educ Today. 2021;99:104786.

Leniston N, Coughlan J, Cusack T, Mountford N, editors. A practice perspective on doctoral education–employer, policy, and industry views. Proceedings of the International Conference on Education and New Developments; 2022.

Ph.D by Reseach Program, (2020).

O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245–51.

Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52:1893–907.

Francis JJ, Johnston M, Robertson C, Glidewell L, Entwistle V, Eccles MP, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25(10):1229–45.

Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18(1):59–82.

Burford J, Kier-Byfield S, Dangeni, Henderson EF, Akkad A. Pre-application doctoral communications: a missing dimension in research on doctoral admissions. Educational Rev. 2024:1–22.

Dobrowolska B, Chruściel P, Pilewska-Kozak A, Mianowana V, Monist M, Palese A. Doctoral programmes in the nursing discipline: a scoping review. BMC Nurs. 2021;20(1):1–24.

Posselt JR. Toward inclusive excellence in graduate education: constructing merit and diversity in PhD admissions. Am J Educ. 2014;120(4):481–514.

Doody S. Interdisciplinary writing should be simple, but it isn’t: a study of meta-genres in interdisciplinary life sciences doctoral programs. McGill University (Canada); 2020.

Cutri J. The third space: fostering intercultural communicative competence within doctoral education. Wellbeing in doctoral education: Insights and guidance from the student experience. 2019:265–79.

Fulton J, Kuit J, Sanders G, Smith P. The role of the professional doctorate in developing professional practice. J Nurs Adm Manag. 2012;20(1):130–9.

Rockinson-Szapkiw A. Toward understanding factors salient to doctoral students’ persistence: the development and preliminary validation of the doctoral academic-family integration inventory. Int J Doctoral Stud. 2019;14:237.

Horta H, Li H, Chan SJ. Why do students pursue a doctorate in the era of the ‘PhD crisis’? Evidence from Taiwan. High Educ Q. 2023.

Terentev E, Bekova S, Maloshonok N. Three challenges to Russian system of doctoral education: why only one out of ten doctoral students defends thesis? Int J Chin Educ. 2021;10(1):22125868211007016.

Anderson V, Gold J. The value of the research doctorate: a conceptual examination. Int J Manage Educ. 2019;17(3):100305.

Keshmiri F, Gandomkar R, Hejri SM, Mohammadi E, Mirzazadeh A. Developing a competency framework for health professions education at doctoral level: the first step toward a competency based education. Med Teach. 2019;41(11):1298–306.

Shin JC, Kim SJ, Kim E, Lim H. Doctoral students’ satisfaction in a research-focused Korean university: socio-environmental and motivational factors. Asia Pac Educ Rev. 2018;19:159–68.

Lusk MD, Marzilli C. Innovation with strengths: a collaborative approach to PhD/DNP integration in doctoral education. Nurs Educ Perspect. 2018;39(5):327–8.

Johnson O, Marus E, Adyanga AF, Ayiga N. The experiences and challenges of doctoral education in public universities compared. J Social Humanity Educ. 2023;3(3):237–52.

Al Makhamreh M, Stockley D. Mentorship and well-being: examining doctoral students’ lived experiences in doctoral supervision context. Int J Mentor Coaching Educ. 2020;9(1):1–20.

Download references

Acknowledgements

The authors extend their appreciation to the National Agency for Strategic Research in Medical Sciences Education (NASR) for funding this research work.

This project was funded by the National Agency for Strategic Research in Medical Sciences Education (NASR). Tehran. Iran. Grant NO. 4020154.

Author information

Authors and affiliations.

Occupational Health and Safety Department, Health Faculty, Qom University of Medical Sciences, Qom, Iran, Islamic Republic of

Alireza Koohpaei

National Agency for Strategic Research in Medical Sciences Education, Ministry of Health and Medical Education, Tehran, Iran, Islamic Republic of

Maryam Hoseini Abardeh & Majid Heydari

Critical Care Quality Improvement Research Center, Shahid Modarres Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran

Shahnaz Sharifi

Education Development Center, Iran University of Medical Sciences, Tehran, Iran, Islamic Republic of

Zeynab Foroughi

You can also search for this author in PubMed   Google Scholar

Contributions

A.K. conceived the study and contributed to the study design, data analysis, drafting, and finalizing of the paper. Z.F., M.H.A. contributed to the data analysis and drafted the paper. Sh. Sh. contributed to data gathering and data entry. M.H.A., A.K., and Z.F. contributed to the study design, interpretation of data and intellectual development of the manuscript as well as critically reviewed the manuscript. MH contributed in writing, critical review and editing of manuscript. All authors read and approved the final version of the paper.

Corresponding authors

Correspondence to Maryam Hoseini Abardeh or Zeynab Foroughi .

Ethics declarations

Ethical approval and consent to participate.

This study was approved by the Ethics Committee of the NASR (No: IR. NASRME. REC. 1402. 073). The transcriptions of participants and their related analysis were anonymized to ensure confidentiality. First, we explained in detail to the interviewees the study objectives. The interview guide and focus group questions were sent to prospective study participants, and their informed consent for participation in the study was obtained prior to their involvement. Following that, since the research presents no risk of harm to interviewees, we acquired verbal consent from the participants as approved by the ethics committee. However, consent was audio recorded, where we guaranteed interviewees their privacy, confidentiality, and anonymity of any information they may provide. Afterward, interviewees made a voluntary choice about participating in the research and were given the right to opt out of the interview as and when they wished.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Koohpaei, A., Abardeh, M.H., Sharifi, S. et al. Investigating the implementation challenges of the research doctoral program and providing related solutions: a qualitative study. BMC Med Educ 24 , 878 (2024). https://doi.org/10.1186/s12909-024-05815-2

Download citation

Received : 01 March 2024

Accepted : 24 July 2024

Published : 14 August 2024

DOI : https://doi.org/10.1186/s12909-024-05815-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research doctorate
• Doctoral program
• Policymaking
• Medical education

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 14.8.2024 in Vol 8 (2024)

A Chatbot (Juno) Prototype to Deploy a Behavioral Activation Intervention to Pregnant Women: Qualitative Evaluation Using a Multiple Case Study

Authors of this article:

Original Paper

• Elisa Mancinelli 1, 2 , BSc, MSc   ; 
• Simone Magnolini 3 , PhD   ; 
• Silvia Gabrielli 2 , PhD   ; 
• Silvia Salcuni 1 , PhD  

1 Department of Developmental and Socialization Psychology, University of Padova, Padova, Italy

2 Digital Health Lab, Centre for Digital Health and Wellbeing, Fondazione Bruno Kessler, Povo, Trento, Italy

3 Intelligent Digital Agents, Centre for Digital Health and Wellbeing, Fondazione Bruno Kessler, Povo, Trento, Italy

Corresponding Author:

Elisa Mancinelli, BSc, MSc

Department of Developmental and Socialization Psychology, University of Padova

Via Venezia 8

Padova, 35131

Phone: 39 3342799698

Email: [email protected]

Background: Despite the increasing focus on perinatal care, preventive digital interventions are still scarce. Furthermore, the literature suggests that the design and development of these interventions are mainly conducted through a top-down approach that limitedly accounts for direct end user perspectives.

Objective: Building from a previous co-design study, this study aimed to qualitatively evaluate pregnant women’s experiences with a chatbot (Juno) prototype designed to deploy a preventive behavioral activation intervention.

Methods: Using a multiple–case study design, the research aims to uncover similarities and differences in participants’ perceptions of the chatbot while also exploring women’s desires for improvement and technological advancements in chatbot-based interventions in perinatal mental health. Five pregnant women interacted weekly with the chatbot, operationalized in Telegram, following a 6-week intervention. Self-report questionnaires were administered at baseline and postintervention time points. About 10-14 days after concluding interactions with Juno, women participated in a semistructured interview focused on (1) their personal experience with Juno, (2) user experience and user engagement, and (3) their opinions on future technological advancements. Interview transcripts, comprising 15 questions, were qualitatively evaluated and compared. Finally, a text-mining analysis of transcripts was performed.

Results: Similarities and differences have emerged regarding women’s experiences with Juno, appreciating its esthetic but highlighting technical issues and desiring clearer guidance. They found the content useful and pertinent to pregnancy but differed on when they deemed it most helpful. Women expressed interest in receiving increasingly personalized responses and in future integration with existing health care systems for better support. Accordingly, they generally viewed Juno as an effective momentary support but emphasized the need for human interaction in mental health care, particularly if increasingly personalized. Further concerns included overreliance on chatbots when seeking psychological support and the importance of clearly educating users on the chatbot’s limitations.

Conclusions: Overall, the results highlighted both the positive aspects and the shortcomings of the chatbot-based intervention, providing insight into its refinement and future developments. However, women stressed the need to balance technological support with human interactions, particularly when the intervention involves beyond preventive mental health context, to favor a greater and more reliable monitoring.

Introduction

User-centered design of digital mental health interventions.

eHealth is a burgeoning field that integrates medical informatics, public health, and business. It encompasses delivering health services and information through the internet and digital technologies. In this domain, e-mental health specifically focuses on leveraging technologies, such as smartphone apps, websites, chatbots, and virtual reality, to enhance and support mental health care [ 1 - 3 ]. e-Mental health holds many advantages, including the increased scalability of mental services, in terms of screening, prevention, and treatment, leading to reduced costs for the broader health care system [ 4 - 6 ]. However, while the potential benefits of digital technology can be considerable, their actual implementation and use, especially within the field of e-mental health, often fall short. The journey from preuse considerations to initial adoption and, crucially, sustained use poses challenges that need careful navigation and understanding. In this regard, a recent review [ 7 ] exploring design methods and approaches for digital tools in mental health emphasized that human-centered design methods, thus those focusing on user experience (UX) rather than just engineering design, are not fully integrated into the field. The reported design approaches are predominantly external, lacking the perspective of the end users for whom the tool is intended. Indeed, when developing digital solutions, it is essential to consider 4 key components: the design issue and solution, the context in which the design occurs, the dynamics and organization of the design activity, and the actors contributing to the design [ 8 - 10 ]. Within the context of e-mental health intervention, the above altogether emphasizes the significance of co-design, a collaborative process strongly involving targeted end users to contribute to all stages of e-mental health intervention development. This inclusive approach encompasses needs assessment, content development, pilot-testing, and finally, dissemination [ 11 ]. The Obesity-Related Behavioral Intervention Trials (ORBIT) model [ 12 ] is instrumental to this end. The ORBIT model, which uses a user-centered design, provides a methodological framework encompassing a pliable and iterative progressive procedure, predefined clinically significant milestones for advancement, and the option to revert to a prior phase of refinement in case of suboptimal outcomes. Its primary emphasis is on pre-efficacy development and testing, yet not failing to incorporate subsequent research phases to illustrate that treatment optimization is viable even for interventions that have attained the efficacy or effectiveness stage [ 12 ].

e-Mental Health in Perinatal Care: A Focus on Prevention Interventions

The World Health Organization (WHO) [ 13 ] has consistently emphasized the significance of identifying and preventing risks, with the WHO and the United Nations Population Fund acknowledging maternal mental health as a pivotal factor in accomplishing the Millennium Development Goals [ 14 ]. The transition to motherhood involves various intrapersonal and interpersonal changes and challenges that can have negative effects on women’s mental health, increasing the risk of developing peripartum depression [ 15 - 17 ]. However, despite the negative repercussions this poses on the women, the child, and the whole family [ 18 ], as well as the broader society [ 19 - 21 ], it often goes untreated. There are various reasons for this. On the one hand, few women proactively seek professional assistance for their mental health problems, mainly due to factors such as lack of mental health literacy; stigma; and practical barriers like childcare, professional, and financial constraints [ 22 ]. By contrast, women face limited access to specialized perinatal mental health services, which is attributed to the capacity constraints of existing services and long waiting times for those in need of support [ 23 , 24 ]. Therefore, many women never receive any support or treatment. Indeed, this situation has sparked interest in the potential of e-mental health. It can circumvent some of the aforementioned barriers, ultimately facilitating a more widespread help-seeking process; this has led to the creation and dissemination of scalable and more far-reaching tools to support the well-being and mental health of perinatal women [ 25 , 26 ]. In this context, the stepped-care model is noteworthy, as its intentions are focused on promoting the dissemination of mental health programs by facilitating coordination between primary and secondary mental health services [ 27 ], and this coordination can be facilitated through e-mental health. This would ultimately align with the evidence that engaging in help-seeking behaviors increases the likelihood of perinatal women seeking further assistance for their depression symptoms [ 28 ]. In this regard, structured, evidence-based interventions such as behavioral activation (BA) might be particularly suitable. BA is a behavioral intervention designed to alleviate symptoms of depression [ 29 - 32 ] by offering individuals practical strategies to improve their adjustment and well-being and supporting participation in enjoyable and positive activities while reducing engagement in behaviors that worsen depressive symptoms [ 29 , 33 ]. As such, these interventions hold great potential as initial broad-case preventive work. However, when specifically focusing on peripartum depression, there appears to be a deficiency in digital prevention and treatment programs at large [ 34 ], and of BA interventions as well [ 35 ], in addressing depression symptoms during pregnancy compared with the postpartum period, thus underscoring the necessity to boost the development and evaluation of primary mental health services.

This Study: Within the Iterative Design Phase

This study arises from the results obtained by a previous exploratory co-design study [ 36 ] investigating the feasibility of an internet-based BA intervention for pregnant women showing subclinical symptoms of depression. As such, it constitutes the second phase of investigation within the “design phase” foreseen by the above-reported ORBIT model [ 12 ]. This prior exploratory study not only aimed to assess the initial feasibility of the intervention but also sought to gather valuable feedback directly from pregnant women. This then guided the adjustment of the intervention’s content and structure while promoting the use of a different digital solution. More specifically, the study aimed to compare a guided and unguided version of the digital intervention, with the guided group involving psychologists who engaged in weekly text message conversations with women to support them in the intervention content revision. In this respect, data suggested that the guided group showed greater adherence and were more willing overall to finish the intervention than the unguided group. Building on this and in line with the existing literature [ 37 , 38 ] highlighting the potential benefit of including chatbots within psychological interventions by fostering intervention adherence through increased engagement and involvement, a new structuring of the BA intervention as a chatbot-delivered intervention was prototyped. Chatbots are artificial intelligence–enabled engagement technologies, falling under the category of technologies that enable interaction with patients through natural language processing by engaging in limited text conversations intending to support subsequent behavior-change tasks [ 39 ]. It is crucial to emphasize that in this context, chatbots are conceptualized as tools suitable for educational purposes, facilitating the acquisition of specific evidence-based techniques or skills [ 40 ] resulting in suitability for application in preventive contexts.

Mindful of the above, this study aims to qualitatively evaluate, through a multiple case study, pregnant women’s experience and perception of a chatbot prototype to deploy a BA preventive support tool and intervention. In this regard, incorporating a dedicated prototype evaluation during co-design can streamline the process of conducting rigorous evaluations in real-world settings during the subsequent evaluative phases, which may involve activities such as pilot-testing and subsequent randomized controlled trials [ 41 ]. Furthermore, women’s desire for improvement and technological advancements of chatbot-based technology in the field of perinatal mental health was also investigated. As such, this study bounds the design and evaluation of the chatbot and prevention intervention it deploys within the ORBIT methodological framework [ 12 ], in favor of a thorough and meticulous evaluation of the intervention design phase regarding both definition and refinement. In line with this, a multiple–case study design is used as it permits the conduct of a comparative analysis of cases, aiming to identify both similarities and differences among them and, thus, in the perception of the chatbot and the content it deploys. In addition, this approach seeks to unveil patterns and themes that arise from the cross-case analysis. By evaluating the phenomenon of interest across different contexts, a multicase study might enhance the validity of findings by investigating in depth how the phenomenon may vary or remain consistent under various circumstances [ 42 ].

Ethical Considerations

Ethical considerations adhered to the guidelines outlined in the Declaration of Helsinki [ 43 ] and European data protection laws (EU GDPR 679/2016). Approval for the study was obtained from the Ethical Committee of the Psychology Department at the University of Padova (approval 5434/2023). Participants provided their informed consent to participation and data publication for scientific reasons.

Participants and Enrolment Procedure

Women aged >18 years and between the 12th and 30th week of gestation could take part in this study. Exclusion criteria were the following: clinically significant depression symptoms (Patient Health Questionnaire-9 [PHQ-9] [ 44 ] score≥15), suicidal ideation (PHQ-9 item 9), present or past history of psychiatric disorders, and experiencing an artificially induced pregnancy. To allow participation, a Google Form link containing the baseline questionnaires was shared through social media platforms (ie, Facebook and Instagram) in pregnancy-related national groups and pages. After the inclusion and exclusion criteria evaluation, women were provided with the information needed to start the interaction with the chatbot in Telegram and sent a copy of the informed consent they had agreed on that was reported within the web-based questionnaire. To uphold confidentiality, each participant was assigned a unique alphanumeric code. Women were granted the autonomy to withdraw from participation at any point without the obligation to provide reasons and without facing any adverse consequence. Furthermore, they were clearly informed that the software (ie, Telegram and the chatbot) did not constitute a medical device, as its use does not extend to the diagnosis, prevention, monitoring, prediction, prognosis, treatment, or alleviation of diseases. It was, instead, clarified that the developed support intervention and related software were exclusively intended for research purposes and used for the sole collection, storage, transmission of data and administration of questionnaires.

A total of 12 women completed the baseline questionnaire. Among them, 2 dropped out after the first interaction (week 1), 2 after completing the interaction in week 2, and 2 following the third interaction (week 3). One participant withdrew after completing the interaction in week 4. Among those who dropped out in the early weeks, 5 reported medical conditions: Crohn disease, risk of miscarriage associated with a shortened cervix and hypertonic pelvic floor, gestational diabetes, hypothyroidism, and fibroma. Ultimately, 5 participants were included in the multiple–case study evaluation, with none reporting any medical conditions. Of them, 4 (80%) participants reached the postintervention questionnaire evaluation, while 1 (20%) had to interrupt the interaction after week 4; however, she agreed to participate in the final semistructured interview. Given that this study aimed to qualitatively evaluate the perception and experience with a chatbot prototype, the decision was made to include this participant despite not finishing the study since she nonetheless was able to engage with the chatbot for more than half of the anticipated interactions.

The Intervention Content and Structuring

This study aligns with the iterative process outlined in the ORBIT model [ 12 ] for intervention design and evaluation. Specifically, it falls within the refined subphase of the initial design phase, in which practical aspects such as mode and agent of delivery, as well as the frequency and duration of contact, are evaluated to identify the most efficient ways to achieve clinical targets. Parallel to this, and in reference to the Digital Product Lifecycle, we care to emphasize that this project is at the beginning stages of the product life cycle, thus moving back and forth between the “definition phase” (in which the product or intervention concepts and related digital requirements are defined) and the “design phase” (which involves prototyping and pilot-testing the product) [ 45 , 46 ].

Accordingly, this study focuses on evaluating a revised version of an intervention based on an evidence-based BA intervention protocol (behavioral activation treatment for depression-revised) [ 47 ]. This revised intervention represents a second evaluation that builds upon exploratory testing conducted in a preceding co-design study [ 36 ]; as such, thorough information on the intervention content and rationality can be found in this prior study paper. However, commencing with the results from this latter study, in this study, the intervention was organized into 6 weekly sessions ( Figure 1 A), omitting the 3 additional ones previously included. The intervention content was streamlined by eliminating separate in-between–session homework. Instead, the essential components of the homework were incorporated within the main sessions or interactions as on-the-moment exercises strategically designed to promote the original intent of the homework. In this context, it is noteworthy that while the original protocol may not explicitly encompass a comprehensive functional analysis, several treatment components seamlessly aligned within such a framework and were further enhanced in the modified version of the intervention. This alignment is underscored by BA’s dual objectives of pinpointing factors that sustain or reinforce depressive behaviors (both positive and negative reinforcement) and identifying positive reinforcers that can support healthy behavioral patterns ( Figure 1 A). This process thus forms the basis for understanding the functional aspects of behavior, laying the groundwork for targeted strategies that can aid the person in autonomously addressing and modifying the maladaptive behavioral patterns effectively.

Moreover, there was a modification in the mode of delivering the intervention. Specifically, the content, previously structured as an e-learning course, was facilitated through a rule-based chatbot named Juno, operationalized within the Telegram platform with the sole purpose of delivering the intervention. As such, information was delivered through text messages, complemented by explanatory videos and images using the Telegram interface. The text messages were sent by Juno, which adhered to a preestablished protocol that had to be followed sequentially, enabling structured dialogues in which women engaged primarily by selecting the predefined buttons to navigate the conversation. Due to the rule-based nature of Juno, individualized feedback was not provided. In this regard, Figure 1 B depicts a simulation of the interaction between Juno and the user, showing how Juno responds and guides the user during the on-the-moment exercises (the reported example is an exercise conducted during week 2 reported in Figure 1 A “The bidirectional link between behavior and emotions”).

Moreover, by using the Telegram interface, participants can access multimedia resources such as videos and images in the multimedia section of the app. In addition, they could scroll back through the chat history to review past topics, although there was not a specific page summarizing the weekly intervention topics. This allowed participants to revisit and reinforce previous discussions as needed while maintaining the logical sequencing of the intervention. Conversations were structured to last around 10 minutes per session.

The Implementation of the Chatbot Juno in Telegram

The chatbot Juno was developed drawing inspiration from the methodology used in designing Motibot, a chatbot dedicated to providing psychosocial support to adults with diabetes mellitus [ 48 ]. Leveraging the capabilities of the Rasa open-source platform [ 49 ], which has been explicitly tailored for chatbot development and training, became viable owing to the domain-agnostic nature of Motibot’s core structure, which provided a remarkably flexible foundation. The Rasa platform seamlessly integrates advanced machine learning techniques and harnesses pretrained embeddings from language models. This integration empowers the construction of a chatbot finely tuned to a specific language. The synergy of machine learning techniques with crafted rules ensures a chatbot that is not only dynamic but also highly responsive. Within Juno, the pivotal role played by natural language understanding [ 50 ] became evident in interpreting user messages while considering the conversational history. A carefully defined set of variables facilitated a smooth transition between turns in the dialogue.

For instance, named entity recognition, a specific natural language understanding task, was used to interpret the intent “say your name” and identify the entity “user’s name.” Juno optimally used Telegram as its user interface, offering numerous advantages to users while streamlining the development process. In addition, Telegram’s built-in support for interactive tools, including buttons, links, and images, enhanced the overall UX. In this regard, enhancing UX involves using personalized interaction time frames. Juno, as part of its intervention process, prompted users at the end of the initial day to specify when they prefer follow-up contacts. This proactive approach assisted users in scheduling their intervention; Juno uses Rasa’s reminder interface to accomplish this task. However, potential server malfunctions can affect this tool. To mitigate such issues, Juno allows users to initiate the interaction (eg, by writing the message “Can we start?”) if the reminder date passes without any notification. Despite being a solution to a possible interaction problem, this approach should maintain a positive UX. Furthermore, it should be acknowledged that in this initial phase of development, the possibility that the chatbot could occasionally overlook an appointment was a possibility.

Moreover, in line with what was reported above, it is noteworthy that Juno follows an expert-written structured script to maintain focus on the intervention content and avoid deviating from the intended topics. Users can provide input by selecting predefined buttons or providing written responses, but they do not receive personalized feedback based on their input. If users attempt to engage with Juno outside the scope of the intervention, Juno informs them that it cannot respond to such queries and returns to the predefined interaction by starting from where they had left off.

With regard to data storage, no further development was required, as the native support of Rasa for storing interactions in a MongoDB database (ie, a universal time stamp) ensures both consistency and the archiving of users’ data (ie, log-in information, the time spent by each user interacting with Juno, etc).

Measurement Instruments

During the baseline assessment, women were asked the following demographic information: age, gestational week, if the pregnancy was physiological or induced through medically assisted techniques, marital status, educational level, category of occupation, living location, past and present psychiatric history, and presence of any medical condition (both pregnancy-related and not). Moreover, during the baseline assessment, participants completed questionnaires assessing psychological symptoms, levels of BA, and perceived environmental reward. The same questionnaires were administered at the end of the sixth week of interactions, facilitated by Juno in the Telegram Chat, for a postintervention evaluation; the UX and user engagement (UE) measures were included in the postintervention assessment.

Psychological Symptoms

Depression symptoms were evaluated through 2 unidimensional self-report tools: the PHQ-9 [ 44 , 51 ] and the Edinburgh Postnatal Depression Scale [ 52 , 53 ]. The PHQ-9 assesses the severity of depression symptoms over the past 2 weeks through 9 items measured on a 4-point Likert scale (0=“not at all”; 3=“almost every day”). Items align with the diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorders , Fourth Edition [ 54 ]. A score of ≤9 indicates mild or no depression symptoms, between 10 and 14 indicates moderate symptoms, and ≥15 indicates severe symptoms. Item 9 specifically assesses suicidal ideation. The Edinburgh Postnatal Depression Scale also assesses the severity of depression symptoms, yet on the previous week and more specifically in association with the perinatal period. It comprises 10 items measured on a 4-point Likert scale (0=“no, not at all”; 3=“yes, always”), with item 10 assessing suicidal ideation. Scores range between 0 to 30, and a score of ≥13 suggests probable depression. Anxiety symptoms were measured through the Generalized Anxiety Disorder-7 [ 51 , 55 ], a unidimensional self-report tool gauging the severity of these symptoms over the past 2 weeks through 7 items measured on a 4-point Likert scale (0=“never”; 3=“almost every day”). Scores range between 0 and 21; a score between 0 and 4 suggests minimal anxiety, between 5 and 9 mild anxiety symptoms, between 10 and 14 moderate anxiety symptoms, and ≥15 severe anxiety symptoms. Finally, stress symptoms were assessed through the Perceived Stress Scale-10 [ 56 , 57 ], a unidimensional self-report tool assessing stress symptoms over the past month using 10 items measured on a 4-point Likert scale (0=“never”; 3=“quite often”). Scores range between 10 and 40, with scores ranging from 0 to 13 suggesting lower stress levels, between 14 and 26 moderate stress levels, and ≥27 high perceived stress levels.

BA Measures

The BA for Depression Scale-Short Form [ 58 ] was used to measure changes in avoidance and activation during BA interventions for depression over the past week. It is a self-report featuring 9 items measured on a 7-point Likert scale (0=“not at all”; 6=“completely”), providing scores for BA, behavioral avoidance, and a total score ranging from 0 to 54. The Environmental Reward Observation Scale [ 59 ], a unidimensional self-report tool, was also used; it measures the level of environmental reward perceived in recent months through 10 items rated on a 4-point Likert scale (1=“strongly disagree”; 4=“strongly agree”). Scores range from 10 to 40.

The UX was evaluated through the Mobile Application Rating Scale [ 60 ], a self-report tool evaluating the quality of an app and its features. Comprising 23 items scored on a 5-point Likert scale (1=“poor”; 5=“excellent”), it assesses 4 dimensions of objective quality: engagement, functionality, esthetics, and information, along with a subjective quality scale. Only subscales related to “information,” “subjective app quality,” and “app-specific” (function) were considered for this study, totaling 17 items. UE was instead evaluated through the User Engagement Scale-Short Form [ 61 ], a short self-report tool assessing UE with a digital solution. With 12 items based on a 5-point Likert scale (1=“strongly disagree”; 5=“strongly agree”), it encompasses factors such as focused attention, perceived usability, esthetic attractiveness, and reward. Higher scores index a more positive evaluation.

Semistructured Interviews

Semistructured interviews, conducted by the first author between October and December 2023, featured 15 main questions tailored to the study. The semistructured interview comprehends 3 main blocks of questions and related probing questions: one focused on women’s personal experience with the intervention content (4 questions), another focused on their experience with the chatbot and the overall platforms (5 questions), and the last one inquired on opinions for future technological advancements (6 questions). Before asking the last block of questions, participants were provided with the definitions of digital intervention and technological advancement within the context of chatbot technologies, as reported in the Multimedia Appendix 1 . The interviews were conducted approximately 10 to 14 days after the participants had finished the interactions with the chatbot Juno; they were conducted either by phone call or through Google Meet, based on the participants’ preference. With the participants’ consent, the interviews were audio recorded for transcription and evaluation.

Data Analysis

All the analyses were computed with RStudio (RStudio IDE). Participants’ questionnaire scores were assessed, and score differences in psychological symptoms and activation levels, between preintervention and postintervention time points, were calculated by subtracting the postintervention scores from the baseline ones. Relying on the qualitative meaning of response points (particularly for the psychological symptoms, measured on a 4-point Likert scale), differences between time points were commented on when they differed by a minimum of +3 or –3 score points.

The semistructured interviews were evaluated in 2 different but complementary manners. First, a purely qualitative descriptive evaluation was conducted by extracting and evaluating the key points reported by each case in the related interview transcript for each question. Subsequently, a text-mining analysis was performed using the R package quanteda [ 62 ]. To this end, (1) the transcripts, written in Italian, were tokenized by using the specific quanteda function and converting uppercase letters into lowercase letters, removing numbers, punctuation, and stop words. Subsequently, (2) user responses were subdivided and grouped based on the question they referred to in separate .txt files. Finally, (3) recurrent words (ie, word stems) and their diagrams (ie, pairs of reoccurring word stems) were extracted; the former were considered recurrent if they appeared at least 3 times, while the latter if they appeared at least 2 times across interviews. The 3-occurrences criterion threshold was defined in line with past research [ 48 ]. In particular, the 3-occurrences criterion for including a stem was chosen based on the assumption that through this, an occurrence is expected to belong to the 5% most recurrent ones. This criterion resulted in the extraction of between 2.4% and 9.2% of the most recurrent stems (average 5.5%) for the different questions, reasonably complying with the assumed 5% threshold. In addition, for a given question, the average occurrence of stems was 1.3; thus, a 3-occurrences threshold was equivalent to the condition of a stem recurring with a frequency corresponding to more than twice the average occurrence.

Cases Presentation

Table 1 shows the participant’s demographic information. Figure 2 shows their scores regarding psychological symptoms, activity level, and environmental reward at baseline ( Figure 2 A) and the postintervention time point ( Figure 2 B), further plotting the difference between the 2 time points ( Figure 2 C). Of the 5 cases, participant E completed the interaction with Juno until (and including) week 4 because of a technical issue with the server provider of the chatbot (the update of the server’s public certificate resulted in a compromised connection between the Rasa server and Telegram, and despite efforts within the support time frame, communication restoration was unsuccessful). As such, her postintervention evaluation measurements are not available. It should also be noted that because of technical issues linked to the temporalization of the interactions, participant B skipped the interaction of week 2, participant A skipped the interaction of week 3 ( Figure 1 ), and participant D skipped the interaction of weeks 3 and 5. Furthermore, all participants had to autonomously prompt the interaction with Juno at least once.

Differences and Similarities Across Cases: Questionnaire Scores

Regarding the trend of change between the 2 time points, women showed comparable levels of psychological symptoms, BA, and environmental reward at baseline, which instead seem slightly different at the postintervention time point. More specifically, participant C stands out as the only one showing a reduction in all psychological symptom variables, with changes ranging from 3 to 4 score points. However, the levels of BA and environmental reward appear seemingly unchanged. By contrast, participant A seems to exhibit a peak in the reduction of stress symptoms and an increase in BA. Interestingly, participant B demonstrated a trend of increase in anxiety symptoms, alongside a trend of reduction in depression symptoms and a notable peak in increased BA.

In contrast, participant D appears to demonstrate a negative peak in BA (ie, a decrease), while the other dimensions seem unchanged. Notwithstanding, it should be stressed that at either time point, none of the participants reported clinically relevant symptoms in terms of depression, anxiety, and stress symptoms. Finally, Figure 3 plots the participants’ evaluation of UX and UE. Taken together, participant B provided, in all dimensions, the lowest UX and UE scores, while participant D had the highest scores. More specifically, all showed a quite high appreciation for the esthetic of the interactions (mean 4, SD 0) and a modest to high perceived usability of the chat (mean 3.67, SD 0.82), which although seems particularly true for participant D, while less so for participant B.

Furthermore, the latter reported a particularly low sense of absorption during the interaction, which was quite low also for participant C. This sense of absorption was instead moderate for participants A and D (mean 2.42, SD 1.17). These 2, together with participant C, also reported a moderate to quite high sense of reward from the interactions (mean 3.33, SD 0.72), instead lower for participant B. A comparable pattern emerged regarding UX-information (mean 4.08, SD 0.63); in addition, participant B, for whom the information was of modest quality, participants A, C, and D instead evaluated them as high-quality information in terms of credible sources, quantity, and clearness. An almost equal score distribution emerged for the app-specific function (ie, the app operation in terms of easy learning, logical flow, and gesture interaction design; mean 3.38, SD 0.75) and subjective quality (ie, the actual availability of using the app; mean 2.69, SD 1.01), with the latter being way lower.

Multimedia appendix 1 shows participants’ specific scores reported in Figures 2 and 3 .

Differences and Similarities Across Cases: Qualitative Evaluation of the Semistructured Interviews’ Answers and Text-Mining Results

A summary of the key concepts that emerged from the answers provided during the semistructured interviews is reported in Tables 2 - 4 , separately for each case. In this regard, it is worth noting the answers provided for q00 regarding the motivation for participation; only participant B reported a more personal motive linked to a desire to enrich her pregnancy experience. Differently, participants C, D, and E (participant E revealed during the semistructured interview that she is a perinatal psychologist and that she participated in the study because she was curious to experience firsthand the potentiality of digital tools in this context) were pushed by curiosity and a personal propensity to help with research. Finally, participant A reported that her curiosity was sparked by seeing one of the institutions that is part of this study.

a d/n: did not know what to answer.

a FAQ: frequently asked question.

b Not applicable.

Focusing on the text-mining analysis performed, the interview length ranged between 27.41 and 60.02 (mean 42.3, SD 12.63) minutes. After deleting the stop words, transcripts included a mean of 1732.8 (SD 795.78) words per participant. Overall, the aggregated results (text mining) are shown in Figures 4 - 6 . The nodes (ie, word stems) dimension illustrates the proportion of concept occurrences across transcripts for a specific question, all appearing at least 3 times. Word stems connected by arrows represent diagrams that have occurred at least twice. The word stems are translated after analysis for inclusion in the plots; therefore, the direction of the arrows reflects the Italian syntax.

Personal Experience With the Chatbot Juno

As for the text-mining aggregated results, the transcript of the interviews regarding participants’ personal experience with Juno included, after deleting the stop words, a mean of 67.56 (SD 28.39) words per question. Results are summarized for each question in Figure 4 . Instead, Table 2 summarizes, separately, the participants’ answers to each question. In this respect, all participants reported feeling negative regarding the technical problems encountered, except participant C, who felt indifferent to them. Noteworthy is that although all skipped at least 1 interaction, participant C is the sole one that had not skipped any interaction, while further reporting that she “knew that this is a research project,” thereby highlighting that she had foreseen some issues to occur. In line with this, the text-mining results highlighted the feelings of displeasure , untimely making the experience less impactful ( made it→less→impactful; q01).

Nonetheless, the experience (in terms of the content of the interactions) was liked and felt very interesting , in particular, the content of the exercises/questions (q02). However, participant A specified that she would have preferred if the latter were proposed in a more structured manner while also allowing for the possibility to continue practicing them in between the interactions to favor a sense of continuity. Furthermore, participant D felt that some of the questions (part of the exercises) were redundant. Participants B, C, and D instead stressed their appreciation for how the broader content was deployed in terms of videos and images, while participant E specifically appreciated how the messages were phrased. The overall content of the interactions, particularly the initial psychoeducation, was felt as pertinent and adequate. Coherently, the text-mining results highlighted that the interactions’ content was felt useful , allowing participants to take a moment to pay attention to themselves (q03). In this regard, they all reported that the content was pertinent to the pregnancy period ( period→pregnancy ) but could also be useful during the postpartum and the future in general , supporting them in asking for help ( ask→help ; q03 and q04) and in general favoring a self-awareness that can transversally be applied to life in favor of well-being. However, focusing on the subjective answers, while participant B felt that the content was suited for the beginning of the second trimester, participant C felt that such a period was already too late and that the support provided by the chatbot was better suited for the emotional tumult of the first trimester.

At last, all women felt the 6-week length of the intervention was adequate , although, given the length of the pregnancy period, they could have followed it even for a longer time (q05). This latter aspect was stressed by all those who had skipped at least 1 week of interaction and not by participant C, who had followed all 6.

UX With the Chatbot Juno in Telegram

The transcript of the interviews regarding participants’ UX with the chatbot Juno in Telegram included, after deleting the stop words, a mean of 69.65 (SD 29.95) words per question. Results are summarized for each question in Figure 5 , while Table 3 reports participants’ answers. With regard to women UX in interacting with Juno, as previously outlined, experiences were quite different, albeit the technical problems with the chatbot Juno have emerged as a matter to particularly account for (q08). In this regard, participant B pointed out the importance of providing clearer guidance, ideally beforehand, on how to autonomously deal with technical issues to help avoid feelings of confusion. Notwithstanding, all women showed appreciation for the esthetic of the material ( esthetic→material ) describing it as cute . Furthermore, it mostly brought the focus of the UX to the way Juno answered their inputs, highlighting the relevance of this aspect, thereby wishing for an increased personalization of the answers (q06). However, despite this, participant A perceived that because of the way messages, in general, were phrased and of the overall interaction flow, these made her at times “forget that there was not a person on the other side.” This is instead different from participant B’s perception, who considered the messages to be a bit sterile. In between these 2 polarities is instead the perception of participants D and E, the former describing them as “sufficiently spontaneous and realistic” and the latter further stressing that, although she felt properly guided by Juno, perceiving clearly that Juno was virtually created made her feel reassured. Coherently, when asked about their opinion on using a chatbot as a means to deploy psychological content (q07), participant A reported that the interactions’ limits (in terms of chatbot freedom) were both a limit and a strength. Nonetheless, overall, women felt that it could be an effective medium that can provide a kind of momentary containment ( type→containment ) and that it might work as a cue to subsequently reach for in- person support. Indeed, they felt that beyond its application in preventive contexts, a psychologist is needed ( go→psychologist→instead ; q07), and even in the context of this study, participant B felt the need for human contact at least by telephone call.

Finally, focusing on the app itself, women all agreed on the convenience ( convenient→app ) of Telegram as an interface, allowing them to avoid downloading another app and describing it as an optimal channel that they already knew and that is easy to use (q09).

Opinions on Future Technological Advancement

The transcript of the interviews’ answers regarding the participants’ desired technological advancement included, after deleting the stop words, a mean of 159.49 (SD 53.01) words per question. Results are summarized for each question in Figure 6 ; Table 4 reports the participants’ answers. Overall, when asked about opinions on future technical advancements, women’s answers were quite cohesive. In line with this, when asked about how they would image an ideal app in the context of perinatal care (q10), the greater focus was on the information content ( content→information ) related to what happens during pregnancy and in the different trimesters ( happens→trimester ) as well psychologically ( well-being→psychological ). It was also focused on the possibility of searching for this information and reading about it ( go→search ) freely.

Furthermore, they reported interest in having a chat with a chatbot within the app mainly to ask personal questions related to their personal experience ( linked to→experience→personal ). In this regard, focusing specifically on the potential technological advancements that could be foreseen for chatbots like Juno (q11), women showed a lack of interest in including vocal commands in terms of sending and receiving audios ( command→vocal→no ) and did not show a particular interest in personalizing the chatbot appearance ( personalization→appearance→chatbot ), albeit recognizing that others might. The sole exception was participant E; she reported voice commands as the first thing she would have liked to add, perceiving it as a way to optimize time. Regarding personalization , this aspect was again prominent among all women, stressing their desire to receive more personalized ( personal ) answers . However, albeit desired, such increased personalization and freedom of the chatbot also emerged as women’s main concern regarding the application of these tools in the mental health context (q12). As such, women reported the need to maintain human→monitoring . Indeed, worries were expressed regarding the kind of information the chatbot might give if unsupervised.

Furthermore, they expressed worries related to increased freedom and resemblance to human interactions, with the idea that this might lead to an overreliance on these tools. Indeed, they stressed the risk of these substituting interactions ( substitute→interaction ) with professionals and psychologists ( support→psychologist→risk→more→freedom→chatbot ), which was not desired. In line with this, participants believed that although a main advantage of these technological→tools is that they can be valuable in supporting psychological→well-being in preventive contexts ( preventive→terms ) or to satisfy specific needs without waiting to make an appointment , they cannot equate a therapeutic→intervention delivered in-person, particularly during pregnancy (q13). To deal with the concerns and risks reported, women agreed on the importance of underlining and reminding of what to expect from these tools ( meaning→tools ) and clearly stating their limits , thereby distinguishing the kind of support that can be received by a physical person versus a digital tool ( digital→person ). This would then also work as a disclaimer, thus preventing them from feeling disappointment when perceiving the limits of these tools (q14). In line with the above, women expressed a strong desire for an app that could be integrated within the health (care) →system, perceiving it as something that could create a shared space that facilitates interactions with gynecologists , thereby allowing the latter to account for women’s psychological well-being together with the medical aspects.

Principal Findings

This study aimed to use a multiple–case study design to evaluate and compare pregnant women’s experience and perception of Juno, a chatbot prototype to deploy a BA preventive support intervention; their opinions regarding desired improvements and technological advancements were also investigated. The insights gained from this study are valuable and in line with previous studies emphasizing the importance and essential nature of evaluating prototypes during the design stages of a digital tool and chatbot in particular [ 41 , 63 ]. Within this context, the adoption of a multicase study design [ 42 ] allowed us to gather valuable in-depth information on the similarities and differences in pregnant women’s perceptions, opinions, and desires while also evaluating the technical issues encountered and their impact on women’s experience [ 63 ].

Focusing on the implementation and operationalization of Juno in Telegram allowed women to benefit from the lack of installation requirements, experiencing an interface within a familiar environment; this is an advancement from the platform used in the previous study [ 36 ]. Instead, feedback regarding the materials’ esthetic and intervention content at large was again appreciated, and the content, in particular, was described as sound and useful. Women expressed specific appreciation for the exercises proposed by Juno as part of the BA intervention, assessing that they favored self-reflection. Differently from the previous study [ 36 ], most women would have liked for the intervention to be longer. This might be linked to the weeks of interaction skipped since only participant C who had completed all 6 interactions would have not lengthened the intervention. Another explanation could, instead, be linked to the change of platform and even more the new structuring of the exercises. Compared with the previous structuring [ 36 ], they have been changed so to be as short, simple, and effortless as possible, and as such, they were turned into on-the-moment reasoning exercises guided by Juno and no longer as in-between–session homework. This altogether seems to have been appreciated by women, except for 1 participant A, who instead stressed that she would have preferred to have the possibility to continue training them autonomously through practical exercises also in between the interactions. The desire for continuity became evident throughout her interview, suggesting that the digital tool was perceived as a companion to turn to when extra support was needed by providing a personal space to freely take care of herself. Participant E also highlighted this, considering it as something with added value, especially during the postpartum period, helping her take care of herself to then potentially better care for the newborn.

Mindful of the above, it is pivotal to remember that women in this study can be deemed “healthy,” as none of them present medical conditions, and all psychological symptom variables were below the clinical thresholds. Notwithstanding, it is worth noting that participant C, who besides having cleared the 6 interactions, also reported the highest symptomatic scores at baseline, showed the greatest and most consistent trend of reduction in symptom scores. However, despite some pretest-posttest changes in symptom variables scores, none of the participants ever reached clinical relevance, so they should be regarded as normal fluctuations in the state of well-being and ill-being occurring during pregnancy [ 64 ]. As such, the positive feedback on the meaning of the content and the exercises proposed is important since within a preventive context, the goal is not symptom reduction or resolution but to emphasize the awareness of psychosocial functioning and the intricate relationship between emotions and behaviors. This would ultimately allow the development of transversally applicable personal resources that can be applied across life situations, thus fostering adaptation capacities at large. Women here appeared to have perceived these benefits, acknowledging that the intervention content was relevant to the pregnancy period and could potentially be helpful during the postpartum and in the future. However, it is worth highlighting a difference in its perceived usefulness as a function of the pregnancy period during which they thought that the intervention should be deployed. Participant B considered it suitable for the early pregnancy period, around the beginning of the second trimester, while participant C suggested it was more beneficial for the emotional tumult of the first trimester. It is noteworthy that both women followed the intervention during the same gestational week. Such individual differences in the perception of need are though important in terms of motivation in following the preventive intervention and in the foreseen impact of the information received.

Notwithstanding these individual differences, consistent patterns across women’s feedback were identified, particularly when asking them how their ideal digital tool developed to provide psychological support during pregnancy should be. The first thing they stressed regarded the content; within a preventive context, all women felt the need for more holistic information in which the medical and the psychological aspects are integrated, helping them understand how the 2 influence each other to then receive guidance in understanding what is “normal” and what is not. They expressed a preference for this information to be readily available, giving them the freedom to access it as they preferred. This could indeed support their empowerment [ 26 ] and aligns with the evidence highlighting that engaging in help-seeking–related behaviors increases the likelihood of perinatal women seeking further assistance in the future if needed [ 28 ]. As for receiving support for their own subjective experience, women pointed to chatbots, seeing them as having the potential to provide a 24/7 means to answer their pregnancy-related questions. This was viewed in the context of containing worries without the need to wait or continually seek assistance for potentially smaller concerns. However, all women consistently emphasized that chatbots should not be intended to substitute in-person support or human relationships more broadly. They highlighted the importance of the perception of contact and vicinity for pregnant women. Furthermore, although recognizing the potential benefits in preventive contexts, in situations of greater need and increased psychological symptoms, women stressed that chatbots should always be accompanied by human monitoring.

Keeping this in mind, it is important to reason about the technical problems encountered with the chatbot Juno. Except for the sole woman who did not skip any interactions (participant C), all other women reported their dissatisfaction with the technical problems encountered. Their reactions varied from feeling that the intervention content became less impactful to experiencing frustration, disappointment, confusion, and a perception of loss of control. These aspects are particularly significant in a mental health context, even if preventive, since such perceptions might reduce the willingness to follow the intervention by hindering their sense of agency. Notably, participant B, who participated with the desire to enrich her pregnancy experience, reported higher and more personal expectations toward the intervention. Consistently, her scores on the UX and UE questionnaires and the results from the semistructured interview suggest that she felt the most negative about the technical problems encountered. In contraposition, participant C, who assessed that she knew that interactions with Juno were “part of a research project,” acknowledged the technical problems but remained indifferent. However, it is crucial to consider the reported feelings of confusion and loss of control. While desiring a more personalized chatbot to receive answers that are more in tune with their individual needs, participants themselves emphasized the importance of clear explanations and reminders about the chatbot’s capabilities and limitations as it becomes more sophisticated and autonomous. This is essential to prevent overreliance on the chatbot and to avoid potential disappointment and iatrogenic effects that could decrease the likelihood of seeking help in the future. When discussing the use of chatbots like Juno as a means to deliver psychological content, participants acknowledged its effectiveness in providing momentary containment and serving as an initial step before seeking further support. However, they also underscored the need for human psychologists or professionals in preventive contexts. Participant B, in particular, expressed a desire for some “human” contact, even by telephone while interacting with Juno. This resonates with a compelling argument made by Sedlakova and Trachsel [ 40 ]; they conducted an epistemic analysis of chatbots’ adoption within mental health or therapeutic contexts, prompting the need to carefully reason about how chatbots can be perceived. As such, in line with women’s desire for increased chatbot personalization but worries linked to its potential increased freedom, the authors [ 40 ] suggested balancing the number of humanlike characteristics and features of chatbots and that their application should be confined to specific functions.

Focusing on the broader real-life application of apps and chatbots in contexts such as the health care system, beyond being highly desired, they were seen as tools that could bridge between women and clinical professionals. Moreover, in line with the above, women reported that having such tools would make them feel like their psychological well-being was accounted for together with their physical well-being since the former is felt neglected. Another aspect that has emerged is that they could help favor self-monitoring and monitoring from the clinician; existing literature does indicate that tools of this nature are acceptable to perinatal women as a means of monitoring mood symptoms [ 65 ]. In this regard, the Interactive Centre of Perinatal Excellence developed by the Australian Centre of Perinatal Excellence [ 66 , 67 ] is noteworthy. It is an interactive digital screening app integrated into the health care system and designed to facilitate screening for perinatal depression and anxiety symptoms. It provides women with feedback on the screening results while generating related reports for the clinician. It can support the prevention of perinatal mental health disorders by empowering women, streamlining the screening process, and saving time and resources for both women and clinicians. In such a context, the inclusion of a tool like Juno within an app to “educate” and guide women through their pregnancy and postpartum while allowing for symptom monitoring might hold great potential; what if, within such an app, the preventive BA intervention deployed through Juno was proposed to women showing mild and/or moderate depression symptoms?

The literature highlights consistent prevalence metrics of depression symptoms throughout the whole pregnancy period (worldwide, 20.7%; Europe, 17.9% [ 68 ]; Italy, 6%-22% [ 69 - 72 ]), pointing at it as among the main predictors of postpartum depression [ 16 , 73 ], with repercussions on the quality of life of women [ 74 ] as well as on the child’s development and well-being [ 75 - 77 ]. These metrics highlight the necessity of collaborative efforts in designing and implementing tailored programs, particularly in primary prevention (to prevent symptoms before they start) and secondary prevention (targeting individuals at risk or with subclinical symptoms) [ 78 ]. This is especially crucial, given the unique characteristics of peripartum depression, referring to its direct association with the challenges and bodily changes inherent to the perinatal period [ 79 ] and stressing the need for tailored intervention programs. In this regard, taken together, our results suggest that Juno holds potential for apps in a preventive context, which is of value considering the paucity of preventive perinatal tools [ 34 ]. However, it has also emerged that within this context, a tool like Juno is not deemed as sufficient. In this regard, the comments made by participant A are emblematic; she felt that what Juno could give within the broader perinatal period was like “a drop in the ocean.” This is further exemplified by the dropout of all women with medical conditions, which suggests that in its current form, the intervention deployed through Juno would have limited application. As such, data indicate that its real-life adoption might be scarce if not inserted within a broader context that can better signify its value while allowing us to account for women’s differences in need, which influences the type and amount of use they would make of it. Furthermore, beyond ensuring a better functioning of the tool itself, a thorough action plan linked to problem resolution should be defined and provided to women (both in research and real-life contexts). However, the evaluation of these issues resonated with literature emphasizing the advantages of incorporating dedicated prototyping and implementation phases during the co-design of digital tools [ 41 ].

Study Limitations and Future Directions

Although the results’ generalizability cannot a priori be expected in this study design, women’s high educational level and residency in northern Italy still represent a limitation. Perinatal depression symptoms tend to be higher among women with lower educational levels [ 80 ], suggesting a potential bias in the sample. In addition, the mentioned sample’s characteristics may reduce the variability of analyzed cases, impacting the generalizability of the findings. A further limitation regards data collection, as it relied on self-reports and semistructured interviews, which are indeed vulnerable to social desirability biases. Moreover, in this study, women’s experience with depression symptoms and their use of e-mental health tools were not measured, thus representing a limit of the study. Nonetheless, assessing these matters could provide valuable insights into their perceptions and potential use of chatbots during pregnancy. These dimensions warrant consideration in future studies to better understand the factors influencing women’s engagement with digital interventions during this critical period.

Being at the beginning of the product life cycle [ 45 , 46 ], referring specifically to the technical problems encountered, while they represent a limitation in the study, their management by the research team was invaluable in providing insights into the software used and the potential of Rasa. This understanding contributes to a more flexible problem-solving approach for addressing current and potential future issues. Proactively addressing such problems helps users maintain a sense of control and proficiency with the tool. In addition, these issues offer important information on how problem resolution, or lack thereof, impacts the overall UX. In this regard, it is noteworthy that the users did not express dissatisfaction with the simplicity of the solution, which primarily operated on rule-based mechanisms. This emphasizes the significance of incorporating user-centered design principles in developing natural language processing solutions that effectively meet end users’ needs and expectations.

Conclusions

In line with this, good practices can be outlined to construct appropriate validity and mitigate any negative effects on the user, thus ensuring ethical standards [ 81 - 86 ]: (1) ground intervention content in evidence-based data pertinent to the perinatal literature; (2) seek input from both end users and clinical professionals to evaluate needs and gather feedback on intervention content and e-mental health tool usability; (3) conduct feasibility and pilot-testing to ensure acceptability, feasibility, efficacy, and effectiveness, along with evaluating e-mental health tool use (both frequency and duration); (4) use adequate measures and evaluate appropriate outcomes to assess intervention success; (5) ensure that end users are provided with a clear informed consent regarding intervention purpose, content, and e-mental health tool capacities, risks, and limits; (6) incorporate safety measures, including clear procedures for managing situations of need and heightened distress, such as providing crisis support services and establishing connections with reference clinicians or public health services, while also monitoring end user mental health; (7) continuously monitor intervention progress to refine effectiveness and minimize potential negative effects; and (8) ensure clinical professionals are properly guided and informed with up-to-date evidence on available e-mental health interventions, their effectiveness, suitability, and safety. Aligning with this, to ensure a consistently high-quality technical solution for end users, substantial investments in assistance and infrastructure are imperative. The insights from this study underscore the importance of prioritizing UX and technical reliability to enhance the effectiveness and adoption of preventive perinatal tools like Juno in real-world contexts. Although Juno already aligns with ethical standards 1 to 5, the results of this study indicate that the tool’s capacities, risks, and limitations need to be greatly reported (point 5). In addition, safety measures were limited to self-reported depression, anxiety, and stress levels, with no specific process for monitoring intervention progress (point 6). Therefore, future developments of Juno should incorporate comprehensive safety measures and test their feasibility and acceptability. This includes integrating technological requirements to establish a more specific procedure for monitoring intervention progress (point 7).

Conflicts of Interest

None declared.

Definitions of digital intervention and technological advancement provided to participants during the semistructured interviews and the participants’ scores.

• Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on e-mental health: a growing field in internet research. J Med Internet Res. Dec 19, 2010;12(5):e74. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Torous J, Bucci S, Bell IH, Kessing LV, Faurholt-Jepsen M, Whelan P, et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. Oct 2021;20(3):318-335. [ FREE Full text ] [ CrossRef ] [ Medline ]
• mHealth: new horizons for health through mobile technologies: second global survey on eHealth. World Health Organization. 2011. URL: https://iris.who.int/handle/10665/44607 [accessed 2024-04-29]
• Fairburn CG, Patel V. The impact of digital technology on psychological treatments and their dissemination. Behav Res Ther. Jan 2017;88:19-25. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim J, Marcusson-Clavertz D, Yoshiuchi K, Smyth JM. Potential benefits of integrating ecological momentary assessment data into mHealth care systems. Biopsychosoc Med. 2019;13:19. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wang Q, Su M, Zhang M, Li R. Integrating digital technologies and public health to fight COVID-19 pandemic: key technologies, applications, challenges and outlook of digital healthcare. Int J Environ Res Public Health. Jun 04, 2021;18(11):6053. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vial S, Boudhraâ S, Dumont M. Human-centered design approaches in digital mental health interventions: exploratory mapping review. JMIR Ment Health. Jun 07, 2022;9(6):e35591. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dorst K, Dijkhuis J. Comparing paradigms for describing design activity. Des Stud. Apr 1995;16(2):261-274. [ CrossRef ]
• Dorst K. Design beyond design. She Ji J Des Econ Innov. Feb 2019;5(2):117-127. [ CrossRef ]
• Dorst K. Design research: a revolution-waiting-to-happen. Des Stud. Jan 2008;29(1):4-11. [ CrossRef ]
• Boyd H, McKernon S, Mullin B, Old A. Improving healthcare through the use of co-design. N Z Med J. Jun 29, 2012;125(1357):76-87. [ Medline ]
• Czajkowski SM, Powell LH, Adler N, Naar-King S, Reynolds KD, Hunter CM, et al. From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. Oct 2015;34(10):971-982. [ FREE Full text ] [ CrossRef ] [ Medline ]
• WHO recommendations on maternal and newborn care for a positive postnatal experience. World Health Organization. URL: https://www.who.int/publications/i/item/9789240045989 [accessed 2024-04-29]
• The millennium development goals report 2013. WHO-UNFPA. URL: https://www.unfpa.org/publications/millennium-development-goals-report-2013 [accessed 2024-04-29]
• Sánchez-Polán M, Franco E, Silva-José C, Gil-Ares J, Pérez-Tejero J, Barakat R, et al. Exercise during pregnancy and prenatal depression: a systematic review and meta-analysis. Front Physiol. 2021;12:640024. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hutchens BF, Kearney J. Risk factors for postpartum depression: an umbrella review. J Midwifery Womens Health. Jan 22, 2020;65(1):96-108. [ CrossRef ] [ Medline ]
• Walker AL, de Rooij SR, Dimitrova MV, Witteveen AB, Verhoeven CJ, de Jonge A, et al. Psychosocial and peripartum determinants of postpartum depression: findings from a prospective population-based cohort. The ABCD study. Compr Psychiatry. Jul 2021;108:152239. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Howard LM, Khalifeh H. Perinatal mental health: a review of progress and challenges. World Psychiatry. Oct 2020;19(3):313-327. [ CrossRef ] [ Medline ]
• Clark M. Maternal depression costs society billions each year, new model finds. Center For Children and Families. 2019. URL: https:/​/ccf.​georgetown.edu/​2019/​05/​31/​maternal-depression-costs-society-billions-each-year-new-model-finds/​ [accessed 2024-04-29]
• Pollack LM, Chen J, Cox S, Luo F, Robbins CL, Tevendale HD, et al. Healthcare utilization and costs associated with perinatal depression among Medicaid enrollees. Am J Prev Med. Jun 2022;62(6):e333-e341. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rokicki S, McGovern M, Von Jaglinsky A, Reichman NE. Depression in the postpartum year and life course economic trajectories. Am J Prev Med. Feb 2022;62(2):165-173. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Button S, Thornton A, Lee S, Shakespeare J, Ayers S. Seeking help for perinatal psychological distress: a meta-synthesis of women’s experiences. Br J Gen Pract. Aug 28, 2017;67(663):e692-e699. [ CrossRef ]
• Fonseca A, Ganho-Ávila A, Lambregtse-van den Berg M, Lupattelli A, Rodriguez-Muñoz MF, Ferreira P, et al. Emerging issues and questions on peripartum depression prevention, diagnosis and treatment: a consensus report from the cost action riseup-PPD. J Affect Disord. Sep 01, 2020;274:167-173. [ CrossRef ] [ Medline ]
• Fonseca A, Silva S, Canavarro MC. Depression literacy and awareness of psychopathological symptoms during the perinatal period. J Obstet Gynecol Neonatal Nurs. Mar 2017;46(2):197-208. [ CrossRef ] [ Medline ]
• Hussain-Shamsy N, Shah A, Vigod SN, Zaheer J, Seto E. Mobile health for perinatal depression and anxiety: scoping review. J Med Internet Res. Apr 13, 2020;22(4):e17011. [ FREE Full text ] [ CrossRef ] [ Medline ]
• van den Heuvel JF, Groenhof TK, Veerbeek JH, van Solinge WW, Lely AT, Franx A, et al. eHealth as the next-generation perinatal care: an overview of the literature. J Med Internet Res. Jun 05, 2018;20(6):e202. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Richards DA. Stepped care: a method to deliver increased access to psychological therapies. Can J Psychiatry. Apr 01, 2012;57(4):210-215. [ CrossRef ] [ Medline ]
• Fonseca A, Gorayeb R, Canavarro MC. Women׳s help-seeking behaviours for depressive symptoms during the perinatal period: socio-demographic and clinical correlates and perceived barriers to seeking professional help. Midwifery. Dec 2015;31(12):1177-1185. [ CrossRef ] [ Medline ]
• Dimidjian S, Barrera M, Martell C, Muñoz RF, Lewinsohn PM. The origins and current status of behavioral activation treatments for depression. Annu Rev Clin Psychol. 2011;7:1-38. [ CrossRef ] [ Medline ]
• Dimidjian S, Goodman SH, Sherwood NE, Simon GE, Ludman E, Gallop R, et al. A pragmatic randomized clinical trial of behavioral activation for depressed pregnant women. J Consult Clin Psychol. Jan 2017;85(1):26-36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jacobson NS, Martell CR, Dimidjian S. Behavioral activation treatment for depression: returning to contextual roots. Clin Psychol Sci Pract. Feb 2001;8(3):255-270. [ CrossRef ]
• Soucy Chartier I, Provencher MD. Behavioural activation for depression: efficacy, effectiveness and dissemination. J Affect Disord. Mar 05, 2013;145(3):292-299. [ CrossRef ] [ Medline ]
• Kanter JW, Manos RC, Bowe WM, Baruch DE, Busch AM, Rusch LC. What is behavioral activation? a review of the empirical literature. Clin Psychol Rev. Aug 2010;30(6):608-620. [ CrossRef ] [ Medline ]
• Mancinelli E, Bassi G, Gabrielli S, Salcuni S. The efficacy of digital cognitive-behavioral interventions in supporting the psychological adjustment and sleep quality of pregnant women with sub-clinical symptoms: a systematic review and meta-analysis. Int J Environ Res Public Health. Aug 03, 2022;19(15):9549. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mancinelli E, Dell'Arciprete G, Pattarozzi D, Gabrielli S, Salcuni S. Digital behavioral activation interventions during the perinatal period: scoping review. JMIR Pediatr Parent. Feb 28, 2023;6:e40937. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mancinelli E, Gabrielli S, Salcuni S. A digital behavioral activation intervention (JuNEX) for pregnant women with sub-clinical depression symptoms: an explorative co-design study. JMIR Hum Factors. May 16, 2024;11:e50098. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Provoost S, Lau HM, Ruwaard J, Riper H. Embodied conversational agents in clinical psychology: a scoping review. J Med Internet Res. May 09, 2017;19(5):e151. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gaffney H, Mansell W, Tai S. Conversational agents in the treatment of mental health problems: mixed-method systematic review. JMIR Ment Health. Oct 18, 2019;6(10):e14166. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kellogg KC, Sadeh-Sharvit S. Pragmatic AI-augmentation in mental healthcare: key technologies, potential benefits, and real-world challenges and solutions for frontline clinicians. Front Psychiatry. Sep 6, 2022;13:990370. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sedlakova J, Trachsel M. Conversational artificial intelligence in psychotherapy: a new therapeutic tool or agent? Am J Bioeth. May 2023;23(5):4-13. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Noorbergen TJ, Adam MT, Teubner T, Collins CE. Using co-design in mobile health system development: a qualitative study with experts in co-design and mobile health system development. JMIR Mhealth Uhealth. Nov 10, 2021;9(11):e27896. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baxter P, Jack S. Qualitative case study methodology: study design and implementation for novice researchers. Qual Rep. Jan 14, 2015;13(4):544-559. [ CrossRef ]
• World Medical Association. Dichiarazione di helsinki della world medical association principi etici per la ricerca biomedica che coinvolge gli esseri umani. Evidence. 1964;5(10):e1000059. [ FREE Full text ]
• Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. Sep 2001;16(9):606-613. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ji X, Abdoli S. Challenges and opportunities in product life cycle management in the context of industry 4.0. Procedia CIRP. 2023;119:29-34. [ CrossRef ]
• Kiritsis D, Bufardi A, Xirouchakis P. Research issues on product lifecycle management and information tracking using smart embedded systems. Adv Eng Inform. Jul 2003;17(3-4):189-202. [ CrossRef ]
• Lejuez CW, Hopko DR, Acierno R, Daughters SB, Pagoto SL. Ten year revision of the brief behavioral activation treatment for depression: revised treatment manual. Behav Modif. Mar 2011;35(2):111-161. [ CrossRef ] [ Medline ]
• Bassi G, Giuliano C, Perinelli A, Forti S, Gabrielli S, Salcuni S. A virtual coach (Motibot) for supporting healthy coping strategies among adults with diabetes: proof-of-concept study. JMIR Hum Factors. Jan 21, 2022;9(1):e32211. [ FREE Full text ] [ CrossRef ] [ Medline ]
• The future of customer experience. RASA Technologies. URL: https://rasa.com/ [accessed 2023-11-30]
• Dinesh PM, Sujitha V, Salma C, Srijayapriya B. A review on natural language processing: back to basics. In: Raj JS, Iliyasu AM, Bestak R, Baig ZA, editors. Lecture Notes on Data Engineering and Communications Technologies. Singapore, Singapore. Springer; 2021:655-661.
• Shevlin M, Butter S, McBride O, Murphy J, Gibson-Miller J, Hartman TK, et al. Measurement invariance of the patient health questionnaire (phq-9) and generalized anxiety disorder scale (GAD-7) across four European countries during the COVID-19 pandemic. BMC Psychiatry. Mar 01, 2022;22(1):154. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Benvenuti P, Ferrara M, Niccolai C, Valoriani V, Cox JL. The Edinburgh postnatal depression scale: validation for an Italian sample. J Affect Disord. May 1999;53(2):137-141. [ CrossRef ] [ Medline ]
• Cox JL, Holden JM, Sagovsky R. Detection of postnatal depression. Development of the 10-item Edinburgh postnatal depression scale. Br J Psychiatry. Jun 2, 1987;150(06):782-786. [ CrossRef ] [ Medline ]
• American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th edition. New York, NY. American Psychiatric Press; 1994.
• Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [ CrossRef ] [ Medline ]
• Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. Dec 1983;24(4):385. [ CrossRef ]
• Mondo M, Sechi C, Cabras C. Psychometric evaluation of three versions of the Italian Perceived Stress Scale. Curr Psychol. Jan 08, 2019;40(4):1884-1892. [ CrossRef ]
• Manos RC, Kanter JW, Luo W. The behavioral activation for depression scale-short form: development and validation. Behav Ther. Dec 2011;42(4):726-739. [ CrossRef ] [ Medline ]
• Armento ME, Hopko DR. The Environmental Reward Observation Scale (EROS): development, validity, and reliability. Behav Ther. Jun 2007;38(2):107-119. [ CrossRef ] [ Medline ]
• Domnich A, Arata L, Amicizia D, Signori A, Patrick B, Stoyanov S, et al. Development and validation of the Italian version of the mobile application rating scale and its generalisability to apps targeting primary prevention. BMC Med Inform Decis Mak. Jul 07, 2016;16(1):83. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O’Brien HL, Cairns P, Hall M. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. Int J Hum Comput Interact. Apr 2018;112:28-39. [ CrossRef ]
• Benoit K, Watanabe K, Wang H, Nulty P, Obeng A, Müller S, et al. quanteda: an R package for the quantitative analysis of textual data. J Open Source Softw. 2020;3(30):774. [ FREE Full text ] [ CrossRef ]
• Yan SJ, Wadley G, Vo T, Xiao Y, Truong H. Design and evaluation of a prototype ‘hibaby’ for pregnancy assistance. In: Proceedings of the 2022 International Conference on Human Machine Interaction. 2022. Presented at: ICHMI '22; May 6-8, 2022:53-59; Beijing, China. URL: https://dl.acm.org/doi/abs/10.1145/3560470.3560477 [ CrossRef ]
• Bjelica A, Cetkovic N, Trninic-Pjevic A, Mladenovic-Segedi L. The phenomenon of pregnancy — a psychological view. Ginekol Pol. Feb 28, 2018;89(2):102-106. [ CrossRef ]
• Varma DS, Mualem M, Goodin A, Gurka KK, Wen TS, Gurka MJ, et al. Acceptability of an mHealth app for monitoring perinatal and postpartum mental health: qualitative study with women and providers. JMIR Form Res. Jun 07, 2023;7:e44500. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Blackmore R, Boyle JA, Gray KM, Willey S, Highet N, Gibson-Helm M. Introducing and integrating perinatal mental health screening: development of an equity-informed evidence-based approach. Health Expect. Oct 24, 2022;25(5):2287-2298. [ FREE Full text ] [ CrossRef ] [ Medline ]
• ICOPE: perinatal mental health digital screening. Center of Perinatal Excellence. URL: https://www.cope.org.au/health-professionals/icope-digital-screening/ [accessed 2024-04-29]
• Yin X, Sun N, Jiang N, Xu X, Gan Y, Zhang J, et al. Prevalence and associated factors of antenatal depression: systematic reviews and meta-analyses. Clin Psychol Rev. Feb 2021;83:101932. [ CrossRef ] [ Medline ]
• Agostini F, Neri E, Salvatori P, Dellabartola S, Bozicevic L, Monti F. Antenatal depressive symptoms associated with specific life events and sources of social support among Italian women. Matern Child Health J. May 11, 2015;19(5):1131-1141. [ CrossRef ] [ Medline ]
• Cena L, Mirabella F, Palumbo G, Gigantesco A, Trainini A, Stefana A. Prevalence of maternal antenatal and postnatal depression and their association with sociodemographic and socioeconomic factors: a multicentre study in Italy. J Affect Disord. Jan 15, 2021;279:217-221. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Corbani IE, Rucci P, Iapichino E, Quartieri Bollani M, Cauli G, Ceruti MR, et al. Comparing the prevalence and the risk profile for antenatal depressive symptoms across cultures. Int J Soc Psychiatry. Nov 14, 2017;63(7):622-631. [ CrossRef ] [ Medline ]
• Giardinelli L, Innocenti A, Benni L, Stefanini MC, Lino G, Lunardi C, et al. Depression and anxiety in perinatal period: prevalence and risk factors in an Italian sample. Arch Womens Ment Health. Feb 29, 2012;15(1):21-30. [ CrossRef ] [ Medline ]
• Caparros-Gonzalez RA, Romero-Gonzalez B, Strivens-Vilchez H, Gonzalez-Perez R, Martinez-Augustin O, Peralta-Ramirez MI. Hair cortisol levels, psychological stress and psychopathological symptoms as predictors of postpartum depression. PLoS One. Aug 28, 2017;12(8):e0182817. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lagadec N, Steinecker M, Kapassi A, Magnier AM, Chastang J, Robert S, et al. Factors influencing the quality of life of pregnant women: a systematic review. BMC Pregnancy Childbirth. Nov 23, 2018;18(1):455. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Betts KS, Williams GM, Najman JM, Alati R. The relationship between maternal depressive, anxious, and stress symptoms during pregnancy and adult offspring behavioral and emotional problems. Depress Anxiety. Feb 30, 2015;32(2):82-90. [ CrossRef ] [ Medline ]
• Field T, Diego M, Hernandez-Reif M, Figueiredo B, Deeds O, Ascencio A, et al. Comorbid depression and anxiety effects on pregnancy and neonatal outcome. Infant Behav Dev. Feb 2010;33(1):23-29. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Talge NM, Neal C, Glover V, Early Stress‚ Translational Research and Prevention Science Network: Fetal and Neonatal Experience on Child and Adolescent Mental Health. Antenatal maternal stress and long-term effects on child neurodevelopment: how and why? J Child Psychol Psychiatry. Mar 2007;48(3-4):245-261. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baker J. Three levels of health promotion/disease prevention. In: Baker J, editor. Contemporary Health Issues. Davis, CA. LibreTexts; 2020.
• Batt MM, Duffy KA, Novick AM, Metcalf CA, Epperson CN. Is postpartum depression different from depression occurring outside of the perinatal period? a review of the evidence. Focus (Am Psychiatr Publ). Apr 2020;18(2):106-119. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Papadopoulou SK, Pavlidou E, Dakanalis A, Antasouras G, Vorvolakos T, Mentzelou M, et al. Postpartum depression is associated with maternal sociodemographic and anthropometric characteristics, perinatal outcomes, breastfeeding practices, and mediterranean diet adherence. Nutrients. Sep 04, 2023;15(17):3853. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bächle TC, Wernick A. The Futures of eHealth, Social, Ethical and Legal Challenges. Berlin, Germany. Alexander von Humboldt Institute for Internet and Society; 2019.
• Balcombe L. AI chatbots in digital mental health. Informatics. Oct 27, 2023;10(4):82. [ CrossRef ]
• Jokinen A, Stolt M, Suhonen R. Ethical issues related to eHealth: an integrative review. Nurs Ethics. Mar 2021;28(2):253-271. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maeckelberghe E, Zdunek K, Marceglia S, Farsides B, Rigby M. The ethical challenges of personalized digital health. Front Med (Lausanne). Jun 19, 2023;10:1123863. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wykes T, Lipshitz J, Schueller SM. Towards the design of ethical standards related to digital mental health and all its applications. Curr Treat Options Psych. Jul 5, 2019;6(3):232-242. [ CrossRef ]
• Wykes T, Schueller S. Why reviewing apps is not enough: transparency for trust (T4T) principles of responsible health app marketplaces. J Med Internet Res. May 02, 2019;21(5):e12390. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 21.03.24; peer-reviewed by AR Yameogo, C-M Huang; comments to author 24.04.24; revised version received 14.05.24; accepted 17.06.24; published 14.08.24.

©Elisa Mancinelli, Simone Magnolini, Silvia Gabrielli, Silvia Salcuni. Originally published in JMIR Formative Research (https://formative.jmir.org), 14.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.