literature review is based on the assumption that

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved August 21, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

UW-Madison Libraries Teaching & Learning

Additional options.

• smartphone Call / Text
• voice_chat Consultation Appointment
• place Visit
• email Email

Chat with a Specific library

• Business Library Offline
• College Library (Undergraduate) Offline
• Ebling Library (Health Sciences) Offline
• Gender and Women's Studies Librarian Offline
• Information School Library (Information Studies) Offline
• Law Library (Law) Offline
• Memorial Library (Humanities & Social Sciences) Offline
• MERIT Library (Education) Offline
• Steenbock Library (Agricultural & Life Sciences, Engineering) Offline
• Ask a Librarian Hours & Policy
• Library Research Tutorials

Search the for Website expand_more Articles Find articles in journals, magazines, newspapers, and more Catalog Explore books, music, movies, and more Databases Locate databases by title and description Journals Find journal titles UWDC Discover digital collections, images, sound recordings, and more Website Find information on spaces, staff, services, and more

Language website search.

Find information on spaces, staff, and services.

• ASK a Librarian
• Library by Appointment
• Locations & Hours
• Resources by Subject

book Catalog Search

Search the physical and online collections at UW-Madison, UW System libraries, and the Wisconsin Historical Society.

• Available Online
• Print/Physical Items
• Limit to UW-Madison
• Advanced Search
• Browse by...

collections_bookmark Database Search

Find databases subscribed to by UW-Madison Libraries, searchable by title and description.

• Browse by Subject/Type
• Introductory Databases
• Top 10 Databases

article Journal Search

Find journal titles available online and in print.

• Browse by Subject / Title
• Citation Search

description Article Search

Find articles in journals, magazines, newspapers, and more.

• Scholarly (peer-reviewed)
• Open Access
• Library Databases

collections UW-Digital Collections Search

Discover digital objects and collections curated by the UW-Digital Collections Center .

• Browse Collections
• Browse UWDC Items
• University of Wisconsin–Madison
• Email/Calendar
• Google Apps
• Loans & Requests
• Poster Printing
• Account Details
• Archives and Special Collections Requests
• Library Room Reservations
• Lesson 1: What is a Literature Review

Literature Reviews in the Sciences

• Introduction
• Lesson 2: Fundamental Research Skills
• Lesson 3: Fundamental Writing Skills
• Lesson 4: Resources

What is a literature review? As stated earlier, a literature review is not a well-defined genre. Broadly speaking, a literature review is a single publication, or an element of a long publication, where a topic or research question is explored by examining a large body of prior publications relevant to that topic or research question. Literature review publications tend to self-identify as literature reviews by using those words in the title and in the abstract and body of the work. Literature review publications have extensive reference lists and a textual narrative that examines the content of the articles in that reference list. Article databases often allow you to limit results to literature reviews with the click of a button. Literature reviews are numerous and not difficult to locate.

Producing a literature review is a much different process than simply finding one to read. Literature reviews are diverse and dynamic because of the interrelationships among four major rhetorical features: type, form, purpose, and audience. Understanding these features can help you create an efficient research and writing plan, and ultimately craft a compelling final document. You may have observed these features and interrelationships among the review articles you’ve read.

Let’s delve into them.

Rhetorical Feature 1: Type

Literature reviews can be classified into many types based on characteristics like how the source material is found, the reason they are produced, and what they look like in their final form. New types continually evolve—a recent  piece of research identified 48 different literature review types !

Let’s look at three fundamental types of published literature reviews:

A narrative literature review is extremely common. You have probably read many literature reviews of this type. You may be tasked to produce a literature review of this type many times in your career. A narrative review uses past published research to tell a story about a topic. In the sciences, a narrative review can bring the reader up to date on an evolving research area, establish the scientific rationale for new research explorations, or provide a history of knowledge on a particular topic. They provide overviews or expert opinions grounded in prior published research on a topic. They are typically produced by a single author, or a small group of authors. A narrative literature review can be educational and informative for subject matter experts or non-experts.

What differentiates a narrative review from other review types is what is missing: a traditional narrative review typically does not follow a standardized method for locating the literature to be reviewed. Traditionally, the assumption is that the reader trusts the expertise of the author, and the review itself doesn’t necessarily follow a standard structure. The narrative review does not typically provide detailed information about how the prior research was located, or why some publications were chosen for inclusion over others. Narrative reviews can disclose their method for locating literature, and clarify the authors’ inclusion/exclusion criteria. This is generally helpful to the reader and adds credibility to the author’s words, but it is not a predominant element of the narrative review text. A traditional narrative review usually summarizes the take-home messages of the literature being reviewed, but doesn’t attempt to synthesize quantitative data from multiple studies into a single finding using statistical methods.

It is likely that most literature reviews you will write will be of the narrative review type.

A systematic review is a literature review that follows a highly specific protocol from start to finish. A systematic review of the literature intends to answer a specific research question. Instead of conducting laboratory or field research, a comprehensive search of all literature is conducted using a strict and reproducible search protocol that is designed ahead of time and made explicit in detail. The literature being searched can be published or unpublished.

Selection of articles to be included in the review also follows predetermined inclusion and exclusion criteria, and should involve multiple reviewers.  A systematic review is actually a research study that uses published literature as data.  Systematic reviews should never be undertaken alone, and like other forms of research, require team collaboration and significant time to conduct. The search and selection process for the literature to be reviewed is a significant portion of the systematic review text.

A narrative review can be conducted with methodological transparency, but that transparency alone does not transform it into a systematic review.

Meta-analysis

A meta-analysis is a sub-type of systematic reviews. A meta-analysis seeks to answer a specific research question using the literature as data. After the rigorous search and selection process is completed, the quantitative data in the individual studies selected for the review is extracted and synthesized (combined) statistically to achieve a result that has higher statistical power than the results of the individual studies. A meta-analysis is a literature review type as well as a research methodology.

The final activity of quantitative synthesis of data from multiple studies is the hallmark characteristic of a meta-analysis and is what transforms a systematic review into a meta-analysis. As a sub-type of systematic reviews, meta-analyses also require comprehensive review of the literature conducted with a specific protocol and adhering to methodological guidelines. Likewise, meta-analyses also investigate a specific research question. The search and selection process, as well as the data analysis, is a significant portion of the published meta-analysis. This type of literature review should also be conducted with a team, and will require significant time.

Questions to ask:

• What type of literature review do I need to write?
• Am I familiar with that type? What are its identifying characteristics?
• Are there guidelines for this type within my discipline that I can use to guide my workflow?

Rhetorical Feature 2: Form

As you might expect, the form that a literature review takes is closely related to its type. Literature reviews might be stand-alone publications that constitute a research study itself, part of a larger research project and publication, or part of a grant proposal.

Stand-alone review

Stand-alone reviews are frequently (but not always) themselves scholarly contributions to a field and published in a peer-reviewed journal. They may, either directly or indirectly, seek to move scholarly conversation and professional practice in a particular direction. Stand-alone reviews present the objective, methodology, and findings of their own internal research question.  Systematic reviews  and  meta-analyses  are most frequently published as stand-alone reviews.

Part of a larger research project

This may be the form that you are most familiar with. It tends to coincide with the  narrative review  type. Literature reviews are a requirement for theses and dissertations, as well as for most peer-reviewed articles (when the article is not itself a stand-alone review). In the case of theses and dissertations, the literature review frequently takes the form of a chapter with its own introduction, clearly defined sections, and conclusion. However, it can also take the form of a chapter section, typically as part of the introduction. In peer-reviewed articles, the literature review likewise takes the form of a section within the larger publication, typically as part of the introduction.

Annotated Article

Open Annotated Article in a new window

Part of a grant proposal

The literature review is a common feature in grant proposals as well, though it may not be as obvious as a dissertation chapter titled “Review of the Literature.” Frequently, a brief  narrative  literature review will be part of the (you guessed it!) “narrative” component of a grant proposal. (Check out  Lesson 4 of the Libraries’ micro-course on Grants & Funding  for more information about writing grant proposals.) In the case of research grants—like you might write for the NSF, NIH, USDA, or UW-Madison Graduate School—the literature review will most frequently occur within an “introduction” or “background” section of the project narrative.

• What form must my literature review take?
• Can I describe the general structure or organization typical of that form?
• What specific formal features have I noticed in examples from my field?

Rhetorical Feature 3: Purpose

Any single literature review often serves multiple purposes, so it will be important for you to understand what it is your literature review is meant to do on a few different levels. As you might guess, the purpose(s) of your literature review will strongly depend on its audience, type, and form.

There is one overriding purpose across nearly all literature reviews, however, which is to situate your research within relevant, ongoing conversations.

Situate your research within relevant, ongoing conversations

Imagine, for a moment, what it would feel like to read a scientific article without a narrative literature review incorporated into the text… You would probably have lots of questions! For instance, “What have other people researched in this area? What did they find? What were the implications, and how does this study support or complicate them? What were the limitations of past research, and how is this study’s approach different?”

The same questions could be asked of a dissertation or grant proposal without a literature review, and systematic reviews and meta-analyses simply wouldn’t exist! The most important thing your literature review needs to do is to summarize and synthesize existing research in a way that creates space for a new perspective—your perspective—to enter the conversation.

Depending on its type, form, and audience, your literature review might also seek to:

• Especially when a literature review is part of a thesis or dissertation project, one of its purposes may be to demonstrate your deep knowledge of the research area to an audience of specialists. The research project itself serves as a testament to your readiness to enter an industrial or scholarly profession, and the literature review specifically testifies to your knowledge of relevant research and your ability to synthesize that research into a coherent narrative.
• If your literature review is part of a grant proposal, it will likely need to communicate the broader impact of your research to an audience of non-specialists. Often, that impact will be framed within the context of real-world problems or challenges that your research aims to address. While your research project as a whole might propose changes in professional practice, public policy, or manufacturing products, the literature review can help you to establish the context for that change by providing an overview of what has and has not been done already.
• In the case of systematic reviews and meta-analyses, one major purpose must be to contribute new knowledge to the field through the review itself. Remember, these types of reviews are research studies themselves, whereas narrative literature reviews summarize and synthesize other research.
• What purpose(s) do I have in writing this literature review?
• What is my literature review supposed to accomplish?
• How is each purpose of my review related to the audience I’m writing for? To the type and form of the review?

Rhetorical Feature 4: Audience

Who will be engaging with your literature review? This element is closely tied to purpose and will impact choices you make while writing. Some audiences may be particularly interested in your literature review due to its type and form. The type and form of your review may also require particular writing conventions.

Specialists

What type of specialized knowledge or experience does your intended audience likely have? Is it likely that your intended audience has deep knowledge of your discipline and area of expertise? You may be able to avoid explanatory passages in your writing by assuming common knowledge or experiences. If your literature review is likely to challenge a status quo within a research area, you may need to devote more text to justifying the challenge. Also, readers who have specialized knowledge in your same area are likely to also know the literature well. Establishing and disclosing your inclusion/exclusion criteria, even in a narrative literature review, can help you respond to challenges.

For instance, suppose a senior faculty member, your advisor, or a peer-reviewer questions why you included certain publications in your review but did not include specified other articles. You may need to defend your choices, and having clearly defined inclusion/exclusion criteria already established will help you form your response to a challenge.

Non-specialists

We are defining a non-specialist as a person whose specialized knowledge is within a different scientific context than yours, or doesn’t run as deeply as yours. If you anticipate that your readership will include persons outside of your specific discipline, you may need to devote space to explaining the concepts and connections you are making so that their significance is readily apparent. For instance, if your literature review is a component of a grant application, the reviewers will likely have deep, specialized knowledge of their own, but not necessarily within your precise research area. It is important to not assume that they will automatically see how your literature review supports your funding need. A non-specialist may also not have a deep knowledge of the literature in your area and, as a critical reader, will want an understanding of your inclusion/exclusion criteria as well.

Questions to Ask:

• What audiences are likely to engage with my literature review?
• What audiences would I like to engage with my literature review?
• What are the concepts, nuances, methods, paradigms, intellectual history, or applications that I understand deeply when I engage with my specialty’s literature? Which of these is important to clarify so a non-specialist can engage purposefully with my literature review?

Rhetorical Features Review

Open Rhetorical Features Review in a new window

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

• What is the purpose of literature review? 
• a. Habitat Loss and Species Extinction: 
• b. Range Shifts and Phenological Changes: 
• c. Ocean Acidification and Coral Reefs: 
• d. Adaptive Strategies and Conservation Efforts: 

How to write a good literature review 

• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• How to write a literature review faster with Paperpal? 
• Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

1. Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 

2. Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 

Find academic papers related to your research topic faster. Try Research on Paperpal  

3. Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 

4. Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 

5. Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 

6. Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

Strengthen your literature review with factual insights. Try Research on Paperpal for free!    

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Write and Cite as you go with Paperpal Research. Start now for free.   

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

Whether you’re exploring a new research field or finding new angles to develop an existing topic, sifting through hundreds of papers can take more time than you have to spare. But what if you could find science-backed insights with verified citations in seconds? That’s the power of Paperpal’s new Research feature!  

How to write a literature review faster with Paperpal?

Paperpal, an AI writing assistant, integrates powerful academic search capabilities within its writing platform. With the Research feature, you get 100% factual insights, with citations backed by 250M+ verified research articles, directly within your writing interface with the option to save relevant references in your Citation Library. By eliminating the need to switch tabs to find answers to all your research questions, Paperpal saves time and helps you stay focused on your writing.   

Here’s how to use the Research feature:  

• Ask a question: Get started with a new document on paperpal.com. Click on the “Research” feature and type your question in plain English. Paperpal will scour over 250 million research articles, including conference papers and preprints, to provide you with accurate insights and citations. 
• Review and Save: Paperpal summarizes the information, while citing sources and listing relevant reads. You can quickly scan the results to identify relevant references and save these directly to your built-in citations library for later access. 
• Cite with Confidence: Paperpal makes it easy to incorporate relevant citations and references into your writing, ensuring your arguments are well-supported by credible sources. This translates to a polished, well-researched literature review. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a good literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. By combining effortless research with an easy citation process, Paperpal Research streamlines the literature review process and empowers you to write faster and with more confidence. Try Paperpal Research now and see for yourself.  

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• How Long Should a Chapter Be?
• How to Use Paperpal to Generate Emails & Cover Letters?

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, academic integrity vs academic dishonesty: types & examples, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , the ai revolution: authors’ role in upholding academic..., the future of academia: how ai tools are..., how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide).

How to Write a Literature Review

What is a literature review.

• What Is the Literature
• Writing the Review

A literature review is much more than an annotated bibliography or a list of separate reviews of articles and books. It is a critical, analytical summary and synthesis of the current knowledge of a topic. Thus it should compare and relate different theories, findings, etc, rather than just summarize them individually. In addition, it should have a particular focus or theme to organize the review. It does not have to be an exhaustive account of everything published on the topic, but it should discuss all the significant academic literature and other relevant sources important for that focus.

This is meant to be a general guide to writing a literature review: ways to structure one, what to include, how it supplements other research. For more specific help on writing a review, and especially for help on finding the literature to review, sign up for a Personal Research Session .

The specific organization of a literature review depends on the type and purpose of the review, as well as on the specific field or topic being reviewed. But in general, it is a relatively brief but thorough exploration of past and current work on a topic. Rather than a chronological listing of previous work, though, literature reviews are usually organized thematically, such as different theoretical approaches, methodologies, or specific issues or concepts involved in the topic. A thematic organization makes it much easier to examine contrasting perspectives, theoretical approaches, methodologies, findings, etc, and to analyze the strengths and weaknesses of, and point out any gaps in, previous research. And this is the heart of what a literature review is about. A literature review may offer new interpretations, theoretical approaches, or other ideas; if it is part of a research proposal or report it should demonstrate the relationship of the proposed or reported research to others' work; but whatever else it does, it must provide a critical overview of the current state of research efforts. 

Literature reviews are common and very important in the sciences and social sciences. They are less common and have a less important role in the humanities, but they do have a place, especially stand-alone reviews.

Types of Literature Reviews

There are different types of literature reviews, and different purposes for writing a review, but the most common are:

• Stand-alone literature review articles . These provide an overview and analysis of the current state of research on a topic or question. The goal is to evaluate and compare previous research on a topic to provide an analysis of what is currently known, and also to reveal controversies, weaknesses, and gaps in current work, thus pointing to directions for future research. You can find examples published in any number of academic journals, but there is a series of Annual Reviews of *Subject* which are specifically devoted to literature review articles. Writing a stand-alone review is often an effective way to get a good handle on a topic and to develop ideas for your own research program. For example, contrasting theoretical approaches or conflicting interpretations of findings can be the basis of your research project: can you find evidence supporting one interpretation against another, or can you propose an alternative interpretation that overcomes their limitations?
• Part of a research proposal . This could be a proposal for a PhD dissertation, a senior thesis, or a class project. It could also be a submission for a grant. The literature review, by pointing out the current issues and questions concerning a topic, is a crucial part of demonstrating how your proposed research will contribute to the field, and thus of convincing your thesis committee to allow you to pursue the topic of your interest or a funding agency to pay for your research efforts.
• Part of a research report . When you finish your research and write your thesis or paper to present your findings, it should include a literature review to provide the context to which your work is a contribution. Your report, in addition to detailing the methods, results, etc. of your research, should show how your work relates to others' work.

A literature review for a research report is often a revision of the review for a research proposal, which can be a revision of a stand-alone review. Each revision should be a fairly extensive revision. With the increased knowledge of and experience in the topic as you proceed, your understanding of the topic will increase. Thus, you will be in a better position to analyze and critique the literature. In addition, your focus will change as you proceed in your research. Some areas of the literature you initially reviewed will be marginal or irrelevant for your eventual research, and you will need to explore other areas more thoroughly. 

Examples of Literature Reviews

See the series of Annual Reviews of *Subject* which are specifically devoted to literature review articles to find many examples of stand-alone literature reviews in the biomedical, physical, and social sciences. 

Research report articles vary in how they are organized, but a common general structure is to have sections such as:

• Abstract - Brief summary of the contents of the article
• Introduction - A explanation of the purpose of the study, a statement of the research question(s) the study intends to address
• Literature review - A critical assessment of the work done so far on this topic, to show how the current study relates to what has already been done
• Methods - How the study was carried out (e.g. instruments or equipment, procedures, methods to gather and analyze data)
• Results - What was found in the course of the study
• Discussion - What do the results mean
• Conclusion - State the conclusions and implications of the results, and discuss how it relates to the work reviewed in the literature review; also, point to directions for further work in the area

Here are some articles that illustrate variations on this theme. There is no need to read the entire articles (unless the contents interest you); just quickly browse through to see the sections, and see how each section is introduced and what is contained in them.

The Determinants of Undergraduate Grade Point Average: The Relative Importance of Family Background, High School Resources, and Peer Group Effects , in The Journal of Human Resources , v. 34 no. 2 (Spring 1999), p. 268-293.

This article has a standard breakdown of sections:

• Introduction
• Literature Review
• Some discussion sections

First Encounters of the Bureaucratic Kind: Early Freshman Experiences with a Campus Bureaucracy , in The Journal of Higher Education , v. 67 no. 6 (Nov-Dec 1996), p. 660-691.

This one does not have a section specifically labeled as a "literature review" or "review of the literature," but the first few sections cite a long list of other sources discussing previous research in the area before the authors present their own study they are reporting.

• Next: What Is the Literature >>
• Last Updated: Jan 11, 2024 9:48 AM
• URL: https://libguides.wesleyan.edu/litreview

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Jul 15, 2024 10:34 AM
• URL: https://guides.lib.udel.edu/researchmethods

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jul 8, 2024 11:22 AM
• URL: https://libguides.northwestern.edu/literaturereviews

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

14 Writing a Literature Review: Introduction

Often the place where thesis students embark on their writing journey, the literature review is also the section where students can struggle the most.

What is a literature review?

A literature review provides an overview of the relevant work that has been done in a field and can be presented in a separate chapter in a traditional thesis, or as a section near the beginning of each publishable paper in a sandwich thesis.

The introduction chapter of a sandwich thesis may also contain a literature review that is broader in scope than those of the individual papers included.

The review of literature is a significant portion of most academic writing, as it orients both the researcher and the reader to what research has been done in an area to date (so as to avoid “re-inventing the wheel”), and also highlights gaps in current knowledge that the present research will seek to address. Literature reviews also indicate that the researcher is knowledgeable in their field, being an active member and participant in it. [1]

According to Kamler and Thomson (2006), there are six key tasks that must be accomplished by a literature review, as illustrated in the following poster. [2]

Why is writing the literature review so often a challenge?

The concept of “the literature review” itself is likely intimidating for graduate students, in part, because of the connotations the different parts of this title evoke when placed together.  First, “the” erroneously signifies that your literature review is one standalone thing, as opposed to something that is integrated and referred to throughout the entire thesis or article. A literature review should not simply be a separate piece completed at the beginning of the research/writing process and then only marginally edited at the end.

Second, the statement “the literature” also carries a potentially intimidating air about it. Literature signifies something of high culture and pretentious importance – impossible to attain, broad in scope, and just frustratingly out of reach.

Finally, “review” gives off the assumption that the reviewer is in the position of a passive audience member who looks on from the outside at all that is going on in the “literature” – not much activity or agency in this positioning.

Types of literature reviews

According to Arlene Fink (2014) in “Conducting Research Literature Reviews”, a literature review surveys books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by doing so, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have explored while researching a particular topic and to demonstrate to your readers how your research fits within a larger field of study.

The following table provides a comparison of the types of literature reviews you may encounter as a graduate student. [3]

Table 1: Types of literature reviews based on research approach

• Christine B. Feak and John M. Swales, Telling a Research Story: Writing a Literature Review (Michigan: The University of Michigan Press, 2009). ↵
• Barbara Kamler and Pat Thomson, Helping Doctoral Students Write: Pedagogies for Supervision (London: Routledge, 2006). ↵
• Maria Grant and Andrew Booth, “A typology of reviews: An analysis of 14 review types and associated methodologies,” Health Information and Libraries Journal 26.2 (2009): 91-108. ↵

McMaster University's Graduate Thesis Toolkit Copyright © 2021 by McMaster University is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

Share This Book

Literature Review: Types of Literature Reviews

• Literature Review
• Purpose of a Literature Review
• Work in Progress
• Compiling & Writing
• Books, Articles, & Web Pages

Types of Literature Reviews

• Departmental Differences
• Citation Styles & Plagiarism
• Know the Difference! Systematic Review vs. Literature Review

It is important to think of knowledge in a given field as consisting of three layers.

• First, there are the primary studies that researchers conduct and publish.
• Second, are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies.
• Third, there are the perceptions, conclusions, opinions, and interpretations that are shared informally that become part of the lore of the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomenon emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomenon. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147.

All content is from The Literature Review created by Dr. Robert Larabee USC

• << Previous: Books, Articles, & Web Pages
• Next: Departmental Differences >>
• Last Updated: Oct 19, 2023 12:07 PM
• URL: https://uscupstate.libguides.com/Literature_Review

Types of Literature Review

There are many types of literature review. The choice of a specific type depends on your research approach and design. The following types of literature review are the most popular in business studies:

Narrative literature review , also referred to as traditional literature review, critiques literature and summarizes the body of a literature. Narrative review also draws conclusions about the topic and identifies gaps or inconsistencies in a body of knowledge. You need to have a sufficiently focused research question to conduct a narrative literature review

Systematic literature review requires more rigorous and well-defined approach compared to most other types of literature review. Systematic literature review is comprehensive and details the timeframe within which the literature was selected. Systematic literature review can be divided into two categories: meta-analysis and meta-synthesis.

When you conduct meta-analysis you take findings from several studies on the same subject and analyze these using standardized statistical procedures. In meta-analysis patterns and relationships are detected and conclusions are drawn. Meta-analysis is associated with deductive research approach.

Meta-synthesis, on the other hand, is based on non-statistical techniques. This technique integrates, evaluates and interprets findings of multiple qualitative research studies. Meta-synthesis literature review is conducted usually when following inductive research approach.

Scoping literature review , as implied by its name is used to identify the scope or coverage of a body of literature on a given topic. It has been noted that “scoping reviews are useful for examining emerging evidence when it is still unclear what other, more specific questions can be posed and valuably addressed by a more precise systematic review.” [1] The main difference between systematic and scoping types of literature review is that, systematic literature review is conducted to find answer to more specific research questions, whereas scoping literature review is conducted to explore more general research question.

Argumentative literature review , as the name implies, examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. It should be noted that a potential for bias is a major shortcoming associated with argumentative literature review.

Integrative literature review reviews , critiques, and synthesizes secondary data about research topic in an integrated way such that new frameworks and perspectives on the topic are generated. If your research does not involve primary data collection and data analysis, then using integrative literature review will be your only option.

Theoretical literature review focuses on a pool of theory that has accumulated in regard to an issue, concept, theory, phenomena. Theoretical literature reviews play an instrumental role in establishing what theories already exist, the relationships between them, to what degree existing theories have been investigated, and to develop new hypotheses to be tested.

At the earlier parts of the literature review chapter, you need to specify the type of your literature review your chose and justify your choice. Your choice of a specific type of literature review should be based upon your research area, research problem and research methods.  Also, you can briefly discuss other most popular types of literature review mentioned above, to illustrate your awareness of them.

[1] Munn, A. et. al. (2018) “Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach” BMC Medical Research Methodology

  John Dudovskiy

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PLoS Comput Biol
• v.9(7); 2013 Jul

Ten Simple Rules for Writing a Literature Review

Marco pautasso.

1 Centre for Functional and Evolutionary Ecology (CEFE), CNRS, Montpellier, France

2 Centre for Biodiversity Synthesis and Analysis (CESAB), FRB, Aix-en-Provence, France

Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications [1] . For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively [2] . Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests [3] . Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read [4] . For such summaries to be useful, however, they need to be compiled in a professional way [5] .

When starting from scratch, reviewing the literature can require a titanic amount of work. That is why researchers who have spent their career working on a certain research issue are in a perfect position to review that literature. Some graduate schools are now offering courses in reviewing the literature, given that most research students start their project by producing an overview of what has already been done on their research issue [6] . However, it is likely that most scientists have not thought in detail about how to approach and carry out a literature review.

Reviewing the literature requires the ability to juggle multiple tasks, from finding and evaluating relevant material to synthesising information from various sources, from critical thinking to paraphrasing, evaluating, and citation skills [7] . In this contribution, I share ten simple rules I learned working on about 25 literature reviews as a PhD and postdoctoral student. Ideas and insights also come from discussions with coauthors and colleagues, as well as feedback from reviewers and editors.

Rule 1: Define a Topic and Audience

How to choose which topic to review? There are so many issues in contemporary science that you could spend a lifetime of attending conferences and reading the literature just pondering what to review. On the one hand, if you take several years to choose, several other people may have had the same idea in the meantime. On the other hand, only a well-considered topic is likely to lead to a brilliant literature review [8] . The topic must at least be:

• interesting to you (ideally, you should have come across a series of recent papers related to your line of work that call for a critical summary),
• an important aspect of the field (so that many readers will be interested in the review and there will be enough material to write it), and
• a well-defined issue (otherwise you could potentially include thousands of publications, which would make the review unhelpful).

Ideas for potential reviews may come from papers providing lists of key research questions to be answered [9] , but also from serendipitous moments during desultory reading and discussions. In addition to choosing your topic, you should also select a target audience. In many cases, the topic (e.g., web services in computational biology) will automatically define an audience (e.g., computational biologists), but that same topic may also be of interest to neighbouring fields (e.g., computer science, biology, etc.).

Rule 2: Search and Re-search the Literature

After having chosen your topic and audience, start by checking the literature and downloading relevant papers. Five pieces of advice here:

• keep track of the search items you use (so that your search can be replicated [10] ),
• keep a list of papers whose pdfs you cannot access immediately (so as to retrieve them later with alternative strategies),
• use a paper management system (e.g., Mendeley, Papers, Qiqqa, Sente),
• define early in the process some criteria for exclusion of irrelevant papers (these criteria can then be described in the review to help define its scope), and
• do not just look for research papers in the area you wish to review, but also seek previous reviews.

The chances are high that someone will already have published a literature review ( Figure 1 ), if not exactly on the issue you are planning to tackle, at least on a related topic. If there are already a few or several reviews of the literature on your issue, my advice is not to give up, but to carry on with your own literature review,

The bottom-right situation (many literature reviews but few research papers) is not just a theoretical situation; it applies, for example, to the study of the impacts of climate change on plant diseases, where there appear to be more literature reviews than research studies [33] .

• discussing in your review the approaches, limitations, and conclusions of past reviews,
• trying to find a new angle that has not been covered adequately in the previous reviews, and
• incorporating new material that has inevitably accumulated since their appearance.

When searching the literature for pertinent papers and reviews, the usual rules apply:

• be thorough,
• use different keywords and database sources (e.g., DBLP, Google Scholar, ISI Proceedings, JSTOR Search, Medline, Scopus, Web of Science), and
• look at who has cited past relevant papers and book chapters.

Rule 3: Take Notes While Reading

If you read the papers first, and only afterwards start writing the review, you will need a very good memory to remember who wrote what, and what your impressions and associations were while reading each single paper. My advice is, while reading, to start writing down interesting pieces of information, insights about how to organize the review, and thoughts on what to write. This way, by the time you have read the literature you selected, you will already have a rough draft of the review.

Of course, this draft will still need much rewriting, restructuring, and rethinking to obtain a text with a coherent argument [11] , but you will have avoided the danger posed by staring at a blank document. Be careful when taking notes to use quotation marks if you are provisionally copying verbatim from the literature. It is advisable then to reformulate such quotes with your own words in the final draft. It is important to be careful in noting the references already at this stage, so as to avoid misattributions. Using referencing software from the very beginning of your endeavour will save you time.

Rule 4: Choose the Type of Review You Wish to Write

After having taken notes while reading the literature, you will have a rough idea of the amount of material available for the review. This is probably a good time to decide whether to go for a mini- or a full review. Some journals are now favouring the publication of rather short reviews focusing on the last few years, with a limit on the number of words and citations. A mini-review is not necessarily a minor review: it may well attract more attention from busy readers, although it will inevitably simplify some issues and leave out some relevant material due to space limitations. A full review will have the advantage of more freedom to cover in detail the complexities of a particular scientific development, but may then be left in the pile of the very important papers “to be read” by readers with little time to spare for major monographs.

There is probably a continuum between mini- and full reviews. The same point applies to the dichotomy of descriptive vs. integrative reviews. While descriptive reviews focus on the methodology, findings, and interpretation of each reviewed study, integrative reviews attempt to find common ideas and concepts from the reviewed material [12] . A similar distinction exists between narrative and systematic reviews: while narrative reviews are qualitative, systematic reviews attempt to test a hypothesis based on the published evidence, which is gathered using a predefined protocol to reduce bias [13] , [14] . When systematic reviews analyse quantitative results in a quantitative way, they become meta-analyses. The choice between different review types will have to be made on a case-by-case basis, depending not just on the nature of the material found and the preferences of the target journal(s), but also on the time available to write the review and the number of coauthors [15] .

Rule 5: Keep the Review Focused, but Make It of Broad Interest

Whether your plan is to write a mini- or a full review, it is good advice to keep it focused 16 , 17 . Including material just for the sake of it can easily lead to reviews that are trying to do too many things at once. The need to keep a review focused can be problematic for interdisciplinary reviews, where the aim is to bridge the gap between fields [18] . If you are writing a review on, for example, how epidemiological approaches are used in modelling the spread of ideas, you may be inclined to include material from both parent fields, epidemiology and the study of cultural diffusion. This may be necessary to some extent, but in this case a focused review would only deal in detail with those studies at the interface between epidemiology and the spread of ideas.

While focus is an important feature of a successful review, this requirement has to be balanced with the need to make the review relevant to a broad audience. This square may be circled by discussing the wider implications of the reviewed topic for other disciplines.

Rule 6: Be Critical and Consistent

Reviewing the literature is not stamp collecting. A good review does not just summarize the literature, but discusses it critically, identifies methodological problems, and points out research gaps [19] . After having read a review of the literature, a reader should have a rough idea of:

• the major achievements in the reviewed field,
• the main areas of debate, and
• the outstanding research questions.

It is challenging to achieve a successful review on all these fronts. A solution can be to involve a set of complementary coauthors: some people are excellent at mapping what has been achieved, some others are very good at identifying dark clouds on the horizon, and some have instead a knack at predicting where solutions are going to come from. If your journal club has exactly this sort of team, then you should definitely write a review of the literature! In addition to critical thinking, a literature review needs consistency, for example in the choice of passive vs. active voice and present vs. past tense.

Rule 7: Find a Logical Structure

Like a well-baked cake, a good review has a number of telling features: it is worth the reader's time, timely, systematic, well written, focused, and critical. It also needs a good structure. With reviews, the usual subdivision of research papers into introduction, methods, results, and discussion does not work or is rarely used. However, a general introduction of the context and, toward the end, a recapitulation of the main points covered and take-home messages make sense also in the case of reviews. For systematic reviews, there is a trend towards including information about how the literature was searched (database, keywords, time limits) [20] .

How can you organize the flow of the main body of the review so that the reader will be drawn into and guided through it? It is generally helpful to draw a conceptual scheme of the review, e.g., with mind-mapping techniques. Such diagrams can help recognize a logical way to order and link the various sections of a review [21] . This is the case not just at the writing stage, but also for readers if the diagram is included in the review as a figure. A careful selection of diagrams and figures relevant to the reviewed topic can be very helpful to structure the text too [22] .

Rule 8: Make Use of Feedback

Reviews of the literature are normally peer-reviewed in the same way as research papers, and rightly so [23] . As a rule, incorporating feedback from reviewers greatly helps improve a review draft. Having read the review with a fresh mind, reviewers may spot inaccuracies, inconsistencies, and ambiguities that had not been noticed by the writers due to rereading the typescript too many times. It is however advisable to reread the draft one more time before submission, as a last-minute correction of typos, leaps, and muddled sentences may enable the reviewers to focus on providing advice on the content rather than the form.

Feedback is vital to writing a good review, and should be sought from a variety of colleagues, so as to obtain a diversity of views on the draft. This may lead in some cases to conflicting views on the merits of the paper, and on how to improve it, but such a situation is better than the absence of feedback. A diversity of feedback perspectives on a literature review can help identify where the consensus view stands in the landscape of the current scientific understanding of an issue [24] .

Rule 9: Include Your Own Relevant Research, but Be Objective

In many cases, reviewers of the literature will have published studies relevant to the review they are writing. This could create a conflict of interest: how can reviewers report objectively on their own work [25] ? Some scientists may be overly enthusiastic about what they have published, and thus risk giving too much importance to their own findings in the review. However, bias could also occur in the other direction: some scientists may be unduly dismissive of their own achievements, so that they will tend to downplay their contribution (if any) to a field when reviewing it.

In general, a review of the literature should neither be a public relations brochure nor an exercise in competitive self-denial. If a reviewer is up to the job of producing a well-organized and methodical review, which flows well and provides a service to the readership, then it should be possible to be objective in reviewing one's own relevant findings. In reviews written by multiple authors, this may be achieved by assigning the review of the results of a coauthor to different coauthors.

Rule 10: Be Up-to-Date, but Do Not Forget Older Studies

Given the progressive acceleration in the publication of scientific papers, today's reviews of the literature need awareness not just of the overall direction and achievements of a field of inquiry, but also of the latest studies, so as not to become out-of-date before they have been published. Ideally, a literature review should not identify as a major research gap an issue that has just been addressed in a series of papers in press (the same applies, of course, to older, overlooked studies (“sleeping beauties” [26] )). This implies that literature reviewers would do well to keep an eye on electronic lists of papers in press, given that it can take months before these appear in scientific databases. Some reviews declare that they have scanned the literature up to a certain point in time, but given that peer review can be a rather lengthy process, a full search for newly appeared literature at the revision stage may be worthwhile. Assessing the contribution of papers that have just appeared is particularly challenging, because there is little perspective with which to gauge their significance and impact on further research and society.

Inevitably, new papers on the reviewed topic (including independently written literature reviews) will appear from all quarters after the review has been published, so that there may soon be the need for an updated review. But this is the nature of science [27] – [32] . I wish everybody good luck with writing a review of the literature.

Acknowledgments

Many thanks to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M. Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P. Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu for insights and discussions, and to P. Bourne, T. Matoni, and D. Smith for helpful comments on a previous draft.

Funding Statement

This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.

University of Pittsburgh Library System

• Collections

Course & Subject Guides

Literature reviews, what is a literature review, a literature review is not, why is a literature review important, research . . hey, anyone can do it.

• Planning Your Review
• Searching Tips
• Getting Materials
• Scholarly Information
• Managing Your Results
• Writing Your Review

Types of Literature Reviews

Different projects involve different kinds of literature reviews with different kinds and  amounts of work. And, of course, the "end products" vary.

• Honors paper
• Capstone project
• Research Study
• Senior thesis
• Masters thesis
• Doctoral dissertation
• Research article
• Grant proposal
• Evidence based practice

Literature Reviews: An Overview for Graduate Students

NCSU Libraries  http://www.lib.ncsu.edu/tutorials/litreview/

Keep in mind that a literature review defines and sets the stage for your later research.  While you may take the same steps in researching your literature review, your literature review is not:

• Not an annotated bibliography in which you summarize each article that you have reviewed.  A lit review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.
• Not a research paper where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

Thanks Dave Kellett   http://www.sheldoncomics.com/archive/100806.html

• Next: Planning Your Review >>
• Last Updated: Aug 19, 2024 10:08 AM
• URL: https://pitt.libguides.com/literaturereview

Communication Studies *: The Literature Review

• Electronic Books
• Citation Tracking
• Conference Proceedings & White Papers
• Find the Full Text
• Broadcasts/Transcripts/Speeches
• Latino Newspapers
• Current News
• General Statistics
• Public Opinion
• Market Research
• Company Research
• Online videos
• Tests/Measures
• DVD Collection
• More Guides
• Research Tools
• RefWorks 2.0
• RefWorks 2.0 Tutorials
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• What Is Scholarly vs. Popular?
• Databases for Literature Review
• Government Websites

A literature review surveys scholarly articles, books and other sources relevant to a particular issue, area of research, or theory, and by so doing, providing a description, summary, and critical evaluation of these works. Literature reviews are designed to provide an overview of sources you have explored while researching a particular topic and to demonstrate to your readers how your research fits into the larger field of study.

All contect is from a Literature Review please refer to the sub-tab under The Literature Review created by Dr. Robert Larabee.

Importance of a Good Literature Review

A literature review may consist of simple a summary of key sources, but it usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• give a new interpretation of old material or combine new with old interpretations,
• trace the intellectual progression of the field, including major debates,
• depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant, or
• usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

The purpose of a literature review is to:

• Place each work in the context of its contribution to the understanding of the research problem being studied,
• Describe the relationship of each work to the others under consideration,
• Identify new ways to interpret, and shed light on any gaps in previous research,
• Resolve conflicts amongst seemingly contradictory previous studies,
• Identify areas of prior scholarship to prevent duplication of effort,
• Point the way in fulfilling a need for additional research, and
• Locate your own research within the context of existing literature.

All contect is from The Literature Review created by Dr. Robert Larabee.

Types of Literature Reviews

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following :

• An overview of the subject, issue or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories (e.g. works that support of a particular position, those against, and those offering alternative approaches entirely),
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence (e.g. primary historical material, case studies, narratives, statistics, recent scientific findings)?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most/least convincing?
• Value -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  The Development of the Literature Review

Four stages : 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not very specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources should I include? 2.  What types of sources should I review (books, journal articles, websites)? 3.  Should I summarize, synthesize, or critique your sources by discussing a common theme or issue? 4.  Should I evaluate the sources? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature reviews. Read them to get a sense of the types of themes you might want to look for in your own research or ways to organize your final review. The bibliography or reference section of sources you've already read are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make your job easier if you first limit scope of the research problem. A good strategy is to begin by searching the HOMER catalog for books about the topic and review their contents for chapters that focus on more specific issues. You can also review the subject indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is very common in the sciences where research conducted only two years ago could be obsolete. However, when writing a review in the social sciences, a survey of the history of the literature may be what is needed because what is important is how perspectives have changed over the years or within a certain time period. Try sorting through some other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to consider what is consider by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronological of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression of revealed a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic (“conceptual categories”) Thematic reviews of literature are organized around a topic or issue, rather than the progression of time. However, progression of time may still be an important factor in a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it will still be organized chronologically reflecting technological developments in media. The only difference here between a "chronological" and a "thematic" approach is what is emphasized the most: the role of the Internet in presidential politics. Note however that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Interbnet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you but include only what is necessary for the reader to locate your study within the larger scholarship framework.

Here are examples of other sections you may need to include depending on the type of review you write:

• Current Situation : information necessary to understand the topic or focus of the literature review.
• History : the chronological progression of the field, the literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : the criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed articles and journals.
• Standards : the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review in this sense is just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence to show that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Use Quotes Sparingly Some short quotes are okay if you want to emphasize a point, or if what the author said just cannot be rewritten in your own words. Sometimes you may need to quote certain terms that were coined by the author, not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute your own summary and interpretation of the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to their own work. Keep Your Own Voice While the literature review presents others' ideas, your voice (the writer's) should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

The most common mistakes made in reviewing social science research literature are that the researcher:

• does not clearly relate the findings of the literature review to the research problem;
• does not take sufficient time to define and identify the most relevent sources to use in the literature review related to the research problem;
• relies exclusively on secondary analytical sources rather than including primary research studies or data;
• uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• does not describe the search procedures that were used in the literature review;
• reports isolated statistical results rather than sythesizing them in chi-squared or meta-analytic methods; and,
• only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions. Consult with a librarian about identifying research databases in other disciplines; almost every discipline has at least one comprehensive database devoted to indexing its research literature.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to this part of writing a research paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. How are they structuring their ideas? What methods have they used to study the problem? What sources have they cited to support of their conclusions? How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've adequately reviewed the research literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a dead end. At this point consider: Does your study extend current research?  Does it forge as new path? Or, does is merely add more of the same thing being said?
• Look at the sources authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been introduced to the research question.
• Search the World of Knowledge Citation database and Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline.
• << Previous: Ratings
• Next: Citation Tracking >>
• Last Updated: Jul 12, 2024 1:43 PM
• URL: https://libguides.usc.edu/communication

Exploring the factors driving AI adoption in production: a systematic literature review and future research agenda

• Open access
• Published: 23 August 2024

Cite this article

You have full access to this open access article

• Heidi Heimberger   ORCID: orcid.org/0000-0003-3390-0219 1 , 2 ,
• Djerdj Horvat   ORCID: orcid.org/0000-0003-3747-3402 1 &
• Frank Schultmann   ORCID: orcid.org/0000-0001-6405-9763 1  

Our paper analyzes the current state of research on artificial intelligence (AI) adoption from a production perspective. We represent a holistic view on the topic which is necessary to get a first understanding of AI in a production-context and to build a comprehensive view on the different dimensions as well as factors influencing its adoption. We review the scientific literature published between 2010 and May 2024 to analyze the current state of research on AI in production. Following a systematic approach to select relevant studies, our literature review is based on a sample of articles that contribute to production-specific AI adoption. Our results reveal that the topic has been emerging within the last years and that AI adoption research in production is to date still in an early stage. We are able to systematize and explain 35 factors with a significant role for AI adoption in production and classify the results in a framework. Based on the factor analysis, we establish a future research agenda that serves as a basis for future research and addresses open questions. Our paper provides an overview of the current state of the research on the adoption of AI in a production-specific context, which forms a basis for further studies as well as a starting point for a better understanding of the implementation of AI in practice.

Explore related subjects

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

The technological change resulting from deep digitisation and the increasing use of digital technologies has reached and transformed many sectors [ 1 ]. In manufacturing, the development of a new industrial age, characterized by extensive automation and digitisation of processes [ 2 ], is changing the sector’s ‘technological reality’ [ 3 ] by integrating a wide range of information and communication technologies (such as Industry 4.0-related technologies) into production processes [ 4 ].

Although the evolution of AI traces back to the year 1956 (as part of the Dartmouth Conference) [ 5 ], its development has progressed rapidly, especially since the 2010s [ 6 ]. Driven by improvements, such as the fast and low-cost development of smart hardware, the enhancement of algorithms as well as the capability to manage big data [ 7 ], there is an increasing number of AI applications available for implementation today [ 8 ]. The integration of AI into production processes promises to boost the productivity, efficiency as well as automation of processes [ 9 ], but is currently still in its infancy [ 10 ] and manufacturing firms seem to still be hesitant to adopt AI in a production-context. This appears to be driven by the high complexity of AI combined with the lack of practical knowledge about its implementation in production and several other influencing factors [ 11 , 12 ].

In the literature, many contributions analyze AI from a technological perspective, mainly addressing underlying models, algorithms, and developments of AI tools. Various authors characterise both machine learning and deep learning as key technologies of AI [ 8 , 13 ], which are often applied in combination with other AI technologies, such as natural language recognition. While promising areas for AI application already exist in various domains such as marketing [ 14 ], procurement [ 15 ], supply chain management [ 16 ] or innovation management [ 17 ], the integration of AI into production processes also provides significant performance potentials, particularly in the areas of maintenance [ 18 ], quality control [ 19 ] and production planning and management [ 20 ]. However, AI adoption requires important technological foundations, such as the provision of data and the necessary infrastructure, which must be ensured [ 11 , 12 , 21 ]. Although the state of the art literature provides important insights into possible fields of application of AI in production, the question remains: To what extent are these versatile applications already in use and what is required for their successful adoption?

Besides the technology perspective of AI, a more human-oriented field of discussion is debated in scientific literature [ 22 ]. While new technologies play an essential role in driving business growth in the digital transformation of the production industry, the increasing interaction between humans and intelligent machines (also referred to as ‘augmentation’) creates stress challenges [ 23 ] and impacts work [ 24 ], which thus creates managerial challenges in organizations [ 25 , 26 ]. One of the widely discussed topics in this context is the fear of AI threatening jobs (including production jobs), which was triggered by e.g. a study of Frey, Osborne [ 27 ]. Another issue associated to the fear of machines replacing humans is the lack of acceptance resulting from the mistrust of technologies [ 28 , 29 ]. This can also be linked to the various ethical challenges involved in working with AI [ 22 ]. This perspective, which focuses on the interplay between AI and humans [ 30 ], reveals the tension triggered by AI. Although this is discussed from different angles, the question remains how these aspects influence the adoption of AI in production.

Another thematic stream of current literature can be observed in a series of contributions on the organizational aspects of the technology. In comparison to the two research areas discussed above, the number of publications in this area seems to be smaller. This perspective focuses on issues to implement AI, such as the importance of a profound management structure [ 31 , 32 ], leadership [ 33 ], implications on the organizational culture [ 34 ] as well as the need for digital capabilities and special organizational skills [ 33 ]. Although some studies on the general adoption of AI without a sectoral focus have already been conducted (such as by Chen, Tajdini [ 35 ] or Kinkel, Baumgartner, Cherubini [ 36 ]) and hence, some initial factors influencing the adoption of AI can be derived, the contributions from this perspective are still scarce, are usually not specifically analyzed in the context of production or lack a comprehensive view on the organization in AI adoption.

While non-industry specific AI issues have been researched in recent years, the current literature misses a production-specific analysis of AI adoption, providing an understanding of the possibilities and issues related to integrating AI into the production context. Moreover, the existing literature tells us little about relevant mechanisms and factors underlying the adoption of AI in production processes, which include both technical, human-centered as well as organizational issues. As organizational understanding of AI in a business context is currently still in its early stages, it is difficult to find an aggregate view on the factors that can support companies in implementing AI initiatives in production [ 37 , 38 ]. Addressing this gap, we aim to systematise the current scientific knowledge on AI adoption, with a focus on production. By drawing on a systematic literature review (SLR), we examine existing studies on AI adoption in production and explore the main issues regarding adoption that are covered in the analyzed articles. Building on these findings, we conduct a comprehensive analysis of the existing studies with the aim of systematically investigating the key factors influencing the adoption of AI in production. This systematic approach paves the way for the formulation of a future research agenda.

Our SLR addresses three research questions (RQs). RQ1: What are the statistical characteristics of existing research on AI adoption in production? To answer this RQ, we conduct descriptive statistics of the analyzed studies and provide information on time trends, methods used in the research, and country specifications. RQ2: What factors influence the adoption of AI in production? RQ2 specifies the adoption factors and forms the core component of our analysis. By adoption factors, we mean the factors that influence the use of AI in production (both positively and negatively) and that must therefore be analyzed and taken into account. RQ3: What research topics are of importance to advance the research field of AI adoption in production? We address this RQ by using the analyzed literature as well as the key factors of AI adoption as a starting point to derive RQs that are not addressed and thus provide an outlook on the topic.

2 Methodology

In order to create a sound information base for both policy makers and practitioners on the topic of AI adoption in production, this paper follows the systematic approach of a SLR. For many fields, including management research, a SLR is an important tool to capture the diversity of existing knowledge on a specific topic for a scientific investigation [ 39 ]. The investigator often pursues multiple goals, such as capturing and assessing the existing environment and advancing the existing body of knowledge with a proprietary RQ [ 39 ] or identifying key research topics [ 40 ].

Our SLR aims to select, analyze, and synthesize findings from the existing literature on AI adoption in production over the past 24 years. In order to identify relevant data for our literature synthesis, we follow the systematic approach of the Preferred Reporting Items for Systematic reviews (PRISMA) [ 41 ]. In evaluating the findings, we draw on a mixed-methods approach, combining some quantitative analyses, especially on the descriptive aspects of the selected publications, as well as qualitative analyses aimed at evaluating and comparing the contents of the papers. Figure  1 graphically summarizes the methodological approach that guides the content of the following sub-chapters.

Methodical procedure of our SLR following PRISMA [ 41 ]

2.1 Data identification

Following the development of the specific RQs, we searched for suitable publications. To locate relevant studies, we chose to conduct a publication analysis in the databases Scopus, Web of Science and ScienceDirect as these databases primarily contain international scientific articles and provide a broad overview of the interdisciplinary research field and its findings. To align the search with the RQs [ 42 ], we applied predefined key words to search the titles, abstracts, and keywords of Scopus, Web of Science and ScienceDirect articles. Our research team conducted several pre-tests to determine the final search commands for which the test results were on target and increased the efficiency of the search [ 42 ]. Using the combination of Boolean operators, we covered the three topics of AI, production, and adoption by searching combinations of ‘Artificial Intelligence’ AND ‘production or manufacturing’ AND ‘adopt*’ in the three scientific databases. Although ‘manufacturing’ tends to stand for the whole sector and ‘production’ refers to the process, the two terms are often used to describe the same context. We also follow the view of Burbidge, Falster, Riis, Svendsen [ 43 ] and use the terms synonymously in this paper and therefore also include both terms as keywords in the study location as well as in the analysis.

AI research has been credited with a resurgence since 2010 [ 6 ], which is the reason for our choice of time horizon. Due to the increase in publications within the last years, we selected articles published online from 2010 to May 8, 2024 for our analysis. As document types, we included conference papers, articles, reviews, book chapters, conference reviews as well as books, focusing exclusively on contributions in English in the final publication stage. The result of the study location is a list of 3,833 documents whose titles, abstracts, and keywords meet the search criteria and are therefore included in the next step of the analysis.

2.2 Data analysis

For these 3,833 documents, we then conducted an abstract analysis, ‘us[ing] a set of explicit selection criteria to assess the relevance of each study found to see if it actually does address the research question’ [ 42 ]. For this step, we again conducted double-blind screenings (including a minimum of two reviewers) as pilot searches so that all reviewers have the same understanding of the decision rules and make equal decisions regarding their inclusion for further analysis.

To ensure the paper’s focus on all three topics regarded in our research (AI, production, and adoption), we followed clearly defined rules of inclusion and exclusion that all reviewers had to follow in the review process. As a first requirement for inclusion, AI must be the technology in focus that is analysed in the publication. If AI was only mentioned and not further specified, we excluded the publication. With a second requirement, we checked the papers for the context of analysis, which in our case must be production. If the core focus is beyond production, the publication was also excluded from further analysis. The third prerequisite for further consideration of the publication is the analysis of the adoption of a technology in the paper. If technology adoption is not addressed or adoption factors are not considered, we excluded the paper. An article was only selected for full-text analysis if, after analyzing the titles, abstracts, and keywords, a clear focus on all three research areas was visible and the inclusion criteria were met for all three contexts.

By using this tripartite inclusion analysis, we were able to analyse the publications in a structured way and to reduce the 3,833 selected documents in our double-blind approach to 300 articles that were chosen for the full-text analysis. In the process of finding full versions of these publications, we had to exclude three papers as we could not access them. For the rest of the 297 articles we obtained full access and thus included them for further analysis. After a thorough examination of the full texts, we again had to exclude 249 publications because they did not meet our content-related inclusion criteria mentioned above, although the abstract analysis gave indications that they did. As a result, we finally obtained 47 selected papers on which we base the literature analysis and synthesis (see Fig.  1 ).

2.3 Descriptive analysis

Figure  2 summarises the results of the descriptive analysis on the selected literature regarding AI adoption in production that we analyse in our SLR. From Fig.  2 a), which illustrates annual publication trends (2010–2024), the increase in publications on AI adoption in production over the past 5 years is evident, yet slightly declining after a peak in 2022. After a steady increase until 2022, in which 11 articles are included in the final analysis, 2023 features ten articles, followed by three articles for 2024 until the cut-off date in May 2024. Of the 47 papers identified through our search, the majority (n = 33) are peer-reviewed journal articles and the remaining thirteen contributions conference proceedings and one book chapter (see Fig.  2 b)).

Descriptive analyses of the selected articles addressing AI adoption in production

The identified contributions reveal some additional characteristics in terms of the authors country base (Fig.  2 c)) and research methods used (Fig.  2 d)). Almost four out of ten of the publications were written in collaboration with authors from several countries (n = 19). Six of the papers were published by authors from the United States, five from Germany and four from India. In terms of the applied research methods used by the researchers, a wide range of methods is used (see Fig.  2 c), with qualitative methods (n = 22) being the most frequently used.

2.4 Factor analysis

In order to derive a comprehensive list of factors that influence the use of AI in production at different levels, we follow a qualitative content analysis. It is based on inductive category development, avoiding prefabricated categories in order to allow new categories to emerge based on the content at hand [ 44 , 45 ]. To do this, we first read the entire text to gain an understanding of the content and then derive codes [ 46 ] that seem to capture key ideas [ 45 ]. The codes are subsequently sorted into distinct categories, each of which is clearly defined and establishes meaningful connections between different codes. Based on an iterative process with feedback loops, the assigned categories are continuously reviewed and updated as revisions are made [ 44 ].

Various factors at different levels are of significance to AI and influence technology adoption [ 47 , 48 ]. To identify the specific factors that are of importance for AI adoption in production, we analyze the selected contributions in terms of the factors considered, compare them with each other and consequently obtain a list of factors through a bottom-up approach. While some of the factors are based on empirical findings, others are expected factors that result from the research findings of the respective studies. Through our analysis, a list of 35 factors emerges that influence AI adoption in production which occur with varying frequency in the studies analyzed by our SLR. Table 1 visualizes each factor in the respective contributions sorted by the frequency of occurrence.

The presence of skills is considered a particularly important factor in AI adoption in the studies analyzed (n = 35). The availability of data (n = 25) as well as the need for ethical guidelines (n = 24) are also seen as key drivers of AI adoption, as data is seen as the basis for the implementation of AI and ethical issues must be addressed in handling such an advanced technology. As such, these three factors make up the accelerants of AI adoption in production that are most frequently cited in the studies analyzed.

Also of importance are issues of managerial support (n = 22), as well as performance measures and IT infrastructure (n = 20). Some factors were also mentioned, but only addressed by one study at a time: government support, industrial sector, product complexity, batch size, and R&D Intensity. These factors are often used as quantitatively measurable adoption factors, especially in empirical surveys, such the study by Kinkel, Baumgartner, Cherubini [ 36 ].

3 Factors influencing AI adoption

The 35 factors presented characteristically in Sect.  2.4 serve as the basis for our in-depth analysis and for developing a framework of influences on AI adoption in production which are grouped into supercategories. A supercategory describes a cluster of topics to which various factors of AI adoption in production can be assigned. We were able to define seven categories that influence AI adoption in production: the internal influences of ‘business and structure’, ‘organizational effectiveness’, ‘technology and system’, ‘data management’ as well as the external influences of the ‘regulatory environment’, ‘business environment’ and ‘economic environment’ (see Fig.  3 ). The factors that were mentioned most frequently (occurrence in at least half of the papers analyzed) are marked accordingly (*) in Fig.  3 .

Framework of factors influencing AI adoption in production

3.1 Internal Environment

The internal influences on AI adoption in production refer to factors that an organization carries internally and that thus also influence adoption from within. Such factors can usually be influenced and clearly controlled by the organization itself.

3.1.1 Business and structure

The supercategory ‘business and structure’ includes the various factors and characteristics that impact a company’s performance, operations, and strategic decision-making. By considering and analyzing these business variables when implementing AI in production processes, companies can develop effective strategies to optimize their performance, increase their competitiveness, and adapt to changes in the business environment.

To understand and grasp the benefits in the use of AI, quantitative performance measures for the current and potential use of AI in industrial production systems help to clarify the value and potential benefits of AI use [ 49 , 54 , 74 , 79 , 91 ]. Assessing possible risks [ 77 ] as well as the monetary expected benefits for AI (e.g. Return on Investment (ROI)) in production plays an important role for adoption decisions in market-oriented companies [ 57 , 58 , 63 , 65 , 78 ]. Due to financial constraints, managers behave cautiously in their investments [ 78 ], so they need to evaluate AI adoption as financially viable to want to make the investment [ 61 , 63 , 93 ] and also drive acceptance [ 60 ]. AI systems can significantly improve cost–benefit structures in manufacturing, thereby increasing the profitability of production systems [ 73 ] and making companies more resilient [ 75 ]. However, in most cases, the adoption of AI requires high investments and the allocation of resources (s.a. personnel or financial) for this purpose [ 50 , 51 , 57 , 80 , 94 ]. Consequently, a lack of budgets and high expected transition costs often hinder the implementation of smart concepts [ 56 , 62 , 67 , 82 , 84 , 92 ]. It is up to management to provide necessary funding for AI adoption [ 53 , 59 , 79 ], which is required, for example, for skill development of employees [ 59 , 61 , 63 ], IT adaptation [ 62 , 66 ], AI development [ 74 ] or hardware deployment [ 68 ]. In their empirical study, Kinkel, Baumgartner, Cherubini [ 36 ] confirm a positive correlation between company size and the intensity in the use of AI technologies. Large companies generally stand out with a higher propensity to adopt [ 53 ] as they have less difficulties in comparison to small firms regarding the availability of resources [ 69 ], such as know-how, budget [ 68 , 84 ] and general data organization [ 68 ]. Others argue that small companies tend to be more open to change and are characterized by faster decision-making processes [ 68 , 93 ]. Product complexity also influences a company’s propensity for AI. Companies that produce rather simple products are more likely to digitize, which in turn offers good starting points for AI adoption. On the other hand, complex product manufacturers (often characterized by small batch sizes) are often less able to standardize and automate [ 36 ]. The company’s produced batch size has a similar influence on AI adoption. Small and medium batch sizes in particular hinder the integration of intelligent technologies, as less automation often prevails here as well. Nevertheless, even small and medium lot sizes can benefit economically from AI [ 36 ]. Since a high R&D intensity indicates a high innovation capability of a company, it is assumed to have a positive influence on AI adoption, as companies with a high R&D intensity already invest heavily in and use new innovations. This in turn speaks for existing competencies, know how and structures [ 36 ].

3.1.2 Organizational effectiveness

This supercategory focuses on the broader aspects that contribute to the effectiveness, development, and success of an organization when implementing AI in a production context. As the factors are interconnected and influence each other, decision makers should consider them carefully.

Users´ trust in AI is an essential factor to enable successful AI adoption and use in production [ 52 , 68 , 78 , 79 , 88 , 90 ]. From the users´ perspective, AI often exhibits the characteristics of a black box because its inherent processes are not fully understood [ 50 , 90 ] which can lead individuals to develop a fear towards the unknown [ 71 ]. Because of this lack of understanding, successful interaction between humans and AI is not guaranteed [ 90 ], as trust is a foundation for decisions that machines are intended to make autonomously [ 52 , 91 ]. To strengthen faith in AI systems [ 76 , 80 ], AI users can be involved in AI design processes in order to understand appropriate tools [ 54 , 90 ]. In this context, trust is also discussed in close connection with transparency and regulation [ 79 ]. User resistance is considered a barrier to implementing new information technologies, as adoption requires change [ 53 , 62 , 92 ]. Ignorance, as a kind of resistance to change, is a main obstacle to successful digital transformation [ 51 , 56 , 65 ]. Some employees may resist the change brought about by AI because they fear losing their jobs [ 52 ] or have other concerns [ 78 ]. Overcoming resistance to technology adoption requires organizational change and is critical for the success of adoption [ 50 , 51 , 62 , 67 , 71 , 80 ]. Therefore, change management is important to create awareness of the importance of AI adoption and increase acceptance of the workforce [ 66 , 68 , 74 , 83 ]. Management commitment is seen as a significant driver of technology adoption [ 53 , 59 , 81 , 82 , 86 ] and a lack of commitment can negatively impact user adoption and workforce trust and lead to skepticism towards technology [ 86 ]. The top management’s understanding and support for the benefits of the adopted technology [ 53 , 56 , 67 , 78 , 93 , 94 ] enhances AI adoption, can prioritize its implementation and also affects the performance of the AI-enabled application [ 55 , 60 , 83 ]. Preparing, enabling, and thus empowering the workforce, are considered the management’s responsibility in the adoption of digital technologies [ 59 , 75 ]. This requires intelligent leadership [ 52 ] as decision makers need to integrate their workforce into decision-making processes [ 75 ]. Guidelines can support managers by providing access to best practices that help in the adoption of AI [ 50 ]. Critical measures to manage organizational change include the empowerment of visionaries or appointed AI champions leading the change and the collaborative development of digital roadmaps [ 54 , 62 ]. To demonstrate management commitment, managers can create such a dedicated role, consisting of an individual or a small group that is actively and enthusiastically committed to AI adoption in production. This body is considered the adoption manager, point of contact and internal driver of adoption [ 62 , 74 , 80 ]. AI initiatives in production do not necessarily have to be initiated by management. Although management support is essential for successful AI adoption, employees can also actively drive integration initially and thus realize pilot projects or initial trials [ 66 , 80 ]. The development of strategies as well as roadmaps is considered another enabling and necessary factor for the adoption of AI in production [ 50 , 53 , 54 , 62 , 71 , 93 ]. While many major AI strategies already exist at country level to further promote research and development of AI [ 87 ], strategy development is also important at the firm level [ 76 , 77 , 81 ]. In this context, strategies should not be delegated top-down, but be developed in a collaborative manner, i.e. by engaging the workforce [ 75 ] and be in alignment with clear visions [ 91 , 94 ]. Roadmaps are used to improve planning, support implementation, facilitate the adoption of smart technologies in manufacturing [ 93 ] and should be integrated into both business and IT strategy [ 62 , 66 ]. In practice, clear adoption roadmaps that provide approaches on how to effectively integrate AI into existing strategies and businesses are often lacking [ 56 , 87 ]. The need for AI-related skills in organizations is a widely discussed topic in AI adoption analyses [ 79 ]. In this context, the literature points both at the need for specific skills in the development and design of AI applications [ 57 , 71 , 72 , 73 , 76 , 93 ] as well as the skills in using the technology [ 53 , 65 , 73 , 74 , 75 , 84 , 93 ] which availability in the firm is not always given [ 49 ]. AI requires new digital skills [ 36 , 50 , 52 , 55 , 56 , 59 , 61 , 63 , 66 , 78 , 80 ], where e.g. advanced analytics [ 64 , 75 , 81 ], programming skills [ 68 ] and cybersecurity skills [ 78 , 93 ] gain importance. The lack of skills required for AI is seen as a major challenge of digital transformation, as a skilled workforce is considered a key resource for companies [ 51 , 54 , 56 , 60 , 62 , 67 , 69 , 70 , 82 , 93 ]. This lack of a necessary skillset hinders the adoption of AI tools in production systems [ 58 , 77 ]. Closely related to skills is the need for new training concepts, which organizations need to consider when integrating digital technologies [ 49 , 50 , 51 , 56 , 59 , 63 , 71 , 74 , 75 ]. Firms must invest in qualification in order to create necessary competences [ 73 , 78 , 80 , 81 , 92 ]. Additionally, education must target and further develop the skills required for effectively integrating intelligent technologies into manufacturing processes [ 54 , 61 , 62 , 83 ]. Regarding this issue, academic institutions must develop fitting curricula for data driven manufacturing engineering [ 64 ]. Another driving factor of AI adoption is the innovation culture of an organization, which is influenced by various drivers. For example, companies that operate in an environment with high innovation rates, facing intense competitive pressures are considered more likely to see smart technologies as a tool for strategic change [ 83 , 91 , 93 ]. These firms often invest in more expensive and advanced smart technologies as the pressure and resulting competition forces them to innovate [ 93 ]. Another way of approach this is that innovation capability can also be supported and complemented by AI, for example by intelligent systems supporting humans in innovation or even innovating on their own [ 52 ].The entrepreneurial orientation of a firm is characterized in particular by innovativeness [ 66 ], productivity [ 63 ], risk-taking [ 86 ] as well as continuous improvement [ 50 ]. Such characteristics of an innovating culture are considered essential for companies to recognise dynamic changes in the market and make adoption decisions [ 51 , 71 , 81 , 84 , 86 , 94 ]. The prevalence of a digital mindset in companies is important for technology adoption, as digital transformation affects the entire organizational culture and behavior [ 59 , 80 , 92 ] and a lack of a digital culture [ 50 , 65 ] as well as a ‘passive mindset’ [ 78 ] can hinder the digital transformation of firms. Organizations need to develop a corresponding culture [ 66 , 67 , 71 ], also referred to as ‘AI-ready-culture’ [ 54 ], that promotes development and encourages people and data through the incorporation of technology [ 71 , 75 ]. With the increasing adoption of smart technologies, a ‘new digital normal’ is emerging, characterized by hybrid work models, more human–machine interactions and an increased use of digital technologies [ 75 , 83 ].

3.1.3 Technology and System

The ‘technology and system’ supercategory focuses on the broader issues related to the technology and infrastructure that support organizational operations and provide the technical foundation for AI deployment.

By IT infrastructure we refer to issues regarding the foundational systems and IT needed for AI adoption in production. Industrial firms and their IT systems must achieve a mature technological readiness in order to enable successful AI adoption [ 51 , 60 , 67 , 69 , 83 ]. A lack of appropriate IT infrastructure [ 68 , 71 , 78 , 91 ] or small maturity of Internet of Things (IoT) technologies [ 70 ]) hinders the efficient use of data in production firms [ 56 ] which is why firms must update their foundational information systems for successful AI adoption [ 53 , 54 , 62 , 66 , 72 , 75 ]. IT and data security are fundamental for AI adoption and must be provided [ 50 , 51 , 68 , 82 ]. This requires necessary developments that can ensure security during AI implementation while complying with legal requirements [ 52 , 72 , 78 ]. Generally, security concerns are common when implementing AI innovations [ 72 , 79 , 91 , 94 ]. This fear of a lack of security can also prevent the release of (e.g. customer) data in a production environment [ 56 ]. Additionally, as industrial production systems are vulnerable to failures as well as cyberattacks, companies need to address security and cybersecurity measures [ 49 , 76 , 88 , 89 ]. Developing user-friendly AI solutions can facilitate the adoption of smart solutions by increasing user understanding and making systems easy to use by employees as well as quick to integrate [ 50 , 72 , 84 ]. When developing user-friendly solutions which satisfy user needs [ 76 ], it is particularly important to understand and integrate the user perspective in the development process [ 90 ]. If employees find technical solutions easy to use, they are more confident in its use and perceived usefulness increases [ 53 , 67 , 68 ]. The compatibility of AI with a firm and its existing systems, i.e., the extent to which AI matches existing processes, structures, and infrastructures [ 53 , 54 , 56 , 60 , 78 , 80 , 82 , 83 , 93 , 94 ], is considered an important requirement for the adoption of AI in IT systems [ 91 ]. Along with compatibility also comes connectivity, which is intended to ensure the links within the overall network and avoid silo thinking [ 59 ]. Connectivity and interoperability of AI-based processes within the company’s IT manufacturing systems must be ensured at different system levels and are considered key factors in the development of AI applications for production [ 50 , 72 , 89 ]. The design of modular AI solutions can increase system compatibility [ 84 ]. Firms deciding for AI adoption must address safety issues [ 51 , 54 , 59 , 72 , 73 , 78 ]. This includes both safety in the use and operation of AI [ 60 , 69 ]. In order to address safety concerns of integrating AI solutions in industrial systems [ 49 ], systems must secure high reliability [ 71 ]. AI can also be integrated as a safety enabler, for example, by providing technologies to monitor health and safety in the workplace to prevent fatigue and injury [ 75 ].

3.1.4 Data management

Since AI adoption in the organization is strongly data-driven, the ‘data management’ supercategory is dedicated to the comprehensive aspects related to the effective and responsible management of data within the organization.

Data privacy must be guaranteed when creating AI applications based on industrial production data [ 49 , 58 , 59 , 60 , 72 , 76 , 78 , 79 , 82 , 88 , 89 , 91 , 94 ] as ‘[M]anufacturing industries generate large volumes of unstructured and sensitive data during their daily operations’ [ 89 ]. Closely related to this is the need for anonymization and confidentiality of data [ 61 , 69 , 70 , 78 ]. The availability of large, heterogeneous data sets is essential for the digital transformation of organizations [ 52 , 59 , 78 , 80 , 88 , 89 ] and is considered one of the key drivers of AI innovation [ 62 , 68 , 72 , 86 ]. In production systems, lack of data availability is often a barrier to AI adoption [ 58 , 70 , 77 ]. In order to enable AI to establish relationships between data, the availability of large input data that is critical [ 62 , 76 , 81 ]. New AI models are trained with this data and can adapt as well as improve as they receive new data [ 59 , 62 ]. Big data can thus significantly improve the quality of AI applications [ 59 , 71 ]. As more and more data is generated in manufacturing [ 85 ], AI opens up new opportunities for companies to make use of it [ 62 ]. However, operational data are often unstructured, as they come from different sources and exist in diverse formats [ 85 , 87 ]. This challenges data processing, as data quality and origin are key factors in the management of data [ 78 , 79 , 80 , 88 , 89 , 91 ]. To make production data valuable and usable for AI, consistency of data and thus data integrity is required across manufacturing systems [ 50 , 62 , 77 , 84 ]. Another key prerequisites for AI adoption is data governance [ 56 , 59 , 67 , 68 , 71 , 78 , 88 ] which is an important asset to make use of data in production [ 50 ] and ensure the complex management of heterogenous data sets [ 89 ]. The interoperability of data and thus the foundation for the compatibility of AI with existing systems, i.e., the extent to which AI matches existing processes, structures, and infrastructures [ 53 , 56 , 84 , 93 ], is considered another important requirement for the adoption of AI in IT systems. Data interoperability in production systems can be hindered by missing data standards as different machines use different formats [ 87 ]. Data processing refers to techniques used to preparing data for analysis which is essential to obtain consistent results from data analytics in production [ 58 , 72 , 80 , 81 , 84 ]. In this process, the numerous, heterogeneous data from different sensors are processed in such a way that they can be used for further analyses [ 87 ]. The capability of production firms to process data and information is thus important to enable AI adoption [ 77 , 86 , 93 ]. With the increasing data generation in the smart and connected factory, the strategic relevance of data analytics is gaining importance [ 55 , 69 , 78 ], as it is essential for AI systems in performing advanced data analyses [ 49 , 67 , 72 , 86 , 88 ]. Using analytics, valuable insights can be gained from the production data obtained using AI systems [ 58 , 77 , 87 ]. In order to enable the processing of big data, a profound data infrastructure is necessary [ 65 , 75 , 87 ]. Facilities must be equipped with sensors, that collect data and model information, which requires investments from firms [ 72 ]. In addition, production firms must build the necessary skills, culture and capabilities for data analytics [ 54 , 75 , 87 , 93 ]. Data storage, one of the foundations and prerequisites for smart manufacturing [ 54 , 68 , 71 , 74 ], must be ensured in order to manage the larg amounts of data and thus realize the adoption of intelligent technologies in production [ 50 , 59 , 72 , 78 , 84 , 87 , 88 , 89 ].

3.2 External environment

The external drivers of AI adoption in production influence the organization through conditions and events from outside the firm and are therefore difficult to control by the organization itself.

3.2.1 Regulatory environment

This supercategory captures the broader concept of establishing rules, standards, and frameworks that guide the behavior, actions, and operations of individuals, organizations, and societies when implementing AI.

AI adoption in production faces many ethical challenges [ 70 , 72 , 79 ]. AI applications must be compliant with the requirements of organizational ethical standards and laws [ 49 , 50 , 59 , 60 , 62 , 75 ] which is why certain issues must be examined in AI adoption and AI design [ 62 , 73 , 82 , 91 ] so that fairness and justice are guaranteed [ 78 , 79 , 92 ]. Social rights, cultural values and norms must not be violated in the process [ 49 , 52 , 53 , 81 ]. In this context, the explainability and transparency of AI decisions also plays an important role [ 50 , 54 , 58 , 70 , 78 , 89 ] and can address the characteristic of AI of a black box [ 90 ]. In addition, AI applications must be compliant with legal and regulatory requirements [ 51 , 52 , 59 , 77 , 81 , 82 , 91 ] and be developed accordingly [ 49 , 76 ] in order to make organization processes using AI clear and effective [ 65 ]. At present, policies and regulation of AI are still in its infancy [ 49 ] and missing federal regulatory guidelines, standards as well as incentives hinder the adoption of AI [ 67 ] which should be expanded simultaneously to the expansion of AI technology [ 60 ]. This also includes regulations on the handling of data (e.g. anonymization of data) [ 61 , 72 ].

3.2.2 Business environment

The factors in the ‘business environment’ supercategory refer to the external conditions and influences that affect the operations, decision making, and performance of the company seeking to implement AI in a production context.

Cooperation and collaboration can influence the success of digital technology adoption [ 52 , 53 , 59 , 72 ], which is why partnerships are important for adoption [ 53 , 59 ] and can positively influence its future success [ 52 , 67 ]. Both intraorganizational and interorganizational knowledge sharing can positively influence AI adoption [ 49 ]. In collaborations, companies can use a shared knowledge base where data and process sharing [ 51 , 59 , 94 ] as well as social support systems strengthen feedback loops between departments [ 79 , 80 ]. With regard to AI adoption in firms, vendors as well as service providers need to collaborate closely to improve the compatibility and operational capability of smart technologies across different industries [ 82 , 93 ]. Without external IT support, companies can rarely integrate AI into their production processes [ 66 ], which is why thorough support from vendors can significantly facilitate the integration of AI into existing manufacturing processes [ 80 , 91 ]. Public–private collaborations can also add value and governments can target AI dissemination [ 60 , 74 ]. The support of the government also positively influences AI adoption. This includes investing in research projects and policies, building a regulatory setting as well as creating a collaborative environment [ 60 ]. Production companies are constantly exposed to changing conditions, which is why the dynamics of the environment is another factor influencing the adoption of AI [ 52 , 63 , 72 , 86 ]. Environmental dynamics influence the operational performance of firms and can favor an entrepreneurial orientation of firms [ 86 ]. In order to respond to dynamics, companies need to develop certain capabilities and resources (i.e. dynamic capabilities) [ 86 ]. This requires the development of transparency, agility, as well as resilience to unpredictable changes, which was important in the case of the COVID-19 pandemic, for example, where companies had to adapt quickly to changing environments [ 75 ]. A firm’s environment (e.g. governments, partners or customers) can also pressure companies to adopt digital technologies [ 53 , 67 , 82 , 91 ]. Companies facing intense competition are considered more likely to invest in smart technologies, as rivalry pushes them to innovate and they hope to gain competitive advantages from adoption [ 36 , 66 , 82 , 93 ].

3.2.3 Economic environment

By considering both the industrial sector and country within the subcategory ‘economic environment’, production firms can analyze the interplay between the two and understand how drivers can influence the AI adoption process in their industrial sector’s performance within a particular country.

The industrial sector of a firm influences AI adoption in production from a structural perspective, as it indicates variations in product characteristics, governmental support, the general digitalization status, the production environment as well as the use of AI technologies within the sector [ 36 ]. Another factor that influences AI adoption is the country in which a company is located. This influences not only cultural aspects, the availability of know-how and technology orientation, but also regulations, laws, standards and subsidies [ 36 ]. From another perspective, AI can also contribute to the wider socio-economic growth of economies by making new opportunities easily available and thus equipping e.g. more rural areas with advanced capabilities [ 78 ].

3.3 Future research directions

The analysis of AI adoption in production requires a comprehensive analysis of the various factors that influence the introduction of the innovation. As discussed by Kinkel, Baumgartner, Cherubini [ 36 ], our research also concludes that organizational factors have a particularly important role to play. After evaluating the individual drivers of AI adoption in production in detail in this qualitative synthesis, we draw a conclusion from the results and derive a research agenda from the analysis to serve as a basis for future research. The RQs emerged from the analyzed factors and are presented in Table  2 . We developed the questions based on the literature review and identified research gaps for every factor that was most frequently mentioned. From the factors analyzed and RQs developed, the internal environment has a strong influence on AI adoption in production, and organizational factors play a major role here.

Looking at the supercategory ‘business and environment’, performance indicators and investments are considered drivers of AI adoption in production. Indicators to measure the performance of AI innovations are necessary here so that managers can perform cost–benefit analyses and make the right decision for their company. There is a need for research here to support possible calculations and show managers a comprehensive view of the costs and benefits of technology in production. In terms of budget, it should be noted that AI adoption involves a considerable financial outlay that must be carefully weighed and some capital must be available to carry out the necessary implementation efforts (e.g., staffing costs, machine retrofits, change management, and external IT service costs). Since AI adoption is a complex process and turnkey solutions can seldom be implemented easily and quickly, but require many changes (not only technologically but also on an organizational level), it is currently difficult to estimate the necessary budgets and thus make them available. Especially the factors of the supercategory ‘organizational effectiveness’ drive AI adoption in production. Trust of the workforce is considered an important driver, which must be created in order to successfully implement AI. This requires measures that can support management in building trust. Closely related to this are the necessary change management processes that must be initiated to accompany the changes in a targeted manner. Management itself must also play a clear role in the introduction of AI and communicate its support, as this also influences the adoption. The development of clear processes and measures can help here. Developing roadmaps for AI adoption can facilitate the adoption process and promote strategic integration with existing IT and business strategy. Here, best practice roadmaps and necessary action steps can be helpful for companies. Skills are considered the most important driver for AI adoption in manufacturing. Here, there is a lack of clear approaches that support companies in identifying the range of necessary skills and, associated with this, also opportunities to further develop these skills in the existing workforce. Also, building a culture of innovation requires closer research that can help companies foster a conducive environment for AI adoption and the integration of other smart technologies. Steps for developing a positive mindset require further research that can provide approaches for necessary action steps and measures in creating a positive digital culture. With regard to ‘technology and system’, the factors of IT infrastructure and security in particular are driving AI adoption in production. Existing IT systems must reach a certain maturity to enable AI adoption on a technical level. This calls for clear requirements that visualize for companies which systems and standards are in place and where developments are needed. Security must be continuously ensured, for which certain standards and action catalogs must be developed. With regard to the supercategory ‘data management’, the availability of data is considered the basis for successful AI adoption, as no AI can be successfully deployed without data. In the production context in particular, this requires developments that support companies in the provision of data, which usually arises from very heterogeneous sources and forms. Data analytics must also be closely examined, and production companies usually need external support in doing so. The multitude of data also requires big data storage capabilities. Here, groundwork is needed to show companies options about the possibilities of different storage options (e.g., on premis vs. cloud-based).

In the ‘regulatory environment’, ethics in particular is considered a driver of AI adoption in production. Here, fundamental ethical factors and frameworks need to be developed that companies can use as a guideline to ensure ethical standards throughout the process. Cooperations and environmental dynamism drive the supercategory ‘business environment’. Collaborations are necessary to successfully implement AI adoption and action is needed to create the necessary contact facilitation bodies. In a competitive environment, companies have to make quick decisions under strong pressure, which also affects AI adoption. Here, guidelines and also best practice approaches can help to simplify decisions and quickly demonstrate the advantage of the solutions. There is a need for research in this context.

4 Conclusions

The use of AI technologies in production continues to gain momentum as managers hope to increase efficiency, productivity and reduce costs [ 9 , 13 , 20 ]. Although the benefits of AI adoption speak for themselves, implementing AI is a complex decision that requires a lot of knowledge, capital and change [ 95 ] and is influenced by various internal and external factors. Therefore, managers are still cautious about implementing the technology in a production context. Our SLR seeks to examine the emergent phenomenon of AI in production with the precise aim of understanding the factors influencing AI adoption and the key topics discussed in the literature when analyzing AI in a production context. For this purpose, we use the current state of research and examine the existing studies based on the methodology of a systematic literature analysis and respond to three RQs.

We answer RQ1 by closely analyzing the literature selected in our SLR to identify trends in current research on AI adoption in production. In this process, it becomes clear that the topic is gaining importance and that research has increased over the last few years. In the field of production, AI is being examined from various angles and current research addresses aspects from a business, human and technical perspective. In our response to RQ2 we synthesized the existing literature to derive 35 factors that influence AI adoption in production at different levels from inside or outside the organization. In doing so, we find that AI adoption in production poses particularly significant challenges to organizational effectiveness compared to other digital technologies and that the relevance of data management takes on a new dimension. Production companies often operate more traditionally and are sometimes rigid when it comes to change [ 96 , 97 ], which can pose organizational challenges when adopting AI. In addition, the existing machines and systems are typically rather heterogeneous and are subject to different digitalization standards, which in turn can hinder the availability of the necessary data for AI implementation [ 98 , 99 ]. We address RQ3 by deriving a research agenda, which lays a foundation for further scientific research and deepening the understanding of AI adoption in production. The results of our analysis can further help managers to better understand AI adoption and to pay attention to the different factors that influence the adoption of this complex technology.

4.1 Contributions

Our paper takes the first step towards analysing the current state of the research on AI adoption from a production perspective. We represent a holistic view on the topic, which is necessary to get a better understanding of AI in a production-context and build a comprehensive view on the different dimensions as well as factors influencing its adoption. To the best of our knowledge, this is the first contribution that systematises research about the adoption of AI in production. As such, it makes an important contribution to current AI and production research, which is threefold:

First, we highlight the characteristics of studies conducted in recent years on the topic of AI adoption in production, from which several features and developments can be deduced. Our results confirm the topicality of the issue and the increasing relevance of research in the field.

Having laid the foundations for understanding AI in production, we focused our research on the identification and systematization of the most relevant factors influencing AI adoption in production at different levels. This brings us to the second contribution, our comprehensive factor analysis of AI adoption in production provides a framework for further research as well as a potential basis for managers to draw upon when adopting AI. By systematizing the relevant factors influencing AI adoption in production, we derived a set of 35 researched factors associated with AI adoption in production. These factors can be clustered in two areas of analysis and seven respective supercategories. The internal environment area includes four levels of analysis: ‘business and structure’ (focusing on financial aspects and firm characteristics), ‘organizational effectiveness’ (focusing on human-centred factors), ‘technology and system’ (based on the IT infrastructure and systems) as well as ‘data management’ (including all data related factors). Three categories are assigned to the external environment: the ‘regulatory environment’ (such as ethics and the regulatory forms), the ‘business environment’ (focused on cooperation activities and dynamics in the firm environment) and the ‘economic environment’ (related to sectoral and country specifics).

Third, the developed research plan as outlined in Table  2 serves as an additional outcome of the SLR, identifying key RQs in the analyzed areas that can serve as a foundation for researchers to expand the research area of AI adoption in production. These RQs are related to the mostly cited factors analyzed in our SLR and aim to broaden the understanding on the emerging topic.

The resulting insights can serve as the basis for strategic decisions by production companies looking to integrate AI into their processes. Our findings on the factors influencing AI adoption as well as the developed research agenda enhance the practical understanding of a production-specific adoption. Hence, they can serve as the basis for strategic decisions for companies on the path to an effective AI adoption. Managers can, for example, analyse the individual factors in light of their company as well as take necessary steps to develop further aspects in a targeted manner. Researchers, on the other hand, can use the future research agenda in order to assess open RQs and can expand the state of research on AI adoption in production.

4.2 Limitations

Since a literature review must be restricted in its scope in order to make the analyses feasible, our study provides a starting point for further research. Hence, there is a need for further qualitative and quantitative empirical research on the heterogeneous nature of how firms configure their AI adoption process. Along these lines, the following aspects would be of particular interest for future research to improve and further validate the analytical power of the proposed framework.

First, the lack of research on AI adoption in production leads to a limited number of papers included in this SLR. As visualized in Fig.  2 , the number of publications related to the adoption of AI in production has been increasing since 2018 but is, to date, still at an early stage. For this reason, only 47 papers published until May 2024 addressing the production-specific adoption of AI were identified and therefore included in our analysis for in-depth investigation. This rather small number of papers included in the full-text analysis gives a limited view on AI adoption in production but allows a more detailed analysis. As the number of publications in this research field increases, there seems to be a lot of research happening in this field which is why new findings might be constantly added and developed as relevant in the future [ 39 ]. Moreover, in order to research AI adoption from a more practical perspective and thus to build up a broader, continuously updated view on AI adoption in production, future literature analyses could include other publication formats, e.g. study reports of research institutions and companies, as well discussion papers.

Second, the scope of the application areas of AI in production has been increasing rapidly. Even though our overview of the three main areas covered in the recent literature serves as a good basis for identifying the most dominant fields for AI adoption in production, a more detailed analysis could provide a better overview of possibilities for manufacturing companies. Hence, a further systematisation as well as evaluation of application areas for AI in production can provide managers with the information needed to decide where AI applications might be of interest for the specific company needs.

Third, the systematisation of the 35 factors influencing AI adoption in production serve as a good ground for identifying relevant areas influenced by and in turn influencing the adoption of AI. Further analyses should be conducted in order to extend this view and extend the framework. For example, our review could be combined with explorative research methods (such as case studies in production firms) in order to add the practical insights from firms adopting AI. This integration of practical experiences can also help exploit and monitor more AI-specific factors by observing AI adoption processes. In enriching the factors through in-depth analyses, the results of the identified AI adoption factors could also be examined in light of theoretical contributions like the technology-organization-environment (TOE) framework [ 47 ] and other adoption theories.

Fourth, in order to examine the special relevance of identified factors for AI adoption process and thus to distinguish it from the common factors influencing the adoption of more general digital technologies, there is a further need for more in-depth (ethnographic) research into their impacts on the adoption processes, particularly in the production context. Similarly, further research could use the framework introduced in this paper as a basis to develop new indicators and measurement concepts as well as to examine their impacts on production performance using quantitative methods.

Benner MJ, Waldfogel J (2020) Changing the channel: digitization and the rise of “middle tail” strategies. Strat Mgmt J 86:1–24. https://doi.org/10.1002/smj.3130

Article   Google Scholar  

Roblek V, Meško M, Krapež A (2016) A complex view of industry 4.0. SAGE Open. https://doi.org/10.1177/2158244016653987

Oliveira BG, Liboni LB, Cezarino LO et al (2020) Industry 4.0 in systems thinking: from a narrow to a broad spectrum. Syst Res Behav Sci 37:593–606. https://doi.org/10.1002/sres.2703

Li B, Hou B, Yu W et al (2017) Applications of artificial intelligence in intelligent manufacturing: a review. Frontiers Inf Technol Electronic Eng 18:86–96. https://doi.org/10.1631/FITEE.1601885

Dhamija P, Bag S (2020) Role of artificial intelligence in operations environment: a review and bibliometric analysis. TQM 32:869–896. https://doi.org/10.1108/TQM-10-2019-0243

Collins C, Dennehy D, Conboy K et al (2021) Artificial intelligence in information systems research: a systematic literature review and research agenda. Int J Inf Manage 60:102383. https://doi.org/10.1016/j.ijinfomgt.2021.102383

Chien C-F, Dauzère-Pérès S, Huh WT et al (2020) Artificial intelligence in manufacturing and logistics systems: algorithms, applications, and case studies. Int J Prod Res 58:2730–2731. https://doi.org/10.1080/00207543.2020.1752488

Chen H (2019) Success factors impacting artificial intelligence adoption: perspective from the telecom industry in China, Old Dominion University

Sanchez M, Exposito E, Aguilar J (2020) Autonomic computing in manufacturing process coordination in industry 4.0 context. J Industrial Inf Integr. https://doi.org/10.1016/j.jii.2020.100159

Lee J, Davari H, Singh J et al (2018) Industrial artificial intelligence for industry 4.0-based manufacturing systems. Manufacturing Letters 18:20–23. https://doi.org/10.1016/j.mfglet.2018.09.002

Heimberger H, Horvat D, Schultmann F (2023) Assessing AI-readiness in production—A conceptual approach. In: Huang C-Y, Dekkers R, Chiu SF et al. (eds) intelligent and transformative production in pandemic times. Springer, Cham, pp 249–257

Horvat D, Heimberger H (2023) AI Readiness: An Integrated Socio-technical Framework. In: Deschamps F, Pinheiro de Lima E, Da Gouvêa Costa SE et al. (eds) Proceedings of the 11 th international conference on production research—Americas: ICPR Americas 2022, 1 st ed. 2023. Springer Nature Switzerland; Imprint Springer, Cham, pp 548–557

Wang J, Ma Y, Zhang L et al (2018) Deep learning for smart manufacturing: methods and applications. J Manuf Syst 48:144–156. https://doi.org/10.1016/J.JMSY.2018.01.003

Davenport T, Guha A, Grewal D et al (2020) How artificial intelligence will change the future of marketing. J Acad Mark Sci 48:24–42. https://doi.org/10.1007/s11747-019-00696-0

Cui R, Li M, Zhang S (2022) AI and procurement. Manufacturing Serv Operations Manag 24(691):706. https://doi.org/10.1287/msom.2021.0989

Pournader M, Ghaderi H, Hassanzadegan A et al (2021) Artificial intelligence applications in supply chain management. Int J Prod Econ 241:108250. https://doi.org/10.1016/j.ijpe.2021.108250

Su H, Li L, Tian S et al (2024) Innovation mechanism of AI empowering manufacturing enterprises: case study of an industrial internet platform. Inf Technol Manag. https://doi.org/10.1007/s10799-024-00423-4

Venkatesh V, Raman R, Cruz-Jesus F (2024) AI and emerging technology adoption: a research agenda for operations management. Int J Prod Res 62:5367–5377. https://doi.org/10.1080/00207543.2023.2192309

Senoner J, Netland T, Feuerriegel S (2022) Using explainable artificial intelligence to improve process quality: evidence from semiconductor manufacturing. Manage Sci 68:5704–5723. https://doi.org/10.1287/mnsc.2021.4190

Fosso Wamba S, Queiroz MM, Ngai EWT et al (2024) The interplay between artificial intelligence, production systems, and operations management resilience. Int J Prod Res 62:5361–5366. https://doi.org/10.1080/00207543.2024.2321826

Uren V, Edwards JS (2023) Technology readiness and the organizational journey towards AI adoption: an empirical study. Int J Inf Manage 68:102588. https://doi.org/10.1016/j.ijinfomgt.2022.102588

Berente N, Gu B, Recker J (2021) Managing artificial intelligence special issue managing AI. MIS Quarterly 45:1433–1450

Google Scholar  

Scafà M, Papetti A, Brunzini A et al (2019) How to improve worker’s well-being and company performance: a method to identify effective corrective actions. Procedia CIRP 81:162–167. https://doi.org/10.1016/j.procir.2019.03.029

Wang H, Qiu F (2023) AI adoption and labor cost stickiness: based on natural language and machine learning. Inf Technol Manag. https://doi.org/10.1007/s10799-023-00408-9

Lindebaum D, Vesa M, den Hond F (2020) Insights from “the machine stops ” to better understand rational assumptions in algorithmic decision making and its implications for organizations. Acad Manag Rev 45:247–263. https://doi.org/10.5465/amr.2018.0181

Baskerville RL, Myers MD, Yoo Y (2020) Digital first: the ontological reversal and new challenges for information systems research. MIS Quarterly 44:509–523

Frey CB, Osborne MA (2017) The future of employment: How susceptible are jobs to computerisation? Technol Forecast Soc Chang 114:254–280. https://doi.org/10.1016/J.TECHFORE.2016.08.019

Jarrahi MH (2018) Artificial intelligence and the future of work: human-AI symbiosis in organizational decision making. Bus Horiz 61:577–586. https://doi.org/10.1016/j.bushor.2018.03.007

Fügener A, Grahl J, Gupta A et al (2021) Will humans-in-the-loop become borgs? Merits and pitfalls of working with AI. MIS Quarterly 45:1527–1556

Klumpp M (2018) Automation and artificial intelligence in business logistics systems: human reactions and collaboration requirements. Int J Log Res Appl 21:224–242. https://doi.org/10.1080/13675567.2017.1384451

Schrettenbrunnner MB (2020) Artificial-Intelligence-driven management. IEEE Eng Manag Rev 48:15–19. https://doi.org/10.1109/EMR.2020.2990933

Li J, Li M, Wang X et al (2021) Strategic directions for AI: the role of CIOs and boards of directors. MIS Quarterly 45:1603–1644

Brock JK-U, von Wangenheim F (2019) Demystifying AI: What digital transformation leaders can teach you about realistic artificial intelligence. Calif Manage Rev 61:110–134. https://doi.org/10.1177/1536504219865226

Lee J, Suh T, Roy D et al (2019) Emerging technology and business model innovation: the case of artificial intelligence. JOItmC 5:44. https://doi.org/10.3390/joitmc5030044

Chen J, Tajdini S (2024) A moderated model of artificial intelligence adoption in firms and its effects on their performance. Inf Technol Manag. https://doi.org/10.1007/s10799-024-00422-5

Kinkel S, Baumgartner M, Cherubini E (2022) Prerequisites for the adoption of AI technologies in manufacturing—evidence from a worldwide sample of manufacturing companies. Technovation 110:102375. https://doi.org/10.1016/j.technovation.2021.102375

Mikalef P, Gupta M (2021) Artificial intelligence capability: Conceptualization, measurement calibration, and empirical study on its impact on organizational creativity and firm performance. Inf Manag 58:103434. https://doi.org/10.1016/j.im.2021.103434

McElheran K, Li JF, Brynjolfsson E et al (2024) AI adoption in America: Who, what, and where. Economics Manag Strategy 33:375–415. https://doi.org/10.1111/jems.12576

Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag 14:207–222. https://doi.org/10.1111/1467-8551.00375

Cooper H, Hedges LV, Valentine JC (2009) Handbook of research synthesis and meta-analysis. Russell Sage Foundation, New York

Page MJ, McKenzie JE, Bossuyt PM et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. https://doi.org/10.1136/bmj.n71

Denyer D, Tranfield D (2011) Producing a systematic review. In: Buchanan DA, Bryman A (eds) The Sage handbook of organizational research methods. Sage Publications Inc, Thousand Oaks, CA, pp 671–689

Burbidge JL, Falster P, Riis JO et al (1987) Integration in manufacturing. Comput Ind 9:297–305. https://doi.org/10.1016/0166-3615(87)90103-5

Mayring P (2000) Qualitative content analysis. Forum qualitative Sozialforschung/Forum: Qualitative social research, Vol 1, No 2 (2000): Qualitative methods in various disciplines I: Psychology. https://doi.org/10.17169/fqs-1.2.1089

Hsieh H-F, Shannon SE (2005) Three approaches to qualitative content analysis. Qual Health Res 15:1277–1288. https://doi.org/10.1177/1049732305276687

Miles MB, Huberman AM (2009) Qualitative data analysis: An expanded sourcebook, 2nd edn. Sage, Thousand Oaks, Calif

Tornatzky LG, Fleischer M (1990) The processes of technological innovation. Issues in organization and management series. Lexington Books, Lexington, Mass.

Alsheibani S, Cheung Y, Messom C (2018) Artificial Intelligence Adoption: AI-readiness at Firm-Level: Research-in-Progress. Twenty-Second Pacific Asia Conference on Information Systems

Akinsolu MO (2023) Applied artificial intelligence in manufacturing and industrial production systems: PEST considerations for engineering managers. IEEE Eng Manag Rev 51:52–62. https://doi.org/10.1109/EMR.2022.3209891

Bettoni A, Matteri D, Montini E et al (2021) An AI adoption model for SMEs: a conceptual framework. IFAC-PapersOnLine 54:702–708. https://doi.org/10.1016/j.ifacol.2021.08.082

Boavida N, Candeias M (2021) Recent automation trends in portugal: implications on industrial productivity and employment in automotive sector. Societies 11:101. https://doi.org/10.3390/soc11030101

Botha AP (2019) A mind model for intelligent machine innovation using future thinking principles. Jnl of Manu Tech Mnagmnt 30:1250–1264. https://doi.org/10.1108/JMTM-01-2018-0021

Chatterjee S, Rana NP, Dwivedi YK et al (2021) Understanding AI adoption in manufacturing and production firms using an integrated TAM-TOE model. Technol Forecast Soc Chang 170:120880. https://doi.org/10.1016/j.techfore.2021.120880

Chiang LH, Braun B, Wang Z et al (2022) Towards artificial intelligence at scale in the chemical industry. AIChE J. https://doi.org/10.1002/aic.17644

Chouchene A, Carvalho A, Lima TM et al. (2020) Artificial intelligence for product quality inspection toward smart industries: quality control of vehicle Non-conformities. In: Garengo P (ed) 2020 9th International Conference on Industrial Technology and Management: ICITM 2020 February 11–13, 2020, Oxford, United Kingdom. IEEE, pp 127–131

Corti D, Masiero S, Gladysz B (2021) Impact of Industry 4.0 on Quality Management: identification of main challenges towards a Quality 4.0 approach. In: 2021 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC). IEEE, pp 1–8

Demlehner Q, Schoemer D, Laumer S (2021) How can artificial intelligence enhance car manufacturing? A Delphi study-based identification and assessment of general use cases. Int J Inf Manage 58:102317. https://doi.org/10.1016/j.ijinfomgt.2021.102317

Dohale V, Akarte M, Gunasekaran A et al (2022) (2022) Exploring the role of artificial intelligence in building production resilience: learnings from the COVID-19 pandemic. Int J Prod Res 10(1080/00207543):2127961

Drobot AT (2020) Industrial Transformation and the Digital Revolution: A Focus on artificial intelligence, data science and data engineering. In: 2020 ITU Kaleidoscope: Industry-Driven Digital Transformation (ITU K). IEEE, pp 1–11

Ghani EK, Ariffin N, Sukmadilaga C (2022) Factors influencing artificial intelligence adoption in publicly listed manufacturing companies: a technology, organisation, and environment approach. IJAEFA 14:108–117

Hammer A, Karmakar S (2021) Automation, AI and the future of work in India. ER 43:1327–1341. https://doi.org/10.1108/ER-12-2019-0452

Hartley JL, Sawaya WJ (2019) Tortoise, not the hare: digital transformation of supply chain business processes. Bus Horiz 62:707–715. https://doi.org/10.1016/j.bushor.2019.07.006

Kyvik Nordås H, Klügl F (2021) Drivers of automation and consequences for jobs in engineering services: an agent-based modelling approach. Front Robot AI 8:637125. https://doi.org/10.3389/frobt.2021.637125

Mubarok K, Arriaga EF (2020) Building a smart and intelligent factory of the future with industry 4.0 technologies. J Phys Conf Ser. https://doi.org/10.1088/1742-6596/1569/3/032031

Muriel-Pera YdJ, Diaz-Piraquive FN, Rodriguez-Bernal LP et al. (2018) Adoption of strategies the fourth industrial revolution by micro, small and medium enterprises in bogota D.C. In: Lozano Garzón CA (ed) 2018 Congreso Internacional de Innovación y Tendencias en Ingeniería (CONIITI). IEEE, pp 1–6

Olsowski S, Schlögl S, Richter E et al. (2022) Investigating the Potential of AutoML as an Instrument for Fostering AI Adoption in SMEs. In: Uden L, Ting I-H, Feldmann B (eds) Knowledge Management in Organisations: 16th International Conference, KMO 2022, Hagen, Germany, July 11–14, 2022, Proceedings, 1st ed. 2022, vol 1593. Springer, Cham, pp 360–371

Rodríguez-Espíndola O, Chowdhury S, Dey PK et al (2022) Analysis of the adoption of emergent technologies for risk management in the era of digital manufacturing. Technol Forecast Soc Chang 178:121562. https://doi.org/10.1016/j.techfore.2022.121562

Schkarin T, Dobhan A (2022) Prerequisites for Applying Artificial Intelligence for Scheduling in Small- and Medium-sized Enterprises. In: Proceedings of the 24 th International Conference on Enterprise Information Systems. SCITEPRESS—Science and Technology Publications, pp 529–536

Sharma P, Shah J, Patel R (2022) Artificial intelligence framework for MSME sectors with focus on design and manufacturing industries. Mater Today: Proc 62:6962–6966. https://doi.org/10.1016/j.matpr.2021.12.360

Siaterlis G, Nikolakis N, Alexopoulos K et al. (2022) Adoption of AI in EU Manufacturing. Gaps and Challenges. In: Katalinic B (ed) Proceedings of the 33 rd International DAAAM Symposium 2022, vol 1. DAAAM International Vienna, pp 547–550

Tariq MU, Poulin M, Abonamah AA (2021) Achieving operational excellence through artificial intelligence: driving forces and barriers. Front Psychol 12:686624. https://doi.org/10.3389/fpsyg.2021.686624

Trakadas P, Simoens P, Gkonis P et al (2020) An artificial intelligence-based collaboration approach in industrial IoT manufacturing: key concepts. Architectural Ext Potential Applications Sens. https://doi.org/10.3390/s20195480

Vernim S, Bauer H, Rauch E et al (2022) A value sensitive design approach for designing AI-based worker assistance systems in manufacturing. Procedia Computer Sci 200:505–516. https://doi.org/10.1016/j.procs.2022.01.248

Williams G, Meisel NA, Simpson TW et al (2022) Design for artificial intelligence: proposing a conceptual framework grounded in data wrangling. J Computing Inf Sci Eng 10(1115/1):4055854

Wuest T, Romero D, Cavuoto LA et al (2020) Empowering the workforce in Post–COVID-19 smart manufacturing systems. Smart Sustain Manuf Syst 4:20200043. https://doi.org/10.1520/SSMS20200043

Javaid M, Haleem A, Singh RP (2023) A study on ChatGPT for Industry 4.0: background, potentials, challenges, and eventualities. J Economy Technol 1:127–143. https://doi.org/10.1016/j.ject.2023.08.001

Rathore AS, Nikita S, Thakur G et al (2023) Artificial intelligence and machine learning applications in biopharmaceutical manufacturing. Trends Biotechnol 41:497–510. https://doi.org/10.1016/j.tibtech.2022.08.007

Jan Z, Ahamed F, Mayer W et al (2023) Artificial intelligence for industry 4.0: systematic review of applications, challenges, and opportunities. Expert Syst Applications 216:119456

Waschull S, Emmanouilidis C (2023) Assessing human-centricity in AI enabled manufacturing systems: a socio-technical evaluation methodology. IFAC-PapersOnLine 56:1791–1796. https://doi.org/10.1016/j.ifacol.2023.10.1891

Stohr A, Ollig P, Keller R et al (2024) Generative mechanisms of AI implementation: a critical realist perspective on predictive maintenance. Inf Organ 34:100503. https://doi.org/10.1016/j.infoandorg.2024.100503

Pazhayattil AB, Konyu-Fogel G (2023) ML and AI Implementation Insights for Bio/Pharma Manufacturing. BioPharm International 36:24–29

Ronaghi MH (2023) The influence of artificial intelligence adoption on circular economy practices in manufacturing industries. Environ Dev Sustain 25:14355–14380. https://doi.org/10.1007/s10668-022-02670-3

Rath SP, Tripathy R, Jain NK (2024) Assessing the factors influencing the adoption of generative artificial intelligence (GenAI) in the manufacturing sector. In: Sharma SK, Dwivedi YK, Metri B et al (eds) Transfer, diffusion and adoption of next-generation digital technologies, vol 697. Springer Nature Switzerland, Cham

Bonnard R, Da Arantes MS, Lorbieski R et al (2021) Big data/analytics platform for Industry 4.0 implementation in advanced manufacturing context. Int J Adv Manuf Technol 117:1959–1973. https://doi.org/10.1007/s00170-021-07834-5

Confalonieri M, Barni A, Valente A et al. (2015) An AI based decision support system for preventive maintenance and production optimization in energy intensive manufacturing plants. In: 2015 IEEE international conference on engineering, technology and innovation/ international technology management conference (ICE/ITMC). IEEE, pp 1–8

Dubey R, Gunasekaran A, Childe SJ et al (2020) Big data analytics and artificial intelligence pathway to operational performance under the effects of entrepreneurial orientation and environmental dynamism: a study of manufacturing organisations. Int J Prod Econ 226:107599. https://doi.org/10.1016/j.ijpe.2019.107599

Lee J, Singh J, Azamfar M et al (2020) Industrial AI: a systematic framework for AI in industrial applications. China Mechanical Eng 31:37–48

Turner CJ, Emmanouilidis C, Tomiyama T et al (2019) Intelligent decision support for maintenance: an overview and future trends. Int J Comput Integr Manuf 32:936–959. https://doi.org/10.1080/0951192X.2019.1667033

Agostinho C, Dikopoulou Z, Lavasa E et al (2023) Explainability as the key ingredient for AI adoption in Industry 5.0 settings. Front Artif Intell. https://doi.org/10.3389/frai.2023.1264372

Csiszar A, Hein P, Wachter M et al. (2020) Towards a user-centered development process of machine learning applications for manufacturing domain experts. In: 2020 third international conference on artificial intelligence for industries (AI4I). IEEE, pp 36–39

Merhi MI (2023) Harfouche A (2023) Enablers of artificial intelligence adoption and implementation in production systems. Int J Prod Res. https://doi.org/10.1080/00207543.2023.2167014

Demlehner Q, Laumer S (2024) How the terminator might affect the car manufacturing industry: examining the role of pre-announcement bias for AI-based IS adoptions. Inf Manag 61:103881. https://doi.org/10.1016/j.im.2023.103881

Ghobakhloo M, Ching NT (2019) Adoption of digital technologies of smart manufacturing in SMEs. J Ind Inf Integr 16:100107. https://doi.org/10.1016/j.jii.2019.100107

Binsaeed RH, Yousaf Z, Grigorescu A et al (2023) Knowledge sharing key issue for digital technology and artificial intelligence adoption. Systems 11:316. https://doi.org/10.3390/systems11070316

Papadopoulos T, Sivarajah U, Spanaki K et al (2022) Editorial: artificial Intelligence (AI) and data sharing in manufacturing, production and operations management research. Int J Prod Res 60:4361–4364. https://doi.org/10.1080/00207543.2021.2010979

Chirumalla K (2021) Building digitally-enabled process innovation in the process industries: a dynamic capabilities approach. Technovation 105:102256. https://doi.org/10.1016/j.technovation.2021.102256

Fragapane G, Ivanov D, Peron M et al (2022) Increasing flexibility and productivity in Industry 4.0 production networks with autonomous mobile robots and smart intralogistics. Ann Oper Res 308:125–143. https://doi.org/10.1007/s10479-020-03526-7

Shahbazi Z, Byun Y-C (2021) Integration of Blockchain, IoT and machine learning for multistage quality control and enhancing security in smart manufacturing. Sensors (Basel). https://doi.org/10.3390/s21041467

Javaid M, Haleem A, Singh RP et al (2021) Significance of sensors for industry 4.0: roles, capabilities, and applications. Sensors Int 2:100110. https://doi.org/10.1016/j.sintl.2021.100110

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Business Unit Industrial Change and New Business Models, Competence Center Innovation and Knowledge Economy, Fraunhofer Institute for Systems and Innovation Research ISI, Breslauer Straße 48, 76139, Karlsruhe, Germany

Heidi Heimberger, Djerdj Horvat & Frank Schultmann

Karlsruhe Institute for Technology KIT, Institute for Industrial Production (IIP) - Chair of Business Administration, Production and Operations Management, Hertzstraße 16, 76187, Karlsruhe, Germany

Heidi Heimberger

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Heidi Heimberger .

Ethics declarations

Conflict of interest.

The authors report no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Heimberger, H., Horvat, D. & Schultmann, F. Exploring the factors driving AI adoption in production: a systematic literature review and future research agenda. Inf Technol Manag (2024). https://doi.org/10.1007/s10799-024-00436-z

Download citation

Accepted : 10 August 2024

Published : 23 August 2024

DOI : https://doi.org/10.1007/s10799-024-00436-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Technology adoption
• AI adoption
• Adoption factors
• Systematic literature review
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 21 August 2024

Evaluating the effectiveness of large language models in abstract screening: a comparative analysis

• Michael Li 1 ,
• Jianping Sun 2 &
• Xianming Tan   ORCID: orcid.org/0000-0002-5478-2269 1 , 3  

Systematic Reviews volume  13 , Article number:  219 ( 2024 ) Cite this article

95 Accesses

Metrics details

This study aimed to evaluate the performance of large language models (LLMs) in the task of abstract screening in systematic review and meta-analysis studies, exploring their effectiveness, efficiency, and potential integration into existing human expert-based workflows.

We developed automation scripts in Python to interact with the APIs of several LLM tools, including ChatGPT v4.0, ChatGPT v3.5, Google PaLM 2, and Meta Llama 2, and latest tools including ChatGPT v4.0 turbo, ChatGPT v3.5 turbo, Google Gemini 1.0 pro, Meta Llama 3, and Claude 3. This study focused on three databases of abstracts and used them as benchmarks to evaluate the performance of these LLM tools in terms of sensitivity, specificity, and overall accuracy. The results of the LLM tools were compared to human-curated inclusion decisions, gold standard for systematic review and meta-analysis studies.

Different LLM tools had varying abilities in abstract screening. Chat GPT v4.0 demonstrated remarkable performance, with balanced sensitivity and specificity, and overall accuracy consistently reaching or exceeding 90%, indicating a high potential for LLMs in abstract screening tasks. The study found that LLMs could provide reliable results with minimal human effort and thus serve as a cost-effective and efficient alternative to traditional abstract screening methods.

While LLM tools are not yet ready to completely replace human experts in abstract screening, they show great promise in revolutionizing the process. They can serve as autonomous AI reviewers, contribute to collaborative workflows with human experts, and integrate with hybrid approaches to develop custom tools for increased efficiency. As technology continues to advance, LLMs are poised to play an increasingly important role in abstract screening, reshaping the workflow of systematic review and meta-analysis studies.

Peer Review reports

Introduction

A systematic review is a rigorous form of research that collates and synthesizes all existing evidence on a specific research question [ 1 ]. It stands as a cornerstone not just in medical research but across diverse academic disciplines. Unlike traditional literature reviews, systematic reviews follow a comprehensive and standardized process, such as the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline [ 2 ], designed to minimize bias and ensure reproducibility. Therefore, these reviews are recognized as one of the zenith levels of evidence in evidence-based research [ 3 ] and play a pivotal role in shaping clinical guidelines, healthcare policies, and informing medical decisions [ 4 ].

Commencing with a well-articulated research question, a typical systematic review launches an exhaustive search strategy that subsequently sweeps through databases, such as PubMed and Embase, supplemented by additional sources such as clinical trial registries and pertinent article reference lists, with aims to capture all relevant studies and mitigate bias. Predetermined inclusion and exclusion criteria guide the subsequent screening and selection of studies, encompassing facts like study design, patient demographics, and intervention types. Reviewers, working independently, appraise each study’s eligibility, reconciling disagreements through discussions or third-party reviews. Data extraction and synthesis follow, either through meta-analysis or narrative synthesis, depending on the heterogeneity of the selected studies.

The practice of conducting systematic reviews has gained substantial popularity with considerable demand within the academic community. A notable reference [ 5 ] to this trend found that approximately 75 trials and potentially 11 systematic reviews are disseminated daily, based on data from around 2010. Moreover, a query for “systematic review” in the Google Scholar database yields approximately 17,000 entries that have been published within the year 2023 alone, which translates into about 51 systematic reviews per day, as observed on November 26, 2023. This expanding volume of literature underscores the critical role that systematic reviews play in the consolidation of research findings across various fields of study.

Despite their pivotal role, executing systematic reviews remains a formidable task due to the abstract screening, a key phase that can be overwhelmingly time-consuming due to its volume. For example, in Polanin et al. [ 6 ], the authors reported that the research staff screened 29,846 abstracts independently (14,923 unique citations were double screened) over the course of 189 days. In addition, the variability in reporting standards, use of jargon, and varied study designs can further complicate the abstract screening process [ 7 ]. Nevertheless, the repetitive nature of the task, combined with cognitive fatigue, can lead to human errors [ 8 , 9 ]. Recent advancements in machine learning (ML) and deep learning propose possible solutions to these challenges. However, traditional ML models, while promising, require domain-specific training, a time-consuming process that often demands manual labeling of datasets [ 10 ].

Advancements in natural language processing (NLP) and artificial intelligence (AI) are opening doors to address challenges in systematic reviews. Large language models (LLMs) like ChatGPT [ 11 ], PaLM [ 12 ], Llama [ 13 ], and Claude [ 14 ] are capturing the research community’s attention. Their collective potential, especially their capability to operate without exhaustive domain-specific training, makes them prime candidates for revolutionizing the systematic review process.

While each of the aforementioned AI tools brings its unique capabilities to the table, the fundamental question remains: How do they stack up, both individually and collectively, in the realm of abstract screening, against the human expert-based process? In pursuit of answers, this research seeks to thoroughly investigate the potential of ChatGPT, Google PaLM, Llama, and Claude in automating the crucial abstract screening phase integral to systematic reviews. Our goal is to rigorously compare the performance of these advanced AI-driven methods with existing machine learning (ML)-based approaches. In doing so, we aim to develop AI strategies that masterfully balance efficiency and accuracy with minimal human intervention, ultimately transforming systematic review practice across disciplines.

The use of NLP for abstract screening is established [ 10 , 15 ]. However, the application of LLMs specifically for this task is a nascent field [ 16 , 17 ]. This emerging area offers significant potential to improve efficiency and accuracy. Our study aims to fill this gap by providing a comprehensive analysis of LLM capabilities in abstract screening, laying the groundwork for future research and application. This is particularly relevant considering the rapid evolution of this technology, highlighting its potential to streamline systematic reviews now and in the future.

The remainder of this paper is structured to provide a comprehensive exploration of our topic. We begin with an in-depth examination of existing methods for abstract screening, including both manual and NLP-based approaches, laying the groundwork for understanding the current state of the field. We then introduce the use of large language model (LLM) tools for abstract screening, detailing our experimental design to meticulously evaluate their performance in this context. Subsequent sections present our empirical findings and results, shedding light on the capabilities and limitations of the AI tools in question. Finally, we engage in a thoughtful discussion, reflecting on the implications of our findings and considering the future trajectory of abstract screening in systematic reviews.

Existing approaches to abstract screening in systematic reviews

In the vast realm of systematic reviews, the critical task of abstract screening serves as a foundational step in curating the highest quality of evidence [ 2 ]. However, this process often presents significant challenges due to the involvement of sifting through large volumes of literature to identify those that align with predefined criteria. Over time, various methodologies, ranging from manual evaluations to sophisticated AI-driven techniques, have been proposed to address the complexities of this task. In this section, we first describe the existing approaches on their operational mechanisms and associated advantages and disadvantages.

Manual approach

Historically, the process of abstract screening was firmly rooted in manual evaluations. In this conventional approach, individual reviewers would scrutinize each abstract against predefined criteria [ 1 ]. The meticulous nature of this method required that multiple experts independently evaluate the literature to ensure both reliability and reduced biases [ 8 ]. While the depth of human expertise brought about nuanced understanding, the manual nature of this method made it both time-consuming and, at times, prone to human error [ 6 , 9 ].

NLP-based approach

As technology evolved, the field witnessed the incorporation of natural language processing (NLP) to automate abstract screening [ 10 ]. In this framework, abstract text undergoes preprocessing and vectorization. Supervised machine learning models, notably the support vector machine (SVM) and the random forest (RF), are then trained on this vectorized data to classify literature based on specific criteria [ 15 ]. The strength of this approach lies in its potential for efficiency. However, its efficacy and accuracy hinges heavily on the availability of a well-curated, labeled training set.

Zero-shot classification

A more recent and innovative approach is zero-shot classification, which was notably highlighted by Xian et al. [ 18 ]. Eschewing the need for an extensive labeled training dataset, zero-shot classification offers the allure of categorizing abstracts without prior domain-specific training. By calculating a probability score for each abstract, researchers obtain a dynamic measure of its alignment with predefined categories. While it does not require training for the machine learning model, the tradeoff is a loss of sensitivity and potential omissions of pertinent studies [ 19 ]. In this study, for a given set of abstracts, we first obtain embeddings (using OpenAI’s text embedding API, the babbage model) for abstracts and for a pre-specified description of an ideal study to include. We indeed use the inclusion/exclusion criteria (see “Examples of abstract screening by using LLMs,” “Automated workflow for streamlining abstract screening via ChatGPT and other tools,” and “ Study design ” sections) as the description of an ideal study to include. We then compute the cosine similarity score between each abstract (its embedding) and the pre-specified description (its embedding). We claim the top 10% (in terms of similarity) of abstracts as positive (i.e., a study should be included).

Hybrid approach

To reconcile the strengths and limitations of these models, a hybrid methodology has emerged [ 18 ]. It cleverly marries the rapid categorization capabilities of zero-shot classification with the precision of traditional machine learning models such as SVM and RF. Here, zero-shot classification provides an initial curation of all abstracts in the training set. Then, based on zero-shot classification’s curation, the abstracts with classification score higher than a pre-specified threshold are selected to undergo manual review to rectify zero-shot classification. The rectified classification then serves as a foundation for training traditional models, which are then employed for screening the broader dataset (e.g., in the testing set). The hybrid approach balances the speed of zero-shot and the precision of traditional ML, and potentially offers enhanced accuracy at reduced human efforts. However, this approach involves the use of multiple methodologies and still relies on well-curated, labeled training data (in this case, a subset of the whole training set). In this study, for a given set of abstracts, we rectify the “positive” abstracts (i.e., the top 10% of abstracts) identified by zero-shot, and then fit a classification model for rectified labels to the text embeddings of the corresponding abstracts using SVM. This learned classification model is then used to predict the class of other abstracts. We added the R code for the zero-shot and hybrid approaches to the GitHub repository https://github.com/mikeli380/LLMAbstractScreening .

Active learning approach

Active learning [ 20 , 21 , 22 ] is an innovative approach to machine learning that optimizes the training process by allowing the model to selectively query a human annotator for labels on the most informative data points. This method is particularly advantageous in scenarios where labeled data is scarce or expensive to obtain. Active learning models can start with minimal datasets, often requiring only one relevant and one irrelevant abstract, making them particularly suitable for tasks such as abstract screening, where the acquisition of large labeled datasets can be prohibitive.

The main advantage of active learning is its efficiency. By focusing on the most informative samples, it reduces the amount of data that needs to be labeled while still effectively training the model. This can significantly reduce the time and resources required for the annotation process. However, the effectiveness of active learning depends heavily on the initial selection of samples and the criteria used to determine the informativeness of subsequent data points. If not well calibrated, the model may request labels for data that do not significantly improve its performance, resulting in an inefficient use of resources. In addition, the iterative nature of querying and updating the model based on new labels can introduce complexity into the training process, potentially requiring more sophisticated infrastructure and oversight than traditional supervised learning methods.

While this study did not test active learning approaches for abstract screening, readers interested in exploring this methodology further are referred to [ 23 ] for detailed information on the application of active learning in abstract screening contexts.

• Large language models

Amidst the evolving methodologies described in the “ Existing approaches to abstract screening in systematic reviews ” section, modern AI tools based on large language models (LLMs), such as ChatGPT, PaLM, Llama, and Claude, are emerging as potential game-changers. Grounded in advanced language processing capabilities, these tools can be tailored to evaluate abstracts against nuanced criteria and offer detailed assessments and classifications. Their prowess signals transformative potential for abstract screening. In this section, we first present two examples to illustrate the potential of LLMs in the context of abstract screening, and then precede with our in-depth discussion on the study’s workflow and design that aims to critically investigate the performance of LLMs in this domain.

Examples of abstract screening by using LLMs

In this subsection, we demonstrate the process of LLM-based abstract screening using ChatGPT with two specified abstracts: (1) Millard et al. [ 24 ] and (2) Zawia et al. [ 25 ]. Using other LLM tools such as Llama, Google PaLM, or Claude for abstract screening is similar and will be skipped for brevity.

In constructing the prompts, we adopted a standardized approach to mimic a typical interaction between a senior researcher and a research assistant. Each prompt was designed to include three critical elements: a brief statement of the topic under review, the exact eligibility criteria as specified in the study guidelines, and the text of the abstract to be reviewed. This approach was designed to simulate a realistic and practical scenario that reflects common academic practices. In addition, to ensure consistency and objectivity in the responses, we required that the results of the LLM be in a binary “yes” or “no” format, directly indicating whether each abstract met the inclusion criteria. This format was chosen to eliminate any ambiguity in the model’s decisions and to facilitate straightforward analysis of its accuracy.

Abstract (1)

ChatGPT prompt: “I would like you to help me with conducting a systematic review on animal depression. I will provide the title and abstract for one journal article and would like you to screen the paper for inclusion. Here are the inclusion criteria: Does the study provide primary data of an animal model of depression or depressive-like phenotype with an appropriate control group? Here is the abstract:

Wistar-Kyoto (WKY) rats show endogenous depressive behavior that can be reversed by antidepressants. Given that WKYs exhibit decreased sensitivity to some antidepressants and treatment-resistant depressed patients often show hypothalamic-pituitary-thyroid (HPT) dysregulation, we examined the behavioral and HPT hormonal responses of WKYs to altered thyroid status. “Euthyroid” WKYs had elevated basal plasma TSH and T(3) levels as compared to Wistars. Hypothyroidism increased TSH levels more in WKYs than in Wistars and increased response latency in the open field test (OFT) of WKYs only. Administration of T(4) and T(3) suppressed plasma TSH equally in both strains. Wistars responded to increased T(3) levels with decreased response latency and increased activity in the OFT, but increased immobility in the forced swim test. In contrast, WKYs responded only to the high T(3) levels with decreased response latency in the OFT. These results suggest the existence of a decreased central nervous system sensitivity to thyroid hormones in WKYs that could be related to their depressive behavior.”

ChatGPT v4 response (accessed on 10/15/2023):

Abstract (2)

The induction of the activity of cerebral ornithine decarboxylase (EC 4.1.1.17) and mRNA by electrical stimulation exhibits regional differences. The effects of the enzyme inhibitor difluoromethylornithine on these regional variations was examined. Administration of this inhibitor resulted in pronounced depression of both basal and induced activity of ornithine decarboxylase in the hippocampus. Basal activity of the enzyme in the neocortex and the cerebellum appeared to be resistant to difluoromethylornithine but the induced enzyme activity was sensitive to the effects of this inhibitor. Susceptibility to difluoromethylornithine may be directly correlated with a slower turnover rate for ornithine decarboxylase. These results suggest that ornithine decarboxylase in the hippocampus may possess a longer half-life than its counterparts in other regions of the brain. Pretreatment with difluoromethylornithine had no effect on the induced ornithine decarboxylase mRNA in the neocortex. Thus, elevated activity of ornithine decarboxylase enzyme, due to electrical stimulation, appears to not have any effect on either the transcription or the decay rate of the induced ornithine decarboxylase mRNA. These findings support the concept of region-specific regulation of cerebral ornithine decarboxylase. ”

From the above two examples presented, it is evident that ChatGPT adeptly comprehends the posed queries and efficiently extracted the inclusion and exclusion criteria. It not only makes clear decisions regarding study inclusion but also articulates its rationale by cross-referencing each criterion.

It is worth noting that the same ChatGPT prompt might result in slightly different responses between different users due to the variability in its response generation, but the conclusion for the eligibility of an abstract is typically consistent. In addition, enhancing the specificity of the criteria within the prompts can further refine the screening process. To illustrate, consider the following enhanced prompts:

Enhanced ChatGPT prompt for abstract (1): “I seek assistance with a systematic review focused on animal depression. I’ll provide the abstract of a particular journal article and would appreciate an assessment for its inclusion based on the following criteria:

The study employs an animal model reflecting depression or a depressive-like phenotype.

It incorporates an appropriate control group.

Primary data is presented within the study.

and here are the exclusion criteria:

The study bypasses the use of an animal model.

The utilized animal model doesn’t characterize depression or a depressive-like phenotype.

Absence of a control group within the study.

The study is primarily a review, devoid of primary data.

Here is the abstract: …”.

With these revised prompts applied to the previously discussed abstracts, the conclusions remain consistent. Nonetheless, we contend that this enhanced structure not only provides a more rigorous framework but also aligns more closely with widely accepted practices in systematic review criteria definition.

Automated workflow for streamlining abstract screening via ChatGPT and other tools

For systematic reviews, particularly when dealing with voluminous data, efficient workflows are paramount. The ChatGPT API (application programming interface) offers a dynamic solution, enabling the automation of abstract screening on a large scale, which circumvents the labor-intensive process of manually inputting abstracts into a chatbot interface. In this subsection, we present an automated workflow for streamlining abstract screening via ChatGPT. Note, though this automated workflow uses ChatGPT as the platform, analogous workflows work for other AI platforms like PaLM, Llama, and Claude.

Automated workflow:

Data collection: The preliminary step entails accumulating a list of titles and abstracts. By utilizing carefully crafted keywords, we retrieve these from PubMed and other pertinent databases. This comprehensive approach ensures the potential inclusion of all relevant studies for a subsequent detailed screening. It is worth noting that while this list is expansive, most of these studies may not find their way into the final meta-analysis post-screening.

Automation through Python: We have devised a python script aimed at harnessing the capabilities of ChatGPT for evaluating the amassed abstracts.

This script interacts with the ChatGPT API (specifically, the GPT-4 version) and, when furnished with tailored prompts, extracts structured responses from ChatGPT.

Typically, the AI’s response commences with a succinct summary, delves into explanations aligned with each criterion, and concludes with a decisive judgment, as exemplified by the examples in the “ Examples of abstract screening by using LLMs ” section.

This automated process efficiently saves ChatGPT’s verdicts on each abstract for ensuing analyses. For instance, it extracts the final decisions regarding the inclusion or exclusion of studies and determines the stance on each pre-specified criterion for every abstract, as exemplified by the last example in the “ Examples of abstract screening by using LLMs ” section.

Additionally, to ascertain the efficiency and cost-effectiveness of this methodology, the script also monitors the time, token usage, and the financial implications of querying the OpenAI API.

In essence, we envision this procedure as delegating the meticulous task of poring over scientific summaries to an AI assistant. This virtual entity meticulously sifts through each summary, determining its alignment with stipulated criteria.

Tuning parameters in the ChatGPT API: The effectiveness of the ChatGPT API is not only dependent on the input data; it is also significantly influenced by adjustable parameters that can refine the responses. Parameters such as temperature, top k, and top p critically affect model performance by modulating the randomness and focus of the output. While fine-tuning these parameters can improve results, it requires significant technical expertise and resources. Defaults, which are rigorously tested by developers, strike a balance between output quality and ease of use, making LLMs more accessible to a wider range of users without the need for complex parameter optimization. While customization holds promise for custom applications, the default settings provide an efficient and practical solution that facilitates wider adoption of LLM technologies.

Given the complexity of fine-tuning these parameters to optimize performance specifically for abstract screening, our study primarily used the recommended default settings for these parameters as provided by the respective platforms (detailed in Table 1 ). This approach was chosen to maintain the feasibility of our experiments and to ensure that our findings are applicable to typical deployment scenarios.

The automated workflow described applies to other LLM tools or different versions of the same tools. Throughout our study, we have tested several popular LLM tools available to us. The specific LLM models and their versions used at the time of our first submission are detailed in Table  1 , along with the tuning parameters.

The field of large language models (LLMs) has evolved rapidly since we started this study in 2023. New models are frequently released, and existing versions are constantly updated. To account for these advances, we have expanded our analysis to include results from the latest versions of previously studied models, as well as a few entirely new ones. As a best practice, we set the temperature parameter to 0 for all latest models. Table 1 now includes these latest models and their versions.

A brief explanation of these parameters is as follows:

Temperature: The temperature controls the randomness of the outputs, with a range from 0 to 2. A temperature value greater than 1 is random and 0 is deterministic. The maximum temperature of 2 gives the most creative and variable outputs.

Max length: The max length is the maximum number of tokens for the model to generate as a response. A single word is generally 2–3 tokens.

Stop sequences: This parameter controls which tokens or phrases will stop the LLM from generating more text.

Top p: When generating text, the model samples from the top p percentage of most likely tokens to generate. The top p is the cumulative probability cutoff for the model’s selection of tokens to generate. Lower top p values mean sampling from a smaller, more top-weighted nucleus.

Top k: When generating text, the model samples from the top k most likely tokens. When deciding the next word to output, the model will select the most likely word when top k is lower.

Frequency penalty: This frequency penalty parameter controls how the LLM penalizes tokens that have already appeared in the input and output text. A frequency penalty of 0 implies that the frequency of the tokens does not impact the LLM and will generate them based on their probability.

Presence penalty: This parameter controls how the LLM penalizes tokens that have not been used recently. A presence penalty of 0 means that the LLM does not care about the recency of the tokens and will generate them based on their probability.

Study design

In our pursuit to assess ChatGPT’s proficiency in abstract screening, we selected certain benchmark databases that have existing performance data from other methodologies. This selection aids in a comparative analysis of performance.

In selecting the systematic reviews for our study, we used a systematic approach guided by specific criteria to ensure relevance and reliability. These studies were selected from the publicly available SYNERGY [ 23 ] dataset, which contains 26 systematic reviews from different disciplines. Key selection criteria included:

Clarity and conciseness of eligibility criteria: The selected studies had well-defined and explicit eligibility criteria. This clarity is essential for accurate replication of the study selection process, which is critical for assessing the performance of LLM tools in an analogous real-world application.

Completeness and cleanliness of data: We ensured that the selected reviews had complete datasets, with all necessary information on included and excluded studies clearly documented, minimizing the risk of ambiguities affecting our analysis.

In addition, to comply with the AMSTAR-2 [ 26 ] guidelines, in particular point 5, we reviewed the methodologies of these reviews to confirm the selection of studies was performed in duplicate and disagreements were resolved by consensus. While our analysis assumes that these systematic reviews adhere to high standards, we recognize the inherent limitations of using pre-existing datasets as a proxy for gold standards in the discussion section.

We picked the following 3 databases from the publicly available SYNERGY dataset [ 23 ]:

Bannach-Brown 2016 [ 27 ]—topic: use of animal models to study depressive behavior

Human-curated (gold standard) results: 1258 excluded abstracts and 230 included abstracts.

We randomly selected 100 excluded abstracts and 100 included abstracts for screening by LLM tools.

Meijboom 2021 [ 28 ]—topic: retransitioning of etanercept in patients with a rheumatic disease

Human-curated (gold standard) results: 599 excluded abstracts and all 32 included abstracts.

We randomly selected 100 excluded abstracts and 32 included abstracts for screening by LLM tools.

Menon 2022 [ 29 ]—topic: methodological rigor of systematic reviews in environmental health

Human-curated (gold standard) results: 896 excluded abstracts and 73 included abstracts.

We randomly selected 100 excluded abstracts and all 73 included abstracts for screening by LLM tools.

For each chosen database, abstracts were categorized as either “cases” (those included based on a gold standard) or “controls” (those excluded per the gold standard). From each category, we randomly selected 100 abstracts (we use all abstracts if there are less than 100 abstracts in that category in a database). These abstracts underwent evaluation by ChatGPT (v4.0) as per our established workflow. Subsequently, ChatGPT’s decisions were juxtaposed against the gold standard to determine sensitivity, specificity, and overall accuracy. The same abstracts were also processed using other LLM tools as listed in Table  1 to record their respective verdicts.

Statistical analysis

To quantify the efficacy of ChatGPT and other AI tools for each database, we calculated the following metrics: (1) sensitivity, (2) specificity, and (3) overall accuracy, where sensitivity is defined as the number of true positives divided by the sum of true positives and false negatives, specificity as the number of true negatives divided by the sum of true negatives and false positives, and accuracy as sum of true positives and true negatives divided by the total number of abstracts. For each metric, associated 95% confidence intervals were also determined. Although it is very common in the field to report F1 score, recall rate, and precision, we believe it is more appropriate to report sensitivity and specificity given this study design. In addition, F1 score, recall rate, and precision can be derived from sensitivity and specificity.

Furthermore, to explore the potential of a unified decision-making process, we combined the decisions from all AI tools using a voting mechanism. The majority decision across the tools was taken as the final verdict for each abstract. For this consolidated approach, we again computed sensitivity, specificity, overall accuracy, and the associated 95% CIs for each database. We also explore the use of latent class analysis (LCA), a model-based clustering approach, to derive consolidated decisions. More details on this LCA approach are provided in the “ Beyond majority voting ” section.

For a given database, 100 cases and 100 controls yield a two-sided 95% confidence interval with a half-width equal to 0.048 when the underline sensitivity (specificity) is approximately 95%.

All statistical analyses were conducted using the R statistical software (version 4.3.1). All tests were two-sided with an alpha level set at 0.05 unless otherwise mentioned.

To improve the transparency and reproducibility of studies using AI tools, we have included the TRIPOD + AI checklist [ 30 ] in our report. This checklist has been adapted to reflect the specifics of our research, which focuses on the evaluation of large language models for abstract screening rather than diagnostic or prognostic modeling. The completed checklist is presented in Table S 1 , to provide readers with a comprehensive overview of our study’s adherence to established reporting standards.

We present the results for each of the 3 databases. For each, we first present the prompts we used when we called LLM tools to screen an abstract, then present the performance data (accuracy, sensitivity, and specificity for each method or LLM tool), followed by a summary of the performance, and a comparison of the performance of the different methods to that based on ChatGPT v4.0.

Results on the Bannach-Brown 2016 database (see Table  2 )

The prompts we used for screening abstracts in this database are as follows:

Conduct a systematic review on animal depression. I provide the title and abstract for one journal article. Provide an overall assessment based on eligibility criteria with only one word answer yes or no with no explanation. Then, for each inclusion or exclusion criterion, answer with only one word, yes if it is included by the inclusion criterion or excluded by the exclusion criterion, and answer no if it does not meet the inclusion criterion or not excluded by the exclusion criterion. After answering all the criteria with yes or no, then provide an overall explanation.

Here is the eligibility criteria: Inclusion Criteria: 1. Any article providing primary data of an animal model of depression or depressive-like phenotype with an appropriate control group (specified above). 2. Animals of all ages, sexes and species, where depression-like phenotype intended to mimic the human condition have been induced. Including animal models where depressive-like phenotypes are induced in the presence of a comorbidity (e.g. obesity or cancer). 3. All studies that claim to model depression or depressive-like phenotypes in animals. Studies that induce depressive behavior or model depression and that also test a treatment or intervention (prior or subsequent to model induction), with no exclusion criteria based on dosage, timing or frequency. 4. Studies measuring behavioral, anatomical and structural, electrophysiological, histological and/or neurochemical outcomes and where genomic, proteomic or metabolomic outcomes are measured in addition to behavioral, anatomical, electrophysiological, histological or neurochemical outcomes. Exclusion Criteria: 1. Review article, editorials, case reports, letters or comments, conference or seminar abstracts, studies providing primary data but not appropriate control group. 2. Human studies and ex vivo, in vitro or in silico studies. Studies will be excluded if authors state an intention to induce or investigate only anxiety or anxious behavior. Studies will be excluded if there is no experimental intervention on the animals (e.g. purely observational studies). 3. Studies that investigate treatments or interventions, but no depressive behavior or model of depression is induced (e.g. toxicity and side-effect studies). 4. Where metabolic outcome measures are the primary outcome measure of a study. Where genomic, proteomic, metabolic or metabolomic outcomes are the sole outcome measures in a study, they will be excluded.

Here is the abstract:

Among all the LLM tools we tested, ChatGPT v4.0 stood out with the highest accuracy (0.945) and specificity (0.960), and satisfactory sensitivity (0.930). Combined decision using major voting seemed to improve sensitivity (0.970) considerably but did not improve specificity (0.870) much. Comparatively, the zero-shot method was less effective across these metrics, while the hybrid method attained superior accuracy (0.961) and specificity (0.982), albeit with reduced sensitivity (0.843). We acknowledge that our findings regarding the zero-shot and hybrid methods differ from those reported in [ 18 ] for the same set of abstracts. This discrepancy could be attributed to the use of distinct sets of embeddings in our analysis compared to the previous study.

For this database, the newer versions of ChatGPT (3.5-Turbo and 4-Turbo) did not improve performance over ChatGPT (v4.0). Gemini-1.0-pro (vs. PaLM 2) and Llama 3 (vs. Llama 2) improved over their older versions, but did not surpass the performance of ChatGPT (v4.0). Claude 3 performed well, but still did not surpass the performance of ChatGPT (v4.0).

Comparison between LLM tools. We compared the performance (sensitivity and specificity) between ChatGPTv4.0 and other LLM tools using the McNemar test and found that ChatGPTv4.0 performed significantly better ( p value = 0.002) than Google PaLM 2 in terms of sensitivity; ChatGPTv4.0 performed significantly better than ChatGPTv3.5 ( p value = 0.008) and better than Llama-2 ( p value < 0.001) in terms of specificity. Combining the decisions of different LLM tools using majority voting did not improve the overall accuracy compared to ChatGPTv4.0. Specifically, there was no statistically significant difference ( p value = 0.134) in sensitivity between the combined decision (majority voting) and ChatGPTv4.0, and the combined decision was significantly worse ( p value = 0.008) than ChatGPTv4.0 in terms of specificity.

Comparison between ChatGPT v4.0 and zero-shot and hybrid methods: We assessed the performance (sensitivity and specificity) of ChatGPT v4.0 against both the zero-shot and hybrid approaches using the McNemar test. Specifically, we aligned the screening results from 100 cases and 100 controls as per the ChatGPT v4.0 method and similarly for the zero-shot and hybrid methods, testing for inconsistencies between these approaches as previously done. Our analysis revealed that ChatGPT v4.0 significantly outperformed the zero-shot method in sensitivity ( p value < 0.001) but showed comparable effectiveness in specificity ( p value = 0.37). Additionally, ChatGPT v4.0 demonstrated marginally superior sensitivity compared to the hybrid method ( p value = 0.07), while its performance in specificity was similar ( p value = 1.00).

Results on the Meijboom 2021 database (see Table  3 )

Conduct a systematic review on transitioning patients from an originator to a corresponding biosimilar.

I provide the title and abstract for one journal article. Provide an overall assessment based on eligibility criteria with only one word answer yes or no, with no explanation. Then, for each inclusion or exclusion criterion, answer with only one word, yes if it is included by the inclusion criterion or excluded by the exclusion criterion, and answer no if it does not meet the inclusion criterion or not excluded by the exclusion criterion.

After answering all the criteria with yes or no, then provide an overall explanation.

Here is the eligibility criteria:

Articles were included if they met the following criteria:

Study involved transitioning from a TNFα inhibitor (including etanercept, infliximab, and adalimumab) originator to a biosimilar

The number of patients who retransitioned was reported or could be calculated

The article was an original research article published in a peer-reviewed journal

The article included baseline characteristics of the patients who transitioned

The article was written in English

The full-text version of the article could be obtained.

Transitioning was defined as patients in whom the biosimilar was introduced after the originator, without treatment with other drugs in between. Retransitioning was defined as restarting the originator directly after discontinuing a biosimilar, without treatment with other drugs in between. In summary, transitioning was defined as switching from the originator to a biosimilar; retransitioning was defined as switching from the originator to a biosimilar and back to the originator. Both transitioning and retransitioning involved changes with the same active biological substance.

Among all the LLM tools we tested, ChatGPT v4.0 stood out with the highest accuracy (0.840), but not with specificity (0.860) or satisfactory sensitivity (0.812). Compared to ChatGPTv4.0, combined decision using major voting did not improve overall accuracy (0.720), but improved sensitivity (1.000) at the sacrifice of specificity (0.630).

Comparison between LLM tools. We compared the performance (sensitivity and specificity) between ChatGPTv4.0 and other LLM tools using the McNemar test and found that ChatGPTv4.0 performed significantly better ( p value < 0.001) than Google PaLM 2, but significantly worse than ChatGPT3.5 ( p value = 0.001) and Llama 2 in terms of sensitivity; ChatGPTv4.0 performed significantly better than ChatGPTv3.5 ( p value < 0.001) and better than Llama 2 ( p value < 0.001), but worse than Google PaLM 2 ( p value = 0.002), in terms of specificity. Combining the decisions of different LLM tools using majority voting did not improve the overall accuracy compared to ChatGPTv4.0. Specifically, there was statistically significant difference ( p value = 0.008) in sensitivity between the combined decision (majority voting) and ChatGPTv4.0, and the combined decision was not significantly worse ( p value > 0.50) than ChatGPTv4.0 in terms of specificity.

For this database, the newer versions of ChatGPT (3.5-Turbo and 4-Turbo) did not improve performance over ChatGPT (v4.0), and Gemini-1.0-pro (vs. PaLM 2) did not improve performance either. However, Llama 3 (vs. Llama 2) improved over its older version and surpassed the performance of ChatGPT (v4.0). Claude 3 also slightly surpassed the performance of ChatGPT (v4.0).

Comparison between ChatGPT v4.0 and zero-shot and hybrid methods: We evaluated the performance of ChatGPT v4.0, focusing on sensitivity and specificity, in comparison with the zero-shot and hybrid approaches, employing the McNemar test as described above. In this analysis, we aligned the screening results from 32 cases and 100 controls for the tests. Our findings indicated that ChatGPT v4.0 significantly surpassed the zero-shot method in sensitivity ( p value = 0.0002) and exhibited marginally improved specificity ( p value = 0.099). Furthermore, ChatGPT v4.0 showed notably higher sensitivity than the hybrid method ( p value < 0.001), although its specificity was comparatively lower.

Results on the Menon 2022 database (see Table  4 )

“Conduct a systematic review on the methodological rigour of systematic reviews in environmental health.

I provide the title and abstract for one journal article.

Provide an overall assessment based on eligibility criteria with only one word answer yes or no, with no explanation.

Then, for each inclusion or exclusion criterion, answer with only one word, yes if it is included by the inclusion criterion or excluded by the exclusion criterion and answer no if it does not meet the inclusion criterion or not excluded by the exclusion criterion.

Here are the eligibility criteria:

To be eligible for inclusion in the SR sample, documents had to fulfill the following criteria:

Identify explicitly as a “systematic review” in their title

Assess the effect of a non-acute, non-communicable, environmental exposure on a health outcome. Environmental exposures can include air and water pollutants, radiation, noise, occupational hazards, lifestyle factors (like diet or physical activity) and lifestyle choices influenced by family and peers (like substance use), social and economic factors (like stress from work or living conditions).

Include studies in people or mammalian models

Be available in HTML format

Abstract X”

Among all the LLM tools we tested, ChatGPT v4.0 stood out with the highest accuracy (0.913) and specificity (0.932), but not with specificity (0.900). Compared to ChatGPTv4.0, combined decision using major voting did not improve overall accuracy (0.884) or sensitivity (0.808), but improved specificity (0.940).

Comparison between LLM tools. We compared the performance (sensitivity and specificity) between ChatGPTv4.0 and other LLM tools using the McNemar test and found that ChatGPTv4.0 performed significantly better than ChatGPT3.5 ( p value < 0.001), Google PaLM 2, and Llama 2 ( p value = 0.02) in terms of sensitivity; ChatGPTv4.0 performed worse than ChatGPTv3.5 and Google PaLM 2, in terms of specificity. Combining the decisions of different LLM tools using majority voting did not improve the overall accuracy compared to ChatGPTv4.0. Specifically, there was statistically significant difference ( p value = 0.008) in sensitivity between the combined decision (majority voting) and ChatGPTv4.0, and the combined decision was not significantly different ( p value = 0.134) than ChatGPTv4.0 in terms of specificity.

For this database, the newer versions of ChatGPT (3.5-Turbo and 4-Turbo) did not improve performance over ChatGPT (v4.0). However, both Gemini-1.0-pro (vs. PaLM 2) and Llama 3 (vs. Llama 2) improved over their older versions and surpassed the performance of ChatGPT (v4.0). Claude 3 also performed well but did not surpass the performance of ChatGPT (v4.0).

Comparison between ChatGPT v4.0 and zero-shot and hybrid methods: We aligned the screening results from 73 cases and 100 controls based on the ChatGPT v4.0 method, and similarly for the zero-shot and hybrid methods, to test for inconsistencies between these approaches, using the McNemar test as done in previous assessments. Our analysis showed that ChatGPT v4.0 significantly outperformed the zero-shot method in both sensitivity ( p value < 0.001) and specificity ( p value = 0.016). In comparison with the hybrid method, ChatGPT v4.0 also demonstrated superior sensitivity ( p value < 0.001) and better specificity ( p value = 0.04).

Monetary cost and time cost

To use the ChatGPT API or other LLM tools, the owners of these platforms charge a predetermined rate for access to the corresponding APIs. These fees are calculated in USD per thousand tokens, where tokens are the basic units used by these LLM platforms to quantify text length. In this context, a token can represent a word, a punctuation mark, or a character. The financial cost of screening 200 abstracts was approximately $6 for ChatGPT v4.0, $0.2 for ChatGPT v3.5, $10 for Llama 2 (using Replicate), while Google PaLM 2 offered its services for free to invited developers. Thus, the cumulative cost of evaluating 200 abstracts across all platforms was approximately $16.2. The cumulative cost of evaluating 200 abstracts across all latest models ($3 for GPT-4-Turbo, $0.05 for GPT-3.5-Turbo, free for Gemini-1.0-pro, $0.05 for Llama-3, $4 for Claude) was less, approximately $7.1. In terms of time efficiency, processing 200 abstracts with each of these LLM tools took approximately 10–20 min using a single thread. However, it is imperative to recognize that abstract screening lends itself well to parallelization. Consequently, one could significantly speed up the process by setting up multiple threads to simultaneously screen different subsets of abstracts, thereby reducing the overall time required for completion. This parallel approach not only increases efficiency, but also ensures that large amounts of data can be processed in a timely manner, making LLM tools even more attractive for large-scale abstract screening tasks. In summary, the monetary and time costs of using LLM tools for abstract screening are negligible compared to manual labeling.

Beyond majority voting

We have expanded our analysis to include a variety of approaches for synthesizing decisions across different LLM tools. Our methodology is inspired by the concept of combining multiple diagnostic tests in the absence of a gold standard, akin to situations where human expert consensus is not available. There are several publications discussing such scenarios [ 31 , 32 ], among which we proposed to use the latent class analysis (LCA) models.

Latent class analysis (LCA) is a statistical method used to identify subgroups within a population, which are not directly observed (hence “latent”) [ 33 , 34 , 35 ]. It is particularly useful when the research interest lies in categorizing individuals into mutually exclusive groups based on their responses to multiple observed variables. In the context of abstract screening, LCA can offer a sophisticated means of integrating decisions from different LLM tools without relying on a gold standard, typically provided by human expert consensus. This approach assumes that the unobserved subgroups (or “latent classes”) explain the dependence between the observed decisions made by each of the LLM tools.

Utilizing the LCA model, we treat the decisions from all LLM tools as dichotomous variables, corresponding to the adherence to each inclusion or exclusion criterion, as well as the overall decision. For instance, within the Bannach-Brown 2016 database (BB2016), there are eight criteria in total with four criteria each for inclusion and exclusion and one overall decision for inclusion, resulting in a total of nine binary items per LLM tool. In our analysis, we incorporated decisions from GPT v4.0, v3.5, and Llama 2. Decisions from Google PaLM 2 were excluded due to a high frequency (10% or more) of incomplete responses. Consequently, for the Bannach-Brown 2016 database, we worked with 27 binary items. For other databases such as Meijboom 2021 (Mj2021) and Menon 2022 (Me2022), the binary items totaled 21 and 15, respectively. It is important to note that LCA models were fitted to the binary data of each database independently.

The LCA model fitting process enables us to calculate the posterior probabilities of each abstract belonging to specific latent classes or subgroups. Abstracts are then categorized based on these probabilities, with assignment to the class for which an abstract has the highest posterior membership probability. The determination of the number of latent classes is a critical step in the LCA model fitting, which requires a priori specification. In our evaluation, we explored models with class numbers ranging from 2 to 6 and utilized the Bayesian information criterion (BIC) to identify the most “optimal” LCA model for our datasets.

Table 5 shows that after applying the Bayesian information criterion (BIC) to determine the most appropriate model for our data, we identified a 3-class model as the best fit for the binary decisions derived from the BB2016 database. Similarly, a 4-class model was optimal for the Mj2021 database, while a 3-class model was again best for the Me2022 database. The confusion matrices generated by the selected LCA models for each database provided a clear juxtaposition between the LLM-assigned classes and the actual labels of the abstracts (see Table  6 ).

The performance metrics derived from these models are noteworthy. For the BB2016 database, should we categorize abstracts within class with label 2 as “included” and assign the others in classes with label 1 or 3 to the “excluded” category, the LCA model achieved a sensitivity rate of 93% and a specificity rate of 96%, indicating a high degree of accuracy in classifying relevant and irrelevant abstracts. In the case of the Mj2021 database, if we view class 1 as “included” group and classes 2 and 3 as “excluded” group, the model achieved a perfect sensitivity rate of 100%, meaning that it correctly identified all relevant abstracts, although the specificity was lower at 79%, indicating a higher rate of false positives. Similarly, for the Me2022 database, considering class 1 to be the “excluded” category and classes 2 and 3 to be the “included” group, the model showed a sensitivity of 94.5% and a specificity of 83%, a good balance between identifying relevant abstracts and avoiding false inclusions.

These results highlight the robustness of the latent class analysis approach to the abstract screening task, providing an effective method for classifying abstracts when a gold standard is not available. The varying levels of sensitivity and specificity across databases underscore the need to tailor the LCA model to the specific characteristics of each dataset as well as further future research.

We have incorporated results from the latest LLM models to increase the robustness and relevance of our findings. However, to maintain consistency and comparability with our original analyses, we have not incorporated these new results into the previously established majority voting or latent class analysis (LCA) approaches. Instead, we have chosen to make all raw data, including results from these newer models, freely available in our GitHub repository. This approach allows interested researchers and practitioners to conduct further investigations or apply alternative methods of analysis. By providing access to this additional data, we aim to promote transparency and enable the broader community to engage with and potentially build upon our work.

This study began with a rigorous exploration of the capabilities of large language models (LLMs) in abstract screening. We used automation scripts developed in Python to interact with the APIs of several LLM tools, including Chat GPT v4.0, Google PaLM 2, and Meta Llama 2, as well as latest versions of these tools. Our central goal was to evaluate the efficiency and accuracy of these tools across three different databases of abstracts, leading us to a complex understanding of their potential in this context.

Large language models (LLMs), particularly ChatGPT, have garnered global attention since their inception. Employing LLMs for abstract screening in systematic reviews is an innovative concept [ 32 , 33 ] and remains underexplored. This study presents the first comprehensive evaluation of LLMs applied to systematic review processes. The findings are encouraging, suggesting that LLMs could revolutionize abstract screening. Specifically, ChatGPT v4.0 exhibited stellar performance across three test scenarios, achieving an accuracy of at least 85%. Furthermore, it attained sensitivity and specificity rates ranging from 80% to an impressive 95%. These exceptional outcomes highlight the substantial promise of LLMs in abstract screening, offering an efficient and capable alternative to the conventional, laborious approaches that typically necessitate extensive human annotation.

However, it is important to acknowledge that we are still in the early stages of integrating LLM tools into the abstract screening field, and they are not without their imperfections—for example, occasionally excluding many (20%) relevant studies, even with the best-performed LLM tools, ChatGPT v4.0. These tools are not a universal solution to all the challenges associated with abstract screening, and they are not ready to completely replace human expertise in this area. Instead, they should be embraced as invaluable assistants in the abstract screening process.

In discussing the limitations of our study, it is important to recognize the constraints associated with our dataset selection and model configuration. We used only three databases from the SYNERGY set, limiting the generalizability of our findings across disciplines and datasets. In addition, the reliance on human-curated labels as a gold standard, while necessary, introduces potential biases due to the retrospective nature of our analysis. These labels may contain errors, and the methodology used in the original reviews to resolve discrepancies may affect the validity of our conclusions.

The parameters chosen for our LLMs—temperature, top k, top p, and prompts—were set to defaults or based on natural conversations to balance output quality with ease of use. While this approach minimizes the need for technical expertise, it may not be optimal for all screening scenarios. In addition, the reliance of LLMs on abstracts alone, as opposed to full-text evaluations, presents a fundamental challenge; critical data influencing inclusion or exclusion may not be present in the abstracts, potentially compromising screening accuracy.

In addition, the rapid development of LLMs and their “black box” nature pose challenges to the transparency that is essential in scientific settings. The environmental impact of using these computationally intensive models is also significant [ 36 ], requiring sustainability considerations. Future research should focus on refining these tools to increase transparency and efficiency, and ensure their responsible development and use in systematic reviews.

Our research suggests that LLM tools are ready to take on a role in abstract screening and are poised to have an immediate and positive impact on the process. Their integration into abstract screening can manifest itself in a few ways. They can serve as autonomous AI reviewers, adding an extra layer of scrutiny and ensuring thoroughness. Our findings suggest that a collective decision, such as one derived from majority voting, can sometimes improve sensitivity, underscoring the potential of LLM tools as a reliable safeguard against oversight, ensuring both comprehensiveness and accuracy.

In addition, LLM tools can facilitate a synergistic partnership with human experts. They are adept at identifying “high-risk” abstracts where different LLM tools have reached different judgments, flagging them for further human evaluation, and promoting a careful and error-free screening process while minimizing human effort.

Another exciting prospect is the integration of LLM tools with hybrid approaches and active learning approach. In this scenario, LLM tools could autonomously annotate abstracts in the training set, minimizing the need for human labeling. Subsequently, these labeled abstracts could be used to train custom NLP models, paving the way for a streamlined and efficient abstract screening process, resulting in significant time and resource savings. Further research is needed to understand how the “uncertainty” in LLM-based decisions, when human-curated labels are not used, affects the performance of the hybrid approach. We also note from the test examples that the performance of hybrid approaches varies widely from example to example and depending on the text embedding tools used. Extensive research is needed to perfect the hybrid approaches.

Another future research direction is to explore how to fine-tune different versions of LLM tools and how to derive collective decisions. One idea is that by using different tuning parameters, one could propose different versions of LLM tools, and if they perform similarly in terms of accuracy but give different decisions (e.g., not highly correlated), we expect that better collective decisions would be obtained, as observed in many ensemble learning experiences [ 37 ]. However, this could be costly and require a lot of exploration.

More importantly, it would be particularly valuable to prospectively test these LLM tools, or integration of these tools with other approaches like zero-shot, active learning, in real-world meta-analysis and systematic review projects. This would provide deeper insights into their practical utility and effectiveness, and a more comprehensive understanding of their impact in live research environments.

In summary, while LLM tools may not be able to fully replace human experts in abstract screening, their ability to transform the screening process is both undeniable and imminent. With continued advances in technology and ongoing refinement, these tools will play a critical role in the future of abstract screening, ushering in a new era of efficiency and effectiveness.

Availability of data and materials

The datasets analyzed during the current study are available in the SYNERGY repository, https://github.com/asreview/synergy-dataset . We have created a GitHub repository https://github.com/mikeli380/LLMAbstractScreening to host our code (R/Python), abstracts, and intermediate results so that interested readers can replicate our findings.

Higgins JPT, et al. Cochrane handbook for systematic reviews of interventions. 2019.

Book   Google Scholar  

Moher D, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7). https://doi.org/10.1371/journal.pmed.1000097 .

Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB,  Richardson WS. Evidence based medicine: what it is and what it isn’t. 1996. Clin Orthop Relat Res. 2007;455.  https://doi.org/10.1136/bmj.312.7023.71 .

Lavis JN, Posada FB, Haines PA, Osei E. Use of research to inform public policymaking. Lancet. 2004;364(9445). https://doi.org/10.1016/S0140-6736(04)17317-0 .

Bastian H, Glasziou P,  Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?. PLoS Med. 2010;7(9).   https://doi.org/10.1371/journal.pmed.1000326 .

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3). https://doi.org/10.1002/jrsm.1354 .

Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3). https://doi.org/10.1056/nejmsa065779 .

Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med. 2002;21(11). https://doi.org/10.1002/sim.1190 .

Wang Z, Nayfeh T, Tetzlaff J, O’Blenis P, Murad MH. Error rates of human reviewers during abstract screening in systematic reviews. PLoS ONE. 2020;15(1). https://doi.org/10.1371/journal.pone.0227742 .

O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1). https://doi.org/10.1186/2046-4053-4-5 .

OpenAI. Introducing ChatGPT Plus. OpenAI Blog; 2023. https://openai.com/blog/chatgpt-plus . Accessed 8 Nov 2023.

Google DeepMind. Introducing PaLM 2. Google Blog; 2023.  https://blog.google/technology/ai/google-palm-2-ai-large-language-model/ . Accessed 8 Nov 2023.

Meta AI Research. Meta and Microsoft introduce the next generation of Llama. Facebook AI Blog; 2023. https://ai.meta.com/blog/llama-2/ . Accessed 8 Nov 2023.

Claude.ai. Claude AI API reference. 2024. https://docs.anthropic.com/en/api/getting-started . Accessed 4 Jun 2024.

Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. 2012. https://doi.org/10.1145/2110363.2110464 .

Alshami A, Elsayed M, Ali E, Eltoukhy AEE, Zayed T. Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions. Systems. 2023;11(7). https://doi.org/10.3390/systems11070351 .

Qureshi R, Shaughnessy D, Gill KAR, Robinson KA, Li T, Agai E. Are ChatGPT and large language models ‘the answer’ to bringing us closer to systematic review automation?  Syst Rev. 2023;12(1).   https://doi.org/10.1186/s13643-023-02243-z .

Xian Y, Lampert CH, Schiele B, Akata Z. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEEE Trans Pattern Anal Mach Intell. 2019;41(9). https://doi.org/10.1109/TPAMI.2018.2857768 .

Moreno-Garcia CF, Jayne C, Elyan E, Aceves-Martins M. A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews. Decis Anal J. 2023;6.  https://doi.org/10.1016/j.dajour.2023.100162 .

Ferdinands G, et al. Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records. Syst Rev. 2023;12(1). https://doi.org/10.1186/s13643-023-02257-7 .

Boetje J, van de Schoot R. The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses. Syst Rev. 2024;13(1). https://doi.org/10.1186/s13643-024-02502-7 .

Yang Y, Loog M. A benchmark and comparison of active learning for logistic regression. Pattern Recognit. 2018;83. https://doi.org/10.1016/j.patcog.2018.06.004 .

De Bruin RJ, Ma Y, Ferdinands G, Teijema J, Van de Schoot. SYNERGY - open machine learning dataset on study selection in systematic reviews. DataverseNL; 2023. https://dataverse.nl/dataset.xhtml?persistentId=doi:10.34894/HE6NAQ .

Millard SJ, Weston-Green K, Newell KA. The Wistar-Kyoto rat model of endogenous depression: a tool for exploring treatment resistance with an urgent need to focus on sex differences. Prog Neuropsychopharmacol Biol Psychiatry. 2020;101. https://doi.org/10.1016/j.pnpbp.2020.109908 .

Zawia NH, Mattia CJ, Bondy SC. Differential effects of difluoromethylornithine on basal and induced activity of cerebral ornithine decarboxylase and mRNA. Neuropharmacology. 1991;30(4). https://doi.org/10.1016/0028-3908(91)90058-J .

Shea BJ, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358. https://doi.org/10.1136/bmj.j4008 .

Bannach-Brown A, Liao J, Wegener G, Macleod M. Understanding in vivo modelling of depression in non-human animals: a systematic review protocol. Evid Based Preclin Med. 2016;3(2). https://doi.org/10.1002/ebm2.24 .

Meijboom RW, Gardarsdottir H, Becker ML, ten Wolde S, Egberts TCG, Giezen TJ. Incidence of and reasons and determinants associated with retransitioning from biosimilar etanercept to originator etanercept. BioDrugs. 2021;35(6). https://doi.org/10.1007/s40259-021-00501-x .

Menon JML, Struijs F, Whaley P. The methodological rigour of systematic reviews in environmental health. Crit Rev Toxicol. 2022;52(3). https://doi.org/10.1080/10408444.2022.2082917 .

Collins GS, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021;11(7). https://doi.org/10.1136/bmjopen-2020-048008 .

Joseph L, Gyorkos TW, Coupal L. Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. Am J Epidemiol. 1995;141(3). https://doi.org/10.1093/oxfordjournals.aje.a117428 .

Hui SL, Walter SD. Estimating the error rates of diagnostic tests. Biometrics. 1980;36(1). https://doi.org/10.2307/2530508 .

Dendukuri N, Joseph L. Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests. Biometrics. 2001;57(1). https://doi.org/10.1111/j.0006-341X.2001.00158.x .

Collins LM, Lanza ST. Latent class and latent transition analysis: with applications in the social, behavioral, and health sciences. 2010.

Google Scholar  

McCutcheon AL. Latent class analysis. Newbury Park: Sage; 1987.

Rillig MC, Ågerstrand M, Bi M, Gould KA, Sauerland U. Risks and benefits of large language models for the environment. Environm Sci Technol. 2023;57(9). https://doi.org/10.1021/acs.est.3c01106 .

Sagi O, Rokach L. Ensemble learning: a survey.  Wiley Interdiscip Rev Data Min Knowl Discov. 2018;8(4). https://doi.org/10.1002/widm.1249 .

Download references

Acknowledgements

The authors thank the reviewers and the Editor for their insightful comments and suggestions, which have greatly improved the quality of this work.

Not applicable.

Author information

Authors and affiliations.

Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA

Michael Li & Xianming Tan

Department of Mathematics and Statistics, University of North Carolina at Greensboro, Greensboro, NC, 27402, USA

Jianping Sun

Department of Biostatistics and Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA

Xianming Tan

You can also search for this author in PubMed   Google Scholar

Contributions

XT conceived the paper. ML and XT devised the study. ML, JS, and XT analyzed the datasets. ML, JS, and XT wrote the manuscript. All authors edited and approved the final manuscript.

Corresponding author

Correspondence to Xianming Tan .

Ethics declarations

Ethics approval and consent to participate, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1: table s1. tripod + ai checklist., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Li, M., Sun, J. & Tan, X. Evaluating the effectiveness of large language models in abstract screening: a comparative analysis. Syst Rev 13 , 219 (2024). https://doi.org/10.1186/s13643-024-02609-x

Download citation

Received : 20 December 2023

Accepted : 11 July 2024

Published : 21 August 2024

DOI : https://doi.org/10.1186/s13643-024-02609-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Abstract screening
• Automated text analysis
• Google PaLM

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]