computer science literature review example

Computer Science and Engineering

• Getting Started
• Keeping Current

Introduction

Gather Your Tools

Determine the Project's Scope

Create the Search Strategy

Determine What Resources to Use

Search, read, refine, repeat.

Saved Searches, Alerts and Feeds

• STEM Biographies & Info
• Writing & Citing

This page focuses on how to do an in-depth literature review for a dissertation, thesis, grant application or lengthy term paper in electrical engineering.  

• For a more general description of what an in-depth literature review is and how it looks, see our guide on " Literature Reviews and Annotated Bibliographies " created by Ed Oetting, history and political science librarian.
• For lower-level engineering undergraduate students who are doing a short term paper, the " How to Research a Topic " page on the " Engineerng Basics" guide may be more applicable.

Library Account Is your library account clear of fines?   If not, you may not be allowed to check out more books nor renew books you already have.  All library notices are sent via email to your "asu.edu" address; if you prefer to receive email at a different address make sure you have forwarded your asu.edu correctly.  Also, make sure that your spam filter allows the library email to come through. 

Illiad (Interlibrary Loan) Account   If you don't already have an ILLiad account, please register for one.  Interlibrary loan services will get you material not available at the ASU Library and also scan or deliver materials from the libraries on the other ASU campuses.

Determine the Project's Scope.

Do you know what you are looking for?  Can you describe your project using one simple sentence or can you phrase the project as a question?  Without a clear idea of the project, you may not be able to determine which are the best resources to search, what terminology should be used in those resources, and if the results are appropriate and sufficient.    

If you're having difficulty getting your project described succinctly, try using a PICO chart to identify the concepts involved:

• P is the popluation, problem, predicament or process
• I is the intervention or improvement
• C is what you'll compare your intervention/improvment to, and
• O is the outcome (or results of the comparison of I and C ) 

For example: 

Your client, the owner of a nuclear power generating facility, has had several less than optimal safety inspections recently.  The inspectors have singled out operator error as a major concern and have required changes in employee training.  But is more training the solution?  The employees complain that the plant's poorly designed control room hampers their ability to respond to non-standard situations.  Could a redesign improve performance and decrease the occurance of unsafe events?   Your client wants more than just your opinion, he wants to see the data to back it up.   So, what can you find in the literature?

Here's one way that the PICO chart could be filled out:     

• P =   nuclear power safety  
• I  = human factors engineering
• C  =  additional training; little or no human factors engineering used  
• O = accident rate or safety inspection comparison

And here are examples of possible search statements:  

• I am looking for ways that human factors engineering can improve safety in the nuclear power industry.
• Is additional training or employing human factors engineering the better method for reducing safety violations in a nuclear power plant? 

Your research will always start with a " P AND I " search; those are the most important pieces of the puzzle.  However, once you have the results from that search, you'll need to know where you want to go with those results; that's when the C and O concepts need to be considered.  

 Also, don't forget --- determine if your project has limits.  For example:

• Are you reviewing the literature only within a specific time frame?
• Are you looking at English-language material only?
• Are you considering research from just the United States or worldwide?
• Are there types of material you won't be covering (trade magazines, patents, technical reports, etc.)?  

Take the simple sentence or question that describes what you are looking for.  What are the concepts in the sentence? Are there synonyms that describe the same concept?   If you filled out a PICO chart, concentrate on the  P (problem) and the  I (intervention) for the concept chart.  

Concept Chart:

Concept 1:   _______  OR _______  OR _______  AND Concept 2:   _______   OR   _______  OR   _______  AND Concept 3:  _______  OR  _______  OR   _______ 

  Example:  

I am looking for ways that human factors engineering can improve safety in the nuclear power industry. 

Concept 1:   nuclear power    OR _ nuclear industry _____   AND Concept 2:   _safety___  OR   _accident prevention____   AND Concept 3: _ human factors engineering ___   

What resources you'll use for your literature review depends on what types of materials you want to find.  

• Background Information The more you know about a topic, the better you'll be able to research it.  You'll be familiar with the terminology, understand the underlining science/technology and be aware of the issues in the field. Most importantly, you'll be able to understand what you've retrieved from your search.  But no matter how much you know before hand you'll likely run across terms and concepts with which you're unfamiliar.    Materials such as encyclopedias, dictionaries and handbooks will not only help you learn about the basics of your topic before you begin your search but they'll also help you understand the terminology used in the documents you found from your literature review.    You'll find these types of resources listed on the Dictionaries and Handbooks pages on this guide.
• Books The large size of books (usually 100-500 pages) allows a topic to be studied braodly, covering many different issues.  Conversely, the large size also allows for a specific aspect of the topic to be covered in great detail.  Because of the time it takes to publish, sci-tech books generally do not contain the most current information. To find print and online books from both the ASU Library as well as in other libraries, see the Books page on this guide.
• Conference Papers Scientists and engineers frequently present new findings at conferences before these findings are written up in journal articles or books.  Not every conference, however, publishes it proceedings.  In some cases, conferences publish only a few of the papers presented but not all.   Many resources that help you find journal articles, may also be used to find conference papers, see the Articles page on this guide.
• Journal and Trade Magazine Articles Articles in journals (also called magazines) are short, usually 5-20 pages in length and cover a specific finding, experiment or project.  Articles in scholary journals are usually written by academics or professional scientists/engineers and are aimed at others at the same level.   Articles in trade journals/magazines are written by the journal staff and report on industry news suchs as sales, mergers, prices, etc.   To find journal and trade magazine articles, use the resources listed on the Articles page on this guide. 
• Patents Patents are grants from governments that gives the inventor certain rights of manufacture.  Patents provide a wealth of information for how a technology is being advanced and by which companies.  It is frequently stated that 80% of the information in patents never appears elsewhere in the literature. 
• To identify patents granted in the U.S. and internationally see the " Searching for Patents " guide.
• To see statistical information for U.S. patents by technology class see the US Patent and Trademark's website.
• Technical Reports Technical reports are part of the "gray literature";  gray literature refers to documents that are not published commercially, hence they are difficult to both identify and find.  Technical reports focus on a specific experiment or research project and are meant to convey the results of the experiment or project back to the funding organization.  In the United States, common sources of technical reports are the government agencies that sponsor research projects.  Reports generated within a private corporation and funded soley by that corporation are seldom ever available to anyone outside of the company.      To find technical reports, use the resources listed on the Technical Reports page on this guide.     

Search, Read, Refine and Repeat

Now it's time to apply your search strategy in the resources you've decided to use.

• Use the Advanced Search feature (or whatever search is set up with the 3 lines of boxes) and enter your search strategy just as you recorded in your search strategy chart.  Don't forget to set your limits.   If the resource only provides a single search box, rearrange your chart from vertical into horizontal so that the search statement looks like this:   (Concept#1 OR synonym) AND (Concept#2 OR synonym) AND (Concept#3 OR synonym) Example: (nuclear power OR nuclear industry) AND (safety OR accident prevention) AND (human factors engineering)
• Examine the results to find the most appropriate items.  Keep your one-sentence project description (and/or your PICO chart) in mind to help you stay on track.
• Export the records/citations you want to keep into a citation manager.
• If there are subjects (may also be called subject headings, index terms, descriptors or controlled vocabulary) assigned to each item, make sure that those also transferred into a citation manager.  If not, add them manually.
• Get the full text of the items 
• Read the full text of the items and look at the subjects assigned to the item and consider:
• Do I have to change (narrow) my topic to something more specific because I'm finding way too much? 
• Do I have to change (broaden) my topic because I can't find enough about it? 
• Is there additional terminology for my topic/concepts that I hadn't included in my original search?
• Redo your search strategy according to what you found in step #6 and rerun the search in the resources again.
• You may need to repeat this cycle several times before you are able to identify the best terminology to use in each resource. 

If there will be several months in between when you search the literature and when you turn in the paper, consider setting up alerts and feeds so that you are notified should new items about your topic appear.   How you set up an alert or feed will vary.  In most cases you'll be required to set up a personal account or profile with the journal or database --- there is no charge for this but you will have to identify yourself and provide an email address.  

For instructions on setting up alerts and feeds, see the " Keep Current " page.

• << Previous: Keeping Current
• Next: Resources >>
• Last updated: Jan 2, 2024 8:27 AM
• URL: https://libguides.asu.edu/computerscience

The ASU Library acknowledges the twenty-three Native Nations that have inhabited this land for centuries. Arizona State University's four campuses are located in the Salt River Valley on ancestral territories of Indigenous peoples, including the Akimel O’odham (Pima) and Pee Posh (Maricopa) Indian Communities, whose care and keeping of these lands allows us to be here today. ASU Library acknowledges the sovereignty of these nations and seeks to foster an environment of success and possibility for Native American students and patrons. We are advocates for the incorporation of Indigenous knowledge systems and research methodologies within contemporary library practice. ASU Library welcomes members of the Akimel O’odham and Pee Posh, and all Native nations to the Library.

How to Write a Good Literature Review?

Last updated: March 18, 2024

• Project Management

It's finally here:

>> The Road to Membership and Baeldung Pro .

Going into ads, no-ads reading , and bit about how Baeldung works if you're curious :)

1. Introduction

In this tutorial, we’ll talk about literature reviews. We’ll explain why they’re important and how to write them.

2. Why Do We Need Research Literature?

Indeed, why? Many students think having a good research idea or original thoughts on a subject is enough.

However, what initially seems original or revolutionary may have already been proposed by other researchers and even proved not to work. Further, not attributing ideas to the original authors could be considered plagiarism; even if not, experts will recognize the ideas and ask why there is no proper attribution.

Also, we should place our idea in the existing knowledge frame since we’re not writing papers for ourselves but for others. Referencing other people’s work shows we know previous approaches and research. It makes an impression that we have a good understanding of the problem we’re writing about. Also, let’s not underestimate the importance of responding to criticism: if readers point out flaws and cite literature we didn’t even try to read, we won’t be able to reply.

Further, the actual value of an idea is in its relation to other ideas. How is it different? Does it reveal why previous theories are wrong? Does it consider the problem from an overlooked perspective? Writing about these points is impossible without knowing the research field’s current state.

Finally, a literature review can stand as a research paper on its own. Its value lies in systematizing the results and letting others get an overview of a research field or a problem.

3. How to Prepare for a Literature Review?

So, how do we write a good review? There’s no universal recipe, but we can roughly separate the process into three stages:

• the initial steps we take to prepare for the review
• carrying out the work
• writing and revising the review

Let’s now get familiar with the preparatory steps:

3.1. Identifying the Search topic(s)

This may appear obvious at first glance, but there’s no literature review without clearly defining what we want to cover.

This step comes after formulating our research question since it will give us starting points. For example, if we’re set to implement a text-to-SQL converter, we should look for articles about natural-language interfaces. However, we’ll be interested in those targeting SQL specifically, not all that deal with this problem or are related to natural language processing .

So, we need to define the scope of the search. It should be sufficiently broad to cover all the relevant sources but also narrow enough to exclude everything unrelated to our research.

3.2. Defining the Purpose of a Review

Equally important is to define the review’s purpose:

Do we want to get familiar with a field to asses if our idea has merit, or do we want to make a comprehensive overview for other researchers?

Is the goal of our review to include all the sources covering the chosen topic(s) and lay out their summaries, or will we engage in analysis and draw conclusions about the state of the literature?

Finally, is the purpose of the review to answer a specific question such as:

• Has a scientific consensus been reached about this or that hypothesis?
• What are the issues with the current methodology?

For example, it’s very common to reproduce medical experiments to see if their results can be replicated. A literature review on a specific drug may just lay out all the studies in which it was used. However, it can also analyze the results and draw conclusions:

• For instance, we may find that 99% of studies successfully replicate the original results
• Similarly, we may find inconclusive results (e.g., 55% of the studies replicate the results, and the confidence intervals contain values lower than 50%)
• Finally, only a small percentage of studies may replicate the original results—for example, only 10% or only those conducted by the authors of the original report

In all three cases, we have a scientific discovery about the drug’s efficacy that is much more credible than the result of a single study.

3.3. Defining the Protocol

We can define the review protocol based on the purpose and the topic. It should be as detailed as necessary and answer all questions about the review methodology:

• What methods will we use to collect the sources?
• What criteria will we use to include or exclude a source?
• Will we use statistical methods to analyze the sources? If so, which methods?

This is where we get different types of reviews. They differ in purpose and protocol (which should reflect the purpose). Rigorous types have clearly defined protocols, whereas informal types don’t specify all the elements or don’t have a protocol at all.

4. Doing the Work

Now, we should follow our review protocol:

4.1. Collecting Sources

We can usually browse electronic databases such as Google Scholar and Semantic Scholar for a list of related articles and books when collecting sources. But, to be as efficient as possible, we’ll have to use the correct keywords and try multiple combinations of synonyms.

All this should be specified in the protocol. For example, we can choose Google Scholar as the search database and the following keywords and search strings:

• Natural Text to SQL
• Natural Language to SQL Queries
• Natural-Language Interfaces
• Natural-Language Interfaces to Relational Databases
• Natural Language Interfaces to RDBMS

Other times, it’s only possible to do offline searches in libraries, museums, and other collections of sources. For instance, a particular manuscript from the 16th century may not be available online, so the only way to study it is to visit the site it’s located in.

Also, an excellent starting point is to check if a quality literature review has been compiled recently. If so, we should start from it to see how other researchers structured their reviews, as that can help us organize our research better.

4.2. Inclusion and Exclusion

If our goal isn’t to cover all the sources related to the topic, we should focus on quality rather than quantity. This calls for filtering criteria, of which there are two types:

• pre-reading and
• post-reading

We define the pre-reading criteria as technical conditions a source has to fulfill to be included in our review before we even read it. For example:

• We may consider only the articles published in peer-reviewed journals or presented at prestigious conferences
• Similarly, we could discard all the sources written more than 50 years ago, justifying this decision by saying that too old sources are obsolete and superseded by more recent studies

The post-reading criteria are concerned with a source’s content. We check them after reading a source, which means we need to read every source thoroughly and answer the following and similar questions:

• Does the experiment have a design flaw?
• Do data support conclusions?
• Are the authors biased?
• Did they discuss their findings from multiple points of view?
• Does the source have any other weaknesses?

The output of evaluating a source should be twofold:

• it should contain a qualitative and/or quantitative summary of the most important information (such as research questions, methodology, and conclusions)
• and an informed decision about whether the source is relevant

In general, if it has any significant weakness, such as bias or statistically unjustified conclusions, we can disregard it. The only exception is if a source has mixed strong and weak parts. In that case, we can cite it for what we deem valid.

Further, we need to be impartial. No matter our personal views or if a source contradicts our ideas, we should evaluate it without bias or prejudice.

It’s important to note that the evaluation step is intertwined with the collection step. We can evaluate a source as soon as we get to it in the collection step.

4.3. Analysis

While collecting the sources, we’ll notice that some are similar. For example, they may share the same or similar methodology, draw the same conclusions, identify the same problem(s), or focus on the same aspect of the research problem. Therefore, we should group similar sources and analyze them together.

Here, we can have two goals:

• to offer the historical perspective
• to determine the current state of the field

Why is the historical analysis important? Some topics get more attention because they’re more interesting or appear more promising. How the “hot” topics gave way to one another will show us how the research field evolved, what approaches have been used or theories hypothesized, what was considered valid until proven incomplete, and why previous methods or ideas were abandoned.

This all leads to the analysis of the current state of the knowledge. What hypotheses and theories do contemporary researchers accept? Are there competing points of view and ongoing debates? What problems have been identified, and what future directions of researchers have been suggested?

To find all that, we should analyze the summaries of the sources we decided to include. Comparing the notes will help us see the bigger picture of the field’s current state.

For example, let’s say we’re interested in the efficacy of a drug. In our protocol, we specify that a study confirms effectiveness if:

• the drug is tested on at least 100 patients in treatment and control groups and
• it heals at least 70% of patients in the treatment group
• it heals statistically significantly more patients than placebo in the control group
• the difference in the percentages of recovered patients is 30% in favor of the treatment group.

Also, let’s say that 67 out of 89 relevant studies confirm the efficacy of a drug in this way. Then, we can say something like this

The literature analysis revealed that 67 out of 89 studies confirmed the drug’s efficacy. The corresponding proportion is 75.28%, with the exact binomial 95%-confidence interval (65%, 83.81%). The evidence shows that the drug is likely to help many patients, although more research is needed to identify factors leading to its failure to do so.

The actual analysis methods (95% confidence interval) should be specified in the protocol.

However, not all review types analyze the collected sources.

5. Writing and Revising the Review

There are two main approaches when it comes to writing. First, we can present sources in chronological order. The other approach is organizing the content by topics.

5.1. Chronological vs. Topic-Centric Reviews

The former is suitable for emphasizing the historical perspective. We’ll identify various periods in the research history and describe them individually. They may differ in methodology, assumptions, or dominant theories.

In the topic-centric approach, a topic can be the research problem, a hypothesis, or a method.

For example, some researchers may use neural networks , whereas others prefer non- AI methods to solve a problem. The two schools of thought need not be separated chronologically.

It can and usually does happen that research is simultaneously active in two or more competing or complementary topics or that there’s a revival of interest in the ideas that were once researched but lost popularity to other, more efficient approaches.

5.2. Reviewing and Revising the Review

After the first draft of the review is completed, we should do the final checks:

• Did we research all topics to achieve a complete understanding?
• Are all sources properly categorized?
• Is our review a story that reads naturally?
• Can we improve the sectioning?
• Are there breaks in the flow? Does each paragraph present precise information and sets the stage for the next one? Is it the same for the sections?
• Are there typos and grammar errors? Are our sentences too complex or easy to read? Are there any ambiguities or passages that can be formulated better?
• Should we research specific topics further?
• Is the review balanced?

Reviewing the first draft right after completing it would be an error. Instead, it’s best to wait a few days to cool off. The goal is to read it when it sounds like another person wrote it. If we review it at that point, we’ll spot its weaknesses more efficiently.

5.3. Getting Help

Of course, asking someone else to read it is always welcome . However, even if they come with criticisms and improvement suggestions that appear unnecessary or without merit, we shouldn’t discard them. Contradicting, confusing or untrue parts can go unnoticed when we read our draft, but another person can spot those weak parts more easily since they didn’t go through our thought process and read what is written, not what we intended to write.

After all, even if it’s about wording, we should consider rewriting those passages. The review is meant to show others we know the field or help them get an overview. Not all the readers will be 100% focused, so if it’s possible to formulate a more straightforward sentence or a paragraph to avoid confusion, we should do it.

6. The Iterative Nature of Writing a Review

This isn’t a linear but iterative process. That means that steps can be repeated or done simultaneously:

For example, after categorizing the papers, we can realize that we need to filter, evaluate, and read the sources they cite. Additionally, an expert we asked to read our first draft may point out that we missed a body of literature. This brings a new pile of papers to process from scratch.

7. Types of Literature Reviews

There are several types of literature reviews, and not all researchers agree on the list and definitions. For example, Grant and Booth (2009) identify fourteen, Booth et al. (2012) find sixteen, and Samnani et al. (2017) nine types of literature reviews.

We won’t delve into details on all the types. Instead, we’ll present and compare the most common ones.

7.1. Narrative Review

This type of review is also known as the traditional.

No particular protocol is defined. Instead, the authors rely on their experience and preferences to collect, filter, and analyze sources on the fly. They summarize the literature covering a broad topic and offer their opinions and conclusions about various sub-topics.

The included body of literature need not be comprehensive since the authors don’t specify a rigorous search strategy. The selection criteria are subjective and not clearly stated, so this type of review is prone to selection biases.

Since the analysis is qualitative and opinionated, subjective biases may play a role in drawing conclusions.

7.2. Systematic Review

This review type is pretty much the opposite of the narrative review.

Its protocol is rigorously defined beforehand and tailored to answer a narrow research question. The search strategy is exhaustive, with the goal of covering all the relevant articles. Selection criteria are also explicitly specified.

The protocol of a systematic review acts as the methodology of an experiment: it needs to be as precise as necessary for the review to be reproducible.

If the sources are quantitative, the analysis will use statistical techniques to synthesize new information (i.e., draw conclusions). This is also called meta-analysis or quantitative meta-analysis. When dealing with qualitative sources, the analysis will also be qualitative.

We usually conduct systematic reviews to make informed recommendations about a policy, drug, or research method studied by scientists.

7.3. Scoping Review

A scoping review is similar to a systematic review. The main difference is that the former doesn’t have a narrow research question. Its purpose is to scan a research field and identify its problems, prevalent themes, knowledge gaps, and limitations.

In doing so, a scoping review follows a standardized and rigorous protocol with a comprehensive search method. This rigor makes it different from the narrative review with which it shares a broadly defined search topic. However, scoping reviews don’t usually assess the sources’ quality.

We don’t use them to make recommendations. Instead, we conduct scoping reviews to find a research topic.

7.4. Summary

Here’s a short summary of the covered review types:

There are others. For example, rapid reviews are conducted in a specified time frame. Their goal is to quickly produce information, so search, selection, and analysis are limited by time constraints. Further, the state-of-the-art reviews cover recent studies and are conducted periodically to scan the current state of a research field.

8. Conclusion

In this article, we talked about writing literature reviews. They’re important because they set up the stage for our research, show that we’re familiar with a scientific field, and are a useful resource for anyone who wants to get an overview of and introduction to it.

A literature review needs a clear research topic and a precisely defined purpose. Depending on that, we differentiate between several types of reviews. Some, like systematic reviews, have rigorous and detailed protocols that specify all the steps of collecting, filtering, and analyzing sources. Other types are more informal, e.g., the narrative review.

Computer Science

• Key resources
• Databases and journal articles
• Standards and patents

What is a literature review?

Finding the gaps and advances in your area, literature review resources, annotated bibliography.

A literature review is a survey and critical analysis of what has been written on a particular topic, theory, question or method.

What is its purpose?

• justify your research
• provide context for your research
• ensure that the research has not been done before
• highlight flaws in previous research
• identify new ways, to interpret and highlight gaps in previous research
• signpost a way forward for further research
• show where your research fits into the existing literature
• Literature reviews guide The Literature reviews guide contains detailed information on the process of searching for and producing literature reviews.
• Scopus A multidisciplinary abstract and citation database of peer reviewed literature, book reviews and conference proceedings.
• Web of Science A collection of citation databases and citation analysis tools covering the sciences, social sciences, arts and humanities.
• IEEE Xplore digital library Provides full-text access to IEEE and IEE transactions, journals, magazines and conference proceedings published since 1988 and current IEEE Standards.
• ACM digital library publications Full text of every article published by the Association for Computing Machinery (ACM) and bibliographic citations from major publishers in computing
• Annual reviews online This database provides review journals from across the sciences, with articles that review significant primary research literature.
• Cited reference searching

Books and other resources for approaches and methods on doing a literature review. See the Literature reviews guide .

An annotated bibliography provides:

• a list of references presenting a brief summary of the main arguments or ideas of each resource.
• a critique or evaluation of the resource's usefulness, reliability, objectivity or bias
• a reflection on how the resource fits into your research.
• << Previous: Multimedia
• Last Updated: Nov 29, 2023 12:58 PM
• URL: https://guides.library.uq.edu.au/computer-science

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

Tales of a migratory marine biologist

Career Feature 28 AUG 24

Nail your tech-industry interviews with these six techniques

Career Column 28 AUG 24

How to harness AI’s potential in research — responsibly and ethically

Career Feature 23 AUG 24

Binning out-of-date chemicals? Somebody think about the carbon!

Correspondence 27 AUG 24

No more hunting for replication studies: crowdsourced database makes them easy to find

Nature Index 27 AUG 24

Partners in drug discovery: how to collaborate with non-governmental organizations

How can I publish open access when I can’t afford the fees?

Career Feature 02 SEP 24

Exclusive: the papers that most heavily cite retracted studies

News 28 AUG 24

Chain retraction: how to stop bad science propagating through the literature

Comment 28 AUG 24

Permanent Researcher Positions for Materials Science

NIMS (Tsukuba, Japan) invites international applications from researchers who can conduct research in materials science.

Tsukuba, Ibaraki (JP)

National Institute for Materials Science (NIMS)

Osaka University Immunology Frontier Research Center Postdoctoral Researcher

IFReC, Osaka University in Japan offers Advanced Postdoc Positions for Immunology, Cell Biology, Bioinformatics and Bioimaging.

Suita Campus, Osaka University in Osaka, Japan

Immunology Frontier Research Center, Osaka University

Tenure-track Associate Professor [Female Only]

Seeking tenure-track assoc. professor for interdisciplinary research in nanoprobe life sciences or related fields at WPI Nano Life Science Institute.

Kanazawa, Ishikawa, Japan (JP)

Nano Life Science Institute, Kanazawa University

Assistant/Associate Professor

Center for Virology and Vaccine Research at Beth Israel Deaconess Medical Center is seeking Assistant or Associate Professor.

Boston, Massachusetts (US)

Beth Israel Deaconess Medical Center (BIDMC)

OPEN FACULTY POSITION-INSTITUTE OF MOLECULAR BIOLOGY, ACADEMIA SINICA, TAIWAN, ROC

One tenure-track faculty position is open to establish an active research program in all disciplines of molecular and cellular biology.

Taipei (TW)

The Institute of Molecular Biology, Academia Sinica, Taiwan

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

How-to conduct a systematic literature review: A quick guide for computer science research

• November 2022
• MethodsX 9(1):101895
• 9(1):101895
• CC BY-NC-ND 4.0

• Mondragon Unibertsitatea

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations
• José Castillo
• Abraham Dávila

• Rungroj Wongmahasiri

• Marcio Biczyk

• Katia Romero Felizardo

• Igor Steinmacher
• Dinda Karunia Putri
• Putri Sarah Zainovi
• Singgih Aulia Aulia Hanip
• Julianto Julianto
• Erna Shevilia Agustian
• Zastra Alfarezi Pratama
• El Bouch Hajar
• Abdellaoui Mohammed
• NDT&E INT
• Satish Chandra
• G. Bala Krishna
• Giulia Alessandri

• J INTELL MANUF

• J CLEAN PROD

• INFORM SOFTWARE TECH

• ROBOT CIM-INT MANUF

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

The Sheridan Libraries

• Computer Science and Information Security
• Sheridan Libraries

Literature Reviews

• How to Access Full Text
• Background: Books and Review Articles
• Articles, Conferences, More
• Google Scholar and Google Books
• Information Security
• LaTeX, MATLAB, Open Source, Overleaf, More
• Problems and Solutions
• Technical Reports
• Avoiding Plagiarism
• Citing Sources This link opens in a new window
• Copyright This link opens in a new window
• Evaluating Information This link opens in a new window
• RefWorks Guide and Help This link opens in a new window
• CAPSTONE HELP
• Organizing/Synthesizing
• Peer Review
• Ulrich's -- One More Way To Find Peer-reviewed Papers

"Literature review," "systematic literature review," "integrative literature review" -- these are terms used in different disciplines for basically the same thing -- a rigorous examination of the scholarly literature about a topic (at different levels of rigor, and with some different emphases).  

1. Our library's guide to Writing a Literature Review

2. Other helpful sites

• Writing Center at UNC (Chapel Hill) -- A very good guide about lit reviews and how to write them
• Literature Review: Synthesizing Multiple Sources (LSU, June 2011 but good; PDF) -- Planning, writing, and tips for revising your paper

3. Welch Library's list of the types of expert reviews

Doing a good job of organizing your information makes writing about it a lot easier.

You can organize your sources using a citation manager, such as refworks , or use a matrix (if you only have a few references):.

• Use Google Sheets, Word, Excel, or whatever you prefer to create a table
• The column headings should include the citation information, and the main points that you want to track, as shown

Synthesizing your information is not just summarizing it. Here are processes and examples about how to combine your sources into a good piece of writing:

• Purdue OWL's Synthesizing Sources
• Synthesizing Sources (California State University, Northridge)

Annotated Bibliography  

An "annotation" is a note or comment. An "annotated bibliography" is a "list of citations to books, articles, and [other items]. Each citation is followed by a brief...descriptive and evaluative paragraph, [whose purpose is] to inform the reader of the relevance, accuracy, and quality of the sources cited."*

• Purdue's OWL (Online Writing Lab) includes definitions and samples of annotations
• Cornell's guide * to writing annotated bibliographies  

* Thank you to Olin Library Reference, Research & Learning Services, Cornell University Library, Ithaca, NY, USA https://guides.library.cornell.edu/annotatedbibliography

What does "peer-reviewed" mean?

• If an article has been peer-reviewed before being published, it means that the article has been read by other people in the same field of study ("peers").
• The author's reviewers have commented on the article, not only noting typos and possible errors, but also giving a judgment about whether or not the article should be published by the journal to which it was submitted.

How do I find "peer-reviewed" materials?

• Most of the the research articles in scholarly journals are peer-reviewed.
• Many databases allow you to check a box that says "peer-reviewed," or to see which results in your list of results are from peer-reviewed sources. Some of the databases that provide this are Academic Search Ultimate, CINAHL, PsycINFO, and Sociological Abstracts.

What kinds of materials are *not* peer-reviewed?

• open web pages
• most newspapers, newsletters, and news items in journals
• letters to the editor
• press releases
• columns and blogs
• book reviews
• anything in a popular magazine (e.g., Time, Newsweek, Glamour, Men's Health)

If a piece of information wasn't peer-reviewed, does that mean that I can't trust it at all?

No; sometimes you can. For example, the preprints submitted to well-known sites such as  arXiv  (mainly covering physics) and  CiteSeerX (mainly covering computer science) are probably trustworthy, as are the databases and web pages produced by entities such as the National Library of Medicine, the Smithsonian Institution, and the American Cancer Society.

Is this paper peer-reviewed? Ulrichsweb will tell you.

1) On the library home page , choose "Articles and Databases" --> "Databases" --> Ulrichsweb

2) Put in the title of the JOURNAL (not the article), in quotation marks so all the words are next to each other

3) Mouse over the black icon, and you'll see that it means "refereed" (which means peer-reviewed, because it's been looked at by referees or reviewers). This journal is not peer-reviewed, because none of the formats have a black icon next to it:

• << Previous: Evaluating Information
• Next: RefWorks Guide and Help >>
• Last Updated: Aug 27, 2024 4:55 PM
• URL: https://guides.library.jhu.edu/computer

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Prevent plagiarism. Run a free check.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved September 2, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

Literature Review Example/Sample

Detailed Walkthrough + Free Literature Review Template

If you’re working on a dissertation or thesis and are looking for an example of a strong literature review chapter , you’ve come to the right place.

In this video, we walk you through an A-grade literature review from a dissertation that earned full distinction . We start off by discussing the five core sections of a literature review chapter by unpacking our free literature review template . This includes:

• The literature review opening/ introduction section
• The theoretical framework (or foundation of theory)
• The empirical research
• The research gap
• The closing section

We then progress to the sample literature review (from an A-grade Master’s-level dissertation) to show how these concepts are applied in the literature review chapter. You can access the free resources mentioned in this video below.

PS – If you’re working on a dissertation, be sure to also check out our collection of dissertation and thesis examples here .

FAQ: Literature Review Example

Literature review example: frequently asked questions, is the sample literature review real.

Yes. The literature review example is an extract from a Master’s-level dissertation for an MBA program. It has not been edited in any way.

Can I replicate this literature review for my dissertation?

As we discuss in the video, every literature review will be slightly different, depending on the university’s unique requirements, as well as the nature of the research itself. Therefore, you’ll need to tailor your literature review to suit your specific context.

You can learn more about the basics of writing a literature review here .

Where can I find more examples of literature reviews?

The best place to find more examples of literature review chapters would be within dissertation/thesis databases. These databases include dissertations, theses and research projects that have successfully passed the assessment criteria for the respective university, meaning that you have at least some sort of quality assurance. 

The Open Access Thesis Database (OATD) is a good starting point. 

How do I get the literature review template?

You can access our free literature review chapter template here .

Is the template really free?

Yes. There is no cost for the template and you are free to use it as you wish. 

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

What will it take for you to guide me in my Ph.D research work?

Thank you so much for all this information. I am unable to download the literature review template and the excel worksheet. When I click the button it takes me to the top of the page. I would really love to use this template, thank you again!

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

How-to conduct a systematic literature review: A quick guide for computer science research

Angela carrera-rivera.

a Faculty of Engineering, Mondragon University

William Ochoa

Felix larrinaga.

b Design Innovation Center(DBZ), Mondragon University

Associated Data

• No data was used for the research described in the article.

Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:

• • Clearly defined strategies to follow for a systematic literature review in computer science research, and
• • Algorithmic method to tackle a systematic literature review.

Graphical abstract

Specifications table

Method details

A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.

Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.

Define PICOC and synonyms

The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.

Planning Step 1 “Defining PICOC keywords and synonyms”.

Formulate research questions

Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).

Research questions examples.

Select digital library sources

The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.

Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.

Define inclusion and exclusion criteria

Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.

Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.

Define the Quality Assessment (QA) checklist

Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.

Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.

Define the “Data Extraction” form

The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .

Planning Step 6 “Define data extraction form”. Examples of fields.

The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.

After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.

Build digital library search strings

Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:

(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”

Gather studies

Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.

Example of Advanced search on Scopus.

In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.

Study Selection and Refinement

The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.

In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.

Performing quality assessment (QA) in Parsif.al.

Data extraction

Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.

Example of data extraction form using Parsif.al.

The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.

Analysis and Report

The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.

The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.

Techniques for bibliometric analysis and examples.

Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple  data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.

[1] Keyword co-relationship analysis using clusterization in vos viewer.

This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.

Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.

The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].

SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.

CRediT author statement

Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.

Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.

1 https://parsif.al/

Data Availability

Supporting systematic literature reviews in computer science: the systematic literature review toolkit

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations.

• Younis H Eisa T Nasser M Sahib T Noor A Alyasiri O Salisu S Hayder I Younis H (2024) A Systematic Review and Meta-Analysis of Artificial Intelligence Tools in Medicine and Healthcare: Applications, Considerations, Limitations, Motivation and Challenges Diagnostics 10.3390/diagnostics14010109 14 :1 (109) Online publication date: 4-Jan-2024 https://doi.org/10.3390/diagnostics14010109
• Nugroho H Surendro K (2024) A Comprehensive Bibliometric Analysis of Missing Value Imputation IEEE Access 10.1109/ACCESS.2024.3357533 12 (14819-14846) Online publication date: 2024 https://doi.org/10.1109/ACCESS.2024.3357533
• Younis H Ruhaiyem N Badr A Abdul-Hassan A Alfadli I Binjumah W Altuwaijri E Nasser M (2023) Multimodal Age and Gender Estimation for Adaptive Human-Robot Interaction: A Systematic Literature Review Processes 10.3390/pr11051488 11 :5 (1488) Online publication date: 15-May-2023 https://doi.org/10.3390/pr11051488
• Show More Cited By

Index Terms

Applied computing

Computers in other domains

Digital libraries and archives

Recommendations

The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature.

Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey ...

Automation of systematic literature reviews: A systematic literature review

Systematic Literature Review (SLR) studies aim to identify relevant primary papers, extract the required data, analyze, and synthesize results to gain further and broader insight into the investigated domain. ...

Towards Sustainability of Systematic Literature Reviews

Background: The software engineering community has increasingly conducted systematic literature reviews (SLR) as a means to summarize evidence from different studies and bring to light the state of the art of a given research topic. While SLR provide ...

Information

Published in.

• SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates, author tags.

• systematic literature survey
• Research-article

Funding Sources

Acceptance rates, upcoming conference, contributors, other metrics, bibliometrics, article metrics.

• 9 Total Citations View Citations
• 770 Total Downloads
• Downloads (Last 12 months) 95
• Downloads (Last 6 weeks) 11
• Curtis H Neate T Vazquez Gonzalez C (2022) State of the Art in AAC: A Systematic Review and Taxonomy Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility 10.1145/3517428.3544810 (1-22) Online publication date: 23-Oct-2022 https://dl.acm.org/doi/10.1145/3517428.3544810
• Konersmann M Kaplan A Kuhn T Heinrich R Koziolek A Reussner R Jurjens J al-Doori M Boltz N Ehl M Fuchs D Groser K Hahner S Keim J Lohr M Saglam T Schulz S Toberg J (2022) Evaluation Methods and Replicability of Software Architecture Research Objects 2022 IEEE 19th International Conference on Software Architecture (ICSA) 10.1109/ICSA53651.2022.00023 (157-168) Online publication date: Mar-2022 https://doi.org/10.1109/ICSA53651.2022.00023
• Napoleao B Petrillo F Halle S (2021) Automated Support for Searching and Selecting Evidence in Software Engineering: A Cross-domain Systematic Mapping 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 10.1109/SEAA53835.2021.00015 (45-53) Online publication date: Sep-2021 https://doi.org/10.1109/SEAA53835.2021.00015
• Ullah H Letchmunan S Zia M Butt U Hassan F (2021) Analysis of Deep Neural Networks for Human Activity Recognition in Videos—A Systematic Literature Review IEEE Access 10.1109/ACCESS.2021.3110610 9 (126366-126387) Online publication date: 2021 https://doi.org/10.1109/ACCESS.2021.3110610
• Felizardo K Carver J (2020) Automating Systematic Literature Review Contemporary Empirical Methods in Software Engineering 10.1007/978-3-030-32489-6_12 (327-355) Online publication date: 28-Aug-2020 https://doi.org/10.1007/978-3-030-32489-6_12
• Bencomo N Götz S Song H (2019) [email protected] Software and Systems Modeling (SoSyM) 10.1007/s10270-018-00712-x 18 :5 (3049-3082) Online publication date: 2-Aug-2019 https://dl.acm.org/doi/10.1007/s10270-018-00712-x

View Options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

View options.

View or Download as a PDF file.

View online with eReader .

Share this Publication link

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

• UWF Libraries

Literature Review: Conducting & Writing

• Sample Literature Reviews
• Steps for Conducting a Lit Review
• Finding "The Literature"
• Organizing/Writing
• APA Style This link opens in a new window
• Chicago: Notes Bibliography This link opens in a new window
• MLA Style This link opens in a new window

Sample Lit Reviews from Communication Arts

Have an exemplary literature review.

• Literature Review Sample 1
• Literature Review Sample 2
• Literature Review Sample 3

Have you written a stellar literature review you care to share for teaching purposes?

Are you an instructor who has received an exemplary literature review and have permission from the student to post?

Please contact Britt McGowan at [email protected] for inclusion in this guide. All disciplines welcome and encouraged.

• << Previous: MLA Style
• Next: Get Help! >>
• Last Updated: Aug 8, 2024 11:00 AM
• URL: https://libguides.uwf.edu/litreview