Source: Foster, M. (2018). Systematic reviews service: Introduction to systematic reviews. Retrieved September 18, 2018, from
Writing a systematic review dissertation isn’t easy because you must follow a thorough and accurate scientific process. You must be an expert in research methodology to synthesise studies. In this article, I will provide a step-by-step approach to writing a top-notch systematic review dissertation.
Table of Contents
However, for students who may find this process challenging and seek professional assistance, I recommend exploring SystematicReviewPro —a reliable systematic review writing service. By signing up and placing a free inquiry and engaging with the admin team at any time, students can avail themselves of an exclusive offer of up to 50% off on their systematic review order. Additionally, there is already a 30% discount running on the website, making it an excellent opportunity to ease your dissertation journey.
As an Undergraduate or Master’s student, you’re are allowed to pick a systematic review for your dissertation. As a PhD student, you can use a systematic review methodology in the second chapter (literature review) of your dissertation. A systematic review is considered the highest level of empirical evidence, especially in clinical sciences like nursing and medicine. When developing new practice guidelines, new services, or new products, systematic reviews are searched and synthesised first on that topic or idea.
The nature of your research topic or research question.
Some research topics or questions strictly conform to qualitative or quantitative methods. For example, if you’re exploring the lived experiences, attitudes, perceptions, and meaning-making in a given population, you’ll need qualitative methods. However, you will require quantitative methods if looking into quantifiable variables like happiness, depression, academic performance, sleep, etc. That said, the nature of your research question should guide you. If your topic is qualitative, you’ll need qualitative studies only. If your topic is quantitative, you’ll need quantitative studies only. Systematic reviews of qualitative studies are less intricate than of quantitative studies. Still, they require a thoughtful approach in synthesizing findings from various qualitative studies.
If you choose to review quantitative studies, you might need to conduct a meta-analysis in your systematic review. A meta-analysis refers to statistical techniques used in pooling findings from various independent studies to compute a summary statistic. For example, in your dissertation, you may aim to investigate the effect of a student well-being programme embedded in university classes on the happiness of university students. Various studies that have investigated the same or a related intervention and quantitively measured happiness among university students must be synthesised together using a statistical technique. The ultimate outcome of that meta-analysis is to provide an overview of the overall trend of the effect of the intervention on university student’s happiness. For more information about how to formulate a research question for a systematic review with a meta-analysis, visit this link.
An example meta-analysis showing the statistical combination of findings from various studies to indicate the overall effect of a psychological intervention on the psychological well-being of university students.
Finding primary studies for your systematic review is the hardest thing you can encounter with this approach. You can choose your topic and plan your journey so well. Upon reaching the point you need primary studies to answer your research question, you get stuck. Retrieving primary studies is challenging because it requires advanced search strategies on various online databases. Doing an advanced search strategy can be an uphill task for someone who has never done a systematic review. This is because, more often than not, depending on the topic, primary studies are not readily available on the Internet. Remember, secondary studies, like systematic reviews and literature reviews, are not eligible for systematic reviews.
Always confirm with your supervisor if you can do a systematic review dissertation. Some supervisors may feel it better for you to do a primary study. So, always confirm with your supervisor before doing much.
Always ensure you’re confident that you can do a systematic review on your own. Writing a systematic review isn’t easy. You need to be aware that doing a systematic review may even be harder than doing interviews or surveys in primary research. Why? A systematic review involves combining many primary studies together in a scientific manner. That means you must have expertise in various research methodologies to know the best way to integrate or synthesise the various studies.
The main advantage of doing a systematic review dissertation is that it saves a lot of time. Conducting interviews or surveys can be time- and resource-consuming. However, with a systematic review, you do everything from your desk. It will save you a lot of time and resources. If you find that you meet many of the requirements of successfully conducting a systematic review, the next step is to engage in the actual process. The step-by-step approach used in writing systematic reviews is outlined below.
The following steps are iterative, meaning you can start over again and again until you meet your research objectives. The step-by-step guide on how to write a systematic review dissertation is summarized in the infographic shown below.
Step-by-step guide on how to write a systematic review dissertation
The starting point of a systematic review is to formulate a research question. As stated above, the nature of your research question will help you make key decisions. For example, you will be able to know which design (quantitative versus qualitative) to consider in your inclusion and exclusion criteria.
The next step is to perform a preliminary search on the Internet to determine if another systematic review has been published. It is not acceptable to repeat what has already been done. Your research should be novel and contribute to a knowledge gap. However, if you find that another systematic review has already been published on your topic. You should consider the publication date.
In most cases, systematic reviews on given topics are outdated. They have not used recent studies published on that topic, thus missing important updates. That can be a good reason you’re conducting your study. Suppose there’s an updated systematic review on your topic. In that case, you should consider reformulating your research question to address a specific knowledge gap.
One unique thing about systematic reviews is that they must be based on a very specific population, intervention/exposure, and assess a specific outcome. Let’s say, for example, you write on Intervention A’s effectiveness in reducing depression symptoms in older frail people. In that case, you must retrieve studies that strictly assess the effectiveness of Intervention A, the outcome being depression symptoms and the population being older frail people.
Therefore, it will be against the principles of a systematic review to focus on Intervention B (different intervention/exposure) on anxiety (different outcomes) in younger people (different populations). Also, depending on your research question, you will need to determine the research design (qualitative versus quantitative) of the studies you will review. Other criteria to consider are the country of publication, the publication date, language, etc.
As said, the main challenge in writing a systematic review is to identify papers. Your literature search should be thorough so that you don’t leave out some relevant studies. Developing a literature search strategy isn’t easy because you must start identifying relevant keywords and search terms for your topic. You must start by knowing common terminologies used in your subject of interest.
Afterward, combine the keywords using Boolean connectors like “AND” & “OR.” For example, suppose my topic is the effectiveness of cognitive behavioural therapy in treating anxiety in adolescents. In that regard, I can combine my keywords as follows: (Cognitive behavioural therapy OR CBT) AND (anxiety) AND (adolescents OR youth). If you use terminologies unknown in your discipline, you will likely not find relevant studies for review.
At this stage, you identify the databases you’ll use to execute your search strategy. When writing a systematic review dissertation, you also need to report the databases that you searched. Commonly searched ones in the field of social and health sciences include PubMed, Google Scholar, Cochrane, PsycInfo, and many others. You need to know how each database works. Also, apart from Google Scholar and PubMed, most of these databases require paid or institutional access. Liaise with your supervisor or librarian to help in identifying good databases for subject and discipline.
When you execute your search strategy on each database, results or search hits will be displayed. This is also another difficult step because of tedious work involved. You start by screening the titles. Then, eliminate results that contain irrelevant titles. You need to be careful at this point because sometimes people eliminate even relevant studies. The title doesn’t need to contain exactly your keywords. Some titles appear totally irrelevant but they actually contain useful data inside.
After screening titles, the next step is to screen abstracts. You may be surprised at this point that the titles you thought were irrelevant actually contain relevant information. For instance, some studies may indicate in the title that their study focused on depression as an outcome when you’re interested in anxiety. However, reading the abstract may surprise you that depression was only a primary outcome. The authors also measured secondary outcomes, among them anxiety. In such an article, you can decide to focus on anxiety results only because they are relevant to your study.
After screening articles identified using various databases, the next step is to augment the search strategy with a manual search. This will ensure you don’t miss relevant studies in your systematic review dissertation. The manual search involves identifying more studies in the bibliographies of the identified articles using a database search. It is also about contacting the authors and experts sourced from the found articles to give access to more articles that may not be found online. Finally, you can also identify key journals from the articles and perform a hand search. For example, suppose I identify the Journal of Cognitive Psychology. In that case, I will visit that journal’s website and perform a manual search there. A properly done manual search can help you identify more articles that you couldn’t have identified using databases only.
After having all your articles intact, the next step is to screen for full-text bodies. In most cases, the titles and abstracts may not contain enough information for screening purposes. You must read the full texts of the articles to determine their full eligibility. At this point, you screen articles identified through database search and manual search altogether. For example, sometimes you may be interested in healthy adolescents. In the abstract, the author of the articles may only report adolescents without providing any specifics about them. Upon reading the full text, you may discover that the authors included adolescents with mental issues that are not within your study’s scope. Therefore, always do a full-text screening before you move to the next step.
Systematic review dissertations can be used to inform the formulation of practice guidelines and even inform policies. You must strive to review only studies with rigorous methodological quality. The quality assessment tool will depend on your study’s design. The commonly used ones for student dissertations include CASP Checklists and Joanna Briggs Institute (JBI) Checklists. You can consult with your supervisor before arriving at the final decision. Transparently report your quality assessment findings. For example, indicate the score of each study under each item of each tool and calculate the overall score in the form of a percentage. Also, always have a cut-off of 65%, and studies whose methodological rigour is below the cut-off are excluded.
The next step is to extract relevant data from your studies. Your data extraction approach depends on the research design of the studies you used. If you use qualitative studies, your data extraction can focus on individual studies’ findings, particularly themes. You can also extract data that can aid in-depth analysis, such as country of study, population characteristics, etc. Using quantitative studies, you can collect quantitative data that will aid your analysis, such as means and standard deviations and other crucial information relevant to your analysis technique. Always chart your data in a tabular format to facilitate easy management and handling.
The data analysis approach used in your systematic review dissertation will depend on the research design. Using qualitative studies, you will rely on qualitative approaches to analyse your data. For example, you can do a thematic analysis or a narrative synthesis. If you used quantitative studies, you might need to perform a meta-analysis or narrative synthesis. A meta-analysis is done when you have homogenous studies (such as population, outcome variables, measurement tools, etc.) that are experimental in nature. Particularly, meta-analysis is performed when reviewing controlled randomized trials or other interventional studies. In other words, meta-analysis is appropriately used when reviewing the effectiveness of interventions. However, if your quantitative studies are heterogenous, such as using different research designs, you must perform a narrative synthesis.
The final step is to produce a written report of your systematic review dissertation. One of the ethical concerns in systematic reviews is transparency. You can improve the transparency of your reporting by using an established protocol like PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses).
Approximate price: $ 22
Delivering a high-quality product at a reasonable price is not enough anymore. That’s why we have developed 5 beneficial guarantees that will make your experience with our service enjoyable, easy, and safe.
You have to be 100% sure of the quality of your product to give a money-back guarantee. This describes us perfectly. Make sure that this guarantee is totally transparent.
Each paper is composed from scratch, according to your instructions. It is then checked by our plagiarism-detection software. There is no gap where plagiarism could squeeze in.
Thanks to our free revisions, there is no way for you to be unsatisfied. We will work on your paper until you are completely happy with the result.
Your email is safe, as we store it according to international data protection rules. Your bank details are secure, as we use only reliable payment systems.
By sending us your money, you buy the service we provide. Check out our terms and conditions if you prefer business talks to be laid out in official language.
Student resources, chapter 1. carrying out a systematic review as a master's thesis.
Here are some other helpful materials for you to read through.
Centre for Review and Dissemination’s guidance for undertaking systematic reviews in health care
www.york.ac.uk/crd/guidance
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .
Prabhakar veginadu.
1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia
2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK
3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India
Associated data.
APPENDIX B: List of excluded studies with detailed reasons for exclusion
APPENDIX C: Quality assessment of included reviews using AMSTAR 2
The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.
The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.
The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.
The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.
Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7
The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5
Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.
An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3
Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.
SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14
The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.
The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.
Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.
Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).
The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.
To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.
From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure 1 . A list of excluded studies with reasons can be found in Appendix B .
Study selection flowchart
Table 1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23
Characteristics of included studies
Author, year | Search strategy (year last searched; no. databases; supplementary searches) | SR design (type of review; no. of studies included) | Topic; subject area | SR objectives | SR authors’ comments on study quality |
---|---|---|---|---|---|
Crumley, 2005 | 2004; Seven databases; four journals handsearched, reference lists and contacting authors | SR; = 64 | RCTs and CCTs; not specified | To identify and quantitatively review studies comparing two or more different resources (e.g., databases, Internet, handsearching) used to identify RCTs and CCTs for systematic reviews. | Most of the studies adequately described reproducible search methods, expected search yield. Poor quality in studies was mainly due to lack of rigor in reporting selection methodology. Majority of the studies did not indicate the number of people involved in independently screening the searches or applying eligibility criteria to identify potentially relevant studies. |
Hopewell, 2007 | 2002; eight databases; selected journals and published abstracts handsearched, and contacting authors | SR and MA; = 34 (34 in quantitative analysis) | RCTs; health care | To review systematically empirical studies, which have compared the results of handsearching with the results of searching one or more electronic databases to identify reports of randomized trials. | The electronic search was designed and carried out appropriately in majority of the studies, while the appropriateness of handsearching was unclear in half the studies because of limited information. The screening studies methods used in both groups were comparable in most of the studies. |
Hopewell, 2007 | 2005; two databases; selected journals and published abstracts handsearched, reference lists, citations and contacting authors | SR and MA; = 5 (5 in quantitative analysis) | RCTs; health care | To review systematically research studies, which have investigated the impact of gray literature in meta‐analyses of randomized trials of health care interventions. | In majority of the studies, electronic searches were designed and conducted appropriately, and the selection of studies for eligibility was similar for handsearching and database searching. Insufficient data for most studies to assess the appropriateness of handsearching and investigator agreeability on the eligibility of the trial reports. |
Horsley, 2011 | 2008; three databases; reference lists, citations and contacting authors | SR; = 12 | Any topic or study area | To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists. | Interpretability and generalizability of included studies was difficult. Extensive heterogeneity among the studies in the number and type of databases used. Lack of control in majority of the studies related to the quality and comprehensiveness of searching. |
Morrison, 2012 | 2011; six databases and gray literature | SR; = 5 | RCTs; conventional medicine | To examine the impact of English language restriction on systematic review‐based meta‐analyses | The included studies were assessed to have good reporting quality and validity of results. Methodological issues were mainly noted in the areas of sample power calculation and distribution of confounders. |
Robson, 2019 | 2016; three databases; reference lists and contacting authors | SR; = 37 | N/R | To identify and summarize studies assessing methodologies for study selection, data abstraction, or quality appraisal in systematic reviews. | The quality of the included studies was generally low. Only one study was assessed as having low RoB across all four domains. Majority of the studies were assessed to having unclear RoB across one or more domains. |
Schmucker, 2017 | 2016; four databases; reference lists | SR; = 10 | Study data; medicine | To assess whether the inclusion of data that were not published at all and/or published only in the gray literature influences pooled effect estimates in meta‐analyses and leads to different interpretation. | Majority of the included studies could not be judged on the adequacy of matching or adjusting for confounders of the gray/unpublished data in comparison to published data. |
Also, generalizability of results was low or unclear in four research projects | |||||
Morissette, 2011 | 2009; five databases; reference lists and contacting authors | SR and MA; = 6 (5 included in quantitative analysis) | N/R | To determine whether blinded versus unblinded assessments of risk of bias result in similar or systematically different assessments in studies included in a systematic review. | Four studies had unclear risk of bias, while two studies had high risk of bias. |
O'Mara‐Eves, 2015 | 2013; 14 databases and gray literature | SR; = 44 | N/R | To gather and present the available research evidence on existing methods for text mining related to the title and abstract screening stage in a systematic review, including the performance metrics used to evaluate these technologies. | Quality appraised based on two criteria‐sampling of test cases and adequacy of methods description for replication. No study was excluded based on the quality (author contact). |
SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.
The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n = 3), literature search ( n = 3), study selection ( n = 2), data extraction ( n = 1), and RoB assessment ( n = 2) (Table 2 ).
Summary of findings from review evaluating systematic review methods
Key elements | Author, year | Method assessed | Evaluations/outcomes (P—primary; S—secondary) | Summary of SR authors’ conclusions | Quality of review |
---|---|---|---|---|---|
Excluding study data based on publication status | Hopewell, 2007 | Gray vs. published literature | Pooled effect estimate | Published trials are usually larger and show an overall greater treatment effect than gray trials. Excluding trials reported in gray literature from SRs and MAs may exaggerate the results. | Moderate |
Schmucker, 2017 | Gray and/or unpublished vs. published literature | P: Pooled effect estimate | Excluding unpublished trials had no or only a small effect on the pooled estimates of treatment effects. Insufficient evidence to conclude the impact of including unpublished or gray study data on MA conclusions. | Moderate | |
S: Impact on interpretation of MA | |||||
Excluding study data based on language of publication | Morrison, 2012 | English language vs. non‐English language publications | P: Bias in summary treatment effects | No evidence of a systematic bias from the use of English language restrictions in systematic review‐based meta‐analyses in conventional medicine. Conflicting results on the methodological and reporting quality of English and non‐English language RCTs. Further research required. | Low |
S: number of included studies and patients, methodological quality and statistical heterogeneity | |||||
Resources searching | Crumley, 2005 | Two or more resources searching vs. resource‐specific searching | Recall and precision | Multiple‐source comprehensive searches are necessary to identify all RCTs for a systematic review. For electronic databases, using the Cochrane HSS or complex search strategy in consultation with a librarian is recommended. | Critically low |
Supplementary searching | Hopewell, 2007 | Handsearching only vs. one or more electronic database(s) searching | Number of identified randomized trials | Handsearching is important for identifying trial reports for inclusion in systematic reviews of health care interventions published in nonindexed journals. Where time and resources are limited, majority of the full English‐language trial reports can be identified using a complex search or the Cochrane HSS. | Moderate |
Horsley, 2011 | Checking reference list (no comparison) | P: additional yield of checking reference lists | There is some evidence to support the use of checking reference lists to complement literature search in systematic reviews. | Low | |
S: additional yield by publication type, study design or both and data pertaining to costs | |||||
Reviewer characteristics | Robson, 2019 | Single vs. double reviewer screening | P: Accuracy, reliability, or efficiency of a method | Using two reviewers for screening is recommended. If resources are limited, one reviewer can screen, and other reviewer can verify the list of excluded studies. | Low |
S: factors affecting accuracy or reliability of a method | |||||
Experienced vs. inexperienced reviewers for screening | Screening must be performed by experienced reviewers | ||||
Screening by blinded vs. unblinded reviewers | Authors do not recommend blinding of reviewers during screening as the blinding process was time‐consuming and had little impact on the results of MA | ||||
Use of technology for study selection | Robson, 2019 | Use of dual computer monitors vs. nonuse of dual monitors for screening | P: Accuracy, reliability, or efficiency of a method | There are no significant differences in the time spent on abstract or full‐text screening with the use and nonuse of dual monitors | Low |
S: factors affecting accuracy or reliability of a method | |||||
Use of Google translate to translate non‐English citations to facilitate screening | Use of Google translate to screen German language citations | ||||
O'Mara‐Eves, 2015 | Use of text mining for title and abstract screening | Any evaluation concerning workload reduction | Text mining approaches can be used to reduce the number of studies to be screened, increase the rate of screening, improve the workflow with screening prioritization, and replace the second reviewer. The evaluated approaches reported saving a workload of between 30% and 70% | Critically low | |
Order of screening | Robson, 2019 | Title‐first screening vs. title‐and‐abstract simultaneous screening | P: Accuracy, reliability, or efficiency of a method | Title‐first screening showed no substantial gain in time when compared to simultaneous title and abstract screening. | Low |
S: factors affecting accuracy or reliability of a method | |||||
Reviewer characteristics | Robson, 2019 | Single vs. double reviewer data extraction | P: Accuracy, reliability, or efficiency of a method | Use two reviewers for data extraction. Single reviewer data extraction followed by the verification of outcome data by a second reviewer (where statistical analysis is planned), if resources preclude | Low |
S: factors affecting accuracy or reliability of a method | |||||
Experienced vs. inexperienced reviewers for data extraction | Experienced reviewers must be used for extracting continuous outcomes data | ||||
Data extraction by blinded vs. unblinded reviewers | Authors do not recommend blinding of reviewers during data extraction as it had no impact on the results of MA | ||||
Use of technology for data extraction | Use of dual computer monitors vs. nonuse of dual monitors for data extraction | Using two computer monitors may improve the efficiency of data extraction | |||
Data extraction by two English reviewers using Google translate vs. data extraction by two reviewers fluent in respective languages | Google translate provides limited accuracy for data extraction | ||||
Computer‐assisted vs. double reviewer extraction of graphical data | Use of computer‐assisted programs to extract graphical data | ||||
Obtaining additional data | Contacting study authors for additional data | Recommend contacting authors for obtaining additional relevant data | |||
Reviewer characteristics | Robson, 2019 | Quality appraisal by blinded vs. unblinded reviewers | P: Accuracy, reliability, or efficiency of a method | Inconsistent results on RoB assessments performed by blinded and unblinded reviewers. Blinding reviewers for quality appraisal not recommended | Low |
S: factors affecting accuracy or reliability of a method | |||||
Morissette, 2011 | Risk of bias (RoB) assessment by blinded vs. unblinded reviewers | P: Mean difference and 95% confidence interval between RoB assessment scores | Findings related to the difference between blinded and unblinded RoB assessments are inconsistent from the studies. Pooled effects show no differences in RoB assessments for assessments completed in a blinded or unblinded manner. | Moderate | |
S: qualitative level of agreement, mean RoB scores and measures of variance for the results of the RoB assessments, and inter‐rater reliability between blinded and unblinded reviewers | |||||
Robson, 2019 | Experienced vs. inexperienced reviewers for quality appraisal | P: Accuracy, reliability, or efficiency of a method | Reviewers performing quality appraisal must be trained. Quality assessment tool must be pilot tested. | Low | |
S: factors affecting accuracy or reliability of a method | |||||
Use of additional guidance vs. nonuse of additional guidance for quality appraisal | Providing guidance and decision rules for quality appraisal improved the inter‐rater reliability in RoB assessments. | ||||
Obtaining additional data | Contacting study authors for obtaining additional information/use of supplementary information available in the published trials vs. no additional information for quality appraisal | Additional data related to study quality obtained by contacting study authors improved the quality assessment. | |||
RoB assessment of qualitative studies | Structured vs. unstructured appraisal of qualitative research studies | Use of structured tool if qualitative and quantitative studies designs are included in the review. For qualitative reviews, either structured or unstructured quality appraisal tool can be used. |
There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n = 4), Hopewell et al. 20 and Crumley et al. 19 ( n = 30), and Robson et al. 14 and Morissette et al. 23 ( n = 4). There were no conflicting results between any of the identified SRs on the same topic.
Overall, the quality of the included reviews was assessed as moderate at best (Table 2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .
3.3.1. methods for defining review scope and eligibility.
Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26
Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”
Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19
In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20
The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22
Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.
Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14
Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.
As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.
Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23
Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.
4.1. summary of findings.
Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.
The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27
Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.
Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.
Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14
One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.
The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.
To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.
There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.
Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.
This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.
The authors declare no conflicts of interest.
APPENDIX A: Detailed search strategies
The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.
Open Access Funding provided by La Trobe University.
Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
Related media.
Hours | |
---|---|
7:30am – 2:00am | |
7:30am – 2:00am | |
7:30am – 2:00am | |
8:00am – 5:00pm | |
Reference Desk | 9:00am – 10:00pm |
I = identify: search for studies that match your criteria, e = evaluate: exclude or include studies, c = collect: extract and synthesize key data, e = explain: give context and rate the strength of the studies, s = summarize: write and publish your final report.
Congratulations!
You've decided to conduct a Systematic Review! Please see the associated steps below. You can follow the P-I-E-C-E-S = Plan, Identify, Evaluate, Collect, Explain, Summarize system or any number of systematic review processes available (Foster & Jewell, 2017) .
P = Plan: decide on your search methods
By now you should have identified gaps in the field and have a specific question you are seeking to answer. This will likely have taken several iterations and is the most important part of the Systematic Review process.
Once you've finalized a research question, you should be able to locate existing systematic reviews on or similar to your topic. existing systematic reviews will be your clues to mine for keywords, sample searches in various databases, and will help your team finalize your review question and develop your inclusion and exclusion criteria. , decide on a protocol and reporting standard, your protocol is essentially a project plan and data management strategy for an objective, reproducible, sound methodology for peer review. the reporting standard or guidelines are not a protocols, but rather a set of standards to guide the development of your systematic review. often they include checklists. it is not required, but highly recommended to follow a reporting standard. .
Protocol registry: Reviewing existing systematic reviews and registering your protocol will increase transparency, minimize bias, and reduce the redundancy of groups working on the same topics ( PLoS Medicine Editors, 2011 ). Protocols can serve as internal or external documents. Protocols can be made public prior to the review. Some registries allow for keeping a protocol private for a set period of time.
Cochrane Database of Systematic Reviews (UGA Login) (Health Sciences)
A collection of regularly updated, systematic reviews of the effects of health care. New reviews are added with each issue of The Cochrane Library Reviews mainly of randomized controlled trials. All reviews have protocols.
PROSPERO (General)
This is an international register of systematic reviews and is public.
Campbell Corporation (Education & Social Sciences)
Topics covered include Ageing; Business and Management; Climate Solutions; Crime and Justice; Disability; Education; International Development; Knowledge Translation and Implementation; Methods; Nutrition and Food Security; Sexual Orientation and Gender Identity; Social Welfare; and Training.
Systematic Review for Animals and Food (Vet Med & Animal Science)
Reporting Standards:
Campbell MECCIR Standards (Education & Social Sciences)
Cochrane Guides & Handbooks (Health & Medical Sciences)
Institute of Medicine of the National Academies: Finding What Works in Healthcare: Standards for Systematic Reviews (healthcare)
Because the purpose of a SR is to find all studies related to your research question, you will need to search multiple databases. You should be able to name the databases you are already familiar with using. Your librarian will be able to recommend additional databases, including some of the following:
Depending on your topic, you may want to search clinical trials and grey literature. See this guide for more on grey literature.
Go here for help with writing your search strategy
Each database you use will have different methods of searching and resulting search strings, including syntax. ideally you will create one master keyword list and translate it for each database. below are tools to assist with translating search strings. .
Includes syntax for Cochrane Library, EBSCO, ProQuest, Ovid, and POPLINE.
The IEBH SR-Accelerator is a suite of tools to assist in speeding up portions of the Systematic Review process, including the Polyglot tool which translates searches across databases.
University of Michigan Search 101 - SR Database Cheat Sheet
Because systematic review literature searches may produce thousands of citations and abstracts, the research team will be screening and systematically reviewing large amounts of results. During screening , you will remove duplicates and remove studies that are not relevant to your topic based on a review of titles and abstracts. Of what remains, the full-text screening of the studies will then need to be conducted to confirm that they fit within your inclusion/exclusion criteria.
The results of the literature review and screening processes are best managed by various tools and software. You can also use a simple form or table to log the relevant information from each study. Consider whether you will be coding your data during the extraction process in your decision on which tool or software to use. Your librarian can consult on which of these is best suited to your research needs.
Data extraction processes differ between qualitative and quantitative evidence syntheses. In both cases, you must provide the reader with a clear overview of the studies you have included, their similarities and differences, and the findings. Extraction should be done in accordance to pre-established guidelines, such as PRISMA.
Some systematic reviews contain meta-analysis of the quantitative findings of the results. Consider including a statistician on your team to complete the analysis of all individual study results. Meta-analysis will tell you how much or what the actual results is across the studies and explains results in a measure of variance, typically called a forest plot.
Systematic review price models have changed over the years. Previously, you had to depend on departmental access to software that would cost several hundred dollars. Now that the software is cloud-based, tiered payment systems are now available. Sometimes there is a free tier level, but costs go up for functionality, number or users, or both. Depending on the organization's model, payments may be monthly, annual or per project/review.
Software list
Tool created by Brown University to assist with screening for systematic reviews.
Review Manager (RevMan) is the software used for preparing and maintaining Cochrane Reviews.
Systematic review tool intended to assist with the screening and extraction process. (Requires subscription)
DistillerSR is an online application designed specifically for the screening and data extraction phases of a systematic review (Requires subscription) Student and Faculty tiers have monthly pricing with a three month minimum. Number of projects is limited by pricing.
It includes features such as text mining, data clustering, classification and term extraction
Rayyan is a free web-based application that can be used to screen titles, abstracts, and full text. Allows for multiple simultaneous users.
"System for the Unified Management, Assessment and Review of Information, the Joanna Briggs Institutes premier software for the systematic review of literature."
PRISMA guidelines suggest including critical appraisal of the included studies to assess the risk of bias and to include the assessment in your final manuscripts. There are several appraisal tools available depending on your discipline and area of research.
Simple overview of risk of bias assessment, including examples of how to assess and present your conclusions.
CASP is an organization that provides resources for healthcare professionals, but their appraisal tools can be used for varying study types across disciplines.
From the Joanna Briggs Institute: "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers."
Johns Hopkins Evidence-Based Practice Model (health sciences)
National Academies of Sciences, Engineering, and Medicine
Document the search; 5.1.6. Include a methods section
List of additional critical appraisal tools from Cardiff University.
Prepare your process and findings in a final manuscript. Be sure to check your PRISMA checklist or other reporting standard. You will want to include the full formatted search strategy for the appendix, as well as include documentation of your search methodology. A convenient way to illustrate this process is through a PRISMA Flow Diagram.
Attribution: Unless noted otherwise, this section of the guide was adapted from Texas A&M's "Systematic Reviews and Related Evidence Syntheses"
Listed below are some of the best examples of research projects and dissertations from undergraduate and taught postgraduate students at the University of Leeds We have not been able to gather examples from all schools. The module requirements for research projects may have changed since these examples were written. Refer to your module guidelines to make sure that you address all of the current assessment criteria. Some of the examples below are only available to access on campus.
These dissertations achieved a mark of 80 or higher:
|
The following two examples have been annotated with academic comments. This is to help you understand why they achieved a good 2:1 mark but also, more importantly, how the marks could have been improved. Please read to help you make the most of the two examples. (Mark 68) (Mark 66) These final year projects achieved a mark of a high first:
For students undertaking a New Venture Creation (NVC) approach, please see the following Masters level examples:
|
|
|
Projects which attained grades of over 70 or between 60 and 69 are indicated on the lists (accessible only by students and staff registered with School of Computer Science, when on campus).
|
These are good quality reports but they are not perfect. You may be able to identify areas for improvement (for example, structure, content, clarity, standard of written English, referencing or presentation quality).
|
|
The following examples have their marks and feedback included at the end of of each document.
The following examples have their feedback provided in a separate document.
|
|
School of Media and Communication . |
|
The following outstanding dissertation example PDFs have their marks denoted in brackets. (Mark 78) (Mark 91) (Mark 85) |
This dissertation achieved a mark of 84: . |
LUBS5530 Enterprise
|
MSc Sustainability
|
|
|
. |
The following outstanding dissertation example PDFs have their marks denoted in brackets. (Mark 70) (Mark 78) |
IMAGES
COMMENTS
Background. A systematic review, as its name suggests, is a systematic way of collecting, evaluating, integrating, and presenting findings from several studies on a specific question or topic.[] A systematic review is a research that, by identifying and combining evidence, is tailored to and answers the research question, based on an assessment of all relevant studies.[2,3] To identify assess ...
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr. Robert Boyle and his colleagues published a systematic review in ...
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question.
A Systematic Review (SR) is a synthesis of evidence that is identified and critically appraised to understand a specific topic. ... This may also include conference papers, conference presentations, theses and dissertations. Some SRs chose to exclude grey literature, whilst others find that it must be included. This may also mean that searches ...
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
by Angela Boland, M. Gemma Cherry and Rumona Dickson. Chapter 1. Carrying Out a Systematic Review as a Master's Thesis. Explore the wealth of resources available across the web. Here are some good places to start. Link to the Campbell Collaboration, an organization that prepares, maintains and disseminates systematic reviews in education, crime ...
undertake systematic reviews as dissertation projects. Students find that they can manage their projects more effectively and meet academic assessment deadlines by conducting a systematic review (SR) dissertation project. Undertaking a systematic review means that the student is not required to go through potentially lengthy ethical
Method details Overview. A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12].An SLR updates the reader with current literature about a subject [6].The goal is to review critical points of current knowledge on a ...
A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr Robert Boyle and his colleagues published a systematic review in ...
Doing a Systematic Review by Angela Boland (Editor); Gemma Cherry (Editor); Rumona Dickson (Editor) If you are a Masters or a PhD student conducting a systematic review for your dissertation or thesis, then this is the book for you! Written by an expert team of authors with years of experience in conducting systematic reviews and supervising ...
A review earns the adjective systematic if it is based on a clearly formulated question, identifies relevant studies, appraises their quality and summarizes the evidence by use of explicit methodology. It is the explicit and systematic approach that distinguishes systematic reviews from traditional reviews and commentaries.
fined criteria in order to answer a research question. The quantitative combination and statistical synthesis of the systema. cally-collected data is what defines a meta-analysis. Here, we first attempt to delineate the basic steps for conducting a systematic review: initial planning, conduc.
Review. Narrative reviews are a discussion of important topics on a theoretical point of view, and they are considered an important educational tool in continuing medical education [].Narrative reviews take a less formal approach than systematic reviews in that narrative reviews do not require the presentation of the more rigorous aspects characteristic of a systematic review such as reporting ...
A systematic review usually involves more than one person in order to increase the objectivity and trustworthiness of the reviews methods and findings. ... In a student dissertation, for example, there may not be the time to be fully systematic in a review of the literature if this is only one small part of the thesis. In other types of ...
Image: https://pixabay.com Steps to conducting a systematic review: PIECES. P: Planning - the methods of the systematic review are generally decided before conducting it. I: Identifying - searching for studies which match the preset criteria in a systematic manner E: Evaluating - sort all retrieved articles (included or excluded) and assess the risk of bias for each included study
Systematic review dissertations can be used to inform the formulation of practice guidelines and even inform policies. You must strive to review only studies with rigorous methodological quality. The quality assessment tool will depend on your study's design. The commonly used ones for student dissertations include CASP Checklists and Joanna ...
Doing a Systematic Review: A Student's Guide. by Angela Boland, M. Gemma Cherry and Rumona Dickson. Chapter 1: Carrying Out a Systematic Review as a Master's Thesis. What Is The Difference Between A Systematic Review And A Meta-Analysis?
by Angela Boland, M. Gemma Cherry and Rumona Dickson. Chapter 1: Carrying Out a Systematic Review as a Master's Thesis. Chapter 3: Defining My Review Question and Identifying Inclusion and Exclusion Criteria. Chapter 5: Applying Inclusion and Exclusion Criteria. Chapter 8: Understanding and Synthesizing Numerical Data from Intervention Studies.
dissertation, a systematic review may replace the literature review section. 2. The SR can be the methodological focus for candidacy exams (e.g. the student does not have a second methodological focus for candidacy). 3. The supervisor and supervisory committee must agree to support the student in conducting a systematic review.
1. INTRODUCTION. Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the "gold standard" of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search ...
Updated presentation on 30 October 2023This presentation covered: what a systematic review (SR) is pros and cons of doing a SR for your dissertation summary of common questions or problems encountered in SR dissertations, and answers or advice general guidance on what to include in your final reportThe slides (see attachments) include links to many further tools, guidance and methods ...
Systematic review tool intended to assist with the screening and extraction process. (Requires subscription) Distiller SR. DistillerSR is an online application designed specifically for the screening and data extraction phases of a systematic review (Requires subscription) Student and Faculty tiers have monthly pricing with a three month ...
Dissertation examples. Listed below are some of the best examples of research projects and dissertations from undergraduate and taught postgraduate students at the University of Leeds We have not been able to gather examples from all schools. The module requirements for research projects may have changed since these examples were written.
le project management (APM), the thesis uses a systematic literature review. Since managing projects in agile way is a relatively new concept compared to the traditional waterfall model, the resu. ts of the review provide an overview of the research conducted in this area. The results are expected t. he.
The extensive and comprehensive systematic review and meta analysis of Shool et al (Citation 2024) complements the work of Liu et al. (Citation 2008), the only systematic review on motorcycle helmet use and injury outcomes in the Cochrane database. Shool and collaborators aimed to identify the underlying causes of the variation in helmet usage ...