Characteristics of the sources, evaluation, and grading of the certainty of evidence in systematic reviews in public health: A methodological study

ObjectivesTo systematically explore how the sources of evidence, types of primary studies, and tools used to assess the quality of the primary studies vary across systematic reviews (SRs) in public health. MethodsWe conducted a methodological survey of SRs in public health by searching the of literature in selected journals from electronic bibliographic databases. We selected a 10% random sample of the SRs that met the explicit inclusion criteria. Two researchers independently extracted data for analysis. ResultsWe selected 301 SRs for analysis: 94 (31.2%) of these were pre-registered, and 211 (70.1%) declared to have followed published reporting standard. All SRs searched for evidence in electronic bibliographic databases, and more than half (n = 180, 60.0%) searched also the references of the included studies. The common types of primary studies included in the SRs were primarily cross-sectional studies (n = 132, 43.8%), cohort studies (n = 126, 41.9%), randomized controlled trials (RCTs, n = 89, 29.6%), quasi-experimental studies (n = 83, 27.6%), case-control studies (n = 58, 19.3%) qualitative studies (n = 38, 12.6%) and mixed-methods studies (n = 32, 10.6%). The most frequently used quality assessment tools were the Newcastle-Ottawa Scale (used for 50.0% of cohort studies and 55.6% of case-control studies), Cochrane Collaboration's Risk of Bias tool (50.7% of RCTs) and Critical Appraisal Skills Program (38.5% of qualitative studies). Only 20 (6.6%) of the SRs assessed the certainty of the body of evidence, of which 19 (95.0%) used the GRADE approach. More than 65% of the evidence in the SRs using GRADE was of low or very low certainty. ConclusionsSRs should always assess the quality both at the individual study level and the body of evidence for outcomes, which will benefit patients, health care practitioners, and policymakers.

Automated summary

This study systematically explored the variations in the sources of evidence, types of primary studies, and tools used to assess the quality in systematic reviews (SRs) in public health. The majority of SRs included cross-sectional studies, cohort studies, and randomized controlled trials. The most commonly used quality assessment tools were the Newcastle-Ottawa Scale, Cochrane Collaboration's Risk of Bias tool, and Critical Appraisal Skills Program. Only a small percentage of SRs assessed the certainty of the evidence, mostly using the GRADE approach. In conclusion, SRs should assess quality at both the individual study and evidence framework level for the benefit of patients, healthcare providers, and policymakers.