This section should describe the critical appraisal process and instruments that will be used in the review process and the procedures for solving disagreements between reviewers.
The goal of critical appraisal (assessment of risk of bias) is to assess the methodological quality of a study and to determine the extent to which a study has excluded or minimized the possibility of bias in its design, conduct and analysis. Bias refers to systematic errors in the design, conduct and analysis of quantitative studies that may impact the validity of inferences from these studies. Critical appraisal of the studies included in a systematic review is performed with the explicit goal of identifying the risk of diverse biases in these studies. JBI uses standardized critical appraisal tools for the assessment of risk of diverse biases encountered in quantitative studies. There are JBI standardized appraisal tools based on study design appropriate for JBI reviews of effectiveness (see Appendix 3.2 regarding the JBI standardized appraisal tools). JBI systematic reviews are required to use these JBI standardized appraisal tools.Reviewers should refer in the review protocol to the JBI standardized critical appraisal checklists and provide references for these checklists. It is not necessary to provide these checklists in appendices of the review protocol. If non-JBI appraisal tools are proposed then these tools should be briefly described and correctly referenced. In this case, an explicit justification for the use of non-JBI appraisal tools should be provided in the review protocol.
Two reviewers should perform independent critical appraisal of retrieved studies using the standardized critical appraisal checklists developed by JBI. The protocol should specify that any disagreements are solved by consensus or by the decision of a third reviewer. In experimental studies (randomized experimental studies and quasi-experimental studies) the most important biases are: selection bias, performance bias, attrition bias, detection bias, and reporting bias. In observational studies the most important biases are: selection bias, information bias, and confounding. The review protocol should specify that reviewers plan to report in narrative form and in tables the results of risk of bias (methodological quality) assessments for each aspect of methodological quality (randomization; blinding; measurement; statistical analysis etc.) for each individual study and the overall risk of bias of the entire set of included studies. The critical appraisal phase of the review should not be treated as a rapid ‘box ticking exercise’ on checklists, but rather as a complex, profound, critical, systematic, thorough examination of the risk of bias of each included study, a solid foundation for an appropriate synthesis of the results.
The review protocol should specify if and how the results of critical appraisal will be used for the exclusion of studies from the review. For example, if studies judged of low methodological quality will be excluded from the review, the details of the circumstances under which such decisions will be made and the explicit criteria or decision rules should be explicitly provided, including explanations for what is considered low methodological quality by reviewers. It is the decision of the review team if they want to exclude from the review studies judged of low methodological quality. Reviewers should explain and justify their criteria and decision rules. The decision as to whether or not to include a study can be made based on meeting a predetermined proportion of all criteria, or on certain criteria being met. It is also possible to weight the different criteria differently. The decisions about the scoring system and the cut-off for inclusion of a study in the review should be made in advance and be agreed upon by all participating reviewers before critical appraisal commences. The review protocol should specify if and how the results of critical appraisal will be used in the synthesis (narrative synthesis or meta-analysis) of the results. It is recommended that the results of critical appraisal should be used in the synthesis phase of the review, for the critical examination of the impact of methodological quality of studies on results (including subgroup analysis or sensitivity analysis). JBI reviewers are encouraged to read the article by Porritt et al (2014) regarding study selection and critical appraisal.