GRADE Approach in Systematic Reviews: How to Assess Certainty of Evidence

Introduction

The GRADE approach is a rating of authors’ confidence in the findings of a systematic review. GRADE is an acronym for the Grading of Recommendations Assessment, Development and Evaluation. You do not grade individual studies using GRADE. Instead, you evaluate the totality of the body of evidence that addresses the outcome and arrive at a rating of high, moderate, low, or very low.

Why go through all this? With a pooled estimate, readers learn the results of the studies. However, they do not learn how much to trust the results. With a small number of unblinded studies, you can still perform a meta-analysis and arrive at a neat number with small decimals. With GRADE, you must say what is wrong with that number. It is utilized by Cochrane, the World Health Organization and other guideline groups.

That body of evidence comes out of your screening and extraction. How to write a systematic review and meta-analysis covers each stage in the sequence GRADE will later read.

Why Is GRADE Important in Evidence Synthesis?

Let’s start with a common mistake. A p-value of 0.03 tells you the result is probably due to random chance. It does not tell you whether the trials were blinded. It does not tell you if trials included your patient population. It does not tell you whether three negative trials are sitting unpublished in someone’s drawer.

That drawer problem has a name and a set of tools for detecting it. Publication bias in meta-analysis covers funnel plots, Egger’s test and the trim-and-fill method, all of which feed directly into the publication bias domain of GRADE.

GRADE assesses all of these. When a result is determined to be “moderate certainty”, it tells the reader that the estimate is probably close to the true value, and that future studies may change it. Confidence intervals do not give this information.

There are three things beginning analysts confuse:

– Study results: what effect the studies found.

– Study limitations: what was wrong with the studies.

– Conflicts of interest: who funded the studies: How confident are you in those estimates? Would a guideline panel say to act?

GRADE addresses the second one. More is required for recommendations. Guideline panels use an evidence-to-decision framework, where they weigh the effect of certainty against the benefits, harms, values, costs, and feasibility of the situation. Thus, high-certainty evidence does not always translate to a strong recommendation. Low-certainty evidence does on occasion.

There are four levels of GRADE evidence certainty

Certainty level	What it means	In practice
High	The true effect is very likely close to the estimate.	New research is unlikely to change the conclusion.
Moderate	The true effect is probably close, but it could be substantially different.	New research could change the estimate.
Low	Confidence in the estimate is limited. The true effect may be substantially different.	New research will likely change the estimate.
Very low	Confidence in the estimate is very limited. The true effect is likely substantially different.	Any estimate is very uncertain

For intervention questions, GRADE starts randomized control trials at high, and observational studies at low, since complete ruling out of confounding is not possible in observational studies.

Use these only as initial guesstimates. You should use trial evidence to rate certainty after you’ve checked all five domains. Sometimes, if there’s a good reason, you can justify rating certainty up based on observational evidence (which we’ll discuss later).

What are the five GRADE domains?

Each domain is a reason to rate certainty down. For each outcome, ask: is this concern not serious, serious (down one level), or very serious (down two)?

 Risk of Bias

Risk of bias covers the flaws in the design, conduct, and analysis of the studies. This includes things like inadequate allocation concealment, having outcome assessors that also know the group assignments, or 30% of the participants being lost to follow-up. Your rating should be reflective of your risk of bias assessment tool, i.e. if you are doing a meta-analysis of randomized trials (RCTs), it should be based on RoB 2.

What RoB 2 asks for, domain by domain, is covered in risk of bias assessment in systematic reviews, including ROBINS-I for non-randomised studies and Newcastle-Ottawa for observational designs.

Using the example above, let’s say there were five trials that reported pain scores, and the three that contributed the most weight (70%) did not blind the participants. Since pain is a subjective outcome and patients who know they received the active treatment may report that they feel better, you may want to rate it down one level.

Inconsistency of Results

Inconsistency of results occurs when studies don’t agree with each other, and you can’t find an explanation. Look at the point estimates. Do they scatter? Do the confidence intervals overlap? Is there a lot of heterogeneity? If each of the studies favors the treatment, but only varies in the degree of effect, then a high I² may not be that concerning.

I², Cochran’s Q and tau squared each answer a different question about that disagreement. Heterogeneity in meta-analysis covers how to read all three and when high inconsistency warrants rating down.

Imagine you have six trials of a weight loss drug. Four of them demonstrate a beneficial effect, and the other two, while not statistically significant, do demonstrate a slight adverse effect. The differences in the dose, the population, and the length of follow-up are similar for all six trials. You have no rationale for the difference you see. Therefore, you would probably rate down.

 Imprecision of Evidence

Imprecision of evidence has to do with random error, and it usually causes more problems for beginners than any other domain. Evidence is imprecise when the confidence interval is so broad that it encompasses more than one of your clinical decisions of interest, or when the analysis is based on very few patients or very few events.

 The new GRADE guidance asks, Where does the confidence interval lie relative to a threshold that is of clinical importance to you? The threshold could be the line of no effect, or the smallest effect that would be of clinical interest to you. If the confidence interval crosses that, usually certainty goes down. Sample size, in and of itself, does not determine this. Also, you could compare the sample size to the optimal information size, meaning the sample size required for a properly powered study.

Finding the right balance of detail and imprecision is difficult. There is a worked example further down.

Publication Bias

Studies with positive results get published more than negative or null results. When this happens, the body of evidence you find overstates the benefits and understates the harms. Publication bias is strongly suspected in the presence of a disproportionate number of small studies (commonly industry-funded), an asymmetric funnel plot, or registered trials with no reported results.

In practice, let’s say you find 6 completed trials in a registry for a supplement. Only the 2 positive trials got published. In this case, GRADE asks if there is “strongly suspected” publication bias. In this case, there likely is.

Publication bias is one signal among several. Publication bias in meta-analysis covers how to test for funnel plot asymmetry with Egger’s test and Begg’s test, and why a significant result doesn’t prove missing studies.

Confirm if upgrading is applicable

Generally, this applies when there are no serious issues in the five domains and observations show a large effect size, or a dose-response relation, or where plausible confounding would likely have reduced the effect observed.

Finalize the rating and justify the decision

It is not necessary to write extensively to justify the decision, but a reader should be able to follow the justification.

Keep the outcomes separate. In a review of a drug to treat heart failure, the effect of the drug may have high certainty based on large, well-conducted trials, while the effect of the drug may have low certainty based on small, open-label trials. Both ratings should be shown in the review.

Always have two independent reviewers evaluate certainty. If there are differences, they discuss their different opinions and, if needed, bring in a third reviewer to facilitate.

A 15% readmission rate would mean 27 fewer readmissions for every 1,000 patients, a range of 60 to 18, giving a moderate level of certainty. So, the drug in question may bring a large reduction in readmission, or increase readmission slightly. Depending on which is the case, a clinician would act differently. So, you rate down one level for serious imprecision, and end with moderate certainty.

Your footnote could say “Downgraded one level for serious imprecision. The 95% CI includes a large reduction in readmissions (60 fewer) as well as an increase (18 more).”

Footnotes are important. Peer reviewers go over them and analyze them line by line to validate your ratings.

Common Mistakes When Applying GRADE

– Reflexively rating down. A concern in a domain doesn’t mean you automatically rate it down. Reflexively rating down is a common error. In fact, there are times when you shouldn’t rate it down.

– Reading a nonsignificant result as “no effect.” A wide CI crossing the line of no effect indicates that we don’t know if the effect exists. This doesn’t mean that the treatment hasn’t worked.

– Rating every outcome the same as the study. We take a domain-by-domain approach to rating.

– Omitting an explanation for a rating. These are difficult to defend, and we are rarely able to validate the reasoning.

Frequently Asked Questions About the GRADE Approach

Q1. What does GRADE mean in a systematic review?

It refers to the process for rating the certainty of evidence in a systematic review. This process is typically used to derive recommendations.

Q2. Is GRADE the same as the quality of evidence?

Pretty much. Older GRADE documents said “quality of evidence.” Later guidance, including the Cochrane Handbook, said to use the term “certainty of evidence.”

Q3. Can observational studies receive a high GRADE rating?

It is infrequent to see a GRADE evaluation of observational studies, since the starting place for observational studies is typically low. However, depending on the outcome, observational studies can move up. For instance, if a study shows a large effect, or if there is a clear dose-response relation, or if a confounding factor would have reduced the effect being measured, observational studies would move up from low to moderate or high.

Q4. What is the difference between GRADE and risk of bias assessment?

Risk of bias tools, such as RoB 2 or ROBINS-I, assess bias for individual studies. GRADE utilizes the judgments of bias, as well as other domains, to reach an overall rating for the body of evidence.

 Q5. Is GRADE mandatory for every systematic review?

Not all systematic reviews require the use of GRADE. It is more common for intervention reviews; for instance, all reviews conducted by Cochrane are required to use GRADE. It is uncommon for qualitative reviews to require the use of GRADE, but rather a similar tool called GRADE-CERQual.

Q6. Which tool is used to create a GRADE Summary of Findings table?

 GRADEpro GDT is the only correct answer. You can make domain judgments, include footnotes, import RevMan data, and export Summary of Findings tables or complete Evidence Profiles ready for publication.

Putting GRADE Into Practice

For each outcome, GRADE comes down to one question: how much should a reader trust this number? Begin with the study design, go through the five domains one at a time, and write a reason for every call you make. The first review takes a while. The second goes a lot faster.

Disclaimer:

Articles published by American Academy of Research & Academics are prepared by our team using information from direct experience, publicly available resources, and educational references. AI tools may be used to assist with drafting, proofreading, and formatting; however, all content undergoes review and approval before publication.
The information provided is intended for educational purposes only. Requirements, policies, and processes may change over time. Readers should consult official sources for the most current information.