Cochrane Risk of Bias Tool: RoB 2 vs ROBINS-I, Domains, Signaling Questions and How to Assess Risk of Bias

Cochrane Risk of Bias Assessment: Why It Matters

A systematic review can be perfect in all domains- it has a well-built search strategy, a sound statistical analysis, and hundreds of included studies- and its conclusions can still be unreliable. Now the question that comes in mind is, how is that possible? The answer is simple. If the studies feeding into the review are flawed, the pooled result inherits those flaws. This is where Cochrane risk of bias assessment comes in. It gives reviewers a structured way to ask whether a study’s result may have been pushed away from the truth by the way the study was designed, run, or reported.

Cochrane offers two main tools. RoB 2 is used for randomized controlled trials, and ROBINS-I is used for non-randomized studies of interventions. This guide explains both tools, their domains, their signaling questions, and how to actually apply them in a systematic review or meta-analysis.

What is the Cochrane Risk of Bias Tool?

It’s a framework for working out whether a study’s result can be trusted, and it deliberately avoids giving studies a score. There’s no “7 out of 10.” Instead, you go through the study section by section, looking at the places where bias usually enters, and you form a judgment about each one.

The word bias is doing specific work here. It means systematic error, not random variation. If a trial’s design pushed the result in one direction, running the study again with twice as many participants will produce the same skewed answer, just with tighter confidence intervals around it.

For anyone doing a meta-analysis, that’s the whole problem in a sentence. Pooling assumes the studies you’re pooling are telling you something real. Suppose six of your ten included trials measured a subjective outcome using unblinded assessors. Your forest plot will still produce a neat summary estimate, and that estimate will still look precise, but it is resting on shaky evidence. Risk of bias assessment is how you flag that for your reader instead of hoping nobody asks.

The domain-by-domain approach exists for a practical reason: bias rarely affects a study evenly. A trial can be watertight in four places and fall apart in the fifth, and a single overall label would hide that completely.

Risk of Bias is Not the Same as Study Quality

Plenty of papers use these terms interchangeably, and it causes real confusion for reviewers who are new to this.

Quality is a loose term. Depending on who you ask, it might cover sample size, funding, how thoroughly the methods were reported, or just whether the paper reads well. Risk of bias asks something much narrower: is there a mechanism by which this particular result could have ended up systematically wrong?

You can see why the two come apart. A multicenter trial with 4,000 participants, a preregistered protocol, and immaculate reporting can still be at high risk of bias if the people rating the outcome knew who received the drug. Meanwhile, a modest 80-patient trial written up in plain language might have nothing wrong with it at all. What you’re assessing is how the study was built, not how impressive it looks.

RoB 2 vs ROBINS-I: Which Risk of Bias Tool Should You Use?

Choosing the right tool is the first decision you make, and getting it wrong invalidates everything that follows. The short version is that RoB 2 is for randomized trials and ROBINS-I is for non-randomized studies of interventions.

Question	Tool to use
Randomized controlled trial?	RoB 2
Non-randomized intervention study?	ROBINS-I
Observational study evaluating the effects of an intervention?	ROBINS-I
RCT with individually randomized participants?	RoB 2
Cluster-randomized trial?	The cluster-randomized variant of RoB 2
Crossover trial?	The crossover variant of RoB 2

The table covers most cases, but study design alone is not always enough to decide. The question you are asking also matters. ROBINS-I was built for studies that estimate the effect of an intervention or exposure, and it compares each study against an ideal randomized trial that could have answered the same question. If a study is not trying to estimate an intervention effect at all, ROBINS-I may not be the right fit either.

Don’t Choose the Tool Based Only on the Study’s Label

“Observational study” is a very broad description. It covers cohort studies comparing two treatments, cross-sectional surveys of symptom prevalence, and case series with no comparison group at all.

Before reaching for ROBINS-I, check what the study is actually estimating. Is there a defined intervention and a defined comparator? Are the authors trying to say that one option produced a different outcome than the other? If yes, ROBINS-I fits. If the study simply describes how common something is, or reports diagnostic accuracy, a different tool is more appropriate.

The same caution applies in the other direction. A paper may call itself randomized without describing any randomization procedure. That does not automatically rule out RoB 2, but it will affect your judgment in the first domain.

RoB 2: The 5 Domains of Risk of Bias

RoB 2 organizes the assessment into five domains. Each domain covers a different stage of the trial, and each one gets its own judgment.

Domain 1:  Bias Arising From the Randomization Process

This domain asks whether the two groups were comparable at the start. Look for how the random sequence was generated, whether allocation concealment was in place so that nobody could predict or influence the next assignment, and whether baseline differences between groups suggest a problem with randomization.

Terms like “randomly allocated” with no further detail are common. Note them, and check the protocol or trial registration before deciding.

Domain 2: Bias Due to Deviations From Intended Interventions

Here you ask whether the trial delivered what it intended. Consider blinding of participants and staff, adherence to the assigned intervention, and whether co-interventions differed between groups.

You also need to look at the analysis. An intention-to-treat analysis keeps participants in their assigned groups. A per-protocol analysis drops those who did not comply, which can reintroduce bias.

Domain 3: Bias Due to Missing Outcome Data

Attrition matters when the people who dropped out differ from those who stayed. Check how much outcome data is missing, whether loss to follow-up was balanced across groups, and whether the reasons for missing data are related to the outcome itself.

Incomplete outcome data is not automatically a problem. Small, balanced, and clearly explained losses are usually fine.

Domain 4: Bias in Measurement of the Outcome

This domain covers how the outcome was assessed. Was the method appropriate and applied the same way in both groups? Were outcome assessors blinded?

Blinding matters more for subjective outcomes such as pain scores or symptom ratings. For objective outcomes such as all-cause mortality, an unblinded assessor is far less likely to influence the result.

Domain 5:  Bias in Selection of the Reported Result

The final domain asks whether the reported result was cherry-picked. Trials often measure multiple outcomes, at multiple time points, using multiple analysis methods. Selective reporting happens when authors present only the version that looked best.

Compare the published paper with the protocol or trial registration. Outcome switching, unreported time points, and analyses that appear without being pre-specified are all warning signs.

ROBINS-I: The 7 Domains of Risk of Bias

ROBINS-I uses seven domains, grouped by when the bias can occur: before the intervention, at the point of intervention, and after it.

Pre-intervention

1. Bias due to confounding. Did other factors influence both the choice of intervention and the outcome, and were they properly handled?
2. Bias in selection of participants into the study. Were participants selected in a way that is related to both the intervention and the outcome, for example by excluding people who had an early event?

At intervention

3. Bias in classification of interventions. Were intervention groups clearly defined, and was group status recorded using information collected at the time rather than assigned afterward?

Post-intervention

4. Bias due to deviations from intended interventions. Did anything happen after the start that pushed groups apart in ways unrelated to the intervention being studied?
5. Bias due to missing data. Were outcome data, intervention status, or confounder data missing for some participants?
6. Bias in measurement of outcomes. Could the way outcomes were assessed differ between groups, especially if assessors knew intervention status?
7. Bias in selection of the reported result. Was the reported result selected from multiple outcome measurements, analyses, or subgroups?

Why ROBINS-I Is More Complex Than RoB 2

In a randomized trial, randomization takes care of confounding for you. In a non-randomized study, nothing does. People end up in one group or another for reasons that often predict the outcome, such as being sicker, older, or seen at a different hospital.

That is why ROBINS-I asks you to list the important confounders before you assess anything, and then to judge whether the study measured and adjusted for them properly. Differences in baseline characteristics that would be a red flag in an RCT are simply expected here, so the question becomes whether they were handled well.

ROBINS-I also asks you to imagine a target trial: the randomized trial that could, in principle, have answered the same question. Each domain then asks how far the real study drifts from that ideal. This causal framework is what makes ROBINS-I harder, and also what makes it useful.

What are RoB 2 and ROBINS-I Signaling Questions?

Signaling questions are the structured prompts inside each domain. They ask about concrete features of the study, and you answer each one with Yes, Probably yes, Probably no, No, or No information.

For example, one RoB 2 signaling question asks whether the allocation sequence was concealed until participants were enrolled and assigned. A ROBINS-I signaling question asks whether the analysis controlled for all important confounding domains.

The most common misunderstanding is to treat these as a checklist and count the answers. They do not work that way. Signaling questions gather evidence; the reviewer still makes the domain judgment. Two “Probably no” answers on minor points may not matter, while a single “No” on allocation concealment often will. The algorithm suggests a judgment, but you can override it if you explain why.

How to Answer Signaling Questions Without Guessing

Use a simple rule: do not infer, look for evidence, record where you found it, then make the judgment.

Check the methods section, the results, supplementary files, the published protocol, and the trial registration entry. Reviewers often find the answer in a registry record rather than the paper itself.

When the study genuinely does not report enough information, answer “No information” instead of guessing. That answer is meaningful, and it usually leads to “some concerns” rather than “high risk.” Missing reporting is not the same as proven bias, though for some domains, such as allocation concealment, silence does raise real doubt. Write down what was missing so your reasoning is visible to readers.

How to Assess Risk of Bias Using RoB 2: Step-by-Step

Step 1: Identify the study design

Confirm the trial is randomized and check whether it is a parallel-group, cluster, or crossover design, since each has its own RoB 2 variant.

Step 2: Define the outcome and the result being assessed

 Risk of bias is judged for one specific result, not for the paper as a whole. Decide which outcome you are assessing before you start.

Step 3: Answer the signaling questions

 Gather information from the methods, results, supplementary material, trial registration, and protocol. Record the supporting text for each answer.

Step 4: Make domain-level judgments

For each of the five domains, decide between low risk, some concerns, and high risk, based on the signaling question answers and your reading of the study.

Step 5: Determine the overall risk of bias

The overall judgment comes from the pattern of domain judgments, not from an average. In general, a result is low risk only if every domain is low risk, and it is high risk if any domain is high risk.

Why You Should Assess Risk of Bias for Specific Results, Not Just the Whole Paper

One trial can be at low risk of bias for mortality and high risk for quality of life, because objective and subjective outcomes are vulnerable to different problems. A single “study-level” rating hides that difference. Assess each result you plan to include in a meta-analysis separately.

How to Assess Risk of Bias Using ROBINS-I

The workflow is similar in shape but starts earlier.

1. Define the intervention and the comparator. Be specific about what is being compared with what.
2. Specify the target trial. Describe the randomized trial that could have answered this question. This becomes your reference point.
3. List the important confounders in advance. Decide, based on subject knowledge, which variables must be accounted for before you look at what the study actually did.
4. Answer the signaling questions for each of the seven domains.
5. Assign a judgment to each domain. ROBINS-I uses low, moderate, serious, and critical risk of bias, plus “no information.”
6. Determine the overall judgment. It is driven by the worst domain, so one critical domain makes the whole result critical.

The target trial idea is worth pausing on. If you cannot describe a trial that would answer your question, the study may not be suitable for ROBINS-I at all.

How to Interpret “Low Risk,” “Some Concerns,” and “High Risk”

RoB 2 uses three levels.

Low risk of bias means there is little concern that this domain could meaningfully change the result.

Some concerns means something is unclear or slightly weak. There is not enough reassurance to call it low risk, but not enough of a problem to call it high risk. Poorly reported studies often land here.

High risk of bias means there is a serious concern that the result may be substantially affected.

The overall judgment is not calculated by counting how many domains were low risk. Four low-risk domains do not cancel out one high-risk domain. The weakest link decides the result.

How to Create a Risk of Bias Traffic-Light Plot Using robvis

A traffic-light plot shows every included study as a row and every domain as a column, with a colored circle for each judgment. It lets readers see at a glance which domains are weak across your evidence base.

robvis is a free R package, also available as a web app, that produces these figures from a simple spreadsheet of your judgments. It also produces a weighted bar plot, which shows the percentage of studies in each category per domain. The traffic-light plot shows study-by-study detail; the bar plot summarizes the whole set.

What a Traffic-Light Plot Does and Doesn’t Tell You

The plot displays your conclusions. It does not display your reasoning. A red circle tells a reader that you judged a domain to be at high risk, but not why.

Always pair the figure with the written justification for each judgment, usually as a supplementary table. Without it, readers cannot check your work, and the figure becomes decoration.

How to Report Risk of Bias in a Systematic Review

Methods: State which tool you used and which version, who carried out the assessments, whether they worked independently and in duplicate, and how disagreements were resolved. Mention any piloting or calibration you did.

Results: Report domain-level judgments and overall judgments for each included result, along with the supporting explanations. Present them in a table or figure, such as a robvis traffic-light plot.

Discussion: Explain what the assessments mean for your conclusions. If your effect estimate rests mostly on high-risk studies, say so, and describe any sensitivity analysis you ran that excluded them.

Common Mistakes Researchers Make When Using Cochrane RoB Tools

1. Using RoB 2 for non-randomized studies. The domains assume randomization happened. Use ROBINS-I instead.
2. Treating risk of bias as study quality. They measure different things.
3. Answering signaling questions from assumptions. If the paper does not say, do not decide that it probably happened.
4. Recording judgments with no supporting evidence. Every judgment needs a note explaining what it was based on.
5. Assessing the study instead of the result. Risk of bias is outcome-specific.
6. Treating missing information as proof of high risk. Unclear reporting usually means “some concerns,” not automatic condemnation.
7. Publishing a traffic-light plot without explanations. The figure needs the reasoning behind it.
8. Mixing tool versions. Do not blend the original Cochrane tool with RoB 2, or ROBINS-I with RoB 2 domains, within one review.

RoB 2 vs ROBINS-I: Quick Comparison

Feature	RoB 2	ROBINS-I
Main use	Randomized trials	Non-randomized studies of interventions
Number of domains	5	7
Confounding	Not a separate domain	Major domain
Randomization assessed	Yes	No
Signaling questions	Yes	Yes
Judgment levels	Low, some concerns, high	Low, moderate, serious, critical
Overall judgment	Yes	Yes

Key Takeaways

The choice of tool comes down to design: RoB 2 for randomized controlled trials, ROBINS-I for non-randomized studies of interventions. RoB 2 works through five domains and ROBINS-I through seven, with the extra weight in ROBINS-I falling on confounding, which randomization already handles in a trial. In both tools, signaling questions guide your thinking rather than score the study, and every judgment applies to a specific result rather than the paper as a whole. Report those judgments, and the reasoning behind them, clearly enough that a reader can follow how you reached them. 

Risk of bias assessment is one step in a much longer process, and most researchers learn it under deadline pressure. The American Academy of Research and Academics offers structured guidance on systematic reviews and meta-analysis, from protocol development and PRISMA reporting through RevMan and risk of bias assessment. If you are working on your first review, or want a second opinion on your methodology, get in touch with our team. 

Disclaimer:

Articles published by American Academy of Research & Academics are prepared by our team using information from direct experience, publicly available resources, and educational references. AI tools may be used to assist with drafting, proofreading, and formatting; however, all content undergoes review and approval before publication.
The information provided is intended for educational purposes only. Requirements, policies, and processes may change over time. Readers should consult official sources for the most current information.