Publication Bias in Meta-Analysis: How to Assess It Using Funnel Plots, Egger’s Test & More

Introduction

You pooled the studies, ran the meta-analysis, and the result came out significant and your meta-analysis paper is ready. Good news, right? Maybe. But ask yourself one question first: what about the studies that never got published? If trials with weak or negative results ended up in someone’s drawer instead of a journal, your analysis never saw them. And your pooled result may look better than the truth.

That’s publication bias. It’s a missing-evidence problem, and it can quietly skew a meta-analysis. Researchers usually look for it with funnel plots, Egger’s regression test, Begg’s test, the trim-and-fill method, and sensitivity analyses. None of these tools can prove publication bias by itself, though. You get the most out of them when you read them together.

What Is Publication Bias?

Publication bias happens when a study’s chance of being published depends on its results. Positive, significant findings tend to get published. Null or disappointing ones often don’t.

Picture two trials of the same drug. The first finds a benefit and lands in a journal. The second finds nothing and is never submitted. A meta-analysis that only picks up the first trial will show a bigger effect than the drug really has.

The Cochrane Handbook uses a wider term for this: risk of bias due to missing evidence. That covers whole studies that were never published, and also results that were measured but left out of a paper. Either way, Cochrane notes that selective non-reporting can make a review overestimate or underestimate an intervention’s effect.

Here’s a quick example. Twelve studies test the same treatment. Eight show a benefit and get published. Four show little or no benefit and don’t. If you can only find the eight, your meta-analysis is working with two-thirds of the evidence, and it’s the most favorable two-thirds.

Key takeaway: Publication bias means some research findings are missing from the literature, and the reason they’re missing is linked to what they found.

Why Does Publication Bias Matter in Meta-Analysis?

A meta-analysis is only as good as the studies you feed into it. If the studies you found aren’t a fair sample of everything that was done, the problem carries straight through to your result.

The most common consequence is an inflated effect size. When studies with bigger benefits are easier to find, the pooled estimate drifts upward, and a treatment can look more useful on paper than it is in real patients.

It also creates false confidence. Pooling data narrows your confidence interval, which feels reassuring. But a narrow interval around the wrong number doesn’t help anyone.

Then there’s the real-world impact. Guidelines, funding decisions, and new research questions often lean on systematic reviews. If missing studies bend the results, those decisions bend with them.

One more thing beginners often miss: publication bias doesn’t only push results up. Usually it does, but missing evidence can pull an estimate down, too.

What Is a Funnel Plot in Meta-Analysis?

A funnel plot is a scatter plot of your included studies, with one dot per study.

• X-axis: effect size (a log odds ratio or mean difference, for example)
• Y-axis: study precision, usually shown as standard error

Big studies are more precise, so they have small standard errors and sit near the top, close to the pooled effect. Small studies are less precise. Their results bounce around more, so they spread out toward the bottom.

If nothing unusual is going on, the dots form a rough upside-down funnel that looks about the same on both sides. If one side looks thin or empty, say the lower left corner, that’s called funnel plot asymmetry.

Asymmetry tells you small-study effects might be present. It doesn’t prove publication bias.

How to Interpret a Funnel Plot for Publication Bias

A lot of people glance at a funnel plot, see it’s lopsided, and write “publication bias” in their results. Try working through three questions instead.

Step 1: Is it roughly symmetrical?

Check whether the smaller studies fall fairly evenly on both sides of the pooled effect. If they do, you probably don’t have an obvious small-study effect.

Step 2: Is anything missing?

Look for a gap where you’d expect studies to be. A classic pattern is an empty patch among the small studies on the “no benefit” side.

Step 3: What else could explain it?

Don’t skip this one. A lopsided funnel can come from:

• Publication or non-reporting bias
• Small-study effects
• Real heterogeneity between studies
• Differences in methods or patient groups
• Weaker methods in the smaller studies
• Chance
• Quirks of the effect measure you used

Cochrane is clear on this point. It treats the funnel plot as a way to explore small-study effects, and it warns against reading asymmetry as proof of publication bias.

How to Assess Publication Bias in a Systematic Review

Publication bias can be assessed using a combination of comprehensive searching, funnel plot inspection, statistical tests for funnel plot asymmetry, and sensitivity analyses. No single method can definitively establish that publication bias is present.

Honestly, your best defense comes before any statistics. A search that covers several databases, trial registries, conference abstracts, and grey literature gives you a much better chance of catching unpublished work. Once you have your studies, these are the main tools for checking them.

1. Funnel Plot

This is the visual check, and it’s usually where you start. It’s good for spotting possible asymmetry. The downside is that it’s subjective. Two reviewers can look at the same plot and disagree, and even a clearly lopsided plot doesn’t point to publication bias specifically.

2. Egger Regression Test

Egger’s test puts a number on what you’re seeing in the funnel plot. It checks whether a study’s effect size is linked to its precision. If the small, less precise studies keep showing bigger effects, the test is likely to come back significant.

A significant result means the funnel is asymmetrical. It doesn’t tell you publication bias is the reason.

Keep in mind that the test isn’t very powerful. Cochrane advises using it only when you have at least 10 studies. It can also mislead in certain cases, like when you’re working with odds ratios or when heterogeneity is high. For binary outcomes, some reviewers use modified versions such as Peters’ or Harbord’s test.

3. Begg Test

Begg’s test (the Begg and Mazumdar rank correlation test) asks the same basic question as Egger’s, but in a different way. Rather than using regression, it checks whether the ranks of the effect estimates line up with the ranks of their variances.

Its main weakness is low power. It can easily miss real asymmetry, especially when you don’t have many studies. That’s one reason you’ll see Egger’s test reported more often.

Egger Test vs Begg Test: What’s the Difference?

The two tests share a goal but take different routes to it.

Feature	Egger Test	Begg Test
Approach	Regression-based	Rank correlation-based
Purpose	Detect funnel plot asymmetry	Detect funnel plot asymmetry
Output	Test statistic and P value	Test statistic and P value
What a significant result means	Evidence of asymmetry	Evidence of asymmetry
Main limitation	Can be affected by heterogeneity and the effect measure used	Generally lower statistical power
Should it be used alone?	No	No

Look at that last row again. Neither test can tell you publication bias exists. Both can only tell you the funnel is lopsided. Figuring out why is still your job.

What Is the Trim-and-Fill Method?

Duval and Tweedie came up with trim-and-fill to answer a “what if” question: how would the pooled effect change if the missing studies were added back?

The method works in a few stages. First, it finds the asymmetry in your funnel plot. Then it trims off the extra small studies on the crowded side and works out where the center of the funnel should really be. Next, it fills in mirror-image studies on the empty side. Finally, it recalculates the pooled effect with those added studies.

The catch is that the filled-in studies aren’t real. They’re statistical guesses. Trim-and-fill also assumes publication bias caused the asymmetry in the first place. If heterogeneity or poor study quality is the real cause, the adjusted estimate can steer you wrong. Treat it as a sensitivity check, not a fix.

Publication Bias vs Small-Study Effects

People use these terms as if they mean the same thing. They don’t.

Publication bias is about missing evidence: whether a study is available depends on what it found.

A small-study effect is simply the pattern where smaller studies show different results from larger ones, often bigger effects.

Publication bias can cause small-study effects. So can other things, such as lower-quality small trials, different patient groups, or small studies that deliver the treatment more intensively. That’s why it’s more accurate to say funnel plots and Egger’s test detect small-study effects rather than publication bias.

How Many Studies Are Needed to Test for Publication Bias?

As a commonly used rule of thumb, tests for funnel plot asymmetry should only be considered when a meta-analysis includes at least 10 studies. Below that, the tests have so little power that they can easily miss real asymmetry.

Having fewer than 10 studies doesn’t mean you’re free of publication bias, though. It just means the statistical tests won’t tell you much. You can still describe how thorough your search was, discuss the risk of missing studies, and list it as a limitation.

Can You Assess Publication Bias in RevMan?

Yes and no. RevMan will draw a funnel plot for you, which is useful for the visual check. But drawing the plot isn’t the same as assessing bias, and RevMan won’t tell you whether bias is there.

According to the Cochrane Handbook, the recommended tests for funnel plot asymmetry aren’t built into RevMan. To run Egger’s test or something similar, most researchers switch to R (with packages like meta or metafor) or Stata. If you’re new to the software, our Stata tutorial for medical research covers the basics.

Publication Bias in Meta-Analysis Example

Here’s a made-up case to tie it together.

You run a meta-analysis of 12 studies on a new intervention. Seven are small and show a large benefit. Five are larger and show a more modest one. Your funnel plot is lopsided, and Egger’s test is significant.

So is this publication bias?

Not necessarily. What you have is evidence of small-study effects. Before you blame missing studies, think about:

• Publication or non-reporting bias
• The quality of the seven small studies
• Clinical differences, like patient groups or doses
• Differences in study methods
• Chance
• Other causes of small-study effects

A sensible next move is a sensitivity analysis. Rerun the numbers with only the larger studies, or only the studies at low risk of bias, and see if the result holds up. With enough studies, meta-regression can also help you check whether study size or quality explains the gap.

How Should You Report Publication Bias in a Systematic Review?

Good reporting lets readers decide for themselves how much to trust your result. Here’s a template you can adapt:

“A funnel plot was visually assessed for asymmetry. Because 12 studies were included, Egger’s regression test was also performed. The results suggested [presence/absence] of small-study effects (P = [exact value]). However, funnel plot asymmetry may have several explanations, including heterogeneity and methodological differences between studies.”

Make sure your write-up covers:

• How many studies were included in the assessment
• The funnel plot
• The effect estimate you used
• The measure of precision you used
• Which statistical test you ran, if any
• The exact P value
• Any sensitivity analyses
• The limits of your assessment

PRISMA 2020 backs this up. It asks authors to show the funnel plot, state the effect estimate and precision measure, and give the exact P value whenever they use a test for funnel plot asymmetry.

Common Mistakes When Assessing Publication Bias

Mistake 1: Calling any lopsided funnel “publication bias.”

Asymmetry has plenty of possible causes. Missing studies are only one of them.

Mistake 2: Running Egger’s test on five studies.

 With so few studies, the test has almost no power. The result won’t mean much either way.

Mistake 3: Reading P > 0.05 as “no publication bias.”

 A non-significant test only means the test didn’t find asymmetry. It may just have been too weak to see it.

Mistake 4: Reporting the trim-and-fill estimate as the corrected answer.

 It’s built on assumptions and imputed studies. Use it to test how sensitive your result is.

Mistake 5: Forgetting about heterogeneity.

 Real differences between studies can create small-study effects without a single study going missing.

Publication Bias Assessment Checklist for Researchers

Run through this list when you plan or review your meta-analysis:

Key Takeaways

• Publication bias means the evidence you can find differs in a systematic way from the evidence that’s missing.
• A lopsided funnel plot doesn’t prove publication bias on its own.
• Egger’s and Begg’s tests check for funnel plot asymmetry and small-study effects, not publication bias directly.
• These tests generally need at least 10 studies to be useful.
• Base your judgment on several pieces of evidence, never on one test.

Frequently Asked Questions

Q1. What is publication bias in simple terms?

It’s when studies with positive or significant results get published more often than studies with negative or null results. As a result, the research you can find may not match everything that was actually done.

Q2. Does an asymmetrical funnel plot always mean publication bias?

No. Heterogeneity, study quality, chance, and the effect measure you chose can all cause asymmetry. Think of it as a sign of small-study effects that you need to look into further.

Q3. What does a significant Egger’s test mean?

It means there’s statistical evidence that your funnel plot is lopsided. It says nothing about why, so you still have to weigh the other possible causes before pointing to publication bias.

Q4. Can I run Egger’s test with fewer than 10 studies?

You can, but you probably shouldn’t. With fewer than 10 studies, the test has low power and its result isn’t reliable. The Cochrane Handbook advises against it.

Q5. Is the trim-and-fill method a reliable way to correct publication bias?

Not by itself. The studies it adds are estimates, not real data, and the method assumes publication bias caused the asymmetry. It works best as a sensitivity analysis to see how much your result could shift.

Q6. How can I reduce publication bias in my systematic review?

Search widely. Go beyond the main databases to trial registries, conference abstracts, dissertations, and other grey literature. It’s also worth emailing study authors to ask about unpublished data.

Final Thoughts

Publication bias is tricky because you’re trying to judge studies you can’t see. Funnel plots, Egger’s test, Begg’s test, and trim-and-fill all help, but each has blind spots. What works best is a thorough search, careful reading of your results, a few sensitivity analyses, and honest reporting of what you found and what you couldn’t rule out. Treat each method as one clue, not the final verdict, and your meta-analysis will be easier to trust.

Disclaimer:

Articles published by American Academy of Research & Academics are prepared by our team using information from direct experience, publicly available resources, and educational references. AI tools may be used to assist with drafting, proofreading, and formatting; however, all content undergoes review and approval before publication.
The information provided is intended for educational purposes only. Requirements, policies, and processes may change over time. Readers should consult official sources for the most current information.