An introduction to EBM (session 5)

skills
beginner
evidence-based medicine
critical appraisal
clinical trials
Published

May 27, 2026

Session outline

  • this session is all about what happens beyond the RCT?
  • we’ll recap some of the issues we discussed last time as motivation
  • then we’ll introduce and discuss a pair of ways of summarising multiple RCTs
  • we’ll relate these two methods back to our earlier discussions about bias
  • and we’ll look at a totally different way of solving some of the problems of RCTs

Recap

  • detecting treatment effects is hard
    • effects are subtle
    • trials have practical problems (like recruitment)
  • studies are often underpowered
    • so they fail to find a real treatment effect
  • that causes harm
    • some valuable treatments get ignored
    • some harmful treatments remain in use

Cochrane story

Data from randomised trials before 1980 of corticosteroid therapy in premature labour and its effect on neonatal death.
Identifier Deaths (Treatment) n (Treatment) Deaths (Control) n (Control)
Auckland 36 532 60 538
Block 1 69 5 61
Doran 4 81 11 63
Gamsu 14 131 20 137
Morrison 3 67 7 59
Papageorgiou 1 71 7 75
Tauesch 8 56 10 71

Forest plots

  • useful intro
  • one row per study
  • box size corresponds with study size (weight)
  • box location corresponds with odds ratio (OR)
  • whiskers (horizonal lines) showing the confidence interval of that OR
  • pooled effect show by the diamond
    • size = total weight
    • horizontal limits = confidence interval
  1. how would you interpret the line representing a study crossing 1?
  2. how can individual lines cross 1, but the pooled diamond not cross 1?

E2: from 1982 to 2020

TipTask
  1. Find the current version of this Cochrane review
  2. What are the main current recommendations?

Cochrane reviews are systematic reviews

A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question (Cochrane library)

Explicit strategies

  • like EBM itself
  • e.g. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)

Meta-analysis

What’s the difference?

  • systematic reviews attempt to include all the relevant studies
  • meta-analyses include some relevant studies
  • that might give rise to biases…

E3: bias and MA/SR

TipTask
  1. Some authors argue that systematic reviews should generally be preferred to meta-analyses on the ground of bias. Can you think of biases that might affect MA, but not SR?
  2. Can you think of different reasons why you might prefer a MA, rather than a SR?

Bad pharma

Ben Goldacre’s Bad Pharma
  • undue influence of industry on drug research
  • plus
  • excessive emphasis on the methods of trials, rather than their conduct
  • plus
  • pressure to perform larger, cleaner, RCTs
  • produces
  • a “murderous disaster” (Goldacre 2012)

Two kinds of validity

Internal and external validity

  • MA/SR can improve internal validity
    • e.g. aggregating improves power, so more likely to detect true effects
  • but they might do this at the expense of reducing external validity
    • e.g. our trial population might be more and more unlike our treatment population
  • and reducing external validity might mean that an intervention might fail in ways that are hard to understand (Cartwright 2012)

Internal vs external validity

Trial and treatment populations

  • wildly dis-similar (Fortin 2006)
  • co-morbidity example
    • a database of 980 general-practice patients were assessed against inclusion critera from 5 blood pressure RCTs
    • of eligible patients “89% to 100% had multiple chronic conditions”
      • mean numbers ranged from 5.5 ± 3.3 to 11.7 ± 5.3

Pragmatic trials

(Patsopoulos 2011)

Final thought

  • so there isn’t going to be a simple answer about where on the precise / pragmatic spectrum we should end up
    • it’s an optimisation problem
  • as EBM started by stressing the importance of explicit/judicious/conscientious use of evidence, we should probably adopt similar standards for our decision making in general
    • explicit about the kind of q we’re answering
    • judicious about the way we use evidence to answer it
    • conscientious in updating, revising, and checking our answers, and our methods

References

Cartwright, Nancy. 2012. “Presidential Address: Will This Policy Work for You? Predicting Effectiveness Better: How Philosophy Helps.” Philosophy of Science 79 (5): 973–89. https://doi.org/10.1086/668041.
Fortin, M. 2006. “Randomized Controlled Trials: Do They Have External Validity for Patients with Multiple Comorbidities?” The Annals of Family Medicine 4 (2): 104–8. https://doi.org/10.1370/afm.516.
Goldacre, B. 2012. Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients. Fourth Estate. https://books.google.co.uk/books?id=JnwXTwEACAAJ.
Patsopoulos, Nikolaos A. 2011. “A Pragmatic View on Pragmatic Trials.” Dialogues in Clinical Neuroscience 13 (2): 217–24. https://doi.org/10.31887/dcns.2011.13.2/npatsopoulos.