Deconstructing Rapid Reviews
•
An Exploration of Knowledge, Traits and Attitudes
Presented by: Shannon Kelly, CADTH Symposium 2015 – Saskatoon, SK
Faculty of Medicine
School of Epidemiology, Public Health and Preventive Medicine
uOttawa.ca
2
Objectives
• To present results from 3 projects on
rapid reviews conducted as part of
MSc. Epidemiology at uOttawa
I.
Defining rapid reviews – a consensus approach
II. Attitudes and perceptions towards rapid reviews
III. Defining rapid reviews – a pragmatic approach
uOttawa.ca
Why do we care about
RAPID REVIEWS?
uOttawa.ca
What is a rapid review?
“…a streamlined approach to synthesizing evidence in a
timely manner - typically for the purpose of informing
emergent decisions faced by decision makers in health
care settings.”
~ Khangura, 2012
uOttawa.ca
Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the
evolution of a rapid review approach. Systematic Reviews. 2012;1:10.
Rapid reviews influence decisionmaking in Canada
• Knowledge users in Canada (everywhere) need support
to make timely, evidence-informed health care decisions
• Systematic reviews and HTA often take too long!
• Delay important policy decisions
• Use less-than-best evidence
• Rapid reviews help fill this void
• Balance rigour, relevance, timelines
…We still need to know so much more…
uOttawa.ca
What do we know?
Ultra-rapid reviews, first test results
uOttawa.ca
What don’t we know?
“In order to consider offering rapid reviews, one first has to
be able to define what they are. Previous reviews on the topic
of rapid reviews describe them as ‘ill-defined,’ ‘not welldefined,’ lacking a single definition, or ‘varying widely in
terms of the language used to describe them.’” (AHRQ,
Hartling, 2015)
“While the concept
of rapid evidence
synthesis, or rapid
review, is not
novel, it remains
a poorly
understood and
yet ill-defined set
of disparate
methodologies
supported by a
paucity of
published,
available scientific
literature”
(Khangura, 2014)
uOttawa.ca
“While rapid review approaches are being
used by HTA producers across the world in
a variety of ways an agreed-upon
definition describing its constitution and
methods is lacking” (Khangura, 2014)
“..evidence is only
slowly emerging as to
which steps in the
systematic review process
may be altered by
increased speed”
(Schünemann, 2015)
“There is little empirical evidence comparing the
continuum of products among rapid reviews
and full systematic reviews, or analysing the
diverse methods used in rapid reviews. It is
important not only to establish transparent
methodologies for rapid reviews, but also to
understand the implications of what is lost in terms
of rigour, bias, and results when methods
associated with full systematic review are
streamlined.” (Ganaan, 2010)
“…a
literature
search by the
authors does
not elucidate
any clear or
final
definition of
what a RR is
and how
exactly the
methodology
of a RR
differs from
a full SR.”
(Harker,
2012)
Needed a research program to
improve knowledge on rapid reviews
An improved understanding of rapid reviews will help us
better meet the needs of decision-makers
uOttawa.ca
A consensus approach
DEFINING RAPID REVIEWS
uOttawa.ca
Objective
• The study will employ a modified Delphi method to
survey of experts in the field of evidence synthesis and
rapid reviews to discover what they consider the key
defining characteristics of rapid reviews to be.
Note: For our purposes, defining characteristics are:
‘one of a number of essential features by which a rapid
reviews can be recognized’.
uOttawa.ca
Method: The Delphi Technique
Definition: a procedure that structures a communication
process among a group of experts.(Linstone & Turoff, 1975)
Characteristics of the Delphi technique:





Staged process, assessment of agreement or consensus
Assessment of the group’s view;
An opportunity for informants to revise their views;
Opportunity to react to and assess differing view points;
Anonymity of informants.
Panelists are selected based on their expertise in the
subject matter; not random.
uOttawa.ca
Participants
Experts targeted for participation were:
1. Subject-matter experts or experienced researchers
with knowledge of a variety of evidence synthesis
methods and practical experience with rapid reviews;
2. Authors with publications relevant to rapid reviews;
and,
3. Delegates presenting pertinent work at recent
meetings, workshops, conferences or symposia.
•
uOttawa.ca
International Cochrane Colloquium, Health Technology
Assessment international (HTAi), CADTH Symposium,
Cochrane Canada Symposium).
The Modified Delphi Process
ROUND 1:
WEB-BASED SURVEY ON KEY
RR THEMES
STUDY SCOPING AND
EXPERT PANEL DEFINITION
Pilot
120 POTENTIAL EXPERTS
(FluidSurveys)
66 EXPERTS
ROUND 2:
DRAFT CHARACTERISTIC
STATEMENTS
Review,
summarize,
analyze, draft
statements
(email)
44 EXPERTS
Review agreement,
refine statements
PUBLICATION
PENDING
ROUND 3:
REVISED STATEMENTS,
EXPERT REVISION OF
AGREEMENT
FINAL 11 CHARACTERISTIC
STATEMENTS
Summarize
results
Review
agreement
CONSENSUS ON 10 OF 11
STATEMENTS
Consensus level = 70%, Approved by TOHREB
uOttawa.ca
(email)
26 EXPERTS
Results: Key Characteristics (1)
Rapid reviews:
• Conducted in less time than a systematic review;
• Have a protocol (at least describing objectives, scope,
PICO, approach);
• Tailor the explicit, reproducible methods conventionally
used in a systematic review in some manner to
expedite the process to meet the needs of a
decision-maker;
• Use a spectrum of approaches to complete an
evidence synthesis related to a defined research
question(s) using the most systematic or rigourous
methods the limited time frame will allow.
uOttawa.ca
RR time to completion
Khangura et al. (2012): SRs take 6 months to 2 years, and RRs ≤ 5 weeks.
Delphi panel: Median time for SR is 36 weeks, while median for RR is 9 weeks.
Delphi Expert Opinion
Systematic Review
Rapid Review
0
20
40
60
80
Time (in weeks)
100
120
Previous RR Samples
uOttawa.ca
GANAAN
2010
HARKER
2012
INAHTA scan
2012
POLISENA
2015
AHRQ
2015
TRICCO
(RR SYMP.
CADTH)
Minimum
3 wk
>3 mo
1 wk
1 wk
5 min
<1mo
Maximum
6 mo
< 24 mo
12 mo
12 mo
8 mo
12 mo
Min = minute; d = days, wk = week; mo = month; yr = year
Results: Key Characteristics (2)
• Balance the risk of the yet undetermined impact of
workflow adaptations to the validity of the review and
its results, with the needs of the knowledge user;
• Report methods and findings transparently, with
appropriate detail required to
– answer the research question;
– meet the requirements of the knowledge user and
allow assessment of confidence in the results;
– inform the audience for which the review is intended;
– while meeting delivery time line agreed upon in
advance.
Consensus (72 - 100%)
uOttawa.ca
Still up for debate…
• Tension: risk of bias/critical appraisal;
– individual study or review level
– distinct and opposing camps
• Achieved consensus on no consensus – RRs as a unique
method;
• Time is still relative…;
• Protocol format, publication/registration;
• Peer review;
• Unpublished (grey) literature;
• Not all rapid reviews meet this definition? Overlap with
some key features of SR, HTA.
Statements based only on the experience of the expert
panel…must be balanced with published literature.
uOttawa.ca
Exploring attitudes and perceptions towards
RAPID REVIEWS
uOttawa.ca
Method: Q methodology
RESEARCH QUESTION
A research
technique that
permits the
systematic study of
subjectivity
(opinions, beliefs
and preferences).
COLLECTION OF OPINION STATEMENTS: CONCOURSE
DEFINING STUDY PARTICIPANTS: P SET
DEVELOPMENT OF REPRESENTATIVE STATEMENTS: Q SET
RANK ORDER OF THE STATEMENTS: Q SORT
FACTOR ANALYSIS
INTERPRETATION
*Adapted from Amin, 2000
uOttawa.ca
Conceptual Q sort layout
We replicated this process using online Q methodology tools:
uOttawa.ca
Sample statements
• When time allows, a comprehensive systematic review of all available evidence should always be conducted.
• Further research comparing the methods and results of rapid reviews and systematic reviews is required before I
decide how I feel about rapid reviews.
• Rapid reviews mean different things to different people.
• Rapid reviews should only precede a more comprehensive and rigorous systematic review.
• The opportunity cost of a comprehensive SR or HTA is too high and it is more advantageous to conduct rapid reviews
when timeliness is a factor.
• All evidence synthesis products, including rapid reviews, SRs, or HTAs, can be conducted very well or very poorly.
• Rapid reviews are comparable to SRs except they are done in a more timely fashion.
• Rapid reviews are ‘quick and dirty' systematic reviews.
• Rapid reviews need to be tailored to the specific needs of the knowledge user.
• Rapid reviews meet the needs of knowledge users.
• It is always appropriate to conduct a rapid review.
• It is important to have minimum standards for the reporting of rapid reviews (e.g., a PRISMA-RR).
• Standardization of rapid review methods may conflict with the needs of knowledge users
• A good quality review of evidence is determined by the methods used, not by the speed at which it is completed.
• It is difficult to tell a rapid review from a systematic review unless very specific nomenclature is used in the title or
description of methods.
• A rapid review cannot be a systematic review.
uOttawa.ca
Summary
Geographic Location
Canada
Age:
• 11 participants performed
the online Q sort
• 50 statements representing
a range of viewpoints and
and subjective opinion on RRs
• Following factor analysis
3 factors extracted
n (%)
18 to 35
2 (18.2)
36 to 50
4 (36.4)
50 or above
5 (45.5)
Female
8 (72.7)
Sex:
n (%)
Education:
n (%)
Doctorate
7 (63.6)
Masters
3 (27.3)
Undergraduate
1 (9.1)
Consider Themselves:
n (%)
Researcher/Producer
8 (72.7)
Knowledge User
2 (18.2)
Neither
Have ever been the
author of a rapid review
1 (9.1)
n (%)
Yes
7 (63.6)
No
4 (36.4)
Unsure
Have used a rapid
review to aid in a policy
or decision-making?
0 (0)
n (%)
Yes
7 (63.6)
No
3 (27.3)
Unsure
uOttawa.ca
n (%)
11 (100)
1 (0.9)
Title
uOttawa.ca
Title
uOttawa.ca
Factor 3. “Pragmatism over principle”
uOttawa.ca
Consensus Statements
Agree
6.
Rapid reviews mean different things to different people.
Disagree
7.
34.
42.
43.
uOttawa.ca
Rapid reviews should only precede a more
comprehensive and rigorous systematic review.
It is appropriate to endeavor to define a single, unique
methodology for rapid reviews.
Any review of evidence that takes longer than 3 months
to produce is not a rapid review.
Any review of evidence that takes longer than 1 month to
produce is not a rapid review.
A pragmatic approach
DEFINING RAPID REVIEWS
uOttawa.ca
Summary of results
• Review of RRs published in journals or as reports by
various agencies and organizations worldwide;
– Supplemented with INAHTA scan to query HTA agencies.
• Screened 1,800+ database records and grey literature
results - Single reviewer, checking by a second;
• Included RR samples were examined for completion
time, nomenclature, study characteristics,
approach/tailoring and other relevant features, including
whether the results are contextualized or framed with
policy implications.
uOttawa.ca
In general
• Findings matched previous work on RR methods:
– “ One species, many breeds” Heterogeneity of approaches,
nomenclature – supports the evidence continuum concept;
– May find more than one approach within a single
organization (e.g., CADTH);
– Methods not well-reported; Self-identifying RRs with no
way to assess what was tailored.
• In addition:
– RRs are getting published;
– Need to focus on contextualization of results, policy
implications for end-user;
– Starting to see RR protocols published (n=4) or registered
in PROSPERO (n=3).
uOttawa.ca
Summary
• Addressed RR knowledge gaps with 3
separate but interrelated studies;
• Experts have identified defining
characteristics that may help
differentiate RRs from other evidence
synthesis approaches;
• Agree to disagree in some areas;
• Three distinct viewpoints on RRs were
ascertained during a Q methodology
study of subjective opinion;
• A selection of RR samples show that
there is room for improvement (without
compromising on the needs of endusers).
uOttawa.ca
A special thank you to Dr. Tammy Clifford and Dr. David Moher,
to CADTH for the initial funding for this student project, and
everyone who has participated in surveys, piloted questionnaires,
shared data and supported this research with their own time
Questions? Thoughts?
uOttawa.ca