Evidence and Organizational
Decision Making:
Never the Twain Shall Meet?
Thomas Rundall, PhD
Academy Health, Annual Research Meeting
June 27, 2006
Research Team
John Hsu, MD, MBA, MSCE
Estee Neuwirth, PhD
Julie Schmittdiel, PhD
Laura Arroyo
Ilana Graetz
Kaiser Permanente Division of Research and Institute for Health Policy
Kaiser Permanente Care Management Institute
Thomas Rundall, PhD
Peter Martelli, PhD Candidate
Rodney McCurdy, MHA, PhD Candidate
U. C. Berkeley, Center for Health Research
Mark Gibson
Pam Curtis, MS
Oregon Health Sciences University, Center for Evidence Based Policy
Financial Support: Agency for Healthcare Research and Quality
Introduction



There is a substantial gap between health services
decision makers and the research community
Studies of health organization decision makers in
Canada, the United Kingdom, and the United States
report that decision makers rarely use research evidence
The research community needs to understand the
reasons for this “gap” and work with managers and
policy makers to close it
Source: Kovner and Rundall, Evidence-based Management Reconsidered.
Frontiers of Health Services Management, Spring 2006.
Rationale for the Study




Using research evidence when making decisions about
the organization, financing and delivery of health care
has great appeal
Decision makers, however, do not appear to use this
information
We queried health organization decision makers about
their definition and use of evidence when making
strategic decisions
The findings of this study will help inform the
construction of an evidence-based management toolbox
Study Design
Qualitative
 Four peer-to-peer discussion groups (full day)
 Questions designed to elicit perceptions of






High-priority decisions
Current levels of research evidence use
Characteristics and definitions of useful evidence
Barriers and facilitators to using evidence
Recommended methods for communicating evidence
Methods



Focus group discussions were held in San Francisco,
Oakland, Chicago, and New York
Purposive sampling of health care senior managers
26 organizations of various types







Private provider groups
Health insurance plans and insurers
Public health systems
Employers and employer groups
Regulatory agencies
Organizations with limited resources and wealthier ones
5-10 managers in each focus group (31 total)
Methods
Moderator posed questions and recorded major themes
 Two researchers recorded all comments and compared
notes with meeting audiotapes
 Using content analysis techniques, two researchers
independently coded the text
 Initial 72.1% agreement on codes; discrepant codes
resolved
 Investigators categorized codes into themes
 Coded themes compared with moderators’ themes

Results

Broad list of high-priority decisions





Purchasing to improve quality and reduce costs
Organization of patient care
Organizational sustainability
Benefit design and coverage
Workforce recruitment development, retention

Limited use of research evidence in decision making

Difficulty distinguishing between evidence-based
medicine and evidence-based management
Results

Broad definition of acceptable evidence





Useful evidence must be






Internal data/measurements
Colloquial evidence
Externally dictated standards
Scientific evidence
Accurate
Applicable
Actionable
Accessible
Research (scientific) evidence often not useful
Disconnect between definitions and evaluation of
evidence
 Lack of experience, resources, and incentives for assessing
evidence
 Desire for tools and mechanisms for validating evidence
Results

Tools need to be concise and focused

Need for processes to help decision makers
appraise evidence

Desire for evidence that is “vetted”
Conclusions
There is a substantial gap between health
services decision makers and the research
community
 Decision makers rarely report using research
evidence and instead rely on other types of
information such as peers with experience.
 Decision makers perceive that existing research
does not meet their needs, both in content and
accessibility.

Next Steps?




Build an evidence-based decision making tool box for
managers
Continue to foster collaborative research between health
services researchers and managers (e.g. The work of the
Center for Health Management Research)
Encourage national organizations to work with
appropriate information technology companies and
experts to improve access to relevant research
Training and re-training of health services researchers to
improve their ability to conduct and communicate
research that will be used by health services managers
and policymakers
Next Steps?




Work with health organizations to build cultures,
processes, and structures that support the use of
research evidence in decision making
Recognize the prevalence of sub-optimal management
decisions
Conceptualize, measure, and assess managerial decision
making
How about a new private/public partnership program:
TRIM (Translating Research into Management)