CIHR Grant Proposal

The Context: interventions are needed to improve quality of care

In the 2001 Institute of Medicine report, “Crossing the Quality Chasm: A New Health System
for the 21st Century,” the authors bemoaned the slow or non-existent translation of evidence based
recommendations and guidelines into clinical practice. (1) Unintentional variations in care, (2) from
surgeries (3) to cancer screening (4) and from mental health (5) to asthma (6), reduce the extent to
which the health care system can affect patient outcomes. Recognizing this problem, clinicians,
managers, and policy-makers now expend enormous efforts (and funds) implementing strategies to
close this ‘quality chasm’. Since quality improvement interventions do not consistently work, (7) there
is increasing recognition of the importance of knowledge translation (KT) research, in which the
determinants of knowledge use and methods for closing the quality gap are systematically evaluated.
(8, 9) For diabetes alone, there have been over 140 randomized trials of interventions aiming to change
how providers care for their patients. (10)
Increasing the utilization of research evidence for clinical decision-making by health care
providers can only be accomplished once the existence of the gap between ideal and actual care is
recognized. Unfortunately, rather than constantly trying to address gaps in learning, providers often
resist learning in areas of weakness. (11) Audit and feedback (AF), defined as a “summary of
performance in a specific area with or without recommendations for action,” (12) is considered an
important tool to improve practice because it may overcome providers’ limited ability to accurately
self-assess (13) by providing data about discrepancies between current practice and target performance,
allowing those who are both motivated and capable to take action.

The Problem: inconsistent estimates of effect in trials featuring audit and feedback

For the proportion of evidence-based practice that is in the control of the health care provider,
AF is thought to increase recognition of suboptimal performance and act as a cue for action. For this
reason, AF is often a foundation of quality improvement interventions. Our team has recently updated
a Cochrane review with 140 randomized trials of AF conducted across many clinical conditions and
settings and found that AF leads to a median 4.3% absolute improvement (interquartile range 0.5% to
16%) in provider compliance with desired practice (not yet published; see Appendix 1). We also
conducted a cumulative meta-analysis of AF trials and found minimal change in the median estimate
of effect over time, suggesting that recently published trials of AF have contributed little new
knowledge to the literature (not yet published; see Appendix 2). Therefore, the appropriate question to
ask is not whether AF is effective, but how to optimize feedback to generate consistently positive
effects, as well as to optimize and maximize effects. If the characteristics of successful AF
interventions could be ascertained and replicated (e.g. those around the 75th percentile, achieving a
16% absolute improvement), profound benefits on a population-level could be achieved. This is
especially true in the current context wherein AF has become more and more common, as funders aim
to increase accountability and performance.

Audit and Feedback: optimal design for improved effectiveness

While it is likely that some variability observed across AF interventions is related to
characteristics of the recipients and their context, much of the effect can be attributed to characteristics
of the AF intervention itself. (14, 15) Such characteristics include the source, frequency, channel, and
content, as well as the specific behavioural changes required of the recipient to improve their ‘score’.
The Cochrane review found that feedback may be more effective when the source is a supervisor or
senior colleague, when it is provided more than once, when it is delivered both verbally and in writing,
and when it includes both explicit targets and an action plan (not yet published; see Appendix 3).
Unfortunately, the Cochrane review was unable to evaluate all of the potentially important AF
components and the conclusions regarding how to best design AF are only tentative as they are drawn
from indirect comparisons (meta-regression). This was necessary because only 18 of the 140 trials
directly compared different ways to design or deliver feedback. Other constraints on the conclusions
from the review included inadequate reporting of the interventions in the trials (precluding the kind of
detailed specification necessary for cumulative optimization) and infrequent use of design features
theorized to enhance AF (meaning that few trials have tested current ideas for optimization).
A vast body of quality improvement-related research from various disciplines such as human
factors and many branches of psychology, including educational, cognitive, behavioural, and
organizational has produced relevant insights regarding how the design of AF may be optimised. (16-
20) For example, psychological theory predicts that including goals and action plans within the
feedback will make it more effective, and the meta-regression in the Cochrane review indicated this to
be the case, but few trials have directly tested this design feature. (18) Despite the promise of
behavioural and cognitive theories for informing how to better design and deliver interventions and for
providing a basis for generalization by illuminating the active ingredients, such theories seem to have
achieved minimal penetrance amongst quality improvement or implementation trialists. (21-23)

Audit and Feedback: moving the science forward to strengthen quality improvement

AF is becoming more widespread in health care and there is emerging evidence that the way it
is designed and delivered can substantially alter its ability to change behaviours of providers and
improve quality of care. There is no evidence of a concerted effort made by investigators in this field
to build upon previous findings to determine how to optimize the design and delivery of this
intervention. Instead of moving the science forward, most trials might be better characterized as
designed based on the principle of “ISLAGATT” (it seemed like a good idea at the time) without
explicitly building on previous research or extant theory. (24) It is reasonable to expect that greater
effort toward building upon previous data and utilization of relevant theory to identify the key, active
ingredients of the interventions would help implementation researchers and quality improvement
stakeholders, including health care system administrators to be more consistently successful and to
produce outcomes that are more generalizable. (25)
Given the many important aspects of AF to be tested, the potential for significant impact, and
the lack of current progress in the field, there is a clear need for coordination of effort and planning a
research program to determine how to best design and deliver AF. Recent research to investigate
methods for optimizing AF includes re-analyses of the Cochrane review of AF informed by
psychological theory, leading to new suggestions regarding feedback design, (18, 26) and a further reanalysis
is ongoing by members of our group funded by CIHR. The intent of this work is to identify
the active ingredients of AF; a better understanding of how AF works lays the basis for more effective
AF design. The results will directly inform the discussions in the proposed meeting since the most
promising designs should then be prospectively tested.

Objectives:

The overall objective is to develop a research agenda to improve the effectiveness of audit and
feedback interventions to reliably improve quality of care. There are two main goals for this meeting
grant. First, we will establish the foundation for a program of research aimed at improving the
effectiveness of AF interventions. This will include 1a) prioritization of the elements of AF design that
should be tested through application of experience, empirical data, and relevant theory; 1b)
recognizing settings that provide opportunities for testing AF design; and 1c) utilizing methodology to
efficiently test and improve upon AF interventions. Second, we will develop a knowledge translation
plan for promoting and further developing this research agenda. This will involve 2a) integrated
collaboration with clinical, administrative, and policy-level stakeholders; 2b) disseminating the results
of the meeting to these stakeholders, as well as to researchers and trainees; 2c) developing a wiki
based platform to support ongoing coordinated collaboration amongst international investigators and to
promote coordinated efforts in the field; 2d) establishing plans for new operating grants that will carry
out the agenda developed in the meeting.

Planned Activities:

Since definitive trials to evaluate AF interventions take a great deal of resources, it is important
to focus efforts on testing the most promising characteristics of AF. Even with worldwide coordination
working on the components identified by these analyses, testing the hundreds of combinations of
elements within AF, not to mention replication in various contexts, would be infeasible. (27) There is a
need to accumulate the experience and knowledge of leading international researchers and knowledge
users in this area to prioritize the key questions to be asked. In line with objectives 1a) and 1b) input
from scientists are needed to determine options most likely to have the greatest impact according to
context, specific target behaviour, and type of recipient, while input from clinical and policy
stakeholders are needed to determine which options are most feasible. New methodological approaches
have been proposed to rule out less promising interventions and to test combinations of components
theoretically postulated to work synergistically together. These include the very promising Multiphase
Optimisation Strategy (MOST), using fractional factorial study designs. (28) In line with objective 1c)
this meeting will provide an opportunity to promote this efficient methodology for future studies.
Prior to the meeting, the organizing committee (NI, JG, JB, AS, ME, MW, MS, MZ, SM – see
Appendix 4) will initiate the process of developing a list of potential components (and categories) of
feedback interventions. (A preliminary version is illustrated in Appendix 5.) The team will organize
resources and references to be distributed to all participants, summarizing the key theoretical
perspectives and evidence base regarding AF. All participants will be asked to review the background
reading ahead of the meeting and each will be requested to provide further references of interest and to
forward additional AF-related components that might be added to the initial list of AF design options.
To maximize the utility of this task, an international team of committed collaborators representing a
variety of research disciplines and clinical contexts have been invited (see letters of collaboration). All
suggestions from collaborators will be collated and presented at the meeting.
The meeting will last two days and will proceed in stages: advancing the list of AF components
to identify all possible variations of feedback interventions that are expected to alter effectiveness;
applying expertise in feedback-relevant theories and practical experience from the field to
systematically narrow down the list of options to focus on those judged most important and also
feasible; discussing potential methodologies for testing these variables; and developing plans to start
purposeful collaborations to move the science forward by identifying promising contexts and clinical
topics for implementing the lessons of the meeting (see Appendix 6 for a tentative schedule). The
meeting will include a mix of didactic and interactive sessions, in small and large groups to facilitate
participation and learning. In line with objective 2a) international participants with relevant expertise
will lead workshops and clinical, administrative, and potential knowledge users will be purposively
integrated in all aspects. In line with objective 2c) the organizing committee will prepare a ‘complete’
list and a ‘focused’ list of AF design options immediately following the meeting and post it on an AFwiki.
This will be coupled with a summary of the meeting minutes and distributed to all participants to
seek additional feedback. Particularly promising opportunities for collaboration in answering the key
questions will be highlighted and encouraged. In line with objective 2d) international collaboration will
be encouraged by working with interested parties to act upon the agenda for AF research by pursuing
new operating grants (including, but not limited to, CIHR KT-related grants such as Knowledge to
Action and PHSI). These activities will be coordinated through an AF-wiki to promote collaboration
and facilitate participation. This wiki will also provide a forum for the development of a prospective
registry of AF studies, enabling researchers to more systematically utilize and build upon extant
knowledge in the field.

Project Team: experience and infrastructure

The nominated PI is a CIHR Research Fellow and the lead author of the latest update of the
Cochrane review of AF (presently in review; see Appendix 1 for summary of results). The organizing
committee has collectively been involved in many trials featuring AF and has a track record of success
working together on related topics. The team represents broad disciplinary expertise and can provide
international perspectives. This committee will: produce the necessary documents in preparation of the
meeting; coordinate and host the meeting; lead the writing for the meeting report and associated
manuscripts; and facilitate new interdisciplinary teams to develop operating grants to act upon the
plans developed. Other participants are key contributors to the current evidence base reviewed above
or interested stakeholders with the clinical expertise and administrative/policy-level influence to
identify and act upon opportunities for future work in the field. Participants have been purposefully
selected to ensure a wide variety of disciplines and clinical contexts are represented (see Appendix 4
and letters of collaboration for more detail).
This work will be facilitated by KT Canada, a CIHR-funded national organization of
researchers committed to knowledge translation research. KT Canada will contribute administrative
support to help in the coordination of this meeting and development of the AF-wiki. The organization
has proven success with this type meeting, conducting a yearly KT Summer Institute in which
international experts are joined with nascent KT-learners from across Canada. This provides an ideal
opportunity to disseminate the results and engage with new investigators and trainees in the field. KT
Canada team-members also have proven success with international collaborations having recently used
a wiki to facilitate new ethics guidelines for cluster-randomized trials (commonly in AF studies).
Crucially, KT Canada has committed to provide up to $10,000 toward the event. In addition, the
meeting will be facilitated by the Women’s College Hospital, who will provide additional in-kind
support to coordinate and host the event (see letters of support and detailed budget).

Deliverables and expected benefits:

The proposed activities supported by this grant will bring together a diverse set of interested
researchers and knowledge users keen to collaborate in order to optimize knowledge translation
interventions. We believe that including multiple research disciplines and integrating clinical,
administrative, and policy knowledge users through the process will ensure that the team both
considers all relevant perspectives and focuses next steps on the most pressing issues. To this end, we
have already garnered interest of a list of key international research experts and critical knowledge
users from Canada and abroad.
As mentioned above, this proposal is organized to accomplish two broad goals: 1) establish an
agenda for a program of research aimed at improving the effectiveness of AF interventions and 2)
develop an integrated KT plan for promoting and achieving the research agenda. In the short term, this
meeting will result in the publication of a meeting report that will inform the design of future AF trials.
This will be disseminated through publication of a discussion paper in Implementation Science,
presentations at KT Canada Seminar Series and Summer Institute, as well as at the NIH Science of
Dissemination and Implementation conference. In addition, a briefing document will be sent to key KT
research funders. In the long-term, the organizing committee will aim to leverage this meeting through
the development of an international collaborative network that will run a series of coordinated AF
trials. The knowledge users will provide key insights for prioritizing the research agenda. Thus,
outcomes of this meeting will allow for an integrated approach to improving our understanding of how
to use AF to close the quality chasm.