A&F+grant+proposal

CIHR Grant Proposal

**The Context: interventions are needed to improve quality of care**
In the 2001 Institute of Medicine report, “Crossing the Quality Chasm: A New Health System for the 21st Century,” the authors bemoaned the slow or non-existent translation of evidence based recommendations and guidelines into clinical practice. (1) Unintentional variations in care, (2) from surgeries (3) to cancer screening (4) and from mental health (5) to asthma (6), reduce the extent to which the health care system can affect patient outcomes. Recognizing this problem, clinicians, managers, and policy-makers now expend enormous efforts (and funds) implementing strategies to close this ‘quality chasm’. Since quality improvement interventions do not consistently work, (7) there is increasing recognition of the importance of knowledge translation (KT) research, in which the determinants of knowledge use and methods for closing the quality gap are systematically evaluated. (8, 9) For diabetes alone, there have been over 140 randomized trials of interventions aiming to change how providers care for their patients. (10) Increasing the utilization of research evidence for clinical decision-making by health care providers can only be accomplished once the existence of the gap between ideal and actual care is recognized. Unfortunately, rather than constantly trying to address gaps in learning, providers often resist learning in areas of weakness. (11) Audit and feedback (AF), defined as a “summary of performance in a specific area with or without recommendations for action,” (12) is considered an important tool to improve practice because it may overcome providers’ limited ability to accurately self-assess (13) by providing data about discrepancies between current practice and target performance, allowing those who are both motivated and capable to take action.

**The Problem: inconsistent estimates of effect in trials featuring audit and feedback**
For the proportion of evidence-based practice that is in the control of the health care provider, AF is thought to increase recognition of suboptimal performance and act as a cue for action. For this reason, AF is often a foundation of quality improvement interventions. Our team has recently updated a Cochrane review with 140 randomized trials of AF conducted across many clinical conditions and settings and found that AF leads to a median 4.3% absolute improvement (interquartile range 0.5% to 16%) in provider compliance with desired practice (not yet published; see Appendix 1). We also conducted a cumulative meta-analysis of AF trials and found minimal change in the median estimate of effect over time, suggesting that recently published trials of AF have contributed little new knowledge to the literature (not yet published; see Appendix 2). Therefore, the appropriate question to ask is not whether AF is effective, but how to optimize feedback to generate consistently positive effects, as well as to optimize and maximize effects. If the characteristics of successful AF interventions could be ascertained and replicated (e.g. those around the 75th percentile, achieving a 16% absolute improvement), profound benefits on a population-level could be achieved. This is especially true in the current context wherein AF has become more and more common, as funders aim to increase accountability and performance.

**Audit and Feedback: optimal design for improved effectiveness**
While it is likely that some variability observed across AF interventions is related to characteristics of the recipients and their context, much of the effect can be attributed to characteristics of the AF intervention itself. (14, 15) Such characteristics include the source, frequency, channel, and content, as well as the specific behavioural changes required of the recipient to improve their ‘score’. The Cochrane review found that feedback may be more effective when the source is a supervisor or senior colleague, when it is provided more than once, when it is delivered both verbally and in writing, and when it includes both explicit targets and an action plan (not yet published; see Appendix 3). Unfortunately, the Cochrane review was unable to evaluate all of the potentially important AF components and the conclusions regarding how to best design AF are only tentative as they are drawn from indirect comparisons (meta-regression). This was necessary because only 18 of the 140 trials directly compared different ways to design or deliver feedback. Other constraints on the conclusions from the review included inadequate reporting of the interventions in the trials (precluding the kind of detailed specification necessary for cumulative optimization) and infrequent use of design features theorized to enhance AF (meaning that few trials have tested current ideas for optimization). A vast body of quality improvement-related research from various disciplines such as human factors and many branches of psychology, including educational, cognitive, behavioural, and organizational has produced relevant insights regarding how the design of AF may be optimised. (16- 20) For example, psychological theory predicts that including goals and action plans within the feedback will make it more effective, and the meta-regression in the Cochrane review indicated this to be the case, but few trials have directly tested this design feature. (18) Despite the promise of behavioural and cognitive theories for informing how to better design and deliver interventions and for providing a basis for generalization by illuminating the active ingredients, such theories seem to have achieved minimal penetrance amongst quality improvement or implementation trialists. (21-23)

**Audit and Feedback: moving the science forward to strengthen quality improvement**
AF is becoming more widespread in health care and there is emerging evidence that the way it is designed and delivered can substantially alter its ability to change behaviours of providers and improve quality of care. There is no evidence of a concerted effort made by investigators in this field to build upon previous findings to determine how to optimize the design and delivery of this intervention. Instead of moving the science forward, most trials might be better characterized as designed based on the principle of “ISLAGATT” (it seemed like a good idea at the time) without explicitly building on previous research or extant theory. (24) It is reasonable to expect that greater effort toward building upon previous data and utilization of relevant theory to identify the key, active ingredients of the interventions would help implementation researchers and quality improvement stakeholders, including health care system administrators to be more consistently successful and to produce outcomes that are more generalizable. (25) Given the many important aspects of AF to be tested, the potential for significant impact, and the lack of current progress in the field, there is a clear need for coordination of effort and planning a research program to determine how to best design and deliver AF. Recent research to investigate methods for optimizing AF includes re-analyses of the Cochrane review of AF informed by psychological theory, leading to new suggestions regarding feedback design, (18, 26) and a further reanalysis is ongoing by members of our group funded by CIHR. The intent of this work is to identify the active ingredients of AF; a better understanding of how AF works lays the basis for more effective AF design. The results will directly inform the discussions in the proposed meeting since the most promising designs should then be prospectively tested.

**Objectives:**
The overall objective is to develop a research agenda to improve the effectiveness of audit and feedback interventions to reliably improve quality of care. There are two main goals for this meeting grant. First, we will establish the foundation for a program of research aimed at improving the effectiveness of AF interventions. This will include 1a) prioritization of the elements of AF design that should be tested through application of experience, empirical data, and relevant theory; 1b) recognizing settings that provide opportunities for testing AF design; and 1c) utilizing methodology to efficiently test and improve upon AF interventions. Second, we will develop a knowledge translation plan for promoting and further developing this research agenda. This will involve 2a) integrated collaboration with clinical, administrative, and policy-level stakeholders; 2b) disseminating the results of the meeting to these stakeholders, as well as to researchers and trainees; 2c) developing a wiki based platform to support ongoing coordinated collaboration amongst international investigators and to promote coordinated efforts in the field; 2d) establishing plans for new operating grants that will carry out the agenda developed in the meeting.

**Planned Activities:**
Since definitive trials to evaluate AF interventions take a great deal of resources, it is important to focus efforts on testing the most promising characteristics of AF. Even with worldwide coordination working on the components identified by these analyses, testing the hundreds of combinations of elements within AF, not to mention replication in various contexts, would be infeasible. (27) There is a need to accumulate the experience and knowledge of leading international researchers and knowledge users in this area to prioritize the key questions to be asked. In line with objectives 1a) and 1b) input from scientists are needed to determine options most likely to have the greatest impact according to context, specific target behaviour, and type of recipient, while input from clinical and policy stakeholders are needed to determine which options are most feasible. New methodological approaches have been proposed to rule out less promising interventions and to test combinations of components theoretically postulated to work synergistically together. These include the very promising Multiphase Optimisation Strategy (MOST), using fractional factorial study designs. (28) In line with objective 1c) this meeting will provide an opportunity to promote this efficient methodology for future studies. Prior to the meeting, the organizing committee (NI, JG, JB, AS, ME, MW, MS, MZ, SM – see Appendix 4) will initiate the process of developing a list of potential components (and categories) of feedback interventions. (A preliminary version is illustrated in Appendix 5.) The team will organize resources and references to be distributed to all participants, summarizing the key theoretical perspectives and evidence base regarding AF. All participants will be asked to review the background reading ahead of the meeting and each will be requested to provide further references of interest and to forward additional AF-related components that might be added to the initial list of AF design options. To maximize the utility of this task, an international team of committed collaborators representing a variety of research disciplines and clinical contexts have been invited (see letters of collaboration). All suggestions from collaborators will be collated and presented at the meeting. The meeting will last two days and will proceed in stages: advancing the list of AF components to identify all possible variations of feedback interventions that are expected to alter effectiveness; applying expertise in feedback-relevant theories and practical experience from the field to systematically narrow down the list of options to focus on those judged most important and also feasible; discussing potential methodologies for testing these variables; and developing plans to start purposeful collaborations to move the science forward by identifying promising contexts and clinical topics for implementing the lessons of the meeting (see Appendix 6 for a tentative schedule). The meeting will include a mix of didactic and interactive sessions, in small and large groups to facilitate participation and learning. In line with objective 2a) international participants with relevant expertise will lead workshops and clinical, administrative, and potential knowledge users will be purposively integrated in all aspects. In line with objective 2c) the organizing committee will prepare a ‘complete’ list and a ‘focused’ list of AF design options immediately following the meeting and post it on an AFwiki. This will be coupled with a summary of the meeting minutes and distributed to all participants to seek additional feedback. Particularly promising opportunities for collaboration in answering the key questions will be highlighted and encouraged. In line with objective 2d) international collaboration will be encouraged by working with interested parties to act upon the agenda for AF research by pursuing new operating grants (including, but not limited to, CIHR KT-related grants such as Knowledge to Action and PHSI). These activities will be coordinated through an AF-wiki to promote collaboration and facilitate participation. This wiki will also provide a forum for the development of a prospective registry of AF studies, enabling researchers to more systematically utilize and build upon extant knowledge in the field.

**Project Team: experience and infrastructure**
The nominated PI is a CIHR Research Fellow and the lead author of the latest update of the Cochrane review of AF (presently in review; see Appendix 1 for summary of results). The organizing committee has collectively been involved in many trials featuring AF and has a track record of success working together on related topics. The team represents broad disciplinary expertise and can provide international perspectives. This committee will: produce the necessary documents in preparation of the meeting; coordinate and host the meeting; lead the writing for the meeting report and associated manuscripts; and facilitate new interdisciplinary teams to develop operating grants to act upon the plans developed. Other participants are key contributors to the current evidence base reviewed above or interested stakeholders with the clinical expertise and administrative/policy-level influence to identify and act upon opportunities for future work in the field. Participants have been purposefully selected to ensure a wide variety of disciplines and clinical contexts are represented (see Appendix 4 and letters of collaboration for more detail). This work will be facilitated by KT Canada, a CIHR-funded national organization of researchers committed to knowledge translation research. KT Canada will contribute administrative support to help in the coordination of this meeting and development of the AF-wiki. The organization has proven success with this type meeting, conducting a yearly KT Summer Institute in which international experts are joined with nascent KT-learners from across Canada. This provides an ideal opportunity to disseminate the results and engage with new investigators and trainees in the field. KT Canada team-members also have proven success with international collaborations having recently used a wiki to facilitate new ethics guidelines for cluster-randomized trials (commonly in AF studies). Crucially, KT Canada has committed to provide up to $10,000 toward the event. In addition, the meeting will be facilitated by the Women’s College Hospital, who will provide additional in-kind support to coordinate and host the event (see letters of support and detailed budget).

**Deliverables and expected benefits:**
The proposed activities supported by this grant will bring together a diverse set of interested researchers and knowledge users keen to collaborate in order to optimize knowledge translation interventions. We believe that including multiple research disciplines and integrating clinical, administrative, and policy knowledge users through the process will ensure that the team both considers all relevant perspectives and focuses next steps on the most pressing issues. To this end, we have already garnered interest of a list of key international research experts and critical knowledge users from Canada and abroad. As mentioned above, this proposal is organized to accomplish two broad goals: 1) establish an agenda for a program of research aimed at improving the effectiveness of AF interventions and 2) develop an integrated KT plan for promoting and achieving the research agenda. In the short term, this meeting will result in the publication of a meeting report that will inform the design of future AF trials. This will be disseminated through publication of a discussion paper in Implementation Science, presentations at KT Canada Seminar Series and Summer Institute, as well as at the NIH Science of Dissemination and Implementation conference. In addition, a briefing document will be sent to key KT research funders. In the long-term, the organizing committee will aim to leverage this meeting through the development of an international collaborative network that will run a series of coordinated AF trials. The knowledge users will provide key insights for prioritizing the research agenda. Thus, outcomes of this meeting will allow for an integrated approach to improving our understanding of how to use AF to close the quality chasm.