AF+meeting+minutes

**Day 1 - December 6th, 2012 **
 * Audit and Feedback Meeting **


 * Large group presentations: Results from Cochrane review of AF: what do we know and where do we go from here? **
 * Noah’s presentation “Using AF to improve quality of care” **
 * Findings from the 2006 Cochrane review
 * 2012 update
 * Meta Regression
 * Format, source, frequency, recipient, baseline performance, risk of bias
 * Plus exploratory analyses
 * 140 trials – AF as a primary component – most were multifaceted
 * Results:
 * Similar to the previous results – AF seems to work
 * Both verbal + written seems to be more effective
 * Given by a senior colleague more effective than from employer representative
 * Moderately frequent vs. only once
 * Goal plus action plan
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">More effective when the feedback attempted to decrease the behaviour
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">AF more effective for prescribing than other behaviours – why? Need to list exactly what is targeted in context – both of situation (material and social) and of other behaviours.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">More effective when baseline performance is lower
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What you were targeting seems to be important – e.g., getting doctors to change their prescribing behaviour---needs more work.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Comments from the group members: **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Should focus on the middle bullets first – as this is what many AF researchers tick off first.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How many behaviours can we look at once?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Important to look at the context
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Try to think about why they may or may not be a modifier
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Personal vs. interpersonal characteristics of the feedback
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Issue of sustainability


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan Michie’s Presentation: “Applying theory to designing AF interventions and evaluations in head to head trials” **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Why Theory? **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">More effective, provides a framework to facilitate (accumulation of evidence, communication across research groups), identifies mechanisms of action

<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">MRC Guidance for developing and evaluating complex interventions (Craig et al 2009, BMJ)


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What Theories? **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">MRC guidance silent on this
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">NICE’s Behaviour Change evidence review (2008)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Identified evidence-based principles of behaviour change (Abraham, Kelly, West, Michie, 2008)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">No guidance on which theories to use
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Starting point for selecting theory
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Understand intervention content
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need a method for specifying content (BCTs)


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Types of A&F **
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Intensive //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">(individual recipients) AND (verbal format) OR (a supervisor or senior colleague as the source) AND (moderate or prolonged feedback)
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Non-Intensive //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> (group feedback) NOT (from supervisor) OR (individual feedback) AND (written) AND (containing info about cost or numbers of tests without personal incentives)
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Moderate //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">(any other combination of characteristics than described in intensive or non-intensive)


 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Problems of categorising by intensity //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> (no theoretical rational, few recommendations for practice offered, mixture of modes of delivery and content)
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">A theory-based approach: //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Specify content as BCTs to allow theoretically based categorisation & analysis; generate theory-based hypotheses concerning effectiveness
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Specifying content: 13 papers from AF review; 28 distinct BCTs grouped into //goal/standard setting, feedback & action planning//
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What kind of theory would help? Self-regulation theory (Carver & Scheier, 1982)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Feedback more effective when goal/target is set
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Most effective where goal/target & action plan

<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Ensure all behaviour change techniques are identified (within and beyond AF; both intervention and control group)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Head to head trials: On what basis does one select intervention components? **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need to have a theory about how AF is working
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What functions are AF playing?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Structure for noticing and reducing discrepancy (target, fb, action plan)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Cue to action
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Reinforcement
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Social support (how it’s given)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Others?


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Summary **<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">:
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Detailed description of intervention a starting pt for identifying mechanisms of action (ie. theory)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Might need to draw on more than one formal theory to generate hypotheses about mechanisms.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">These hypotheses should guide intervention design, optimisation, evidence synthesis and trial design
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Group Discussion: **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Anne Sales: Does feedback have a role in promoting motivation? SM – yes-absolutely
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Feedback more than once is more effective – how we should apply the theory in terms of multiple cycles of feedback….Susan: want feedback to be unpredictable of when it is coming.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Survey response rates are low – survey physicians to see if they are reducing discrepancy first
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Theories of feedback; theories of goals – should we be going for a theory of AF? Susan: yes – that’s what we are trying to do – a program theory. Can we think about generating a theory of AF from more general theories?


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Heather Colquhoun: “A SR of AF interventions: the good, the bad, and the ugly” **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">To whom: 51% given to individuals; rest group (18%), both (16%), unclear (14%); 92% FB given to target person
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What was given? 79% given about behaviour, 14% on outcomes, 32% other (cost, accuracy of diagnosis, risk data, education, barriers to change) 58% on an individual’s performance, 64% group performance, mostly aggregate (81%)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">*Comment: Why cost not part of behaviour – it is actionable? Susan – these are outcomes of behaviour – feedback on behaviour is the most effective.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Another participant comment: It’s a construct that can be pulled out. Like what we did.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Comment: How much were classified as education – correct solution information?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">36% was graphical; 14% was unclear.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">89% addressed the behaviour to be changed; 6% no; 5% unclear.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Clear comparison in the fb? 74% yes. (mostly others previous performance 68% – very few multiple comparisons)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How delivered? Face to face (44%)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How much? Total fb : 24% unclear; clear 76% (33/107 given only once, only 27 given more than 4x)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Comment: self activated/real time – have it in front of you? Were there any of these?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Comment: giving it the same way each time (e.g. 3 X the same way…would there be a difference? )
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Comment: Anne Sales – new information to people – highly variable; background issue that is important.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Lag time between collection of the fb and the provision of the fb not clearly stated (37%); if yes – mostly months.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Rational for AF in the trial? 36% was empirical; 28% was intuitive (AF has never been used, but thought it would be a good idea), 26% no rational (just appear in the intervention), 9% used theory to design the intervention.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Running list of the articles that included the AF reports – only 7 did this.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Summary – some consistency, but also wide variation, reporting is poor, rationale for the intervention is lacking


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">AFTER LUNCH.... **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Large Group Discussion – Key Thoughts to Consider for the Meeting: **
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan Michie //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: What might be in a good intervention (includes theory)? The criteria that you would use to evaluate? Nature of the study design for the evaluation?
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Sylvia Hysong: //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Better idea of what the priority criteria is (e.g., the 5 things you would want to test because....)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Cumulative meta-analysis– shouldn’t discount 140 trials – clearly a lot of theory has gone into it. Don’t want to get into “paralysis by analysis” by spending a lot of time working thru this and not doing new trials.
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Merrick: //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Unconfident about the prospects of a detailed predictive theory. But, explicit ideas of how something will work seems like a good idea.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Not really sure what the “best bet” looks like
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Jeremy: //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Goal of the meeting: same mistakes have been made over the years. If we can start to think about some of the basics of what we should do – this would be helpful – so how to take it forward in the field. This is a “call to action” – to be more effective, rather than just repeating trials.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How do we best do it is the real question
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Jamie //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Brehaut: the cumulative meta-analysistells us that we are not getting better – most likely because we don’t have any theory. Very little theory went into the development of these trials –need to develop more thoughtful AF.
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan Michie //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: 3 research questions that we talked about 1.) Can we design a better AF intervention as a whole? 2.) What is the relative contribution of components within an AF intervention? 3) What are the mechanisms by which the effects are happening?
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Anne Sales //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: We don’t want to necessarily come up with a top 5 today, but to leave tomorrow with some possible studies?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Even if you don’t think you have theory – you have theory! Need a program theory – some things you deprioritize and some things become more vivid. Can trade off one program theory for another.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Plan: To suggest where things could move forward.

__<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Other group’s additions: __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Acceptability (fits into above list), actionability, strength of evidence in relation to the guideline (fit with SMART category)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Group exercise 1: Susan’s group presentation **
 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Goal standard audit //**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Visibility of required behaviour
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Modifiability of behaviour (co-morbidity of pts, extent to which it is automated, habit strength)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Engagement with setting the goal
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Number of goals
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Quality of goal
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">SMART goals (achievable, relevant, time bound).
 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Feedback: //**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Competence of deliverer (e.g., how to give fb)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Seriousness of outcome
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Profile/importance of outcome in society
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Salience/vividness/emotional connotations of fb
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Recipient seeks fb rather passively receiving it
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Type/aggregation of behaviours
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Multiple behaviours
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Framing(positive vs. negative/gain vs. loss)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Complexity of target (different types, draw on theory)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Credibility of source
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Goal orientation (motivation, mastery vs. learning) of recipients
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Intrusiveness of intervention vs. part of daily routine
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Visibility of the feedback
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Timeliness
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Interpretation, time to review
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Opportunity to reflect and discuss (e.g. on causes of behaviour, instances of successful performance)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Support from management/leaders for feedback given
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Graded feedback ( starting positively)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Multimodality
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Format re new media
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Linked to key messages
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Feedback to teams
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Longitudinal feedback - to tell a story

__<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Other group’s additions: __
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How feedback is collected (e.g., self-reported), counts vs. priorities (% achieved or appropriateness),
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Jeremy //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: cognitive understanding of the AF (comprehensibility of the feedback)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Timing, frequency – hard to tease out
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Context – picking apart things piece by piece isn’t going to work; interventions need to consider if they are patient level, organizational level etc, what are the elements that are modifiable?


 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Action Plan – the group had little to say on this. //**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Involvement with creating action plans
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Nature of action plans ( e.g. “ if then plans”)
 * __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Others groups addition: __
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Availability of tools for action
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Who controls the action? //(Anne Sales)//
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Who’s developing the action plan? //(Jeremy)//


 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Characteristics of the recipients //**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Past experience of AF as more or less useful
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Alignment with goals, values etc of recipient re. receiving feedback and changing behaviour
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Knowledge of the goals, and shy they are there
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">allegiance to goals
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Perceived social comparison with groups/individuals/social norm
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Trust of the organization
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Autonomy of recipients
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Intuitive vs. reflective orientation
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Learning style
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Authority of recipient
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Degree to which recipients are organized/self-regulation
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Education level/reflectiveness
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Responsibility of practice
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Union membership
 * __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Other group’s additions: __
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Was the recipient a volunteer?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Personality traits of the recipients
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Is feedback given to individual or a group?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Time that AF is being delivered
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Considerations of equity (fairness of comparison)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Motivation for change
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Perceived consequences of achieving the goal
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Self efficacy (confidence in being able to do it)


 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">General/ External aspects //**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Combined with other interventions/elements (how to define these?)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Financial, or other consequences of AF
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Whether AF is evaluated as a process
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Automation of practice
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Preparation/anticipation for intervention in advance (e.g., present audit details, and that they will be assessed)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Time spent on intervention
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Engagement in designing AF
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Alignment with goals/values, etc of recipient
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Comment on automation – control behaviour through system design – via technologies, financial incentives, public reporting – is it different from AF?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Patient activation (patient feedback for a physician)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Action planning as a co-intervention
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Prevalence of the condition being audited (e.g., if it’s rare – doesn’t matter how much you get) ---AF may be much better suited for common conditions, as opposed to rare/expensive interventions. __Note__: Heather added that based on the extraction – only 3 instances where this was reported in the literature.
 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Mechanism of Action: //**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Relationship between components and outcome (e.g. Curvilinear)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Is this really point a mechanism (Anne)? Association between type of AF and the type of outcome. Basically - more is not always better – not necessarily linear.
 * __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Other group’s additions __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">:
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">R. Foy’s comment: We’ve been discussing elements of an intervention and other things that are apart of context. Hard to generalize from one study to another if we don’t describe these in our studies. What’s the risk of repeating this work? Number of existing taxonomies describing context.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">***Susan: //what is AF most relevant for// in terms of type of behaviour, situation etc., //nuts and bolts// of the actual intervention, //context// in which it happens, //mechanisms// of action...

<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">FOLLOWING SMALL GROUP SESSIONS – **RESEARCH AREAS THAT SHOULD BE ADDRESSED TO MOVE THE AF FIELD FORWARD**
 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Jill Francis’s Group //**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Discussed the need to have a publication for the design principals of AF
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need for reporting guidelines – but then steered away from this idea – they don’t always work that well
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Discussion about how clear those principals could be. (e.g., could we say “don’t do AF once” do we have the evidence to support that?)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Leaders of AF – many areas and pieces that can be manoeuvred
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need to define AF in terms of its prototypical and discretionary elements – what needs to be there in order to be minimally defined?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need for a common language
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need to look at the variation in effects of AF trials that only used 1 time AF (variation of effects or case studies)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Movement towards enrolling everyone in the study – every study for the control arm would have the AF---worked effectively in child health.


 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Anne Sales Group //**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Taxonomy would be great; so start with a Delphi process?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Key questions: (“Meta questions”)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">When is AF the preferred intervention and what is it conditioned on? Likely around the characteristics of the topic/target group.
 * 1) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What is it that needs to be tailored about an AF intervention that is sensitive to context/topic/participants?
 * 2) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Meta-synthesisof non RCT – there is another literature that doesn’t get included in the Cochrane reviews


 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Merrick’s group //**
 * **<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">5 research questions **
 * 1) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What is the impact of engaging the recipient in (design, implementation) vs. not engaged
 * 2) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Impact of combining AF with one of the following:
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Incentives/penalties (financial, CME, licensing)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Tools/practise aids (clinical decision tool)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Educational or academic detailing, group learning, supplementary
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Practise redesign (coaches, facilitation, mentorship)
 * 1) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Presence of face to face feedback vs. non personal feedback (degree of skill of person providing the feedback)
 * 2) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Take the top quintile of AF studies and replicate their interventions
 * 3) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">The 5 most important aspects of AF to study are:
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Frequency; individual vs. group; benchmarks, in person vs. electronic delivery, number of targets
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Developing strategies for replicating and implementing successful interventions in other settings, looking at sustainability
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What is the yield of AF focussed on - low performing vs. on average performing practitioners (where the target is important of course!)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Is there a different impact If feedback is given to different professions providing the same kind of service (e.g., physician vs. midwife)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Does it have a different effect if given to office staff vs clinician only + clinician


 * **//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan’s group – types of research questions //**
 * 1) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What tasks are most amenable to effective AF?
 * 2) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How to optimise -
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Don’t need placebo, but have standard feedback as control
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Individualised data, comparison data, feedback of recent performance, include written/visual feedback
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Is audit on its own an intervention – if people are aware that their performance is being measured without feedback?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Are Cochrane findings a starting point from which to generate more effective interventions?
 * 1) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Relative effectiveness of, and interactions between components
 * 2) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What components are most promising? Hypotheses
 * 3) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">More time to reflect on the feedback
 * 4) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Level of aggregation of feedback data should be aligned to performance to be changed
 * 5) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Te AF targets to team based or organisational goals
 * 6) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Engage recipients in iteratively designing and implementing AF


 * 1) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What are mechanisms of action of AF interventions e.g., time may work, identify goals for improvement, plans to improve performance, greater depth of processing
 * 2) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How to standardise language for describing AF interventions?
 * 3) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What paradigm are we in regarding user-centre design; adaptive interventions – methodological innovation + qualitative studies of recipients to look at user experiences


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Jeremy’s summary comments from Day 1: **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">AF seems to be effective, but it’s a disorganized field.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Want to develop a position/guidance paper to improve the value of the science.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Agreement about foundational work that needs to get done:
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need to have an agreement about language
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need better reporting of existing studies
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Broad meta questions (e.g., when it might be useful).
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Sometimes AF can be done, but it might not be the most optimal thing to do
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How can we optimise the effects of AF?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Can think about in terms of intrinsic features (how can we do it better), absence or presence of co-interventions, comparing it to other interventions, extrinsic features – how does the context modify AF?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Sustainability (research to service world)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Understanding the mechanism better
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Head to head trials – engaging healthcare systems/integrated approach
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Can we provide a menu of priorities that would be useful?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">General comments from the group: **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan – still quite a lot of overlap in the ideas. Reflect on what the “buckets” are – members can go to whichever group their expertise/interests are.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Merrick – reducing the list is needed – we need a fairly formal approach of doing this. Which attributes/co interventions go well with AF is not very different from when we should use AF.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan – methodologies – thinking about engineering design principles

**<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Day 2: December 7th, 2012 ** **<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Developing an active, international AF research collaboration **

//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Robbie Foy‘s Presentation // //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Steve Ornstein’s Presentation // //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Sue Wells’s Presentation – Audit and Feedback NZ style // //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">David Price’s Presentation // //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Stakeholder point of view: // <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Response: Robbie Foy – written feedback that may or may not be shared with staff/department. [Action plans – given suggestions and support materials, meetings]. David Price ---“Audit and dissemination” – they get a score card that is posted, sometimes do quality improvement projects. Steve – “passive dissemination” on a website, reviewing with providers and staff. <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Response from Steve – Could systematically evaluate them at a low cost. <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">No clear reasonsocial influences, risk aversion, my patients are different, believing that they aren’t based on gold standard recommendations etc. //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Sylvia Hysong //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: responding to the question about how uptake was measured – could track and monitor whether people were logging into and checking the report – via the website. They could see that uptake dropped after the financial incentive was removed. Another study – survey of recipient’s reactions to feedback. //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Noah //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: how clear is it what people are supposed to do with the feedback? Is there an opportunity to automate their responses to the feedback? //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Anne Sales //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: Suggestion to start with places that have less audit and feedback experience (patient settings/long term care etc) Other settings should be considered … //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Jeremy //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">:
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Review of Day 1 **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Intrinsic, extrinsic factors (topic, recipient) and mechanisms of action
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Position paper – what is the state of the science/best practice in general
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Key questions to ask for people developing AF interventions/elements to include
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Key questions to move the field forward
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Real life instances where AF is occurring – Laboratories **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Blood Transfusions study (transfusions costly/given to people who don’t need them)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">4 streams/stages (developing/piloting/code/enhanced fb)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Clustered randomised trial on user fb incorporating economic modeling and process evaluation
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Enhanced fb compared with usual fb
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Enhanced context (approximating goal setting?) – Does the fb identify the behaviours interested in – is it related back to a goal?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Enhanced delivery (approximating to action planning?) – Who gets the fb; do they make an action plan; does the team respond /use the fb?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">2x2 factorial cluster randomised trial; targeted behaviour: red cells and platelets
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">152 hospitals
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Practice Partner Research Network (PPR net ) – over 200 practices in the US
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Practice based learning and research organization to improve health care in its member practices and elsewhere in US
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Aim to turn clinical data (EMR) into actionable info; empirically test theoretical sounds interventions using the EMRs ; disseminate successful interventions
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">In all PPRNET studies - they use AF as the control condition.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Do summary measures of care; number of measures that practices are above the PPR benchmark
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">This info isn’t really actionable – what about the pts that need care? They have a patient level report for this.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">A list of all measures that they report on – # of patients, number that meet the criteria, and number that don’t, each pt level report has context sensitive information
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Primary care organisations
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Each person gets a unique NHI (linked with age, gender, ethnicity, NZDep score)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">2002 – Cardiovascular risk assessment – guidelines put into computerized decision support (~ 400 practices)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">PREDICT – at first ~ 3% had a cardiovascular risk assessment; now about 50%
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Report provided to GPs and nurses – could see practice/GP level data – could see which pts they needed to see. Found that it wasn’t helpful – denominator = eligible population assessed
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Widespread implementation, AF query tools for enrolled population, for multiple conditions, variable PHO use – benchmarking, CME, MOH reporting
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Anonymised linkage of CVD risk profiles to outcomes; NHI is linked to national and regional data
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">ProCare radiology point of care commissioning – gives them decision support about the appropriateness of their order, cost info, waiting time for routine/urgent, budget, balance.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Before –overspending budget by 30% - after remained within budget every year for past 4 years
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Before – 25% ultrasound referrals clinically inappropriate, currently 0.1% inappropriate
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Before waiting time for ultrasound 4 weeks to 6 months....now waiting times less than 5 days
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Kaiser (consists of medical groups & health plan; 9 different States in the US)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Regional health plan that partners with independent Permanente medical group (Kaiser Permanente) – independent entities that work together
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">All use the same EMR
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">HMO research network
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Beginning to introduce AF into some of their research projects – not an AF trial per se
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What does AF add to CME?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">More attention needs to be paid to best evidence to networks
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Targeting funding for something like this…
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Chronic pain – looking for a pre/post project around practice guidelines
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Inclusion of decision makers/policy makers in the work
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Discussion/questions about the “laboratories” presented: **
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Anne Sales //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: If doing feedback directly - how is it delivered? How do you measure uptake?
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Jeremy //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> (to Steve) When doing these “tweaks” – have you been reporting these?
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Mary //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> – why weren’t people not complying with guidelines? Why giving blood when no indication to do so?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Not easy – reporting mechanism exists outside of electronic health record – any action taken would be within the electronic health record.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Action planning is often not individualized
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Components might be automated…
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">These presentations show that there are many different ways of doing this.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Embedding research into routine settings – research ethics, consent etc.

//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Merrick //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: framework of pragmatic randomized trial design – must be careful not to confuse the number of subjects with the rigor of the design.

//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: patents in past data would be useful, but it would be nice to have collaboration around the issues that we want to study - informal international collaborations are possible.


 * __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">AFTER LUNCH __**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Mark Chignell’s presentation “Audit and Feedback Science: Lessons from Cathedral Building” **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Science vs. Iterative design
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Aerodynamics – Science
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Cockpit Design – Iterative Design
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">E.g., early altimeters led to “controlled flight into terrain”
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Human Factors: tools of the trade
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Guidelines and Heuristics (E.g., use a knob that is this big and requires that much torque)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Requirements Analysis (Find out what people (and the situations) require)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Conceptual Design (Based on general principles, relevant science, requirements
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Iterative Design (Prototype, test (user testing), refine)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">AF guidelines and requirements – good
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Iterative design of AF tools – Good
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">High level theory of feedback intervention – Good
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Detailed theory in specific, but important areas – Good
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Theories of everything relating to AF – may be impossible
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">RCTs – not good for complex multivariate situations
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Controlled experiments need to have fractional factorial designs that sacrifice higher order interactions to allow more factors to be considered
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Response Surface Analysis is a similar approach (regression) using continuous variables.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Yes, we can do science as well as iterative design. But there will probably never be a science of Smartphone design. Maybe we need a bit more of a design focus at this time.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan Michie on behalf of Linda Collins “Unpacking the black box”: Engineering more potent behavioural interventions to improve public heath” **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What is the black box? //Treatment Package Approach//: encourages people to put a lot of stuff in it in order to get an effect; doesn’t take into account efficientness, cost effectiveness, time effectiveness etc
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Can borrow ideas from engineers and build optimized behavioural interventions
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Resource Management Principle //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> – how engineers think; what I need to find out; what are the resources that I have
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Objective is identity 1 of the 2-3 best approaches and choose efficient experimental design
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Start with the resources you already have
 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Multiphase Optimization Strategy (MOST) //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">– a framework and not a procedure
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Optimized – “best possible solution given constraints”– this is the goal that you want to achieve.
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Many components that we can study – idea is to have a factorial design with combinations and come up with a fraction intervention design. Draw on a theory ….
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Is the intervention the best possible given the constraints vs. evaluation (statistically significant)?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">IF= not best possible given constraints + not statistically significant =optimize using effect size as criterion; IF= not best possible + statistically significant = intervention can probably be improved; IF = best possible given constraints + not statistically significant = different intervention strategy needed; //IF= best possible + significant = what to aim for//
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">This approach puts more emphasis and strategic approach into the early stages of intervention design.


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Discussion: **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How do you know if the intervention is the ‘best possible’? You start off with your constraints (time, resources etc) and make a judgement about what can be achieved within that constraint. (objective function data)


 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Anne Sales //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: we don’t know the effect size of different variations of the component parts; how do you get those effect sizes?


 * //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Susan Michie //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">: have 3 approaches. 1.) kitchen sink approach – don’t do that; 2.) do many studies – takes a lot of time, misses the point of complexity about synergistic interaction between components 3.) this approach – accepting the idea of complexity and using empirical knowledge, theoretical principles etc


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Lets you rule out certain pairs of components that are of no value, not feasible etc.

//<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Robbie Foy & Anne Sales Group // <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">In thinking about how to prioritise these, we might use a basic framework/set of criteria, i.e., //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Identified themes to explore: // //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Prototypical: //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Components of AF //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Discretionary: //<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> Action plans, modes of delivery __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Assumptions: __
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What sorts of __extrinsic factors__ need to be considered in the design and evaluation of AF? **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Is there a good **evidence base** or distinct lack of an evidence base for considering or measuring this factor in a study?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Is there a good **theoretical base** or underpinning rationale for considering or measuring this factor in the study
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What practical factors or **feasibility** issues need to be considered?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">To what degree can these factors be sufficiently **well defined, understood and operationalized** in practice, studies and reviews?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How **costly** is it to implement/measure; how parsimonious is the intervention with this included or factored in?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Other general questions to consider: (for “what should happen next” paper) **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What external conditions are likely to optimise our ability to design and evaluate AF or enable us to make an optimal decision about whether or not to use AF?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Can we decide which sorts of external factors should ideally be reported to inform judgments about generalizability or feasibility of studies?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What do evidence users need to know?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">By extrinsic factors, we refer to: **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Context (probably includes several of the following…)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Characteristics of recipients
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Characterizes of deliverers
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Characteristics of targeted behaviour/topic characteristics
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Characteristics of potentially available audit measures
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Co-interventions (always being mindful of cost)
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Organizational and/or setting characteristics
 * __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Mechanisms of Action __****<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> – Jill Francis’s Group **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">How theory is ranked or grouped in a way that it can be used as resources– how do you build those theories
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Level of interventions and what there targets are
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">A need to explain variability and impact of interventions
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Are there generalizable outcomes of AF – not so convinced of this
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">* A need to create a directory of resources to inform design of interventions to structure problem definition and solution design
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">There may be a role for design and evaluation manual – so how do you design them and how do you evaluate them?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Theory itself is a problem – There are career incentives for theory fiddling
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Are audit and feedback two separate or linked processes?
 * __<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Intrinsic Factors __****<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> – Susan Michie’s Group **
 * 1) **<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What is an intrinsic factor? **
 * 1) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Should only be used for an issue that is regarded as important
 * 2) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Interactions between co-interventions and how intrinsic factors work
 * 3) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Minimal standard for feedback – (comparison data, feedback of recent performance, include written/visual feedback)
 * 4) **<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Criteria for selecting components **
 * 5) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Empirical findings
 * 6) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Theory/principles from relevant literature/disciplines
 * 7) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Dominant beliefs/Frequent current practice
 * 8) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Practitioner experience
 * 9) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Cost
 * 10) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Current priorities and policy
 * 11) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Current practice and possibilities (e.g., quality of data, measures)
 * 12) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Feasibility and screening work
 * 13) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Designs to:
 * 14) **<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Testable hypotheses – //Areas that may generate successful hypotheses// **
 * 15) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Level of aggregation of feedback data should be aligned to performance to be changed – individual vs. practice/team
 * 16) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Complexity/nature of feedback (e.g., number of indicators reported on; number of behaviours; bundles; order/priority of indicators; tailoring)
 * 17) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Format of feedback
 * 18) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Mode of delivery of feedback – e.g., paper, electronic, social
 * 19) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Frequency of feedback
 * 20) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Time to reflect/act on feedback
 * 21) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Engagement in design and delivery of feedback vs. top down imposed
 * 22) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Role for intermediate outcomes/process measures
 * 23) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Providing information/commentary alongside the feedback to add to/interpret the feedback
 * 24) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Including action plans- type e.g., if – then
 * 25) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Training of recipients to understand and act on feedback
 * 26) **<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Constraints/considerations **<span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> – **criteria to use to select what to study**
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Translate activity into evidence
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Maximise number of components to test within resources
 * 1) <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Sample size – collaborations across groups/sharing data etc.


 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Jeremy’s closing summary: **
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Current state of the science about how we use AF is problematic
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Repeat studies that don’t let us advance our knowledge base
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">New from today - examples of how AF is used; a lot of variability
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Need to keep the collaboration going
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Starting points of a definition of AF; prototypical elements of AF; common co-interventions
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">When is AF useful? How do we optimise it under those circumstances?
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Recognition that there are modifiers – need to understand them
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Looking in terms of extrinsic/intrinsic was useful; how to prioritise – using current empirical knowledge, using theories from a broad range of disciples, practical issues
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Framework about how to prioritise
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">“Intrinsic group” – hypotheses -
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">“Mechanisms group” – develop and test both program and mid level theory
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;"> How do we advance the field? Collaborative thinking, embracing the different disciples