Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health...
-
Upload
jonathan-weeks -
Category
Documents
-
view
216 -
download
0
Transcript of Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health...
![Page 1: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/1.jpg)
Scoring Goals or Changing Scoring Goals or Changing the Game:the Game:
What Impacts Should We What Impacts Should We Measure?Measure?
Jonathan LomasCanadian Health Services Research
Foundation Presentation to ESRC Symposium: ‘New
Approaches to Assessing the Non-Academic Impact of Social Science’London, May 12-13, 2005
![Page 2: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/2.jpg)
My question is, are we having an impact?
![Page 3: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/3.jpg)
Is Research Ready for Action?Is Research Ready for Action?Medline search (1966-02] to identify articles stating:
a. “need more research” or “need less research” Need more 161/162
Need less 1/162
b. “more questions than answers” or “more answers than questions”
More questions 163/166
More answers 3/166David, AS. BMJ 2002; 323:1462-3
![Page 4: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/4.jpg)
Assessing the Impact of What?Assessing the Impact of What?The Research Produced?
A single research study published in a journalA summary of some research studies written in plain language and posted on a web siteA systematic review with targeted dissemination of key messages to potential usersA body of research knowledge developed and discussed face-to-face with potential users
![Page 5: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/5.jpg)
Assessing the Impact of What? Assessing the Impact of What? (cont)(cont)
The Research Production Process?All the activities of a research commissioning or granting agency (including training)All the activities of a research production facility (e.g. institute, department, university)The activities of a potential research user organization and its staffThe entire ‘research regime’ in a country
![Page 6: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/6.jpg)
Where I WorkWhere I Work
CHSRF’s mission:
“To support evidence-based decision-making in the healthcare system”
Our ultimate desired impact is cultural change in the research and healthcare systems
Changing the game, not just scoring goals
![Page 7: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/7.jpg)
The Discipline of Objectives & Logic The Discipline of Objectives & Logic ModelsModels
Yogi Berra on objectives:“If you don’t know where you’re going, you might not get there”
Lewis Carroll on logic models:“If you don’t know where you’re going, any road will take you there”
![Page 8: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/8.jpg)
Decision MakersPolicy Makers
Organized Interests e.g. drug companies,
professional associations
Service Professionals
Managers
‘Client’ & Public
Decision Maker Diversity
![Page 9: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/9.jpg)
Why Social Science isn’t IBM or General Electric
ResearchersUniversity-
based
Stakeholder-based
System-based
Management Consultants
Policy Makers
Organized Interests e.g. drug companies,
professional associations
Service Professionals
Managers
Patient & Public
Decision Makers
PR
OB
LEM
S
SO
LUTIO
NS
![Page 10: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/10.jpg)
Decision Makers
Researchers
Evidence-Based Decision-Making
Research Funders
Knowledge Purveyors
Receptor Capacity
Critical Evaluation P
RO
BLE
MS
SO
LUTIO
NS
Funding and Training Vehicles
Priority-setting structures
ISSUES & PRIORITIES
PRIORITY
TOPICS
IDEAS
RESEARCH
EVIDENCE
Other InfluencesPersonal Experience,
Anecdote, Wants, Interests, Myths, Assumptions, etc.
Synthesis and Influence
Linkage and
Exchange
![Page 11: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/11.jpg)
CHSRF’s ObjectivesCHSRF’s Objectives
1. To increase health system decision-makers’ appreciation of the value of research
2. To increase the production of research relevant to the needs of health system decision- makers
3. To increase the availability and acquisition of needed research by health system decision-makers
4. To increase the appraisal and application of needed research by health system decision makers
![Page 12: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/12.jpg)
Increased Appreciation of Value of Increased Appreciation of Value of ResearchResearch
Programs and activities:overall ‘linkage & exchange’ approachconsultations with users for prioritiescase study presentations of value
Measurable outcomes:# of decision-makers participating in researchexpenditures on research commissioning by system% of employees in system with research training
![Page 13: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/13.jpg)
Increased Production of Relevant Increased Production of Relevant ResearchResearch
Programs and activities:priority theme-based program-fundingcommissioned syntheses on current issuesapplied training programsencourage university incentives for applied research
Measurable outcomes:amount of research in priority theme areasself-reported awareness/use of research syntheses# of graduates with applied research skills
![Page 14: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/14.jpg)
Increased Availability/Acquisition of Increased Availability/Acquisition of ResearchResearch
Programs and activities:plain-language research summaries (1:3:25, Mythbusters)face-to-face exchanges on timely topicscreation/support of knowledge networkscreation/support for knowledge brokering
Measurable outcomes:self-reported awareness of disseminated researchself-reported follow-up contact with researchersself-reported use of web-based and other resources for research evidence acquisition (audit computer bookmarks)
![Page 15: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/15.jpg)
Increased Appraisal/Application of Increased Appraisal/Application of ResearchResearch
Programs and activities:training users in research appraisal and applicationorganisational best practices in research use
Measurable outcomes:self-reported application of researchchanges in organisational structures and processes to better accommodate researchIncreased sense of ‘decision-certainty’ where synthesised research is available
![Page 16: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/16.jpg)
The Attribution ChallengeThe Attribution Challenge
What, if any, is/was the role of research versus all the other influences on behaviour?
![Page 17: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/17.jpg)
Adapted from Philip Davies, 2005
Program or Intervention Effectiveness
Types of Research Evidence
Implementation Evidence
Organizational Evidence
Economic/ Financial Evidence
Ethics Evidence
Forecast Evidence
Attitudinal Evidence
ExperimentalQuasi-Experimental
Counterfactual
SurveysAdmin DataComparativeQualitative
Cost-BenefitCost-Effectiveness
Cost-UtilityEconometrics
ExperimentalQuasi-Experimental
QualitativeTheories of Change
Public ConsultationDistributional Data
MultivariateRegression
SurveysQualitative
![Page 18: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/18.jpg)
Adapted from Philip Davies, 2005
Types of Research Evidence
Program or Intervention Effectiveness
Implementation Evidence
Organizational Evidence
Economic/ Financial Evidence
Ethics Evidence
Forecast Evidence
Attitudinal Evidence
ExperimentalQuasi-Experimental
Counterfactual
SurveysAdmin DataComparativeQualitative
Cost-BenefitCost-Effectiveness
Cost-UtilityEconometrics
ExperimentalQuasi-Experimental
QualitativeTheories of Change
Public ConsultationDistributional Data
MultivariateRegression
SurveysQualitative
ResearchEvidence
![Page 19: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/19.jpg)
Adapted from Philip Davies, 2005
Combining Research and Colloquial Evidence for Information
ResearchEvidence
Professional Experience &
Expertise
Political Judgement
Resources
ValuesHabits & Tradition
Lobbyists & Pressure Groups
Pragmatics & Contingencies
![Page 20: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/20.jpg)
A Final Zany IdeaA Final Zany Idea
The ‘impact file’ - a deductive approach to assessing impact
Add photo of impact file
![Page 21: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/21.jpg)
What’s In the Impact File?What’s In the Impact File?
T F H M & Y D G W - testimonialsAltered career trajectory (researcher or decision-maker)Changes in similar organization’s programs & processes - ‘lateral impact’Better communication of research and its implications - disseminationAwareness of research by decision-makers - acquisitionChanged decisions based on research – applicationChanges in researchers’ or decision-makers’ structures and processes - ‘cultural change’
![Page 22: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/22.jpg)
Maybe Settling for Second Best Maybe Settling for Second Best is OK!is OK!Using an impact file is:
Motley, ad-hoc and not comprehensive or systematicBiased to individual not organizational responsesQualitative and potentially non-generalizable
But, as Churchill said of democracy:“Many forms of government have been tried … No one pretends that democracy
is perfect … Indeed, it has been said that democracy is the worst form of government, except all those other forms that have been tried from time to time.” House of Commons, 1947
![Page 23: Scoring Goals or Changing the Game: What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium:](https://reader036.fdocuments.net/reader036/viewer/2022070306/5515fe4455034694308b4aa3/html5/thumbnails/23.jpg)
THANK YOU!THANK YOU!
www.chsrf.caor
www.fcrss.ca