On evidence in HTA: Bringing patient, caregiver and community experience in

Taking account of what can’t be counted: The place of qualitative evidence in HTA

Fiona A. Miller, PhD @FionaAliceMillAssociate Professor, IHPME, THETACADTH SymposiumApril 11, 2016, Ottawa

2

3

What is evidence?

• Origins (Upshur, 2001)

– Middle English: via Old French from Latin evidentia, from evident- 'obvious to the eye or mind' (Oxford)

• Definitions (Upshur, 2001)

– something that makes plain or clear; an indication or sign (Dictionary.com)

– something that furnishes proof (Merriam-Webster)• Facts alone are not evidence (Madjar & Walton, 2001)

– “facts can only become evidence in response to some particular question” (Chandler et al, 1994)

4

What questions might HTA need to answer?

Describing/ Characterizing• Patients

– The meaning and experience of illness

• Technologies– The meaning and

experience of current and proposed technologies

• Services/ Systems– The arrangement and

operation of current and proposed services

Interrogating/ Explaining• Patients

– How social, cultural and economic contexts condition illness

• Technologies– How social, cultural and

economic contexts condition the effects of technology

• Services/ Systems– How services and systems

condition experiences and outcomes

– What makes some interventions work better than others

5

Answering “other” questions: The need for “other” evidence

• From and about persons– ‘knowledge in the possession of people’ (JBI, 2014)– Physically, socially, culturally embodied and embedded

• Only partially captured in the clinical and economic evidence traditionally used in HTA

6

Sources of “other” evidence

Research-based• Often “qualitative”– Interviews, focus groups– Ethnographic

observation– Questionnaires,

documents• Descriptive• Interpretive• Theory

Opinion-based• Patient experts - Expert

opinion– Input– Participation– Deliberative engagement

• Descriptive• Interpretive• Judgment

Often seeks to “give voice”

May use research-based methods

7

ON RELEVANCE AND RIGOURThe quality of “other” evidence

8

Relevance

• On relevance– Being ‘pertinent to…’, ‘bearing upon…’, ‘connected with…’,

or ‘appropriate to…’, ‘…the matter at hand’, as well as ‘germane’, ‘apropos’, ‘material’, and ‘applicable’ (Dobrow et al, 2015)

• Opinion has a priori relevance– Local, specific• (And other functions beyond the probative)

• Research has complex relevance – Aggregative function – persistence of themes– Configurative function – unanticipated connections

between themes (Sandelowski et al, 2012)– Comparative function – the contingency of current practice

9

Rigour

• On rigour– The quality of being extremely thorough, exhaustive, or

accurate … meticulous, thorough, careful, diligent, scrupulous, exact, precise, correct

• A function of act and actor– Appropriate– Expert– Trustworthy

10

Critical appraisal tools• QARI, JBI– Critical appraisal of

qualitative evidence

• NOTARI, JBI– Critical appraisal of text

and opinion based evidence

11

QARI Critical Appraisal Criteria (JBI, 2014)

• 1. Congruity between the stated philosophical perspective and the research methodology

• 2. Congruity between the research methodology and the research question or objectives

• 3. Congruity between the research methodology and the methods used to collect data

• 4. Congruity between the research methodology and the representation and analysis of data

• 5. There is congruence between the research methodology and the interpretation of results

• 6. Locating the researcher culturally or theoretically • 7. Influence of the researcher on the research, and vice-versa, is addressed • 8. Representation of participants and their voices • 9. Ethical approval by an appropriate body • 10. Relationship of conclusions to analysis, or interpretation of the data

12

NOTARI Critical Appraisal Criteria (JBI, 2014)

• 1. Is the source of opinion clearly identified? • 2. Does the source of opinion have standing in the field of

expertise? • 3. Are the interests of patients/clients the central focus of the

opinion? • 4. Is the opinion’s basis in logic/experience clearly argued? • 5. Is the argument that has been developed analytical? Is the

opinion the result of an analytical process drawing on experience or the literature?

• 6. Is there reference to the extant literature/evidence and any incongruence with it logically defended?

• 7. Is the opinion supported by peers?

13

Rigour

• On rigour– The quality of being extremely thorough, exhaustive, or

accurate … meticulous, thorough, careful, diligent, scrupulous, exact, precise, correct

• A function of evidence systems (Pang et al, 2003)

– Stewardship– Financing: securing and allocating funds– Creating and sustaining resources– Producing, utilizing and synthesizing evidence

14

Health “evidence” systems (Pang et al, 2003)

15

In conclusion …

• There are questions that cannot be answered without “other” evidence– Concerning the embodied and embedded ways in which persons

understand and experience illness and its management, and how services and systems do and could condition these realities

• Such evidence can be generated through research (e.g., qualitative research) and/or expert opinion– Which can be more or less relevant– And more or less robust

• The quality of such evidence is a function– Of acts and actors– And of evidence systems

• If we need this evidence, we need systems to support its quality and use

16

References

• Upshur, RE. (2001). The status of qualitative research as evidence. Morse, Janice M., Janice Swanson, and Anton J. Kuzel. (Eds.) The nature of qualitative evidence. Sage, 2001: 5-26

• Madjar, I & Walton, JA. (2001). What is problematic about evidence? Morse, Janice M., Janice Swanson, and Anton J. Kuzel. (Eds.) The nature of qualitative evidence. Sage, 2001: 5-26

• Chandler, JK., Davidson, AI., Harootunian, HD. (1994) Editor’s Introduction, Questions of evidence: Proof, practice, and persuasion across the disciplines. Chicago: University of Chicago Press, 1-8

• The Joanna Briggs Institute. (2014) Joanna Briggs Institute Reviewers’ Manual: 2014 edition. The Joanna Briggs Institute

17

References

• Sandelowski, M., Voils, C. I., Leeman, J., & Crandell, J. L. (2012). Mapping the mixed methods–mixed research synthesis terrain. Journal of mixed methods research, 6(4), 317-331.

• Pang, T., Sadana, R., Hanney, S., Bhutta, Z. A., Hyder, A. A., & Simon, J. (2003). Knowledge for better health: a conceptual framework and foundation for health research systems. Bulletin of the World Health Organization, 81(11), 815-820.

• Dobrow, MJ., Miller, FA., Frank, C., Brown, AD., (2015) Understanding Relevance of Health Research: Considerations in the Context of Research Impact Assessment, Commissioned by the Ontario SPOR Support Unit, IHPME White Paper Series

18

Questions?