Measuring Outcomes to Evaluate Dementia Programs and Interventions
Strategies, Challenges, and Benefits
Michael Lepore, PhDStephanie Hughes, MPP
Elizabeth Blount, MA
Part of the National Alzheimer’s and Dementia Resource Center webinar series sponsored by the Administration for Community Living.
www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute.
Measuring Outcomes to Evaluate Dementia Programs and Interventions
Strategies, Challenges and Benefits
Michael Lepore, PhDStephanie Hughes, MPP
2
National Alzheimer’s and Dementia Resource Center Technical Assistance Webinar
January 30, 2019
Overview of Learning Objectives
• Identify multiple methods for collecting outcome data
• Describe risks of using outcome data collected from proxies instead of from people with dementia
• Describe the benefits of using validated outcome measures
• Provide resources for identifying validated measures
• Describe considerations in selecting a measure(s)
What are Measurable Outcomes?
A “measurable outcome” is an observable end-result that describes how a particular intervention benefits consumers
– It may demonstrate functional status, mental well-being, knowledge, skill, attitude, awareness or behavior
Outcomes from projects targeting individuals living with dementia and their caregivers include, but are not limited to:
– change in quality of life of the person living with dementia
– change in unmet needs
– change in stress and/or burden
– change in caregiver self-efficacy
– change in the responsiveness or cost-effectiveness of a service delivery system
A measurable outcome is different than a measurable “output”, such as: the number of clients served; the number of training sessions held; or the number of service units provided.
Why Measure Outcomes
Outcome measurement results can be used to:
– Decide if you should continue doing what you are doing
– Decide whether and how to change what you are doing
– Encourage participation and support from others, including program participants and funders
Sample Outcome Measure
6
Target population
Intervention Output Goal Outcome Goal Outcome Measure
Outcome Measurement
Persons with dementia who live alone
Care of People with Dementia in their Environments (COPE)
COPE provided to 100 persons with dementia who live alone
Improve quality of life of persons with dementia who live alone
Quality of Life in Alzheimer’s Disease (QOL-AD)
Implement QOL-AD pre-COPE and 30-day post-COPE
Common Outcome Measurement Methods (1/2)
Surveys/Questionnaires: Ask participants questions and record their answers
– Questions may be open- or closed-ended Closed-ended questions (e.g., yes/no) provide quantitative data Open-ended questions usually generate qualitative data, but answers can
be coded to yield quantitative data
Observation: Directly observe operations and activities– Useful when the required information can be obtained by watching and
listening– Can answer questions about whether a program is being delivered and
operated as planned and can better understand both the situation and context
Interviews: Conversation between interviewer and person who can provide input on issue being evaluated
– Follow a script or guide, which may include probing questions – Questions are usually open-ended and data are usually qualitative, but
quantitative data may be collected
Common Outcome Measurement Methods (2/2)
Focus groups: Small-group discussions – Moderator uses a question guide to introduce topics of interest and
probe for discussion– Data are usually qualitative but a short quantitative survey can be
conducted as part of the focus group to obtain quantitative information
Secondary data: Use existing data sources (e.g., medical records)– Abstract outcome data from secondary data sources – Data are usually quantitative and standardized but not tailored to the
evaluation
Capacity to Participate in Data Collection (1/2)
Some people with dementia, such as those in early to moderate stages of Alzheimer’s disease, have capacity to participate in data collection (Karlawish, 2008)
However, dementia is progressive, so capacity of people with dementia to participate declines over time (Moye, Karel, Gurrera, & Azar, 2006)
– Whether an individual is capable of decision making depends on whether he or she can communicate relatively consistently, understand basic information about choices, evaluate the implications of different choices, and rationally comprehend the risks and benefits associated with different options (Karel et al., 2010; Woods & Pratt, 2005).
– Because these capacities fluctuate over time and different decisions require different levels of capacity, a diagnosis of dementia—or even a specific score on a cognitive test—may not be the only criterion for determining decision-making capacity (Woods & Pratt, 2005).
Capacity to Participate in Data Collection (2/2)
There is no “gold standard” for determining capacity (Mitty, 2012)
Several decisional capacity assessment tools exist that can aid researchers in determining capacity among people with dementia (Cacchione, 2011).
– Evaluation to Sign Consent (Resnick et al., 2007)
– Hopemont Capacity Assessment Interview (Edelstein, 1999)
– Capacity to Consent to Treatment Interview (Marson, Ingram, Cody, & Harrell, 1995)
– MacCarthur Competence Assessment Tool-Treatment (Grisso & Applebaum, 1998).
Limited decisional capacity among individuals with dementia also may be indicated by difficulties with naming and delayed memory (Moye et al., 2006)
Proxies: Pros and Cons
Engaging proxies, such as caregivers or family members, is a valuable option for data collection when the person with dementia does not have capacity to participate or provide informed consent
– However, proxy decision-makers commonly make research enrollment decisions based on what they think will maximize the well-being of the person with dementia, not a substituted judgment of what the person with dementia would decide (Karlawish et al., 2008)
– Data provided by proxies commonly differ from data provided by people with dementia (Clarke & Keady, 2002; Sands, Ferreira, Stewart, Brod, & Yaffe, 2004).
People with dementia have been found to have higher hopes for their quality of life than their caregivers do for them (Thorgrimsen et al., 2003)
Selecting an Appropriate Outcome Measure
Does it measure the right outcome(s)?
Is it reliable?
Is it valid?
What are data collection considerations?
Measuring the Intended Outcomes
• Stakeholder agreementDevelop a clear logic
model
• Choose one or more outcome(s)
• Subtle variationsGet specific
• Intensity of intervention• Timing of data collection
Confirm the outcome(s) is realistic
Reliable and Valid Measures
• Reliability – “The extent to which a measure produces the same results when used repeatedly to measure the same thing”
• Validity – “The extent to which a measure actually measures what it is intended to measure.”From Evaluation: A Systematic Approach by Rossi, Lipsey and Freeman
Valid but not reliable
Reliable but not valid
Measures the intended outcome
Saves time
Adds credibility to results
May allow some comparability
Benefits of Using Existing Validated Measures
Obtaining Validated Measures
• National Alzheimer’s and Dementia Resource Center• Evaluating Dementia Services and Supports: Instrument Resource
Listhttps://nadrc.acl.gov/node/70
• 73 instruments• 25 domains – e.g. burden, formal supports, cognitive
functioning, risk assessment, unmet needs, behavioral symptoms, quality of life
• Easy to access• Scoring, administration• Permissions
Considerations in Selecting a Measure
• Who is collecting the data?• What level of expertise is needed?• What training is needed?
• Who is responding?• Person with dementia, caregiver, professional • Cognitive capacity
• Other considerations• Mode – Phone, in person, electronic, paper• Burden – Number of questions, amount of time required• Flow – When and how it will fit into the intervention
process
Frequency of Data Collection
Frequency Typical Use Pros ConsOne time Very brief
interventions Least resource intensive
Little respondent time required
Respondent self-reporting introduces bias
Pre/post Training sessions, measuring knowledge change
Allows measure of change
Relatively little time required
Only measures immediate impact
Pre/post with follow-up
More intensive interventions that impact well-being over time or involve practice change
Allows measure of impact over time
Resource-intensive
References Cacchione, P.Z. (2011). People with dementia: Capacity to consent to research participation. Clinical Nursing Research, 20, 223-
227. Clarke, C.L, & Keady, J. (2002). Getting down to brass tacks: A discussion of data collection with people with dementia. In H.
Wilkinson (Ed.), The perspectives of people with dementia: Research methods and motivations (pp. 25-46). London, UK: Jessica Kingsley Publishers.
Edelstein, B. (1999). Capacity assessment interview manual and scoring guide. Morgantown, WV: West Virginia University Press. Grisso, T., & Applebaum, P.S. (1998). Assessing competence to consent to treatment: A guide for physicians and other health
professionals. New York, NY: Oxford University Press. Hughes, S. J., Walberg, D., Lepore, M., Gould, E., & Walsh, E. G. (2018). Evaluating dementia services and supports: Instrument
resource list. Administration on Aging, Administration for Community Living. Karlawish, J. (2008). Measuring decision making capacity in cognitively impaired individuals. Neurosignals, 16, 91-98. Karel, M.J., Gurrera, R.J., Hicken, B., & Moye, J. (2010). Reasoning in the capacity to make medical decisions: The consideration
of values. Journal of Clinical Ethics, 21(1), 58-71. Marson, D. C., Ingram, K. K., Cody, H. A., & Harrell, L. E. (1995). Assessing the competency of patients with Alzheimer’s disease
under different legal standards. Archives in Neurology, 52, 949-954 Mitty, E.L. (2012). Decision-making and dementia. The Hartford Institute for Geriatric Nursing, New York University, College of
Nursing. Moye, J., Karel, M.J., Gurrera, R.J., & Azar, A.R. (2006). Neuropsychological predictors of decision-making capacity over 9
months in mild-to-moderate dementia. Journal of General Internal Medicine, 21, 78-83. Resnick, B., Gruer-Baldani, A.L., Pretzer-Aboff, I., Galik, E., Buie, V.C., Russ, K., & Zimmerman, S. (2007). Reliability and validity
of the Evaluation to Sign Consent Measure. The Gerontologist, 47, 69-77. Rossi, P., Lipsey, M., & Freeman, H. (2004). Evaluation: A Systematic Approach. Thousand Oaks, CA: Sage Publications, Inc. Sands, L.P., Ferreira, P., Stewart, A.L., Brod, M., & Yaffe, K. (2004). What explains differences between dementia patients’ and
their caregivers’ ratings of patients’ quality of life? The American Journal of Geriatric Psychiatry, 12(3), 272-280. Thorgrimsen, L., Selwood, A., Spector, A., Royan, L., de Madariaga Lopez, M., Woods, R.T., & Orrell, M. (2003). Whose quality
of life is it anyway? The validity and reliability of the Quality of Life-Alzheimer’s Disease (QoL-AD) scale. Alzheimer Disease & Associated Disorders, 17(4), 201-208.
Woods, B., & Pratt, R. (2005). Awareness in dementia: Ethical and legal issues in relation to people with dementia. Aging andMental Health, 9(5), 423-429.
More information
Michael Lepore, [email protected]
Stephanie Hughes, [email protected]
National Alzheimer’s and Dementia Resource Center, nadrc.acl.gov Evaluating Dementia Services and Supports: Instrument
Resource List https://nadrc.acl.gov/node/70
Measuring Outcomes to Evaluate Dementia Programs and Interventions: Strategies, Challenges, and Benefits
Elizabeth BlountJanuary 30, 2019
• Evidence-based information, coaching and support program for adults with chronic health conditions and their caregivers
• Telephonic/e-mail delivery method
• Personalized and tailored coaching
• Provides: ongoing support, help finding practical solutions to present concerns, future planning/prevention, and adjusts to changes in care
• Service delivery period determined by licensed site
BRI CARE CONSULTATION: PROGRAM OVERVIEW
Key Components of Program
• Initial Assessment: gradual identification of present or future concerns, both medical and non-medical
• Action Plan: evolving set of simple and achievable tasks to address concerns step-by-step, begins in first call
• Maintenance and Support: regularly maintained contacts to monitor care situation (more frequent as needed) and respond to any changes
BRI CARE CONSULTATION: PROGRAM OVERVIEW
Program Development History
• Nine different research demonstrations/projects over 2 decades
• Program demonstration in Georgia:– 2010-2013– Funded by Administration on Aging/Administration for Community
Living– 3 phases:
• Original 3 Year Grant Demonstration: 2010-2013 • Atlanta Regional Commission (ARC) Pilot Project: 2013-2014 • BRI Care Consultation Statewide Expansion: 2013-present
BRI CARE CONSULTATION: PROGRAM OVERVIEW
• Challenges identified by Georgia Division of Aging Services (GDAS) and Area Agencies on Aging (AAAs) with Georgia Alzheimer’s Disease or Related Dementias (ADRD) families:
– Growing waiting lists – Underutilized formal services – Programs serving both caregiver and ADRD care recipient– Effective service abilities using face-to-face methods
• BRI Care Consultation program chosen because:– High efficiency level: cost effectiveness of telephonic model – Ability to contact care consultant – Care Consultation Information System (CCIS) guided program
delivery – Effective in wide range of agencies
PROGRAM IMPLEMENTATION HISTORY IN GEORGIA
• Program delivered at three AAAs: – Atlanta Regional Commission – Heart of Georgia Altamaha– Legacy Link
• Eligibility criteria for program participation included:– Care receivers with/without a formal dementia diagnosis– Care receiver living in the community, with a primary caregiver
(living in the home or elsewhere) – Care receiver living in community without an identified primary
caregiver– Ability to read and speak English
ORIGINAL GRANT DEMONSTRATION
• Instruments utilized to measure outcomes same as original studies that tested efficacy of program
• Measure outcomes established for project:
– Enrollment– Attrition– Caregiver Depression – Caregiving Confidence or self-
efficacy – Caregiver Social Isolation – Caregiver Health Strain
– Caregiver Needs Met-including total needs met and eight categories of needs, including:• Informal help• Dementia diagnosis• Daily tasks• Respite • Legal/financial • Support• Service access• Living arrangements
ORIGINAL GRANT DEMONSTRATION: OUTCOMES
• Results:– Enrollment and Attrition:
• Enrolled- exceeded overall goal• Attrition-slightly higher than national average
– Depression:• Decrease in caregivers of persons with high behavior
problems– Caregiver Confidence or self-efficacy:
• Increase in total sample and caregivers of persons with high behavior problems
ORIGINAL GRANT DEMONSTRATION: RESULTS
• Results:– Social Isolation
• Decrease in caregivers of persons with high behavior problems and high cognitive impairment
– Health Strain• Decrease in caregivers of persons with high behaviors
problems– Needs Met
• Increase in total sample
ORIGINAL GRANT DEMONSTRATION: RESULTS
• Goals identified for expansion:– Maximize the use limited resources– Enhance sustainability of aging network through fee-for-
service options– Infuse evidence-based programs throughout the aging
network – Improve coordination between formal and informal care– Improve the focus on prevention and person-centered
care– Improve availability of services in rural areas
STATEWIDE EXPANSION: GOALS
• Continued scaling of the program in Georgia:• Implementation Sites:
• 9 direct, 3 referring • Ongoing Training and Technical Assistance Provided:
• RCI: providing additional, comprehensive, tailored • BRIA: overall, part of standard licensing agreement
• Families Served:• No longer serving only ADRD families
• Private Pay/Fee-for-Service Model Development • Data Collection:
• Continually through Care Consultation Information System (CCIS)
STATEWIDE EXPANSION: WHERE WE ARE NOW
Bass, D. M., Clark, P.A., Looman, W.J., McCarthy, C.A., & Eckert, S. (2003). The Cleveland Alzheimer’s managed care demonstration: Outcomes after 12 months of implementation. Gerontologist, 43(1), 73-85.
Bass, D., Easom, L., Primetica, B., & Holloway, C. (2015). Reflections on Implementing the Evidence-Based BRI Care Consultation with RCI in Georgia. Generations, 39(4), 49–56.
Clark, P. A., Bass, D. M., Looman, W.J., McCarthy, C.A., & Eckert, S. (2004). Outcomes for patients with dementia from the Cleveland Alzheimer’s Managed Care Demonstration. Aging and Mental Health, 8(1), 40-51.
Easom, L. & Holloway, C. (2015). Georgia BRI Care Consultation (Unpublished grant final report). Rosalynn Carter Institute for Caregiving, Americus, GA.
Kohout, F.J., Berkman, L.F., Evans, D. A., et al (1993). Two shorter forms of the CES-D (Center for Epidemiological Studies Depression) depression symptoms index. J Aging Health, 5:179-93.
Sterling, E. W. (2014). An Analysis of Case Management and Care Consultation in the Atlanta Area (Unpublished report). Sterling Results, LLC, Marietta, GA.
REFERENCES
Social media:
@rcicaregiving
@benrose1908
@georgiadepartmentofhumanservices
Website:
www.rosalynncarter.org
www.benrose.org
https://aging.georgia.gov
KEEP UP WITH US
Thank You!
Contact information
Michael Lepore, [email protected]
Stephanie Hughes, [email protected]
Elizabeth Blount, [email protected]
Top Related