Richard Lennox, PhDVP of Commercial ScienceChestnut Global Partners

Measuring Outcomes in EAP and Workplace Programs: Intro of a new Suite

 

NBC EAP Group MeetingSan DiegoSeptember 9, 2009

David Sharar, PhDManaging DirectorChestnut Global Partners  

Performance measurement in EAP is more like a Rorschach test than a defined term.

 

"I need to move beyond 'counting noses' and begin to quantify the actual impact of EAP intervention to ensure our continued viability". Michael Hack, LCSWManager, Global Workplace Solutions - ConocoPhillips, Houston, Texas

Performance is in the Eye of the Beholder

• Purchaser: Are my EAP dollars being spent

wisely?

• Provider: (primary vendor or affiliate): Does good

service or results produce rewards?

• Client (employee or family member): Is the EAP

responsive to my personal needs?

State of performance measurement in EAP

• Substandard performance is largely invisible• Many measures are blunt, incomplete, distorting,

exaggerated (e.g. Utilization rates)• We lack common definitions & standard markers

of success• Under capitation, “marginal” performance

receives the same rate as “optimal”

Which results in:

• Purchasers use the one metric everyone knows: PRICE

• Purchasers and their agents lack understanding of what an EAP should accomplish

• The link between Outcomes and Price is unclear (and not known)

• So, employers buy EAP on Price and Promises

Why measure Outcomes?

• Describe the effects or results of our interventions (outcome measures)

• Improve an aspect of the process of care (process measure), which in theory leads to better outcomes

• Make comparisons across vendors or program models

• Counterforce to deflation of EAP rates

Issues with “Employer-Driven”

• Many employers don’t know a good measure from a bad one

• Lack of senior management engagement

• Over-reliance on consultants and brokers who do not understand EAP

• Competing goals between finance, HR, benefits, occupational medicine, etc.

• Performance measurement is difficult-and this difficulty is not always appreciated

Why are outcomes so difficult?

Natural Problems• Confounding factors beyond your control

• Sample size too small to produce effect

• Long delays when measuring over time

• Low frequency of interesting outcomes

Why are outcomes so difficult?

Human Problems• Inadequate information systems

• No extra funding (vendor bears cost)

• Accessing employer data

• Point of measurement complexity

• Insufficient level of clinical detail

• How does one address poor outcomes?

Criteria for selecting measures:

• Does the measure serve to enhance the productivity or well-being of employees? (e.g. is it “mission critical”?

• Is the measure based on science or opinion? (and if opinion, is there consensus)?

• Is it feasible? (resource availability, automated data collection, and statistically meaningful comparisons)

Outcome Variables

• Defined– Occur after the intervention– Caused by the intervention– Not by some other variable

• Is closely linked to the intervention– Not some spurious factor– Affect by other factors

Multiple Determinism

“The characteristic of a variable to be affected by a large number of factors in addition to the condition in question, increasing the risk that a failure to changed the variable is incorrectly attributed to a weak intervention rather than to the fact that the outcome is a poor indicator of effectiveness.”

Proximal Outcomes

• Variable central to intervention

• Linked to the specific goals of the intervention– Occur at the end of EAP intervention or

discharge from treatment

Medial Outcomes

• Quality of Life– Impact of the intervention on the client’s life

experience– Not directly related to the goals of intervention,

but not too far removed– In the real world

• At work, or home

– ADL (activities of daily living)– Also affected by other illnesses or disorders

Distal Outcomes

• Only indirectly related to the goal of intervention– Health Care use – clients is not told to avoid

using medical care

• Caused by numerous other factors

Clinical Treatment Evaluation

• Proximal Outcomes• Most important for testing efficacy• Provides the most convincing test of whether

intervention works as planned• Will have the strongest effects

– Best for establishing sample size• Best for establishing the ability to move clinical

markers.

Effectiveness Studies

• Designed to evaluate ability of EAP to affect post-clinical variables.

• Best suited for medial outcomes

• Quality of life: Does the intervention improve the clients life beyond simply reducing distress?

• Can the client function at work?

EAP Evaluation

• Not focused on efficacy• Interested in the real world at work• Can make better use of distal outcomes• Economic analysis can be linked• Smaller effects• Large sample• Secondary data

Take Home Message

• Understand the proximity of your outcome to the intervention its self.

• Incorporate multiple determinism of an outcome in the expected effect size and the statistical power.

• Don’t burden you intervention by trying to show extraneous value that will be difficult to detect in any efficacy study

CGP's New Suite of EAP Outcome Measures

• SHORT but empirically valid

• Easy to administer

• Workplace focused (5 subscales, 5 items per subscale)

• Norms with non-clinical population being established for comparisons

• FREE of charge (but we would like the data)

 

Absenteeism

caused you miss to work altogether.

made you late for work.

caused you to take off early.

pulled you away from your normal work location while still at work

required you to be on the phone, e-mail or internet while at work.

Response Key: 1= strongly disagree; 2= somewhat disagree; 3 = neutral; 4 = somewhat agree; 5 = strongly agree

Presenteeism

I had a hard time doing my work because of my personal problems.

. My personal problems kept me from concentrating on my work..

Because of my personal problems I was not able to enjoy my work.

My personal problems made me worry about completing my tasks.

I could not do my job well because of my personal problems.

Response Key: 1= strongly disagree; 2= somewhat disagree; 3 = neutral; 4 = somewhat agree; 5 = strongly agree

Work Engagement

I feel stimulated by my work.

I often think about work on my way to the work site.

I feel passionate about my job.

I am often eager to get to the work site to start the day.

I often find myself thinking about my work at home.

Response Key: 1= strongly disagree; 2= somewhat disagree; 3 = neutral; 4 = somewhat agree; 5 = strongly agree

Life Satisfaction

My life is nearly perfect.

I am not very satisfied with my life as a whole.

So far, my life seems to be going very well.

There isn’t anything about my life that I would change if I could.

I am very disappointed about the way my life has turned out.

Response Key: 1= strongly disagree; 2= somewhat disagree; 3 = neutral; 4 = somewhat agree; 5 = strongly agree

Workplace Distress

I often feel anxious at work.

Thinking about being at work makes me upset.

I am unhappy most of time at work

I dread going into work

I can’t wait to get away from work.

Response Key: 1= strongly disagree; 2= somewhat disagree; 3 = neutral; 4 = somewhat agree; 5 = strongly agree