Psychometric Properties of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group Adult Visit Survey

September 11, 2012Naomi Dyer, PhD, Joann Sorra, PhD, Scott Smith, PhD, Westat

Paul Cleary, PhD, YaleRon Hays, PhD, RAND

CAHPS is a multiyear initiative of AHRQ to support and promote the assessment of consumers’ experiences with health care

Numerous CAHPS surveys have been created for different health care settings such as :

Health Plan Surgical Care Home Health Care Hospital In-center Hemodialysis Nursing Home Clinician & Group

Background

Westat, under contract with AHRQ, houses the database for Clinician and Group CAHPS (CG-CAHPS) and reports on these data

Background

There are several CG-CAHPS Surveys:

CG-CAHPS 12-month Survey (Adult and Child)

CG-CAHPS 12 month Survey with Patient-Centered Medical Home

CG-CAHPS Visit Survey (Adult and Child)

CG-CAHPS Database is the newest component of the CAHPS Databases

The CG-CAHPS Adult Visit Survey database received the most data in 2011

Background

The CG-CAHPS Visit surveys were created to allow patients to report on and evaluate their experiences during their most recent outpatient medical visit

The 12 month versions ask about their experiences in the previous year

Surveys and related material available at:

http://cahps.ahrq.gov/clinician_group/

Background

Goal: Develop a survey to assess patients' experiences with medical groups and clinicians.

Development of the Visit Survey began in 2008

Field testing in 2009 to compare with the 12-month survey version

Results led to the Access to Care items in the survey reverting to a 12-month reference period rather than visit-specific

Survey Development

CG-CAHPS Composites

3 composites

1. Access to Care (5 items)2. Doctor Communication (6 items)3. Courteous/Helpful Staff (2 items)

2 overall rating items

1. Overall Doctor Rating (11-point scale; 0 = “Worst doctor possible”, 10 = “Best doctor possible”)

2. Recommend the Doctor’s Office to family and friends (3-point scale; “No” to “Yes, definitely”)

Various socio-demographics including overall health, age, gender, and education

Access to Care Composite

All 5 items reference a 12-month period

4-point scale (1 = Never, 2 = Sometimes, 3 = Usually, 4 =Always)

Access to Care Composite

In the last 12 months,

How often did you get an appointment for care you needed right away as soon as you needed?

How often did you get an appointment for routine care as soon as you needed?

When you phoned this doctor’s office during regular office hours, how often did you get an answer to your medical question that same day?

When you phoned this doctor’s office after regular office hours, how often did you get an answer to your medical question as soon as you needed?

How often did you see this doctor within 15 minutes of your appointment time?

Doctor Communication Composite

All six items reference the most recent office visit

3-point scale (1 = Yes, definitely; 2 = Yes, somewhat; 3 = No)

During your most recent visit, did this doctor…

Explain things in a way that was easy to understand?

Listen carefully to you?

Give you easy to understand information about your health questions or concerns?

Seem to know the important information about your medical history?

Show respect for what you had to say?

Courteous/Helpful Staff Composite

Both items reference the office visit

3-point scale (1 = Yes, definitely; 2 = Yes, somewhat; 3 = No)

During your most recent visit, were clerks and receptionists at this doctor’s office…

As helpful as you thought they should be?

Treat you with courtesy and respect?

CG-CAHPS Analysis Dataset

Adult Visit Survey Data from the CG-CAHPS Comparative Database 103,442 responses 469 practice sites

There were 5 screener questions that determined if the composite item was to be answered by respondents 4 screener questions for Access to Care 1 screen question Doctor Communication

Only non-missing data for the composite items were included in the analysis

CG-CAHPS Analysis Dataset

93% of respondents did not phone their doctor after regular hours

Because of the high percentage of skips, “How often did you get an appointment for care you needed right away as soon as you needed?” was dropped from analysis

Remaining 4 Access to Care composite items had responses between 46% to 98% of respondents

Final CG-CAHPS Analysis Dataset 450 practice sites 21,318 responses

Characteristics of Analysis Dataset

All respondents received paper/mail surveys

89% of sites were Family Practice or Internal Medicine

69% of sites were owned by a hospital or integrated delivery system

67% of respondents were female;

81% were 45 years or older

Psychometric Analyses and Criterion

Individual and Multilevel Confirmatory Factor Analysis (CFA) Factor loadings above 0.40 Acceptable model fit indices

• CFI > 0.95 • RMSEA < 0.06 • SRMR < 0.08 (Individual, Between- and Within-)

Individual Internal Consistency Reliability Analysis Cronbach’s α ≥ 0.70

Psychometric Analyses and Criterion

Practice Site Reliability Analysis Reliability ≥ 0.70

Examined reliability by practice size categories

1 clinician 2 to 3 clinicians 4-9 clinicians 10-13 clinicians 14-19 clinicians 20 or more clinicians

Correlation analysis among the composites and global ratings Individual and practice site levels

Individual Level CFA Results

All items within composites had individual factor loadings above 0.40 with average loadings of Access to Care: 0.68 Doctor Communication: 0.76 Courteous/Helpful Staff: 0.86

All model fit indices met criteria CFI: 0.97 RMSEA: 0.05 SRMR: 0.04

Multilevel CFA Results

All items at practice site level had factor loadings above 0.40 Between site factor loadings range: 0.59 to 0.99 Within site factor loadings rage: 0.45 to 0.99

All model fit indices met criteria except between-practice site SRMR CFI: 0.97 RMSEA: 0.03 Between SRMR: 0.10 Within SRMR: 0.05

Internal Consistency Reliability

All items at individual level had Cronbach’s alpha above 0.70

Access to Care α = 0.77

Doctor Communication α = 0.89

Courteous/Helpful Staff α = 0.85

Practice Site Reliability

Practice site reliability estimate were acceptable for all site with at least 4 clinicians

For a site with 1 clinician• Only Access to Care had reliability above 0.70

For a site with 2-3 clinicians• Access to Care and Courteous/Helpful Staff had

reliability above 0.70

The average number of respondents for 1 clinician and 2-3 clinicians was less than 100

Smaller sites need more respondents per practice to increase reliability to acceptable levels

Individual Correlations

Composites and Global Ratings 1 2 3 4

1. Access to Care

2. Doctor Communication 0.35

3. Courteous/Helpful Staff 0.29 0.25

4. Overall Doctor Rating 0.39 0.52 0.22

5. Recommend Doctor’s Office 0.34 0.52 0.29 0.47

All correlations were significant (p < 0.05)

Strongest correlation was between Doctor Communication and Overall Global Rating Items

Practice Site Level Correlations

All correlations were significant (p < 0.05)

Composites and Global Ratings 1 2 3 4

1. Access to Care

2. Doctor Communication 0.45

3. Courteous/Helpful Staff 0.57 0.41

4. Overall Doctor Rating 0.42 0.75 0.34

5. Recommend Doctor’s Office 0.52 0.76 0.43 0.76

Strongest correlation was between Doctor Communication and Recommend Doctor’s Office

Summary

Overall, both the individual level and multilevel confirmatory factor analysis results provided support for the survey’s three composites

The CG-CAHPS composites have acceptable individual-level internal consistency reliability

The reliability remains acceptable across sites with four to twenty or more clinicians

Email: naomidyer@westat.com