Challenges of Screening for Disability in Surveys:Challenges of Screening for Disability in Surveys:

Example of the NSIP Survey on the Prevalence of Example of the NSIP Survey on the Prevalence of

Disability Among Active Members in National ServiceDisability Among Active Members in National Service

Challenges of Screening for Disability in Surveys:Challenges of Screening for Disability in Surveys:

Example of the NSIP Survey on the Prevalence of Example of the NSIP Survey on the Prevalence of

Disability Among Active Members in National ServiceDisability Among Active Members in National Service

Presented by:Holly Matulewicz, Research Associate

National Service Inclusion Project at the Institute for Community Inclusion

October 17, 2008

Overview of Today’s SessionOverview of Today’s Session

• Introduction to NSIP and the project

• Defining & Measuring the construct

– Why is prevalence important?– Conceptualizing and operationalizing the construct of

disability– Examples in federal & cross-national surveys– Challenges in survey administration for prevalence

• NSIP Project at ICI– Process for developing / testing CNCS instrument

Introduction to NSIP and Purpose of the ProjectIntroduction to NSIP and Purpose of the Project

• What is the National Service Inclusion Project (NSIP)?

• Training• Technical assistance

• How did this task emerge?

• What are the goals of the task?

Why is prevalence important?Why is prevalence important?

• Why would we need to know how many people have a disability in a given population?

• How might these data be used?

• What challenges would you expect in collecting these data?

ExerciseExercise

• Take 2 min as an individual and construct your definition of disability.

• Brief group sharing of ideas.

Conceptualizing Construct of DisabilityConceptualizing Construct of Disability

• Complex task! Implications of the construct - status, identity.

• Context of the definition: social, political, economic, ethical.

• No single definition of disability among large national surveys and polls (different purposes / audiences).

Evolution of the definitionEvolution of the definition

Medical Model

• Physical, mental, or psychological condition that limits a person’s activities.

• Linked to conditions.

• Problem residing in the individual.

Social Model

• Arises from interaction of a person’s functional status with the physical, cultural, and policy environments.

• Neither person nor environment specific.

• Environment should be designed to accommodate all.

Nagi’s theory on disabilityNagi’s theory on disability

• Sociologist / epidemiologist - 1st to recognize environment plays a role in definition of disability.

• Disability = a relational concept which goes beyond individual limitations & attributes & includes actual physical & social environments as well as the reaction of others.

• Possible to have a disability construct used:– Across the US?– Internationally / cross-nationally?– What are the complexities involved?

Disability Defined Disability Defined • Americans with Disabilities Act (ADA) of 1990 defines disability

as: – physical or mental impairment that substantially limits one or

more life activities, – a history or record of such an impairment, or – Regarded by others as having such an impairment.

• Under Federal Social Security Disability Act, "disability" means the "inability to engage in any substantial gainful activity by reason of any medically determinable physician or mental impairment which can be expected to last  for a continuous period of not less than 12 months or result in death."

• World Health Organization (WHO) created an International Classification of Functioning and Disability (ICF). Breaks disability construct into three dimensions:– 1) Medical or health condition; 2) Functional limitation and

participation restrictions; 3) Environmental restrictions

Operationalizing construct of DisabilityOperationalizing construct of Disability

• How will you identify disability, based on your construct / definition?

• How will you measure the construct?

• How many items will you need to do so?

– If unlimited amount …– If only 1 item … or 3, 6, 10 items … – As a stand-alone instrument or subsection of

whole

Overview: 3 Types of Disability DefinitionsOverview: 3 Types of Disability Definitions

• Functional /Activity / Work Limitation: – Because of a health condition/disability, the person

experiences limits in the ability to walk, talk, think, see, hear, remember, etc. The condition lasts at least six months.

• Specific condition/disease: – The specific disorder (Down Syndrome, Multiple Sclerosis,

Macular Degeneration), it is often paired with a severity and onset measure.

• Self-report: – Do you have a disability?

Functional/Activity LimitationFunctional/Activity Limitation

• Easier to measure functional limitation. Many scales exist (e.g. activities of daily living scales).

• Need to combine this with severity measures.

• Use in self-report surveys has disadvantages because two people with the same functional limitation may report it differently.

– Anchor problem: Who is the person comparing themselves to in assessing their functional impairment.

– Someone who cannot do something may report no difficulty because he / she sees it as not relevant / no difficulty in something not done at all.

Work Limitation QuestionsWork Limitation Questions

• A subset of functional or activity limitation is work limitation.

• (Do you/Does anyone in this household) have a health problem or disability which prevents (you/them) from working or which limits the kind or amount of work (you/they) can do? (Y/N)

• Problems: – It is over-inclusive. People with broken legs, pregnant, short

term injuries may say yes.

– It is under-inclusive because people with disabilities who work might say no. People who have visual impairments, deafness, or learning disabilities often report a condition but say that they have no activity limitation.

– Any others?

Disability as measured by Health Condition onlyDisability as measured by Health Condition only

• Example: Do you have one of the following conditions: – Asthma– Gastrointestinal disorder– Depression– Visual impairment

• Survey measurement issues:– Number of conditions is too large – Ability to determine severity per condition important as

not everyone with a condition has a disability.

Self-report measureSelf-report measure

• Do you have a disability? AND / OR Do you have a disability or health condition that causes a functional limitation lasting more than 6 months?

• Problems: • Specific groups are less likely to say yes

– Older people– People with cognitive disabilities– People who are deaf– People with a stigmatizing condition

• Some people with disabilities may not fit the definition above.– People with a history of a disability or who may qualify as having a

disability because they are perceived to have a disability. (e.g. people with facial scars or deformity).

• Some people may say yes to it and not have a disability– Particular disorders can not be counted as disability (actively SA,

pyromania, “deviance clause”)– Some claim to be disabled with no confirming diagnosis (LD, ADHD)

What measures are being used?What measures are being used?• In an effort not to reinvent the wheel,

let’s look at what is currently being used:

– By who– How (who do they include / exclude)– For what purpose– With what results (for prevalence).

Examples of Federal & International MeasuresExamples of Federal & International Measures• Behavioral Risk Surveillance Survey (BRFSS)

• Survey of Income and Program Participation (SIPP)

• Current Population Survey (CPS)

• National Health Interview Survey (NHIS)

• Census 2000 Long form & American Community Survey (ACS)

• CDC Healthy People 2010 measures

• UN Measures (with NCHS staff from US)

Federal Measures: BRFSSFederal Measures: BRFSS

• RDD telephone survey of non-institutional pop >18 yrs

• 1998 module on disability from 36,842 respondents in 11 states & DC. RR of 59.2%

• Prevalence rates range by state – 21.8% in AL to 17.1% SC

• Used 2 Qs: – Are you limited in any way in any activities because of an

impairment or health problem? (y/n)

– If you use any equipment or help from others to get around, what do you use? (wheelchair, walker, cane, another person)

Federal Measures: SIPPFederal Measures: SIPP

• Not designed as survey of health or disability - but these concepts so closely connect with program participation it cannot meet goals without including these measures

• Panel survey with topical modules; monthly admin, sample sizes from 14,000 to 36,700 interviewed households. Widely cited prevalence rate of 20.6% based on 1994 data

• Non-institutionalized civilians only – uses a “kitchen sink” definition of disability– Itemized specific functional activities, ADLs, and instrumental

activities of daily living. – Examples:

• Seeing ordinary newspaper print • Hearing normal conversation• Having speech understood

Federal Measures: CPS (1981-2004)Federal Measures: CPS (1981-2004)• Survey of 50,000 HH only (NONE in institutions,

schools, hospitals) in 754 sample areas.

• Conducted monthly by Census for BLS.

• Purpose - provide primary indicator for labor force / income stats. Q on disability = work limitation:– Does anyone in this HH have a health problem or

disability which prevents them from working or limits the kind of work they can do?

– Strengths / weaknesses of this measure?

• In 2002, data showed 22M (10%) ages 16-64 reported work disability

Federal Measures: NHISFederal Measures: NHIS

• Disability defined as being limited in activity caused by chronic health condition. Focuses on activity limitation in major activity of life, defined by age:

• <5 years - play• 5-17 yrs - attending school• 18-69 working or keeping home• >70 yrs - ability to care for self / home without

assistance from another person

• Data as of 2002 reported 34M (12%) of non-institutionalized civilians limited in usual activities

Federal Measures: Census 2000Federal Measures: Census 2000• 281.4 M people in 115.9M housing units

• Long form (1/6 HH) had disability Q

• Broken into 6 categories:– Sensory– Physical– Mental– Self-care– Going outside the home– Employment Disability

Federal Measures: ACSFederal Measures: ACS

• Designed to replace Census long form.

• Conducted by Census to 3M HHs in 2005. Starting in 2006 – included the 2.5% of US population living in “group quarters.”

• As of 2002, data shows 41M or 13% of non-institutionalized population with disability.

Census 2000 / ACS Measures cont.Census 2000 / ACS Measures cont.

• Does this person have any of the following long-lasting conditions:– Blindness, deafness, or a severe vision or hearing

impairment?– A condition that substantially limits one or more basic

physical activities such as walking, climbing stairs, reaching, lifting, or carrying?

• IF 15 yrs or older: Because of a physical, mental, or emotional condition lasting 6 months or more, does this person have any difficulty in doing any of the following activities?– Going outside the home alone to shop or visit a doctor’s

office?– Working at a job or business?

Federal Measure: CDC Healthy People 2010 Federal Measure: CDC Healthy People 2010

Developed standardized items to be used in all Healthy People 2010 surveys.

1. Are you limited in any way in any activities because of a physical, mental, or emotional problems (y/n)

2. Do you now have any health problems that require you to use special equipment such as a cane, wheelchair, a special bed, or special telephone? (y/n)

UN MeasuresUN Measures• UN commissioned a team of experts to create a

cross-national disability measure

• Broad design - able to compare across cultures. Draft Qs include (scale for response cats)

– Do you have difficulty seeing, even if wearing glasses?– Do you have any difficulty hearing, even if using a

hearing aid? – Do you have any difficulty walking or climbing steps?– Do you have difficulty remembering or concentrating?– Do you have difficulty with self-care, such as washing or

dressing?– Because of a physical, mental, or emotional health

condition, do you have difficulty communicating (ex. understanding others or others understanding you).

Challenges posed for data collectionChallenges posed for data collection

• Measures– With limited space - challenge to construct Qs

straightforward / easy to answer– Respondents fear of potential stigma associated with

reporting– Others?

• Administration:– Not all modes are accessible to all populations– Non-response at unit and item level – Others?

Summary: What to Do?Summary: What to Do?• Disability measures are not uniform and there is no

consensus about what questions to use. Pattern of questions may help you get some indication of disability prevalence. Use multiple types of questions within one survey.

• Limit the use of the word “disability” when possible.

• Consider including questions that ask about disability-specific service use in which eligibility is required.– Ever use VR, SPED, DMH, DMR– Ever received SSI, SSDI

• Consider including questions that ask about equipment, devices, and AT.– Wheelchair, scooter, leg braces– Guide dog, service dog, white cane, JAWS, Braille– Hearing aides, CART, ASL interpreters

Developing the NSIP Survey InstrumentDeveloping the NSIP Survey Instrument

Goal of Project: Develop a survey instrument that can be used to ascertain the prevalence rate of disability among active members in:

• Americorps National, Americorps Direct, NCCC, VISTA

Steps in the project: 1. Conducted literature review on federal and international

measures of disability.2. Convened survey work group to draft instrument. 3. Seek IRB Approval for Cognitive Testing. 4. Conducted 9 cognitive interviews with former CNCS

members. 5. Produce summary of key findings for CNCS. 6. Produce final survey and toolkit for states on how to

conduct the survey (drafts available for conference attendees).

Convened survey work group to draft instrument Convened survey work group to draft instrument • Work group comprised of experts in fields of disability, sociology,

and survey methodology. Defined areas of disability to measure:

– ADD / ADHD– Autism– Chronic Health Conditions– Communications / Speech– Developmental disabilities, cognitive impairments, learning disabilities, – Disability – general– Mobility– Psychiatric or Mental Health, – Sensory (hearing / vision)– Short stature / little person– Substance abuser– Traumatic Brain Injury

• Developed strategy for how to field the instrument to get high response rate, even among persons without disability (to best gather prevalence estimates).

Conducted Cognitive interviewsConducted Cognitive interviews• Conducted in 1:1 format.

• Respondents complete the questions and provide feedback to researchers during & after administration, using techniques such as:

• Direct observation (body language, eye movement).• “Think aloud” (walk through thought process).• Debriefing (sensitivity, burden).• Validity testing (define term in own words).

• 9 case maximum, per OMB guidelines, prior to OMB approval.

Findings from Cognitive InterviewsFindings from Cognitive Interviews

• Confirmation of “disability” being a term to avoid in the title and cover letter. – Strong, consistent feedback on this issue. – Described disability as “loaded” or “negative connotation.”

• Finalized title as “CNC Survey on Health, Well-Being, and Member Diversity.”

• Provide more emphasis on purpose in cover letter, stress themes of “diversity” and “inclusion” which resonated.

• Removed introductory text to series on MH to minimize any extra attention to sensitive topics.

• Added “social security survivors benefits” as response option in series on benefits.