Making Making Family Family

Outcome Outcome DataData

RepresentatiRepresentative ve

of the Stateof the State

Monday, August 27, Monday, August 27, 20072007

Rosanne Griff-Rosanne Griff-CabelliCabelli

rosanne.griff-cabelli@state.de.us

Sue CampbellSue Campbellsusan.campbell@state.de.us

Delaware Birth to Delaware Birth to Three Three

Early Intervention Early Intervention SystemSystem

Purpose of the Purpose of the Original Family Original Family

SurveySurveyThe survey was conducted in order to

•obtain information about families’ perceptions of and satisfaction with services they received from the Child Development Watch Program (CDW)

•determine families’ satisfaction with the CDW office facilities and staff interactions

•determine the impact of CDW and Birth to Three services on families’ quality of life

Topics Addressed Topics Addressed in the Original Family in the Original Family

SurveySurvey Did parents think their ability to parent theirchild with a disability improved?

Did parents think their child(ren)’s development improved?

Did parents feel that CDW was responsive to their needs?

Did parents feel that they had input into the services their children received?

Were families satisfied with CDW and staffinteractions?

Did CDW services have a positive impact onfamilies’ quality of life?

2006 Family Outcomes to be Assessed2006 Family Outcomes to be Assessed

Outcome 1:Families know their rights and advocateeffectively for their child

Outcome 2:Families Understand their child’s abilitiesand special needs

Outcome 3:Families help their child develop and learn

Reasons for a PilotReasons for a Pilot

OSEP Family OutcomesDevelop and integrate OSEP outcomesquestions into existing survey

Expand responses for existing survey itemsFrom a three point response to a six pointresponse

Test strategy of telephone interview for surveyingfamilies

Developing the Developing the New Family SurveyNew Family Survey

Delaware Ongoing Program Evaluation

Committee (OPEC) made recommendations:

Added questions to increase depth and range of response for the three family outcomes

Responses reported as single questions and by clusters– look at trends since 1999

Expand responses of the existing survey froma range of three to a range of sixBEFORE: Yes; Less than I’d Like; NoAFTER: Very strongly agree; Strongly agree; Agree; Disagree; Strongly disagree; Very strongly disagree

Piloting the New Family SurveyPiloting the New Family Survey

Conducted in Fall 2005

Telephone contact families randomly selected letter sent to families before calls made Calls made during day and evening

Demographics of 2005 Demographics of 2005 Telephone Pilot Family Survey Telephone Pilot Family Survey Respondents (N=49)Respondents (N=49)• Gender (child’s)

26 male(53%)22 female (45%)

• Race/Ethnicity38 Caucasian (79%)7 AfricanAmerican (15%)1 Hispanic (2%)1 Asian (2%)1 Not Reported (2%)

• Family Income11 >$100,000 (23%)19 $50-100,000 (41%)9 $20-49,999 (19%)6 <$20,000 (13%)2 Not Reported

• Location24 New Castle County (49%)25 Kent or Sussex County (51%)

2005 Federal Outcomes2005 Federal OutcomesMeasures Internal Measures Internal ConsistencyConsistency

Federal Outcome Topic

Number of

Questions

Chronbach Alpha Coefficie

nt

% of Families indicating “yes”

Families Know their Rights

4 .846 91.1%

Families Effectively Communicate their Children’s Needs

5 .715 95.1%

Families help their Children Develop and Learn

4 .808 93.4%

1.Questions from pilot found appropriate for use

2.Better to use telephone survey than mail distribution method

3.Stratified sample rather than random sample

Pilot resulted in the following:Pilot resulted in the following:

2006 Telephone Interviews 2006 Telephone Interviews with Familieswith Families

Representative sample is predetermined byinformation in data system (ISIS)•Geographic region•Race/ethnicity•Length of time in CDW (state focus)

Sample matrix currently includes 12 data cells•Each cell contains 30 families to be sampled•Total of 360 families to be sampled

Can always add specific demographics if

need to focus on specific populations

Small cell size

Sampling MatrixSampling Matrix

Geographic Region

Ethnicity Length of Time in Program

North

Caucasian < 18 months (30 families)

> 18 months (30 families)

African American

< 18 months (30 families)

> 18 months (30 families)

Other < 18 months (30 families)

> 18 months (30 families)

South

Caucasian < 18 months (30 families)

> 18 months (30 families)

African American

< 18 months (30 families)

> 18 months (30 families)

Other < 18 months (30 families)

> 18 months (30 families)

Number of Families by Region, Number of Families by Region, Ethnicity, and Ethnicity, and

Months in CDWMonths in CDW

Region

African American

Caucasian Other

Total

Less than 18Months

More than 18Months

Less than 18Months

More than 18Months

Less than 18Months

More than 18Months

North 110 57 221 147 112 37 684

South

52 26 104 60 35 15 292

Total 162 83 325 207 147 52 976

Received list of 976 families in CDW684 in New Castle County (CDW

North)292 in Kent and Sussex Counties

(CDW South)

Before the Telephone Interviews…Before the Telephone Interviews…

850 of these families were mailed:

1. a cover letter explaining the purpose of the interview

2. an information sheet describing• the usefulness of family feedback• assurances of confidentiality• examples of some information that would be asked• additional contact numbers if families had

questions about the interview

Number of Letters Mailed by Number of Letters Mailed by Region, Ethnicity, and Months Region, Ethnicity, and Months in CDW in CDW (N=850)(N=850)

EthnicityAfrican

AmericanCaucasian Other

Length of Time in Program

Less than 18

Months

Greater

than 18

Months

Less than 18

Months

Greater

than 18

Months

Less than 18

Months

Greater

than 18

Months

State

1st Mailing

112 83 120 120 147 52

2nd Mailing

6 0 6 0 0 0

3rd Mailing

44 0 100 60 0 0

Total 162 83 226 180 147 52

Letters Mailed and Return RateLetters Mailed and Return Rate

Telephone calls were made to all of the families from the original list, unless letters were returned due to incorrect addresses, until 224 families had completed the Family SurveyFamily Survey. .

The families were contacted between one and four times.

Calls would be made during the day, in the evenings, and on the weekends

Reasons families were not contacted via telephone:

•Telephone was disconnected•Number was wrong •Families were found not to meet the

criteria for completing the telephone interview

Reasons why Families were not ContactedReasons why Families were not Contacted

Reasons families were not contacted via telephone:

•97 Telephone was disconnected (35.4%)

•63 Number was wrong (23.0%)•59 Mail returned (21.5%)•28 No phone number listed (10.2%)•9 Less than six months in CDW

(3.3%)•5 Phone not in service (1.8%)•14 Other reasons (5.1%)

275 families

How did Delaware Determine How did Delaware Determine Family Outcome Data is Family Outcome Data is Representative?Representative?

1.Contract with the Center for Disabilities Studies, University of Delaware to moderate biased response

2. Sample according to demographics of Annual Child Count

Cost Data Preliminary AnalysisCost Data Preliminary Analysis

Delivery Method Mailed Survey

Telephone Interview

Year of Implementation 2004 2006

Sampling Method Random SampleStratified Random

Sample

Number Contacted 203 surveys 576 candidates

Number Completed 100 surveys 224 interviews

Return Rate 49.3% of sample 38.9% of sample

% of the Population 10.80% 22.90%

Cost/Person Contacted $3.23/family $2.34/family

Cost/Completed Survey $6.57/survey $8.91/survey

Methodology Mailed survey up to 4 times Mailed letter

  Telephone call reminder

Telephone Interviews

Costs IncludedMailed letter and

survey Mailed letter

  Return envelope Telephone calls

  Return mailing  

 Survey reproduction

costs  

  Phone call reminders  

ConclusionsConclusions

1.The expanded response possibilities allows for deeper analysis of families’ responses (move to 6 point scale)

2.The telephone survey allows for individual responses and clarification of questions

3.The telephone survey allows for the immediateidentification of the sample demographics, and allows for targeting of underrepresented categories. Data collection will continue until all categories are appropriately represented

4. The telephone survey also allows for immediatefeedback as to why families chose not to participate and the demographics of who chose not to participate

Lessons Learned for Efficient Lessons Learned for Efficient Family Surveying and Adequate Family Surveying and Adequate RepresentationRepresentation1.Contact families at various times

of the day within a shorter timeframe (1-2 weeks) rather than making multiple attempts at a larger timeframe.

2.Request updated information from CDW offices to contact families who have disconnected phone numbers, no telephone numbers or wrong telephone numbers. If phone number still not available, mail survey.

3.Send a follow-up letter to families asking families to call back to schedule an interview