Stemming the Rising Tide of Nonresponse

Response Rate ~arch Team

NASS RESPONSE RATE RESEARCH TEAM

FCSM 2018 Research and Policy Conference March 8, 2018

Gerald Tillman Census and Survey Division

Chief, Survey Administration Branch Business Council Sponsor

Todays Objectives

Rate Research Team

➢Research and Review of Nonresponse Literature➢Development of RRRT Work Plan➢Most Notable & Noteworthy Accomplishments➢ Lessons Learned➢Q&A

2

Nonresponse Literature Review

• A report from the American Academy of Political & Social Science - January2013 edition:

– Article: Explaining Rising Nonresponse Rates in Cross-Sectional Surveys (Brick and Williams)

• The common practice in surveys is to group the reasons for nonresponse into3 major categories:

– Noncontact (inaccessible), refusals, and other reasons

• “Other reasons” category typically are: (Consistent w/Groves & Couper 1998)– Language problems– Being away during data collection– Poor health

• More recent research on cell phones shows growth in cell-phone onlypopulation may effect the extent of coverage and nonresponse(Brick et al. 2006)

3

Nonresponse Literature Review

Six reasons for nonresponse:

(1) Failure of the data collector to locate or identify thesample unit

(2) Failure to make contact with the sample unit(3) Refusal of the sample unit to participate(4) Inability of the sample unit to participate(5) Inability of the data collector and sample unit to

communicate(6) Accidental loss of the data or questionnaire

(Lynn, 2008)

4

The Nonresponse Tidal Wave

.. ········· ··········. ··········

········· ..

r r r r r r r r r r r r

5

15.0

20.0

25.0

30.0

35.0

40.0

45.0

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

Crops APS Non-Response, 2002-2017, by quarter

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

June

Se

p De

c M

ar

June

Se

p De

c M

ar

June

Se

p De

c M

ar

June

Se

p De

c M

ar

June

Se

p De

c

NASS Response Rate Research Team

- ,_'l ,-- L -Response Rate Research Team

Our Goal

We are working to improve response rates by : • Strengthening survey processes• Decreasing respondent burden• Leveraging relationships• Improving enumerator training• Communicating more accurately and consistently with all stakeholders• Adding value by listening to producers and stakeholders

Team Membership

The team has been in existence since 2016. All NASS Divisions are represented on the team.

6

Where to Start

RRRT created a survey with 13 questions related to response rate:

• 85 pages of feedback from our 12 Regional Field Offices (RFOs)

• 588 pages of feedback from 158 (NASDA) National AssociationState Departments of Agriculture - Field & Phone SupervisorEnumerators

• 41 pages of feedback from 62 NASS Headquarters and NationalOperations Division staff

• Completed 6 Farmer Feedback Listening Sessions (CA, ID, MI, MO,ND, & MO)

7

13 Sub-Teams

Rate Research Team

❑ Undeliverable As Addressed (UAA)/Disconnect Team❑ Stakeholder Relations Team❑ NASDA Training Team❑ Survey Timeline Team❑ Respondent Burden Index Team❑ Deadwood Team❑ Sample Review Team❑ Callout Review Team❑ Strategic Optimized Sample Selection Team❑ Enhanced Data Collection Team❑ Increasing Cooperation & Engagement (ICE) Team❑ Farmer’s Feedback Team❑ Inaccessible Investigative Team

8

Most Notable Accomplishments

H ndln N mb r

pon~ R R

• nd Ml Ing Phon

_____ .....,. ._._ ..... _...,.. ___ ....,. ___ _

-- - I l .......,.,....._. .,__

H ndllng Und llv r Return d M lllngs

A Addr ss d

✓

✓

Tips for Sur y Mn nilr IOf)

Deadwood Sub-Team Deadwood – Records on our sample frame that are active, but in reality there is some evidence they could be out of business or scope.

Approved Operational Decision – DM-02-18, Decision Memorandum from NASS Administrator, January 24, 2018.

UAA/Disconnect Sub-Team

9

Investigating the Effect of Distributing a Brochure to Boost Response Rates

For 111or1 infvnnltion .. ..,. ... _ .. _O.S•-IIIIIIC--•m41,1tl...,__.,.

a.-•• ...... -.-&11 .... -•-•111.,.,._ 11 .... tc. .... tAr-•~ .,..,_,.,.. .. ....... ,..._.,,"" ... ... ---.---.. .. ••MelfUlllllfl-ft.tNlf ..,_,.,,,_,111111,_ ,_,. .,..--·~-·--·-.. -BIi_,, ____ , .... ~--,._., ... v~,,-i...m. S,llW-~,._~,_-- ...... •~Y'Jll--___ ,p,-'!Milllltl

,~ ... -~--Wft;I .. .,,., ........ ~ ~-ltrd> ... nmt,....,. ,....,. ____ ""' .... --·-----•-'-•-~ -~--........... ..,...,..,, ...... 1w11111 ... - ...... --......... ....,. Llt'INI_,.._, _ _, .. -

I.JSOi\,,;,;,,. _____ _

.-·V€: ----

A~t 90,000 ftrrners ~ cht quaiurty Acrffr, Producrion, end Stow (lPSJ Surv1y one tt> w t1111'$ per yea Dtpend1fl on whtrt you 1M ind tht tommodltift you p. you m,y rtc11Yt o,- or 11101' ol tlwlt quwrty IIMy1

,, I ---- Marth /,\• t.,\ - .. 1ro_ .. __

~4- ' ' . __ ,_.,,_ I

I, r ' r

June __ ... _,..,_,_, _...,...,.. .. _,fl_ -September ......... -ll'.OI_ ,, .. ,.,....,.,.,,._,,_lllif"'

f ...... I'' ~ .......... c., I .__._,.,.,.,... _ .. _.,._,..

Deambtt ........... -..,,.r,i.~ ••l"lliPllll~""pllb ______ ., __ ..

., ......,,.. .~------~ .~,..,_.,.,..,.... ..,... ___ __ I

,- aa.,.. ........ ...,. .... i. ~9N•---l• c.a-,....,-...

I. 11m,-,--•••• I·=------....... ... .,..-,_,....._ ....,.,_ ·---.... -·--·-----·-,. .,., ... __ -~-.... -•-..----... """~""-~ , ,1i1n.,. ... -... .... ... •Mtt■ ... 111!'1 -~-.........

.,.;.

•llinrllrl~•-- ~ -. -.. _,_ ___ ,._ .... = :;r,..- . ' •·

. '-· ···· '~ ·~~ -. -

10

Investigating the Effect of Distributing a Brochure to Boost Response Rates

Stakeholder Relation Sub-Team Split-Plot Sample Design

▪ Mailed a brochure title “Why They Matter” with survey questionnaires for halfthe sample

▪ Rated the previous year’s response rates levels (low, medium, high) as ablock factor

▪ States within each level were randomized to either receive a brochure or not

▪ The response variable focus is the current year’s total useable response rate

▪ The predicator variables: (Brochure: 1 = Yes, 0 = No), % response ratecomparison, and capture differences of brochure use comparison to 2016

▪ Data analysis: (1) A random intercept logistic model fitted for the 2017 totaluseable response rate; (2) A regression model fitted for the difference inresponse rates between 2016 and 2017

11

Investigating the Effect of Distributing a Brochure to Boost Response Rates

Studentized Residuals 3 -------- 30 --- --- ...... _

'" 0 ij .. 2 ::, ~ C 20 ., -- 0 ~ .. --- ~ 0 0 .. -- ~ :;; D:: -- cc: Cl.

~ ---+ -1 10 C 0 0 5!- -2 0 ., 0 D:: -0 -2 -1 0 2 -3.2 -1 .6 0 1 .6 3.2 ..

0 Lin ear Predictor Residual ., u C f! ., ~ 2 C -1 0 0 0

/ 0

'" '" ::, ::, ~ 0 ~ 0 ~ od/ ~

-2 cc: -1 cc: -1 ✓•

High Low Medium -2 -2

Response Rate Category (2016) 0

-2 -1 0 I Brochure - No --+- Yes I Quantile

Figure 1: Plots of least-squares means from the Figure 2: Plots of residuals from the regression random intercept logistic regression model model

2017 March Prospective Acreage Planting Survey

12

Sample Review Sub-Team

▪ Developed and streamlined aggregate and detail level response history data

▪ HR_PCT_XYR = % of complete responses for 1 year, 3 year, and 5 yearsrespectively

▪ HR_Surveys_Xyr = Total number of surveys for 1 year, 3 year, and 5 yearsrespectively

▪ HR_Mode_Xyr = Preferred mode of completion for 1 year, 3 year, and 5 yearsrespectively

▪ The team is working on new propensity score models:▪ Model #1 – Identify records most likely to be completed via field enumeration▪ Model #2 – Identify records most likely to be completed via Mail/CATI

▪ Discovering ways to incorporate impact: quantiles, variables/strata, breakpoints

▪ Benefit: Provides a clear cutoff for potential high impact operations

13

Respondent Burden Index (RBI) Sub-Team

• Review the old Joint Burden Index process

• Implemented and integrated an RBI calculation

• A high burden indicator will be loaded in Survey Management Service

• This will reduce staff time to determine best data collection strategy

• The burden is indicated with the use of these variables x1 (number of surveys),x2 (number of contacts), x3 (total OMB survey time)

• Based on level of burden, surveys involved, and response history the best datacollection strategy can be planned

• Other possible things that could help in the decision process:– Create a profile of your high burden records– Ask field supervisors/enumerators what they know about these operations– Review existing record level comments

14

Stemming the Rising Tide of Nonresponse

15

50.00

55.00

60.00

65.00

70.00

75.00

80.00

85.00

90.00

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

Crops APS Useable Response, 2002-2017, by quarter

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

Jun

Sep

Dec

Mar

Ju

n Se

p De

c M

ar

June

Se

p De

c M

ar

June

Se

p De

c M

ar

June

Se

p De

c M

ar

June

Se

p De

c M

ar

June

Se

p De

c

Stemming the Rising Tide of Nonresponse

cu ... ................................... ~ ······~ --~--

······································································ ······

Response Rate Research Team

Useable Response March 2002-2017 90.0

65.0

70.0

75.0

80.0

85.0

60.0

55.0

50.0 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

Useable Response June 2002-2017

55.0

60.0

65.0

70.0

75.0

80.0

85.0

90.0

50.0 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 16

Stemming the Rising Tide of Nonresponse

···················

··············································••

················································· ····················

'········· ~ ··················· -----, ·····················~ ~-~··-~..,-., .... ~ ··········•• ....... .

Response Rate Research Team

Useable Response September 2002-2017 90.0

60.0

65.0

70.0

75.0

80.0

85.0

55.0

50.0 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

Useable Response December 2002-2017 90.0

65.0

70.0

75.0

80.0

85.0

60.0

55.0

50.0 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

17

Lesson Learned

“Routine is the enemy of instinct. So break the mold! While it’s important to establish routines, schedules, and systems of operation, it’s just as important to know when to change

them. Routines without ongoing assessment lead to stagnation and mediocrity. Most individuals, teams, and

organizations rise to a challenge or fall to the familiar. It’s better to change and fail than to settle for the status quo.”

(Bishop TD Jakes, 2014)

18

RRRT Team Members

Jill Bishop Barbara Rater Valbona Bejleri Shirley Samson Andrew Dau Jamila Sani Gail Gregory Marcella Simmons* Andy Higgins Jodie Sprague John Hilton Gerald Tillman Greg Lemmons Shareefah Williams Dan Lofthus Tyler Wilson Beckie McCracken* Linda Young

Over 150 NASS employees participated as Members of the 13 RRRT Sub-Teams

*NASDA Supervisors (Field & Phone)

19

Selected References

Brick, Michael J. and Williams, Douglas (2013). Explaining Rising Nonresponse Rates in Cross-Sectional Surveys 2013. Westat.

Brick, Michael J., Dipko, Sarah, Presser, Stanley, Tucker, Clyde, and Yuan, Yangyang (2006). Nonresponse bias in a dual frame sample of cell and landline numbers. Public Opinion Quarterly 70 (5): 780-93

Groves, Robert M. and Couper, Mick P. (1998). Nonresponse in household interview surveys. New York, NY: John Wiley.

Lynn, Peter (2008). “The Problem of Nonresponse”. International Handbook of Survey Methodology, 2008. Institute for Social and Economic Research (ISER). University of Essex

20

Stemming The Rising Tide of Nonresponse

21