Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

Developing a User Interface for Large-scale, Multi-mode, Survey Data Collection

Jennifer Beck & Elizabeth SinclairU.S. Census Bureau

User Focus 2012October 19, 2012

Saturday, January 5, 2013

Economic Census• A measure of the health of U.S. businesses and the American economy

every five years

• Collects detailed information on revenue and business performance

• Goes out to more than 4 million businesses across industries:1. Multi-unit businesses = businesses with multiple locations2. Single-unit businesses = businesses with a single location

• Businesses receive a combination of forms tailored to their specific business activities– more than 800 different survey forms

• The number of form combinations creates unique multi-mode data collection challenges

Economic Census reporting options

• The 2012 Economic Census has three different modes for data collection:

1. Paper forms2. Surveyor – a software application3. Internet

Paper form

Surveyor

Internet option

2012 Economic Census Internet option

• Only available to single-unit businesses:– A pilot for future Economic Censuses– Alternative to Surveyor software

• Respondents log into a Web interface through Census website

• Respondents fill out and submit all information online

Multi-mode data collection• Offering multiple modes of data collection has

benefits:– Can save costs– Good customer service

• Designing for multiple modes is challenging:– Mode consistency– Mode capabilities and constraints

Multi-mode challenges in the Economic Census

• The paper forms, Surveyor (software program) and the Web interface “share” a creation database

• The database contains “images” that will make up the form pages for all three modes

• Because the systems “share” pages, the pages have to work for all three modes:

– The “shared” design creates limitations and usability issues, especially for Surveyor and the Web

– Changes made for the Web also make the same changes to Surveyor

2012 Economic Census Internet option: Usability testing goals

1. Evaluate the overall function and performance of the online interface

2. Assess respondents’ reactions to the interface

Participants• 17 respondents from small businesses

• Diverse job titles: accountants, office managers, business owners

• From three different industries: – Retail/Service companies– Manufacturing companies– Construction companies

MethodologyWhere: • Met at respondents’ place of business

What: • Respondents used their own computers (or iPad)

How: • Respondents filled out the census forms• Completed tasks to test features of the interface• Used standard “think-aloud” protocol

Usability findings• The Web interface performed well during the

testing

• We uncovered some key findings about the interface:1. Questionnaire navigation2. Scrolling3. Response quality control

Questionnaire navigation

• Survey navigation:– Forward and backward paging– Questionnaire navigation – response path through

a survey

• Skip patterns:– Questionnaire navigation path instructions for

respondents– Help respondents locate and navigate to the

relevant form sections

Skip pattern: Paper questionnaire

Skip pattern: Web interface

Using skip patterns• Skip patterns can have a mode-specific advantage:

– Paper:• Require reading, understanding, and following skip instruction

navigation• Respondents do not always follow skip patterns on paper forms

well• Can create response errors

– On the Web:• Skip patterns can be automated• Directly navigate user down the correct questionnaire navigation

path• Can reduce response error

Problems with skip patterns: Usability findings

• The Web Interface (and Surveyor) had no automated skip patterns– Questions still contained skip instructions– Respondents had to page through irrelevant sections– Due to timing and programming complexity

• Respondents were confused as to how to navigate to the appropriate section of the questionnaire– Expected forward navigation to take them to the appropriate screen– Expected to be able to click “Remarks” to navigate to that section

• Lack of automation was especially problematic for the Web interface– Web interface had no navigation tree– Surveyor software has a navigation tree

Surveyor navigation tree

Fixing problems with skip patterns

The recommendations:1. Program automated skip patterns to navigate the

respondent to the next questions (and remove skip instructions)OR2. Include a navigation tree

The resolutions:• Skip instruction stayed with the question

– Shared image database made changing question layout and presentation not possible

• Programming complexities and time constraints prevented adding a navigation tree

Scrolling • Economic Census has a number of “long”

questions– Tend to be detailed business activities and cumulative

calculations– Do not fit on a single “page”

• Long questions require mode-specific solutions:– Paper:

• Pagination– Web interface (and Surveyor):

• Paging• Scrolling

Surveyor

Internet

Choosing between paging and scrolling

• Paging creates mode-consistency problems:– Paper:

• Easy to “flip” back and forth between the question sections– Web interface (and Surveyor):

• Requires forward and backward navigation to go back and forth between the question sections

• Breaking up the item across pages can create response errors

• Scrolling “preserves” mode consistency– The 2012 Economic Census Web interface employed scrolling

Problems Scrolling: Usability findings

• Scrolling created mode-specific problems:1. Long items were “intimidating” to respondents2. Length attenuated the benefits of scrolling over

paging

• Scrolling was especially problematic for the Web interface– Timing out:

• Long items take longer to fill out• Data save linked to navigating to the next page• Respondents could lose data

Fixing problems with scrollingThe recommendations:

1. Break up the items over multiple pages2. (Make changes to the question to maintain mode consistency) 3. Include a navigation tree

The resolutions:• Questions maintained scrolling design:

– Shared image database made changing question layout and presentation not possible

• Not enough time to re-program and add a navigation tree• Time out increased to 50 minutes• Data saved on time out

Response Quality Control• Data quality is a big concern:

– Follow-up is expensive– Post-collection editing is expensive and time consuming

• Respondents find data edits helpful – Provides a “check” against errors– Communicates the importance of the information

• Controlling for data quality is mode-specific advantage:– Paper forms offer no “built-in” data quality control– The Web interface allows for “real time data editing”– Can link responding to satisfying quality requirements

• The web interface had edit triggers for individual items and for of the

Item-specific data edits: Errors Errors = items respondents must fill out in order to submit their data to Census

Item-specific data edits: Warnings

Warnings = items we would like respondents to fill out *

* Respondents can still submit their data to Census with out filling them out

Overall data edits: Form ReviewForm Review= indicated problem sections with color coding

• One-click navigation to problem section

Response quality control: Usability findings

• Respondents liked the quality control features:– Item-specific data edits:

• The edit messages were clear and helpful• Liked having the “extra check”

– Form Review:• Really liked the layout of the form review• Liked the ease of navigating to problem sections

• Capabilities were examples of successful design features

• Provide in quality-control measure not possible in a paper mode

Challenges to designing the 2012 Economic Census Web interface

• The Economic Census is a large data-collection undertaking

• Presents challenges for how to maintain consistency between modes and optimize mode advantages

• More work needs to be done to refine web data collection for future Economic Censuses

Contact Information

Jen Beck:jennifer.l.beck@census.gov

Elizabeth Sinclair:elizabeth.sinclair@census.gov

Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

Documents

Transcript of Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

Public Information Meeting #1 Hamilton and Tipton Co Final... · Jennifer Beck. INDOT Project Manager, Greenfield District. jbeck@indot.in.gov 317-525-4995. PROJECT TEAM OVERVIEW

Beck Final

AAPSE 2003 Using Surveys to Determine Language Needs Presented by: Suzanne Snedeker, Ph.D. Sabina Swift, Ph.D. Jennifer Weber.

The Beck Protocol - Dr Bob Beck

Terry Carwile Ray Beck Jennifer Riley Gene Bilodeau Byron ... · Terry Carwile Ray Beck Jennifer Riley Gene Bilodeau Byron Willems Tony Bohrer Position sought: Mayor Mayor Council

Artists & reports SePTember 2013 40 · 2013-09-10 · 40 GroSSe JubiläumSauSGabe Timeline – 40 Jahre durch S muSikbuSineSS Tom beck STefanie heinzmann nica & Joe Jennifer ruSh

Welcome! [] · Melodie Beck (Lead) Erick Scheminske . Terry Anderle . Bill Levine . Rich Hull . Ann Renaud . Jeff Stalter . Jennifer Dye . FUNCTIONAL TEAMS . Budget . Tom Gamache

MEASURING SEXUAL IDENTITY IN NCHS SURVEYS Jennifer Madans, Ph.D. U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES Centers for Disease Control and Prevention.

Beck Indictment

MEDO DE QUEDA EM INDIVÍDUOS COM ARTRITE ......BAI (Beck Anxiety Inventory – Inventário Beck de Ansiedade) e BDI (Beck Depression Inventory – Inventário Beck de Depressão).

BECK CIRCULAR SILENCERS LES SILENCIEUX CIRCULAIRES BECK ...beck-gmbh.net/media/wysiwyg/downloads/pdf/Beck_Produktbroschue… · Die beck Rohrschalldämpfer sind für den wirkungsvollen

Beck Ulrich

Stanford Cornea Project 1. 2 Laura Hartman, Dale Waters, Rachel Parke- Houben, Curtis W. Frank Stayce Beck, Luo Luo Zheng, Yuhua Hu Jennifer Cochran Resmi.

Metabolic Syndrome, Diabetes, and Cognitive Impairment in the Era of Combination Antiretroviral Therapy Allen McCutchan 1, Jennifer Marquie-Beck 1, Scott.

Terapia Cognitiva Beck Clase Beck

Beck online: Speichern auf einem USB-Stick · PDF file>> beck-online SS beck-shop >> beck-akademie beck-stellenmarkt >> beck-community beck-aktuell beck-online DIE DATE-NBANK Suche:

full PDF scorecard here · Beck, Jennifer B ... Van Drew, Jeff F Vitale, Joseph F. F Watson, ... NEW JERSEY FILM AND TELEVISION PROJECT AND EMPLOYMENT INCENTIVE PROGRAM

Beck 2.0 - Community Elemente Beim Beck Verlag

COMMITTEE ON OPINIONS JENNIFER BECK and …ogtf.lpcnj.org/2009/2009300iK/Beck v OHare - Trial Court (Feinberg...advantages and disadvantages of “monetizing” some of its ... available

Ulrich Beck