Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

Developing a User Interface for Large-scale, Multi-mode, Survey Data Collection

Jennifer Beck & Elizabeth SinclairU.S. Census Bureau

User Focus 2012October 19, 2012

Saturday, January 5, 2013

Economic Census• A measure of the health of U.S. businesses and the American economy

every five years

• Collects detailed information on revenue and business performance

• Goes out to more than 4 million businesses across industries:1. Multi-unit businesses = businesses with multiple locations2. Single-unit businesses = businesses with a single location

• Businesses receive a combination of forms tailored to their specific business activities– more than 800 different survey forms

• The number of form combinations creates unique multi-mode data collection challenges


Economic Census reporting options

• The 2012 Economic Census has three different modes for data collection:

1. Paper forms2. Surveyor – a software application3. Internet


Paper form


Surveyor


Internet option


2012 Economic Census Internet option

• Only available to single-unit businesses:– A pilot for future Economic Censuses– Alternative to Surveyor software

• Respondents log into a Web interface through Census website

• Respondents fill out and submit all information online


Multi-mode data collection• Offering multiple modes of data collection has

benefits:– Can save costs– Good customer service

• Designing for multiple modes is challenging:– Mode consistency– Mode capabilities and constraints


Multi-mode challenges in the Economic Census

• The paper forms, Surveyor (software program) and the Web interface “share” a creation database

• The database contains “images” that will make up the form pages for all three modes

• Because the systems “share” pages, the pages have to work for all three modes:

– The “shared” design creates limitations and usability issues, especially for Surveyor and the Web

– Changes made for the Web also make the same changes to Surveyor


2012 Economic Census Internet option: Usability testing goals

1. Evaluate the overall function and performance of the online interface

2. Assess respondents’ reactions to the interface


Participants• 17 respondents from small businesses

• Diverse job titles: accountants, office managers, business owners

• From three different industries: – Retail/Service companies– Manufacturing companies– Construction companies


MethodologyWhere: • Met at respondents’ place of business

What: • Respondents used their own computers (or iPad)

How: • Respondents filled out the census forms• Completed tasks to test features of the interface• Used standard “think-aloud” protocol


Usability findings• The Web interface performed well during the

testing

• We uncovered some key findings about the interface:1. Questionnaire navigation2. Scrolling3. Response quality control


Questionnaire navigation

• Survey navigation:– Forward and backward paging– Questionnaire navigation – response path through

a survey

• Skip patterns:– Questionnaire navigation path instructions for

respondents– Help respondents locate and navigate to the

relevant form sections


Skip pattern: Paper questionnaire


Skip pattern: Web interface


Using skip patterns• Skip patterns can have a mode-specific advantage:

– Paper:• Require reading, understanding, and following skip instruction

navigation• Respondents do not always follow skip patterns on paper forms

well• Can create response errors

– On the Web:• Skip patterns can be automated• Directly navigate user down the correct questionnaire navigation

path• Can reduce response error


Problems with skip patterns: Usability findings

• The Web Interface (and Surveyor) had no automated skip patterns– Questions still contained skip instructions– Respondents had to page through irrelevant sections– Due to timing and programming complexity

• Respondents were confused as to how to navigate to the appropriate section of the questionnaire– Expected forward navigation to take them to the appropriate screen– Expected to be able to click “Remarks” to navigate to that section

• Lack of automation was especially problematic for the Web interface– Web interface had no navigation tree– Surveyor software has a navigation tree


Surveyor navigation tree


Fixing problems with skip patterns

The recommendations:1. Program automated skip patterns to navigate the

respondent to the next questions (and remove skip instructions)OR2. Include a navigation tree

The resolutions:• Skip instruction stayed with the question

– Shared image database made changing question layout and presentation not possible

• Programming complexities and time constraints prevented adding a navigation tree


Scrolling • Economic Census has a number of “long”

questions– Tend to be detailed business activities and cumulative

calculations– Do not fit on a single “page”

• Long questions require mode-specific solutions:– Paper:

• Pagination– Web interface (and Surveyor):

• Paging• Scrolling


Paper


23


24


25


26


27


28


29


30


31


32


33


34


35


36


37


38


39


Surveyor


41


42


43


44


45


46


47


48


49


50


51


Internet


53


54


55


56


57


58


59


60


61


62


Choosing between paging and scrolling

• Paging creates mode-consistency problems:– Paper:

• Easy to “flip” back and forth between the question sections– Web interface (and Surveyor):

• Requires forward and backward navigation to go back and forth between the question sections

• Breaking up the item across pages can create response errors

• Scrolling “preserves” mode consistency– The 2012 Economic Census Web interface employed scrolling


Problems Scrolling: Usability findings

• Scrolling created mode-specific problems:1. Long items were “intimidating” to respondents2. Length attenuated the benefits of scrolling over

paging

• Scrolling was especially problematic for the Web interface– Timing out:

• Long items take longer to fill out• Data save linked to navigating to the next page• Respondents could lose data


Fixing problems with scrollingThe recommendations:

1. Break up the items over multiple pages2. (Make changes to the question to maintain mode consistency) 3. Include a navigation tree

The resolutions:• Questions maintained scrolling design:

– Shared image database made changing question layout and presentation not possible

• Not enough time to re-program and add a navigation tree• Time out increased to 50 minutes• Data saved on time out


Response Quality Control• Data quality is a big concern:

– Follow-up is expensive– Post-collection editing is expensive and time consuming

• Respondents find data edits helpful – Provides a “check” against errors– Communicates the importance of the information

• Controlling for data quality is mode-specific advantage:– Paper forms offer no “built-in” data quality control– The Web interface allows for “real time data editing”– Can link responding to satisfying quality requirements

• The web interface had edit triggers for individual items and for of the


Item-specific data edits: Errors Errors = items respondents must fill out in order to submit their data to Census


Item-specific data edits: Warnings

Warnings = items we would like respondents to fill out *

* Respondents can still submit their data to Census with out filling them out


Overall data edits: Form ReviewForm Review= indicated problem sections with color coding

• One-click navigation to problem section


Response quality control: Usability findings

• Respondents liked the quality control features:– Item-specific data edits:

• The edit messages were clear and helpful• Liked having the “extra check”

– Form Review:• Really liked the layout of the form review• Liked the ease of navigating to problem sections

• Capabilities were examples of successful design features

• Provide in quality-control measure not possible in a paper mode


Challenges to designing the 2012 Economic Census Web interface

• The Economic Census is a large data-collection undertaking

• Presents challenges for how to maintain consistency between modes and optimize mode advantages

• More work needs to be done to refine web data collection for future Economic Censuses


Contact Information

Jen Beck:[email protected]

Elizabeth Sinclair:[email protected]

72


Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

Documents

Transcript of Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)