AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Casey Langer Tesfaye and Susan White Can a...

AAPOR 69th Annual ConferenceAnaheim, CA May 17, 2014

Casey Langer Tesfaye and Susan White

Can a survey of U.S. High Schools be Replaced or Reduced through Web Searches? The Successes and Complications of an

Experimental Strategy

Background

•Nationwide Survey of High School Physics

•Multiphase survey Sample: public & private HS via NCES

Refreshed quadrennially, Redrawn periodically Phase 1: schools, Phase 2: teachers

Phase 1: Schools

•Does your school offer Physics?•Names & e-mail addresses of physics teachers

Background

•Multimode survey Mode: paper, web, phone

Problem 1

•Paper survey is really expensive

Problem 2

•Schools are oversurveyed•One IRB is not enough

More and more schools have custom research applications or processes

Problem 3

•Address lists out of date Some schools close or move

•Some schools difficult to reach by phone

•Eliminate paper survey altogether•Reduce contact with schools•Filter out closed schools faster

Proposed solution

•Begin with web searches•Call remaining schools

Strategy

•3 minute limit•Zoned form

Evaluating the effort

• Responses Response rate Data quality Research process requests

• $ savings

Outcome

•Overall response rate lower, but still over 90%

Outcome

Mode # of Responses % of total Responses

Web search 1061 29%

E-mail 894 25%

Phone call 1663 46%

Total 3618 100%

Data Quality

•Difficult to measure•Many issues to consider

Training and Streamlining

•Googling is a skill Google evolves

•Search strategies and techniques vary Evolve over time

There is a tradeoff between encouraging creative search strategies and evaluating the quality of the information yielded

Data Sources varied

•Sources varied Search results & search pages (e.g. Physics

Olympic teams, activities rosters) School websites, class websites, department

pages, grade-level pages, parental resources Great schools websites LinkedIn and other social networking sites

Quality Control

•Many schools share the same name Double check school location

•Many cities share the same name Use full/er address to verify

•Many teachers share the same name Listings with last names only are common

Information varied

• Very different configurations of information for each school Some easier to navigate, some had more or

less information or differently targeted information (e.g. sports teams)

Different definitions (e.g. Upper School)

Information varied

•Search results vary Date of updates varies Site content varies Websites go up & down, sometimes cached,

sometimes redesigned

What makes a complete entry?

•Goal of form was staged Stage 1:1. Is physics information available?2. Is physics teacher information available?

What makes a complete entry?

•Goal of form was staged Stage 1:1. Is physics information available?2. Is physics teacher information available? Stage 2:3. Then is there any helpful contact information?

- What constitutes helpful contact information?

New classification: helpful, but not complete

Data Quality

•Every mode of survey has data quality issues Web searches yielded more inaccurate

information that in the past Inaccurate information leaks into teacher

survey E.g. “I don’t teach physics” “I teach physics, but

I have no students this year” More of a need for a streamlined system for

corrections

•Web search, email & phone call cheaper than Paper survey, email & phone call

•Web search & phone calls roughly equal (but web had the added benefit of filtering out closed schools faster and reducing contact with schools)

In Sum…

•Web searches save $ •Web searches lead to less research process requests and potential rejections

But…•Web searches cannot be the only mode•A system that uses web searches:

Must have streamlined procedures for updating incorrect information

The web continually evolves

•More information online every day•Google continually improving and streamlining search results

• Training cannot be static Develop search and quality standards that can

evolve over time Develop a system that encourages web search

staff to discuss their strategies Use these interactions to understand what

information you are collecting and where you are collecting it from

In the future

•We will continue the web searches But we will strengthen and streamline the

feedback loop

Thanks!

•Questions, comments?Casey Langer Tesfayeclanger@aip.org

AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Casey Langer Tesfaye and Susan White Can a...

Documents

Transcript of AAPOR 69 th Annual Conference Anaheim, CAMay 17, 2014 Casey Langer Tesfaye and Susan White Can a...

Tesfaye Debela FINAL.pdf

ОТЧЁТ AAPOR О БОЛЬШИХ ДАННЫХ

TILAHUN BEKELE TESFAYE - 213.55.95.56

Zone of Ambivalence AAPOR Presentation

NewUniversitiesStudy(Tesfaye, S.)

Gerald Taylor_Camac, Camay y Camasca

CONSUMER BUYING BEHAVIOR BY Tesfaye Tumdedo

2016 aapor mari harris

Africa’s Regional Integration Arrangements · Africa’s Regional Integration Arrangements History and Challenges Tesfaye Dinka and Walter Kennes1 September 2007 1 Tesfaye Dinka

Virtual travel project Espana Camay, Oscar,Teressa, Victoria.

SUPER DISCOUNT STORE - Philenewsmedia.philenews.com/prosfores/cyprus-is/superdiscountstore.pdf · CAMAY XEPIQN HAND SOAP CAMAY CAMAY 4x90wocg-0 baby skincare HEAD & SHOULDERS LAMnoYAN

AAPOR Report on Big Data

colemansdata.shoptocook.comcolemansdata.shoptocook.com/shoptocook/Content... · CAMAY Classic Camay Soap 21$5 Spray Nine Cleaner ra 599 Assorted Alibi Dish Liauid 3/$5 55 ct Sunlight

Camay Soap

2016 aapor michael wild

Final Thesis (Tesfaye Tsegaye)

Camay Final Report

AAPOR Conference 2013

Abel Tesfaye took - fusd.net

AAPOR 69 th Annual Conference 5/16/2014 Casey Langer Tesfaye American Institute of Physics Reporting Data in the Social Media World Panel Title: Practical.