Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining...
Transcript of Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining...
![Page 1: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/1.jpg)
Combining Crowdsourcing and
Google Street View to Identify
Street-level Accessibility Problems
Kotaro Hara, Vicki Le, Jon E. Froehlich
Human-Computer Interaction Lab
Computer Science Department
University of Maryland, College Park
makeability lab
![Page 2: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/2.jpg)
I want to start with a story…
![Page 3: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/3.jpg)
You Your Friend
![Page 4: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/4.jpg)
![Page 5: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/5.jpg)
![Page 6: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/6.jpg)
The problem is not just that
there are inaccessible areas of
cities, but also that there are
currently few methods for us
to determine them a priori
![Page 7: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/7.jpg)
Can we use Google Street View to find
sidewalk accessibility problems?
Could crowdworkers perform tasks to find, label, and
assess the severity of accessibility problems?
![Page 8: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/8.jpg)
ProjectSidewalk
Labeling Interface
![Page 9: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/9.jpg)
Validation Interface
ProjectSidewalk
![Page 10: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/10.jpg)
Physical Street Audits
Background and Related Work
![Page 11: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/11.jpg)
Physical Street Audits
Physical Street Audits
![Page 12: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/12.jpg)
Street audits are conducted by governments
and/or community organizations.
Time-consuming and expensive
![Page 13: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/13.jpg)
Video Recording Sampson et al. 1999
![Page 14: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/14.jpg)
Top-Down Satellite Imagery Taylor et al., 2011
![Page 15: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/15.jpg)
Omnidirectional Streetscape Imagery Clarke et al., 2010; Rundle et al., 2011; Taylor et al., 2011; Guy & Truong, 2011
![Page 16: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/16.jpg)
![Page 17: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/17.jpg)
Key Point High-level of concordance
between physical audit and
Street View based audit.
![Page 18: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/18.jpg)
Mobile Crowdsourcing SeeClickFix.com
![Page 19: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/19.jpg)
Mobile Crowdsourcing NYC 311
![Page 20: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/20.jpg)
Mobile Crowdsourcing NYC 311
These mobile tools can be used as complementary
techniques to our GSV approach
![Page 21: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/21.jpg)
Problem Categories
![Page 22: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/22.jpg)
Missing Curb Ramp
Missing Curb Ramp Object in Path Surface Problem Ending Sidewalk Other
![Page 23: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/23.jpg)
Object in Path
Missing Curb Ramp Object in Path Surface Problem Ending Sidewalk Other
![Page 24: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/24.jpg)
Surface Problem
Missing Curb Ramp Object in Path Surface Problem Ending Sidewalk Other
![Page 25: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/25.jpg)
Prematurely Ending Sidewalk
Missing Curb Ramp Object in Path Surface Problem Ending Sidewalk Other
![Page 26: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/26.jpg)
Other
Two curb ramps positioned
too close to each other
Other
Missing Curb Ramp Object in Path Surface Problem Ending Sidewalk
![Page 27: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/27.jpg)
Research Questions
Can motivated workers identify sidewalk
accessibility problems using Street View?
Can crowd workers perform this task?
![Page 28: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/28.jpg)
STUDY ONE: RESEARCH TEAM LABELERS
STUDY TWO: WHEELCHAIR USER LABELERS
STUDY THREE: MECHANICAL TURK LABELERS
![Page 29: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/29.jpg)
STUDY ONE: RESEARCH TEAM LABELERS
STUDY TWO: WHEELCHAIR USER LABELERS
STUDY THREE: MECHANICAL TURK LABELERS
![Page 30: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/30.jpg)
What accessibility problems exist in this image?
![Page 31: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/31.jpg)
R1 R2 R3
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 32: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/32.jpg)
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
![Page 33: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/33.jpg)
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
x2
Researcher 1
![Page 34: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/34.jpg)
x4
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
Researcher 2
![Page 35: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/35.jpg)
Researcher 3
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
x8
![Page 36: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/36.jpg)
Researcher 1
Researcher 2
Researcher 3
![Page 37: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/37.jpg)
There are multiple ways to examine the labels.
![Page 38: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/38.jpg)
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table Image Level Labels
This table tells us what accessibility
problems exist in the image
![Page 39: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/39.jpg)
Pixel Level Labels
Labeled pixels tell us where
the accessibility problems
exist in the image.
![Page 40: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/40.jpg)
Why do we care about image level vs. pixel level?
![Page 41: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/41.jpg)
Coarse Precise
Point Location
Level
Sub-block
Level
Block
Level (Pixel Level) (Image Level)
![Page 42: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/42.jpg)
Coarse Precise
Point Location
Level
Sub-block
Level
Block
Level (Pixel Level) (Image Level)
![Page 43: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/43.jpg)
Coarse Precise
Point Location
Level
Sub-block
Level
Block
Level (Pixel Level) (Image Level)
Pixel level labels could be used for
training machine learning algorithms
for detection and recognition tasks
![Page 44: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/44.jpg)
Coarse Precise Localization
Spectrum
Point Location
Level
Sub-block
Level
Block
Level
Specification
Spectrum
Multiclass Object in Path
Curb Ramp Missing
Prematurely Ending Sidewalk
Surface Problem
Binary
Problem
No Problem
(Pixel Level) (Image Level)
Two Accessibility Problem Spectrums Different ways of thinking about accessibility problem labels in GSV
Coarse Precise
![Page 45: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/45.jpg)
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
Problem
Multiclass label Binary Label
Sidewalk Ending
Surface Problem
Other
![Page 46: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/46.jpg)
![Page 47: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/47.jpg)
Dataset
Manually curated 229 static Street View images
of sidewalks from metropolitan area (Baltimore, DC,
LA, and New York)
Dataset consists of 47 Curb Ramp Missing, 66
Object in Path, 67 Surface Problem, and 50
Prematurely Ending Sidewalk, and 50 images
with no problems
Average image age was 3.1 (SD=0.8) years old
![Page 48: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/48.jpg)
Primary Study 1 Question
Can motivated workers provide consistent labels?
![Page 49: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/49.jpg)
Study Method
3 researchers individually labeled
229 static images
We used Fleiss’ kappa to measure image
level binary and multiclass label agreement
between researchers
![Page 50: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/50.jpg)
These are researcher labels
![Page 51: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/51.jpg)
R1 R2 R3
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 52: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/52.jpg)
R1 R2 R3
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 53: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/53.jpg)
R1 R2 R3
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 54: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/54.jpg)
We observed moderate to substantial
agreement between researchers
Researchers have consistent perspective
towards what constitutes sidewalk
accessibility problems
Result
![Page 55: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/55.jpg)
STUDY ONE: RESEARCH TEAM LABELERS
STUDY TWO: WHEELCHAIR USER LABELERS
STUDY THREE: MECHANICAL TURK LABELERS
![Page 56: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/56.jpg)
![Page 57: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/57.jpg)
Study Method
3 wheelchair users
Independently labeled 75 subset of 229 Street
View images
Think-aloud and sessions were video
recorded
30 min post-study interview
We used Fleiss’ kappa to measure agreement
between wheelchair users and researchers
![Page 58: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/58.jpg)
Here is the recording from the study session
![Page 59: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/59.jpg)
![Page 60: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/60.jpg)
Result
Strong agreement between wheelchair users’
labels and researchers’ labels
Wheelchair users and motivated workers share
the similar perspective of what constitute
sidewalk accessibility problems
![Page 61: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/61.jpg)
STUDY ONE: RESEARCH TEAM LABELERS
STUDY TWO: WHEELCHAIR USER LABELERS
STUDY THREE: MECHANICAL TURK LABELERS
![Page 62: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/62.jpg)
We need ground truth to evaluate turkers’ tasks
![Page 63: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/63.jpg)
Majority Vote
![Page 64: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/64.jpg)
R1 R2 R3 Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 65: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/65.jpg)
R1 R2 R3 Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 66: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/66.jpg)
R1 R2 R3 Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 67: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/67.jpg)
R1 R2 R3 Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 68: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/68.jpg)
R1 R2 R3 Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 69: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/69.jpg)
We took majority vote of researcher labels across
all 229 images to produce ground truth dataset
![Page 70: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/70.jpg)
Ground Truth
Turker
Per Image Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Correct
Correct
Correct
Wrong
This turker scored
3 out of 4
![Page 71: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/71.jpg)
Primary Study 3.1 Question
How accurate can turkers label sidewalk
accessibility problems?
![Page 72: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/72.jpg)
Study Method
We hired 185 turkers in total from AMT
We batched 1-10 image labeling task into 1 HIT
and paid $0.01-0.05 per HIT
We asked turkers to watch 3 min. tutorial video.
Task showed up after they finished watching first
half
Neglected Other because it was < 0.6% of the
entire label
![Page 73: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/73.jpg)
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
Kotaro Hara
Timer: 00:07:00 of 3 hours
10 3 hours
![Page 74: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/74.jpg)
High-level Results
81% accuracy without quality control
93% accuracy with quality control
![Page 75: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/75.jpg)
I want to show some positive turker labeling examples
![Page 76: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/76.jpg)
TURKER LABELING EXAMPLES
Curb Ramp Missing
![Page 77: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/77.jpg)
TURKER LABELING EXAMPLES
Curb Ramp Missing
![Page 78: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/78.jpg)
TURKER LABELING EXAMPLES
Object in Path
![Page 79: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/79.jpg)
TURKER LABELING EXAMPLES
Object in Path
![Page 80: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/80.jpg)
TURKER LABELING EXAMPLES
Prematurely Ending Sidewalk
![Page 81: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/81.jpg)
TURKER LABELING EXAMPLES
Prematurely Ending Sidewalk
![Page 82: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/82.jpg)
TURKER LABELING EXAMPLES
Surface Problems
Object in Path
![Page 83: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/83.jpg)
TURKER LABELING EXAMPLES
Surface Problems
Object in Path
![Page 84: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/84.jpg)
And now some negative examples…
![Page 85: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/85.jpg)
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
No Curb Ramp
![Page 86: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/86.jpg)
No Curb Ramp
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
Incorrect Object in Path label. Stop
sign is in grass.
![Page 87: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/87.jpg)
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
No problems in this image
![Page 88: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/88.jpg)
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
![Page 89: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/89.jpg)
T1 T2 T3 Maj. Vote
3 Turker Majority Vote Label
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
T3 provides a label of low quality
![Page 90: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/90.jpg)
To look into the effect of turker majority vote
on accuracy, we actually had 28 turkers label
each image
![Page 91: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/91.jpg)
28 groups of 1:
We had 28 turkers
label each image:
![Page 92: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/92.jpg)
28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
![Page 93: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/93.jpg)
28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
![Page 94: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/94.jpg)
28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
![Page 95: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/95.jpg)
28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
4 groups of 7:
3 groups of 9:
![Page 96: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/96.jpg)
Multiclass Classification
Problem Object in Path
Curb Ramp Missing
R1 R2 R3
Sidewalk Ending
Surface Problem
Researcher Maj. Vote
Turker
Correct
Correct
Correct
Wrong
Binary Classification
![Page 97: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/97.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image Level Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Multiclass
Accuracy
![Page 98: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/98.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
![Page 99: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/99.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Accuracy saturates
after 5 turkers
![Page 100: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/100.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Stderr: 0.2% Stderr=0.2%
![Page 101: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/101.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Binary
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
![Page 102: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/102.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Binary
Accuracy 1 L
ab
el Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 L
ab
els
Problem
![Page 103: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/103.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
80.6%
86.9% 89.7% 90.6% 90.2%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Binary
Accuracy 1 L
ab
el Problem
![Page 104: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/104.jpg)
Primary Study 3.2 Question
Can turkers validate other turkers’ label to filter
out mistakes?
Validators Labelers
![Page 105: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/105.jpg)
Kotaro Hara
Timer: 00:07:00 of 3 hours
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
3 hours 10
![Page 106: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/106.jpg)
After quality control, accuracy increased from
81% to 93%
![Page 107: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/107.jpg)
Limitations and Future Work
![Page 108: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/108.jpg)
Increase Scalability
Build a automated crawler to collect images
from Street View
Allow turkers to “walk” and control camera
angle in Street View and label sidewalk
accessibility problems
Volunteer website
![Page 109: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/109.jpg)
Computer Vision to Automate Accessibility Attribute Detection
![Page 110: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/110.jpg)
SVM based Sliding Window Approach
![Page 111: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/111.jpg)
Accessibility Aware Navigation System
![Page 112: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/112.jpg)
Summary
makeability lab
81% labeling accuracy with no quality control
93% labeling accuracy with quality control
A Google Faculty Research Award
kindly sponsored this work.
![Page 113: Combining Crowdsourcing and Google Street View to Identify … · 2016-03-08 · Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems Kotaro](https://reader033.fdocuments.net/reader033/viewer/2022060404/5f0edff07e708231d4415d06/html5/thumbnails/113.jpg)
Questions?
Kotaro Hara | @kotarohara_en
Victoria Le | [email protected]
Jon E. Froehlich | @jonfroehlich
makeability lab
Combining Crowdsourcing and Google Street View
to Identify Street-level Accessibility Problems
A Google Faculty Research Award
kindly sponsored this work.