Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine...
-
Upload
jon-froehlich -
Category
Technology
-
view
538 -
download
1
description
Transcript of Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine...
![Page 1: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/1.jpg)
Human Computer Interaction Laboratory
makeability lab
CHARACTERIZING PHYSICAL WORLD ACCESSIBILITY AT SCALE USING CROWDSOURCING, COMPUTER VISION, & MACHINE LEARNING
![Page 2: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/2.jpg)
My Group Started in 2012
![Page 3: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/3.jpg)
![Page 4: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/4.jpg)
Human-Computer Interaction Lab
![Page 5: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/5.jpg)
![Page 6: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/6.jpg)
BenShneiderman BenBederson JonFroehlich JenGolbeck LeahFindlater
MarshiniChetty
JennyPreece
AllisonDruin MonaLeighGuh
a TammyClegg JuneAhn EvanGolub TimClausner KentNorman
IraChinoy
KariKraus
AnneRose CatherinePlaisa
nt
computer
science
hcil
JessicaVitak
NiklasElmqvist NicholasDiakopoulos
![Page 7: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/7.jpg)
@jonfroehlich Assistant Professor Computer Science
Making in the HCIL
@jonfroehlich
Assistant Professor
Computer Science
31st HCIL Symposium
May 29, 2014
HCIL Hackerspace Founded in 2012
![Page 8: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/8.jpg)
HCIL Hackerspace Looking North
![Page 9: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/9.jpg)
HCIL Hackerspace Looking South
![Page 10: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/10.jpg)
Three Soldering Stations HCIL Hackerspace
![Page 11: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/11.jpg)
Craft/Textile Station HCIL Hackerspace
![Page 12: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/12.jpg)
Two Mannequins HCIL Hackerspace
![Page 13: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/13.jpg)
Wall of Electronic Components HCIL Hackerspace
![Page 14: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/14.jpg)
Quadcopters HCIL Hackerspace
![Page 15: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/15.jpg)
Two 3D-Printers HCIL Hackerspace
![Page 16: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/16.jpg)
One CNC Machine HCIL Hackerspace
![Page 17: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/17.jpg)
Physical Making HCIL Student Leyla Norooz
![Page 18: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/18.jpg)
Electronics Making HCIL student Tansy McBurnie
![Page 19: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/19.jpg)
E-Textile Design HCIL Student Michael Gubbels showing SFF
![Page 20: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/20.jpg)
Collaborative Working HCIL students Joseph, Cy, Matt, and Jonah
![Page 21: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/21.jpg)
Student Sewing HCIL student Matt sewing
![Page 22: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/22.jpg)
Fun! HCIL students Kotaro Hara and Allan Fong
![Page 23: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/23.jpg)
More Fun! HCIL students Sean, Michael, Alexa, and me
![Page 24: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/24.jpg)
Human-Computer Interaction Lab
![Page 25: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/25.jpg)
Human Computer Interaction Laboratory
makeability lab
CHARACTERIZING PHYSICAL WORLD ACCESSIBILITY AT SCALE USING CROWDSOURCING, COMPUTER VISION, & MACHINE LEARNING
![Page 26: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/26.jpg)
30.6 million U.S. adults with mobility impairment
![Page 27: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/27.jpg)
15.2 million use an assistive aid
![Page 28: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/28.jpg)
Incomplete Sidewalks Physical Obstacles Surface Problems No Curb Ramps Stairs/Businesses
![Page 29: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/29.jpg)
![Page 30: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/30.jpg)
The National Council on Disability noted that
there is no comprehensive information on
“the degree to which sidewalks are
accessible” in cities.
National Council on Disability, 2007
The impact of the Americans with Disabilities Act: Assessing
the progress toward achieving the goals of the ADA
![Page 31: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/31.jpg)
The lack of street-level
accessibility information can
have a significant impact on
the independence and
mobility of citizens
cf. Nuernberger, 2008; Thapar et al., 2004
![Page 32: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/32.jpg)
I usually don’t go where I don’t
know [about accessible routes] -P3, congenital polyneuropathy
![Page 33: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/33.jpg)
https://www.flickr.com/photos/johnhanleyphoto/5991029700/sizes/l
“Man in Wheelchair Hit By Vehicle
Has Died From Injuries”
-The Aurora, May 9, 2013
![Page 34: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/34.jpg)
http://youtu.be/gWuryTNRFzw
![Page 35: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/35.jpg)
![Page 36: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/36.jpg)
http://accesscore.org
This is a mockup interface based on walkscore.com and walkshed.com
![Page 37: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/37.jpg)
http://accesscore.org
This is a mockup interface based on walkscore.com and walkshed.com
![Page 38: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/38.jpg)
![Page 39: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/39.jpg)
How might a tool like AccessScore:
Change the way people think about and
understand their neighborhoods
Influence property values
Impact where people choose to live
Change how governments/citizens make
decisions about infrastructural investments
![Page 40: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/40.jpg)
AccessScore would not change how people navigate
the city, for this we need a different tool…
![Page 41: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/41.jpg)
NAVIGATION TOOLS ARE NOT ACCESSIBILITY AWARE
![Page 42: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/42.jpg)
Routing for: Manual Wheelchair
1st of 3 Suggested Routes 16 minutes, 0.7 miles, 1 obstacle
!
!
!
!
A
B
Route 1 Route 2
Surface Problem Avg Severity: 3.6 (Hard to Pass)
Recent Comments: “Obstacle is passable in a manual chair but not in a motorized chair”
Routing for: Manual Wheelchair
A
1st of 3 Suggested Routes 16 minutes, 0.7 miles, 1 obstacle
!
Click to rate severity
ACCESSIBILITY AWARE NAVIGATION SYSTEMS
![Page 43: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/43.jpg)
![Page 44: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/44.jpg)
Where is this data going to come from?
![Page 45: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/45.jpg)
Safe Routes to School Walkability Audit Rock Hill, South Carolina
Walkability Audit Wake County, North Carolina
Walkability Audit Wake County, North Carolina
TRADITIONAL WALKABILITY AUDITS
![Page 46: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/46.jpg)
Safe Routes to School Walkability Audit Rock Hill, South Carolina
Walkability Audit Wake County, North Carolina
Walkability Audit Wake County, North Carolina
TRADITIONAL WALKABILITY AUDITS
![Page 47: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/47.jpg)
http://www1.nyc.gov/311/index.page
MOBILE REPORTING SOLUTIONS
![Page 48: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/48.jpg)
http://www1.nyc.gov/311/index.page
MOBILE REPORTING SOLUTIONS
![Page 49: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/49.jpg)
![Page 50: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/50.jpg)
Similar to physical audits, these tools are built for
in situ reporting and do not support remote,
virtual inquiry—which limits scalability
Not designed for accessibility data collection
![Page 51: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/51.jpg)
MARK & FIND ACCESSIBLE BUSINESSES
wheelmap.org axsmap.com
![Page 52: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/52.jpg)
MARK & FIND ACCESSIBLE BUSINESSES
wheelmap.org axsmap.com
Focuses on businesses
rather than streets &
sidewalks
Model is still to report
on places you’ve
visited
![Page 53: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/53.jpg)
Our Approach: Use Google Street View (GSV) as a massive data source
for scalably finding and characterizing street-level accessibility
![Page 54: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/54.jpg)
HIGH-LEVEL RESEARCH QUESTIONS
1. Can we use Google Street View (GSV) to find street-
level accessibility problems?
2. Can we create interactive systems to allow minimally
trained crowdworkers to quickly and accurately
perform remote audit tasks?
3. Can we use computer vision and machine learning
to scale our approach?
![Page 55: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/55.jpg)
ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
TOWARDS SCALABLE ACCESSIBILITY DATA COLLECTION
![Page 56: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/56.jpg)
ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
![Page 57: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/57.jpg)
ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
![Page 58: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/58.jpg)
ASSETS’12 GOALS:
1. Investigate viability of reapproprating online map
imagery to determine sidewalk accessibility via
crowd workers
2. Examine the effect of three different interactive
labeling interfaces on task accuracy and duration
![Page 59: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/59.jpg)
WEB-BASED LABELING INTERFACE
![Page 60: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/60.jpg)
WEB-BASED LABELING INTERFACE FOUR STEP PROCESS
1. Find and mark accessibility problem 2. Select problem category
3. Rate problem severity 4. Submit completed image
![Page 61: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/61.jpg)
WEB-BASED LABELING INTERFACE VIDEO
Video shown to crowd workers before they labeled their first image
http://youtu.be/aD1bx_SikGo
![Page 62: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/62.jpg)
WEB-BASED LABELING INTERFACE VIDEO
http://youtu.be/aD1bx_SikGo
![Page 63: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/63.jpg)
THREE LABELING INTERFACES
Point-and-click Rectangular Outline Polygonal Outline
Pixel Granularity
![Page 64: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/64.jpg)
Los Angeles
DATASET: 100 IMAGES
New York
Baltimore
Washington DC
![Page 65: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/65.jpg)
DATASET BREAKDOWN
34 29 27 11 19 0
10
20
30
40
No Curb Ramp Surface Problem Object in Path Sidewalk Ending No Sidewalk Accessibility Issues
Manually curated 100 images from urban neighborhoods in LA, Baltimore, Washington DC, and NYC
![Page 66: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/66.jpg)
DATASET BREAKDOWN
34 29 27 11 19 0
10
20
30
40
No Curb Ramp Surface Problem Object in Path Sidewalk Ending No Sidewalk Accessibility Issues
Manually curated 100 images from urban neighborhoods in LA, Baltimore, Washington DC, and NYC
Used to evaluate false positive
labeling activity
![Page 67: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/67.jpg)
DATASET BREAKDOWN
34 29 27 11 19 0
10
20
30
40
No Curb Ramp Surface Problem Object in Path Sidewalk Ending No Sidewalk Accessibility Issues
Manually curated 100 images from urban neighborhoods in LA, Baltimore, Washington DC, and NYC
Used to evaluate false positive
labeling activity
This breakdown based on majority vote data from 3 independent researcher labels
![Page 68: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/68.jpg)
Our ground truth process
![Page 69: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/69.jpg)
What accessibility problems exist in this image?
![Page 70: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/70.jpg)
R1 R2 R3
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 71: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/71.jpg)
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
![Page 72: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/72.jpg)
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
x2
Researcher 1
![Page 73: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/73.jpg)
x4
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
Researcher 2
![Page 74: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/74.jpg)
Researcher 3
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
x8
![Page 75: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/75.jpg)
Researcher 1
Researcher 2
Researcher 3
![Page 76: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/76.jpg)
There are multiple ways to examine the labels.
![Page 77: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/77.jpg)
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table Image Level Analysis
This table tells us what accessibility
problems exist in the image
![Page 78: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/78.jpg)
Pixel Level Analysis
Labeled pixels tell us where
the accessibility problems
exist in the image.
![Page 79: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/79.jpg)
Why do we care about image level vs. pixel level?
![Page 80: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/80.jpg)
Coarse Precise
Point Location
Level
Sub-block
Level
Block
Level (Pixel Level) (Image Level)
![Page 81: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/81.jpg)
Coarse Precise
Point Location
Level
Sub-block
Level
Block
Level (Pixel Level) (Image Level)
![Page 82: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/82.jpg)
Coarse Precise
Point Location
Level
Sub-block
Level
Block
Level (Pixel Level) (Image Level)
Pixel level labels could be used for
training machine learning algorithms
for detection and recognition tasks
![Page 83: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/83.jpg)
Coarse Precise Localization
Spectrum
Point Location
Level
Sub-block
Level
Block
Level
Class
Spectrum
Multiclass Object in Path
Curb Ramp Missing
Prematurely Ending Sidewalk
Surface Problem
Binary Problem No Problem
(Pixel Level) (Image Level)
TWO ACCESSIBILITY PROBLEM SPECTRUMS Different ways of thinking about accessibility problem labels in GSV
Coarse Precise
![Page 84: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/84.jpg)
Object in Path
Curb Ramp Missing
R1 R2 R3
Researcher Label Table
Problem
Multiclass label Binary Label
Sidewalk Ending
Surface Problem
Other
![Page 85: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/85.jpg)
To produce a single ground truth dataset, we used majority vote.
![Page 86: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/86.jpg)
R1 R2 R3 Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 87: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/87.jpg)
R1 R2 R3 Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
After you talk about majority vote labels in this slide, fade
in labels that are currently shaded.
![Page 88: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/88.jpg)
R1 R2 R3 Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
![Page 89: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/89.jpg)
ASSETS’12 MTURK STUDY METHOD Independently posted 3 labeling interfaces to MTurk.
Crowdworkers could work with only one interface.
For training, turkers required to watch first 1.5 mins of
3-min instructional video.
Hired ~7 workers per image to explore avg accuracy
Turkers paid ~3-5 cents per HIT. We varied number of
images/HIT from 1-10.
![Page 90: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/90.jpg)
ASSETS’12 MTURK DESCRIPTIVE RESULTS Hired 132 unique workers
Worked on 2,325 assignments
Provided a total of 4,309 labels (AVG=1.9/image)
![Page 91: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/91.jpg)
MAIN FINDINGS: IMAGE-LEVEL ANALYSIS
0%
20%
40%
60%
80%
100%
Point-and-click Outline Rectangle
AVERAGE ACCURACY Higher is better
0
10
20
30
40
50
Point-and-click Outline Rectangle
MEDIAN TASK TIME (SECS) Lower is better
![Page 92: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/92.jpg)
MAIN FINDINGS: IMAGE-LEVEL ANALYSIS
83.0% 82.6% 79.2%
0%
20%
40%
60%
80%
100%
Point-and-click Outline Rectangle
AVERAGE ACCURACY
All three interfaces performed similarly. This is without quality control.
0
10
20
30
40
50
Point-and-click Outline Rectangle
MEDIAN TASK TIME (SECS) Higher is better Lower is better
![Page 93: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/93.jpg)
MAIN FINDINGS: IMAGE-LEVEL ANALYSIS
83.0% 82.6% 79.2%
0%
20%
40%
60%
80%
100%
Point-and-click Outline Rectangle
32.9
41.5 43.3
0
10
20
30
40
50
Point-and-click Outline Rectangle
AVERAGE ACCURACY MEDIAN TASK TIME (SECS)
All three interfaces performed similarly. This is without quality control.
Point-and-click is the fastest; 26% faster than Outline & 32% faster than Rectangle
Higher is better Lower is better
![Page 94: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/94.jpg)
ASSETS’12 CONTRIBUTIONS:
1. Demonstrated that minimally trained crowd workers
could locate and categorize sidewalk accessibility
problems in GSV images with > 80% accuracy
2. Showed that point-and-click fastest labeling interface
but that outline faster than rectangle
![Page 95: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/95.jpg)
ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
![Page 96: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/96.jpg)
ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
![Page 97: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/97.jpg)
CHI’13 GOALS:
1. Expand ASSETS’12 study with larger sample.
• Examine accuracy as function of turkers/image
• Evaluate quality control mechanisms
• Gain qualitative understanding of failures/successes
2. Validate researcher ground truth with labels from
three wheelchair users
![Page 98: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/98.jpg)
Los Angeles
DATASET: EXPANDED TO 229 IMAGES
New York
Baltimore
Washington DC
![Page 99: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/99.jpg)
CHI’13 GOALS:
1. Expand ASSETS’12 study with larger sample.
• Examine accuracy as function of turkers/image
• Evaluate quality control mechanisms
• Gain qualitative understanding of failures/successes
2. Validate researcher ground truth with labels from
three wheelchair users
![Page 100: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/100.jpg)
GROUND TRUTH: MAJORITY VOTE 3 RESEARCHER LABELS
How “good” is our ground truth?
![Page 101: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/101.jpg)
![Page 102: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/102.jpg)
IN-LAB STUDY METHOD Three wheelchair participants
Independently labeled 75 of 229 GSV images
Used think-aloud protocol. Sessions were video recorded
30-min post-study interview
We used Fleiss’ kappa to measure agreement between
wheelchair users and researchers
![Page 103: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/103.jpg)
Here is an example recording from the study session
![Page 104: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/104.jpg)
![Page 105: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/105.jpg)
IN-LAB STUDY RESULTS Strong agreement (κmulticlass=0.74) between wheelchair
participants and researcher labels (ground truth)
In interviews, one participant mentioned using GSV to
explore areas prior to travel
![Page 106: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/106.jpg)
CHI’13 GOALS:
1. Expand ASSETS’12 study with larger sample.
• Examine accuracy as function of turkers/image
• Evaluate quality control mechanisms
• Gain qualitative understanding of failures/successes
2. Validate researcher ground truth with labels from
three wheelchair users
![Page 107: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/107.jpg)
CHI’13 GOALS:
1. Expand ASSETS’12 study with larger sample.
• Examine accuracy as function of turkers/image
• Evaluate quality control mechanisms
• Gain qualitative understanding of failures/successes
2. Validate researcher ground truth with labels from
three wheelchair users
![Page 108: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/108.jpg)
CHI’13 MTURK STUDY METHOD Similar to ASSETS’12 but more images (229 vs. 100)
and more turkers (185 vs. 132)
Added crowd verification quality control
Recruited 28+ turkers per image to investigate
accuracy as function of workers
![Page 109: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/109.jpg)
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
Kotaro Hara
Timer: 00:07:00 of 3 hours
10 3 hours
Labeling Interface
![Page 110: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/110.jpg)
Kotaro Hara
Timer: 00:07:00 of 3 hours
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
3 hours 10
Verification Interface
![Page 111: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/111.jpg)
Kotaro Hara
Timer: 00:07:00 of 3 hours
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
3 hours 10
Verification Interface
![Page 112: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/112.jpg)
CHI’13 MTURK LABELING STATS Hired 185 unique workers
Worked on 7,517 labeling tasks (AVG=40.6/turker)
Provided a total of 13,379 labels (AVG=1.8/image)
Hired 273 unique workers
Provided a total of 19,189 verifications
CHI’13 MTURK VERIFICATION STATS
![Page 113: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/113.jpg)
CHI’13 MTURK LABELING STATS Hired 185 unique workers
Worked on 7,517 labeling tasks (AVG=40.6/turker)
Provided a total of 13,379 labels (AVG=1.8/image)
CHI’13 MTURK VERIFICATION STATS Hired 273 unique workers
Provided a total of 19,189 verifications
Median image labeling time vs. verification time: 35.2s vs. 10.5s
![Page 114: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/114.jpg)
CHI’13 MTURK KEY FINDINGS 81% accuracy without quality control
93% accuracy with quality control
![Page 115: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/115.jpg)
Some turker labeling successes...
![Page 116: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/116.jpg)
Turker Labels Curb Ramp Missing
TURKER LABELING EXAMPLES
Curb Ramp Missing
![Page 117: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/117.jpg)
Turker Labels Curb Ramp Missing
TURKER LABELING EXAMPLES
Curb Ramp Missing
![Page 118: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/118.jpg)
TURKER LABELING EXAMPLES
Object in Path
![Page 119: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/119.jpg)
TURKER LABELING EXAMPLES
Object in Path
![Page 120: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/120.jpg)
TURKER LABELING EXAMPLES
Prematurely Ending Sidewalk
![Page 121: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/121.jpg)
TURKER LABELING EXAMPLES
Prematurely Ending Sidewalk
![Page 122: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/122.jpg)
TURKER LABELING EXAMPLES
Surface Problems
![Page 123: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/123.jpg)
TURKER LABELING EXAMPLES
Surface Problems
![Page 124: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/124.jpg)
TURKER LABELING EXAMPLES
Surface Problems
Object in Path
![Page 125: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/125.jpg)
TURKER LABELING EXAMPLES
Surface Problems
Object in Path
![Page 126: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/126.jpg)
And now some turker failures…
![Page 127: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/127.jpg)
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
No Curb Ramp
![Page 128: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/128.jpg)
No Curb Ramp
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
Incorrect Object in Path label. Stop
sign is in grass.
![Page 129: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/129.jpg)
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
Surface Problems
![Page 130: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/130.jpg)
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
Surface Problems
Tree not actually
an obstacle
![Page 131: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/131.jpg)
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
No problems in this image
![Page 132: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/132.jpg)
TURKER LABELING ISSUES
Overlabeling Some Turkers Prone to High False Positives
![Page 133: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/133.jpg)
T1 T2 T3 Maj. Vote
3 Turker Majority Vote Label
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
T3 provides a label of low quality
![Page 134: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/134.jpg)
To look into the effect of turker majority vote on
accuracy, we had 28 turkers label each image
![Page 135: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/135.jpg)
28 groups of 1:
We had 28 turkers
label each image:
![Page 136: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/136.jpg)
28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
![Page 137: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/137.jpg)
28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
![Page 138: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/138.jpg)
28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
![Page 139: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/139.jpg)
28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
4 groups of 7:
3 groups of 9:
![Page 140: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/140.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image-Level Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Multiclass
Accuracy
![Page 141: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/141.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image-Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
![Page 142: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/142.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image-Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Accuracy saturates
after 5 turkers
![Page 143: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/143.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image-Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Stderr: 0.2% Stderr=0.2%
![Page 144: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/144.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image-Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Binary
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
![Page 145: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/145.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image-Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Binary
Accuracy 1 L
ab
el Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
4 L
ab
els
Problem
![Page 146: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/146.jpg)
78.3%
83.8%
86.8% 86.6% 87.9%
80.6%
86.9%
89.7% 90.6% 90.2%
50%
60%
70%
80%
90%
100%
1 turker (N=28) 3 turkers (N=9) 5 turkers (N=5) 7 turkers (N=4) 9 turkers (N=3)
Ave
rage
Imag
e-le
vel A
ccur
acy
(%)
Error bars: standard error
Image-Level Accuracy
Multiclass
Accuracy
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem 4 L
ab
els
Binary
Accuracy 1 L
ab
el Problem
![Page 147: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/147.jpg)
81.2%
85.8% 88.1% 89.3%
91.8% 92.7% 90.7%
50%
60%
70%
80%
90%
100%
EVALUATING QUALITY CONTROL MECHANISMS Image-Level, Binary Classification
1 labeler 1 labeler, 3 verifiers
(majority vote)
1 labeler, 3 verifiers
(zero tolerance)
3 labelers (majority vote)
3 labelers (majority vote)
3 verifiers (majority vote)
3 labelers (majority vote)
3 verifiers (zero tolerance)
5 labelers (majority vote)
![Page 148: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/148.jpg)
81.2%
85.8% 88.1% 89.3%
91.8% 92.7% 90.7%
50%
60%
70%
80%
90%
100%
EVALUATING QUALITY CONTROL MECHANISMS Image-Level, Binary Classification
1 labeler 1 labeler, 3 verifiers
(majority vote)
1 labeler, 3 verifiers
(zero tolerance)
3 labelers (majority vote)
3 labelers (majority vote)
3 verifiers (majority vote)
3 labelers (majority vote)
3 verifiers (zero tolerance)
5 labelers (majority vote)
3 labelers + 3 verifiers = 93%
![Page 149: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/149.jpg)
CHI’13 CONTRIBUTIONS:
1. Extended and reaffirmed findings from ASSETS’12
about viability of GSV and crowd work for locating
and categorizing accessibility problems
2. Validated our ground truth labeling approach
3. Assessed simple quality control approaches
![Page 150: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/150.jpg)
ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
![Page 151: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/151.jpg)
ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
![Page 152: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/152.jpg)
All of the approaches so far relied purely on manual
labor, which limits scalability
![Page 153: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/153.jpg)
& Manual Labor Computation
![Page 154: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/154.jpg)
Automatic Workflow Adaptation for Crowdsourcing Lin et al. 2012; Dai et al. 2011 ; Kamar et al. 2012
![Page 155: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/155.jpg)
Computer Vision & Streetview Goodfellow et al., 2014; Chu et al., 2014; Naik et al., 2014
![Page 156: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/156.jpg)
Tohme 遠目 Remote Eye ・
![Page 157: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/157.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・ Design Principles
1. Computer vision is cheap (zero cost)
2. Manual verification is far cheaper than manual labeling
3. Automatic curb ramp detection is hard and error prone
4. Fixing a false positive is easy, fixing a false negative is
hard (requires manual labeling).
![Page 158: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/158.jpg)
![Page 159: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/159.jpg)
The “lack of curb cuts is a primary
obstacle to the smooth integration
of those with disabilities into the
commerce of daily life.”
-P3, congenital polyneuropathy
Kinney et al. vs. Yerusalim & Hoskins, 1993
3rd Circuit Court of Appeals
![Page 160: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/160.jpg)
“Without curb cuts, people with
ambulatory disabilities simply
cannot navigate the city”
-P3, congenital polyneuropathy
Kinney et al. vs. Yerusalim & Hoskins, 1993
3rd Circuit Court of Appeals
![Page 161: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/161.jpg)
Dataset
svDetect Automatic Curb
Ramp Detection
svCrawl Web Scraper
Tohme 遠目 Remote Eye ・
![Page 162: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/162.jpg)
svCrawl Web Scraper
svDetect Automatic Curb
Ramp Detection
Tohme 遠目 Remote Eye ・
Curb Ramp Detection on
Street View image
False positives False negatives = missed curb ramps
![Page 163: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/163.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
Tohme 遠目 Remote Eye ・
![Page 164: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/164.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
Tohme 遠目 Remote Eye ・
![Page 165: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/165.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
Tohme 遠目 Remote Eye ・
svVerify can only fix false positives, not false negatives! That is, there is no way for a worker to add new labels at this stage!
![Page 166: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/166.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 167: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/167.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
.
![Page 168: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/168.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 169: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/169.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
Complexity: Cardinality:
Depth: CV:
0.14 0.33 0.21 0.22
![Page 170: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/170.jpg)
svVerify
Manual Label
Verification
Tohme 遠目 Remote Eye ・
Complexity: Cardinality:
Depth: CV:
0.14 0.33 0.21 0.22
Predict presence of false
negatives with linear SVM
and Lasso regression
svCrawl Web Scraper
Dataset
svLabel Manual Labeling
![Page 171: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/171.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svVerify
Manual Label
Verification
Tohme 遠目 Remote Eye ・
Complexity: Cardinality:
Depth: CV:
0.82 0.25 0.96
0.54
![Page 172: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/172.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 173: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/173.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 174: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/174.jpg)
Google Street View Intersection Panoramas and GIS Metadata
3D Point-cloud Data
Top-down Google Maps Imagery
Scraper & Dataset
![Page 175: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/175.jpg)
Saskatoon
Los Angeles
Baltimore
Washington D.C.
Washington D.C.
Baltimore
Los Angeles
Saskatoon
![Page 176: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/176.jpg)
Washington D.C.
Dense urban area
Semi-urban residential areas
Scraper & Dataset
![Page 177: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/177.jpg)
Washington D.C. Baltimore Los Angeles Saskatoon
* At the time of downloading data in summer 2013
Scraper & Dataset
Total Area: 11.3 km2
Intersections: 1,086
Curb Ramps: 2,877
Missing Curb Ramps: 647
Avg. GSV Data Age: 2.2 yrs
![Page 178: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/178.jpg)
How well does GSV data reflect
the current state of the physical
world?
![Page 179: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/179.jpg)
![Page 180: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/180.jpg)
Vs. Vs.
![Page 181: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/181.jpg)
Washington D.C. Baltimore
Physical Audit Areas
GSV and Physical World
> 97.7% agreement
273 Intersections
Dataset | Validating Dataset
Small disagreement due to construction.
![Page 182: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/182.jpg)
Washington D.C. Baltimore
Physical Audit Areas
273 Intersections
> 97.7% agreement
Dataset
Key Takeaway Google Street View is a viable source of curb ramp data
![Page 183: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/183.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 184: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/184.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 185: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/185.jpg)
AUTOMATIC CURB RAMP DETECTION
1. Curb Ramp Detection
2. Post-Processing Output
3. SVM-Based Classification
![Page 186: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/186.jpg)
Deformable Part Models Felzenszwalb et al. 2008
Automatic Curb Ramp Detection
http://www.cs.berkeley.edu/~rbg/latent/
![Page 187: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/187.jpg)
Deformable Part Models Felzenszwalb et al. 2008
Automatic Curb Ramp Detection
http://www.cs.berkeley.edu/~rbg/latent/
Root filter Parts filter Displacement cost
![Page 188: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/188.jpg)
Automatic Curb Ramp Detection
Multiple redundant
detection boxes
Detected Labels Stage 1: Deformable Part Model
Correct 1
False Positive 12
Miss 0
![Page 189: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/189.jpg)
Automatic Curb Ramp Detection
Curb ramps shouldn’t be
in the sky or on roofs
Correct 1
False Positive 12
Miss 0
Detected Labels Stage 1: Deformable Part Model
![Page 190: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/190.jpg)
Automatic Curb Ramp Detection
Detected Labels Stage 2: Post-processing
![Page 191: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/191.jpg)
Automatic Curb Ramp Detection
Detected Labels Stage 3: SVM-based Refinement
Filter out labels based on
their size, color, and position.
Correct 1
False Positive 5
Miss 0
![Page 192: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/192.jpg)
Automatic Curb Ramp Detection
Correct 1
False Positive 3
Miss 0
Detected Labels Stage 3: SVM-based Refinement
![Page 193: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/193.jpg)
Automatic Curb Ramp Detection
Correct 6
False Positive 11
Miss 1
Detected Labels Stage 1: Deformable Part Model
![Page 194: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/194.jpg)
Automatic Curb Ramp Detection
Correct 6
False Positive 6
Miss 1
Detected Labels Stage 2: Post-processing
![Page 195: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/195.jpg)
Automatic Curb Ramp Detection
Correct 6
False Positive 4
Miss 1
Detected Labels Stage 3: SVM-based Refinement
![Page 196: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/196.jpg)
Some curb ramps
never get detected
False positive
detections
Automatic Curb Ramp Detection
Correct 6
False Positive 4
Miss 1
![Page 197: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/197.jpg)
Some curb ramps
never get detected
False positive
detections
Automatic Curb Ramp Detection
Correct 6
False Positive 4
Miss 1
These false negatives are expensive to correct!
![Page 198: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/198.jpg)
Used two-fold cross validation to evaluate CV sub-system
![Page 199: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/199.jpg)
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Pre
cisi
on (
%)
Recall (%)
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
Precision
Higher, less false positives
Recall
Higher, less false negatives
![Page 200: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/200.jpg)
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Pre
cisi
on (
%)
Recall (%)
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
Goal: maximize
area under curve
![Page 201: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/201.jpg)
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Pre
cisi
on (
%)
Recall (%)
Stage 1: DPM
Stage 2: Post-Processing
Stage 3: SVM
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
![Page 202: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/202.jpg)
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Pre
cisi
on (
%)
Recall (%)
Stage 1: DPM
Stage 2: Post-Processing
Stage 3: SVM
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
![Page 203: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/203.jpg)
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Pre
cisi
on (
%)
Recall (%)
Stage 1: DPM
Stage 2: Post-Processing
Stage 3: SVM
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
More than 20% of
curb ramps were
missed
![Page 204: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/204.jpg)
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Pre
cisi
on (
%)
Recall (%)
Stage 1: DPM
Stage 2: Post-Processing
Stage 3: SVM
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
Confidence
threshold of -0.99,
which results in
26% precision and
67% recall
![Page 205: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/205.jpg)
Occlusion Illumination
Scale Viewpoint Variation
Structures Similar to Curb Ramps Curb Ramp Design Variation
Automatic Curb Ramp Detection
CURB RAMP DETECTION IS A HARD PROBLEM
![Page 206: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/206.jpg)
Can we predict difficult intersections & CV performance?
![Page 207: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/207.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 208: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/208.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 209: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/209.jpg)
Automatic Task Allocation | Features to Assess Scene Difficulty for CV
Number of connected streets from metadata
Depth information for intersection complexity analysis
Top-down images to assess complexity of an intersection
Number of detections and confidence values
![Page 210: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/210.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 211: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/211.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 212: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/212.jpg)
3x
Manual Labeling | Labeling Interface
![Page 213: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/213.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 214: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/214.jpg)
svCrawl Web Scraper
Dataset
svDetect Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel Manual Labeling
Tohme 遠目 Remote Eye ・
![Page 215: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/215.jpg)
2x
Manual Label Verification
![Page 216: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/216.jpg)
2x
Manual Label Verification
![Page 217: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/217.jpg)
Automatic Detection and
Manual Verification Automatic Task Allocation
Can Tohme achieve equivalent or better accuracy at a lower
time cost compared to a completely manual approach?
![Page 218: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/218.jpg)
STUDY METHOD: CONDITIONS
Manual labeling without
smart task allocation
& vs.
CV + Verification without
smart task allocation
Tohme 遠目 Remote Eye ・
vs.
Evaluation
![Page 219: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/219.jpg)
Accuracy Task Completion Time
Evaluation
STUDY METHOD: MEASURES
![Page 220: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/220.jpg)
Recruited workers from Mturk
Used 1,046 GSV images (40 used for golden insertion)
Evaluation
STUDY METHOD: APPROACH
![Page 221: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/221.jpg)
RESULTS
Labeling Tasks Verification Tasks
# of distinct turkers: 242 161
1,270 582 # of HITs completed:
# of tasks completed: 6,350 4,820
# of tasks allocated: 769 277
Evaluation
We used Monte Carlo simulations for evaluation
![Page 222: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/222.jpg)
84% 88% 86%
0%
20%
40%
60%
80%
100%
Acc
ura
cy M
easu
res
(%)
Precision Recall F-measure
Manual
Labeling CV and Manual
Verification
&
94
0
20
40
60
80
100
Task
Co
mp
letio
n T
ime /
Sce
ne (s)
Manual
Labeling CV and Manual
Verification
& Tohme 遠目 Remote Eye ・
Tohme 遠目 Remote Eye ・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
ACCURACY COST (TIME)
![Page 223: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/223.jpg)
84%
68%
88%
58%
86%
63%
0%
20%
40%
60%
80%
100%
Acc
ura
cy M
easu
res
(%)
Precision Recall F-measure
Manual
Labeling CV and Manual
Verification
&
94
42
0
20
40
60
80
100
Task
Co
mp
letio
n T
ime /
Sce
ne (s)
Manual
Labeling CV and Manual
Verification
& Tohme 遠目 Remote Eye ・
Tohme 遠目 Remote Eye ・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
ACCURACY COST (TIME)
![Page 224: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/224.jpg)
84%
68%
83% 88%
58%
86% 86%
63%
84%
0%
20%
40%
60%
80%
100%
Acc
ura
cy M
easu
res
(%)
Precision Recall F-measure
Manual
Labeling CV and Manual
Verification
&
94
42
81
0
20
40
60
80
100
Task
Co
mp
letio
n T
ime /
Sce
ne (s)
Manual
Labeling CV and Manual
Verification
& Tohme 遠目 Remote Eye ・
Tohme 遠目 Remote Eye ・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
ACCURACY COST (TIME)
![Page 225: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/225.jpg)
84%
68%
83% 88%
58%
86% 86%
63%
84%
0%
20%
40%
60%
80%
100%
Acc
ura
cy M
easu
res
(%)
Precision Recall F-measure
Manual
Labeling CV and Manual
Verification
&
94
42
81
0
20
40
60
80
100
Task
Co
mp
letio
n T
ime /
Sce
ne (s)
Manual
Labeling CV and Manual
Verification
& Tohme 遠目 Remote Eye ・
Tohme 遠目 Remote Eye ・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
13% reduction
in cost
ACCURACY COST (TIME)
![Page 226: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/226.jpg)
svControl
Automatic
Task Allocation svVerify
Manual Label
Verification
svLabel Manual Labeling
Evaluation | Smart Task Allocator
~80% of svVerify tasks were correctly routed
~50% of svLabel tasks were correctly routed
![Page 227: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/227.jpg)
svControl
Automatic
Task Allocation svVerify
Manual Label
Verification
svLabel Manual Labeling
Evaluation | Smart Task Allocator
If svControl worked perfectly, Tohme’s cost would drop to 28% of a manually labelling approach alone.
![Page 228: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/228.jpg)
Example Labels from Manual Labeling
![Page 229: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/229.jpg)
Evaluation | Example Labels from Manual Labeling
![Page 230: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/230.jpg)
Evaluation | Example Labels from Manual Labeling
![Page 231: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/231.jpg)
Evaluation | Example Labels from Manual Labeling
![Page 232: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/232.jpg)
Evaluation | Example Labels from Manual Labeling
![Page 233: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/233.jpg)
Evaluation | Example Labels from Manual Labeling
![Page 234: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/234.jpg)
This is a driveway.
Not a curb ramp.
Evaluation | Example Labels from Manual Labeling
![Page 235: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/235.jpg)
Evaluation | Example Labels from Manual Labeling
![Page 236: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/236.jpg)
Evaluation | Example Labels from Manual Labeling
![Page 237: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/237.jpg)
Examples Labels from CV + Verification
![Page 238: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/238.jpg)
Raw Street View Image
Evaluation | Example Labels from CV + Verification
![Page 239: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/239.jpg)
False detection Automatic Detection
Evaluation | Example Labels from CV + Verification
![Page 240: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/240.jpg)
Automatic Detection + Human Verification
Evaluation | Example Labels from CV + Verification
![Page 241: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/241.jpg)
Automatic Detection
Evaluation | Example Labels from CV + Verification
![Page 242: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/242.jpg)
Automatic Detection + Human Verification
Evaluation | Example Labels from CV + Verification
![Page 243: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/243.jpg)
False verification
Automatic Detection + Human Verification
Evaluation | Example Labels from CV + Verification
![Page 244: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/244.jpg)
UIST’14 CONTRIBUTIONS:
1. First CV system for automatically detecting curb
ramps in images
2. Showed that automated methods could be used to
improve labeling efficiency for curb ramps
3. Validated GSV as a viable curb ramp dataset
![Page 245: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/245.jpg)
TOWARDS SCALABLE ACCESSIBILITY DATA COLLECTION
ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
Improve CV Algorithms
Public Website Release &
New UI
Temporal Tracking
Other Physical World Sampling
Techniques
The Future
![Page 246: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/246.jpg)
![Page 247: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/247.jpg)
8,209 Intersections in DC
![Page 248: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/248.jpg)
8,209 Intersections in DC
BACK OF THE ENVELOPE CALCULATIONS Manually labeling GSV with our custom interfaces
would take 214 hours
With Tohme, this drops to 184 hours
We think we can do better
Unclear how long a physical audit would take
![Page 249: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/249.jpg)
FUTURE WORK: COMPUTER VISION Context integration & scene understanding
3D-data integration
Improve training & sample size
Mensuration
![Page 250: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/250.jpg)
![Page 251: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/251.jpg)
![Page 252: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/252.jpg)
FUTURE WORK: FASTER LABELING & VERIFICATION INTERFACES
![Page 253: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/253.jpg)
FUTURE WORK: TRACK PHYSICAL ACCESSIBILITY CHANGES OVER TIME
![Page 254: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/254.jpg)
FUTURE WORK: ADDITIONAL SURVEYING TECHNIQUES
Transmits real-time imagery of
physical space along with
measurements
![Page 255: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/255.jpg)
THE CROWD-POWERED STREETVIEW ACCESSIBILITY TEAM!
Kotaro Hara Jin Sun Victoria Le Robert Moore Sean Pannella
Jonah Chazan David Jacobs Jon Froehlich
Zachary Lawrence
Graduate Student
Undergraduate
High School
Professor
![Page 256: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/256.jpg)
Flickr User: Pedro Rocha https://www.flickr.com/photos/pedrorocha/3627562740/
Flickr User: Brooke Hoyer https://www.flickr.com/photos/brookehoyer/14816521847/
Flickr User: Jen Rossey https://www.flickr.com/photos/jenrossey/3185264564/
Flickr User: Steven Vance https://www.flickr.com/photos/jamesbondsv/8642938765
Flickr User: Jorge Gonzalez https://www.flickr.com/photos/macabrephotographer/6225178809/
Flickr User: Mike Fraser https://www.flickr.com/photos/67588280@N00/10800029263//
PHOTO CREDITS
Flickr User: Susan Sermoneta https://www.flickr.com/photos/en321/344387583/
![Page 257: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/257.jpg)
This work is supported by:
Faculty Research Award
Human Computer Interaction Laboratory
makeability lab
![Page 258: Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning](https://reader033.fdocuments.net/reader033/viewer/2022052906/558926ebd8b42ab32f8b46e5/html5/thumbnails/258.jpg)
Human Computer Interaction Laboratory
makeability lab
CHARACTERIZING PHYSICAL WORLD ACCESSIBILITY AT SCALE USING CROWDSOURCING, COMPUTER VISION, & MACHINE LEARNING