Jeffrey P. BighamUniversity of Washington CSEjbigham@cs.washington.edu

Enabling Blind Users to Blaze Trails Through the Web

Tessa Lau and Jeffrey NicholsIBM Almaden Research

{tessalau, jwnichols}@us.ibm.com

start

Scale of Access

37 Million Blind People in the World(125 million with low vision)

Increasing Aging Population Situational Impairments

Access technology benefits 25% of web users1

[1] “The Market for Accessible Technology. “ Forrester Research, 2003.

Web Potential for Blind People

Web Advantages Convertible to Accessible Form Shop From Home

Blind People Read 7x as Many Books

The Law U.S. Government Requires Accessibility1,2

NFB vs. Target Corporation2

Making Access Possible is Not Enough Should be Intuitive and Fun

[1] Section 508 of the U.S. Rehabilitation Act.[2] Americans with Disabilities Act.

X

X

X

X

h

h

h

e b b

START

Keyboard Shortcuts

X

X

X

X

<span class=“heading”>

e b b

START

<span class=“heading”>

<span class=“heading”>

.heading { font-weight: bold; }

Keyboard Shortcuts

Task: Placing Book on Hold

1. Enter “Book You Want”

2. Click “Find” Button

3. Find Correct Result

4. Click “Place Hold” Button

5. Click “Request” Button

6. Verify Queue Position

Task: Placing Book on Hold

1. Enter “Book You Want”

2. Click “Find” Button

3. Find Correct Result

4. Click “Place Hold” Button

5. Click “Request” Button

6. Verify Queue Position

Task: Placing Book on Hold

1. Enter “Book You Want”

2. Click “Find” Button

3. Find Correct Result

4. Click “Place Hold” Button

5. Click “Request” Button

6. Verify Queue Position

Task: Placing Book on Hold

1. Enter “Book You Want”

2. Click “Find” Button

3. Find Correct Result

4. Click “Place Hold” Button

5. Click “Request” Button

6. Verify Queue Position

How can blind people more effectively complete web tasks?

Avoid Linear Searches Direct Users to Important Content

Users Maintain Control Interface should be understandable Guide users not direct them

Flexible and Adaptable Work in full web domain

Compatibility with Tools Users KnowSupplement to existing tools (screen readers)

CoScripter1:Recording and Replaying Tasks

Pseudo-Natural Language Commands

[1] Little et al. “Koala: capture, share, automate, personalize business processes on the web.” CHI 2007.

1. go to “http://www.united.com”2. enter your “flight number” (e.g., 0168) into the “Flight number” textbox3. enter your “airport” (e.g., SJC) into the “From:” textbox4. click the “Check” button

CoScript Repository

Example

TrailBlazerPlacing a book on hold at the Seattle Public Library.

Set Focus & Provide Contextual Help

Step 2 of 15: select “Books” from the “Search” listbox

“Search” Listbox

“Previous Step” Button

“Play from Here” Button

“Next Step” Button

Starts Here

...

...

Step description.

Inline PlaybackControls

Formative Evaluation

Generalizing with Suggestions

Choose from one of thefollowing suggestions:

Click the “Check” button.Enter into the “From:” textbox.Enter into the “To:” textbox.Select from the “Date:” listbox.Select from the “Time:” listbox.

Providing Suggestions

User enters a short task description (query) “flight status on united” “place hold on book at Seattle library”

(etc.)

Machine learning model ranks all actions.

FeaturesTask DescriptionRelated Scripts from the Script RepositoryUser’s History of Actions

Apply existing scripts to new tasks or new sitesNo need to explicitly find script first

Click the “profile” link.Click the “Home” link.Click the “Books” link.…Enter into the “Name” textbox.Enter into the “Zip” textbox.…Click the “Submit” button.Click the “Check Now” button.…Turn on the “Yes” Radio Button.….

Example Action List

x1 x2 x6

…

y

0.080.270.100.760.20

Generalizing Scripts to New Sites

“flight status on united”Task Description

Repository

1. go to “www.aa.com”2. enter your “Flight Number” (e.g. 144) into the “Flight Number:” textbox3. click the fifth “GO” button

Flight Status on American Airlines

1. go to “http://www.southwest.com/”2. select “SJC” from the “Select the Departure City:” listbox3. enter your “flight number” into the first “Enter the Flight Number” textbox4. click the first “View Flight Status Information” button

Flight Status on Southwest Airlines

Click “my profile” linkClick “Flight Information” linkEnter into “Mileage Plus” textboxEnter into “flight number” textboxClick “Check” button

Possible Actions

similarity( , )

Click “my profile” linkEnter into ‘password’ password boxClick “Login” buttonEnter into “flight number” textboxClick “Check” button

0.080.880.100.360.20

Using History to Match Sub-TasksLast Action

Repository

Update Facebook Status

Reserve UW Meeting Room

Possible Actions

1. go to “www.facebook.com”2. enter your “facebook username” into the “username” textbox3. enter your “facebook password” into the password box4. Click “Login” button5. …

7. …8. enter your “CSE login” into the “username” textbox9. enter your “CSE password” into the password box10. …

0.000.870.460.140.04

0.000.750.420.13

“enter into ‘username’ textbox”

similarity( , )

Evaluation

Tested on 102 Actions 15 Most Popular CoScripts User-provided title used as task description 6.8 steps per script 42 possible actions

Leave-One-Out Cross-Validation Removed from repository too

Top 5 in 76% of cases

CoScriptRepository(5.1 words on avg.)

Suggestions Provided

Ongoing and Future Work

User Studies What happens when TrailBlazer is wrong? Incentives vs. Cost

Better Model Continuous Learning Popularity, Past Behavior, etc.

Improved Web Access Voice Over Mobile Phones Small Screens

Conclusion

TrailBlazer makes web access more efficient Record, replay and share trails Targets non-visual use Uses existing repository of trails

Suggestion Component Finds appropriate trails Generalizes existing scripts to new tasks and sites

The End

Thanks to:Allen Cypher, Clemens Drews, Jimmy Lin, Yevgen Borodin, Charles Chen and T.V. Raman.

Want to hire me?

jbigham@cs.washington.eduwww.jeffreybigham.com

Directing Users to Content

Direct to Content Avoid inefficient linear searches

5 Participants/20 Regions Mechanical Turk

3 Commonalities Semantic Desciption (78%) Heading of Region (53%) Words Used in Region (37%)

Clip the “search results” region

Directing Users to Content

Direct to Content Avoid linear searches

5 Descriptions of 20 Regions Mechanical Turk 27 Users

Clip the “search results” region

Clip the region containing “batting average”

0.080.270.100.760.20

Generalizing Scripts to New Sites

“flight status on united”Task Description

Repository

1. go to “www.aa.com”2. enter your “Flight Number” (e.g. 144) into the “Flight Number:” textbox3. click the fifth “GO” button

Flight Status on American Airlines

1. go to “http://www.southwest.com/”2. select “SJC” from the “Select the Departure City:” listbox3. enter your “flight number” into the first “Enter the Flight Number” textbox4. click the first “View Flight Status Information” button

Flight Status on Southwest Airlines

Click “my profile” linkClick “Flight Information” linkEnter into “Mileage Plus” textboxEnter into “flight number” textboxClick “Check” button

Possible Actions Actions a Scripts s1: enter1: flight1: number1: textbox

2: button1: city2: click1: departure3: enter4: flight3: go1: information1: listbox4: number1: select1: SJC1: status2: textbox1: view2: your

Similarity = cos(a, s)

Click “my profile” linkEnter into ‘password’ password boxClick “Login” buttonEnter into “flight number” textboxClick “Check” button

0.080.880.100.760.20

Matching Partial ScriptsLast Action

Repository

Update Facebook Status

Reserve UW Meeting Room

Possible Actions Action a

Scripts s

1: enter2: password1: box

box: 0.9button: 0.1click: 0.0CSE: 1.2enter: 2.5facebook: 1.2go: 0.0login: 0.1password: 1.8textbox: 1.6

W(si) = d * W(si-1) + cos(t, si)

1. go to “www.facebook.com”2. enter your “facebook username” into the “username” textbox3. enter your “facebook password” into the password box4. Click “Login” button5. …

7. …8. enter your “CSE login” into the “username” textbox9. enter your “CSE password” into the password box10. …

0.30 * 0.000.30 * 0.00 + 0.870.30 * 0.87 + 0.200.30 * 0.46 + 0.000.30 * 0.14 + 0.00

0.30 * 0.00 + 0.000.30 * 0.00 + 0.750.30 * 0.75 + 0.200.30 * 0.43 + …

0.000.870.460.140.04

0.000.750.420.13

d = 0.30

“enter into ‘username’ textbox”