Slides byYevgen Borodin

(slides adapted for Psych 384, 3/3/09)

Department of Computer Science, Stony Brook University

A Vision for a Universally Accessible Web

The web is designed for those who can filter out irrelevant information

Non-Visual Web BrowsingJaws, Windows Eyes, Hal

Serial audio interface

Shortcut-driven navigation in HTML DOM-tree

Navigation between links, headers, lines, etc.

Inaccessible images, links, multimedia, etc.

How blind people browse the webLandmarksSearchesList of linksHeadingsStructureSpeed of speechStatic sites [Bigham, et. al., ASSETS’07]

What is HearSay?Started:

Motivated by work in AIInformation extraction from web pagesProcess modelingAdded an audio interface

Now :Working with HKSB and HKNCCollaborate with IBM, UWSeveral faculty members, Ph.D., MS.,

undergrad

HearSay 3 Free!!!Multi-platformFocused on Web browsingFlexible multimodal interfaceSupports text-to-speech enginesSupports voice recognition engines

Improving navigation would make web browsing more efficient.Segment pagesIdentify patternsAdd 2D navigationSummarize content

All this helps, but...

Project Goals

Filter out irrelevant informationDiscover relevant informationProvide quick access to relevant content

Evaluate the usability of HearSayCompare HearSay to other screen readersDistribute a stable version of the program for

free 

ScenariosRelevancy in Ad-hoc Web Browsing

Relevancy when Web content changes

Relevancy in Online Transactions (e.g. shopping, paying bills)

But what is relevant?

Manual Annotations of Content (the user could tell us what is relevant)

Beginning of Main Content

Search Button

Needed for Manual Labeling:

Provide an interface for creating annotations

Store annotations in a database

Query the database when the page loads

Apply the metadata to the page

Provide an interface for reviewing the annotations

Collaborative authoring of accessibility metadata[Takagi et al, 2008]

Social network connecting end-users and volunteers [http://socialaccessibility.alphaworks.ibm.com/]

Accessibility Commons (AC) DB to store metadata[Kawanaka, Borodin et al, 2008]

Web-based infrastructure for sharing metadata

Benefits of Social Accessibility

Shortens the time for accessibility renovations

Supported: headings, ALT tags, and titles

Workshop at UW – formed a consortiumDefined the Accessibility Commons DB schemaIdentified the object addressing methods: XPath,

MD5, URI

Automatic labeling of content to support web transactions

Labeling contentNon-visual web transactions are difficult

Consider all problems with non-visual browsingNeed to locate relevant concepts (buttons and

links)

Relevant concepts are similar across websitesSome variations, e.g. “add to cart”, “add to bag”Different labels, e.g. “Search”, “Go”, “Find”Evolution of relevance and form over time

Or instead of annotating elements, the whole process could be automated. Ex: AT&T Log-in Page

AT&T: Account Overview Page

AT&T: Make a Payment Page

AT&T: Confirm Payment Details

Macro Recording InterfaceCreate a recording (non-)visuallySave recording with a descriptionVoice interface to replay the macro-recordingPage (in)dependenceCustomizing what is readSpecifying variables

Context-Directed Browsing

Something has Just Changed…?

Dynamic Web ContentDynamic content:

Our actions often cause changeWe pay attention to changes of contentNew information is often in the changesAffects relevancy of information

Types of updates:Page refresh, redirect, JavaScript and AJAX

updates

Source of updates:User-invoked and Timer-based

Another example:

Dynamic Content Paradigm

Treat any content changes as “updates”:AJAX, JavaScript, refresh, redirectNavigation by following linksUsing back and forward buttons

Analyze and diff the updated Web contentProvide interface for reviewing the changes

Page Refresh

Filtering Repeated Content

User-Centric GoalsDiscover and present relevant information

firstMinimize access-time to relevant informationKeep users focused on tasks and informationFacilitate multi-tasking and refocusingEnable automation of repetitive tasksKeep the context of user actionsMinimize system distractions

HearSay for the SightedBrowsing on handhelds

Browsing over the phone

Browsing on-the-go

Other services

Web Accessibility in Handhelds

Mobile Browsing ProblemsData Transfer Cost is High

Connection is Slow

Small Screens

Lots of Scrolling

Context-driven Browsing

External CollaboratorsAccessibility Group at IBM JapanAccessibility Group at Google

HKSB and HKNCArizona State University

Conferences: ASSETS, W4A, CSUN 

ConclusionWeb Accessibility is an important problemGlimpse of Interesting ApproachesMuch remains to be done:

E.g. Integration, Robustness (Specification and Verification)

SonificationOther modalities – touch, pen, ..Extensive end user studies to probe mental models

to drive technology development (feedback)Other Disabilities – cognitive, motor impairment,

etc.36

Questions?Comments?

Concerns?Suggestions?

www.cs.sunysb.edu/~hearsay

mailto:borodin@cs.sunysb.edu

www.cs.sunysb.edu/~borodin