Gestures Matter

Gestures Matter observing remote collaborationpresented by: Marie SilverstrimNEXT SLIDE 20 years ago music120 some years ago Music

Music: Every Rose Has Its Thorn by PoisonVogue by MadonnaIce Ice Baby by Vanilla Ice

NEXT SLIDE 20 yrs ago Movies

References:http://www.musicstack.com/album/poison/every+rose+has+it's+thornhttp://www.musicstack.com/item/168457115http://music.yahoo.com/vanilla-ice/albums/best-of-vanilla-ice-platinum-disc--189120896

220 some years ago Movies

Movies: Pretty WomanBatmanHoney I shrunk the KidsBack to the Future II

NEXT SLIDE 20 yrs ago Politics

References:http://www.imdb.com/title/tt0096895/http://www.imdb.com/title/tt0096874/http://www.imdb.com/title/tt0100405/http://www.imdb.com/title/tt0097523320 some years ago Politics

We just had an election last week so who was elected 20 years ago?

Politics:President: George Bush Senior, 1989First Gulf War, 1990-91Americans with Disabilities Act, 1990

NEXT SLIDE 20 yrs ago Technology

References:http://www.whitehouse.gov/about/presidents/georgehwbushhttp://humanwarvelous.blogspot.com/2011/05/gulf-war-1991-1-17-1991-2-28-this-war.htmlhttp://disabilitysavvy.com/services/ada-assessments/

4

20 some years ago Technology

And now, finally, technology, where were we?

Technology:Sonys discman made music more portableExcept when they skippedPhones had just become mobilecar phones because too big to actually carryWWW fathered by Tim Berners-LeeHTML, HTTP, URL (1989-1991)

NEXT SLIDE Honey I shrunk the world

References:http://www.radiomuseum.org/r/sony_discman_d_131.html http://blog.wellscti.com/blog/bid/29537/If-you-build-it-he-will-come-The-First-WELLS-CTI-Blog http://appleinsider.com/articles/07/10/18/road_to_mac_os_x_leopard_safari_3_0.html

5Honey, I shrunk the world.The need to be in the same place as your team is diminishing

Teams can begin working remotely

New issues of remote collaboration come into the picture

SO, all these things are pointing to people doing more things out and about - Taking calls away from home - Looking up data online instead of an encylcopedia - Global issues coming closer to home

What does this mean for us in HCI?How does it affect our working environments?

NEXT SLIDE Gestures. Whered that come from?

6So. Gestures Whered that come from?Sara BlySpecializes in qualitative, observational techniquesManaged the Collaborative Systems Group at Xerox PARC 1986-1994Currently owns Sara Bly Consulting in Portland, Or

MA in Education from Stanford UnivPhD in Comp Science from UC DavisBA in Math from Univ of Kansas

Sara By Interested in collaborationobservational techniquesworking at PARC right when everything is going remote So, She does a study.

Sara BlyMA in Education from Stanford UnivPhD in Comp Science from UC DavisBA in Math from Univ of Kansas

Specializes in qualitative, observational techniquesCurrently owns Sara Bly Consulting in Portland, OregonManaged the Collaborative Systems Group at Xerox PARC 1986-1994

NEXT SLIDE 20 yrs ago A Use of Drawing Surfaces in Different Collaborative Settings

7A Use of Drawing Surfaces in Different Collaborative Settings, Bly 1988http://tiny.cc/d36bmwHypothesize: How do you think the actions of: Gesturing, Writing, and Drawing will vary between interaction modes:Face to Face, Media link, and Telephone?

Study consisted of 3 types of 2 person drawing sessions Face to FaceSeparated with Media linkSeparated with Telephone linkObservations suggest that the process may be as important as the final productPull designers togetherContribute to verbal communicationIncrease focusFurther research neededSo, she studied how interactions varied depending on how the interaction occurred- Face to Face- Media Link (video)- Telephone

STOP! HYPOTHOSIZE!!! - Same use? Who has most/least gestures? - What do we think the difference will be between Gestures for video and telephone?- Who has most/least Drawing? Writing?SHOW RESULTSTalk about Hypothesis vs Study Results

Blys observations.Further Research Needed: - to understand the role of drawing space activities in the design process - how that might be accommodated by computer-supported collaborative drawing tools.

Recognized John Tang in the acknowledgements

NEXT SLIDE Further research was done by Tang (PRONOUNCED TONG)

FULL URL:http://dl.acm.org/citation.cfm?id=62266.62286&coll=DL&dl=GUIDE&CFID=176888124&CFTOKEN=83842534

8And further research was done.

John C. TangWorking with Bly at PARC while completing his PhDBS, MS, PhD in Design Division of Mech. Eng. from Stanford Univ.Interested in research for:Supporting distributed collaborationUnderstanding how users integrate across devices to accomplish tasksAt Microsoft Research since 2008Previously at IBM Research, Sun Microsystems, Inc.Tang was working with Bly during her researchSo, he went on and completed his PhD on this group collaboration concept

Since then, kept up interest in the area, and currently working at Microsoft Research

NEXT SLIDE Who was paying attention to Tang?9Saul GreenbergFull professor of Comp. Sci. at Univ of Calgary, CanadaEducation:BS Microbiology and ImmunologyDiploma of Education, 1978MS Comp. Sci., 1984 Univ of CalgaryPhD Comp. Sci., 1989 Univ of CalgaryLeads GroupLab at U of CalgaryFocuses on HCI, UbiComp, CSCWResearch has common thread of situated interaction

Who was paying attention?

Saul Greenberg and his team were paying attentionThey built their own tool, based on Tangs findings

Numerous Awards: holds the NSERC/iCORE/Smart Technologies Industrial Chair in Interactive Technologies a University Professorship - a distinguished University of Calgary award recognizing research excellence received the CHCCS Achievement award in May 2007 was also elected to the prestigious ACM CHI Academy in April 2005 for his overall contributions to the field of HCI

Common thread ofsituated interaction: which considers how computer technology fits within the fabric of peoples day to day activities. includes: how such technology blends naturally in the flow of peoples work practices how people socialize and work together through technology how that technology fits within peoples physical environment

NEXT SLIDE Lets See Greenbergs version10So lets see it (Dr. Greenbergs version)http://www.youtube.com/watch?v=yeqAa6nbz1w0:52 3:45

http://www.youtube.com/watch?v=yeqAa6nbz1w0:52 3:45

Implemented Tangs pointsBUT, gestures limited to pointing, and EMPHATIC pointing, not as rich

NEXT SLIDE But, before we look at Tangs research, one more thing, What is Collaboration?11But first what is Collaboration?Definition: To work jointly with others or jointly especially in an intellectual endeavormerriam-webster.com/dictionary/collaborateIs that a good definition? Is it missing anything?(Denning & Yaholkovsky 2008)What about Cooperation? Is that the same as Collaboration?Websters definition

Questions: Is that a good definition for our purposes? What about other words, Cooperation, Coordination? This was also brought up in the other reading

NEXT SLIDE Collaboration vs other words definition page

Reference:Denning, P.J. & Yaholkovsky, P., 2008. Getting to we. Communications of the ACM, 51(4), p.19. Available at: http://portal.acm.org/citation.cfm?doid=1330311.1330316.12Collaboration. What it is.(Denning & Yaholkovsky 2008)Information SharingCoordinationCooperationCollaborationExchanging data and messagesRegulating people and things for common understandingPlaying together in the same game under the same agreed upon rulesCreating solutions through the synergistic actions of a groupBetter explanation of Collaboration.Not a definition, but draws lines between the wordsMain point, Collaboration is CREATING TOGETHER

NEXT SLIDE Board game example

Denning, P.J. & Yaholkovsky, P., 2008. Getting to we. Communications of the ACM, 51(4), p.19. Available at: http://portal.acm.org/citation.cfm?doid=1330311.1330316.13Collaboration. What it is.(Denning & Yaholkovsky 2008)Information SharingCoordinationCooperationCollaborationExchanging data and messagesRegulating people and things for common understandingPlaying together in the same game under the same agreed upon rulesCreating solutions through the synergistic actions of a groupreading the instructions on the board gameresolving who is going first, who is the banker, clearing up house rulesactually playing the board game by the house rulescombining two games together for a brand new gameSo, what exactly does that mean?

Board game analogyBut, since were talking about computers, lets put it in those terms

NEXT SLIDE Technology examples

Denning, P.J. & Yaholkovsky, P., 2008. Getting to we. Communications of the ACM, 51(4), p.19. Available at: http://portal.acm.org/citation.cfm?doid=1330311.1330316.14Collaboration. And what its not.Information SharingCoordinationCooperationCollaborationExchanging data and messagesRegulating people and things for common understandingPlaying together in the same game under the same agreed upon rulesCreating solutions through the synergistic actions of a groupemailblogschatsfile sharingproject mgmt auctionsshopping cartsonline paymentwikisdiscussion boardsSecond Lifebrainstormingcharrattesconsensus workshopStrauss method(Denning & Yaholkovsky 2008)Technology examples of each of the break downs.

Focus on Collaboration:Brainstorming: we all knowCharrattes: break into sub groups, who discuss and report back to larger group divide and conqueorConsensus Workshop:Set the context - introduce the focus questionBrainstorm the ideasCluster the ideasName the idea clustersReview and action.Strauss Method: Looking at data to come up with the theory, interview to test theory (sampling), compare sample w theory

NEXT SLIDE Back to Tangs Research

Longer description of Charrettescharrettes often take place in multiple sessions in which the group divides into sub-groups. Each sub-group then presents its work to the full group as material for future dialogue. Such charrettes serve as a way of quickly generating a design solution while integrating the aptitudes and interests of a diverse group of people

Strauss Method: Looking at data to come up with the theoryGrounded theory Approach:theoretical sensitive coding: generating theoretical strong concepts from the data to explain the phenomenon researchedtheoretical sampling: deciding whom to interview or what to observe next according to the state of theory generation, and that implies starting data analysis with the first interview, and writing down memos and hypotheses earlycomparebetween phenomena and contexts to make the theory strong15Back to Tangs Research: BasicsGoal: Understand collaborative workIdentify resources and hindrancesGuide development of tools accordinglySetup:Video groups of 3-4 people working on a design project around paper or whiteboardAnalyze video for actions and functions of those actionsMake observationsWhat is todays natural environment for collaboration?Tang wanted to continue Blys observations:Better understand collaborative work processIdentify the situation and THEN guide toolsSetup:Tang wanted a Natural environment setupGroups around paper, or whiteboardVideoed the interaction to analyze later

Question:What is TODAYs NATURAL environment? Is it still paper and white board? What else?

NEXT SLIDE What was the key data that Tang was looking for when analyzing the data?

16J. C. Tangs Research: Key DataFunctions to collaborate: Store information = note takingExpress ideas = sharingMediate interaction = moderating

Actions performed:Listing = no spatial importanceDrawing = sketching, may include textGesturing = context mattersMatrix of functions and actions to perform them

NEXT SLIDE How did Tang organize the data?

17

J. C. Tangs Research: AnalysisAnalysis: Hows the breakdown of actions and functions? Is this enough? Too much? How do you define purposeful gestures? Is yawning bored or tired? Clicking pen angry, thinking, or nothing at all?How would adding qualitative measurements change the observations? useful versus non useful?How does the background of the group affect the gestures used?- Thanks to LulaSO, How did Tang analyis all this data in the videos? How did he simplify it down to something measurable?He had this handy dandy table. What do we think of the table?

Questions:How is the breakdown of actions and functions? Enough? Too much? Missing anything?

How do you define purposeful gestures? Is yawning bored or tired? Does it matter?

How would adding qualitative measurements change the observations? useful vs non useful Can we know?

Does the background of the group affect how they interact with each other? Are the basics still there?

NEXT SLIDE Actual results found

18J. C. Tangs Research: Results

40% express60% mediate20% store 60% express 20% mediate 100% store

66% list33% draw66% draw33% gesture33% draw66% gestureRegardless what we think of it, this is what he usedAnd he found that Lists are boring. They store info, and not much else

So, lets look at drawing and gesturingDrawing is VERY richGesturing is mostly mediation, but substantially expressive. Apparently we all play charades a lot.

So why did VIDEO have so LITTLE gesturing?How do we mediate when we dont have drawing and gesturing to help?

NEXT SLIDE So, putting these findings into words

19J. C. Tangs Research: FindingsGesturesUsed for mediation and expression of ideasAll gestures need contextBut theres no record, cant refer back to a gestureCan be used for storage via mimicking

DrawingMostly for expression of ideas, also store & mediationCreating artifacts is problematic (recording gestures)Timing matters, creating and discussion of graphicsContext still matters who created the mark?Is no implementation of gestures and drawing better than badly done implementation?I caught a fish THIS big!Gestures are really important. Statement like I caught a fish THIS big has no meaning without gestures. Tone helps a little, but still not clearNO RECORD of gestures. If you miss it, or forget it, its gone

Drawing:

Creating artifacts is problematic: while one person is writing down what the group just decided upon, the recorder is either left behind in the discussion, or the entire group is waiting for the recorder to finish. Disrupts the flow of the idea generation process

Timing matters: If I draw something and say we need to add THIS and THIS and draw something else, and your computer froze, the whole process just fell apart. Audio and picture needs to match

Questions: Is no implementation of gestures and drawings better than badly done implementations?If youre watching a conference where the video and audio are slightly off, do you stop watching or listening or both?

NEXT SLIDE More findings

20J. C. Tangs Research: FindingsFluent Intermingling of ActionsFor research purposes, actions and functions are separated, but in reality, multiple actions for single function, and rapid switching between actionsMediationTurn taking cued by proximity and gesturesParallel working decreases bottlenecks and focusContext sets public and private spaceSpatial OrientationSets ownership and audience, public vs private spaceHow many people is too many people for collaboration? Does it change for F2F vs Virtual?Fluent Intermingling. People dont stop and change modes, most actions have multiple meanings

Mediation is important for smooth flow

Spatial Orientation is important, gives private vs public perspective

Questions: At what number of people does collaboration stop working? Is that number the same for physical space and virtual space?

NEXT SLIDE Take Aways

21J. C. Tangs Research: Take AwaysObserved needs for a Collaborative workspaceHand gestures, in context, convey informationProcess of creating drawings conveys information beyond the final productProximity and simultaneous access to work space are key to mediating the groups interactionFluently intermix actions and functions in work spaceNeed common view AND spatial orientation in work space

Was this news then? Is this news now?Do we use these findings well in development today?List of observations: 5 of them:

Questions: Was this news 20 years ago? Is it news today?

Do we use these findings well, or at all? in development today? - shosmer (shannon?)Within the HCI community?Within Industry / Corporate world?

NEXT SLIDE Why do we care?

22J. C. Tangs Research: Why do we care?Laid foundations for role of human factors in shared visual spacesNeed to include insights to human factors in system development

Provided an early example of robust groupware developmentBegin with observations of actual processLook beyond the activity and end resultAccording to Greenberg, this is why we care- include insights of human factors in system development- look at ACTUAL PROCESS, not just end result or activity in name

NEXT SLIDE IN SHORT

23In Short.Design based on [human factors] should transcend technology in terms of the collaborative experience it offers the group.Saul GreenbergDESIGN based on human factors should TRANSCEND TECHNOLOGY in terms of the collaborative EXPERIENCE it offers to the group

Any comments on this quote?

DONE WITH TANG

NEXT SLIDE 2007. A MERE 5 years ago

24

2007, Just a mere 5 years ago

Fergie Big girls dont cry

Spiderman 3

43 president George W Bush

BEST Cell phone by CNET: Top Cell Phone was AT&T HTC TILTThe good:The AT&T Tilt features a slide-out full QWERTY keyboard and a tilting screen. The Windows Mobile 6 smartphone also offers the full gamut of wireless options, including 3G and GPS, push e-mail, a 3-megapixel camera, and support for AT&TMusic and Video.The bad:Speakerphone quality wasn't the greatest, and talk-time battery life was on the shorter side. The Tilt was sluggish at times, and picture quality was subpar. We also had problems acquiring a GPS fix.The bottom line:The AT&T Tilt promises to be the carrier's most powerful smartphone for business users with its full range of wireless options, Windows Mobile 6, and innovative tilt screen.

BEST MP3: IPOD CLASSICVideo screen 80 Gb

NEXT SLIDE How does this paper on sign language relate to group collaboration?

References:http://www.whitehouse.gov/about/presidents/georgewbushhttp://www.imdb.com/title/tt0413300/http://www.uulyrics.com/music/fergie/song-big-girls-dont-cry-personal-album-version/http://reviews.cnet.com/smartphones/at-t-tilt/4505-6452_7-32638455.html25How does this other paper relate to group collaboration?Research follows principles demonstrated by J. Tang, highlighted by S. Greenberg:Need to include insights to human factors in system developmentBegin with observations of actual processPerformed iterative research incorporating feedback from Deaf communityGestures arent just helpful for a deaf person using an interpreter, they are essentialPrinciples demonstrated by Tang and highlighted by Saul are implemented- started with actual process- iterative research- feedback- included human factors in the system design

In Tangs paper, Gestures were helpful, but you could get by without them. Mediation and expression may be harder, or not as richBut for deaf community, Gestures made by interpreters are not just NICE TO HAVE, they are ESSENTIAL for communication

NEXT SLIDE Who did the research? UNC team

26

University of North Carolina team

Karl GyllstromPhD in CS,UNCNow CS post doc researcherKatholieke Universiteit Leuven - NetherlandsDavid StottsCS Prof, UNCAdvisor for Millers PhDInterests: CSCW,Collaborative UIDorian Miller2009 PhD in CS, UNCDissertation on this topicNow works at IBM RTPJames CulpMBA, UNC

Dorian Miller did his PhD disseration on this, and is now working at IBM Research Triangle Park in Raleigh-Durham areaDorian Millers full dissertation:https://www.cs.unc.edu/cms/publications/dissertations/miller.pdf

Karl Gyllstrom was also involved, hes now doing post doc research in the Netherlands

David Stotts was the advisor for Millers dissertation, hes involved in CSCW and Collaborative UI

James Culp, kind of odd man out, he got his MBA from UNC

NEXT SLIDE Their research: Semi-Transparent Video Interfaces to Assist Deaf Persons in Meetings

27Semi-transparent Video Interfaces to Assist Deaf Persons in MeetingsWhen a person focuses on one source of data, misses other sources of dataparticularly bad for deaf persons watching an interpreter

Investigates overlays of semi-transparent video useful for note takinguseful for video conferencinguseful for aiding communication between hearing and deaf persons without an interpreterWhat is the problem?

When you focus on one piece of data, you miss another piece of dataproblem for everyone, but particularly bad for deaf persons watching an interpreter

Investigates overlay of semi transparent videoAllows user to see two views at once

NEXT SLIDE How bad could it be?

28How bad could it be?http://www.youtube.com/watch?v=20HjI9UNPKA04:25 - 5:25

Think about:Listen for detail information, imagine taking notesHow much would you lose by looking away?Lydia Callis doing ASL interpretation for Mayor Bloomberg in NYC for Hurricane Sandy

Watch video, thinking about the detail of information involved, and imagine trying to take notes of important infoHOW MUCH DO YOU LOSE BY LOOKING AWAY?

http://www.youtube.com/watch?v=20HjI9UNPKA4:25 - :52, pause play second bit with muting for note taking up through 5:25

Discusion from the Think abouts

NEXT SLIDE Group Meeting Scenario problem

29Group Meeting Scenario problemMultiple areas to focus on: interpreter, presenter, slides, taking notescan use periphery vision, but if locations are too far apart, no gooddeaf person is cut off from conversation when taking notesProblem:

Cant focus on multiple things at once if they are too far apart

Deaf person is cut off from conversation when not watching the interpreter

NEXT SLIDE SOLUTION

30

Projects interpreter onto laptopCan take notes while still following conversationGood bitsUser can set size, transparency, and recordIndependent: can point camera wherever neededTransparent or OpaqueBad bitsTradeoff with real estate and details of videoRecording takes up large amount of memoryEye contact between interpreter and deaf person lostGroup Meeting Interface SolutionAny other Good or Bad bits that should be highlighted? Features that could highlight or help fix those items?Solution: Project the interpreter onto the laptop

Good and Bad bits to this solution

GOOD -> user has control of the display, both in what is captured, and how it is displayed on the screenNEEDS HELP -> tradeoff of real estate used vs detail of video, recoding is a memory hog, eye contact is being lost

Question: Anything else that stands out to you? Features that could highlight or fix issues?- what about syncing note taking to the video? - Jen

NEXT SLIDE Personal Meeting Scenario problem

31Personal Meeting Scenario problemSwitching between application and video display is problematicvideo display great for 2 deaf persons to communicate through sign languagewhen switch to work in an application, all communication is inferred from mouse movementPersonal Meeting

Video conferencing is great, but what happens when you switch to non-person view?Again, communication is cut off, or resorts to chat, interpreting mouse movements

NEXT SLIDE SOLUTION

32FaceTop application developed at UNCImages of both parties are overlaid on top of desktopGood: Can watch persons gestures while using an applicationBad: Details still lost in semi transparency, washed-out lookGesturing away from camera is imprecise, handled by showing both video feedsPersonal Meeting Interface Solution

Same question:Any other Good or Bad bits that should be highlighted?

Features that could highlight or help fix those items?Facetop application

BOTH people are projected in addition to the applications, can see everything at once

Again, Good and Bad bits to this solution

GOOD -> users can watch the other person and application at the same time. Much richer interactionNEEDS HELP -> details lost in semi transparency, and gesturing away from monitor to be caught by the camera is imprecise

Question: Slightly different scenario, but same questionAnything else that stands out to you? Features that could highlight or fix issues?

NEXT SLIDE Feedback from Deaf Community

33Feedback from Deaf communityIteratively tested with various persons and communitiesGood:Useful in situations where student taking own notes helps, like math classesUseful in situations with multiple deaf persons meeting and trying to transcribe minutesUseful in situations with copy signingUseful for video conferencing, could comfortably and successfully complete task

So, the UNC team was good and did iterative testing with various persons and communities

Good things found: - see slide, self explanatory

NEXT SLIDE Needs improvement34Feedback from Deaf communityIteratively tested with various persons and communitiesNeeds Improvement:Semi transparent video diffuses application and strains eyesLocating the cameras problematic, consider central camera with streamed output as a room featureInterpreters and deaf persons give each other feedback through eye contact, lost with videoHad to finger spell slower to allow for video rates

And things needed improvement:

Semi transparent diffusing is hard on eyesLoss of eye contact, impersonalizes communicationVideo issues -> camera placement, video rates for finger spelling

NEXT SLIDE Why do we as an HCI community care?35Why do we care?Gestures are lost in space for everyone

Better collaborative groupware

Utilizing existing technology in new ways

Extension to online capabilities?- Thanks to Jen and DougIs this a crutch or a tool? Does changing the audience change the answer?- Thanks to DaryaIs there a point of information overload?- Thanks to several folksGestures are lost for everyone so, capturing them is good, better collaborative groupware ideas for everyone

The idea of utilizing technology in different new ways. Borrowing from each other and tweaking is good.

Questions:

Extension to online capabilities Jen and Doug

Is this a tool or a crutch? Does the audience change the answer? - Darya

NEXT SLIDE THE END! Thank you!36

The End.Thank you!

THE END.37