The Glass Class Designing Wearable Interfaces May 1, CHI 2014 Mark Billinghurst HIT Lab NZ University of Canterbury mark.billinghurst@canterbury.ac.nz Hayes Raffle Glass team Google [x] hraffle@google.com

1: Introduction Mark Billinghurst Director of HIT Lab NZ, University of Canterbury PhD Univ. Washington Research on AR, mobile HCI, Collaborative Interfaces More than 250 papers in AR, VR, interface design Sabbatical in Glass team at Google [x] in 2013 Hayes Raffle Interaction Research Lead, Google Glass PhD MIT Media Lab Ran a couple of companies Launched a few products Won a few awards Published many papers in HCI Major changes in computing How do you Design for this? Course Goals In this course you will learn Introduction to head mounted wearable computers Understanding of current wearable technology Key design principles/interface metaphors Relevant human perceptual principles Rapid prototyping tools Areas for future research Hands on experience with the technology What You Wont Learn Low level programming Glass Mirror API, GDK, Vuzix SDK, etc Designing for non-HMD based interfaces Watches, fitness bands, etc How to develop wearable hardware optics, sensor assembly, etc Evaluation methods Experimental design, statistics, etc Schedule 9:00 Introduction (Mark + Hayes) 9:05 Overview/History (Mark) 9:20 Evolution and Design Principles (Hayes) 9:45 Prototyping Tools + Best Practices (Mark / Hayes) 10:20 Break/Demo 10:30 Concept Design Exercise (Design group / Hayes) 10:50 Wearable Technologies (Lecture group / Mark) 11:20 Design Presentations (Design group / Hayes) 11:50 Research Directions (Mark + Hayes) 12:20 Finish Design Group (33 People) If your name is on this list you are in the Design Group Yang Wang Konstantino Kapetaneas Preethi Srinivas Tony James Kate Vogt Aneesh Tarun Josh Andres Maria Maim Bram Reurings Luke Mill Tuck-Voon How M Gill Janaki Kumar Melinda Knight M Calkins Mike Tissenbaum Samantha Tse Kal McDowd Adora Tam Oscar Meruvia Mike Chen Anita Hoechtl Merlin Stone Ashoomi Dohlakia Icy Zhu Zdenek Mikovec Cristina Manresa-Yee Christian Winkler Angela Noh Amyris Fernandez Deborah Ptak Arne Renkema-Padmos Thomas Fritz Display Demos You Can Try Google Glass Display Glass UI, AR demos, Games, multimedia capture Vuzix M-100 Display Monocular display Epson BT-100, Epson BT-200 See through displays, Junaio markerless tracking Brother AirScouter display Projected see-through image Recon Snow Micro-display integrated into ski goggles CHI Wearables Exhibit Online at http://wcc.gatech.edu/exhibition 2: Overview/History A Brief History of Time Trend smaller, cheaper, more functions, more intimate Time pieces moved from public space onto the body 18th Century 20th Century 13th Century A Brief History of Computing Trend Smaller, cheaper, faster, more intimate Moving from fixed to handheld and onto body 1950s 1980s 1990s Room Desk Lap Hand Head What is a Wearable Computer ? A computer that is: Portable while operational Enables hands-free/hands-limited use Able to get the users attention Is always on, acting on behalf of the user Able to sense the users current context Rhodes, B. J. (1997). The wearable remembrance agent: A system for augmented memory. Personal Technologies, 1(4), 218-224. In Other Words .. A computer that is .. Eudaemonic: User considers it part of him/herself Existential: User has complete control of the system Ephemeral: System always operating at some level Mann, S. (1997). Wearable computing: A first step toward personal imaging. Computer, 30(2), 25-32. Wearable Computing Computer on the body that is: Always on Always accessible Always connected Other attributes Augmenting user actions Aware of user and surroundings Augmented Interaction Rekimoto, J., & Nagao, K. (1995, December). The world through the computer: Computer augmented interaction with real world environments. In Proceedings of the 8th annual ACM symposium on User interface and software technology (pp. 29-36). The Ideal Wearable Persists and Provides Constant Access: Designed for everyday and continuous user over a lifetime. Senses and Models Context: Observes and models the users environment, mental state, its own state. Augments and Mediates: Information support for the user in both the physical and virtual realities. Interacts Seamlessly: Adapts its input and output modalities to those most appropriate at the time. Starner, T. E. (1999). Wearable computing and contextual awareness (Doctoral dissertation, Massachusetts Institute of Technology). Wearable Attributes fafds History of Wearables 1960-90: Early Exploration Custom build devices 1990 - 2000: Academic, Military Research MIT, CMU, Georgia Tech, EPFL, etc 1997: ISWC conference starts 1995 2005+: First Commercial Uses Niche industry applications, Military 2010 - : Second Wave of Wearables Consumer applications, Head Worn Thorp and Shannon (1961) Wearable timing device for roulette prediction Audio feedback, four button input Ed Thorp Thorp, E. O. (1998, October). The invention of the first wearable computer. In Wearable Computers, 1998. Second International Symposium on (pp. 4-8). IEEE. Keith Taft (1972) Wearable computer for blackjack card counting Toe input, LED in Glasses for feedback Belt computer Shoe Input Glasses Display Steve Mann (1980s - ) http://wearcomp.org/ MIT Wearable Computing (1993-) http://www.media.mit.edu/wearables/ Enabling Technologies (1989+) Private Eye Display (Reflection Technologies) 720 x 280 dipslay Red LED Vibrating mirror Twiddler (Handykey) Chording keypad Mouse emulation MIT Tin Lizzy (1993) General Purpose Wearable Doug Platt, Thad Starner 150 MHz Pentium CPU 32-64 Mb RAM 6 Gb hard disk VGA display 2 PCMCIA slots Cellular modem http://www.media.mit.edu/wearables/lizzy/lizzy/index.html Thad Starner 1998 Early Wearable Computing Early Technology Computing Belt or Backpack Displays Head Mounted, LCD Panel, Audio Input Devices Chording Keyboard, Speech, Camera Networking Wireless LAN, Infra-Red, Cellular US Military Wearables (1989- ) Early experimentation 386 computer, VGA display GPS, mapping software Land Warrior (1991-) Integrated wearable system Camera, colour display, radio Navigation, reports, photos Zieniewicz, M. J., Johnson, D. C., Wong, C., & Flatt, J. D. (2002). The evolution of army wearable computers. IEEE Pervasive Computing, 1(4), 30-40. Wearables at CMU (19912000) Industry focused wearables Maintenance, repair Custom designed interface Dial/button input Rapid prototyping approach Industrial designed, ergonomic http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html Early Commercial Systems Xybernaut (1996 - 2007) Belt worn, HMD, 200 MHz ViA (1996 2001) Belt worn, Audio Interface 700 MHz Crusoe Symbol (1998 2006) Wrist worn computer Finger scanner Prototype Applications Remembrance Agent Rhodes (97) Augmented Reality Feiner (97), Thomas (98) Remote Collaboration Garner (97), Kraut (96) Maintenance Feiner (93), Caudell (92) Factory Work Thompson (97) Mobile AR: Touring Machine (1997) University of Columbia Feiner, MacIntyre, Hllerer, Webster Combines See through head mounted display GPS tracking Orientation sensor Backpack PC (custom) Tablet input Feiner, S., MacIntyre, B., Hllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217. MARS View Virtual tags overlaid on the real world Information in place Backpack/Wearable Systems 1997 Backpack Wearables Feiners Touring Machine AR Quake (Thomas) Tinmith (Piekarski) MCAR (Reitmayr) Bulky, HMD based Piekarski, W., & Thomas, B. (2002). ARQuake: the outdoor augmented reality gaming system. Communications of the ACM, 45(1), 36-38. PCI 3D Graphics Board Hard Drive Serial Ports CPU PC104 Sound Card PC104 PCMCIA GPS Antenna RTK correction Antenna HMD Controller Tracker Controller DC to DC Converter Battery Wearable Computer GPS RTK correction Radio Example self-built working solution with PCI-based 3D graphics Columbia Touring Machine Mobile AR - Hardware HIT Lab NZ Wearable AR (2004) Highly accurate outdoor AR tracking system GPS, Inertial, RTK system HMD First prototype Laptop based Video see-through HMD 2-3 cm tracking accuracy 2008: Location Aware Phones Nokia NavigatorMotorola Droid 2009 - Layar (www.layar.com) Location based data GPS + compass location Map + camera view AR Layers on real world Customized data Audio, 3D, 2D content Easy authoring Android, iPhone Wearable Evolution Backpack+HMD: 10+ kg Handheld + HMD Separate sensors .... UMPC 1.1GHz 1.5kg still >$5K Scale it down more: Smartphone$500 Integrated 0.1kg billions of units 1997 2003 2007 Google Glass (2011 - ) Hardware CPU TI OMAP 4430 1 Ghz 16 GB SanDisk Flash,1 GB Ram 570mAh Battery Input 5 mp camera, 720p recording, microphone GPS, InvenSense MPU-9150 inertial sensor Output Bone conducting speaker 640x360 micro-projector display Google Glass Specs Other Wearables Vuzix M-100 $999, professional Recon Jet $600, more sensors, sports Opinvent 500 Euro, multi-view mode Motorola Golden-i Rugged, remote assistance Ex: Recon Instruments Snow Ski display/computer Location, speed, altitude, phone headset http://www.reconinstruments.com/ Projected Market dsfh Summary Wearables are a new class of computing Intimate, persistent, aware, accessible, connected Evolution over 50 year history Backpack to head worn Custom developed to consumer ready device Enables new applications Collaboration, memory, AR, industry, etc Many head worn wearables are coming Android based, sensor package, micro-display Evolution + Design Principles Last year Last week NowForever The Now machine Focus on location, contextual and timely information, and communication. Why Glass? Leadership vision "Computing should just be more comfortable" "Google should do the hard work, and you should have a chance to live, have a good life, and get on with it." As technology becomes more personal and immediate, it can start to disappear. Distant Intimate Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Early prototyping Concept video Reality What we learned Transparent displays are tricky Colors are funny and inconsistent. You can only add light to a scene, not cover anything up. Motion can be disorienting. Clarity, contrast, brightness, visual field and attention are important. Reading Some things dont work Immersion Some things dont work Details Some things dont work Design principles The world is the experience Get the interface and interactions out of the way. Micro Interactions The position of the display and limited input ability makes longer interactions less comfortable. Using it shouldnt take longer than taking out your phone. A rear view mirror Don't overload the user. Stick to the absolutely essential, avoid long interactions. Be explicit. As personal as it gets Recognize and adapt to the user not the other way around. Glass is the most personal device you own. It operates closer to your most valuable senses and your environment, and it knows more about both. Glass should adapt, be personal and recognize the wearer; be aware of what consists his identity (physiology, memory, preferences, environment) and connect to it. As personal as it gets Glass is the most personal device you own. It should recognize and adapt to you not the other way around. For the closest people and most important moments Glass should be for prioritizing your closest people and creating value for the whole group, not just the wearer. Examples Search Picture Messaging PhoneNavigation Video CallVideo Platform How are people using Glass for creative expression? A world of stories In Viewpoint of Billions by David Datuna, Glass allows viewers to unlock images and video with interactive experiences. Social action First-person journalist Tim Pool broadcasts an intimate view of Istanbul protests. 'I want to show you what it's like to be there as best I can, even if that ends with me running full-speed into a cafe and rubbing lemons all over my face after being tear-gassed' Sharing and connecting Conductor Cynthia Johnston Turner shares a 1st person experience with her orchestra. Personal expression Alexander Chens Viola through Glass. Our tools are becoming more intimate and immediate. We can craft a future of learning, creative expression and empathy. Distant Intimate 4: Prototyping Tools How can we quickly prototype Wearable experiences with little or no coding? Why Prototype? Quick visual design Capture key interactions Focus on user experience Communicate design ideas Learn by doing/experiencing Prototyping Tools Static/Low fidelity Sketching User interface templates Storyboards/Application flows Interactive/High fidelity Wireframing tools Mobile prototyping Native Coding Important Note Most current wearables run Android OS eg Glass, Vuzix, Atheer, Epson, etc So many tools for prototyping on Android mobile devices will work for wearables If you want to learn to code, learn Java, Android, Javascript/PHP Typical Development Steps Sketching Storyboards UI Mockups Interaction Flows Video Prototypes Interactive Prototypes Final Native Application Increased Fidelity & Interactivity Sketched Interfaces Sketch + Powerpoint/Photoshop/Illustrator GlassSim http://glasssim.com/ Simulate the view through Google Glass Multiple card templates GlassSim Card Builder Use HTML for card details Multiple templates Change background Own image Camera view GlassSim Samples Glass UI Templates Google Glass Photoshop Templates http://glass-ui.com/ http://dsky9.com/glassfaq/the-google-glass-psd-template/ Sample Slides From Templates Application Storyboard http://dsky9.com/glassfaq/google-glass- storyboard-template-download/ ToolKit for Designers Vectoform Google Glass Toolkit for Designers http://blog.vectorform.com/2013/09/16/google-glass- toolkit-for-designers-2/ Sample cards, app flows, icons, etc Application Flow Limitations Positives Good for documenting screens Can show application flow Negatives No interactivity/transitions Cant be used for testing Cant deploy on wearable Can be time consuming to create Transitions Series of still photos in a movie format. Demonstrates the experience of the product Discover where concept needs fleshing out. Communicate experience and interface You can use whatever tools, from Flash to iMovie. Video Sketching See https://vine.co/v/bgIaLHIpFTB Example: Video Sketch of Vine UI UI Concept Movies Interactive Wireframes Interactive Wireframing Developing interactive interfaces/wireframes Transitions, user feedback, interface design Web based tools UXpin - http://www.uxpin.com/ proto.io - http://www.proto.io/ Native tools Justinmind - http://www.justinmind.com/ Axure - http://www.axure.com/ UXpin - www.uxpin.com Web based wireframing tool Mobile/Desktop applications Glass templates, run in browser https://www.youtube.com/watch?v=0XtS5YP8HcM Proto.io - http://www.proto.io/ Web based mobile prototyping tool Features Prototype for multiple devices Gesture input, touch events, animations Share with collaborators Test on device Proto.io - Interface Demo: Building a Simple Flow Gesture Flow Scr1 Scr2 Scr3 Scr4 Scr5 Scr6 Ta p Swipe Start Transitions Demo Justinmind - http://www.justinmind.com/ Native wireframing tool Build mobile apps without programming drag and drop, interface templates web based simulation test on mobile devices collaborative project sharing Templates for Glass, custom templates User Interface - Glass Templates Web Simulation Tool Comparing Wireframe Tools Tool Web Native Wearable Template Interaction Uxpin X X Proto.io X X Justinmind X X X Axure X X X Wireframe Limitations Cant deploy on Glass No access to sensor data Camera, orientation sensor No multimedia playback Audio, video Simple transitions No conditional logic No networking Processing for Wearables Processing Programming tool for Artists/Designers http://processing.org Easy to code, Free, Open source, Java based 2D, 3D, audio/video support Processing For Android http://wiki.processing.org/w/Android Strong Android support Generates Android .apk file Processing - Motivation Language of Interaction Sketching with code Support for rich interaction Large developer community Active help forums Dozens of plug-in libraries Strong Android support Easy to run on wearables http://processing.org/ http://openprocessing.org/ Development Enviroment Basic Parts of a Processing Sketch /* Notes comment */ //set up global variables float moveX = 50; //Initialize the Sketch void setup (){ } //draw every frame void draw(){ } Importing Libraries Can add functionality by Importing Libraries java archives - .jar files Include import code import processing.opengl.*; Popular Libraries Minim - audio library OCD - 3D camera views Physics - physics engine bluetoothDesktop - bluetooth networking http://toxiclibs.org/ Processing and Glass One of the easiest ways to build rich interactive wearable applications focus on interactivity, not coding Collects all sensor input camera, accelerometer, touch Can build native Android .apk files Side load onto Glass Example: Hello World //called initially at the start of the Processing sketch void setup() { size(640, 360); background(0); } //called every frame to draw output void draw() { background(0); //draw a white text string showing Hello World fill(255); text("Hello World", 50, 50); } Demo Hello World Image PImage img; // Create an image variable void setup() { size(640, 360); //load the ok glass home screen image img = loadImage("okGlass.jpg"); // Load the image into the program } void draw() { // Displays the image at its actual size at point (0,0) image(img, 0, 0); } Demo Touch Pad Input Tap recognized as DPAD input void keyPressed() { if (key == CODED){ if (keyCode == DPAD) { // Do something .. Java code to capture rich motion events import android.view.MotionEvent; Motion Event //Glass Touch Events - reads from touch pad public boolean dispatchGenericMotionEvent(MotionEvent event) { float x = event.getX(); // get x/y coords float y = event.getY(); int action = event.getActionMasked(); // get code for action switch (action) { // let us know which action code shows up case MotionEvent.ACTION_DOWN: touchEvent = "DOWN"; fingerTouch = 1; break; case MotionEvent.ACTION_MOVE: touchEvent = "MOVE"; xpos = myScreenWidth-x*touchPadScaleX; ypos = y*touchPadScaleY; break; Demo Sensors Ketai Library for Processing https://code.google.com/p/ketai/ Support all phone sensors GPS, Compass, Light, Camera, etc Include Ketai Library import ketai.sensors.*; KetaiSensor sensor; Using Sensors Setup in Setup( ) function sensor = new KetaiSensor(this); sensor.start(); sensor.list(); Event based sensor reading void onAccelerometerEvent() { accelerometer.set(x, y, z); } Sensor Demo Using the Camera Import camera library import ketai.camera.*; KetaiCamera cam; Setup in Setup( ) function cam = new KetaiCamera(this, 640, 480, 15); Draw camera image void draw() { //draw the camera image image(cam, width/2, height/2); } Camera Demo Timeline Demo Create Card Class load image, card number, children/parent cards Timeline Demo Load cards in order Translate cards with finger motion Swipe cards in both directions Snap cards into position Native Coding Overview For best performance need native coding Low level algorithms etc Most current wearables based on Android OS Need Java/Android skills Many devices have custom API/SDK Vusix M-100: Vusix SDK Glass: Mirror API, Glass Developer Kit (GDK) Mirror API + Glass GDK Glassware and Timeline Glassware and Timeline Static Cards Static content with text, HTML, images, and video. e.g. notification messages, news clip Live Cards Dynamic content updated frequently. e.g. compass, timer Immersions Takes over the whole control, out from timeline. e.g. interactive game Glassware Development Mirror API Server programming, online/web application Static cards / timeline management GDK Android programming, Java (+ C/C++) Live cards & Immersions See: https://developers.google.com/glass/ REST API Java servlet, PHP, Go, Python, Ruby, .NET Timeline based apps Static cards - Text, HTML, media attachment (image & video) - Standard and custom menu items Manage timeline - Subscribe to timeline notifications - Sharing with contacts - Location based services Mirror API GDK Glass Development Kit Android 4.0.3 ICS + Glass specific APIs Use standard Android Development Tools GDK add-on features Timeline and cards Menu and UI Touch pad and gesture Media (sound, camera and voice input) GDK Glass Summary Use Mirror API if you need ... Use GDK if you need ... Or use both Hardware Prototyping Fake Display 3D print Thingiverse model see http://www.thingiverse.com/thing:65706 Have the social impact of Google Glass without the cost Build Your Own Wearable MyVu display + phone + sensors Beady-i http://www.instructables.com/id/DIY- Google-Glasses-AKA-the-Beady-i/ Rasberry Pi Glasses Modify video glasses, connect to Rasberry Pi $200 - $300 in parts, simple assembly https://learn.adafruit.com/diy-wearable-pi-near-eye-kopin-video- glasses Physical Input Devices Can we develop unobtrusive input devices ? Reduce need for speech, touch pad input Socially more acceptable Examples Ring, pendant, bracelet, gloves, etc Prototyping Platform Arduino Kit Bluetooth Shield Google Glass Example: Glove Input Buttons on fingertips Map touches to commands Example: Ring Input Touch strip, button, accelerometer Tap, swipe, flick actions How it works Bracelet Armband Gloves 1,2, 3,4 Values/ output Summary Prototyping for wearables is similar to mobiles Tools for UI design, storyboarding, wireframing Android tools to create interactive prototypes App Inventor, Processing, etc Arduino can be used for hardware prototypes Once prototyped Native Apps can be built Android + SDK for each platform Other Tools Wireframing pidoco FluidUI Rapid Development Phone Gap AppMachine Interactive App Inventor WearScript App Inventor - http://appinventor.mit.edu/ Visual Programming for Android Apps Features Access to Android Sensors Multimedia output Drag and drop web based interface Designer view app layout Blocks view program logic/control App Inventor Designer View App Inventor Blocks View Orientation Demo Use wearable orientation sensor WearScript JavaScript development for Glass http://www.wearscript.com/en/ Script directory http://weariverse.com/ Best Practices (Dos + donts) Dont design an app Glass OS is time-based model, not an app model. X Know what makes Glass different than a phone Glass has certain superpowers. Remember what these superpowers are and use them to augment the experience youre designing. X Dont just port your mobile experience over to Glass It wont work. It will be too busy. It will be hard for users to quickly understand and navigate your content. The constraints on Glass are simply too strict. Bad idea. Even inverted, aside from the need to properly format each screen, the layout simply contains too much information for Glass. Good idea! Design for the (hyper)now When is my next meeting? How many calories have I eaten today? Can I get a burger for lunch? Spend 90% of your time thinking about what people want to know (in sport or elsewhere) at any given moment. The more you know about what info people need and currently dont have - the more compelling your design will be. Do one thing at a time Design for emotion thumbs up viewsnap - running Make it glanceable Seek to rigorously reduce information density. Successful designs afford for recognition, not reading. Bad Good Reduce the number of info chunksX You are designing for recognition, not reading. Reducing the total # of information chunks will greatly increase the glanceability of your design. 1 2 3 1 2 3 4 5 (6) Test done by Morten Just using a watch Design single interactions to be faster than 4 s Eye movements For 1: 1 230ms For 2: 1 230ms For 3: 1 230ms For 4: 3 (52/17) 690ms For 5: 2 460ms ~1,840ms Eye movements For 1: 1-2 460ms For 2: 1 230ms For 3: 1 230ms ~920ms 1 2 3 1 2 3 4 5 (6) Test done by Morten Just using a watch Test the glanceability of your design Test your design indoors + outdoors White is your new black On the device, black is not blackX Your mock Device simulation Establish hierarchy with color - not font size White is your and grey is your or . Footer text - establishing time, attribution, or distance - is the only place where you see a smaller font size used. If you have brand-specific typography - use it Use relative information display calendar card - in 10 minutes, Fri - viewsnap Remember, people have an ever-growing ecosystem of wearablesX Each device should be used when its most relevant and when its the easiest interaction available. The Glass screen is just one part of the experience Do view your design on the device 1. Download Android Design Preview 2. Plug in Glass 3. Terminal command: java -jar AndroidDesignPreview-0.3.2.jar 4. Drag the red rectangle over your work to take a look 5. Check out the developer site for more in-depth guidelines 5: Concept Design Exercise Design Group (33 People) If your name is on this list you are in the Design Group Yang Wang Konstantino Kapetaneas Preethi Srinivas Tony James Kate Vogt Aneesh Tarun Josh Andres Maria Maim Bram Reurings Luke Mill Tuck-Voon How M Gill Janaki Kumar Melinda Knight M Calkins Mike Tissenbaum Samantha Tse Kal McDowd Adora Tam Oscar Meruvia Mike Chen Anita Hoechtl Merlin Stone Ashoomi Dohlakia Icy Zhu Zdenek Mikovec Cristina Manresa-Yee Christian Winkler Angela Noh Amyris Fernandez Deborah Ptak Arne Renkema-Padmos Thomas Fritz 6: Wearable Technologies Wearable System Some Key Aspects Display Technologies Input Devices Interaction Metaphors Perceptual Factors Ergonomics Cognitive Aspects Display Technologies Key Properties of HMD Field of View Human eye 95 deg. H, 60/70 deg. V Resolution > 320x240 pixel Refresh Rate Focus Fixed/manual Size, Weight < 350g for long term Power Types of Head Mounted Displays Occluded See-thru Multiplexed Optical see-through HMD Virtual images from monitors Real World Optical Combiners Optical See-Through HMD Epson Moverio BT-200 Stereo see-through display ($700) 960 x 540 pixels, 23 degree FOV, 60Hz, 88g Android Powered, separate controller VGA camera, GPS, gyro, accelerometer Strengths of optical see-through Simpler (cheaper) Direct view of real world Full resolution, no time delay (for real world) Safety Lower distortion No eye displacement see directly through display Video see-through HMD Video cameras Monitors Graphics Combiner Video Video See-Through HMD Vuzix Wrap 1200DXAR Stereo video see-through display ($1500) Twin 852 x 480 LCD displays, 35 deg. FOV Stereo VGA cameras 3 DOF head tracking Strengths of Video See-Through True occlusion Block image of real world Digitized image of real world Flexibility in composition Matchable time delays More registration, calibration strategies Wide FOV is easier to support wide FOV camera Multiplexed Displays Above or below line of sight Strengths User has unobstructed view of real world Simple optics/cheap Weaknesses Direct information overlay difficult Display/camera offset from eyeline Wide FOV difficult Vuzix M-100 Monocular multiplexed display ($1000) 852 x 480 LCD display, 15 deg. FOV 5 MP camera, HD video GPS, gyro, accelerometer Display Types Curved Mirror off-axis projection curved mirrors in front of eye high distortion, small eye-box Waveguide use internal reflection unobstructed view of world large eye-box See-through thin displays Waveguide techniques for thin see-through displays Wider FOV, enable AR applications Social acceptability Opinvent Ora Waveguide Methods See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable Holographic Hologram diffracts light Limited FOV Colour bleeding Diffractive Slanted gratings Total internal reflection Costly, small FOV Waveguide Methods See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable Clear-Vu Reflective Several reflective elements Thinner light guide Large FOV, eye-box Reflective Simple reflective elements Lower cost Size is function of FOV Comparison Chart Input Technologies Input Options Physical Devices Keyboard Pointer Stylus Natural Input Speech Gesture Other Physiological Twiddler Input Chording or multi-tap input Possible to achieve 40 - 60 wpm after 30+ hours Chording input about 50% faster than multi-tap cf 20 wpm on T9, or 60+ wpm for QWERTY Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., & Looney, E. W. (2004, April). Twiddler typing: One-handed chording text entry for mobile phones. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 671-678). ACM. Virtual Keyboards In air text input Virtual QWERTY keyboard up to 20 wpm On real keyboard around 45-60+ wpm Word Gesture up to 28 wpm On tablet/phone Word Gesture up to 47 wpm Handwriting around 20-30 wpm A. Markussen, et. al. Vulture: A Mid-Air Word-Gesture Keyboard (CHI 2014) Unobtrusive Input Devices GestureWrist Capacitive sensing Change signal depending on hand shape Rekimoto, J. (2001). Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on (pp. 21-27). IEEE. Unobtrusive Input Devices GesturePad Capacitive multilayered touchpads Supports interactive clothing Skinput Using EMG to detect muscle activity Tan, D., Morris, D., & Saponas, T. S. (2010). Interfaces on the go. XRDS: Crossroads, The ACM Magazine for Students, 16(4), 30-34. Issues to Consider Fatigue Gorrilla Arm from free-hand input Comfort People want to do small gestures by waist Interaction on the go Can input be done while moving? Interaction on the Go Fitts law still applies while interacting on the go Eg: Tapping while walking reduces speed by > 35% Increased errors while walking Lin, M., Goldman, R., Price, K. J., Sears, A., & Jacko, J. (2007). How do people tap when walking? An empirical investigation of nomadic data entry.International Journal of Human-Computer Studies, 65(9), 759-769. Interface Metaphors Information Display Head Stabilized Fixed View Body Stabilized 3 DOF Tracking World Stabilized 6 DOF Tracking Spatial Cues for Wearable Info Billinghurst, M., Bowskill, J., Dyer, N., & Morphett, J. (1998, March). An evaluation of wearable information spaces. In Virtual Reality Annual International Symposium, 1998. Proceedings., IEEE 1998 (pp. 20-27). IEEE. Spatial cues sign. improve performance No difference between audio and visual cues Spatial Conferencing 3+ attendees can be distinguished with spatialized audio but could not without spatialized audio Billinghurst, M., Bowskill, J., Jessop, M., & Morphett, J. (1998, October). A wearable spatial conferencing space. In Proceedings of ISWC, 1998. (pp. 76-83). IEEE. Organizing Tools in a Halo Display Biocca, F., Tang, A., Lamas, D., Gregg, J., Brady, R., & Gai, P. (2001). How do users organize virtual tools around their body in immersive virtual and augmented environment?: An exploratory study of egocentric spatial mapping of virtual tools in the mobile infosphere. Media Interface and Network Design Labs, Michigan State University, East Lansing, MI. User Attention Metaphors Cognitive continuums (a) Input, (b) Output Increase cognitive load from left to right Notification Interruptions Gradually increase engagement Reduce attention load Receiving SMS on Glass Bing Tap Swipe Glass Show Message Start Reply User Look Up Say Reply Nomadic Radio (2000) Spatial audio wearable interface Sawhney, N., & Schmandt, C. (2000). Nomadic radio: speech and audio interaction for contextual messaging in nomadic environments. ACM transactions on Computer-Human interaction (TOCHI), 7 (3), 353-383. Spatial Audio Metaphor Messages/Events arranged depending on time of day Notification Interruptions Dynamic scaling of incoming message based on interruptibility of the user Busy = silence Availble = preview Layered Audio Notifications Background ambient audio Notification scale depending on priority Perception Issues AR Perceptual Issues Environment: Issues related to the environment itself. Capturing: Issues related to digitizing the environment Augmentation: Issues related to the design, layout, and registration or AR content Display device: Technical issues associated with the display device. User: Issues associated with user perceiving content. E. Kruijff, J. E. Swan, and S. Feiner. Perceptual issues in augmented reality revisited. 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2010, pp. 3--12. Cognitive Issues in Mobile AR Three categories of issues Information Presentation displaying virtual information on the real world Physical Interaction content creation, manipulation and navigation Shared Experience collaboration and supporting common experiences Li, Nai, and Henry Been-Lirn Duh. "Cognitive Issues in Mobile Augmented Reality: An Embodied Perspective." Human Factors in Augmented Reality Environments. Springer New York, 2013. 109-135. Depth Cues Pictorial: visual cues Occlusion, texture, relative brightness Kinetic: motion cues Relative motion parallax, motion perspective Physiological: motion cues Convergence, accommodation Binocular disparity Two different eye images Use the Following Depth Cues Movement parallax. Icon/Object size (for close objects) Linear perspective To add side perspective bar. Overlapping Works if the objects are big enough Shades and shadows. Depends on the available computation Information Presentation Amount of information Clutter, complexity Representation of information Navigation cues, POI representation Placement of information Head, body, world stabilized View combination Multiple views Twitter 360 www.twitter-360.com iPhone application See geo-located tweets in real world Twitter.com supports geo tagging Wikitude www.mobilizy.com Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Information Filtering Information Filtering Use context to support information filtering Show less information on AR mode (uncluttered screen) Text Representation Object space In general, does not fit wearable applications Billboards Screen space Annotations placed on a 2D plane, usually parallel to the projection plane. Better legibility of text. Outdoor background textures Changing outdoor illuminanation, and text drawing styles. Billboard and green text drawing styles are recommended. Active text drawing styles did not perform better relative to static styles. Screen Space Annotation AR Navigation Problem how to AR to assist with navigation to POI Multi-View Navigation Use multiple views (ego vs exo-centric) Map/top down view long distance navigation AR View nearby navigation, situational awareness Radar view in AR view Wearable Display: Limited FOV Transitioning Between Views Seamless spatial awareness Cognitive Models Resource Competition Framework Mobility tasks compete for cognitive resources with other tasks the most important given higher priority RCF is a method for analyzing this, based on: task analysis modelling cognitive resources a resource approach to attention Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005, April). Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 919-928). ACM. RCF Key Assumptions Four Key Assumptions 1. Functional Modularity: cognitive system divided into functionally separate systems with diff. representations 2. Parallel Module Operation: cognitive modules operate in parallel, independent of each other 3. Limited Capacity: cognitive modules are limited in capacity with respect to time or content 4. Serial Central Operation: central coordination of modules (eg monitoring) is serial Cognitive Interference Structural interference Two or more tasks compete for limited resources of a peripheral system eg two cognitive processes needing vision Capacity interference Total available central processing overwhelmed by multiple concurrent tasks eg trying to add and count at same time Cognitive Resources & Limitations asdfasdf Using RCF 1. Map cognitive faculty to task 2. Look for conflicts/overloads 3. Analyse for competition for attention 4. Look for opportunities for technology to reduce conflicts/competition Example: Going to work .. Which is the most cognitively demanding? Application of RCF Busy street > Escalator > Caf > Laboratory. But if you made Wayfinding, Path Planning, Estimating Time to Target, Collision Avoidance easier? Ergonomics Where to put Wearables? Places for unobtrusive wearable technology Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., & Martin, R. (1998, October). Design for wearability. In Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 116-122). IEEE. Where to Place Trackpad? User study 25 people different postures Front of thigh most preferred, torso/upper arm worst Thomas, Bruce, et al. "Determination of placement of a body-attached mouse as a pointing input device for wearable computers." 2012 16th International Symposium on Wearable Computers. IEEE Computer Society, 1999. Where do users want Wearables? 29% on clothing 28% on wrist 12% on Glasses Tool/Task Design Paradigm Lin, R., & Kreifeldt, J. G. (2001). Ergonomics in wearable computer design. International Journal of Industrial Ergonomics, 27(4), 259-269. Social Perception How is the User Perceived? TAT Augmented ID 7: Design Exercise Building on Glass Design Exercise Design for Glass Guidelines Design for Glass Dont get in the way Guidelines Design for Glass Dont get in the way Keep it timely Guidelines Design for Glass Dont get in the way Keep it timely Avoid the unexpected Guidelines Design for Glass Dont get in the way Keep it timely Avoid the unexpected Design for people Guidelines Platform Platform Platform - Mirror API Platform If you could build anything on Glass, what would it be? Sprint - Ideas Person (mom, family, astronomer) 60 seconds Sprint - Ideas Place (in the kitchen, car, hike) 60 seconds Sprint - Ideas Function (dance, stargaze, sequence dna) 60 seconds Sprint - Final deliverable Make a poster - (Hayes will demo) Title One liner Picture Pitch What is it? What does it do? Why is it a good idea? Sprint - Focus 10 min - Ideas to create a product to solve need 3 min - Share with your group Sprint - Focus Sprint - Focus 3 min - Pick a concept Sprint - Focus 15 min - Dive deeper, develop, poster Sprint - Focus 15 min - Poster Sprint - Lightning 2 minute pitch 7: Design Presentations Design Concepts 8: Research Directions Challenges for the Future (2001) Privacy Power use Networking Collaboration Heat dissipation Interface design Intellectual tools Augmented Reality systems Starner, T. (2001). The challenges of wearable computing: Part 1. IEEE Micro,21(4), 44-52. Starner, T. (2001). The challenges of wearable computing: Part 2. IEEE Micro,21(4), 54-67. Interface Design Gesture Interaction Capturing Behaviours 3 Gear Systems Kinect/Primesense Sensor Two hand tracking http://www.threegear.com Gesture Interaction With Glass 3 Gear Systems Hand tracking Hand data sent to glass Wifi networking Hand joint position AR application rendering Vuforia tracking Performance Full 3d hand model input 10 - 15 fps tracking, 1 cm fingertip resolution User Study Gesture vs. Touch pad vs. Combined input Gesture 3x faster, no difference in accuracy Meta Gesture Interaction Depth sensor + Stereo see-through https://www.spaceglasses.com/ Collaboration Social Panoramas Ego-Vision Collaboration Wearable computer camera + processing + display + connectivity Current Collaboration First person remote conferencing/hangouts Limitations Single POV, no spatial cues, no annotations, etc Sharing Space: Social Panoramas Capture and share social spaces in real time Enable remote people to feel like theyre with you Key Technology Google Glass Capture live panorama (compass + camera) Capture spatial audio, live video Remote device (desktop, tablet) Immersive viewing, live annotation Awareness Cues Where is my partner looking? Enhanced radar display, Context compass Interaction Glass Touchpad Input/Tablet Input Shared pointers, Shared drawing User Evaluation Key Results Visual cues significantly increase awareness Pointing cues preferred for collaboration Drawing on Glass difficult, ranked low in usability Intellectual Tools Context Sensing Using context to manage information progressive information display as user shows interest Context from Speech Gaze Real world Wearable AR Display Ajanki, A., Billinghurst, M., Gamper, H., Jrvenp, T., Kandemir, M., Kaski, S., ... & Tossavainen, T. (2011). An augmented reality interface to contextual information. Virtual reality, 15(2-3), 161-173. Gaze Interaction AR View More Information Over Time OpenSource Eyetracker Could use open source eyetracking Open Shades Eye Tracker http://www.wearscript.com/en/latest/eyetracking.html 9: Resources Glass Resources Main Developer Website https://developers.google.com/glass/ Glass Apps Developer Site http://glass-apps.org/glass-developer Google Design Guidelines Site https://developers.google. com/glass/design/index?utm_source=tuicool Google Glass Emulator http://glass-apps.org/google-glass-emulator Other Resources AR for Glass Website http://www.arforglass.org/ Vandrico Database of wearable devices http://vandrico.com/database Books Programming Google Glass Eric Redmond Rapid Android Development: Build Rich, Sensor-Based Applications with Processing Daniel Sauter Contact Details Mark Billinghurst email: mark.billinghurst@hitlabnz.org twitter: @marknb00 Hayes Raffle email: hraffle@google.com Feedback + followup form goo.gl/6SdgzA