Interaction modalities, technologies, and tools for

interactive art Jorge C. S. Cardoso

http://jorgecardoso.eu

Objectives

•  Overview of most common non-desktop interaction and computing paradigms

•  Examples of technologies and tools that can be used to implement those interactions o  Available tools at EArtes/UCP

• Most prominent people/references in these areas

•  No code (for now)

Topics Gestural interaction Tangible interaction Brain interaction

Wearable computing Physical computing Digital public displays Mobile computing

Gestural interaction

Photo credits: Flickr user artmakesmesmile: http://www.flickr.com/photos/artmakesmesmile/169193698/

Gestural interaction

"A gesture is a motion of the body that contains information. Waving goodbye is a gesture. Pressing a key on a keyboard is not a gesture because the motion of a finger on its way to hitting a key is neither observed nor significant. All that matters is which key was pressed".

-- Kurtenbach and Hulteen (1990), cited by Billinghurst & Buxton (2011)

Gestural interaction

Technologies/Tools (ordered by the complexity of gestures: simplest to more complex) •  Touch surfaces •  Computer vision techniques •  Wiimote •  Kinect •  MoCap

Gestural interaction: Touch surfaces

•  Mostly planar, rectangular, glass(y) surfaces •  Various technologies for detecting touch

o  which have different limits for the number of simultaneous touches it can detect

•  Gestures o  Fingers only (small devices) o  Hands/forearms (large(r) devices, depending on technology

used) o  2d movements only

•  Number of users o  one - personal devices (phones, tablets) o  few - large surfaces (tables, walls)

§  depends on the orientation of the surface

Gestural interaction: Touch surfaces

Tablets/phones

•  Singing fingers o  Jay Silver o  https://vimeo.com/12010952

Single-person interaction, fingers

Gestural interaction: Touch surfaces

Medium/Large surfaces

•  Touch-Wall @ EArtes o  Laser light plane (LLP) technique

(see Sandler 2012) o  Multi-touch surface (in

development) o  Vertical orientation o  One-two person interaction o  Fingers, hands

Gestural interaction: Touch surfaces

Medium/Large surfaces

•  Bookstore @ EArtes

Gestural interaction: Touch surfaces

Medium/Large surfaces

•  Interactive tables (Reactables) o  Diffused illumination technique o  Horizontal orientation o  Up to four people interacting o  Fingers, hands shapes o  Three available @ EArtes

Gestural interaction: Touch surfaces

Very large surfaces

•  Massive Curved Multi-Touch Screen o  Reality touchscreen University of Groningen o  http://www.youtube.com/watch?v=AlWFtF06RFo

Several-person interaction, fingers, hands

Gestural interaction: computer vision

Computer vision (CV) techniques (non depth-cameras)

•  Various kinds of surfaces/scenes

•  Gestures o  Varied, depends on positioning of camera

§  fingers §  hands

§  whole body

o  mostly 2d, but can also infer distance o  not very precise

§  Can be very precise with controlled conditions

Gestural interaction: computer vision

Video Place

•  Myron Krueger

•  http://www.youtube.com/watch?v=dmmxVA5xhuo

Gestural interaction: computer vision

Edge/Swap

•  Tiago Dionisio, Rudolfo Quintas

•  http://www.youtube.com/watch?v=OSKpXZjECtY •  http://www.youtube.com/watch?v=ryX4Vq2-IKE

Gestural interaction: computer vision

BES Interactive Video Wall

•  YDreams •  http://www.youtube.com/watch?v=ijrZQbSOONQ

Gestural interaction: computer vision

Balance

•  Jorge Coutinho •  https://vimeo.com/40506946

Gestural interaction: computer vision

Sketchsynth

•  Billy Keyes •  http://vimeo.com/42053193 •  http://golancourses.net/2012spring/05/13/billy-keyes-

final-project-sketchsynth/

Gestural interaction: computer vision

All eyes on you

•  Britzpetermann •  https://vimeo.com/33186969

Gestural interaction: computer vision

Knee deep

•  Emily Gobeille & Theodore Watson •  https://vimeo.com/8805152

Gestural interaction: computer vision

Night lights

•  YesYesNo •  http://vimeo.com/8525186

Gestural interaction: computer vision

aB

•  Jorge Coutinho •  http://vimeo.com/29524291

Gestural interaction: Wiimote

Wiimote •  Nintendo game controller

o  Wireless: uses bluetooth to connect to a computer

o  Has various accelerometer sensors to sense movement

o  IR sensor for pointing

•  Gestures o  3d movement, rotation o  combined with the sensor bar, allows

pointing gestures

•  Various wiimotes available @ EArtes

Gestural interaction: Wiimote

Wiimote

•  Wiimote hacks o  Johnny Chung Lee o  http://www.ted.com/talks/

johnny_lee_demos_wii_remote_hacks.html

Gestural interaction: Wiimote

Pizza breakout

•  Bruno Santos

Gestural interaction: Wiimote

Wii Drum High Tutorial

•  He Zhao •  http://vimeo.com/2406224

Kinect •  XBox game controller

o  Depth camera: senses the distance of each pixel to the camera (Borenstein 2012).

•  Gestures o  Detects a "point cloud" - pixels and their

distances to the camera o  With extra software it's possible to

detect limbs (skeleton) positioning o  More accurate than simple cameras

•  Two available @ EArtes

Gestural interaction: Kinect

Gestural interaction: Kinect

Kinect Physics Tutorial for Processing

•  Video showing the result of Processing code that uses Kinect

•  http://www.creativeapplications.net/processing/kinect-physics-tutorial-for-processing/

Gestural interaction: Kinect

YScope medical interface

•  YDreams •  Gestural interface for an operating room •  http://www.youtube.com/watch?v=91F6zErnCrs

Gestural interaction: Kinect

Control the Humanoid Robot by Kinect

•  Using the skeleton detected by the Kinect to drive a robot

•  ikaziso •  http://www.youtube.com/watch?v=w8BmgtMKFbY

Gestural interaction: Kinect

Unnamed soundsculpture

•  Daniel Franke & Cedric Kiefer •  Using various Kinect to model a dancer and then

process the data into visual effects

•  https://vimeo.com/38505448

•  honorary mention from the Prix Ars Electronica

Gestural interaction: Kinect

Make the line dance

•  "1024" •  Detection of skeleton + projection into human

body •  https://vimeo.com/21308228

Gestural interaction: Kinect

Fragmentos de um olhar

•  Pedro Serrano

Gestural interaction: LeapMotion

Leap Motion

•  New controller for hand interactions •  https://leapmotion.com/

Gestural interaction: MoCap

MoCap - Motion Capture

•  MoCap Room

•  High-speed infrared cameras detect markers placed on actor's body

•  Gestures o  Highly accurate gestures

•  (Vicon MoCap available @ EArtes)

Gestural interaction: MoCap

T(ether)

•  Tangible Media Group •  Interaction with virtual volumetric

data •  http://vimeo.com/42173010

Gestural interaction: MoCap

Portrait of the ghost drummer

•  Odaibe •  Visualization of the process of

playing on a drum kit •  http://vimeo.com/34682556 •  Non-interactive

Gestural interaction: computer vision

Computer Vision Tools

•  Processing o  BlobDetection o  JMyron

•  Max/Msp o  Jitter o  jit.cv external

•  Community Core Vision o  http://ccv.nuigroup.com/

Gestural interaction: Wiimote

Wiimote Tools

•  Processing/Java o  Motej - http://motej.sourceforge.net/ o  Wigee - http://www.wiigee.org/

•  Max/Msp o  aka.wiiremote external -

http://www.iamas.ac.jp/~aka/max/

Gestural interaction: Kinect

Kinect Tools

•  http://openkinect.org/wiki/Main_Page •  Book "Making things see" (Borenstein 2012) •  Processing/Java

o  OpenKinect - http://www.shiffman.net/p5/kinect/ o  SimpleOpenNI -

http://code.google.com/p/simple-openni/ o  http://www.creativeapplications.net/processing/

kinect-physics-tutorial-for-processing/ •  Max/Msp

o  jit.freenect.grab - http://jmpelletier.com/freenect/

Gestural interaction: Touch surfaces

Tools

•  Web apps o  jQMultiTouch library

§  http://dev.globis.ethz.ch/jqmultitouch/

Gestural interaction

References

•  Billinghurst & Buxton (2011). Gesture Based Interaction, http://www.billbuxton.com/input14.Gesture.pdf

•  Borenstein, Greg (2012). Making Things See: 3D vision with Kinect, Processing, Arduino, and MakerBot, O'Reilly Media / Make

•  Buxton, Bill (2012). Multi-Touch Systems that I Have Known and Loved, http://www.billbuxton.com/multitouchOverview.html

•  Kurtenbach, G. & Hulteen, E. (1990). Gestures in Human-Computer Communications. In B. Laurel (Ed.) The Art of Human Computer Interface Design. Addison-Wesley, 309-317, http://www.billbuxton.com/inputManuscript.html

•  Levin, Golan. Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers, http://www.flong.com/texts/essays/essay_cvad/

•  Sandler, Seth (2012). LASER LIGHT PLANE MULTITOUCH TECHNIQUE, http://sethsandler.com/multitouch/llp/

Tangible interaction

Photo credits: Flickr user kedume: http://www.flickr.com/photos/kedume/746632510/sizes/l/in/photostream/

Tangible interaction

"[...] tangible interfaces give physical form to digital information, employing physical artifacts both as representations and controls for computational media. TUIs couple physical representations (e.g., spatially manipulable physical objects) with digital representations (e.g., graphics and audio), yielding user interfaces that are computationally mediated, but generally not identifiable as "computers" per se." - Ullmer & Ishii (2000)

Tangible interaction

Picture from Ullmer and Ishii (2000).

Tangible interaction

Tangible Media Group site

•  http://tangible.media.mit.edu/projects/

Tangible interaction

Reactable

•  Tangible musical instrument o  Users manipulate physical objects to create and control the sound

o  Can be used as a general tangible framework

•  http://www.youtube.com/watch?v=Mgy1S8qymx0

Tangible interaction

Little Boxes

•  ECAL/Joelle Aeschlimann •  Music boxes for iPad •  https://vimeo.com/45704273

Tangible interaction

Sketch-a-TUI

•  Alexander Wiethoff, et al. •  Set of printable objects for tangible prototypes •  http://vimeo.com/38793875 •  Paper

o  http://tei-conf.org/12/Main/Sketchatui

•  Printable objects o  http://project-premium.org/sketch-a-tui/index.html

Tangible interaction

Birds on Paper •  Chen-Wei Chiang, et al. •  Drawing music •  http://www.youtube.com/watch?v=qqkkNotSQMo •  Paper

o  http://tei-conf.org/12/Main/Birdsonpaper

Tangible interaction

Chronotape •  Peter Bennett •  Tangible timeline with annotations •  https://vimeo.com/28191512 •  Paper

o  http://tei-conf.org/12/Main/Chronotape

Tangible interaction

Splash Controllers

•  Luc Geurts and Vero Vanden Abeele •  Uncareful Manipulation of Water •  Paper

o  http://tei-conf.org/12/Main/Splashcontrollers

Tangible interaction

Skube

•  Andrew Nip, Ruben van de Vleuten, Malthe Borch, and Andrew Spitz

•  Tangible interface to Last.fm & Spotify Radio •  http://www.creativeapplications.net/maxmsp/skube-

tangible-interface-to-last-fm-spotify-radio/

Tangible interaction

Faustine

•  Carlos Caires & Jorge Cardoso •  Interactive video installation inspired by the short story

"Morel's Invention" •  https://vimeo.com/19043202

Tangible interaction

The garden of time

•  Carlos Caires & Jorge Cardoso •  Interactive video installation inspired by the short story

"The Garden of Forking Paths" •  https://vimeo.com/24836769

Tangible interaction

Tools

•  Reactable •  TUIO - Fiducial tracking

o  http://www.tuio.org/?software o  Processing o  Max/MSP o  CCV

•  Wiimotes •  Other off-the-shelf hardware/sensors

Tangible interaction

References •  G. W. Fitzmaurice, H. Ishii, and W. A. S. Buxton, “Bricks: Laying the

Foundations for Graspable User Interfaces,” in Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’95, 1995, pp. 442–449.

•  H. Ishii and B. Ullmer, “Tangible bits,” in Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’97, 1997, pp. 234–241.

•  Ullmer, B. and Ishii, H. (2000). “Emerging frameworks for tangible user interfaces,” IBM Systems Journal, vol. 39, no. 3, pp. 915–931, Jul. 2000.

Brain interfaces

Photo credits: Flickr user Saad Faruque: http://www.flickr.com/photos/cblue98/7254347346/sizes/l/in/photostream/

Brain interfaces

Brain-Computer Interfaces (BCI) or Mind-Machine Interface (MMI)

Using brain signals to interact with computers.

•  Usually based on EEG readings •  Various commercial devices available •  http://en.wikipedia.org/wiki/Brain

%E2%80%93computer_interface#Low-cost_BCI-based_Interfaces

Brain interfaces

Câmara Neuronal

•  Adolfo Luxúria Canibal, João Martinho Moura, Miguel Pedro Guimarães

•  http://projects.jmartinho.net/3486412/Camara-Neuronal-Video-Teaser

Brain interfaces

Brainwave beats

•  Collin Cunningham •  Midi notes from brain signals •  Uses the Neurosky Mindset device •  http://blog.makezine.com/2011/05/05/collins-lab-

brainwave-beats/

Brain interfaces

Mansion Maze 3 BCI Game Trailer

•  Eric Rosendale, Ian McCabe, Veronica Cole, Aaron DeChamplain, and Matt Stephan

•  BCI Game •  http://www.youtube.com/watch?v=sdYAIrr9nDw

Brain interfaces

Mind Control of Lego NXT Telepresence Robot

•  Mark Bruce •  Controling a lego robot using brain signals •  Uses EPOC BCI device •  https://www.youtube.com/watch?v=nX741DZw8l4

Brain interfaces

Tools

•  Neurosky Mindset (available @ EArtes) •  EPOC neuroheadset - http://emotiv.com/emortal/cart/ •  Processing library

o  Neurosky Mindset Processing §  http://jorgecardoso.eu/processing/

MindSetProcessing/ •  http://en.wikipedia.org/wiki/Brain

%E2%80%93computer_interface#Low-cost_BCI-based_Interfaces

Wearable computing

Geordi La Forge, Star Trek The Next Generation character wearing the VISOR

Wearable computing

"Wearable computing is the study or practice of inventing, designing, building, or using miniature body-borne computational and sensory devices. Wearable computers may be worn under, over, or in clothing, or may also be themselves clothes" (Mann, 2012)

Wearable computing

Image from: http://www.eyetap.org/research/wearables/wearcomp/wearables.html

Image from: http://www.interaction-design.org/

Wearable computing

Google Project Glass

•  https://plus.google.com/+projectglass/posts •  https://www.youtube.com/watch?

v=9c6W4CCU9M4

(how will our world be in 5/10 years, when we all start wearing devices even more powerful than Glass?)

Wearable computing

XS Labs

•  http://xslabs.net/

Wearable computing

Blazer 01.2

•  A light emitting bracelet •  Can be used to "draw" letters in the air

•  Vincent leclerc •  http://www.uttermatter.com/blazer/012/index.htm#

Wearable computing

cidZiy

•  Capacitive sensing textile surface

•  xslabs •  http://www.youtube.com/watch?v=zH4J6Uaf88A

Wearable computing

Kukkia

•  Kukkia flowers frame the face and slowly open and close over time

•  xslabs •  http://www.youtube.com/watch?v=R5fd6AwGMhg

Wearable computing

See U in my Tee

•  Inês Petiz •  T-shirts that react to the presence of other t-shirts •  http://wearable-technology.blogspot.pt/

Wearable computing

References •  Mann, Steve (2012): Wearable Computing. In: Soegaard, Mads and Dam,

Rikke Friis (eds.). "Encyclopedia of Human-Computer Interaction". Aarhus, Denmark: The Interaction Design Foundation. Available online at http://www.interaction-design.org/encyclopedia/wearable_computing.html

Physical computing

Physical computing

"Physical computing, in the broadest sense, means building interactive physical systems by the use of software and hardware that can sense and respond to the analog world. (...) In the broad sense, physical computing is a creative framework for understanding human beings' relationship to the digital world. In practical use, the term most often describes handmade art, design or DIY hobby projects that use sensors and microcontrollers to translate analog input to a software system, and/or control electro-mechanical devices such as motors, servos, lighting or other hardware." - (Physical computing 2012)

Physical computing

Physical computing book

•  O'Sullivan & Igoe 2004

Physical computing

reaDIYmate

•  Olivier Mével and Marc Chareyron

•  http://www.readiymate.com

Physical computing

An Instrument for the Sonification of Everyday Things

•  Dennis P Paul •  http://

www.creativeapplications.net/processing/an-instrument-for-the-sonification-of-everyday-things/

Physical computing

Piano playing stairs

•  http://www.youtube.com/watch?v=DuNWkKBokts

Physical computing

What the phonics

•  Andrew Spitz & Momo Miyazaki •  https://vimeo.com/45747333

Physical computing

Barcode band

•  Kang woon jin, Lee ha lim, Kim yong duk •  https://vimeo.com/41132461

Physical computing

Syndyn

•  André Rangel and Anne-Kathrin Siegel •  https://vimeo.com/20859799

Physical computing

Little Bits

•  Hardware components that can be assembled together •  http://market.littlebits.cc/ •  https://vimeo.com/45276780

Physical computing

MakeyMakey

•  Circuit board that connects to a computer via USB and allows everyday objects to become controllers

•  http://www.makeymakey.com/

Physical computing

Lego Mindstorms NXT

•  Available @ EArtes •  Lego kit with

o  motors o  distance sensor o  light sensor o  button o  lego parts...

Physical computing

Tools •  Arduino - http://www.arduino.cc/ •  Galago - http://outbreak.co/galago •  Digispark - http://digistump.com/ •  Raspberry pi - http://www.raspberrypi.org/ •  MakeyMakey -

http://www.makeymakey.com/ •  LittleBits - http://littlebits.cc/ •  Lego NXT Mindstorms (available @EArtes)

o  http://mindstorms.lego.com/en-us/Default.aspx

•  3d printer, CNC, etc.

Physical computing

References •  Physical computing. (2012, July 21). In Wikipedia, The

Free Encyclopedia. Retrieved 14:37, September 22, 2012, from http://en.wikipedia.org/w/index.php?title=Physical_computing&oldid=503476286

•  Dan O'Sullivan, Tom Igoe. (2004). Physical Computing: Sensing and Controlling the Physical World with Computers, Thomson Course Technology, Boston, Massachusetts,

Digital public displays

Photo credits: Flickr user bmann: http://www.flickr.com/photos/boris/5104900447/sizes/l/in/photostream/

Digital public displays

Digital displays located in public areas, allowing anyone to see/interact with them

•  Traditionally, public displays have been used to broad information from one active source to millions of passive receivers

•  Digital public displays can change this o  Turning receivers into active receivers o  Supporting dynamic locally relevant information o  Supporting user-generated content (many to many

content distribution) •  Everyone can interact

o  Everyone can see you interacting o  Everyone sees the results from the interaction

Digital public displays

Advertising

Digital public displays

Honda Interactive Billboard using SMS and Bluetooth

•  Users can "start" the car via SMS •  http://www.youtube.com/watch?v=OudaJGtwdQ8

Digital public displays

BBC Billboard

•  Voting billboards, users can SMS their vote •  http://www.bbc.co.uk/blogs/theeditors/2007/06/

live_in_new_york_city.html

Digital public displays

Audio Jack Billboards

•  Promoting a TV show "Big Love" •  Users can listen to people's thoughts by plugin

headphones into the billboard •  http://www.trendhunter.com/trends/audio-jack-billboard-

for-big-love

Digital public displays

Nokia's N90 interactive billboard

•  YDreams •  The billboard snaps people's pictures and displays them

on the billboard •  http://www.youtube.com/watch?v=UWgIswo3lV4

Digital public displays

Mini Billboard Interactive

•  The billboard displays the names of the Mini car owners passing by

•  http://www.nytimes.com/2007/01/29/business/media/29cooper.html

Digital public displays

Nikon D700

•  The billboard plays flashing cameras as people pass by. •  http://www.thecoolhunter.net/article/detail/1570/nikon-

d700-guerrilla-style-billboard

Digital public displays

Ambient displays

•  Information about their surroundings •  Information visualization, aesthetics, art

Digital public displays

Hello Wall

•  (Prante et al. 2003) •  Can display various abstract information patterns •  Can display group activity

Digital public displays

Informative Art ambient display

•  Skog et al. 2003 •  Display bus information (only the bus number 16 - two in

each direction) o  Size of square: amount of time before bus leaves o  Color: Time to leave to the bus stop o  Position: right - buses to the city center; left - buses from

the city center

Digital public displays

Information Percolator

•  Heiner et al. 1999 •  A water bubble display

o  Can be used to display various types of ambient information

•  http://www.cs.cmu.edu/~hudson/video.html

Digital public displays

Entertainment/Waiting times

Digital public displays

Mobilenin

•  Scheible, Jürgen and Ojala, Timo •  Paper

o  http://www.mendeley.com/research/mobilenin-combining-multi-track-music-video-personal-mobile-phones-public-display-multi-user-interac

o  Allows users to vote on the next video clip

Digital public displays

Public Youtube Player

•  Allows users to have some control over the videos that are played

•  at the EArtes' Bar display

Digital public displays

Everybody Votes

•  Users can express their opinions regarding various questions and suggest their own questions

•  at the EArtes' Bar display

Digital public displays

References

Heiner, J. M., Hudson, S. E., & Tanaka, K. (1999). The Information Percolator  : Ambient Information Display in a Decorative Object. Proceedings of the 12th annual ACM symposium on User interface software and technology - UIST ’99 (pp. 141–148). New York, New York, USA: ACM Press. doi:10.1145/320719.322595

Prante, T., Röcker, C., Streitz, N., Stenzel, R., Magerkurth, C., van Alphen, D., & Plewe, D. (2003). Hello.Wall - Beyond Ambient Displays. Video Track and Adjunct Proceedings of the 5th Intern. Conference on Ubiquitous Computing (UBICOMP’03). Seattle, Wash., USA.

Skog, T., Ljungblad, S., & Holmquist, L. E. (n.d.). Between aesthetics and utility: designing ambient information visualizations. IEEE Symposium on Information Visualization 2003 (IEEE Cat. No.03TH8714) (pp. 233–240). IEEE. doi:10.1109/INFVIS.2003.1249031

Mobile computing

Mobile computing

"Mobile computing is human–computer interaction by which a computer is expected to be transported during normal usage. Mobile computing involves mobile communication, mobile hardware, and mobile software." - (Mobile computing 2012)

Mobile computing

X-Fetch

•  Daniel Santos, João Rema •  City-wide game, peddy paper like •  Players have to discover a set of clues spread across

the city (Porto) using a mobile device that plays videos pointing to next clues and that tracks players GPS locations

•  http://jpn.c2com.up.pt/2005/10/29/xfetch_quando_o_real_e_o_virtual_se_confundem.html

Mobile computing

•  GPS Drawing

Jeremy Wood http://www.gpsdrawing.com/gallery.html

Mobile computing

Mobile mobile

•  http://www.theophane.co.uk/mobile-mobile •  https://vimeo.com/8118881

Mobile computing

Pom

•  Jorge Coutinho •  Jogo Pong estendido a seis jogadores •  https://vimeo.com/18579379

Mobile computing

References

•  Mobile computing. (2012, September 8). In Wikipedia, The Free Encyclopedia. Retrieved 17:56, September 30, 2012, from http://en.wikipedia.org/w/index.php?title=Mobile_computing&oldid=511364202