USING VISION SENSORS FORINNOVATIVE HCI

- CHALLENGES FOR THE NEXT INTERACTIVE MEDIA -

Hideki Koike Tokyo Institute of Technology

OUTLINE

➤ Vision-based HCI

➤ Challenges for the next interactive media

➤ Environment: Interactive Surface on the Water

➤ Contents: Spectating Ball Sports from Ball’s POV

➤ Advertisement: Unaware Gaze Navigation

➤ Lessons learned from our prototype development

VISION-BASED HCI

NEW I/O DEVICES HAVE BEEN CHANGING HCI

IBM PC Macintosh DataLand

CUI GUI PUI

I: keyboard

O: character display

I: keyboard, mouse

O: bitmap display

I: gesture recognition/ speech recognition

O: large display/wearable

ADVANCES IN COMPUTER VISION

➤ Hardware:

➤ Camera: smaller, higher resolution, and less expensive.

➤ GPU: faster and less expensive.

➤ Software:

➤ Open software (e.g. OpenCV, AR Toolkit)

wired unwired

SPECIAL USE CAMERAS

➤ Example:

➤ far/near infrared, polarization, high-speed, depth, mocap, action cam, eye tracking, light field

➤ Can see what human cannot

➤ Enable natural and intuitive UI

far infrared camera polarization camera depth camera

CHALLENGES FOR THE NEXT INTERACTIVE MEDIA

AQUATOP DISPLAY: INTERACTIVE SURFACE ON THE WATER

PROTOTYPE

WATER SPECIFIC GESTURES

Poking Stroking

Scooping Hitting

WATER SPECIFIC GESTURES

poking as selecting

stroking as swiping

scooping as dragging

COMPARING TO TRADITIONAL UI

selection page scroll transition exit app.

PC’s GUI mouse click slider click&drag win. close

button

Touch UI touch swipe touch&drag home button

AquaTop poke stroke scoop hit

IMMERSION BY FLUID SIMULATION

AQUATOP DISPLAY [2012]

EVALUATION: VISIBILITY

0.26 g/l

1.10 g/l

EVALUATION: POINTING ACCURACY

poking touching

AWARD

➤ Laval Virtual 2013: Grand Prix

➤ METI Innovative Technologies 2013: Special Award (Culture)

➤ TV Tokyo World Business Satellite 2013: Toretama Award & Kickstarter Award

➤ and more.

BALLCAM!: SPECTATING BALL SPORTS FROM

BALL’S POV

BALL’S POV

➤ 360 degree omnidirectional video streaming

➤ Ball’s Point of View

time

PROTOTYPE

➤ What is difficult?

➤ The ball is spinning, sometimes very fast (> 600rpm)!

➤ How to stabilize the video from spinning ball?

first prototype: The iBall [2011]

second prototype

BALLCAM! [2013]

OVERVIEW

Image Subsampling

View Expansiontriplet image sixlet image

Motion Interpolation

Distortion Correction

STEP 1: CORRECTING LENS DISTORTION AND ROLLING SHUTTER

P : back-projection

K: calibration matrixR(t): camera rotation

Gauss-Newton method

STEP 2: SELECTING IMAGES USING INTENSITY

STEP 3: EXTENDING VIEWS USING MULTIPLE FRAMES

frame t+1

frame t+2

triplet image N

sixlet image

triplet image N+1

frame t

STEP 4: IMAGE INTERPOLATION

imageinterpolation

Mf: forward mapping

Mb: backward mapping

MI: Identity mapping

BALLCAM! [2013]

OUR GOAL

➤ Rugby World Cup in Japan [2019]

➤ Tokyo Olympic [2020]

➤ World Cup Soccer [2022]

20162011 2012 2013 2014 2015

UNAWARE GAZE NAVIGATION

EXPERIMENTS

EXPERIMENT 1

no blur(σ=0) σ=5

experiment image

heat map

EXPERIMENT 1

no blur(σ=0) σ=5

experiment image

heat map

EXPERIMENT 1

blur strength and the time to fixation

EXPERIMENT 2

EXPERIMENT 2

USAGE SCENARIO

LESSONS LEARNED FROM OUR PROTOTYPE DEVELOPMENT

HOW IDEAS BECOME TO A UNIQUE PRODUCT

new(?) idea

rapid prototyping

advanced implementation

unique product

NEW IDEA(?)

➤ In most cases, it is not.

➤ Survey other work carefully.

➤ OK. It seems to be a new idea!

➤ However, 10,000 people in the world are inspiring the same idea at this moment.

RAPID PROTOTYPING

➤ Let’s make a prototype ASAP.

➤ The idea may be not so interesting as you thought.

➤ The idea may be more interesting than you thought.

➤ OK. It seems to be interesting!

➤ However, 100 people in the world are making the same prototype at this moment.

ADVANCED IMPLEMENTATION

➤ Let’s do serious implementation.

➤ Work harder.

➤ However, you may face very difficult technical issues.

➤ Use the wisdom of predecessors.

➤ You may be only one who finally develop the system!

CONCLUSION

➤ Vision-based HCI is a key to the next HCI.

➤ Effective use of special use cameras enables natural and intuitive interaction.

➤ Challenges for the next interactive media:

➤ Environment, Contents, and Advertisement.

➤ Rapid prototyping and advanced implementation make your ideas unique.

REFERENCES➤ AquaTop Display

➤ Y. Matoba et.al.: “AquaTop Display: A True Immersive Water Display System”, Proc. on ACM SIGGRAPH 2013, Emerging Technologies, Article 4, 2013.

➤ H. Koike et.al.: “AquaTop Display: Interactive Water Surface for Viewing and Manipulating Information in a Bathroom”, ACM Int’l Conf. on Interactive Tabletops and Surfaces (ITS’13), pp.155-164, 2013.

➤ BallCam

➤ R.Funakoshi et.al.: “Activity-Aware Video Stabilization for BallCam”, Proc. on ACM User Interface Software and Technology (UIST’16 Adjunct), pp.197-198, 2016.

➤ R. Funakoshi et.al.: “Synthesizing Pseudo Straight View form A Spinning Camera Ball”, 7th Augmented Human Int’l Conf. (AH’16), pp.30:1-30:4, 2016.

➤ K. Horita et.al.: “Experiencing the Ball’s POV for Ballistic Sports”, Proc. of the 4th Augmented Human Int’l Conf. (AH’13), pp.128-133, 2013.

➤ Unaware Blur

➤ H. Hata et.al.: “Visual Guidance with Unnoticed Blur Effect”, Int’l Working Conf. on Advanced Visual Interfaces (AVI2016), pp.28-35, 2016.