“WaterTouch”: An Aquatic Interactive MultimediaSensory Table based on Total Internal Reflection in Water

Steve Mann, Ryan Janzen, Jason HuangUniversity of Toronto

http://wearcam.org/watertouch/

ABSTRACTWe propose a water-based multitouch multimedia user-interfacebased on total-internal reflection as viewed by an underwatercamera. The underwater camera is arranged so that noth-ing above the water surface is visible until a user touches thewater, at which time anything that penetrates the water’ssurface becomes clearly visible.

Our contribution is twofold: (1) computer vision using un-derwater cameras aided by total internal reflection; (2) hy-peracoustic signal processing (frequency shifting) to capturethe natural, acoustically-originating sounds of water ratherthan using synthetic sounds.

Using water itself as a touch screen creates a fun and play-ful user interface medium that captures the fluidity of thewater’s ebb and flow. In one application, a musical instru-ment is created in which acoustic disturbances in the water(received by underwater microphones or hydrophones) arefrequency-shifted to musical notes corresponding to the lo-cation in which the water is touched, as determined by theunderwater computer vision system.

Categories and Subject DescriptorsH.5.2 [Info. systems]: Interfaces & pres.—User Interfaces;I.4.9 [Computing Methodologies]: Image Processing andComputer Vision; J.2 [c.apps.]: Physical sciences & engi-neering; J.5 [c.apps.]: Arts & humanities—Music

General TermsTheory, Design, Human Factors

KeywordsHydraulophone, Total internal reflection, Tangible User In-terface, ORGANic User Interface, Human Computer Inter-action, Musical Instrument, Touch Screen, Waterplay

We wish to thank Jonti-Craft, Inc. for providing their “Sensory

Table” into which we installed the WaterTouch.

* Area Chair: David Shamma

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.All figures copyrighted by the authorsMM’11 November 28–December 1, 2011, Scottsdale, Arizona, USA.Copyright 2011 ACM 978-1-4503-0616-4/11/11 ...$10.00.

1. INTRODUCTION

1.1 Background and related workWater has a history of use in human computer interac-

tion [11], public sculpture [9], and water therapy [2].Water has also been used in hydraulophones [8], which

use pressurized jets of water as a tactile user-interface. Ingeneral, tangible user interfaces have proven themselves inmany applications of human-computer interaction. For ex-ample Ishii’s bottles [3], and the Music Cube [1] form notableexamples of tangible interfaces.

Benjamin Franklin invented a musical instrument whichhe called the armonica, making sound from fingers rubbingon tuned pieces of glass while wet. Sometimes also referredto as glass harmonica, this instrument consisted of a row ofspinning glass disks or goblets all mounted to a single metalshaft spun by a foot treadle.

While playing glass armonica underwater, we found, inour previous work [7], that the water imparted interestingattributes to the sound, but we wanted some additional ver-satility, and the option to have a high Q-factor (less damp-ing) at certain times during our performances. In order toachieve this, we used a spinning glass cylinder, which pro-duced sound continuously along its entire length.

We picked up the sound by way of a geophone (contactmicrophone) in the cylinder, and transmitted it wirelesslyto a computer. A computer vision system, connected to acamera, detected the 1-dimensional position where the rodwas touched.

This information was used to control the attributes ofone or more (depending on the number of fingers touch-ing) bandpass filters affecting the original physical sound.The instrument was used in a variety of public performances(street performances, underwater performances, etc.). SeeFig 1(c),(d).

Glass was chosen so as to make an approximate refractiveindex match with the liquid (water) in which it was im-mersed. With this matching of refractive indices, the glasscylinder appeared almost invisible in the water. Using un-derwater cameras looking upwards, at an angle greater thanthe critical angle of total-internal-reflection, the image ofthe fingers was strongly enhanced, such that the cameracould much more easily pick up the fingers while ignoringeverything else in the scene (Fig 1(c),(d)). Additionally,tiny water waves produced by the sound vibrations (acous-tic sound production in the glass) were visible, so that thecamera could actually pick up some of the ripples from thesound waves in the scene.

925

(a) (b) (c) (d)

Figure 1: WaterTouch uses true total internal reflection. (a) Placing the camera underwater, and having it look up at an angle, makes itpossible to see the fingers as they disturb the light gathered by total internal reflection. This gives rise to a hyper-sensitivity in the image plane,making visible subtle motion- and sound-induced ripples in the water. With carefully constructed computer vision, a high-speed camera canfunction as an optical pickup of acoustic phenomena, much like an electric guitar having an optical pickup. Together with an array of geophones,hydrophones, and microphones, this provides a multiply acoustic instrument having a richly acoustic physicality. Fingers that penetrate the surfacecan appear as if floating in a disembodied state (b), making computer vision, tracking along the flat air-water boundary layer, and accurate 3Dlocalization trivial. (c) and (d) show our earlier experiments with an underwater friction-idiophone, where underwater (plus wearable overhead)cameras “look” at a glass cylinder and detect 1-dimensional motion.

1.2 Relation to other touch screensMany touch screens work on the principle of frustrated

total internal reflection [12][10] (i.e. affecting an evanescentwave by touching the surface outside it). WaterTouch workson true total internal reflection, i.e. the user actually pene-trates the surface material.

Water gives depth and dimensionality to the surface. Itsrandomness, unpredictability, and unique properties, createa fun, plaful, and fascinating form of interaction that evena 2 year old child can immediately relate to!

2. WATERTOUCHWe now describe the watertouch interface. Unlike our

early 1-dimensional experimentation which required glass,this work purely uses the surface-tension layer of water asthe sole surface of visual and acoustic interaction.

It is based on a product called a “Sensory Table” manu-factured by Jonti-Craft, a company based in Wabasso, MN,USA. The product consists of a transparent acrylic tub onwheels that can be filled with sand or water to give childrena variety of sensory experiences.

The name ‘Sensory Table’ arises because the sand or waterare used to stimulate the senses, not because the table hasany sensors in it. But we can easily add sensing apparatusto turn the Sensory Table into a human-computer interface.

Typically we use one or more underwater cameras, alongwith underwater listening devices (hydrophones) and a dataprojector. Projectors serve two roles: (1) structured lightingfor active computer vision; (2) display of information to theuser. This can be achieved by an infrared projector or otherinfrared light source for the structured vision, and a visiblelight projector for the display of information to the user.

Unlike a solid touch screen, where only the tips of the fin-gers touch a two-dimensional (2D) surface, the WaterTouchuses computer vision to capture finger motion in three di-mensions (3D), while also capturing some of the randomor idiosyncratic behaviour of water that generally gives theuser-experience an aquatic feel. See Fig 2.

Fingers that penetrate the surface appear as if floating ina disembodied state, as shown in Fig 1(b). Inserting a blackbackground at the far end of the pool causes the undersideof the water surface to also become black, and when lit fromabove, the fingers become very clearly distinguishable.

Figure 2: WaterTouch, with projector and active computer vision(infrared illumination, infrared and visible light sensing), underwaterusing total internal reflection. The use of water also gives a volumetriceffect visible throughout the water. Unique properties of water createan interesting and sensory-rich space for play and exploration.

3. TOTAL INTERNAL REFLECTIONUNDER A RIPPLED WATER SURFACE

To achieve total internal reflection, the camera’s angle ofviewing (akin to angle of incidence for light emission) mustbe beyond the critical angle at the water surface. We useSnell’s law: sin θ1

sin θ2=n2

n1(1)

where n1 and n2 are the indices of refraction of water andair, respectively, and θ1 and θ2 are the angles of incidenceand transmission. The critical angle when looking from un-derwater up at a flat air-water interface is θc

.= 48.6◦ (found

by setting θ2 to 90◦ for the onset of total internal reflection).The camera angle θCAM must be sufficiently shallow to

ensure total internal reflection over the entire viewing areaunder the water surface. This requirement becomes evenmore harsh when surface waves are present.

Ripples or waves are created in our water trough by themovement of fingers. Capillary waves are the most com-mon type of water waves for this size of setup, as opposedto deep water gravity waves. Small capillary waves can be

926

(a) (b)Figure 3: (a) Camera viewing the water surface from underneath, with total internal reflection (side view). The worst-case scenario has surfacewaves propagating in the x-direction either towards or away from the camera, which gives the most strict requirement on the camera angle toachieve total internal reflection. (b) Overhead illustration of the underwater total-internal-reflection haptic area.

approximated as sinusoidal waves, as in:

zw(~r, t) = A sin(k(~r · r̂t)− ωt) (2)

with ~r as position in (x, y), t as time, r̂t as the direction ofwave travel, k as the wave number, and ω as the angularfrequency of the water wave.

For total internal reflection, we need the top peripheralviewing angle (see Fig 3(a)) to be sufficiently large:

θP1 > θc + θw(~r, t) (3)

at all times, where θw(~r, t) is the angle of the water surface.To be true at all times, we consider a wave travelling in thex direction, moving towards or away from the camera.

max {θw(~r, t)} = sup

{tan−1 ∂zw

∂~r: ~r, t

}(4)

= sup

{tan−1 ∂zw

∂x

∣∣∣∣y=0

: x, t

}(5)

= sup{

tan−1 (Ak cos(kx− ωt))}

(6)

= tan−1(Ak)max (7)

= tan−1

(2πA

λ

)max

(8)

where λ is the wavelength.For a camera with viewing angle α, we have θP1 = θCAM−

α2

, and so we place the camera at an angle such that:

θCAM >α

2+ sin−1

(n2

n1

)+ tan−1

(2πAmaxλmin

)(9)

to maintain total internal reflection over the viewing area.

4. TRANSFORMING FROM IMAGECOORDINATES TO PHYSICAL,RECTANGULAR COORDINATES

Since the camera is placed on an angle, the water surfacearea viewed by it is trapezoidal. The trapezoidal regionsubtends a solid rectangular cone of α degrees high and βdegrees wide, for a camera aspect ratio of β : α.

We make a transformation to relate what is viewed on thecamera to the real, physical space along the water surface.

A point (X,Y ) in a XMAX by YMAX image, transforms tothe physical coordinates:

x = dCAM tan(θCAM − α

(X

XMAX− 1

2

))(10)

y = x tan(β(

YYMAX

− 12

))(11)

with an origin as shown in Fig 3, and camera depth dCAM .

The total size of the haptic area is: (for θCAM + α2 < 90◦)

Lx = dCAM ·(tan

(θCAM + α

2

)− tan

(θCAM − α

2

))(12)

Ly,1 = dCAM · 2 tan(β2

)tan

(θCAM − α

2

)(13)

Ly,2 = dCAM · 2 tan(β2

)tan

(θCAM + α

2

)(14)

Although many modern television displays and camerasare HDTV (16:9 aspect ratio), much of the underwater videocameras commonly in use still use the NTSC standard (4:3aspect ratio). This aspect ratio is also standard for VGA,SVGA, DVGA. So, as an example, for cameras with a 4:3aspect ratio, it can be helpful to orient the camera in asideways orientation, so β : α = 3 : 4. This is a way to makethe haptic surface more square than rectangular, so Lx isnearly equal to (Ly,1 + Ly,2)/2.

5. MULTITOUCH IMAGE PROCESSINGOF UNDERWATER FINGERS

Detecting fingers and interpreting their motions involvedseveral steps. The first was to use skin tone masking, basedon ratios of red:green, red:blue and green:blue for each pixel.

After masking according to a set of possible skin tones,the masked image was converted to greyscale, and two iter-ations of erosion were applied to reduce noise in the fingerboundaries: first, vertical erosion, then horizontal erosion.

We then applied canny edge detection to extract contoursfrom a binary version of the eroded image. The contourboundary was numerically expanded by 20% in order to fillin gaps between finger sections that the masking may haveopened up. The set of all enclosed regions of the given framewere unioned with those from the previous frame, creatinga set-union type of moving average filter.

Canny edge detection was applied a second time to themodified finger regions, and then their centroids were calcu-lated. The (X,Y ) coordinates were transformed to physical(x, y) coordinates. Finger area created a third dimension, z.

The result was a tactile user-interface that was responsiveto finger motion. We designed a system that created musi-cal tones, where the vertical placement of fingers controlledfrequency and horizontal placement controlled amplitude.

Furthermore, the surface was multitouch, which meantthat multiple fingers inserted in the water created a musicalchord. For example, three fingers could be inserted to playa D-minor chord, and so on.

6. FINGERS VS. BUBBLESWe chose to embrace the idiosyncracies of the water sur-

face, using a separate detection system that detects bubbles

927

and splashes independently from the fingers, thus creatinga new user input space.

Fluid flow creates a large set of phenomena in responseto finger motion. Surface tension, alone, leads to variouskinds of waves, bubbles, splashes, atomizations and dropletmergers. All of these complex phenomena are controllable,and the human being at a young age gets a very keen senseof how they function, intuitively.

We created a separate detection scheme for bubbles andsplashes using colour matching based on the hues associatedwith the reflections of waves and bubbles at various angles.

As a demonstration, we had the bubble interface controla polyphonic tone generator, and the result was a gamut ofcontrollable sounds in sync with the visual turbulent splashes.

7. LIQUID PROJECTION SCREENWater in the vessel was able to make the projection dis-

play visible, given a small concentration of impurities in or-dinary drinking water. With water alone, the projection iscast in a similar way as sunlight making visible rays of lightpermeating a deep lake.

It is also possible to use other materials as a projectionscreen. Oil, such as vegetable oil, floats on the water surfaceand adds a different scattering characteristic for viewing theprojection.

8. WATERTOUCH AS A HYPERACOUSTICMUSICAL INSTRUMENT

Inherent in the WaterTouch system is a high degree ofidiosyncratic behavior that is specifically aquatic. Everytime the surface is touched, ripples and waves are generated.

These disturbances, some turbulent and some laminar,create underwater sound, over a wide spectral range. A greatdeal of the frequency spectrum extends below the range ofhuman hearing. Since many of these disturbances are con-trollable by fingers, we chose to embrace these sounds bycapturing them and shifting them up to desired notes on amusical scale, where the degree of shift depended on the lo-cation of touch on the surface. In this way, our WaterTouchsystem, when used as a musical instrument, is a hyperacous-tic instrument.

Unlike a hyperinstrument[5] in which position sensors,etc., add synthetic sounds to an acoustic instrument, hyper-acoustic instruments use the acoustically-originated soundas their primary computer input/output and process thatsound based on additional sensors or inputs [6].

To make this new hyperacoustic instrument as expressiveas possible, we wished to bring the subsonic and ultrasonicsounds into the audible range. In a way similar to (but notthe same as), superheterodyne radio reception, signals canbe downshifted and upshifted by means of using an oscil-lator in the process of frequency-shifting and various formsof selective sound filtration. However, unlike what happensin a superheterodyne receiver, we prefer to scale frequencieslogarithmically rather than linearly, in order to better matchthe frequency distribution of human perception. [7]

This digital signal processing is, in a general sense, afiltering operation, which may be highly nonlinear in cer-tain situations. The frequency-shifter was implemented ona computer system having a broadband analog to digitalconverter, i.e. an A to D converter that responds all theway down to 0Hz (DC), and up to approx. 40kHz. Com-puter vision was performed on the same computer system,with the resulting multitouch (x, y) coordinates controlling

multiple amplitudes and pitches, depending on the numberof fingers inserted. One mode had a continuum of availablepitches, while another mode constrained the pitches to adiatonic scale.

What we have done is brought the ultra-low frequencies(easily controllable by a user when touching the water [4])into the audible range, in response to a real-time visual mul-titouch analysis of finger motion across the water surface.

9. CONCLUSIONSWe successfully implemented and demonstrated “Water-

Touch”, a user-interface based on a Sensory Table (WaterTable). The apparatus was equipped with sensors that re-sulted in a water-based touch screen that captured the ran-domness and idiosyncratic behavior of water waves, ripples,bubbles, and similar phenomena, in response to user input.

The use of total internal reflection, in a pool with an un-derwater camera, created a situation in which tracking be-came very simple. For example, fingers inserted into the wa-ter were immediately visible, completely disembodied, againsta pure black background. This made the computer visiontracking extremely easy. Moreover, the simple algebraic pro-jective geometry of the air-water boundary made it very easyto locate, in multiple dimensions, anything that penetratedthe water’s surface.

10. FUTURE WORKThe use of a lake as a touch surface is being explored as

part of a grander vision of turning an entire lake into theworld’s largest musical instrument. Our hope is that thiswork helps with the creation of an Ontario Water Centre tohelp teach the world about the importance of clean lakes,rivers, and about water conservation (ongoing collaborationwith the Ladies of the Lake, Georgina, Ontario, Canada).

11. REFERENCES[1] M. B. Alonso and D. V. Keyson. MusicCube: making digital

music tangible. ACM CHI, 2005.

[2] K. Devereux et al. Effects of a water-based program on women65 years and over: a randomised controlled trial. AustralianJournal of Physiotherapy, 52(2):102–8, 2005.

[3] H. Ishii. Bottles: A transparent interface as a tribute to markweiser. IEICE Trans. on Info. and Systems, pages Vol. E87–D,No. 6, pp. 1299–1311, June 2004.

[4] R. Janzen and S. Mann. Arrays of water jets as user interfaces:Detection and estimation of flow by listening to turbulencesignatures using hydrophones. In Proc. ACM MM ’07,September 24-29, Augsburg, Germany, pages 505–8, 2007.

[5] T. Machover. Hyperinstruments: A composer’s approach to theevolution of intelligent musical instruments. In W. Freeman,editor, Cyberarts. Spartan Books, San Francisco, 1991.

[6] S. Mann, R. Janzen, and R. Lo. Hyperacoustic instruments:Computer-controlled instruments that are not electrophones. InProc. ICME2008, pages 89–92, 2008.

[7] S. Mann, R. Janzen, R. Lo, and C. Aimone. Inventing...to makea badly tuned...instrument play in perfect harmony. In Proc.ICMC ’07, Aug. 27-31, volume 1, pages 105–112, 2007.

[8] S. Mann, R. Janzen, and M. Post. Hydraulophone designconsiderations: Absement, displacement, and velocity-sensitivemusic keyboard in which each key is a water jet. In Proc. ACMMM ’06, October 23-27, Santa Barbara, pages 519–528, 2006.

[9] P. Richards. The Wave Organ. Exploratorium (acc. 2005-8),http://www.exploratorium.edu/visit/wave organ.html.

[10] H. Seichter et al. Multitouch interaction for tangible userinterfaces. IEEE International Symposium on Mixed andAugmented Reality, pages 213–214, 2009.

[11] K. van Mensvoort. Datafountain: Money translated to water.Online documentation by the inventor,http://www.koert.com/work/datafountain/, 2005.

[12] D. Yeliussizov et al. Multi-touch sensing using frustrated totalinternal reflection. Proc. App. Inf. and Comm. Tech. (AICT),pages 1–4, 2009.

928