IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. X, NO. X, MONTH 2016 1

Deformable Model-Based Methods forShape Control of a Haptic Jamming Surface

Andrew A. Stanley, Student Member, IEEE, and Allison M. Okamura, Fellow, IEEE

Abstract—Haptic Jamming, the approach of simultaneously controlling mechanical properties and surface deformation of atactile display via particle jamming and pneumatics, shows promise as a tangible, shape-changing human-computer interface.Previous research introduced device design and described the force-displacement interactions for individual jamming cells.The work in this article analyzes the shape output capabilities of a multi-cell array. A spring-mass deformable body simulationcombines models of the three actuation inputs of a Haptic Jamming surface: node pinning, chamber pressurization, and celljamming. Surface measurements of a 12-cell prototype from a depth camera fit the mass and stiffness parameters to the deviceduring pressurization tests and validate the accuracy of the model for various actuation sequences. The simulator is used todevelop an algorithm that generates a sequence of actuation inputs for a Haptic Jamming array of any size in order to match adesired surface output shape. Data extracted from topographical maps and three-dimensional solid object models are used toevaluate the shape-matching algorithm and assess the utility of increasing array size and resolution. Results show that a discreteLaplace operator applied to the input is a suitable predictor of the correlation coefficient between the desired shape and thedevice output.

Index Terms—Tactile display, haptic I/O, particle jamming, shape-changing interface.

F

1 INTRODUCTION

G RAPHICAL user interfaces (GUIs) provide an ac-cessible channel for humans to interact with

computers, serving as an essential driver in the large-scale adoption of the personal computer. However,even with the development of touch screens andmulti-touch interfaces, GUIs lack many of the touch-based features accessed during interaction with real,physical objects. To restore the touch-based elementsthat are lost at the screen interface between humansand computers, GUIs can be supplemented with hap-tic interfaces. Commercial haptic interfaces range fromwell-timed vibrations of a screen to three-dimensionalforce feedback through a tabletop kinesthetic robot,but even these options miss some of the richnessand information content afforded by tangible userinterfaces [1] and real objects.

Creating a dynamic tangible interface capable ofphysically rendering the broad range of objects orscenes that a GUI can visually render remains achallenge, with several implementations proposed byresearchers. One such implementation, Haptic Jam-ming [2], uses a combination of particle jamming andpneumatics to create a tactile display with simulta-neously controllable stiffness and deformable geom-

• A. A. Stanley and A. M. Okamura are with the Department ofMechanical Engineering, Stanford University, 418 Panama Mall, Bldg660, Stanford, CA 94305.E-mail: astan@stanford.edu; aokamura@stanford.edu

This work was supported by the National Science Foundation (1217635),U.S. Army Medical Research and Materiel Command (USAMRMC;W81XWH-11-C-0050), and a National Science Foundation GraduateResearch Fellowship.

Fig. 1. This paper develops methods to convert objectsfrom a 3-D model into a surface that a Haptic Jammingtactile display can render.

etry. One inherent advantage of the Haptic Jammingmethodology for displaying programmable surfacesover pin arrays or rigid mechanisms more commonlyused for dynamic shape output displays is that itsflexible, continuous surface can create a more nat-ural feel for organic objects or medical simulation,rather than sharp, pixelated features. However, theuse of a non-rigid structure as a basis for the dynamic

2 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. X, NO. X, MONTH 2016

surface creates challenges for modeling and controlof its configuration. While previous research [2][3]characterized the shape of ballooning single cells andthe mechanical properties of the force-displacementinteractions that human users experience, the exactsequence of cell jamming, node pinning, and chamberpressurization that results in a desired surface config-uration is unknown.

Section 2 describes existing methods for deformablebody modeling, other shape changing interfaces, anda more detailed background on Haptic Jamming. Sec-tion 3 develops a framework to simulate the flexiblebody dynamics of a Haptic Jamming surface and allof its possible input actuations. Section 4 fits mass andstiffness parameters from the model to a 12-cell pro-totype by recording surface heights during separatesequences of chamber pressurization with the cellsunjammed and jammed, and uses a depth camera tovalidate that the model holds for various actuationsequences. Based on the insights gained from thisdynamic modeling, Section 5 proposes an algorithmto control the shape output by arrays of arbitrarydimensions. In conjunction with the simulator fromSection 3, this shape control algorithm can validate thecapabilities of a Haptic Jamming surface to match de-sired shape configurations, ranging from topograph-ical maps to computer-aided design (CAD) modelsof three-dimensional objects like the one shown inFigure 1. A correlation coefficient evaluates how wellthe output matches the desired input shape, and theresults in Section 5 elucidate the characteristics thatmake a desired surface configuration achievable.

2 BACKGROUND

This work builds on previous research across a num-ber of fields, including haptics, human-computer in-teraction, graphics, and surgical simulation. The fol-lowing subsections outline existing devices and meth-ods for shape control, background information onHaptic Jamming, and modeling deformable bodies.

2.1 Shape-Changing Interfaces and ControlA tactile display that can change shape both underthe control of the computer and the user, concep-tualized as “Digital Clay” [4], represents an idealfor an advanced form of 3D computer input andoutput user interface. Display implementations in-clude arrays of shape memory alloys [5] and pneu-matics [6][7]. Pin arrays [8], and more recently theircombination with depth sensors and projectors, haveallowed greater dynamic physical affordances forsurface interactions [9]. For the majority of thesedisplays, the large number of actuators means thatshape control primarily involves down-sampling thedesired shape to the resolution of the display andpositioning each element accordingly. The SmartMeshmulti-loop mechanism [10] is based on extendable

(a) Four-Cell Array

Unjammed Cells

Jammed Cells

Pressurized Air Chamber

Pinned Node

(b) Mechanism Illustrations

Fig. 2. (a) Various configurations of a four-cell HapticJamming display (© IEEE 2013 [2]) and (b) side-viewcutaway illustrations of a larger array.

links arranged in a double-layer square grid, andthe Formable Object [11] uses a parallel rigid-bodystructure with kinematics optimized to render basicshapes like cylinders and spheres. For a “Digital Clay”display consisting of hydraulic actuators arrangedalong the rows and columns of an array [12], the res-olution of the interface can more feasibly be increasedbecause the number of actuators scales linearly ratherthan polynomially as the size of the array grows [13].

2.2 Haptic JammingA Haptic Jamming surface, shown in Figure 2a as firstdeveloped in [2], consists of a hollow silicone mem-brane molded into a flat array of thin cells. The cellsare filled with granular material such that applyinga vacuum to any individual cell can rapidly switchit from a soft, flexible state to rigid, “jammed” state.The silicone surface also creates a seal over a pressure-regulated air chamber beneath it such that increasingthe pressure will balloon soft, “unjammed,” cells orgroups of cells outward, while the rigid cells remainlocked in their current configuration. The device in-cludes mechanisms to pin the nodes, i.e. locations onthe surface where corners of multiple cells connect, atany given height. This can prevent rigid cells fromrotating about one another, as illustrated in Figure2b, and provides an extra dimension of shape outputcapability. Various sequences of cell jamming, nodepinning, and chamber pressurization create shapesof various size and complexity with controllable me-chanical properties across the surface. Genecov etal. [14] investigated the human perception of indi-vidual Haptic Jamming cells with a psychophysicalstudy. More recently, Li et al. examined the force-displacement characteristics of a human perceptionof a granular jamming chamber in series with apneumatic chamber for multi-finger palpation [15]. A

STANLEY AND OKAMURA: DEFORMABLE MODEL SHAPE CONTROL FOR HAPTIC JAMMING 3

twelve-cell prototype with the ability to pin nodes atany given height [3] demonstrated the feasibility ofcreating more complex surface geometries with largerarrays.

2.3 Deformable Body Modeling

A review article [16] by Meier et al. presents a com-parison of the most prominent deformable model-ing methods. Heuristic modeling approaches ofteninvolve linking points or volumes with springs ordeformable splines and updating the model basedon the interactions between them. Continuum me-chanical modeling approaches resolve the differentialequations governing the full body, most commonlythrough the finite element method (FEM). AlthoughFEM techniques typically require significant computa-tional time, they have shown promise for the controlof flexible bodies in soft robotics [17]. Meier et al. [16]found that the spring-mass model and the boundaryelement method (BEM) represented the most promis-ing approaches in terms of computation, topology,and biomechanical realism.

Heuristic spring-mass models suffer from non-constancy of the volume of an object, which can re-duce realism in objects that normally maintain a con-stant volume. Such models also have the potential forinstability of the integration schemes they implementas the number of springs and point masses in a modelgrows. A Haptic Jamming surface, however, does notmaintain a constant volume in reality, and its rela-tively thin structure limits the number of point massesnecessary to create an accurate model, suggesting thespring-mass model as a favorable approach. WhileFEM and BEM methods typically produce the mostaccurate results for solids with consistent materialproperties [18], they require modeling the materialas a continuous mass, which is not the case for aHaptic Jamming surface filled with discrete particlesthat falls outside the realm of traditional hyperelasticmaterials [19]. Furthermore, this work focuses on theshape control of a Haptic Jamming surface and thestudy of how increased size and resolution affectsoutput performance, so although an FEM approachcould ultimately produce a more accurate simulation,a spring-mass approach provides the desired utilitywithout requiring parallelization onto a GPU for iter-ative algorithm testing or model adjustments.

3 DEFORMABLE MODELING OF HAPTICJAMMING

Control of the shape of a Haptic Jamming displayrequires a model of the behavior of the surface andits response to all possible actuation inputs. Previousshape models were limited to a single cell balloon-ing from pressurization without any cell jamming ornode pinning [2]. To understand the shape-output

Node

Cell Boundary

Cell Interior

Fixed Array

Boundary

Fig. 3. A screen shot from a 3×4-cell version of thesimulator used to model the dynamics of a particlejamming tactile display with point masses and spring-dampers. Cell interior point masses also include atorsional spring (not shown) along each of the two axesas explained in Section 3.3.

characteristics of a multi-cell array requires a signifi-cantly more complicated model to include the effectsof interactions between neighboring cells of varyingrigidity, pinning the nodes between cells, and differentcombinations of orders of operations.

3.1 Spring-Mass Model of an Array

We model the surface as a deformable body using aspring-mass model, similar to those described in [16],such that it consists of an array of point masses con-nected by spring-dampers. We implemented the dy-namic model in C++ using the CHAI3d libraries [20]as a wrapper for OpenGL to allow visualization ofthe surface and to permit force interactions via com-mercial kinesthetic haptic interfaces. A screen shot ofa sample 3×4 array is shown in Figure 3. The fixedboundaries of the array are marked by red points,interiors of cells are marked by blue points, pinnablenodes are marked by yellow points, and bordersbetween cells are marked by green points. Cells ina jammed state have a slightly darker shade, similarto how cells in the real array become darker as theair is vacuumed out of them, packing the granulestogether more densely. We chose a grid mesh over thestandard triangle and T-2 meshes for easy scaling ofthe array size. This allows us to include any numberof rows and columns of cells and any number ofpoint masses per cell edge. Additionally, the gridmesh facilitates ensuring that all boundary pointsare equally connected to neighboring cells, which isparticularly important for the nodes at the corners ofcells which can be pinned at any height. Furthermore,a grid mesh enables the methods that we developed tomodel chamber pressurization and particle jammingof individual cells described in Sections 3.2 and 3.3,

4 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. X, NO. X, MONTH 2016

pi pL

pR

pB

pA

r A/B

r R/L

ni ^

Fig. 4. The surface normal at point i can be approxi-mated by normalizing the cross product of the vectorsconnecting its neighbors along each axis of the meshas explained in Equation (3).

respectively. Based on the 3×4 square-cell array de-veloped in [3], we set up our simulation frameworkto use square cells; their size and quantity can easilybe adjusted by changing constants in the simulation.

For a spring connecting point A to point B, thespring force acting on point B is

~FB/A

s = −k(∥∥∥~rB/A

∥∥∥− leq

)rB/A, (1)

where ~rB/A is the position vector to point B frompoint A, leq is the equilibrium length of the spring, kis the spring stiffness, and rB/A is a unit vector in thedirection of ~rB/A. The spring force acting on point A,~FA/B

s , is simply −~FB/A

s . For the corresponding damperwith linear damping coefficient b, the damping forceacting on point B is

~FB/A

d = −b(~vB/A · rB/A)rB/A, (2)

where ~vB/A is the velocity of point B relative topoint A, and its dot product with rB/A gives thecomponent of that relative velocity that is along theaxis of the spring-damper. Iterating through all of thespring-dampers in the system and adding gravity toeach point allows the acceleration of each point to becalculated as the sum of the forces acting on it dividedby its mass.

To avoid the complexity of calculating and reevalu-ating intermediate states, an explicit Euler integrationscheme updates the velocities and positions of thenon-fixed points based on the calculated accelerationvalues. Fixed points are held in place by simply notupdating their positions after the integration in eachtime step. The points on the edge of the array arealways fixed, and the pinnable nodes at the cornersof cells can be toggled between fixed and not fixed topin a node at its current height.

3.2 Air Chamber PressurizationTo simulate the effects of pressurizing the air chamberbeneath the surface, we apply an outward force to

each point mass proportional to the gauge pressureat that time step. Because there is no implicit functiondefining the surface, we estimate the normal vector ateach point using the locations of the points surround-ing it. One example of modeling pressurization inthin membranes [21] uses the weighted average of thenormal to each plane around a vertex in a triangularmesh; for a grid mesh this method would add severaladditional computations for each point.

To work around this, we define the surface normalat a point as

ni =~rR/L ×~rA/B∥∥∥~rR/L ×~rA/B

∥∥∥ , (3)

where ~rR/L is the vector to the point to the right ofpoint i from the point to its left, and~rA/B is the vectorto the point above it in the mesh from the point belowit. Figure 4 illustrates this concept for estimating thesurface normal. Taking the cross product of ~rR/L and~rA/B in this order ensures that the resulting vectorwill be pointing outward from the surface, regardlessof whether those points create a local profile thatis convex or concave. The resulting force from thechamber pressure on each point

~Fi

P = (PgaugeAspp)ni (4)

proportional to the gauge pressure, Pgauge, and theaverage surface area per point, Aspp can then be addedinto the summation of forces before calculating theacceleration in each time step of the integrator.

3.3 Simulation of Particle Jamming with TorsionalSpringsUnlike node pinning and chamber pressurization, theforces associated with jamming of individual cellsdo not have an existing modeling framework thatfits a spring-mass simulation. When no vacuum isapplied to a cell, the granular material inside canflow freely, so the flexible membrane surroundingit primarily dictates the forces associated with anydeformations. Given the elasticity of the silicone usedin a Haptic Jamming device, the spring-mass modeldescribed above characterizes an unjammed cell fairlywell. Previous research [3] developed spring-dampermodels to describe the force-displacement characteris-tics of particle jamming cells in compression, showinghow the stiffness and damping parameters increase athigher levels of applied vacuum. This result would atfirst suggest one could model jamming in a spring-mass model by simply increasing the stiffness anddamping parameters in each spring-damper within ajammed cell. However, the rheological models devel-oped in [3] describe point compressions and do notdefine overall shape deformations at various levels ofjamming, which involves some level of compliance asthe surface can take on and hold new forms.

STANLEY AND OKAMURA: DEFORMABLE MODEL SHAPE CONTROL FOR HAPTIC JAMMING 5

Beyond the issue of “stiff integrators” reported in[16], the method of increasing the spring stiffness failsto describe the behavior of a particle jamming systemfor more fundamental reasons. The application of avacuum to a particle jamming cell causes the cell tobecome rigid in its current configuration, whateverthat configuration may be. An increase in the stiffnessof all of the springs in one cell of the surface simula-tion would increase the forces trying to hold that cellflat, which could accurately simulate a cell that wasjammed while it was flat but would not model a celljammed in any other configuration.

To simulate the rigidity of a jammed cell withoutcreating an underlying three-dimensional volumetricstructure of point masses requires a torsional stiffnessthat maintains the angles between points. To definea torsional spring at a point mass along one axis ofthe mesh involves that point, pi, as well as the pointsimmediately preceding and following it, which wedenote pA and pB . Given these three points, we candefine the vectors from pi to pA and pB as ~rA/i and~rB/i, respectively. We can then calculate the angle θbetween these two vectors

θ = acos

~rA/i ·~rB/i∥∥∥~rA/i∥∥∥∥∥∥~rB/i

∥∥∥ (5)

as the arccosine of their dot product divided by theirmagnitudes. By this definition, the possible values ofθ between the vectors in 3D space are limited to rangefrom 0 to π. This and the following vector operationsare illustrated in Figure 5. Each torsional spring hasan associated equilibrium angle, θeq, that we initializeto π for all of the torsional springs in our flat meshgrid. When disturbed from its equilibrium angle, thetorsional spring will apply forces to the points to pushthem back toward equilibrium angle such that themagnitude, τ , of the resulting torque is proportionalto this deviation from equilibrium and the torsionalstiffness, kt of the spring

τ = −kt(θ − θeq). (6)

An intermediary unit vector

τ =~rA/i ×~rB/i∥∥∥~rA/i∥∥∥∥∥∥~rB/i

∥∥∥ sin(θ)(7)

normal to the plane created by the three points isnecessary to calculate the forces on each point. Theresulting forces on points A and B are calculated byrearranging the fundamental ~τ = ~r × ~F relationship,where ~τ = τ τ , so that the equations

~FA

t = τ~rA/i × τ∥∥∥~rA/i

∥∥∥ and ~FB

t = ττ ×~rB/i∥∥∥~rB/i

∥∥∥ (8)

guarantee that the force vectors, ~FA

t and ~FB

t will liein the plane created by the three points and normal

pi

pA

pB

r B/i

τ - out of plane^

r A/i

θ

FtA

FtB

Fti =-(Ft

A+FtB)

θ-θeq

Fig. 5. We use three points to define a torsionalspring along one axis of a grid mesh in a spring-massmodel, shown here with an equilibrium angle of π. Theresulting force vectors are defined in Equations (5)-(9).

to each point’s respective position vectors from pointi. Finally, to ensure that the torsional spring appliesonly a net torque to the simulation with no net force,a resultant force

~Fi

t = −(~FA

t + ~FB

t ) (9)

equal and opposite to the sum of the forces appliedto points A and B must be applied to point i. Likethe pressure forces in Equation 4, these forces areall added into the summation before integrating theaccelerations in each time step.

Points on the interior of cells, shown in blue inFigure 3, have two torsional springs each, one alongeach axis of the mesh. We group the torsional springsby cell to facilitate jamming of individual cells in thesimulation. The framework for modeling these tor-sional springs also provides a method to account forthe compliant nature of granular material-filled cells,where cells become less compliant as the vacuum levelincreases. We model this cell compliance, cc as the rateat which the equilibrium angles of its torsional springschange in each time step t

θteq = θt−1eq + cc(θt − θt−1eq ) (10)

such that they move progressively towards their cur-rent angles in each time step. Thus, the stiffnessand compliance coefficients in a torsional spring actas a spring and damper in series with adjustableparameters, or a Maxwell model, as developed in [3]to describe a jammed cell in compression. The linearsprings from the silicone stiffness act as a spring inparallel to the Maxwell model, resulting in the Zenermodel developed to describe an unsupported cell.When a cell is in its unjammed state, all of its torsionalsprings have their stiffness, kt, set to zero and theircompliance, cc, set to one. To jam a cell, the torsionalstiffness is increased and the compliance decreased toform a more rigid body in its current configuration.The accompanying video shows the simulation withvarious combinations of actuation inputs, along withclips of the real device for comparison.

6 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. X, NO. X, MONTH 2016

4 EXPERIMENTAL VALIDATION

To validate that our deformable body modeling meth-ods accurately simulate a real Haptic Jamming sur-face, we built a 3×4-cell array. The design and con-struction are similar to the device developed in [3] butwith slightly larger 38 mm cell widths to allow moreaccurate surface capture by a Kinect v2 (MicrosoftCorporation) depth camera mounted 600 mm abovethe surface. At this range, the Kinect depth sensor hasa resolution of 1 mm and an accuracy of 2-4 mm[22], and we interpolate the last ten frames beforeeach measurement in an attempt to minimize thesensor noise. We use this published accuracy as thebenchmark for the performance of our model.

4.1 Parameter Fitting

In order to validate the model with the 12-cell array,we first fit its parameters. Since the primary goal ofthis work is to model and control the static shapeoutput of the surface, we focus here on methods tofit the mass, normal spring stiffness, and torsionalspring stiffness parameters to the device. We used twotests, an unjammed and a jammed incremental pres-surization, to fit these three parameters. The first test(the unjammed pressurization test) involves startingthe surface in its unjammed and unpinned state andincrementally increasing the air pressure beneath thesurface from 0 kPa to 2.41 kPa while recording thesurface state with the Kinect at each increment. Weconducted five separate trials of this test as shown inFigure 6.

In this unjammed pressurization test, the heightsstart negative with low chamber pressure becausethe weight of the surface causes it to sag below itsunstretched flat configuration. Once the force fromthe air pressure matches the force from gravity, thesurface flattens. We use the average zero-crossingpressure (0.301 kPa) multiplied by the flat area ofthe surface in square meters (0.0174 m2 for our 12-cell array) divided by gravity to calculate the effectivemass parameter for the simulation (0.55 kg). With themass parameter known, we simulate the unjammedpressurization test with a variety of normal springstiffnesses. Plotting the center heights of the surfacesfrom these simulations at each incremental pressureshows a curve with a similar shape to the exper-imental data that either flattens out as the normalspring stiffness increases or steepens as the stiffnessdecreases. We use a modified version of the bisectionmethod, described in Algorithm 1, to find the value ofthe normal springs’ stiffness that best fits the data ac-cording to the coefficient of determination (r2) value.Because increasing the spring stiffness monotonicallyflattens the curve, this method does not suffer fromdependencies on the initial guess or finding localrather than absolute extrema. For our device, a normal

Pressure (kPa)

0 0.5 1 1.5 2 2.5 3 3.5

He

igh

t (m

m)

-30

-20

-10

0

10

20

30

40

50

60

r2 = 0.99

r2 = 0.90

Simulated Unjammed Pressurization

Simulated Jammed Pressurization

Measured Unjammed Pressurization

Measured Jammed Pressurization

Fig. 6. Surface heights during sequential pressureincreases in the device with all cells unjammed and allnodes unpinned provide the data necessary to fit themass and normal stiffness parameters in the dynamicmodel. Pressure increases with the cells jammed in aflat configuration provide the surface height data to fitthe torsional stiffness parameters.

Algorithm 1 Modified Bisection Method1: lastStepDirection← increasing2: while step > minStepSize do3: if fit(stiffness + step) > fit(stiffness - step) then4: stiffness← stiffness + step5: stepDirection← increasing6: else7: stiffness← stiffness− step8: stepDirection← decreasing9: if stepDirection 6= lastStepDirection then

10: step← step/211: lastStepDirection← stepDirection

spring stiffness of 64.5 N/m fit the data with an r2

value of 0.99.After setting the value of the normal spring stiffness

parameter, we conduct the second test (the jammedpressurization test) to find the value of the torsionalspring stiffness parameter. In this test we first pressur-ize the chamber beneath the unjammed surface untilit levels out and then jam all of its cells in that flatstate. From this state, we then remove the underlyingpressure before recording the jammed surface heightswith chamber pressures incrementing from 0 kPa to3.45 kPa. As with the unjammed inflation test, weuse Algorithm 1 to adjust the value for the torsionalspring stiffness parameter in the simulation until wefind the best fit to the experimental data. For ourdevice, we found that a τ of 504 N/radian, whichwith four point masses per 38 mm cell edge equatesto a torsional stiffness of 4.79 Nm/rad, fits the datawith an r2 value of 0.90.

STANLEY AND OKAMURA: DEFORMABLE MODEL SHAPE CONTROL FOR HAPTIC JAMMING 7

Starting Configuration After First Sequence After Second Sequence After Third Sequence

Me

asu

rem

en

tP

hysic

al D

evic

eS

imu

latio

n

Fig. 7. From its starting configuration through a series of sequences involving pinning selective nodes, jammingselective cells, and increasing chamber pressure, the model continuously fits the output shape of the actualdevice within the measurement accuracy of the depth camera.

4.2 Output ComparisonWe now use the device to verify that the modelholds for inputs and conditions beyond simply in-crementing pressure with all nodes unpinned and allcells either unjammed or jammed. For this test, wedeveloped a set of actuation sequences that subject thedevice to a variety of types of deformation. Betweeneach of the actuation sequences we capture the surfacestate with the depth camera to compare it againstthe surface that the simulator outputs for the sameactuation sequence. As shown in Figure 7, we start thedevice with no cells jammed, no nodes pinned, andno pressure in the air chamber. In the first actuationsequence, we pressurize the chamber to the levelingpressure (0.301 kPa) before jamming four of the sixcells on the right side of the array and pinning thefour nodes in the middle and right side. In the secondsequence, we increment the pressure to 0.414 kPabefore jamming the cell and pinning the node inthe bottom left corner. Finally, from this state weincrement the pressure to 1.73 kPa and record thelast surface snapshot to compare with the simulatoroutput for this same set of actuation sequences. Forthe four states shown in Figure 7, the mean absolutedifference between the simulated surface and themeasured surface of the physical device are 2.05 mm,3.39 mm, 3.42 mm, and 2.36 mm respectively, whichall fall within the range of the published accuracy ofthe depth sensor [22].

5 SHAPE CONTROL

Controlling the shape of a Haptic Jamming surfacepresents an interesting problem because its systemdynamics are nonlinear and largely configuration de-pendent. We define the starting state as an array withno cells jammed, no nodes pinned, and a chamberpressure equal to ambient pressure. The only forces

acting on the point masses at this starting state aregravity, reaction forces at the edges of the array,and the internal linear spring-dampers connecting allpoint masses to their respective neighbors, resulting ina soft mesh that sags a little bit under its own weight(Figure 8a). For the purpose of achieving a desiredsurface shape we can focus on the position of eachpoint mass.

5.1 Actuation Sequence Planning

For any given surface there are three types of possibleactuations: toggling the rigidity of a cell by jammingor unjamming it, toggling the fixed state of a pinnablenode, or adjusting the chamber pressure beneath thesurface to any value between ambient and the max-imum pressure of the supply. Because the results ofthese actuations are largely configuration dependent,applying the same set of actuations in a differentorder can result in vastly different output shapes. Forexample, jamming all of the cells before increasingthe chamber pressure will result in a rigid surfacewith approximately the same shape as the startingconfiguration, whereas increasing the pressure beforejamming will result in a rigid surface that is balloonedoutward, as illustrated with a set of three actions inFigures 8b-8d. As a result, the number of possiblesurface configurations grows exponentially with thenumber of actuations applied, which limits the effec-tiveness of typical dynamic programming methods toexplore and search the possible configurations withan optimized substructure.

Thus, rather than performing a comprehensivesearch of the possible configurations of a surface, wematch a desired surface shape using two key featuresdemonstrated by the dynamics of a Haptic Jammingsurface as modeled in Section 3. First, the height ofan unpinned node can only increase with increasing

8 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. X, NO. X, MONTH 2016

(a)

(b)

(c)

(d)

(e)

( f )

Pin g jam g pressurize

Pin g pressurize g jam

Pressurize g jam g pin

Unpin without

pressure increase

Unpin

with

pressure

increase

Fig. 8. From the starting configuration (a), the same three actions of pinning all of the nodes, jamming all but onecell, and increasing the chamber pressure applied in different orders results in different shape outputs (b), (c),and (d). The only way an increase in chamber pressure can result in a decrease in curvature of an unjammed cellis if this action is combined with another action, such as the unpinning of neighboring nodes, but the net effect ofthe pressure increase is still an increase in curvature. This is demonstrated in the transitions from configuration(b) to configurations (e) and (f).

chamber pressure, regardless of the jammed state ofneighboring cells and pin states of any other nodes.Second, the curvature of an unjammed cell can onlyincrease with increasing chamber pressure, regardlessof the current states of neighboring cells or nodes.The magnitude with which the height of an unpinnednode or the curvature of an unjammed cell increasesdue to an increase in chamber pressure still dependslargely on the state of neighboring cells and nodes, butthese heights and curvatures will not decrease so longas no states of neighboring cells or nodes are toggledsimultaneously with the chamber pressure increase.For example, unpinning all nodes bordering a celland increasing the chamber pressure simultaneouslymight decrease the curvature of that cell, as shownin the transition from Figures 8b-8f, but that cell inthe resulting configuration will have greater curva-ture than it would have had if the nodes had beenunpinned without increasing the chamber pressure, asshown in the transition from Figures 8b-8e. Therefore,the net effect of the chamber pressure increase wasan increase in the curvature of the cell. Given thesefeatures of the dynamics of a Haptic Jamming surface,we developed Algorithm 2 to control the shape of thesurface to match a desired goal configuration.

Algorithm 2 begins with a starting configuration ofan unpressurized chamber with all cells unjammedand all nodes unpinned, and then incrementally in-crease the chamber pressure and run the simulatoruntil it reaches static equilibrium. If any node in thesimulator’s array configuration passes the height ofthe corresponding point in the desired goal configu-ration, then that node is pinned at that height, andif any cell’s curvature in the simulator’s array con-figuration exceeds the curvature of the correspondingcell in the goal configuration, then that cell is jammed.

Algorithm 2 Array Configuration Matching1: while pressure < maxPressure do2: pressure← pressure + increment3: while equilibriumReached 6= true do4: updateStates(array, timeStep)5: equilibriumReached← checkForEquilibrium()6: allNodesPinned← true7: for each pinnableNode in array do8: zGoal← goalConfig.pinnableNode.z9: if pinnableNode.z >= zGoal then

10: pinnableNode.fixed← true11: else12: allNodesPinned← false13: allCellsJammed← true14: for each cell in array do15: curvatureGoal← goalConfig.cell.curvature16: if cell.curvature >= curvatureGoal then17: cell.jammed← true18: else19: allCellsJammed← false20: if allNodesPinned && allCellsJammed then21: break

The algorithm finishes when the maximum chamberpressure is exceeded or if all nodes and cells arepinned and jammed, respectively. Figure 9a shows anexample input goal configuration for a 6×8 cell arraywith 4 point masses per cell edge, developed from atopographical map of Australia (as explained furtherin Section 5.2). Figure 9b shows the resulting outputfrom Algorithm 2 and the accompanying video showsthe algorithm running on the simulator.

The algorithm requires a method to check if the ar-ray simulation has reached static equilibrium, as wellas a method to estimate the curvature of each cell. We

STANLEY AND OKAMURA: DEFORMABLE MODEL SHAPE CONTROL FOR HAPTIC JAMMING 9

(a) Input Desired Heights

(b) Simulator Output

Fig. 9. The simulator uses Algorithm 2 to try to recreatetopographic map data in a 6×8 cell grid.

define static equilibrium as any state where the sumsof the magnitudes of the velocities and accelerationsof all the points are both within a tolerance ε of zero,a condition that we can easily check each time theEuler integrator updates the states in the dynamicsimulation. To estimate the curvature of a cell, we canuse the θ angles already calculated for each torsionalspring in that cell, where each point mass withinthe cell has two torsional springs, oriented alongthe horizontal and vertical axes. We define the localsingle-axis curvature, κi, for each torsional spring atpoint i as

κi =

{π − θi, ~rA/i · ni +~rB/i · ni < 0θi − π, otherwise

}(11)

using the same vector notations defined in Figures 4and 5. This equation requires two lines to define thesign of the curvature due to the fact that θ is limited tovalues between 0 and π, regardless of the orientationof the torsional spring in 3D space. If the vectorspointing from the center of the torsional spring to eachof its endpoints are, on average, against the directionof the normal vector pointing out from the surface atthat point, as given by the sum of the dot products,then the torsional spring is bending inwards and thecell is ballooning outwards, which we define as apositive curvature. Conversely, if the cell is concaveat that point, we define the curvature as negative.We estimate κc, the curvature of cell c, as the averagevalue of κi for all torsional springs in the cell.

This averaging method for estimating the curvatureof the cell does lose some information about theoverall shape. The method ignores any eccentricity in

the cell shape, so a cell that is ballooned smoothlyacross its surface could have the same value of κc asa cell that is bent sharply on one side and flat on theother. Furthermore, saddle points where the curvatureis positive along one axis and negative along anotherwould have curvatures calculated close to zero. How-ever, the averaging method for estimating curvaturesuffices for the way it is used in this algorithm.

5.2 Topographic Map Displays

One possible application of a Haptic Jamming sur-face display is a refreshable tactile topographic map.Visual two-dimensional topographic maps typicallyuse constant elevation lines or coded colors overlaidonto regions to show elevation changes from featureslike mountains and valleys. A tactile display thatphysically changes its geometry to match the slopeswithin a region could provide an additional, intuitiveinformation channel, or even an alternative methodto convey maps and topography to the visually im-paired. Many three-dimensional globes use texturedsurfaces to present topographic information in exactlythis manner, but a refreshable display like HapticJamming could essentially zoom in on any region ofinterest to create its surface features on a larger scalebefore reconfiguring for a different region of interest.

The dynamic simulator developed in Section 3 pro-vides an efficient means to test the potential effective-ness of a Haptic Jamming interface to display topo-graphical maps, evaluate the capability of Algorithm2 to recreate desired shapes, and quantify the relationbetween the number of cells in an array and theresulting detail of the shapes it can create. Creatinga goal configuration from a topographical map forinput into the algorithm requires a desired heightfor each point mass in the simulator, so we selectedmaps with pixel values easily converted to heights.In the original 2D images of Figure 10 we invertthe hue channel to construct 3D data so that pixelswith low hue values (red-orange) correspond to highpeaks and high hue values (blue-purple) correspondto low valleys or oceans. We then down-sample theresulting gray-scale image so that each pixel maps toone corresponding point mass in the simulator. Forexample, a simulation of a 3×4 cell array with 4 pointmasses per cell edge would use a 13×17 pixel imagebecause of the extra row and column of point massesnecessary to complete the array boundary. The desiredpoint heights are scaled by their gray-scale intensity inthe image and the length dimensions of the simulatedarray before they are used as the goal configurationof the incremental pressurization algorithm.

We ran the simulator using topographical mapsof Australia and the San Francisco Bay Area as thedesired goal configuration for sixteen different arraysizes listed in Table 1. The surface heights of the de-sired inputs and corresponding simulator outputs for

10 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. X, NO. X, MONTH 2016

TABLE 1Sizes of arrays used to evaluate the shape control methods in Section 5

# of Rows 1 2 3 3 4 4 5 5 5 6 6 6 7 7 8 9# of Columns 2 3 4 5 5 6 6 7 8 7 8 9 9 10 11 12

# of Cells 2 6 12 15 20 24 30 35 40 42 48 54 63 70 88 108

Original 2D Image Constructed 3D Data

Input into 3x4 Simulator Output from 3x4 Simulator

Input into 6x8 Simulator Output from 6x8 Simulator

Input into 9x12 Simulator Output from 9x12 Simulator

Fig. 10. A 2D topographical image of Australia isconverted to 3D data, which is then down-sampledat various lower resolutions to create desired inputshapes (left column) for the dynamic simulator. Theheights of the points from the corresponding simulatoroutputs after running Algorithm 2 are displayed in theright column.

three sample array sizes are shown for the Australiamap in Figure 10. Qualitatively, these images providea number of insights about the effectiveness of HapticJamming surfaces for conveying topographical mapdata. As one would intuitively expect, the surfacefails to display sharp peaks, as there is no way forit to create a convex feature smaller than the size ofa cell. The surface can create sharp valleys, but thesesharp valleys can only occur at the nodes or along thesurface boundary. Similarly, the surface cannot createany tall features along its outer boundaries, regardlessof their profile.

To quantitatively evaluate these results we used

the two-dimensional version of Pearson’s r correla-tion coefficient. Such normalized cross correlations arecommonly used for template matching in images [23].This correlation coefficient is calculated as

r =

∑i

∑j

(Aij −A

) (Bij −B

)√√√√(∑

i

∑j

(Aij −A

)2)(∑i

∑j

(Bij −B

)2) , (12)

where Aij and Bij are the intensities of each pixel inthe two images and A and B are the mean intensitiesof each image. In this case, the pixel intensities corre-spond to heights of points on the surface. Correlationcoefficients range from −1 to 1, where r = 1 indicatesan exact match, r = −1 indicates an exact inversion,and r = 0 indicates no correlation.

Because the Haptic Jamming interface is a contin-uous surface with non-constant curvatures, there ismore dimensionality to the shape output than the mrows × n columns of cells, or even than the m×n×ppoints per cell edge used in the simulation. To accountfor this continuity, we scale the images created by theoutput point heights back up to the size of the originalimage using bicubic interpolation before calculatingtheir correlation coefficients with that original image.

The calculated correlation coefficients for both ofthe topographical maps and all of the array sizeslisted in Table 1 are plotted in Figure 11. The surfacescreated to match Australia’s topographical map resultin much higher correlation coefficients than those forthe San Francisco Bay Area. This occurs in large partbecause Australia is an island, whereas the featuresfor the SF Bay Area run off the edge of the image, aneffect that the Haptic Jamming surface cannot capablydisplay. One interesting result shown by Figure 11is that, while correlation coefficients tend to increasewith the array size as one would intuitively expect,they do not increase monotonically. More cells do notalways lead to higher correlation coefficients because,as the array size changes, the locations of the pinnablenodes relative to the key image features can shiftaround. If the nodes in one array size align partic-ularly well with the topographical valleys, it is likelythat they will not align quite as well in the array atthe next size up, and the correlation coefficient willgo down as a result. Furthermore, Figure 11 illustratesthat the payoff in increased correlation for increasingthe number of cells tends to level off rather quickly,which might be due to the fact that the number of

STANLEY AND OKAMURA: DEFORMABLE MODEL SHAPE CONTROL FOR HAPTIC JAMMING 11

Number of Cells in Array

0 20 40 60 80 100

Co

rre

latio

n C

oe

ffic

ien

t

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Australia

SF Bay

Pear

Human Heart

Pineapple

Femur

Stanford Bunny

Bicycle

Fig. 11. Correlation coefficients between 3D topo-graphical map data or desired surfaces created fromSTL files and the shape output by a Haptic Jammingsurface at a variety of array sizes.

cells grows quadratically as the number of rows andcolumns increase linearly.

5.3 Rendering 3D Models

With typical kinesthetic haptic devices like a Phan-tom Premium or a Novint Falcon, 3D models canbe physically rendered in a virtual environment byconstructing “metaballs” around each point such thatthe force it applies to the cursor is defined by acontribution function dependent on the distance fromthe cursor to that point [24]. The contribution functionis typically defined to be “compact” such that eachpoint only contributes force feedback if the cursorlocation falls within its radius of influence, and these“metaballs” can also be converted into implicit sur-faces [25]. While this permits efficient haptic renderingof highly complex surfaces with any contours, it limitsthe rendering at any given time step to the location ofthe cursor. In contrast, shape-changing interfaces likea Haptic Jamming surface allow multi-point contact sothat the user can not only trace the boundary of thesurface but also obtain a sense of the object’s overallform factor in his or her hand.

Ideally, a Haptic Jamming surface or any othershape-changing interface would have the capacity torender the shape of any three-dimensional object. Thiswould allow, for example, a designer using CADsoftware or an online shopper to hold and feel avirtual version of the product he or she is develop-ing or considering purchasing. In reality, the outputcapabilities of these interfaces often fall somewhere

(a) STL File Rendering (b) Height-Scaled Intensity

(c) Desired Input (d) Simulator Output

Fig. 12. An STL file (a) is converted to a gray-scaleimage (b) by coloring each face with an intensity cor-responding to the z-height of its vertices. Surfaces thatfold underneath the object, like the end of the aortain this human heart, are lost in the rendering, whichsuits the output capabilities of tactile display interfaceslike a Haptic Jamming surface. This image converts todesired input heights (c) with the resulting output (d)from the simulator for a 8×11 cell grid with four pointsper cell edge in the simulation.

in between two and three dimensions in the sensethat they typically cannot display all of the three-dimensional details or contours, particularly concavesurfaces that fold underneath an object.

To account for this shortcoming, we first convert the3D model data into a 2D image before converting backto 3D heights that we can input as a desired surfaceinto the shape control algorithm. We start with an STLfile, a set of vertices and triangular surfaces commonlyused to encode 3D objects, like the one shown inFigure 12a. We then color this rendering in gray scalesuch that the intensity of each face corresponds to thez-height of its vertices before taking a top-view 2Dsnapshot. The gray scale intensities in this resultingimage are then rescaled to the range 0−1 such that thelowest surfaces visible from the top view, rather thanthe lowest surfaces in the 3D model, become the zeroheight. Figure 12b shows the resulting 2D image fromthis process for the rendering shown in Figure 12a.

From the resulting 2D gray scale image, we use thesame methods described in Section 5.2 to construct3D height data to input into the shape control algo-rithm. Figure 12c shows a sample input for an 8×11cell array and Figure 12d shows its resulting shapeoutput. We calculated correlation coefficients for eachtest object with a variety of array sizes. As with thetopography evaluation, the results shown in Figure 11demonstrate that the correlation coefficient tends toincrease with array size, but not always smoothly, asthe misalignment of pinnable nodes from key features

12 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. X, NO. X, MONTH 2016

can reduce the correlation of the resulting surface tothe desired shape even if the array size increases.

5.4 Predicting Output Correlation

The results of the simulations run to calculate thedata shown in Figure 11 gives us some intuitionregarding an expectation for the correlation coefficienta Haptic Jamming surface can achieve for a giveninput shape. Smoother input shapes tend to result inhigher correlation coefficients than complex texturesor sharp bumps, and the display struggles to recreatelarge features on or near the edge of the array. Wesought a metric that could quantitatively reinforcethese insights by providing a prediction of the outputcorrelation that the surface would produce for a giveninput shape. While the gradient provides the slope ofa surface by summing the partial derivatives along itshorizontal axes, this does not necessarily describe howchallenging it will be for the Haptic Jamming surfaceto replicate it. For example, a surface ballooned out-ward is just as feasible to display as a completely flatsurface, even though the ballooned surface will havelarge gradient magnitudes at its edges. The sum of thesecond partial derivatives, or Laplace operator, givenby

∆f = ∇2f =∂2f

∂x2+∂2f

∂y2(13)

in two dimensions, helps account for the complexityin displaying shapes with large rates of change ofslope like bumpy textures or sharp peaks.

Mirroring our methods in Sections 5.2 and 5.3,where we convert between point heights in an inputshape and the intensity of pixels in an image, weapproached this problem from a digital image pro-cessing perspective. The discrete Laplace operator [26]has proven effective in applications from edge detec-tion [27] to shape analysis and segmentation [28]. Thisis calculated for each pixel as the sum of the differ-ences between that pixel and its nearest neighbors,achieved by convolving the image with a 3×3 filterkernel.

We zero pad the image for applying the filter atthe edges, rather than replicating boundary pixels asis more commonly done in digital image processing.This helps account for the fact that the Haptic Jam-ming surface cannot display features that run off theedge of the image, as demonstrated by the San Fran-cisco Bay Area example. The exact values in the kernelfor a Laplacian filter vary with a shape parameter, αL,ranging from 0 to 1, that defines the weights of thediagonal pixels relative to those directly neighboringthe center pixel. For example, αL = 1

3 gives thediagonals one-third of the total weight, resulting in

HL =

.25 .5 .25.5 −3 .5.25 .5 .25

(14)

0.7 0.75 0.8 0.85 0.9 0.95 1

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 − (Mean Intensity of Laplacian Filtered Input)

Corr

ela

tion C

oeffic

ient

r2 = 0.758

Australia

SF Bay Area

Pear

Human Heart

Pineapple

Femur

Stanford Bunny

Bicycle

Fig. 13. After filtering the input images with a discreteLaplace operator and inverting, shown to the left for9×12 cell arrays in the same order as the plot legend,the mean intensity provides a good prediction of thecorrelation coefficient with the shape created by thesimulator. Marker sizes scale with the size of the array.

TABLE 2Effects of the filter kernel weighting

αL 0 18

14

12

1

r2 0.7555 0.7583 0.7545 0.7412 0.7293

αL is the weighting of the filter kernel and r2

is the resulting coefficient of determination forfitting Laplace filtered image intensity to outputimage correlation

where HL is the filter kernel to convolve with theimage to calculate the discrete Laplace operator. InFigure 13, we plot the correlation coefficients againstthe mean intensity of the inverted image after apply-ing the Laplace filter with αL = 1

8 , with the inverse ofthe images created by the filtering shown to the left ofthe plot. High image intensity after applying the filterindicates sharp changes in slope, so it seems reason-able that the inverse of this leads to high correlationcoefficients from the simulator output. Changing αLdoes not affect the coefficient of determination (r2)very much when fitting the mean magnitudes of theinverted images to the correlation coefficients of theresulting simulator output, as shown in Table 2.

The slope of the linear fit changes with the numberof point masses per cell edge used in the simulationbecause it affects the resolution, and therefore sharp-ness of peaks, of the resized image used to calculatethe desired heights for the input surface. Running thesimulation with ten points per edge instead of four,for example, would result in lower mean intensitiesfor all of the Laplace filtered input images without

STANLEY AND OKAMURA: DEFORMABLE MODEL SHAPE CONTROL FOR HAPTIC JAMMING 13

affecting the output image correlation much. How-ever, so long as the number of points per cell edge ismaintained between simulations, the Laplace filteredinput image remains a good predictor for comparingoutput correlations.

6 CONCLUSION AND FUTURE WORK

The deformable surface of a Haptic Jamming ar-ray represents a novel form of tangible computerinterface. Using the deformable body methods formodeling the actuation inputs developed in Section3, we can simulate an array of any dimension tostudy the behavior of the surface shape under anysequence of inputs, which we validate with a 12-cellphysical device in Section 4. The simulator led to ourdevelopment of a shape control algorithm in Section 5.As a result, we can test the capabilities of the HapticJamming framework to recreate a variety of surfaceshapes from map topography to three-dimensionalsolid models, and evaluate how the output shapecorrelation varies with the number of cells in thesurface. A discrete Laplace operator applied to theinput image gives a good prediction of the correlationwith the simulator output. While this evaluation high-lights the limits of the device to create sharp peaks, italso demonstrates the benefits of using a deformablesurface as a shape-changing display, particularly forcreating organic objects with continuous contours noteasily recreated by the more pixelated shape-changingdevices that rely on arrays of rigid objects or linkages.However, this combination of shape and mechanicalproperty control also prevented us from finding a setof closed-form expressions to describe both the sur-face dynamics and the actuation inputs. An optimiza-tion based on closed-form expressions would facilitatefinding the simulation parameters, as demonstrated inadaptive radiotherapy [29] and cardiac imaging [30].Ideally, we would be able to construct a matrix like aJacobian (used for typical robotic manipulators) thatwe could invert to find the input forces needed for adesired output.

The shape control algorithm could potentially beimproved in a couple of ways. First, the algorithmrelies upon simulating the surface until static equi-librium is reached in between each action. However,a node or cell could feasibly be pinned or jammed,respectively, while the surface is still moving, poten-tially offering an opportunity to create more compli-cated curvatures within cells. Second, the algorithmviews cell jamming as a binary action, whereas inreality the vacuum level in each cell can vary alonga continuous scale. To effectively improve the shapecontrol algorithm with either of these features wouldrequire very precise matching all of the dynamic coef-ficients in the simulation to the real physical surface inuse, rather than just the stiffness and mass coefficientsfit in Section 4. Such a task would be challenging

given the current variability in the manufacturing ofthese surface displays. One could develop machinelearning algorithms to find values of dynamic coef-ficients for a specific array, or use particle filteringmethods to identify physical properties of deformableobjects as proposed in [31]. It could also help speedup the simulation enough to run learning or searchalgorithms by parallelizing it onto a GPU, as wasachieved for simulating soft tissue deformation withcutting and haptic feedback [32] and neurosurgicalsimulation [33]. Furthermore, as the correlation be-tween the desired input shapes and the resultingoutput shapes does not increase monotonically withincreasing cell size, the results in Section 5 suggestthat the output could be improved by incorporatingsome image rescaling and translation to optimizealignment of key points in the input shapes withthe nodes for the given array dimension. The currentalgorithm is also limited in that it requires a specificstarting configuration. Some applications, like zoom-ing in and out on a topographical map, could benefitfrom an algorithm that finds an optimal sequence ofactuations from any starting configuration and wouldalso require the capability of the algorithm to run inreal time.

Alternatively, the addition of sensors to the de-vice to detect its shape in real-time could allow foran entirely different shape control technique usingfeedback control, or some combination of feedbackwith the algorithms proposed in this work as a feed-forward term. These sensors, whether they would beembedded into the device like the capacitive sensorsin [34] or an external depth camera like the MicrosoftKinect used in Section 5, would enable a broaderspectrum of user interaction modalities.

ACKNOWLEDGMENTS

The authors thank Kenji Hata and Samuel Schorr fortheir help framing the shape control as a search prob-lem and testing classic AI algorithms on the simulator.Images and STL files for testing the topography and3D model rendering methods are from the followingsources: Virtual Oceana, California Department of Wa-ter Resources, Thingiverse, GrabCAD, and the MatlabCentral File Exchange.

REFERENCES[1] H. Ishii and B. Ullmer, “Tangible bits: Towards seamless

interfaces between people, bits and atoms,” in ACM SIGCHIConference on Human Factors in Computing Systems, 1997, pp.234–241.

[2] A. A. Stanley, J. C. Gwilliam, and A. M. Okamura, “Hapticjamming: A deformable geometry, variable stiffness tactiledisplay using pneumatics and particle jamming,” in IEEEWorld Haptics Conference, 2013, pp. 25–30.

[3] A. A. Stanley and A. M. Okamura, “Controllable surface hap-tics via particle jamming and pneumatics,” IEEE Transactionson Haptics, vol. 8, no. 1, pp. 20–30, 2015.

[4] J. Rossignac, M. Allen, W. Book, A. Glezer, I. Ebert-Uphoff,C. Shaw, D. Rosen, S. Askins, P. Bosscher, J. Gargus, I. Llamas,

14 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. X, NO. X, MONTH 2016

and A. Nguyen, “Finger sculpting with Digital Clay: 3D shapeinput and output through a computer-controlled real surface,”in Shape Modeling International, 2003, pp. 229–231.

[5] P. Taylor, A. Moser, and A. Creed, “A sixty-four element tactiledisplay using shape memory alloy wires,” Displays, vol. 18, pp.163–168, 1998.

[6] C.-H. King, M. O. Culjat, M. L. Franco, J. W. Bisley, E. Dutson,and W. S. Grundfest, “Optimization of a pneumatic balloontactile display for robot-assisted surgery based on human per-ception.” IEEE Transactions on Biomedical Engineering, vol. 55,no. 11, pp. 2593–600, 2008.

[7] H. Iwata, H. Yano, and N. Ono, “Volflex,” in ACM SIGGRAPH2005 Emerging technologies, 2005, p. 31.

[8] R. Howe, W. Peine, D. Kantarinis, and J. Son, “Remote pal-pation technology,” IEEE Engineering in Medicine and BiologyMagazine, vol. 14, no. 3, pp. 318–323, 1995.

[9] S. Follmer, D. Leithinger, A. Olwal, A. Hogge, and H. Ishii, “in-FORM: Dynamic physical affordances and constraints throughshape and object actuation,” in User Interface Software andTechnology, 2013, pp. 417–426.

[10] A. Mazzone and A. Kunz, “Sketching the future of theSmartMesh wide area haptic feedback device by introducingthe controlling concept for such a deformable multi-loopmechanism,” Links, vol. 3, p. 248, 2005.

[11] S. Klare and A. Peer, “The Formable Object: a 24-degree-of-freedom shape-rendering interface,” IEEE/ASME Transactionson Mechatronics, no. 99, pp. 1–12, 2014.

[12] H. Zhu and W. J. Book, “Control Concepts For Digital Clay,”in IFAC Symposium on Robot Control, 2003.

[13] R. Winck, J. Kim, W. J. Book, and H. Park, “Command genera-tion techniques for a pin array using the SVD and the SNMF,”in 10th IFAC Symposium on Robot Control, vol. 10, no. 1, 2012,pp. 411–416.

[14] A. M. Genecov, A. A. Stanley, and A. M. Okamura, “Perceptionof a Haptic Jamming Display: Just Noticeable Differences inStiffness and Geometry,” in IEEE Haptics Symposium, 2014, pp.333–338.

[15] M. Li, T. Ranzani, S. Sareh, L. D. Seneviratne, P. Dasgupta,H. A. Wurdemann, and K. Althoefer, “Multi-fingered hapticpalpation utilizing granular jamming stiffness feedback actu-ators,” Smart Materials and Structures, vol. 23, no. 9, pp. 1–11,2014.

[16] U. Meier, O. Lopez, C. Monserrat, M. C. Juan, and M. Alcaniz,“Real-time deformable models for surgery simulation: a sur-vey,” Computer Methods and Programs in Biomedicine, vol. 77,no. 3, pp. 183–197, 2005.

[17] C. Duriez, “Control of elastic soft robots based on real-timefinite element method,” in IEEE International Conference onRobotics and Automation, 2013, pp. 3982–3987.

[18] A. Nealen, M. Muller, R. Keiser, E. Boxerman, and M. Carlson,“Physically based deformable models in computer graphics,”in Computer Graphics Forum, vol. 25, no. 4. Wiley OnlineLibrary, 2006, pp. 809–836.

[19] E. Sifakis and J. Barbic, “FEM simulation of 3D deformablesolids: a practitioner’s guide to theory, discretization andmodel reduction,” in ACM SIGGRAPH 2012 Courses. ACM,2012, p. 20.

[20] F. Conti, F. Barbagli, R. Balaniuk, M. Halg, C. Lu, D. Morris,L. Sentis, E. Vileshin, J. Warren, O. Khatib, and K. Salisbury,“The CHAI libraries,” in Eurohaptics, 2003, pp. 193–205.

[21] Y. Chen and G. Medioni, “Fitting a surface to 3-D points usingan inflating balloon model,” in CAD-Based Vision Workshop,1994, pp. 266–273.

[22] H. Furntratt and H. Neuschmied, “Evaluating pointing ac-curacy on Kinect v2 sensor,” in International Conference onMultimedia and Human-Computer Interaction (MHCI), 2014.

[23] J. P. Lewis, “Template matching using fast normalized crosscorrelation,” in Vision Interface, vol. 95, no. 120123, 1995, pp.15–19.

[24] G. Wyvill, C. McPheeters, and B. Wyvill, “Data structure forsoft objects,” The Visual Computer, vol. 2, no. 4, pp. 227–234,1986.

[25] A. Leeper, S. Chan, and K. Salisbury, “Point clouds can berepresented as implicit surfaces for constraint-based hapticrendering,” in IEEE International Conference on Robotics andAutomation, 2012, pp. 5000–5005.

[26] M. Belkin, J. Sun, and Y. Wang, “Discrete laplace operator

on meshed surfaces,” in ACM Symposium on ComputationalGeometry, 2008, pp. 278–287.

[27] L. J. van Vliet, I. T. Young, and G. L. Beckers, “A nonlinearlaplace operator as edge detector in noisy images,” ComputerVision, Graphics, and Image Processing, vol. 45, no. 2, pp. 167–195, 1989.

[28] M. Reuter, S. Biasotti, D. Giorgi, G. Patane, and M. Spagnuolo,“Discrete laplace–beltrami operators for shape analysis andsegmentation,” Computers & Graphics, vol. 33, no. 3, pp. 381–390, 2009.

[29] E. Coevoet, N. Reynaert, E. Lartigau, L. Schiappacasse, J. De-quidt, and C. Duriez, “Introducing interactive inverse femsimulation and its application for adaptive radiotherapy,” inMedical Image Computing and Computer-Assisted Intervention–MICCAI 2014. Springer, 2014, pp. 81–88.

[30] H. Delingette, F. Billet, K. C. Wong, M. Sermesant, K. Rhode,M. Ginks, C. A. Rinaldi, R. Razavi, and N. Ayache, “Personal-ization of cardiac motion and contractility from images usingvariational data assimilation,” IEEE Transactions on BiomedicalEngineering, vol. 59, no. 1, pp. 20–24, 2012.

[31] S. Burion, F. Conti, A. Petrovskaya, C. Baur, and O. Khatib,“Identifying physical properties of deformable objects by us-ing particle filters,” in IEEE International Conference on Roboticsand Automation, 2008, pp. 1112–1117.

[32] H. Courtecuisse, H. Jung, J. Allard, C. Duriez, D. Y. Lee,and S. Cotin, “Gpu-based real-time soft tissue deformationwith cutting and haptic feedback,” Progress in Biophysics andMolecular Biology, vol. 103, no. 2, pp. 159–168, 2010.

[33] G. R. Joldes, A. Wittek, and K. Miller, “Real-time nonlinearfinite element computations on gpu–application to neurosur-gical simulation,” Computer Methods in Applied Mechanics andEngineering, vol. 199, no. 49, pp. 3305–3314, 2010.

[34] S. Follmer, D. Leithinger, A. Olwal, N. Cheng, and H. Ishii,“Jamming User Interfaces: Programmable Particle Stiffnessand Sensing for Malleable and Shape-Changing Devices,” inACM Symposium on User Interface Software and Technology, 2012,pp. 519–528.

Andrew A. Stanley (S’10) received the BSEdegree in mechanical engineering and ap-plied mechanics from the University of Penn-sylvania in 2011 and the MS degree in me-chanical engineering from Stanford Univer-sity in 2013. He is currently working towardthe PhD degree at Stanford University, sup-ported by a US National Science FoundationGraduate Research Fellowship. He is a stu-dent member of the IEEE.

Allison M. Okamura (S’98-A’00-M’03-SM’09-F’11) received the BS degree fromthe University of California at Berkeley in1994, and the MS and PhD degrees fromStanford University in 1996 and 2000,respectively, all in mechanical engineering.She is currently Associate Professor inthe mechanical engineering departmentat Stanford University. She has been anassociate editor of the IEEE Transactions onHaptics, an editor of the IEEE International

Conference on Robotics and Automation Conference EditorialBoard, and co-chair of the IEEE Haptics Symposium. Her awardsinclude the 2009 IEEE Technical Committee on Haptics EarlyCareer Award, the 2005 IEEE Robotics and Automation SocietyEarly Academic Career Award, and the 2004 NSF CAREER Award.She is an IEEE Fellow.