Expression Method of Human Locomotion Records for Path

Planning and Control of Human-symbiotic Robot System based on

Spacial Existence Probability Model of Humans

Rui FUKUI1, Hiroshi MORISHITA2 and Tomomasa SATO1

1University of Tokyo, 2HMI Corp.E-mail {fukui,hiroshi,tomo}@ics.t.u-tokyo.ac.jp

Abstract

In this paper, a novel describing method of humanlocomotion trajectory record was proposed, and meth-ods of its application to the path planning and con-trol of human-symbiotic mobile robots were indi-cated. The novel describing method, named as Ex-istence Record Probability Map (abbreviated ERPMhereafter), was generated through the following threesteps: 1) measuring human locomotion path usingpressure distribution sensor floor, 2) applying a prob-ability potential model of human body to the mea-sured path, and 3) integrating such probability poten-tial maps for some time period. From an ERPM, a)an existence probability value, b) ridge lines of thepotential, and c) Q-value at a point on a ridge line,were elicited. As a usage of these information, meth-ods of a) finding a position where a collision probabil-ity of a mobile robot and a human is high, b) findingmajor trajectories of human locomotion, and c) eval-uation of path width to avoid collision with a human,were indicated.

1 Introduction

A number of researches have been conducted on mo-bile robot system, and path planning to avoid colli-sion has become one of the major areas. In the caseof factory, space, and nuclear plant application, pathplanning to avoid collision with an object is empha-sized. On the other hand, development of humanoidsand welfare robots are quite active, and in the nearfuture, a human-robot symbiosis environment, whererobots assist humans in ordinary houses, may be-come feasible. In this case, not only collision preven-tion but also more sophisticated control like keepingsome distance with humans which is regarded as be-ing comfortable for the humans, will become essen-tial. To realize such human-robot symbiosis environ-ment, the following two techniques are important:

• Technique to measure the positions of humans

and robots simultaneously by the sensory equip-ments

• Technique to control the position and action ofrobots based on the measured human position

As for the first technique, there seems to be two ap-proaches: One is to install sensory equipments in theenvironment, and the other is to install them in therobots and humans such as wearable position trans-mitter and so on. The authors think the former onemore feasible because sensory equipments in the en-vironment do not restrict the ordinary movement ofhumans and robots. Intelligent Room of MIT [1],the Aware Home of Georgia Tech [2], Neural Net-work House of University of Colorado [3] are the ex-amples sensory environment system to measure thepositions of humans and robots. In the IntelligentRoom, human position is measured by vision sensor,and floor type sensor is used for personal identifica-tion in the Aware Home. The Neural Network Houseis equipped with sensors to monitor motion, temper-ature, light and sound. However, these intelligentenvironments are developed mainly for informativesupport to humans, namely they are not intended tocreate human-robot symbiosis environments for thepurpose of mechanical support to humans. In thenear future, not only informative support but alsomechanical support such as fetching things out of thereach of a human, will become important. To realizethis mechanical support, it is essential to develop thetechniques to measure and accumulate the locomo-tion of humans correctly, and to process the informa-tion to the shape suitable for path and motion plan-ning. From this viewpoint, this paper firstly proposesthe method to create a potential map named ERPM,which describes the major path of human locomotionas a potential, and secondly proposes methods to uti-lize the characteristic values elicited from ERPM forpath planning of human symbiosis mobile robots. Inchapter 2, methods to measure the locomotion recordof humans are explained. In chapter 3, the ERPM,

a novel expression of human locomotion trajectoryrecords and the characteristic values which can beelicited from it are discussed. In chapter 4, methodsof applying the ERPM and the characteristic val-ues made from actual human locomotion data to thepath planning and control of human symbiosis mo-bile robots are shown. Chapter 5 is the conclusion.

2 Measurement of human locomotion bythe sensor floor

Before explaining a novel expression method for hu-man locomotion records, this section describes mea-surement method of human movement by sensorfloor which can measure the pressure distribution asa bitmap image.

2.1 Outline of the sensor floor

To measure pressure distribution, sensor floor[4]shown Figure 1 was used for detecting human loco-motion trajectory. This sensor floor covers 2m×2marea with 256×256=65,536 pressure measurementpoints. The sensor floor consists of 16 floor sensor

Figure 1: Pressure Sensor Distributed Floor to Ob-tain Footprints of Humans

units. Figure 2 shows the details of floor sensor unit.As showed in Figure 2(right), each floor sensor unit iscomposed of upper sheet and bottom pattern. Whenpressure is applied on the upper sheet, it deformsand touches the bottom pattern to make the short-circuit. This short-circuit is detected by the con-troller. The distribution of switch ON and switchOFF information are converted to a bitmap imageand is put on the serial bus line as the output. Table1 shows the specification of the sensor floor.

Figure 2: Floor Sensor Unit

Table 1: Sensor Floor Specification

Sensing Area 2000 × 2000 mm

Sensor Pitch 7 mm

Output Data ON-OFF(1bit)

Transition Pressure 25 kPa

Sampling Frequency Approx. 16 Hz

Data Transfer Rate 115200 bps

Data Type Compressed Character

2.2 Data obtained from the sensor floor

Figure 3 is the typical data obtained from the sensorfloor. The figure displays the data of human walkingon bare foot. While human is walking on the sen-sor floor, immediately after the human foot touchesthe sensor floor, small blob image comes out. Thenthe size of blobs becomes larger and reaches to itsmaximum when all foot backside touches the floor,and becomes smaller as his foot separates from theupper sheet. This dynamic transition is captured asa series of bitmap images.

3 Novel expression method of human lo-comotion record (ERPM) and its char-acteristic values

This section describes the calculation algorithm toobtain ERPM, and the characteristic values elicitedfrom the ERPM. Figure 4 shows the flow of dataprocessing. There are five processing steps in theflow. The ERPM is obtained through the first threesteps, and the characteristic values are elicited in thelatter two steps.

1. Interpolation of human footprints by B-splinecurve.

2. Expansion of B-spline curve width.

3. Accumulating potential maps.

4. Extraction of ridge lines from ERPM.

Figure 3: Pressure Data Obtained when Walking onBare Foot

5. Calculation of Q-value at each points on a ridgeline of ERPM.

DataProcess

CalculatingQ-value at eachPoints on RidgeLine

CalculatingRidgeLine ofPotential Map

AccumulatingPotential Maps

Convert To SinglePotential Map

Interpolationby B-SplineCurve

Estimated HumanWalking Path

Human Footprintsof walking

Potential Mapof a single Path

Potential Mapof several Paths

RidgeLineCoordinates

Q-value Map

(1)

(2)

(3)

(4)

(5)

(A)

(B)

(C)

(D)

(E)

(F)

Figure 4: Data Processing Flow

3.1 Interpolation of human footprints by B-spline curve

As described before, discrete footprint image can beobtained from the sensor floor. An image of one footdoesn’t always become a single blob. In case whenone footprint consist of two blobs, one is an imageof tiptoe and the other is an image of heel. In thesecases human foot position is estimated by calculat-ing the gravity center of two or more blobs. By re-peating this step, touch down points of each foot areobtained and a B-spline curve that interpolate thosepoints is calculated. This B-spline curve is defined as

human walking path. More research should be doneto determine which is the best among various inter-polation methods, but in this paper we use B-splinecurve for convenience.

3.2 Expansion of B-spline curve width

In the previous step, we got the estimated humanwalking path. In this step, we need to express hu-man body width. There may be two kinds of hu-man body width. One is the breadth of shouldersthat occupies the space consistently. The other isthe region where hands and feet pass by beyond thebreadth of shoulders while walking. The latter regionshould be treated by a probability method. To real-ize this, probability model shown Figure 5(A) shouldbe arranged along the B-spline curve. Figure 5(A)shows that the probability becomes lower in the out-side of the cylinder. The parameters of this coniccylinder model need to be determined by experimen-tal method. In this paper, we used cone and cylin-der like Figure 5(B),(C) for simplicity, and obtainedqualitatively proper results.

DISK(C)

60pixel

1

60pixel

30

CONE(B)

IDEAL(A)

Figure 5: Models Used to Expand B-spline Line

3.3 Accumulating Potential Maps

When a human walks in a room, even if start and endpoint are fixed, the routes he takes varies accordingto the subject and to the cases. However a typicalroute emerges after many trips. To express such situ-ation, human walking data that have same start areaand same goal area are processed by step(1),(2), andERPM is made by accumulating the output.

3.4 Extraction of ridge lines from ERPM

There can be found some ridge lines in ERPM, andthey can be reagarded as typical human locomotionroutes because because these ridge lines connect thepeaks of ERPM. Usually there shows middle size

ridge lines in addition to the typical route. Sort-ing the extracted ridge lines by its height makes itpossible to detect a typical route.

3.5 Calculation of Q-value at each points ona ridge line

Even on the same typical ridge line, the potentialvalue is not constant. For example, 1) If there isnarrow area in the path where only one person canpass the area at the same time, all people must walkon that point. Therefore the potential of the pointbecomes high and the shape of potential map be-comes sharp. In the opposite case, 2) If the width ofthe road is enough for several people passing throughat the same time, people can select their route freely,and the potential becomes low and dull. To treat thisnumerically, a virtual cross section perpendicular tothe typical route is assumed. The shape of this crosssection is generally a hill shape. Firstly h is definedas the height of the hill, and w as the width of hillat the half height. Secondly a value Q = h/w isadopted, and Q is called as “Q-value”. 1)This Q-value becomes big when the ERPM is sharp at thepoint, and 2) becomes small when the potential isdull. This value can be used for indicating concen-tration degree of human walking to the typical route.

4 Experiments and Discussion

4.1 Experiments

Figure 6 shows the setup of experiments. 1m×1mGoal and Start areas are prepared at the downleftand upright corners of 2m square floor. Two chairsare used as obstacles that restrict human walkingpath. In the case of experimental setup 1, subjectsare supposed to walk between the two chairs. In thecase of experimental setup 2, subjects are supposedto walk around the two chairs. Figure 7 is the foot-prints image obtained in the experimental setup 1.In the Figure 8, “©”marks denote the gravity cen-ter of footprints image, and the line is the B-splinecurve that interpolates these points. 100 cones shownin Figure 5(B) are arranged on the B-spline curve inFigure 8, and by calculating enveloping surface ofcones the potential of single path shown Figure 9 isobtained. Figure 10 displays the potential of singlepath made by arranging disks shown Figure 5(C) in-stead of cones. Comparing Figure 10 with Figure9, Figure 9 potential seems to reflect the human lo-comotion in the sense that probability of existencebecomes lower, getting apart from the curve. Figure11 shows the potential calculated by accumulatingcone shape potentials shown in Figure 5(B) insteadof calculating enveloping surface. However this cal-culation seems to be inadequate for single path po-tential because the potential values at the center and

FloorSensor

StartArea

GoalArea

Subject

Obstacle(Chair)

StartArea

GoalArea

Setup 1

Setup 2

Figure 6: Experimental Setup

the ends on the same route are different.

Under the experimental setup 1 of Figure 6, subjectswere asked to walk from start area to goal area freelyfor 120 times. These 120 footprints images are inter-polated by B-spline curve and transformed to sin-gle path potential maps by arranging cones on thecurve. Figure 12 shows the accumulated potentialmap(ERPM) of 120 single path potential maps. Atthe center area the potential values are high, becausethe walking path is restricted by the two chairs. Startarea and goal area have certain width, and there aremany choices of routes for subjects. Therefore thepotential spread broad. Similarly, experiment undersetup 2 of Figure 6 was carried out for 120 times.Figure 13 shows the potential map(ERPM). In thisexperiment setup, there are obstacles at the center offloor, and two routes are available for subjects, rightand left routes. These two routes are clearly shownin the potential map.

Figure 14, Figure 15 is the ridge lines extractedfrom Figure 12, Figure 13 potential maps respec-tively. In Figure 14, there are main ridge line run-ning from downleft to upright, and some extra ridgelines around start area and goal area. This resultmeans that some subjects didn’t walk through themain typical route, but took other routes. For exam-ple, a subject seems to have entered the floor fromleft floor edge and exited from right floor edge. Inthe experimental setup 2 of Figure 15, some extra

Figure 7: An Example of Pressure Data Obtainedin Experimental Setup 1

50 100 150 200 250

50

100

150

200

250

Figure 8: Interpolation of Footprints by B-splineCurve

routes can be seen near the floor edges in addition tothe main routes. Among many ridge lines of Figure14, one ridge line was selected as a shortest coursefrom downleft start area to upright goal area. Fig-ure 16 shows the extracted ridge line. Figure 17 isthe graph of Q-value change along the selected ridgeline. Left edge of graph corresponds to the donwleftcorner of Figure 17 map, and right edge to the up-right corner. Q-value is quite high at the center offloor and relatively low at the corner.

4.2 Discussion

Adoption of temporal weight. In the experi-ments of this paper, each single path potential areaccumulated with the same weight. However if thepassage of time from data sampling moment to thepresent is considered in the way that weight param-eters are changed depending on the passage of time,

Figure 9: Expansion of Walking Path by EnvelopingCone-shaped Potentials

Figure 10: Expansion of Walking Path by Envelop-ing Disk-shaped Potentials

dynamic potential map can be obtained. In such adynamic potential map, typical route and Q-valueswill be changing continuously. That is to say, ifthe reciprocal number of the passage of time is usedfor weight parameter, past walking data is treatedlightly and latest walking data is emphasized.

Application for robot path planning. In a hu-man symbiotic robot system, robot and human livetogether. In such a system, path planning for robotmovement is significant in the following two points1) Robot must fit into human daily life, 2) Robotshouldn’t disturb human life. In the former case,when mobile robot fits into human life, robot canmoves on the typical route of human walking route.In this case robot should be controlled as movingalong the ridge line of ERPM. In the latter case,robot is thought not to disturb human life. If robotmoves along the ridge line, robot may collide with hu-man, and such situation is thought to be inadequate.In such case, robot should check the Q-value at each

Figure 11: Expansion of Walking Path by Accumu-lating Cone-shaped Potentials

Figure 12: Potential Map made by AccumulatingSeveral Paths (Experimental Setup 1)

points of the ridge line. Low Q-value indicates thatthe route along the ridge line have extra space forrobot, and the robot can go through this extra space.Conversely high Q-value means the route don’t haveso much extra space and people can’t walk away fromthe ridge line. A robot may come across people atsuch point because people must walk tightly alongthe ridge line, so the robot must confirm the vacancyon the route before going into such point and passthere as quick as possible.

5 Conclusion

In this paper, a novel expression method of humanlocomotion records was proposed, and basis for itsapplication to the path planning and control of hu-man symbiosis mobile robots was shown. In order toget an expression, the following two procedures arenecessary: 1) measuring the human locomotion pathby using pressure distribution sensor floor, and 2)producing 2D human existence probability map byapplying the human existence probability model tothe path and by making integration of them for some

Figure 13: Potential Map made by AccumulatingSeveral Paths (Experimental Setup 2)

Figure 14: Ridge Lines Extracted from the Poten-tial Map (Experimental Setup 1)

Figure 15: Ridge Lines Extracted from the Poten-tial Map (Experimental Setup 2)

Figure 16: One Ridge Line of Potential Map (Ex-periment 1)

0 5 10 15 20 25 30 3515

20

25

30

35

40

45

50

55

60[Q-value Map of one ridge line]

Point Number

Q-value

Figure 17: Q-value Transition One RidgeLine(Experiment 1)

time period. This map is the probability expressionof human locomotion records, and the authors namedit as Existence Record Probability Map (ERPM). Inthe experiments, 2m×2m pressure distribution sen-sor floor was used to measure the footprints gener-ated during the human locomotion. Two types ofpath settings were experimented by setting two ob-stacles on the floor. 120 trips in each setting wererecorded and ERPMs were calculated respectively.From these ERPMs, ridge lines were extracted, andQ-values were calculated. In this paper, characteris-tic values extracted from ERPM and its applicationmethod to the path planning and control of humansymbiosis robots were discussed. This applicationmethod is summarized as follows: a) Direct usage ofERPM value: probability of a human at a certainpoint is calculated by dividing the ERPM by totalnumber of trips. Therefore when a mobile robot goesinto such point that ERPM value is high, specialcare should be taken to avoid collision with a hu-

man. b) Usage of ridge line of ERPM: the ridge linesof ERPM show the typical locomotion path of hu-mans. Therefore, when one wants to control the mo-bile robot to follow the same path as humans travel,this ridge line can be used as a path for the mobilerobot. c) Usage of Q-value on the ridge line: the Q-value which is obtained from the cross section figureperpendicular to the ridge line, shows the allowancefor the mobile robot path to the typical human lo-comotion path. Therefore in case the mobile robotshould be programmed to take the same path as hu-mans take and also to minimize the disturbance tothe humans, this Q-value plays a important role. IfQ-value is low, the mobile robot can take a littleapart path from the typical human locomotion path.When the Q-value is high, the path is narrow and itis impossible to take a path apart from the typicalhuman path. In this case, the mobile robot is re-qeuested to proceed after confirming that no one ison the typical path and to pass that area as quick aspossible. Introducing the temporal weight onto theERPM will emphasize the newly recorded paths, andthis time-considered ERPM will be useful to describethe changing environments.

Acknowledgments

The authors would like to express their great thanksto our colleagues Taketoshi Mori, Tatsuya Harada,and Masamichi Simosaka for the fruitful discussion.

References

[1] Nicholas Hanssens, Ajay Kulkarni, RattapoomTuchinda, and Tyler Horton “Building Agent-BasedIntelligent Workspaces” The International Workshopon Agents for Business Automation. Las Vegas, NV,2002

[2] Kidd, Cory D., Robert J. Orr, Gregory D. Abowd,Christopher G. Atkeson, Irfan A. Essa, Blair MacIn-tyre, Elizabeth Mynatt, Thad E. Starner and WendyNewstetter. “The Aware Home: A Living Laboratoryfor Ubiquitous Computing Research” the Second In-ternational Workshop on Cooperative Buildings, Oc-tober 1999.

[3] Mozer, M. C. “The neural network house: An envi-ronment that adapts to its inhabitants.” the Amer-ican Association for Artificial Intelligence SpringSymposium on Intelligent Environments, pp.110-114,Menlo Park, CA: AAAI Press. 1998.

[4] H.Morishita, Rui Fukui and Tomomasa Sato “HighResolution Pressure Sensor Distributed Floor forFuture Human-Robot Symbiosis Environments”,IEEE/RSJ IROS 2002 Proceedings, pp.1246-1251,EPFL, Lausanne, Switzerland, September 2002.