Crowd Behaviors Analysis and Abnormal Detectionbased on Surveillance Data

Jing Cui, Weibin LiuInstitute of Information ScienceBeijing Jiaotong University

Beijing Key Laboratory of AdvancedInformation Science and Network Technology

Beijing 100044, Chinawbliu@bjtu.edu.cn

WeiWei XingSchool of Software EngineeringBeijing Jiaotong UniversityBeijing 100044, China

Abstract—Crowd analysis and abnormal trajectories detectionare the hot topics in computer vision and pattern recognition.As more and more video monitoring equipments are installed inpublic places for public safety and public management, researchesbecome urgent to learn the crowd behavior patterns throughthe trajectories obtained by the intelligent video surveillancetechnology. In this paper, the FCM (Fuzzy c-means) algorithm isadopted to cluster the source points and sink points of trajectoriesthat are deemed as critical points into several groups. Naturally,the trajectory clusters can be acquired. After refining them,the feature information statistical histogram for each one whichcontains the motion information will be built after refining thetrajectory clusters with Hausdorff distances. Eventually, the localmotion coherence between test trajectories and refined trajectoryclusters will be used to judge whether they are abnormal.

Keywords: Crowd analysis, abnormal trajectories detection,FCM, feature information statistical histogram

I. INTRODUCTION

As more and more video monitoring equipments are in-stalled in public places for public safety and public manage-ment, researchers can learn the motion patterns of crowdsand do further studies by analyzing the observed data. Asthe traditional methods can not be applied in the analysis ofunstructured situations, to overcome this problem, we proposea new approach. The approach adopted in our research focuseson the motion patterns learning and abnormal trajectories de-tecting. Our approach has several advantages: Firstly, FCM[8]is used to cluster the source and sink points, and the hidden un-structure information of the unstructured scene will be learned.Secondly, according to the hidden unstructure information,we will get the training trajectory clusters. And the parallelcoordinates which can represent data in high-dimension areused to describe the motion patterns of crowd. Thirdly, wecan judge which trajectory cluster the test trajectory mostpossibly belongs to with the hidden unstructured informationand our training trajectory clusters learned before. Then we justneed to make a compared between the test trajectory and thecluster which it most possibly belongs to, instead of the whole

This research is partially supported by National Natural Science Foundationof China (61370127, 61100143), Program for New Century Excellent Talentsin University (NCET-13-0659), Fundamental Research Funds for the CentralUniversities (2014JBZ004), Beijing Higher Education Young Elite TeacherProject (YETP0583). {Corresponding author: Weibin Liu, wbliu@bjtu.edu.cn}

clusters. As a result,the computational efficiency is improvedgreatly.

II. RELATED WORK

The data observed by monitoring equipments in a sceneusually can not be studied directly. Researchers, like Sugimuraet. al. [2], proposed a method for tracking persons in thecrowd. After transforming the observed data into trajectories oftracking objects, the crowd behaviors can be analyzed. Crowdbehavior analysis has three major aspects: motion patternslearning, abnormal behaviors detection and behaviors predic-tion. Next, we will briefly describe some of the achievementson them.

Generally speaking, motion patterns learning is the primarystep in the related studies. It practices the regular motiontrajectories, namely, motion patterns, by using the observeddata. For instance, Fatih Porikli et.al[3] learned the trajectorypatterns by computing affinity matrices and applying eigen-vector decomposition. Few years later, an improved DBSCAN(Density-Based Spatial Clustering of Applications with Noise)method was used to divide the motion flows into differentpatterns[4].

Abnormal behaviors detection aims at identifying themovement behaviors which are obviously different from othermotion tracks or have low probabilities of occurrence by usingthe motion patterns discovered before.Claudio Rosito Junget.al.[1] proposed a approach that used 4-D histogrm to makeabnormal detection. Stauffer et.al. [5] modeled each pixel as amixture of Gaussians, used an on-line approximation to updatethe model at the same time.

Behaviors prediction is a subject which attracts muchattention. Researchers desire to forecast the next moving regionor semantic behavior based on the priori knowledge andmotion patterns of moving objects. Josh Jia-Ching Ying et.al[6]combined the geographic features and the semantic featuresof users’ trajectories together, and then it evaluated the nextlocation of a mobile user based on the frequent behaviors ofsimilar users in the same cluster.

In addition, other aspects of crowds also appeal to scholars.Jan Sochman et.al [7] proposed an automatic on-line inferenceof social groups based on the Social Force Model in crowdedscenarios.

3

Figure 1: Framework of crowd analysis and abnormal detection

III. OVERVIEW OF OUR FRAMEWORK

The framework of our approach roughly includes fivestages: 1) trajectory preprocessing; 2) critical points clustering;3) rough trajectory clusters refining; 4) analysis of the behav-iors of crowd; 5) abnormal trajectories detection, as shown inFigure 1.

IV. CROWD MOTION PATTERNS LEARNING

A. Trajectory Preprocessing

In most researches, each trajectory will be represented bya sequence of flow vectors.

Fn = {f1, f2, ..., fn} (1)

Where n is the length of the trajectory. The flow vector inEquation (1) is formed by a tetrad as follow:

fn =< x(tn), y(tn), θ(tn), v(tn) > (2)

It contains the spatial, direction and velocity information ofthe trajectory at time.

As the persons pass the scene from different regionsand the speeds of them are not uniform, the trajectories ofmoving objects need a pre-process to get a unity length. Inthe preprocessing stage, we use linear interpolation to getnormalization. In that case, we can perform the followingstudy more conveniently. The trajectory of each object has itsunique features. The detailed information of it usually can notbe studied directly. Hence the researchers always make somepreprocessing to match their needs. For example, if personswho pass the scene through different regions or with variantvelocities, the length of their trajectories tracked by the cameramay have a different value. For next experiments, we should leteach trajectories be represented by the same number of flowvectors. That is to say, the number of sample points withineach trajectory should be unified so that we can assess thesimilarity between trajectories.

The common trajectory normalization methods containresample and smoothing. As the trajectory of each object isdescribed by a flow vectors set in chronological order, there weuse the resample based on one dimension linear interpolation to

make all the trajectories described by the flow vector set withthe same number. Eventually, the resample points computedby our algorithm can help to access the similarities betweentrajectories in the next processes easily.

B. Critical Points Clustering

In order to analyze the behaviors of crowd, we’d better tocluster the similar trajectories into one same group for furtherresearch. In our research, we extract the critical points of alltracks: source point and sink point of each trajectory, whichusually appear at the edge regions of the scene. As so far,researchers have developed a lot of clustering methods , it ispointed out that the standard FCM algorithm is robust to thescaling transformation of the dataset, while others are sensitiveto such transformation.

In our research, the FCM algorithm is performed for criticalpoints clustering. And then we can obtain N points groups.Obviously, we can get N ∗ N motion patterns of crowdtrajectories roughly.

C. Build Feature Information Statistical Histograms for Re-fined Clusters

After the previous works, we have gotten the rough similartrajectories. A group of similar trajectories means a patternof crowd behaviors. In order to learn the behavior rulesbetter, we should find the center trajectory whose sum ofthe Hausdorff distances to other trajectories in its cluster isminimum. And then those trajectories that have unreasonableHausdorff distances to the center trajectory of its cluster shouldbe removed. As a result we achieve refined trajectories clustersand then build the feature information statistical histograms forthem.

After the previous works, we have gotten the rough similartrajectories. In order to learn the behavior rules better, weshould remove the trajectories that have unreasonable Haus-dorff distances to the center trajectory of its cluster to refinetrajectory clusters, and transform sample (x, y) into (x, y, θ, v)to build the feature information statistical histograms whichcan describe the probability distributions of trajectories in thescene.

In order to get the histograms. First of all, weshould discretize each trajectory as a sequence of(xd

i (t), ydi (t), θ

di (t), v

di (t)). The feature information statistical

histogram Hk related to the kth cluster is built by spreadingthe according to a kernel g

Hk(xd, yd, θd, vd) =

Nk!

i=1

Nf (i)!

t=1

g(xd − xdi (t) ,

yd − ydi (t), θd − θdi (t), vd − vdi (t))

(3)

Where Nk is the number of the trajectories in kth clusterand Nf(i) is the length of the ith trajectory in cluster k andg(x, y, θ, v) is the spreading kernel.

g(x, y, θ, v) = g1(x, y)g2(θ)g3(v) (4)

4

In order to build the feature information statistical his-togram for each cluster, three steps should be followed:

1) The first step is to discretize the information of eachpoint in ith trajectory .we change the location of the mov-ing object (x, y) to a discrete value (xd, yd), where xd ∈{0, 1, ..., Nx−1}, yd ∈ {0, 1, ..., Ny−1} andNx is the numberof lines of the image, Ny is the number of columns of theimage.

2) Then we transform the local direction vector θ intoa discrete value θd, which has Nθ levels, and each levelcomprises a circular sector with internal angle:

θd ∈ {θ0, θ1, ..., θN−1}, θj =j

Nθ2π (5)

At the same time, the velocity v is turned into a discretevalue vd within Nv levels (low, middle and high)

vd =

"

0(low speed) , if 0 ≤ v < vt1(mediu speed) , if vt ≤ v < vh2(high speed) , if vh ≤ v

(6)

vt = µv − kσv, vh = vt = µv + kσv (7)

where k control the velocity, set k=2. In each cluster, we willobtain an array about the velocity of trajectories, µv is themean of the array, and σv is its variance.

3) The third step is to compute the histograms withEquation (3) and Equation (4). For g(x, y) in Equation (4),a discrete Gaussian function is the best choice. And σ is thestandard deviation of function. The normal spatial distributionin n-dimensional space is shown as in Formula (8).

g(r) =1

√2πσ2

Ne−

r2

2σ2(8)

As the research works on a 4-dimensional space, we cantransform the above equation into Formula (9).

g1(x, y) =1

2πσ2e−

x2 + y2

2σ2(9)

Then we compute the g2(θ) , the second part of Equation(4), as in the following equation,

g2(θ) = max{0, 1−|θ|ang∆θNθ

} (10)

Where θ ∈ {θ0, ..., θNθ−1}, θj is a discrete direction value,∆θNθ = 2π/Nθ and |θ|ang = min{|θ|, 2π − |θ|}.

The last item of the spreading kernel is g3(v) = δ(v)(11), δ(v) is the discrete Dirac Delta(unit impulse) function.Eventually, we can get a histogram with dimensions Nx ×Ny × Nθ × Nv for each cluster. It should be noticed that afeature information statistical histogram is computed for eachcluster and it represents a motion pattern.

V. ABNORMAL TRAJECTORY DETECTION

According to the result of previous experiments, the refinedclusters and their histograms have been obtained to help usjudge whether the test trajectories are abnormal or not. We willuse the following formula to calculate the local consistencybetween the given trajectory and the kth cluster histogram Hk.

dk(t) = Hk(xd(t), yd(t), θd(t), vd(t)) (12)

If they are local consistent at time t, the value of dk(t) willbe large. Otherwise the value will be small. These abnormaltrajectories are that most sample points of them have low levelvalues of dk(t).

We should choose a threshold value Tk for each cluster. Ifdk < Tk(13), we will regard the trajectory as local abnormalat time t. Each trajectory of kth cluster should be calculatedthe value of dik(t), then we can obtain a set Dk of kth clusteras in Equation (14)

Dk =Nk#

i=1

Nf (i)#

t=1

{dik(t)} (14)

where Tk is r-quantile of the distribution of Dk.

VI. EXPERIMENT RESULTS

This section will demonstrate the results of experimentbased on the previous work. All the experiment data are fromthe pedestrian trajectory database of Edinburgh University. Inour research, we draw 20000 pedestrian trajectories randomlyto analyze and study the motion patterns of them.

A. Critical Points Clustering

Firstly, we extract the source and sink points of all thetrajectories in our database. According to Figure 2(a)-(b), wecan see that the entry regions and exit regions are almost bidi-rectional;So we merge source set and sink set to a whole pointset for further learning. After that, we should group all thepoints into several clusters by FCM. In this study, the numberof clusters that can lead the best result will be 12. All pointclusters are shown in Figure 2(c). Through grouping the criticalpoint set into several clusters some structure information of thegiving scene can be learnt at this stage.

(a) source points distribution (b) sink points distribution

(c) all critical points set (d) density distributionof two setsFigure 2: Point clusters learning

5

group(1-7) group(2-7) group(3-12)

group(4-10) group(5-12) group(6-12)Figure 3: Several important rough trajectory clusters

B. Trajectory Motion Patterns

In the first step, 12 point clusters can be found. Then,we regard the trajectories which appeared at region A anddisappeared at region B as a rough trajectory cluster. That isto say, we have achieved 144 rough trajectory clusters. Severalprominent trajectory clusters are shown in Figure 3.

Secondly, with the purpose of learning crowd motionregularity easily and accurately, we need to refine the trajectoryclusters obtained before. A group of similar trajectoriesmeansa pattern of crowd behaviors. In order to learn the behaviorrules better, we should find the center trajectory whose sumof the Hausdorff distances to other trajectories in its cluster isminimum. And then those trajectories that have unreasonableHausdorff distances to the center trajectory of its cluster shouldbe removed. As a result we achieve refined trajectories clustersand then build the feature information statistical histograms forthem.

Moreover, building feature information statistical his-tograms for refined clusters can helps us learn the probabilitydistribution of trajectories. In order to describe all the featureinformation about the clusters, we adopt the parallel coordi-nates as shown in Figure 4.

Parallel coordinates is a common method for the highdimension data visualization. . A high dimension data point(xd, yd, θd, vd, Hk(xd, yd, θd, vd)) can be expressed as a bro-ken line. The inflection points of it are located at each parallelaxis and they can show the value of corresponding dimension.What’s more, the value of Hk(xd, yd, θd, vd) shows that thesum of the probabilities of each location sample point in clusterk belongs to the statistic class (xd, yd, θd, vd).

C. Abnormal Detection

Eventually, we extract 5000 new trajectories to make abnor-mal detections. The critical points of these test trajectories are

(a) the histogram of cluster1-2 (b) the histogram of cluster4-7

Figure 4: Feature information statistical histograms of clusters

(a) critical points of test tracks (b) critical point clustersFigure 5: Status of the critical points of test trajectories

divided into 12 groups according to the regions they belongedto as shown in Figure 5.

Moreover the status of test trajectories is displayed inTABLE I. Each element in TABLE I represents the numberof test trajectories that most possible belong to each trainingcluster learned before. Then the later work can determine thecorrectness of our preliminary judgments.

TABLE I: ENTRY-EXIT REGION TRANSFERRING MATRIXOF TEST TRAJECTORIES

Exit Region1 2 3 4 5 6 7 8 9 10 11 12

1 61 6 0 14 17 1 44 4 3 9 1 12 1 74 16 190 12 7 22 121 4 63 17 533 0 6 15 22 0 0 19 12 13 4 0 57

EntranceRegion 4 12 216 22 45 64 2 44 644 7 67 54 115

5 18 6 1 65 6 0 18 14 0 34 6 86 0 2 1 4 2 3 1 0 1 3 0 87 54 26 14 26 19 1 62 93 0 16 25 228 26 111 12 586 9 1 71 7 0 242 69 349 0 11 21 0 0 1 1 0 37 3 0 2410 1 92 13 60 8 1 22 109 1 71 16 611 5 13 1 34 3 3 24 82 0 14 13 2112 2 54 50 120 8 8 31 41 13 12 24 18

Next, we detect the local motion coherence between testtrajectories and histogram set of refined trajectory clusters,the value of r mentioned in section V is set to 20. InFigure 6, figure (a) shows most of trajectory 42305(the indexof this trajectory in test set is 42305) are local coherencewith the histogram of cluster2-12. Then in (b), most of thetest trajectory 39206 the index of it is 39206 are not localcoherence with the same histogram.So we regard trajectory39206 as an abnormal one.

The trajectory, most parts of which are coherence withthe histogram will be regarded as normal. Otherwise it willbe reckoned as an abnormal trajectory. Several prominentdetection results are shown in Figure 7. The refined trajectory

(a) dk of trajectory 42305 (b)dk of trajectory 39206

Figure 6: Local motion coherence of two trajectories betweenhistogram of cluster2-12

6

Figure 7: Test trajectories detection

clusters on the left, the normal trajectories for them on themiddle and the abnormal trajectories on the right.

In order to describe the effect of our abnormal detectionstep better, we also prepare another test set with 10000trajectories. The status of it is displayed in TABLE II. Andseveral detection results are show in Figure 8. It proves theeffct of our approach again.

TABLE II: ENTRY-EXIT REGION TRANSFERRING MATRIXOF TEST SET2

Exit Region1 2 3 4 5 6 7 8 9 10 11 12

1 71 12 0 62 32 0 314 45 3 13 34 382 5 107 18 245 18 3 41 134 5 95 47 1053 0 13 22 28 0 0 22 10 17 5 2 94

EntranceRegion 4 56 414 27 38 310 4 79 980 15 223 108 288

5 33 11 4 255 12 0 77 45 0 66 27 206 0 3 0 2 0 3 0 3 1 1 5 57 350 58 32 47 80 2 159 217 2 39 84 1038 54 116 11 843 28 1 228 22 0 380 105 999 3 13 22 1 0 1 2 0 37 9 0 2910 2 113 19 101 50 0 29 199 1 72 28 1511 30 49 5 65 24 2 75 118 0 33 26 6412 34 134 104 292 17 4 130 112 15 13 53 21

We can see that if the pass regions of a test trajectory aredifferent from most members in its corresponding cluster, itwill be found out as an abnormal one, just as most in Figure 7and 8.Even though some trajectories seem similar with othermembers in corresponding cluster in spatial aspect, the motiondirection or moving velocity may have distinct differences.They are also deemed to have another motion tendency andalso be regarded as abnormal, like the examples in the thirdline of Figure 7.

VII. CONCLUSIONIn this paper, we have done some progressive work in the

crowd behaviors analysis and abnormal trajectory detection:Experiment result indicated that the output (normal trajecto-ries) produced accurately by our method was mostly coherentwith the test cluster. The advantage of our approach is thatthe hidden unstructure information of the unstructured scene

Figure 8: Abnormal detection for test set2

are learned on the training step, so we can judge whichtrajectory cluster the test trajectory most possibly belongs towith it. Then on the detecting step, the test trajectory justneeds to compare with the trajectory cluster which has a highmotion consistency with it, instead of the whole clusters. Asa result,the computational efficiency is improved greatly.

In the future, the following work can be carried outas improvements of the method: the more optimal clusteralgorithm for critical points clustering should be implementedfor learning the cluster number automatically, and the behaviorprediction can use the motion patterns obtained in section IVto predict the next moving region and semantic behavior.

REFERENCES[1] C.R.Jung, L.Hennemann and S.R.Musse. Event Detection using Trajec-

tory Clustering and 4-D Histograms. IEEE Transactions on Circuits andSystems for Video Technology. Vol. 18, November 2008, pages 1565-1575,

[2] D.Sugimura, K.M.Kitani, T.Okabe, Y.Sato and A.Sugimoto. Using In-dividuality to Track Individuals:Clustering Individual Trajectories inCrowds using Local Appearance and Frequency Trait. 12th IEEE Inter-national Conference on Computer Vision,Kyoto, September 2009, pages1467-1474;

[3] F.Porikli. Learning Object Trajectory Patterns by Spectral Clustering.IEEE International Conference on Multimedia and Expo. vol.2,Taipei,June 2004(ICME), pages 1171-1174;

[4] W.He and Z.Liu. Motion Pattern Analysis in Crowded Scenes by usingDensity based Clustering. 9th International Conference on Fuzzy Systemsand Knowledge Discovery, Sichuan,May 2012(FSKD2012), pages 1855-1858;

[5] C.Stauffer and W.E.L.Crimson. Learning Patterns of Activity Using Real-Time Tracking. IEEE Transactions on Pattern Analysis And MachineIntelligence. Vol.22, August 2000, pages 747-757.

[6] J.J.Ying, W.C.Lee, T.C.Weng and V.S.Tseng. Semantic Trajectory Miningfor Location Prediction. ACM SIGSPATIAL GIS11,Chicago, November2011, pages 34-43;

[7] J.Sochman and D.C.Hogg. Who Knows WhoInverting the Social ForceModel for Finding Groups. IEEE International Conference on ComputerVision Workshops,Barcelona,November 2011, pages 830-837;

[8] J.C.Bezdek , B.Ehrlich, W.Full. FCM: the fuzzy c-means clusteringalgorith. Comput Geosciences, Vol.10,October 1984,pages 191-203.

7