Background Representation and Subtraction forPan-Tilt-Zoom Cameras

Luís Alves José GasparInstituto Superior Técnico / UL

luis.miguel.alves@tecnico.ulisboa.pt, jag@isr.tecnico.ulisboa.pt

Abstract—The security provided by video surveillance systemsin public and private places made camera networks ubiquitous.Current systems are mostly composed by fixed cameras. Theuse of pan-tilt-zoom (PTZ) cameras is increasing but at a slowpace. PTZ cameras offer a wider field-of-view, upon control ofthe pan, tilt and zoom degrees of freedom, but lack automatedsurveillance services.

The automatic detection of events in PTZ cameras is one ofthe steps towards the reduction of required, intensive, human-computer interaction work for targets tracking. When one eventis detected, tracking can be started. Given the prediction ofthe pose of each target, scheduling algorithms can thereforeassign visual targets to cameras and control methodologies giveinstructions to the cameras. The work here described is focusedon PTZ cameras geometric and background modeling which willbe important in future works on the detection of events.

In this work, we propose the improvement of PTZ came-ras background model, by introducing calibration methodolo-gies suitable for this kind of cameras and a multi-resolutionbackgroud model. Background subtraction methodologies arediscussed with the goal of detailing a better detection system forPTZ cameras which can take advantage of data representationswith different resolutions for potentially interesting areas.

I. INTRODUCTION

The need of video surveillance systems in public and privateplaces, combined with lower costs provided by the recentadvances on video compression and communications madecamera networks ubiquitous. Nowadays, the usage of pan-tilt-zoom (PTZ) cameras in surveillance systems is increasing butvideo surveillance systems are still mainly composed by staticcameras.

Although PTZ cameras provide a better resolution and flex-ibility in selecting different field-of-views (FOVs), their usagein automated surveillance systems is still limited comparedto static cameras. Apart from having to deal with conven-tional video processing challenges, like handling illuminationchanges, target total or partial occlusions or geometric changesof targets, they lack automation schemes required to takeadvantage of the pan, tilt and zoom degrees of freedom,without introducing intensive, repetitive, human labor.

Figure 1 lists common building blocks of a visual multi-target tracking system. Detection and segmentation of objectsof interest is the first step in an automated visual event de-tection application. Background subtraction can help with theobject detection, however, traditional background subtractionalgorithms rely on principles that are not applicable for movingcameras. During the last years, some extensions of backgroundsubtraction for moving cameras were created for situations

Detection Tracking

PTZ Camera

Scheduling

Figure 1: General Surveillance System.

where the camera motion is known, such as stationary mountedPTZ cameras, or for when the scene geometry is restricted, likedrone cameras. In those extensions, the background subtrac-tion is generalized by using motion compensation.

Lately, object-specific foreground detectors have been grow-ing in popularity. These detectors can identify objects but relyon a training phase supported by large training datasets. Whenthe detector is trained, an object is found by exhaustivelyscanning the image and applying the detector for each sectionand scale of the image. However, it is difficult to train adetector for appearance changes, like people changing clothes.

Background subtraction for moving cameras appears as oneinteresting alternative. The targets appearance just needs to bedifferent from the background. For example, it is not importantwhether a person uses a costume (Carnival, Halloween, etc)making an appearance too different from a person. Anotherexample, parking lots, even if a car is not observed parkingor leaving, it can be detected that the car entered or left thepark.

A. Related Work

In the early 1970s, information technologies, as wellas adoption of video surveillance technologies, began tospread [17]. Nowadays, video surveillance systems are ex-panding quickly in Europe and North America while on Chinathey are already part of the society.

With the interest of society in such systems, video surveil-lance became an active research and development (R&D)field. Within Instituto Superior Técnico, one finds also muchR&D, and one finds in particular supervised works as theMSc. thesis of Diogo Vicente [18], Tiago Castanheira [3],Pedro Silva [14] and Tiago Marques [10]. These MSc. workstackle the multiple components of a surveillance systemsnamely detection, tracking and scheduling and are briefly

detailed in section II-C of chapter II. The surveillance systemscomponents are also described in detail in section II-C.

The work in this thesis is focused on the aspects ofbackground representation and subtraction. Sinha et al. [15]provides a seminal work on PTZ cameras calibration andGalego et al. [6], [7], proposes an automatic calibration methodfor those cameras which take into account radial distortion.Both works represent a big contribution to PTZ camerasmodeling and are detailed in chapter III.

B. Problem Formulation

Previous work using PTZ cameras in a surveillance [12] isjust another example demonstrating the difficulty of havingonline detection and tracking of objects in a surveillancesystem scenario.

One of the first difficulties arise from the high resolution im-agery that a PTZ camera can generate. With all the backgroundimages from different zoom levels available one needs todefine a good background representation to store and use them.Multiple approaches to the camera background representationcan be explored to overcome this problem. This will be themain problem addressed in this thesis.

Once the background model is well defined, the challengeis to have background subtraction algorithms applied to highresolution background models. The results of events detectioncan then feed with data the tracking and scheduling algorithmsalready available.

C. Document Structure

Section 1 introduces the problem to approach in the thesis.Section 2 presents the pin-hole camera model followed by adiscussion on the state-of-the-art on background subtractionmethodologies. It also provides an overview of the state-of-the-art on vision based surveillance systems. Section 3 presentsin further detail different aspects of PTZ cameras geometricmodeling such as the zoom and the radial distortion effects.A discussion on calibration methologies for PTZ cameras isalso documented.. Section 4 describes the aspects related toomnidirectional background representations namely the multi-resolution representations and the memory concerns. Section5 provides an overview of the different experiments executedas well as the results obtained. Section 6 summarizes thework performed and highlights the main achievements in thiswork. Moreover, section 7 proposes further work to extend theactivities described in this document.

II. BACKGROUND AND RELATED WORK

In this chapter, we start by defining the model used inall the work and some aspects regarding camera calibration.Next we describe the state of the art regarding backgroundsubtraction methodologies and the assumptions on which theyare supported. Lastly we introduce the context of the work ina surveillance system ending with the focus of the thesis inthe topic of video surveillance systems.

A. Pin-hole Camera Model and Calibration

The model we will use in this work is the pin-hole model,with the camera center of rotation fixed and coincident with thecenter of projection while it is rotating and zooming. Theseassumptions are valid when the PTZ camera is used in anoutdoor or in a large environment, situations where the camerarotation is small compared to the distance to the scene beingobserved.

To process each scene frames and make considerationsabout the world, namely, the background, we need a functionthat back-projects a pixels [u v]T , with u representing the pixelcolumn and v the pixel line, into world coordinates [X Y Z]T ,assuming a Z value, and vice versa, i.e. projecting [X Y Z]T

to [u v]T .1) Projection: In the pin-hole camera model a 3D point

X = [X Y Z]T in 3D projective plane P 3 is projected into thecamera image plane P 2 at coordinates x = [u v]T . Lower-caseletters represent 2D points while 3D points are represented bycapital letters and the image plane is represented in blue.

The relationship between the 3D point X and the imageprojection x can be given by f : P 3 → P 2 such that x ∼ PXwith P a 3 by 4 rank-3 matrix denoted as camera projectionmatrix given by

P = Kz [R t] , Kz =

f z s czx

0 f z czy

0 0 1

(1)

in which K represents the camera intrinsic parameters, Rthe rotation matrix, t the translation vector and ∼ symbolrepresents an equality up to a scale factor. Together, thematrices K and t represent the camera extrinsic parameters.The matrix K is expressed in function of s, the camera pixelskew parameter (also named x:y pixel ratio) associated to everycamera, f the focal length and c = (cx,cy) the principal point.Both f and c depend only on the camera zoom z thereforethey are represented as f z and cz = (cz

x,czy).

The relation between a world coordinate X = [x y z 1] inhomogeneous coordinates and an image point x is given by a3 x 4 matrix P called projection matrix, defined up to scaleby a factor λ such that

λx = PX . (2)

2) Back Projection: Given the camera equation 2, andknowing the world coordinates of a point allows obtainingthe image coordinates in pixels. The inverse operation is notpossible since the matrix P is not a square matrix. Knowingthe image coordinates([u v]T ) in pixels we can estimate the[X Y ] values for each Z coordinate. To overcome this, all 3Dpoints are defined as belonging to a line that goes through theoptical center and has the same direction of the optical ray.

3) Calibration: The main goal of camera calibration is tofind the camera matrix P since it contains all the camerainformation necessary to relate x and X. The 3 x 4 matrixP has 12 elements, but since the scale factor is arbitrary, ithas 11 degrees of freedom. There are 3N equations (3 foreach point projection) but, since each projection introduces

one additional unknown λ, we need at least 6 points in orderto have the problem well defined.

One option for the geometric calibration of a camera isto use the Direct Linear Transformation (DLT). Section IIIdescribes in more detail some of the available methods foran automated and non-automated calibration of PTZ cameraswhich may include DLT.

B. Background Subtraction

Background subtraction is based on the assumption thatthe difference between the current image and the backgroundimage is caused by a moving object in the scene. Most ofthe scientific literature available denotes moving objects asforeground since most of the moving objects are part of theforeground.

In general, background subtraction requires assumptions as:the camera and its parameters are fixed; the scene does nothave illumination changes; the background is static and noobjects move or are inserted into the background; the initialbackground does not contain sleeping objects that will movelater into the foreground; the background and the foregroundcan be separated by a threshold when differencing the currentimages and the background. In practice, these assumptionsare usually hard to meet due to the wide range of real worldconditions.

We present three different methodologies which have somesimilarities but with different concepts to reach the samegoal of obtaining a precise object segmentation. Each methodintroduces an improvement over the previous one.

1) Basic Background Subtraction: With Basic BackgroundSubtraction (BBS), as in [11], the foreground mask is com-puted by getting the absolute difference between the currentframe image and a static background image and comparing theresult with a threshold previously defined. All the image areaswhere the result is higher than the threshold will be consideredas active areas.

The BBS has a number of drawbacks. Firstly it is notrobust to different lightning and scene conditions since thethreshold is not dynamic. Its value might be suitable forsome conditions and bad others. It also depends on previousbackground estimation. While for static cameras it might bepossible to do this estimation using an empty scene as thebackground, for PTZ cameras that can constantly change theirorientation and zoom level, it might be difficult to get a goodbackground estimation, making this method not reliable forthese cameras. Additionally, if parts of the background startmoving, like leafs of a tree, they will also be detected.

2) Frame Differencing: Frame differencing consists of sub-tracting two adjacent images and then filter the stationarybackground by comparing against a threshold. As previousmethod, the areas where the result of the subtraction is higherthan the threshold are classified as foreground whereas theother areas are classified as background.

This method quickly adapts to environment changes andcamera movements although when objects part of the fore-ground stop, they will not be detected. Objects that areuniformly colored will also be a problem when applying this

method since in this case, when we subtract the current frameand the previous frame, only the leading and trailing edge willbe detected, leaving a lot of pixels not recorded. This behaviormakes it difficult to understand if the object is moving towardsor away from the camera.

A possible solution to improve frame differencing is a three-frame differencing method as [19]. Now, not only the pastframe is taken into account but also a future frame. Thisrequires either that the algorithm is not running online or that amotion model of the objects is known, so we can predict wherethe object will be. The results will however differ dependingon the time interval between the frames selected since objectshave different sizes and will move at different speeds.

3) Motion Images Applied to Background Subtraction:Motion history images (MHI), a concept described originallyby James W. Davis in [5] and further developed in [4]represent a method that instead of focusing on trajectories ofthe object, focuses on motion patterns. The MHI is constructedby successively layering selected image regions on top ofeach other over time. This framework allows the representationof both the position and temporal history of moving objectsby marking each pixel with the last time instant it wasdeclared foreground, making it an interesting application forbackground subtraction algorithms. The slowly moving objectsthat are often missed by frame differencing are now taken intoaccount as foreground.

C. Surveillance with Pan-Tilt-Zoom Cameras

A common goal of surveillance systems is the detectionand tracking of moving targets like pedestrians, vehicles, etc,in the environment. With the rapid growth in the numberof surveillance cameras, the demand and development ofautomated methodologies for detection and tracking based onimage processing techniques also increased.

Currently, most surveillance systems are based on fixedwide field-of-view (FOV) cameras. Wide FOV comes at thecost of losing resolution. With the usage of PTZ camerasin surveillance systems it is possible to adapt the FOV andtherefore the area of the environment covered by the camera.

The challenge is to find an optimal control policy fortracking the maximum number of targets with the maximumresolution possible. Having the PTZ parameters, namely thethree degrees of freedom for the control, the challenge of usinga scheduling policy is how the degrees of freedom can beused in order to have a fair use of zoom level together witha good number of targets being tracked. A study, a proposedsolution, and an implementation is provided in the MSc. thesisof Tiago Marques [10] while on MSc. thesis of Pedro Silva[14], a multi-target tracking algorithm is proposed.

In this work, we will focus in the first step of detection oftargets in a surveillance system where images are processedand the detection and segmentation of objects of interestoccurs.

1) Freely Moving Cameras versus PTZ Cameras: Most oftoday’s video streams are captured from moving platformslike camera phones, cameras mounted on ground vehicles ordrones, or mounted on robots. While moving cameras, mostly

with fixed zoom, can have translations during video streams,cameras used in video surveillance systems, like PTZ cameras,do not usually translate and have the zoom as a degree offreedom. PTZ cameras can be categorized as a type of movingcameras and share some of the common problems when itcomes to background/foreground segmentation.

In [13], Sheikh et al. proposed a pixel-wise labeling of fore-grounds and backgrounds for freely moving cameras. It uses anorthographic camera model and assumes that the backgroundis the main rigid entity of the image in order to get three basictrajectories corresponding to the camera motion. On the otherhand, Kwak et al. [8] does not use orthographic projections butperspective ones. Motion information is explicitly determinedwith the help of a particles filter to estimate poses.

In the case of PTZ cameras equipped with odometry on thepan, tilt and zoom motion degrees of freedom, one does needto estimate the camera motion. In other words, conventionalmotion estimation methodologies based on feature tracking arenot required.

2) Background Subtraction for PTZ Cameras: Consideringstatic cameras, foreground can be defined as all the visualinformation that is not present in the static background. Formoving cameras, like PTZ cameras, this distinction is ambigu-ous. Several background subtraction methods were developedin the last years.

In [5], Davis et al. introduced the concept of motionhistory images where images are built by successfully layeringselected image regions on top of each other. This methodfocuses on motion patterns over object trajectories.

Later, James Davis [4] developed the motion history imagesconcept, addressing previous problems on limited recognitionby building a gradient-based motion pyramid to extract localmotion flow directly from motion history images.

The background subtraction approaches referred in thissection are computationally inexpensive to represent, charac-terize, and recognize target motions with fixed cameras.Therepresentations described here capture both the position andtemporal history of a moving object they can be used to selectthe foreground and the background pixels.

In our case, we want to consider PTZ cameras, and needtherefore to deal with the background apparent motion due tothe camera motion.

3) PTZ Cameras Modeling and Background Representa-tion: Previous dissertations on tracking and scheduling withPTZ cameras was already developed in Instituto SuperiorTécnico. In Tiago Castanheira MSc. thesis [3], it is proposedan automatic system to track buses in a parking lot by using anExtended Kalman Filter (EKF). The EKF revealed satisfactoryresults for position estimation although all the testing was onlydone under a simulation environment. A multi-target visualtracker with the help of PTZ cameras is presented in PedroSilva MSc. thesis [14]. The tracker combines kernel-based andparticle filter trackers using a tridimensional target model. Anheuristic scheduling algorithm to coordinate a set of cameras isalso developed. In the case of Tiago Marques MSc. thesis[10],an information theory approach is used to solve the schedulingand control problems.

Regarding cameras calibration, previous work on PTZ ca-meras automatic calibration was developed by Galego et al. [6]regarding radiometric effects inherent in the operation ofthose cameras. From the same author, in [7], a method forPTZ cameras auto-calibration over the cameras zoom range isproposed.

The MSc. thesis of Diogo Leite [9] also proposed anautomatic calibration method over the camera’s zoom rangeand a new methodology for automatic surveillance control fortracking and prediction targets trajectories based on memoryof past events is presented.

D. Focus of the Thesis

In the thesis the focus will be in proposing an improvedbackground model that allows the complete FOV represen-tation of a PTZ camera. The model defined should providea satisfactory framework for events detection so that, in thefuture, the detection algorithm could provide data to previ-ously developed algorithms on multi-targets tracking and PTZscheduling and control algorithms like the ones presented inthis section. To accomplish the background model a goodstudy of PTZ cameras geometric model is need.

III. PAN-TILT-ZOOM CAMERAS GEOMETRIC MODELING

With this section, we introduce the constrains added byhaving a camera that can zoom in and out and how theradial distortion effect is noticeable for different zooms. Themodeling of this aspects is essential to ensure a precisebackground representation is obtained.

A. Zoom Model

Differently from cameras with fixed optics, PTZ camerasallow zoom in/out operations. From a geometric point of view,the zoom transformation can be seen as an 2D homography.

Assuming a pin-hole camera model, given a set of pointsx and x′, image of X but taken at two different time instantsby a camera that is zooming, we can compute a projectivetransformation that maps each x to x′. These points, x andx′, are related to X as x = K[R t]X and x′ = K′[R′ t]X wheret = 0. Therefore, x′=R′R−1K−1x. The matrix K is the cameraintrinsic matrix for a particular zoom level. If we assume thecamera is zooming with a fixed center of projection, we canwrite x′ = K′K−1x.

The 2D homography can be written as Hzoom = K′K−1.

B. Radial Distortion Model

Many cameras, including PTZ cameras at minumum zoom,will slightly deviate from the pin-hole model due to radialdistortion. Figure 2 was acquired with an Axis 5512 PTZCamera at the maximum zoom out level (0%) when showing achess calibration pattern in front of the camera and illustrateshow noticeable the distortion of the lens is, in particular inthe image corners. The red lines placed afterwards on top ofthe pattern represent a grid with the same size of the chesssquares and illustrate in an easier way the effect of radial

Figure 2: Image acquired with Axis 5512 PTZ Camera at themaximum zoom out level (0%) with a perfect grid overlay inred color.

distortion. It is visible that the chess pattern squares are notsquares anymore in the image acquired.

Radial distortion is therefore an important subject to modelbefore introducing the PTZ camera images into further pro-cessing algorithms, such as the construction of a panoramicbackground representation.

The radial distortion makes a 3D point X be projected toa point xd = [xd yd ]

T different from the expected, undistorted,point xu = [x y]T . The point is deviated according to a radialdistortion function L where[

xdxd

]= L(r)

[xy

],r =

√x2 + y2,L(r) = 1+ l1r2 + l2r4 (3)

with r being the radial distance, the distance from point xuto the center of distortion (x0,y0). The point xd can then bewritten as

xd = (1+ k1 ‖xu‖2 + k2 ‖xu‖4)(xu− x0)+ x0 . (4)

The radial distortion for each zoom level, Lz, can beparameterized by (xz

c,yzc,k

z1,k

z2). Constraining the principal

point (cx,cy) to be equal to the distortion center x0 = [x0 yc],the radial distortion function for each zoom level can beparameterized by the coefficients kz

1 and kz2.

C. Camera Calibration based on a Standard Toolbox

Another important aspect to model in PTZ cameras is theinstrinsics parameters of a camera. Jean Ives Bouguet’s [1]developed a method that estimates the intrinsic parameters of acamera by capturing images of a planar chess pattern at variousorientations. The developed method outputs not only the focallength and the principal point but also the skew coefficient andthe radial distortion.

This method is considered a manual method since it needsa person to go in the front of the camera with the pattern andlater on, a person to identify the points of the pattern in theimage. Automatic methods for camera calibration are availableand documented by Agapito et al. [2] and Galego et al. [7]but due to the lack of textured regions in our test environment,those methods described were not suitable in several regions.

D. Calibration Based on Focal Length Parameter

An alternative method to Jean-Yves Bouguet Camera Cal-ibration Toolbox to obtain the focal length parameter in

function of the zoom level of a PTZ camera is based on realworld measurements with a measure tape.

From the relation between a world coordinate X = [X Y Z]and an image point x = [x y z] described in equation 5, focallength value can be estimate knowing the X and Z coordinateof a world point and the respective u image point.

uv1

=

Su 0 00 Sv 0cx cy 1

XYZ

⇔{λu = SuX

λ = Z⇔ Su =

ZX

u (5)

E. Calibration based on a Grid on the Ground Plane

Following previous section, where a calibration method-ology is based in scene measures, we propose a calibrationmethodology for a PTZ camera based on a calibration patternplaced in the ground floor which estimates the intrisics of thecamera given the camera height, the distance from the camerato the pattern and the tilt angle of the camera.

w1zw

1Y

{w1}

w1X

pz

pY

pX

{w2}

{p}

w2z

w2Y

w2X

{c}

d1

d2

tilt

Figure 3: Referentials used in the calibration method proposedwith the use of a calibration pattern.

In the setup it is assume the X-axis of the calibration patternreferential is parallel to the X-axis of the camera referential(in a ab notation where b represents the axis in a referentialthen we must have px // cx) which in practice is obtainedby enforcing that the X-axis of the camera is parallel to theground plane. In other words it means that the camera just tiltson an axis parallel to the ground plane or that the camera doesnot roll with respect to the ground plane. Figure 3 illustratesthe referentials used in the setup.

From the assumption that the X-axis of the camera refer-ential is parallel to the ground plane referential the remainingreferentials are built accordingly. The main world referenceframe, {w1}, is constructed by first projecting the origin of thecamera referential to the ground plane at the origin, definingthat the X-axis of the world referential is parallel to the X-axis of the camera referential (cx // ground plane). Next, wedefine the Z-axis in the world referential to be on the groundand on the plane with the X-axis of the X-axis of the patternreferential being parallel to the X-axis of world referential.TheX-axis of the pattern is therefore parallel to the X-axis of theworld referential (px // w1x) and the Z-axis of the patternreferential is parallel to the Z-axis of the world referential(pz // w1z).

Recalling Equation 1 that maps a 3D data point into a 2Dimage point if the extrinsics of the camera are defined bya series of transformations based on the camera height, thedistance from the camera to the pattern and tilt of the camerawe can write a 2D image point by

m∼ K [R | t]p︸ ︷︷ ︸cTp

M, cTp =cTw1

w1Tw2wTp (6)

where the transformations cTw1 , w1Tw2 and w2Tp are defined infunction of the distance d1, the distance from the origin of thepattern referential, pO, to the origin of the {w1} referential,w1O, along the Z-axis of the {w1} referential (or Z-Axis of{p} referential), the distance d2, defined as the distance fromthe origin of {p} referential, pO, to the origin of the {w1}referential along the X-axis of the {w1} referential (or X-Axis of {p} referential), the distance h is the camera heightrelatively to the ground and lastly of the angle θ whichrepresents the tilt angle (Figure 3).

If we define a 3D point in the pattern as pM =[i.∆x 0 j.∆z 1]′ a 2D point m=(i, j) in the image can thereforebe defined as

m∼ K

(Rx(θ)

i.∆x0

j.∆z

+Rx(θ)

d2hd1

) (7)

where ∆x and ∆z are the calibration pattern rectangles sizeshorizontally and vertically, respectively.

In Equation 7 both the image points and the world pointswere measured leaving the intrinsics matrix K and the angle θ

as the unknowns. Since the angle θ can be measured from thecamera when acquiring the image of the calibration pattern,Equation 7 can be used to solve a optimization problem andfind the best values for the intrinsic matrix.

F. Radial Distortion Estimation with Line Fitting

Since we are dealing with a variable zoom camera andtherefore a variable focal length parameter, for low zoomlevels, where the focal length has the lowest values, the radialdistortion effect is noticeable.

This section describes a procedure based in line fitting inorder to estimate a good approximation of the radial distortion.

If we take figure 2 to have the squares matching the red lineswe can create an optimization problem in which the radialdistortion coefficients are tweaked so the points x and xd areas closest as possible.

Defining the optimization problem as a line fitting problem,each red line which contains a set of undistorted points definedby a linear equation and each set of distorted points whichshould belong to the line must have the lowest possibledeviation from that line. The described logic is applied to boththe horizontal and vertical lines.

IV. PAN-TILT-ZOOM CAMERAS BACKGROUND MODELING

On chapter II we described some background subtractionmethodologies. All of them had an important element incommon: the correct estimation of the background. Without

the background well defined, the foreground identification willalso fail. For PTZ cameras, most of the assumptions madein section II-B are not valid. We have now three degrees offreedom - pan, tilt and zoom - which have to be consideredto build a correct background representation.

A. Background Representation

To make an omnidirectional representation we only need topan and tilt the camera. The zoom is also a degree of freedomthat can be explored to improve the representation of the scene.An option is to have multiple levels of zoom included in theomnidirectional representation.

1 frame WxH

min. zoom (max. FOV)

1 frame WxH

medium zoom

1 frame WxH

max. zoom (min. FOV)

Figure 4: Zooming camera resolution representation.

In a real scenario, a camera that allows the zoom operationbut does not pan neither tilts, can fill one complete pyramidand leaves partially empty other neighbor pyramids. In fig-ure 4, a zooming camera is presented to illustrate how theresolution and the FOV are related in such cameras.

In figure 5, the zooming camera defines multiple (side-by-side) pyramids although it can only fill them partially, as thecamera is assumed here to be just zooming in or out withoutpan and/or tilt.

1 frame = A1 + B

1 + C

1 + D

1 + E

1min. zoom

(min. number of pixels per

degree of FOV)

max. zoom

(max. number of pixels per

degree of FOV)

1 frame = A2 + B

2 + C

2

1 frame = A3

A1

B1

C1

D1

A2

B2

A3

C2

E1

Figure 5: Zooming allows multiple FOVs and allows thedefinition of multiple pyramids.

A pan-tilt-zoom camera allows defining multiple pyramids(as the zoom-only camera) and allows filling the completeFOV in the pyramids. Figure 6 illustrates the advantage ofhaving a PTZ camera to represent the complete FOV. It isworth noting that some data will be redundant. Frame f4 isredundant given f1 in opposite to f6. Frame f5 is partiallyredundant given f2. A balance in the acquisition must be done,interpolating data when possible.

B. Setting-up in a Real Scenario

In some environments, e.g. outdoors, large portions ofthe scene may contain regions which are not desired to bemonitored.

f1

f2

f3

f4

f5

f6

Figure 6: Zoom (lens) fixed camera allows acquiring framesf1, f2, f3 and f4 but not f5, f6 and many others not representedin the image. To directly acquire f6, a camera rotation isneeded.

For instance, in a surveillance system of an outdoor parkinglot, the sky or the walls do not require a high detail incomparison to objects of interest, like cars or people. To avoidrepresenting those parts, two possible approaches can help inthe decision of zooming or not. One, described in [16], is toavoid zooming into textureless regions like for example thesky, walls, roads. The second is to capture the depth value ofeach pixel and decide according its value. Regions like thesky or buildings far away do not need to be zoomed in. Inorder to have the corresponding depth value for each pixel ofthe scene, we need an auxiliary depth measuring device. Theauxiliary device can be a camera used to calibrate the on-sitePTZ camera during the setup of system and later it can beused to work in parallel with the main PTZ camera and getthe depth values.

In the impossibility of having an analysis regarding thetexture of the scene prior or during the acquisition, savingthe images acquired from all the zoom levels provides theflexibility to decide later which regions are useful for theneeded applications. With this option we can provide a om-nidirectional background representation for each zoom levelused.

C. Omnidirectional Background Representation

To represent the background of a PTZ camera we pickthe cube representation since it handles a complete sphericalFOV, 360◦ x 360◦, as opposed to the planar or cylindricrepresentations.

Two main functions are needed to use the backgroundmodel: first the computation of a back-projection for eachimage point and then the back-projections of those points inthe cube faces [6].

1) Back Projection and Identification of the Correct CubeFace: If we know the camera intrinsic parameters and thepan and tilt angle orientations used for acquiring the specificimage, we can back-project a 2D point into a 3D using theequation

[X Y Z]T = (K R)−1m (8)

with K being the intrinsic parameters matrix and R the rotationmatrix. To obtain the 2D coordinates of a 3D point on the faceof the cube, we need firstly to determine which right face ofthe cube to project the point.

Knowing the 3D point information and the size of a cubeface, we can find a latitude angle, dependent on longitude,known as critical latitude, ϕc, which indicates whether theoptical may be pointing to the top or bottom faces.

Given the critical latitude and by converting the 3D point co-ordinates to spherical coordinates longitude, θ= arctan(X/Z),and latitude, ϕ = arctan(−Y/

√X2 +Z2), we can determine

which face of the cube corresponds a specific point.2) Projection to a Cube Face: Now that we know the

cube faces for each point, the next step is the mapping ofthose points into the corresponding cube faces. The mappingcorresponds to the projection of the back-projection of thepreviously computed image points using a projection matrixPWF = KF

[RWF 03×1

]where KF is an intrinsic matrix

characterizing the resolution of the cubes faces and RWF arerotational matrices defining optical axis orthogonal to eachcube face.

An image point m is therefore mapped on a cube face mFi

as mFi ∼KFRWFR−1K−1mi.

V. BACKGROUND SUBTRACTION ON THEOMNIDIRECTIONAL REPRESENTATION

The omnidirectional background representation introducedin the previous section allows to collect image data acquiredby the pan-tilt-zoom camera, during background construction.In an installation phase, or upon required updates due to scenechanges, the pan-tilt-zoom sweeps along the ranges of the pan,tilt and zoom degrees of freedom and integrates the acquiredimages.

During operation, one acquires one image at a specificsetting of pan-tilt-zoom and then reads from the backgroundrepresentation a corresponding image with the same settings.This allows using the background methodologies introducedin chapter 2.

The zoom degree of freedom in practice is limited to asmall range, chosen to allow detecting scene changes, e.g.humans moving, with a number of pixels larger than a min-imum threshold. The sweeping of the pan-tilt-zoom degreesof freedom implies acquiring multiple values for each point(pixel) of the cube face, which are then averaged to the fillthe pixel value.

In other words, the omnidirectional background representa-tion allows to collect image data acquired at multiple zoomlevels, but only saves according to the resolution chosen forthe cube faces. The choice of the resolution of the cubefaces involves a tradeoff between accuracy and model memory.Choosing a large resolution allows saving background imagesacquired at higher zoom levels with almost no loss of data.However, the required computer memory necessary to hold thecube faces also grows fast.

VI. EXPERIMENTS AND RESULTS

In this chapter, following the methods for PTZ cameracalibration described in section III, we present the results ofrunning algorithms based on the described methologies forcamera calibration in a real scenario setup at Computer andRobot Vision Lab, Técnico Lisboa. During all the experiments

described in this chapter, the PTZ camera used was AXIS 5512PTZ Camera.

A. Camera Calibration based on a Standard Toolbox

The first results are based on the calibration method whichuses Jean-Yves Bouguet Camera Calibration Toolbox forMATLAB, described in section III-C.

The toolbox requires a prior acquisition of a set of imageswith a calibration pattern. Computing the calibration valuesfor each zoom level with this method is very time consumingso most of the values are interpolated from the acquired data.

Figure 7 shows the resulting curves for the principal pointand radial distortion parameters. The focal length calibrationparameter results are shown later in figure 8

(a)

(b)

Figure 7: Principal point calibration parameter, cc, evolution(a) and radial distortion calibration parameter, kc, evolution (b)for Axis 5512 PTZ Camera zoom range with the respectiveerror per point.

Focal length, which represents the distance between thecenter of the lens and its focus, as expected, increases whenthe zoom level also increases.

The principal point represents the intersection of the opticalaxis and the image plane. The point is very often the centerof the image plane or it is close to it. In the values obtainedthis behavior is not observed.

As seen in equation 4 the higher the values of radialdistortion, the bigger is the difference between the expected

projection point and the real and observed projected point. Asthe zoom increases, in this experience the radial distortionincreases significantly which will lead to a big differencebetween the undistorted point and the distorted point observed.The order of magnitude of the values does not match sincethe expected difference from the undistorted point to thedistorted point is around 1-2 pixels and with a radial distortioncoefficient of more than 0.5 it is not possible to reach thatdifference.

Sinha et. al [15] presented the results of a successfullycalibration done for two distinct PTZ cameras which highlightsthe unexpected values obtained in the calibration done withJean-Yves Bouguet Camera Calibration Toolbox. Although thefocal length follows the same behaviour as the calibration donein [15] both the principal point and the radial distortion haveunexpected behaviors for higher zoom levels.

Some reasons as the 3D noise from the calibration patterndue to curling of the pattern or 2D noise resultant of imprecisecorners detection, low light or image compression can bethe culprit of the not expected values. On the other side, itis possible that the Jean-Yves Bouguet Camera CalibrationToolbox could not be suitable for calibration of high zoomlevels as the toolbox requires distinguishable poses of thecalibration pattern and, at high zoom levels, it struggles tounderstand the difference between poses.

B. Focal Length Single Parameter Calibration

To measure different focal length values the bar is set atdifferent distances from the camera. The measures start at adistance, in the ground plane, from the camera to the bar, of 2meters to 15,5 meters, the physically possible distance in thelaboratory. The results of the focal length parameter can befound in figure 8.

As expected, the focal length increases as the zoom levelincreases. It is worth nothing that moving the bar around 11 to13 meters from the camera produced a low zoom level changepossibly due to the digital zoom of the camera which allowsthe camera to crop an image in a centered area with the sameaspect ratio as the original image, keeping the same zoomlevel.

C. Calibration based on a Grid on the Ground Plane

The results for the focal length parameter obtained aftersolving an optimization problem based on Equation 7 to finda solution for our unknown, the intrinsics matrix can be foundin figure 8 along with previous experiences results.

The results obtained with this methodology confirm thesuspicion raised in section VI-B regarding the calibrationvalues obtained with the Yves Bouguet Camera CalibrationToolbox for high zoom levels. With the two experiencesdetailed in sections VI-B and VI-C we prove that for highzoom levels the toolbox does not provide a good estimationof the intrinsics parameters of AXIS 5512 PTZ camera.

D. Radial Distortion Estimation with Line Fitting

For the figure 2 presented in previous chapter, after theoptimization problem we obtain figure 9 which illustrates the

0 20 40 60 80 100

Zoom level (%)

0

0.5

1

1.5

2

2.5

3

3.5

4

Foca

l L

ength

(m

)

104

MATLAB Tolbox Calibration (interpolated)

MATLAB Toolbox Calibration

Figure 8: Focal length parameter value evolution with differentcalibration methods for Axis 5512 PTZ camera zoom range.

initial pattern grid (inner grid) and the resulting grid (outergrid) after line fitting.

100 200 300 400 500 600

Horizontal pixels

100

150

200

250

300

350

400

450

500

550

Ver

tica

l pix

els

Figure 9: Initial pattern grid (inner grid) and the resulting grid(outer grid) after line fitting for maximum zoom out.

For this specific example we obtain the results displayed intable I.

Table I: Measures obtained after solving the line fitting opti-mization problem.

Zoomlevel(%)

Radialdistortioncoefficient

(k1)

Radialdistortioncoefficient

(k2)

Horizontalprincipal

point(pixels)

Verticalprincipal

point(pixels)

Focallength

(meters)

0 0,4625 -0,6165 352,0015 287,9988 408,67205 -0,0655 0,0123 352,0544 287,8751 409,8461

The values obtained are within the expected order of mag-nitude and as expected, the values decrease as the zoom valuesincrease.

VII. OMNIDIRECTIONAL BACKGROUND REPRESENTATION

As described in section IV-C, we chose a cube as thesolution to represent a scene in an omnidirectional way. Forthe purpose of testing this solution for different zoom levels

(a) (b)

Figure 10: Cube mosaic and calibration for a zoom level of40% considering the radial distortion effects (left) and nottaking into account the radial distortion (right).

and the effects of a good calibration, this section introducestwo experiments.

In the first experiment, Diogo Leite [9] used a single datasetwith two different calibration parameters. These experienceswere performed to assess the importance of having a calibra-tion which takes into account radial distortion described inIII-B for lower zoom levels. When radial distortion was notmodeled the results are noticeable different comparing to theresults including radial distortion. The visual appearance ofthe mosaic improved clearly.

In the second experiment we use all the knowledge acquiredwith previous experiments with camera calibration for differ-ent zoom levels of a PTZ camera, detailed in section VI-A,VI-B and VI-C, to build a cube mosaic with higher resolutionusing images acquired with a higher zoom levels.

Examples of the cube mosaics using a calibration basedon Jean-Yves Bouguet Camera Calibration Toolbox for azoom level of 40% allows testing this calibration solution andreiterate the conclusion of previous section which states thatthe calibration toolbox is not suitable for higher zoom levels.

Figure 10 (a) illustrates a section of cube mosaic andcalibration for a zoom level of 40% without considering theradial distortion effects while figure 10 (b) illustrates thesame conditions of zoom but considering the radial distortioneffects. The differences are barely visible due either to a badcalibration or because the radial distortion effects start to benegligible when having higher zoom values as proved by Sinhaet. al [15] and in this work with the results obtained in sectionVI-D and table I.

VIII. BACKGROUND SUBTRACTION WITHPAN-TILT-ZOOM CAMERAS

To illustrate that background subtraction methods can beapplied to the mosaics of 40% zoom level obtained, figure11 presents two mosaics obtained after applying the manualcalibration methods based on manual measures to build thecube representation of a scene where a USB drive is placedon top of table and later disappears. Figure 11 also showsthe result of applying Basic Background Subtraction to bothmosaics. Although the object we want to detect is very small

and the mosaics are not exactly stitched together, it is possibleto detect the object being removed from one instant to anothersince we are leveraging the increased resolution due to thehigher zoom of the camera.

Mosaic without object Mosaic with object Detected pixels

Figure 11: Results of a background subtraction algorithm onmosaics using datasets 181002t4 (no object) and 181002t5(with the object) acquired with 40% of zoom.

IX. CONCLUSION AND FUTURE WORK

The work described in this report explored three main sub-jects applied to surveillance systems based on PTZ cameras:background subtraction methodologies, camera calibration andomnidirectional background representations. The first subjectdescribed the basics of background subtraction and identifiedthe problems of extending to moving (rotating and zooming)cameras, with the same performance, some of the methodolo-gies already developed for static cameras.

The second subject was tackled since a background rep-resentation could not be done without the camera calibrated.In our work, due to the lack of textured regions in our testenvironment, automatic calibration methods described werenot suitable in several regions. To perform the calibration ofthe AXIS 5512 PTZ camera available at the test environmentin function of the zoom level, several alternatives based oncalibration patterns and real measures at the environment werestudied.

Jean-Yves Bouguet Camera Calibration Toolbox showed tobe a good alternative for low zoom values but providing incor-rect values for high zoom levels possibly due to the 3D noisefrom the calibration pattern, 2D noise resultant of imprecisecorners detection, low light or image compression. For higherzoom levels the calibration of the focal length parameter basedin real environment measures and the calibration based on acalibration pattern on the floor proved to be a good but sloweralternative to automatic methods. The results obtained followthe same behavior as previous experiments developed by Sinhaet al. [15].

As referred, the definition of an omnidirectional backgroundmodel is a major component in this project. The work of Sinhaet al. [16] is seminal on the aspect that pan-tilt-zooms cameraswere used to construct an high resolution scene representation.

We were able to create a simple framework for multi-resolution omnidirectional representations of the backgroundwithout taking into consideration the memory size problemsbut using the calibration methods supported by manual mea-sures of the scene to project the images into a cubic represen-tation of the environment. With those representations it was

proved to be possible to run a simple background subtractionalgorithm.

Future work, namely the development of the dissertationwork, is focused on extending multi-resolution omnidirectionalrepresentations of the background to events detection algo-rithms. This work shows that is possible to build backgroundrepresentations even with the lack of textured regions. Asolution to solve the memory size problems resulting fromsuch representations is a topic which needs more time andresearch investment.

REFERENCES

[1] Jean-yves bouguet camera calibration toolbox for matlab. http://www.vision.caltech.edu/bouguetj/calib_doc/. Accessed: 2018-07-10.

[2] Lourdes Agapito, E. Hayman, and I. Reid. Self-calibration of rotatingand zooming cameras. International Journal of Computer Vision,45(2):107–127, Nov 2001.

[3] Tiago Castanheira and José Gaspar. Multitasking of smart cameras.Master’s thesis, Universidade de Lisboa, Nov 2013.

[4] J. W. Davis. Hierarchical motion history images for recognizing humanmotion. In Proceedings IEEE Workshop on Detection and Recognitionof Events in Video, pages 39–46, 2001.

[5] J. W. Davis and Aaron F. Bobick. The representation and recognitionof human movement using temporal templates. In Proceedings of the1997 Conference on Computer Vision and Pattern Recognition (CVPR’97). IEEE Computer Society, 1997.

[6] Ricardo Galego, Alexandre Bernardino, and José Gaspar. Vignettingcorrection for pan-tilt surveillance cameras. In VISAPP 2011 - Proceed-ings of the International Conference on Computer Vision Theory andApplication, pages 638–644, 01 2011.

[7] Ricardo Galego, Alexandre Bernardino, and José Gaspar. Auto-Calibration of Pan-Tilt Cameras Including Radial Distortion and Zoom,pages 169–178. Springer Berlin Heidelberg, 2012.

[8] Suha Kwak, Taegyu Lim, Woonhyun Nam, Bohyung Han, and Joon HeeHan. Generalized background subtraction based on hybrid inferenceby belief propagation and bayesian filtering. In 2011 InternationalConference on Computer Vision, pages 2174–2181, Nov 2011.

[9] Diogo Leite and José Gaspar. Target tracking with pan-tilt-zoomcameras. Master’s thesis, Universidade de Lisboa, Nov 2011.

[10] Tiago Marques and José Gaspar. Cooperating smart cameras. Master’sthesis, Universidade de Lisboa, Nov 2015.

[11] J. C. Nascimento and J. S. Marques. Performance evaluation of objectdetection algorithms for video surveillance. IEEE Transactions onMultimedia, 8(4):761–774, Aug 2006.

[12] J. C. Neves and H. Proença. Dynamic camera scheduling for visualsurveillance in crowded scenes using markov random fields. In 201512th IEEE International Conference on Advanced Video and SignalBased Surveillance (AVSS), pages 1–6, Aug 2015.

[13] Y. Sheikh, O. Javed, and T. Kanade. Background subtraction for freelymoving cameras. In 2009 IEEE 12th International Conference onComputer Vision, pages 1219–1225, Sept 2009.

[14] Pedro Silva and José Gaspar. Vision based multi-target tracking.Master’s thesis, Universidade de Lisboa, Nov 2014.

[15] Sudipta N. Sinha and Marc Pollefeys. Pan-tilt-zoom camera calibrationand high-resolution mosaic generation. Comput. Vis. Image Underst.,103(3):170–183, Sept 2006.

[16] Sudipta N. Sinha, Marc Pollefeys, and Seon Joo Kim. High-resolutionmultiscale panoramic mosaics from pan-tilt-zoom cameras. In ICVGIP,2004.

[17] Ola Svenonius and Fredrika Björklund. Video Surveillance and SocialControl in a Comparative Perspective. 01 2013.

[18] Diogo Vicente and José Gaspar. Event detection with pan-tilt cameras.Master’s thesis, Universidade de Lisboa, Nov 2009.

[19] H. Zhang and K. Wu. A vehicle detection algorithm based on three-frame differencing and background subtraction. In 2012 Fifth Interna-tional Symposium on Computational Intelligence and Design, volume 1,pages 148–151, Oct 2012.