SWITCHER:A stereo algorithm for Ground Plane Obstacle Detection

Y.Zheng', D.G.Jones*, S.A.Billings\ J.E.W.Mayhew*, J.P.Frisby*

•Artificial Intelligence Vision Research Unit, University of Sheffield.'Department of Control Engineering, University of Sheffield.

Sheffield, S10 2TN.

A stereo algorithm for detecting and locating obstaclesraised above a ground plane is described. It is based onapplying a simple multi-model approach using hypothesistests to the grey levels of pre-processed stereo images.The algorithm has been successfully applied to realimages captured from a moving vehicle in order torecover putative obstacle locations.

1. INTRODUCTIONWe describe an algorithm, called SWITCHER, for analys-ing stereo images for the purposes of vehicle obstacledetection. The algorithm has been developed as part of aproject1 whose overall aim is the demonstration of astereo vision system capable of warning the driver, humanor automaton, of obstacles in the path of the vehicle thatconstitute a potential hazard given the current vehicle tra-jectory. It is assumed that the terrain over which the vehi-cle is travelling is a reasonably even surface, such as apath, road or factory floor.

In brief, SWITCHER finds disparity matches as follows:

(a) Each stereo image is first subjected to a stage of lowlevel image pre-processing which includes: imagerectification to parallel camera geometry without aliasing,high pass filtering, and grey scale equalisation.

(b) A window of grey levels from the left image is incre-mentally moved in 1 pixel shifts across the right image,subtracted from the right image at each location, and thevariance of the differences so obtained is calculated foreach disparity shift. Shifts are restricted to the rangeexpected for obstacles protruding from the ground plane.

(c) A statistical test (the F-test) is used to decide whichdisparity shift gives the smallest expected variance in thedifferences produced by subtraction. This test is appliedsequentially to points along rasters and a decision is madeto switch from the disparity selected for the previouslyconsidered window location only if there is evidence of asignificantly lower variance of the difference at a newdisparity value: hence the name SWITCHER for this algo-rithm.

It is demonstrated using natural images collected fromvehicle-mounted cameras that SWITCHER is capable of

1 We gratefully acknowledge the support of Mr A CSleigh of RSRE Malvern, who has arranged the funding forthis project. RSRE also provided the vehicle used underthe direction of Mr J Sherlock, and we thank Mr G Ed-wards for his part in assisting with image capture.

delivering a disparity map from which the scene coordi-

nates of a potential obstacle can be derived.

2. SPECIFICATION OF THE PROBLEMThe problem of detecting and locating obstacles raisedabove a ground plane by stereo vision may be studiedwithin the following framework.

Defining any two matching image rasters

l(x) = six) + nix)

r(x) = s£x) + nr(x) (1)

where x is the image coordinate of corresponding pointson the ground plane, l(x) and r(x) are intensities recordedby the left and the right cameras respectively, and nix)and nr(x) are noise components presumed to arise fromcamera noise, image capture, image pre-processing, etc.When no obstacle is present, and taking s to be related tothe photometric properties of points on scene surfaces, wemake the following assumption

= s£x) (2)

That is, for the purposes of analysis, we assume that thepre-processing stages are sufficient to eliminate any prob-lems arising from the stereo projections failing to possessphotometric invariance. Experiments to date show thatthis assumption is reasonable. Indeed we have obtainedgood results for several images sequences without imagepre-processing other than rectification.

When an obstacle is present whose surface plane has auniform disparity d, then

six) = sAx-d) (3)

and the problem will be specified as: find d and determinethe domain of x upon which eq.(3) holds.

3. SOME NOTATION AND ASSUMPTIONSInitially, to simplify the description, assume that only oneobstacle is present along a raster. The extension of thealgorithm to several obstacles can easily be made. Also,assume the disparities of the obstacle and the groundplane are each uniform along a raster (this assumption isnot restrictive given a reasonably small window size).The study will be carried out on a raster basis, i.e., theimage signals l(x) and r(x) are both one-dimensional. Theextension to two-dimensions is straightforward.

91AVC 1989 doi:10.5244/C.3.16

Fig.l shows the simplest case in which only one obstaclewith uniform disparity d across the raster is present. A isthe width of the obstacle along the raster.

r(x)

M

Fig. 1For ease of reference in the following analysis, severaldefinitions are introduced. Let Pb(x) and P0(x) be thebackground and the obstacle image signals of the left ras-ter respectively. Note that both of them are functions ofthe pixel position x. Let GP, OC and OB be defined asthree regions of x with the following properties

GP = {xl six) = Pb(x) and sr(x) = Pb(x)} (4)

OC = {xl six) = Pb(x) and sr(x) = P0(x+d)} (5)

OB = {xl six) = P0(x) and s&c-d) = P0(x)} (6)

with d > 0.

Clearly GP represents the region of the ground planewhich can be seen by both cameras; in Fig.l this is theregion where 1 < x < a and a+d+A+l < x < M. RegionOC represents the occlusion part that can be seen by theleft camera but not the right; in Fig.l OC is the regionwhere a+l < x < a+d. The obstacle which can be seen byboth cameras but with a forward phase shift of the rightsignal relative to that of the left is located in region OB;this region is given by a+d+l <x< a+d+A in Fig.l.

Now define various differences as follows

eo(x) = - r(x),

- r{x-\),

where dm is the maximum expected disparity of an obsta-lcle, and the set e0 to

formulation.can be regarded as a multi-model

4. OBSTACLE DETECTIONIt is obvious from the above definitions that when no obs-tacle is present, i.e. for all xe GP,

eo(x) = nix) - n&c),

ei(x) = six) - sjx-l) + nix) - nr(x-l),

ejjx) = six) - s^x-dj + nix) - nfc-dj

Assume the noise signals nix) and n£x) are normally andindependently distributed with zero mean and variance a2,and assume they are uncorrelated with s, and j ^ . Then thezeroth difference eo(x) is also normally distributed with

zero mean and variance 2CTJ;. However, the variances ofthe rest of the differences ex(x), . . . ,ed (x) are allexpected to be higher than 2CTJ unless the ground planehas no texture. Similarly, for all xeOB, the differencesare given by

eo(x) = six) - sr(x) + nix) - nr(x),

= nix) ~ nr{x-d),

+ nix) -

and e/x) becomes normally distributed with zero meanand variance 2a2

n. The variances of the rest of thedifferences are expected to be higher than 2a2

n unless theobstacle surface has no texture. The above observationsuggests the use of the variances of the differences as thekey to locate an obstacle and to determine its disparity.A simple method to detect the position of an obstaclewould be to calculate the zeroth difference eQ(x) only andthen set up a threshold such that if in some domain Q,eo(x) is above the threshold for almost all xe Q, an obsta-cle may be present in Q. This is the basic idea used byMallot et al [1] in their obstacle detection method. It isquick to implement, but does not provide information onthe disparity of the obstacle. Also, problems arise in set-ting a threshold suitable for the particular image texturespresented by the scene under consideration. It was thislatter difficulty that led us to design SWITCHER, whichavoids the problem of setting thresholds by using a statist-ical test. We are however continuing to investigate Mal-lot et al's approach, not least because of its simplicity andlow computational cost [4].

5. THE NECESSITY FOR A STATISTICALTESTThe method introduced here can recover disparity infor-mation. The idea is to set up a window of size W ondifferences <?,(*)> i=0,l,...,dm, estimate the variance of thedifferences in the window and then make comparisons.The window size W is taken as an odd number for con-venience, e.g., the k"1 window on the zeroth difference isthe set

{ eo(x), x = k-^-,...Jc,...M~- }

Let e,<x) be the i'h difference after the mean has beenremoved. The variance of this difference in the k'h win-dow can then be estimated by

whereW-\

•4m

(7)

As noted above, provided all xeGP in the k'h window,a^k,W) has the minimum expected value 2a\. Similarly,provided all xeOB in the k'k window, o2JJc,W) has theminimum expected value 2a\. Thus the exercise of locat-

92

ing an obstacle is reduced to finding the minimumexpected value of df(k,W), i=0,l dm at every window.The disparity d at the k'h pixel of the left image will thenbe given by

E[ min E[ df(k,W)i

(8)

where E[z] denotes the expected value of the random vari-able z. However, when some x in the k* window lies inthe region OC, or some xeOB and some xeGP, the k'h

window enters the occlusion parts and all variances of thedifferences are expected to be greater than the minimum

The basic problem now is to solve eq.(8). Suppose in the(k-l)th window, d=0; in the k'h window, it is found that6^ < do for some i=p*O. A decision has to be made hereon whether d* < do by chance or &l actually comes froma population with expectation of mean less than that ofdo- The above situation clearly calls for a statistical test.If, after the statistical test, it is found that dp is notsignificantly smaller than do, then the disparity remains atzero. Otherwise the decision d=p can be made. The sta-tistical test used is the F-test where, in the above example,

if certain conditions are met by l{x) and r(x).

6. APPLICATION OF THE F-TEST TOOBSTACLE DETECTIONConsider the case when xe GP. (Similar arguments hold inthe case xeOB.) Then,

eo(x) = nix) - nXx)

ei(x) = s,(x) - sXx-i) + nix) - njjc-i)

where

To use the F-table to compare do and dp for some i=p*O,it is necessary that both e0 and ep are from two indepen-dent populations with normal distributions. The abovetwo equations show that this requirement cannot besatisfied completely. To see this, assume both st and srare normally distributed with the same variance c£ (Onthe whole, this assumption will be valid.) Then the twodifferences e0 and ep are correlated due to the noise signaln;, and their correlation is

2* + R(p)/2

where

R(p) = E[

s, and sr are the means of si and sr respectively. Thus toreduce their correlation, the image must have asufficiently large signal-to-noise ratio2

S _ <*M

N a .

The second point to note in applying the F-test is that theF-table is produced on the basis of two populations havingthe same variance. Hence in deciding if dp1 issignificantly smaller than do, the hypothesis to be tested isthat both of them have the same expectation. When thishypothesis is accepted, it seems that the disparity could beeither 0 or p . At this point, the decision made in the{k-\)<h window will influence the present decision in thek window, the argument being that a decision to changedisparity must be supported by adequate evidence.Although such a criterion makes the change of disparity adamped process, it reduces noise in the disparity estima-tion. Indeed, in order to avoid too hasty switching to anew disparity we have explored switching only if threesuccessive F-tests are all significant. That procedure canbe effective in avoiding false alarms but it is not alwaysnecessary and its potential is the subject of current experi-mentation [4].

The third point is that the decision on whether dp1 issignificantly smaller than do is a one-sided test on the F-distribution.Thus, suppose in the (Jfc-1)* window it was found that d=q(i.e. q is the presently chosen disparity value). Using Hfor hypothesis and denoting the level of significance bya, the steps taken to test the hypothesis of d=q in the k""window, which together define the SWITCHER algo-rithm, are as follows.

1. H: d=q.2. Choose a.3. Calculate 6f(k, W), i = 0,l,...,dm.

4. Find p such that dp1 = min 6? .

5. If p = q, accept H.

6. Hp*q, findF = - | .

7. Reject H if F > F^QV-hW-l) and set d=p.Noise on the disparity estimation will be introduced forthe occlusion parts as none of the differences will havethe minimum expected variance. The occlusion parts aretransition periods between different disparities. Providedthe image has sufficiently large signal-to-noise ratio andprovided the window size is properly chosen, false esti-mates of disparity for the occlusion parts should not resultin false disparity estimates when xe GP or xe OB.

6.1. Two-dimensional windowsInstead of working on a single raster, the above study caneasily be extended to include several rasters provided anobstacle present on these rasters has a reasonably uniformdisparity from one raster to its neighbouring rasters. Infact, in the demonstrations which follow matching win-dows straddling three rasters are used.

We have examined both theoretically and experimentallythe effects of varying the window size (Zheng et al, in

2 Experiments using artificial random dot textures haveshown that S/N ratios at least as low as 5:1 are satisfactory,and experience with natural images suggests that evenlower values can be tolerated.

93

preparation). Our main conclusion is that the smallestobstacle that can be reliably detected is about half thewidth of the window but that vulnerability to noiseincreases as window size is reduced. The choice of 17 x 3provides a good compromise for the image sequencesinvestigated to date.

7. IMPROVEMENTS TO PERFORMANCE

7.1. The Spatial Coherence ConstraintSome of the road scenes we have investigated have verylow S/N ratios caused by poor dynamic range in the TVcamera response when dealing with low intensity regionswithin otherwise brightly lit scenes. We have found thatthe problem of the false alarms triggered in such regionscan be met in large part by employing a spatial coher-ence constraint. The idea here is that certain 'noisespikes' in the disparity map could not arise from the scenewithout having given rise to above-ground plane dispari-ties recorded 'behind' them unless the cameras happenedto be looking at a thin lamina so oriented that it is seen'edge on'. It is safe to assume that such circumstances areunlikely. Hence the absence of supporting disparitiesabove the ground plane gives a firm basis for excludingthese noise spikes. We have found this constraint a high-ly effective noise cleaner.

7.2. Exploiting the ground plane disparityWe have found it advantageous both in speed and in noiseelimination to limit the range of disparties that issearched to about + 40 pixels. This range is ample tocapture obstacles in the images explored to date (but seeremarks in section 10 on exploiting temporal coherencebetween successive stereo pairs).

8. EXPERIMENTS USING SWITCHER ONNATURAL IMAGESSWITCHER has been evaluated off-line using a selectionof 256 x 256 stereo image pairs recorded on video fromcameras mounted 52 cm apart and 67 cm above theground on the RSRE experimental vehicle.

8.1. Image Pre-processingEach image pair was first subjected to a series of geometr-ical and grey level transformations. The aim of this imagepre-processing was to recover the underlying texturewithin each raster in a form most suited to the simplegrey level subtraction operation upon which SWITCHERis founded.

First, a point transform rectification with anti-aliasing bylocal neighbourhood Gaussian smoothing to sub-pixel acu-ity was applied to the images to achieve zero verticaldisparity between corresponding left and right rasters.Knowledge of the camera geometry required for this stagewas provided in the present experiments by an implemen-tation of Tsai's algorithm [3] in AIVRU's TINATOOLimage processing environment [5].

Second, local variations in grey levels caused by pho-tometric variance between images, low frequency drifts,

etc were then reduced to acceptable levels by high-passfiltering with a low cut-off frequency of 2 cycles/deg.

Finally, a two band log grey scale equalisation filter, witheach band symmetrically placed around the mean greylevel, was applied to enhance homogeneity of texturalinformation in the signals by emphasising their low ampli-tude portions.

The last two stages are not always necessary forSWITCHER to perform adequately. Experiments are inprogress to investigate when they are required [4].

8.2. Variance Residue Measurements &Disparity Selectionvs 11 A 1 7 x 3 window from the left image was incre-

mentally moved in 1 pixel steps across the right image,

subtracted from the right image at each location, and the

variance calculated for the differences so obtained. A

selected disparity value for each location was obtained by

comparing variances using the F-test described above with

a 5% significance level throughout.

9. RESULTSThe results from applying SWITCHER to two samplestereo pairs are shown in figures 2 and 3. The rectifiedstereo images are shown first (a - right image, b - leftimage, as required for crossed-eye fusion), followed bythe results of high pass filtering and grey scale equalisa-tion of the left image, (c). The disparity map returned bySWITCHER is shown in (d), and (e) illustrates the resultof imposing on that map the constraint of spatial coher-ence.

9.1. The Post Image PairIn figure 2e the disparity map returned for a region wherethe ground plane has no obstacle is clearly distinguishablefrom a region where the post-like obstacle is present. Thesmooth slanted obstacle disparity surface is bounded oneach side by noisy regions caused by occlusions. It hasproved a simple matter to estimate the x,y,z scene coordi-nates of the obstacle from this map.

Comparison of the equation for the recovered groundplane with that provided from the same images by a fastHough Transform disparity mapping technique [2], showsthat SWITCHER provides accurate disparity information.

9.2. The Kerb Image PairThe right hand portion of the disparity map (3e) derivesfrom the ground plane and is again reasonably flat,whereas the left hand portion from the kerb region isnoisy. The latter is probably due to differences betweenthe image projections of the high frequency grassy tex-ture above the kerb, and a choice of window size unsuitedto the rapid changes in disparity in the grassy area. Evenso, SWITCHER'S output is quite sufficient for the pur-poses of obstacle detection assuming a knowledge of theground plane.

94

Fig 2 (a) Fig 2 (b)

Fig 2 (c) Fig 2 (d)

Fig 2 (e)

10. CONCLUSIONSThe SWITCHER algorithm has been shown to be capableof delivering information regarding obstacles from realstereo images captured from a moving vehicle. In experi-ments to date, in no case is an obstacle missed and prob-lems posed by false alarms appear slight [4]. Work is inhand to investigate versions of SWITCHER whichpreserve these attributes while reducing as far as possiblethe computational cost.

Information about a putative obstacle, delivered as arough estimate of the x,y,z scene coordinates of the obsta-cle, could be used to control a region of interest process-ing strategy aimed at checking in more detail the imagezone where the obstacle is recorded. Work is planned touse SWITCHER in this way within a control architecturethat integrates information about the scene from a continu-ous stream of stereo image pairs collected on a movingvehicle, to provide advance warning about obstacles likelyto cause a collision given the current vehicle trajectory.

95

\r

Fig 3 (a) Fig 3 (b)

Fig 3 (c)

Fig 3 (e)

Fig 3 (d)

11. REFERENCES1. H. A. Mallot, E. Schulze and K. Storjohann, 'Neural

Network Strategies for Robot Navigation', in G. Dreyfusand L. Personnaz (eds.) Proc. nEuro '88 (Paris, June1988).

2. P. McLauchlan and J. E. W. Mayhew, 'Needles: AStereo Algorithm for Texture', Workshop on ImageUnderstanding and Machine Vision, Cape Cod Mas-sachusetts June 1989.

3. R. Y. Tsai, 'An efficient and accurate camera calibrationtechnique for 3D machine vision', Proc IEEE CVPR 861986, pp 364-374.

4. D. G. Jones, Y. Zheng, S. A. Billings, J. E. W. Mayhewand J. P. Frisby, 'Experiments on stereo algorithms forground plane obstacle detection', In preparation.

5. J. Porrill, S. B. Pollard, T. P. Pridmore, J. B. Bowen,J. E. W. Mayhew and J. P. Frisby, 'TINA: TheSheffield Vision System', Proceedings of the Ninth Inter-national Joint Conference on Artificial Intelligence, Milan1987, pp 1138-1144.

96