CENTER FOR

MACHINE PERCEPTION

CZECH TECHNICAL

UNIVERSITY IN PRAGUE

BACHELORTHESIS

ISSN

1213-2365

Road-plane Estimation from

Video-sequences

Yurii Bizinskyi

CTU�CMP�2013�14

May 24, 2013

Available atftp://cmp.felk.cvut.cz/pub/cmp/articles/matousek/Bizinskyi-TR-2013-14.pdf

Thesis Advisor: Ing. Martin Matou²ek, Ph.D.

Research Reports of CMP, Czech Technical University in Prague, No. 14, 2013

Published by

Center for Machine Perception, Department of CyberneticsFaculty of Electrical Engineering, Czech Technical University

Technická 2, 166 27 Prague 6, Czech Republicfax +420 2 2435 7385, phone +420 2 2435 7637, www: http://cmp.felk.cvut.cz

Czech Technical University in Prague

Faculty of Electrical Engineering

Department of Computer Science and Engineering

Bachelor's Project

Road-plane Estimation from Video-sequences

Yurii Bizinskyi

Supervisor: Ing. Martin Matou²ek, Ph.D.

Study Programme: Software technologies and management, Bachelor

Field of Study: Software Engineering

May 24, 2013

Aknowledgements

I'd like to express sincere thanks to my supervisor Ing. Martin Matou²ek, Ph.D., for hisvaluable comments, support and for the time he devoted to our consultations. Many thanksto Ing. Ond°ej Sychrovský for his valuable advises on VHDL language and ADTF framework.

v

vi

Declaration

I hereby declare that I have completed this thesis independently and that I have listed allthe literature and publications used.I have no objection to usage of this work in compliance with the act 60 Zákon £. 121/2000Sb.(copyright law), and with the rights connected with the copyright act including the changesin the act.

In Dolní B°eºany on May 24, 2013 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii

viii

Abstract

This work solves a problem of real-time estimation of the horizon line in a video-sequencefrom a camera placed in a moving car. The horizon line is retrieved from decomposition ofhomography between neighbouring frames of the sequence. The homography is estimatedusing RANSAC applied to a set of correspondences. This set is calculated by running theKL tracker (implemented on FPGA chip) on a set of areas of interest taken from the part ofthe frame which contains road. Estimated horizon line can be used in many video-sequenceanalysis tasks.

Abstrakt

Tato práce °e²í problém odhadu linie horizontu v realném £ase z video-sekvence pochaze-jící z kamery umist¥né v pohybujícím se aut¥. Linie horizontu je vypo£tená dekompozicihomogra�e mezi sousednimí snímky. Homogra�e je spo£tená pomocí metody RANSACaplikované na mnoºin· korespondenci. Tato mnoºina je vysledkém b¥hu KL trackeru (im-plementovaného na FPGA £ip·) který zpracovává mnoºin· oblasti zájmu získanou z £ástísnímk· obsahujicí vozovku. Linie horizontu m·ºe být pouºitá v °ád¥ uloh zabývajících seanalýzou video-sekvenci.

ix

x

Contents

1 Introduction 1

1.1 Task formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Areas of interest tracking 5

2.1 Work area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Areas of interest and search areas . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 Shift prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 Tracking algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 FPGA implementation of areas of interest tracking 11

3.1 Restrictions and simpli�cations . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Robust homography estimation 15

5 Horizon estimation 17

6 Results 19

7 Conclusion 25

7.1 Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

A DVD content 29

xi

xii

Chapter 1

Introduction

In the last decade the number of road accidents in EU decreased by 26% [4] despite the10% increase in the number of vehicles per capita [9]. This tendency may be the result ofthe increasing number of vehicles equipped with some of the active safety systems like ABSor ESP which proven [1][2] to be e�ective. Those systems showed that road safety can begreatly improved and that is why the extensive research is being devoted to other activesafety systems.

One of the most evolving and challenging categories of active safety systems are vision-based systems, e.g. Collision Warning, Lane Departure Warning, Intelligent HeadlightsControl, Pedestrian Detection and others. A lot of those systems can be found in upper-class vehicles but, as ABS and ESP practice shows, to remarkably improve road safety suchsystems should appear in middle-class and lower-class vehicles.

To achieve this goal those system should be cost-e�cient. The best scene recognitionis achieved by using stereo-pair of images taken from two cameras but to make the systemcheaper the same information should be retrieved using a single camera. It naturally bringsproblem of lack of 3D position information about detected objects if they area moving.In this case some additional information should be provided. One such information is theknowledge of position of horizon line in the scene. Horizon line can be used as a geometricrestriction which helps to �lter detected objects. For example, in a vehicle detection systembased on wheels detection the knowledge of horizon line position restricts the detected wheelsbe below it.

1.1 Task formulation

This work solves a problem of the horizon line (Fig.1.1) estimation in a video-sequence.Video-sequence is taken from vehicle-mounted camera with 25 frames per second as a se-quence of gray-scale 738 × 435 frames. Frames are processed in previous-to-current pairs.Camera, used to capture video, has calibration matrix [7]

K =

975.45 0 350.450 978.38 215.220 0 1

. (1.1)

1

2 CHAPTER 1. INTRODUCTION

Figure 1.1: Horizon

The searched horizon line may not always match the visible horizon line, like in case of upor downhill movement or when there is no visible horizon at all (in dense urban conditions,tunnels, parking buildings etc). Frames preprocessing and processing of the results of featuretracker are implemented in C/C++ as plug-in to the ADTF [3] framework while the featuretracker is designed using VHDL and runs on FPGA to achieve required frame-rate.

The algorithm designed to solve the task runs in a loop over a sequence of frames anddoes following operations on every frame:

• Work area is cut from original frame and a set of areas of interest from work area isstored until the next frame arrives. If this is not the �rst frame then a set of searchareas is stored (Chapter 2).

• FPGA implemented feature tracker runs on every pair of respective area of interestand search area to �nd a correspondence (Chapter 3).

• Homography between work areas is estimated from the results of feature tracker usingrobust estimation by RANSAC (Chapter 4).

1.1. TASK FORMULATION 3

• Estimated homography is modi�ed and then decomposed to rotation, translation andnormal vector of the road plane. Horizon line is estimated from the estimated normalvector (Chapter 5).

4 CHAPTER 1. INTRODUCTION

Chapter 2

Areas of interest tracking

This chapter deals with estimation of correspondences between 2 frames. Tracking featuresin consequent frames of original video-sequence is used. The tracking is a di�cult taskbecause not only the position of tracked features changes but also their scale. To ease thistask the scale factor is removed.

2.1 Work area

To remove the scale factor every frame from camera is preprocessed, transformed into avertical view of the road, a view to the prede�ned area of the frame (Fig. 2.1(a), 2.1(b)).This area is further called work area. Transition between work areas of neighbour framesis mostly a horizontal motion. Additionally there is a small rotation, and vertical cameramovement contributing as a scale factor. Dimensions and position of work area are �xedaccording to camera type and position. Naturally, road isn't the only object in the workarea. Cars, walls and other objects could eventually get into the work area but becausethe estimation algorithm, described in Chapter 4, is robust the success rate is high enough.Transformation from original frame to work area is done using base homography that havebeen calibrated manually,

Hb =

8.42e− 4 1.32e− 3 −5.67e− 1−5.56e− 08 2.76e− 3 −8.24e− 1−5.56e− 09 3.62e− 06 −7.09e− 4

. (2.1)The result of applying this homography to original frames can be seen in Figures 2.1(c)and 2.1(d).

2.2 Areas of interest and search areas

Areas of interest are small parts (Fig.2.2(b)) of the Ak−1 work area with their centres lyingon the prede�ned grid (Fig.2.2(a)). The correspondences to these parts between Ak−1 andAk are searched. Size of the area of interest is chosen so that it is large enough to containwell recognizable texture. Width and hight could be di�erent but we choose it to have an

5

6 CHAPTER 2. AREAS OF INTEREST TRACKING

(a) Ik−1 (b) Ik

(c) Ak−1 (d) Ak

Figure 2.1: Work area. Previous (a) and current (b) frame work areas shown as the yellowpoligon, (c,d) work areas geometrically transformed by Hb.

odd number of pixels so the area of interest has natural centre. We have used two di�erentsizes of the areas, 21 and 37 pixels. We denote the set of pixel coordinates relative to anarea centre as w.

Search areas are parts of the Ak work area with their centres lying on the predictedcoordinates of respective areas of interest. Size of the search area is 20px wider and higherthan respective dimensions of the area of interest.

2.3 Shift prediction

To facilitate or in some cases to allow algorithm to work successfully a prediction of area ofinterest shift is calculated for every area. If there were no predictions it could easily happenthat area's of interest shift between neighbour frames is too large (bigger than its own sizefor small areas and high velocity) that there'll be no way to �nd correspondence. Also,

2.4. TRACKING ALGORITHM 7

(a) Set of areas of interest (b) Selected area of interest (the read areain a)

Figure 2.2: Areas of interest

good prediction increases speed of searching of the correspondences to the areas of interest.The set of predictions is further designated as P = {p1, p2, ..., pn}. This set is computed byapplying calculated homography Ha to the area's of interest centres grid. In case the videosequence starts from steady position there is no need of any initial prediction. Otherwise, theinitial prediction should be provided from some external source and because such externalsource isn't aware of the number of areas of interest, same prediction is used for all areas,P0 = {p1, p2, ..., pn}, pi = pj ,∀i, j = {1, 2, ..., n}.

2.4 Tracking algorithm

Areas of interest tracking is done using KL feature tracker [8]. It's overview is given below.This part of the horizon plane search algorithm is implemented using VHDL on FPGA.

1. Algorithm starts with two work areas Ak and Ak−1, a set of areas of interest S ={si}, si = (xi, yi), from area Ak−1 and a set of predictions P . The goal is to calculatethe set of correspondences C = {ci}, ci = (x′i, y′i), in work area Ak using S and P .

2. The gradient of whole Ak−1 is calculated (approximated by di�erence):

∇Ak−1(x, y) =[ ∂

∂xAk−1(x, y)∂∂yAk−1(x, y)

]=

[Ak−1(x+1,y)−Ak−1(x−1,y)

2Ak−1(x,y+1)−Ak−1(x,y−1)

2

]. (2.2)

3. For every area of interest si the process of correspondence ci lookup is started:

(a) Components of weight matrix are calculated as sum in the area of interest,

W =∑u∈w∇Ak−1(u+ si)

(∇Ak−1(u+ si)

)T=

[gxx gxygxy gyy

]. (2.3)

8 CHAPTER 2. AREAS OF INTEREST TRACKING

(b) The initial position of search area is ci = si + pi.

(c) Using linear interpolation the image patch of the same size as area of interest,centred at ci, is cut from work area Ak.

(d) Shift vector h is then calculated

h =∑u∈w

(∇Ak−1(u+ si)T [Ak−1(u+ si)−Ak(u+ ci)]

)W−1 (2.4)

(e) Correspondence ci for area of interest si is updated as ci ← ci + h.(f) If absolute value of calculated shift h is bigger than � = 0.0001 the calculation is

repeated from (c). In VHDL design the � is set to 0 due to precision.

4. The output of the algorithm is the set of correspondences C (Fig. 2.3(a) 2.3(c)).

(a) High success rate 21× 21 (b) High errors rate 21× 21

(c) High success rate 37× 37 (d) High errors rate 37× 37

Figure 2.3: Correspondences. Ak images are shown.

The sample results of the tracking algorithm are shown in Figure 2.3. Result is highlydependent on the road texture and depends less on the size of the area of interest. Featureless

2.4. TRACKING ALGORITHM 9

texture does not provide enough data to decide in which direction to move while searching forthe correspondence. So, it may happen that the search stops immediately on the predictedposition without moving anywhere or runs through textureless part of the image up to theimage boundary (Fig. 2.3(b)). Such mistakes are �ltered by RANSAC described in Chapter 4.

10 CHAPTER 2. AREAS OF INTEREST TRACKING

Chapter 3

FPGA implementation of areas of

interest tracking

Tracking algorithm described in the previous chapter is implemented on the FPGA chip usingVHDL. Considering full independence of the areas of interest is was chosen to implementparallel design. The full parallelism could not be achieved because data are passed to chip insequential manner. Processing of the input data is divided into three parts � receiving data,calculating gradients and weight components and searching for correspondences. The �rsttwo are done by the same block which does required computations per pixel immediatelyafter receiving a new pixel. The last part, search for correspondences, is not started untilthe complete set of input data for one area of interest has been received. So, if there is morethan one independent functional block present then processing of next set of data is startedimmediately after previous set had been passed to chip, as long as there are free processingblocks left, while previous blocks work on the correspondences tracking. It was decided thatdesign should work on the 125MHz frequency (the reference frequency of the PCIe interfaceused for FPGA connection), which e�ectively means having 8 ns per clock cycle.

3.1 Restrictions and simpli�cations

The main restriction of the implementation are limited resources of the FPGA chip. Originaltracking algorithm described in Sec. 2.4 required holding 3+1 full work areas at once. The�rst one was used as source of the areas of interest, second and third to hold calculatedgradients from the �rst one and the last one that is being searched for the areas of interestmatch. If areas of interest were cut and placed into separate storage then it increased thespace requirement which reduced the amount of processed areas of interest at the sametime. On the other hand, if areas of interest were accessed through o�set to original workareas then the complexity of synthesized design did not allow to achieve required frequency.That is why the decision was made to reduce the amount of work done by FPGA chip tosimplify the design. The task to hold both work areas and cut appropriate areas of interestwas taken out from the chip. In this case the cut data blocks (Fig. 3.2(a)) are passeddirectly to free functional block (described further). But the major simpli�cation was touse only integer values in all computations which, as tests showed, signi�cantly reduced the

11

12 CHAPTER 3. FPGA IMPLEMENTATION OF AREAS OF INTEREST TRACKING

quality of the result on areas with featureless texture. The main reason was the complexityof implementation of sub-pixel interpolation which also required the use of �oating pointarithmetics.

3.2 Implementation

The whole design consists of three main blocks: patcher, tracker and arbiter (Fig. 3.1).Patcher and tracker work always in pair and comprise one functional block. Patcher receivesand holds incoming data, computes gradients and weight matrix components. Tracker doescorrespondences search. And because there is more than one functional block present anarbiter is used to manage them. Arbiter manages all functional blocks, assigns new datato free blocks and gathers results and sends them to the output. The whole design issynchronized to the clock.

Figure 3.1: Functional blocks

3.2.1 Input/Output data

Block of data (Fig. 3.2(a)), for each area of interest, consists of an identi�er, area of interest(Section 2.2), search area (Section 2.2) and padding if needed. The need of padding dependson the size of the area of interest and search area and the fact that data are passed to FPGAin 4-byte chunks per clock cycle. So the whole block of data has to be padded to 4-byteboundary. Patcher requires data to be passed in per-byte manner so the 4-byte chunks areunpacked by the unpacker block to one byte per clock cycle. Unpacker blocks input whilepassing unpacked bytes. Output data (Fig. 3.2(b)), for each area of interest, consists of anidenti�er, x and y relative coordinates of the centre of the found correspondence.

3.2. IMPLEMENTATION 13

(a) Input data.

(b) Output data.

Figure 3.2: Structure of data passed to/from FPGA.

(a) Structure of patcher. (b) Registers shifting. Register marked green is usedin calculations.

Figure 3.3: Internal structure

3.2.2 Patcher

This block bu�ers and stores incoming data for the Tracker (Fig. 3.3(a)). It also computesgradients (2.2) of the area of interest for both dimensions and computes weight compo-nents (2.3). Gradients are computed in the same way as in (2.2) except that they arecomputed only for the area of interest instead of whole work area. Division by two is doneusing right bit shift. No other operations here require division. Calculation of weight com-ponents is pipelined. Storage is organised as an array of registers. To eliminate randomaccess to this array, we have used the FIFO style to �ll the registers (Fig. 3.3(b)). It allowsus to have every new pixel at the same last position of the array so all calculations are doneusing this one register. All registers are shifted one pixel further before inserting new pixel.This architecture signi�cantly reduces the complexity of this part of �nal design. When thewhole block of data was received and preprocessed the patcher signals tracker that it canstart searching for the correspondence.

14 CHAPTER 3. FPGA IMPLEMENTATION OF AREAS OF INTEREST TRACKING

3.2.3 Tracker

This block does actual searching of the correspondence. At the beginning the tracker startswith the centre of the area of interest matched to the centre of the search area. It meansthat coordinates of the correspondence calculated by tracker are relative to search area'scentre. Giving that the coordinates of the search area's centre depend on prediction whichis unknown to tracker the recalculation to absolute coordinates in work area is done outsideFPGA. Tracker uses weight matrix which has constant value and is determined by (2.3). Dueto the fact of usage integer values only we do not calculate its inverse immediately. That iswhy only determinant

d = gxxgyy − g2xy (3.1)

is calculated at the beginning and is used later at the end of every iteration when �nal shiftis calculated using slightly modi�ed (2.4)

h =

[ ∑x∑y

]TW−1 =

[ ∑x∑y

]T [gyy −gxy−gxy gxx

]1

d, (3.2)

where∑

x and∑

y are sums over x and y coordinates of the term in (2.4). The result ofdivision by d in (3.2) is rounded to the nearest integer and is used in the next iteration. Ifthe shift is not zero the centre of the area of interest is placed to the shifted position andcalculation is repeated, otherwise tracking is �nished. Multiplications done in (3.2) have 32-bit integer operands. DSP block on FPGA can multiply only operands up to 25x18 bits sothe built-in multiplication operator was expanded to sequential combination of several DSPs.Such design could not meet the timing requirements. That's why we have used the pipelinedMultiplier IP Core. The same reasoning was to use Divider IP Core for �nal division by d.

Arbiter

Arbiter keeps track of how many patcher-tracker pairs are busy and checks if there are anyresults to be passed to the output port. If there are new data on the input and there is afree patcher-tracker pair then the data are redirected to this pair and the pair is marked asbusy. If there are no free pairs then the arbiter blocks input and waits. If more than onetracker �nishes its job at the same time then the arbiter passes their results to the outputport in consequent clock cycles one at a time. After tracker had �nished its work and itsresult had been passed to the output port the corresponding patcher-tracker pair is markedas free.

Tracker implemented with described design on FPGA di�ers from MATLAB implemen-tation mainly by not using the �oating-point arithmetics. The comparison of both imple-mentations is shown in Chapter 6.

Chapter 4

Robust homography estimation

The set of correspondences is further being used to estimate homography that best matchesthem. Estimation should take into account a possible high error rate in the search of thecorrespondences (Fig. 2.3(b), 2.3(d)). The best possible result is if correspondences forsuch areas of interest are �ltered out and do not a�ect the �nal homography estimation.The Random sample consensus [6] algorithm is used for this task. Its adaptive version whichcalculates homography Ha between work areas using set of correspondences C is summarizedhere:

1. Set required number of iterations N and counter j,

N ←∞, j = 0.

2. While N > j

(a) Select a random sample of 4 correspondences and compute the homography Hafrom them.

(b) Check if Ha can be inverted by calculating the reciprocal condition number. If itis lesser than 1−12 then go to (a).

(c) Calculate the symmetric transfer error d⊥ for each area of interest si ∈ S and itscorrespondence ci ∈ C,

di⊥ = ||si −H−1a (ci)||2 + ||ci −Ha(si)||2.

(d) Compute the number of inliers n (Fig. 4.1) consistent with Ha as the number ofcorrespondences for which di⊥ < T

2, where threshold T = 2 px.

(e) If n is the new maximum then remember current set of inliers.

(f) Update required number of iterations

N ← log(1− p)log(1− (1− �)4)

,

where k is a number of areas of interest, � = 1−n/k, and probability p = 0.9999.

15

16 CHAPTER 4. ROBUST HOMOGRAPHY ESTIMATION

(g) j ← j + 1

3. Re-estimate Ha from all inliers.

The quality of the RANSAC's result depends on the quality of the result from trackingalgorithm. If tracker provides too inconsistent set of correspondences it may happen thatRANSAC estimate homography from incorrect correspondences marking the correct one asoutliers. This is the worst case scenario because it in�uences the prediction calculation forthe next frame. In Figures 4.1(a),4.1(c) we can see a lot of inliers and a few outliers whichare �ltered. Figures 4.1(b),4.1(d) show that too many mistakes highly in�uences the resultof RANSAC. The blue rectangles depict the correspondences which represent the real motionof the car because the car is turning left in this part of the the sample video-sequence.

(a) (b)

(c) (d)

Figure 4.1: Inliers (red) and outliers (yellow). (a, c) obvious mistakes are �ltered, (b, d) toomany mistakes. Blue rectangles depict correct correspondences that were wrongly �lteredout.

Chapter 5

Horizon estimation

Homography Ha between work areas, estimated using RANSAC from the set of correspon-dences, is used to calculate homography Hk between original images. To do this, the reversaltransformation using Hb is done including application of calibration matrix K:

Hk = K−1(H−1b HaHb)K. (5.1)

This homography is decomposed to rotation matrix and translation vector using algorithmdescribed in [5] as

Hk = dR+ tnT , (5.2)

where n is a normal vector of the plane along which the camera moves and in our case it isthe road plane.

Decomposition ends up with two possible results with only one being correct for currentimage and other being just its mirror. To select the correct one it is su�cient to check bothresults on the edges of the image. Only one of them �ts into it.

Applying transposed calibration matrix to the normal n gives the horizon line

L = K−Tn. (5.3)

which can be used in original image without any further modi�cations (Fig.1.1).

Depending on the results of the feature tracker and RANSAC the horizon may be esti-mated correctly (Fig. 5.1(a)), with minor error (Fig. 5.1(b)), major error (Fig. 5.1(c)) or notestimated at all (Fig. 5.1(d)).

17

18 CHAPTER 5. HORIZON ESTIMATION

(a) Good (b) Worse

(c) Bad (d) No horizon

Figure 5.1: Quality of the horizon estimation.

Chapter 6

Results

The algorithm was tested on the sample video-sequences from real environment. For everysequence the initial prediction was manually computed because none of them started fromsteady vehicle position. Tests of the algorithm on sample video-sequences in MATLAB showthat the algorithm is designed correctly and the only improvement to be done is the additionof the Kalman �lter. On the opposite, the quality of output from FPGA implementationwithout �oating-point computations is very low. Figures 6.1(a), 6.1(b), 6.1(c), 6.1(d) showthat the modelled feature tracker has high success rate even in di�cult conditions like fea-tureless road texture and road turns while the FPGA implemented feature tracker fails tocorrectly �nd correspondences. In Figure 6.1(a) both implementations have very similarresults, lines mostly overlap. This is the �rst frame of video-sequence so both of them useinitial prediction and so have same starting conditions. But in Figure 6.1(b), 20 framespast the beginning, only four correspondences match. Figure 6.1(c) shows that there are nomatches and that FPGA implementation had lost track of movement. Figure 6.1(d) showsthat while the tracker from matlab implementation recovers from road turn, the FPGAtracker tells that the car is still turning right. But in case of good road texture the quality ofFPGA tracker is very good, which can be seen in Figures 6.2(a), 6.2(b), 6.2(c), 6.2(d) wheretextureless and textured road turn are shown.

Simple reconstruction of camera tracks is shown in Figure 6.3. There it is seen how theincorrect handling of road turn in�uences further feature tracking until textured part of theroad is reached. Handling of road turns is poor and processing of frames after the road turnis very often totally broken. Even if road turn has enough features to be handled correctly itoften happens that the road after the road turn is featureless so the FPGA implementationcan not catch the change in the direction of movement. In this case prediction stays thesame as it was for road turn which further worsens result.

As it was mentioned, the only signi�cant di�erence between the matlab and the FPGAimplementation is that the latter does not use �oating-point arithmetics and sub-pixel inter-polation. Search of the correspondences very often stops in the �rst iteration because nextiteration requires sub-pixel movement. It can be seen in Figure 6.4 which shows the meanvalues of prediction for each axis. Flat parts of the series mean that prediction does notchange between frames. Not only it di�ers from mean values of matlab but in general suchstrait movement is barely possible.

19

20 CHAPTER 6. RESULTS

(a) Frame 1 (b) Frame 20

(c) Frame 75 (d) Frame 125

Figure 6.1: Comparison between matlab and FPGA feature tracker. Test video-sequence 44.Red - FPGA, green - matlab.

21

(a) FPGA, textureless (b) matlab, textureless

(c) FPGA, textured (d) matlab, textured

Figure 6.2: Textureless vs textured road turns.

22 CHAPTER 6. RESULTS

44 45

46 47

48 49

50 51

Figure 6.3: Reconstructed camera movement, sequences 44 to 51.

23

44 45

46 47

48 49

50 51

Figure 6.4: Mean predicition values, sequences 44 to 51.

24 CHAPTER 6. RESULTS

Chapter 7

Conclusion

This project addressed all the tasks in the assignment. The algorithm was developed andimplemented using required technologies. During the development the widely used tools suchas KL tracker or RANSAC were adapted and integrated.

Major modi�cations of KL tracker were done to comply to restrictions of FPGA chip.Sub-pixel interpolation was removed to avoid �oating-point calculations. Multiplication byinverse weight matrix was cut to multiplication of respective items and division by deter-minant, again to avoid making �oating point calculations. Because of capacity restrictionsinstead of passing whole work area only areas of interest and their respective search areaswere passed to FPGA one by one. FPGA result for whole work area was then gatheredbefore running RANSAC.

ADTF framework plug-in which does the input preprocessing and FPGA results process-ing was implemented. Wrapper of selected functions from BLAS and LAPACK libraries,which provides intuitive implementation of matrix operations, was written.

Results (Chapter 6) show that the FPGA implementation of the algorithm must usesub-pixel interpolation and �oating-point arithmetics. Without it the algorithm can not beused. On the featureless parts of the road the correspondences are not found correctly andhandling of road turns is very bad.

7.1 Improvements

The must be improvement is the implementation of the sub-pixel interpolation and usage of�oating-point arithmetics. When it is done it can further be improved by application of theKalman �lter to the estimated road homography.

Data storage in FPGA design was based on registers. Instead of registers the onboardRAM can be used. It may reduce the complexity of the design and allow to use the randomaccess to data. It can also free up resources which can ease the task of implementation ofthe sub-pixel interpolation.

25

26 CHAPTER 7. CONCLUSION

Bibliography

[1] National Highway Tra�c Safety Administration. Statistical Analysis of the E�ectivenessof Electronic Stability Control (ESC) Systems, 2007. URL .

[2] National Highway Tra�c Safety Administration. The Long-Term E�ect of ABS in Passen-ger Cars and LTVs, 2009. URL .

[3] Audi Electronics Venture GmbH. Automotive Data and Time-Triggered Frame-work. URL .

[4] EU road safety database or national publications. Road safety evolution in EU, 2012.URL .

[5] Olivier Faugeras and F. Lustman. Motion and structure from motion in a piecewiseplanar environment. Technical Report RR-0856, INRIA, June 1988. URL .

[6] Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm formodel �tting with applications to image analysis and automated cartography. Comm. ofthe ACM, 24(6):381�395, June 1981.

[7] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision.Cambridge University Press, New York, NY, USA, 2 edition, 2003. ISBN 0521540518.

[8] Bruce D. Lucas and Takeo Kanade. An iterative image registration technique with anapplication to stereo vision. International Joint Conference on Arti�cial Intelligence,pages 674�679, 1981.

[9] The World Bank. Motor vehicles (per 1,000 people), 2013. URL .

27

http://www-nrd.nhtsa.dot.gov/Pubs/810794.pdfhttp://www-nrd.nhtsa.dot.gov/Pubs/810794.pdfhttp://www-nrd.nhtsa.dot.gov/Pubs/811182.pdfhttp://www.audi-electronics-venture.de/aev/brand/en/services/development_tools/adtf.htmlhttp://www.audi-electronics-venture.de/aev/brand/en/services/development_tools/adtf.htmlhttp://ec.europa.eu/transport/road_safety/specialist/statistics/http://hal.inria.fr/inria-00075698http://hal.inria.fr/inria-00075698http://data.worldbank.org/indicator/IS.VEH.NVEH.P3/countries/EU?display=graphhttp://data.worldbank.org/indicator/IS.VEH.NVEH.P3/countries/EU?display=graph

28 BIBLIOGRAPHY

Appendix A

DVD content

item sub-item description

src.zip archive with sources| c++ ADTF plug-in C++ source code| matlab MATLAB implementation of the algorithm| vhdl FPGA design of KL trackerthesis source code of this thesisvideo test video-sequences with drawn horizon and correspondences

f* � FPGA, m* � MATLABthesis.pdf this thesis

Table A.1: DVD content

29

IntroductionTask formulation

Areas of interest trackingWork areaAreas of interest and search areasShift predictionTracking algorithm

FPGA implementation of areas of interest trackingRestrictions and simplificationsImplementation

Robust homography estimationHorizon estimationResultsConclusionImprovements

DVD content