Automatic Camera Calibration for Image Sequences of a

Football Match

Flávio Szenberg (PUC-Rio)

Paulo Cezar P. Carvalho (IMPA)

Marcelo Gattass (PUC-Rio)

reference pointsobject points

Juiz Virtual

reference pointsobject points

Juiz Virtual

Proposed algorithm

Computation of the planar projective transformation

Camera Calibration

Filtering to enhance lines

Next image in the sequence

Detection of long straight-line segments

First image of the sequence

Line recognition

Line readjustment

Computation of the initial planar projective transformation

Filtering to enhance lines

The Laplacian of Gaussian (LoG) filter is applied to the image with threshold.

010

141

010

121

242

121

16

1

Gaussian filter

Laplacian filter

Detection of long straight line segments

Segmentation is done in the image to locate long straight line segments candidate to be field lines.

This procedure is divided in two steps:

• Eliminating pixels that are not in a straight line.

• Determining straight lines segments.

Eliminating pixels that are not in astraight line

The image is divided, by a regular grid, in rectangular cells.

For each of theses cells, the eigenvalues, 1 2 of the covariance matrix, given below, are computed.

If 2 = 0 or 1/ 2 > M (given) then

eigenvector of 1 is the predominant direction

else

the cell does not have a predominant direction

Eliminating pixels that are not in astraight line

n

ii

n

iii

n

iii

n

ii

yyyyxx

yyxxxx

0

2

0

00

2

'''

'''

Cells with pixels forming straight line segments:

Eliminating pixels that are not in astraight line

Determining straight line segments

The cells are traversed in such a way that columns are processed from left to right and the cells in each column are processed bottom-up. Each cell is given a label: If there is no predominant direction in a cell, discard it. Otherwise check the three neighbors to the left and the neighbor

below the given cell. If any has a predominant direction similar to that of the current cell, then it receives the label of that cell;

otherwise, a new label is used for the current cell.

Determining straight line segments Group the cells with the same label Merge the groups that correspond to segments that lie on

the same line. At the end of the process, each group provides a straight

line segment.

Field lines recognition From the set of segments, the field lines are detected and

the field is recognized.Model-based recognition method [Grimson90]

• Set of restrictions

F1 F7 F6F5F4F3F2

f1:

f2:

Interpretation TreeF1

F6F2

F3

F4

F5 F7

Model

f1

f2

f3

f4

f5

Visualization

f6

f7

F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2

The node {f1: F1, f2:F6 , f3:F3} is discarded because it violates the

restriction:The line representing F6 must be between the lines

representing F1 and F3.

Field lines recognitionDiscarding nodes

F1

F6F2

F3

F4

F5 F7

Model

f1

f2

f3

f4

f5

Visualization

f6

f7

F1 F7 F6F5F4F3F2

F1 F7 F6F5F4F3F2

f1:

f2:

Interpretation Tree

F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2 F1 F7 F6F5F4F3F2

f3:

Field lines recognitionChoosing the best solution

F1

F6F2

F3

F4

F5 F7

Model

f1

f2

f3

f4

f5

Visualization

f6

f7

• In general, there are several feasible interpretations;

• We choose the one where the sum of the length of the matched segments is maximum.

f1 : f2 : F3 f3 : f4 : F1 f5 : F6 f6 : F4 f7 : F7

f1 : f2 : f3 : F3 f4 : F1 f5 : F6 f6 : F4 f7 : F7

WINNER

F7

f1

f2

f3

f4

f5

Visualization

f6

f7

F1

F6F2

F3

F4

F5 F7

Model

Field lines recognitionFinal result

f1 : f2 : F3 f3 : f4 : F1 f5 : F6 f6 : F4 f7 : F7

f1

f2

f3

f4

f5

Visualization

f6

f7

F1

F6F2

F3

F4

F5

Model

f1 : f2 : F3 f3 : f4 : F1 f5 : F6 f6 : F2 f7 : F5

or

Computation of the initial planar projective transformation

A planar projective transformation corresponding to the recognized lines is found (using points of intersection and vanishing points as reference points).

points of intersection

vanishing points

Line readjustment

tolerance of the line readjustment

Line readjustment

reconstructed line

tolerance

image points

new located line

The new located line is obtained by a least square method

Computation of the final planar projective transformation

After relocating the field lines, a better reconstruction of the field can be obtained.

Camera calibration

Camera is calibrated using Tsai’s method for reconstruction of elements not on the plane of the field.

• For the first image, we apply the camera calibration process proposed.

• In order to optimize the process from the second image on, we take advantage of the previous image calibration.

• The final planar projective transformation for the previous image is used as a initial transformation for the current image.

Working with a sequence of images

Computation of the planar projective transformation

Camera Calibration

Next image in the sequence

Line readjustment

Results

First scene Last scene

The artificial sequence

First scene Last scene

The real sequence

Results

Computer: Pentium III 600MHz

The sequence of test has 27 frames

The time of processing: 380 milliseconds

(< 900 milliseconds needed to real-time)

Results (accuracy)

Field’s PointsCorrect

CoordinatesReconstructed Coordinates Error

x y z u v u v105.0 68.00 0.00 81.707 216.584 81.731 215.972 0.612

88.5 13.84 0.00 230.117 78.133 228.747 77.525 1.499

88.5 54.16 0.00 1.236 183.463 0.424 183.197 0.854

99.5 24.84 0.00 259.039 134.206 258.566 133.815 0.614

99.5 43.16 0.00 146.690 174.826 146.067 174.484 0.711

105.0 30.34 0.00 269.817 155.102 269.629 154.697 0.446

105.0 30.34 2.44 270.921 181.066 270.215 180.863 0.735

105.0 37.66 2.44 224.101 194.645 223.291 194.407 0.845

105.0 37.66 0.00 223.405 170.271 223.082 169.876 0.510

Average Error 0.696Tab. 1 - Comparison between the correct and reconstructed coordinates for the first scene.

Results (accuracy)

Field’s PointsCorrect

CoordinatesReconstructed Coordinates Error

x y z u v u v105.0 68.00 0.00 97.167 205.940 96.791 205.585 0.517

88.5 13.84 0.00 243.883 66.434 243.549 66.022 0.530

88.5 54.16 0.00 16.101 173.174 15.655 172.623 0.709

99.5 24.84 0.00 273.344 124.029 273.125 123.715 0.382

99.5 43.16 0.00 160.672 164.798 160.366 164.421 0.486

105.0 30.34 0.00 284.160 145.173 283.992 144.914 0.309

105.0 30.34 2.44 285.241 171.290 284.886 171.090 0.407

105.0 37.66 2.44 238.127 184.768 237.744 184.538 0.447

105.0 37.66 0.00 237.462 160.349 237.252 160.063 0.355

Average Error 0.452Tab. 2 - Comparison between the correct and reconstructed coordinates for the last scene.

Conclusions

• The algorithm presented here has generated good results even when applied to noisy images extracted from TV.

• The algorithm can be used in widely available computers (no specialized hardware is necessary).

• Processing time is well below the time needed for real-time processing. Extra time can be used, for example, to draw ads and logos on the field.

Future work

• Investigate processes for smoothing the sequence of cameras by applying Kalman filtering.

• Develop techniques to track other objects moving on the field, such as the ball and the players.

• Draw objects on the field behind the players, to give the impression that the players are walking on them.