Marcin IWANOWSKI

Warsaw University of Technology, Institute of Control and Industrial Electronics

An application of mathematical morphology to filter ing and feature extraction for pattern recognition

Abstract. Mathematical Morphology is a very efficient tool for image processing and analysis, based on non-linear local operators. In this paper mathematical morphology is applied to pattern recognition. The appropriate practical methods of morphological image processing are described, as well as its theoretical background. An example of an application of described methods to recognition of car number plates is also presented. Mathematical morphology is applied to filter the image and to extract some topological features from each character. These features are later included in a feature vector, which is recognized by using the decision tree. The aim of the paper is to proof that mathematical morphology is a powerful technique which could be efficiently applied in the engineering practice to pattern recognition.

Streszczenie. Morfologia matematyczna jest bardzo efektywnym narzędziem komputerowego przetwarzania i analizy obrazów opartym na nieliniowych operatorach lokalnych. W artykule morfologia matematyczna została stosowana do rozpoznawania wzorców. Opisane zostały odpowiednie praktyczne metody morfologicznego przetwarzania obrazów, jak również ich teoretyczne podstawy. Pokazany jest także przykład zastosowania opisanych metod do rozpoznawania tablic rejestracyjnych samochodów. Morfologia matematyczna jest stosowana do filtracji obrazu oraz do ekstrakcji z każdego znaku pewnych jego cech topologicznych. Cechy te zostają następnie umieszczone w wektorze cech, który jest następnie rozpoznawany z wykorzystaniem drzewa decyzyjnego. Celem artykułu jest wykazanie, że morfologia matematyczna jest wydajna techniką znajdującą swoje zastosowanie w praktyce inżynierskiej do rozpoznawania wzorców Keywords : image processing, mathematical morphology, feature extraction, pattern recognition. Słowa kluczowe : przetwarzanie obrazów, morfologia matematyczna, ekstrakcja cech, rozpoznawanie wzorców. 1. Introduction Mathematical morphology is a very efficient tool for image processing and analysis, based on non-linear local operators. In this paper mathematical morphology is applied to pattern recognition. The morphological operators are applied to filter the image and to produce the features necessary for recognition. The algorithms for feature extraction by mathematical morphology are described, as well as its theoretical background. A practical application is also presented. It is devoted to analysis of binary images, containing printed characters – images of car number plates. Mathematical morphology is used to filter the image and to extract some information (features) from each character. These features are later measured and included in a feature vector. It contains the special kind of topological information - the homotopy of the character, which equals in the case of printed characters to the number of holes in the character shape. The homotopy is calculated in the original character and in the character modified by superimposing of four masks. Also, the number of horizontal and vertical lines is computed as well as the proportions of the character. In the paper the complete identification scheme is presented, beginning from the input graytone image of number plate, till the final result of recognition. The paper is divided into 5 sections. The first one is this introduction. In the next one the necessary image processing background including the basic image operations, the introduction to mathematical morphology and labeling algorithm are described. In the third section the feature extraction methods are described. In the section 4, the practical application to car number plate identification is described. The last section concludes the paper.

2. Necessary image processing background

2.1. Types of images and basic operations Two-dimensional digital image is usually represented by a matrix of pixels. It is a mapping of the coordinates’ space into a space of pixel values: (1) Iyxf →× },...,0{},...,0{:

where x,y are the sizes of the image f along the x- and y-axis, respectively. Depending on the possible values of pixels (set I in Eq.1) different types of images are defined, three of which are used by methods described in this paper. For }1,0{=I binary

images are defined, for }maxgray,...,0{=I - the graytone ones

(maxgray is the maximum grayvalue - value of white - usually equal to 255) and for },...,{ 0 nllI = - the labeled (or

mosaic) images. Last two kinds of images usually have the same representation, but their meaning is different. Values of the graytone image represent a real intensity of light, while the values of labeled image are the labels describing the affiliation of pixels into given partitions.

Fig.1. Graytone (a), labeled (b) – graytones represent labels, and

binary image (c).

To obtain the binary image from the graytone one the thresholding operation is applied:

(2) [ ]

≤≤

=otherwise

bjifaifjifT ba

),(

0

1),()(],[

If binary image contains separated objects (groups of neighboring pixels – connected components) it can be transformed into the labeled image by means of the operation of labeling. The algorithm of labeling is described later in this section. A partition with label l of labeled image could be extracted into a separated binary image: (3) [ ]

=

=otherwise

ljififjifT l

),(

0

1),()(][

The binary images can be combined using the intersection and sum operators:

==

=∩otherwise

jigjififjigf

1),(),(

0

1),]([

(4)

==

=∪otherwise

jigjififjigf

0),(),(

1

0),]([

In case of graytone and binary images one defines the inversion of the image (for binary images maxgray = 1): (5) ),(maxgray),( jifjif C −=

2.2. Morphological image processing Image processing by using mathematical morphology [1,2,3,4] is based on two nonlinear local operators: minimum (infinimum) and maximum (supremum) applied within the neighborhood of every image pixel. This neighborhood is defined using the structuring element . It is translation invariant and shape of pixel’s neighborhood. Various structuring elements are in common use in the morphological image processing. The most widely applied are elementary and directional ones. The elementary structuring element contains the closest pixel neighborhood, which consists in 4, 6 or 8 neighbors depending on the image grid used (4, 6 or 8 connectivity). In this paper the 8-connectivity taking into account. The directional structuring element contains the neighboring pixel in the given direction. This allows one to filter or detect the directional features. Examples of structuring elements are shown on Fig. 2.

Fig.2. 8-connectivity (a), structuring elements: elementary (b),

directional (c)-(h). Dilation and erosion of image f with structuring element B are defined as, respectively: (6) [ ] ,),(max),)](([

,vjuifjif

BvuB ++=

∈δ

(7) [ ]),(min),)](([,

vjuifjifBvu

B ++=∈

ε

where p stands for the single image pixel. The operators are applied to all the pixels of image f. Dilation and erosion of given size n are defined as respectively: )...))(....()(

)( fftimesn

BBBn

B−

= δδδδ

(8) )...))(....()(

)( fftimesn

BBBn

B−

= εεεε

Fig.3. Input object (a), its erosion (b), dilation (c), directional vertical

dilation (d) and directional horizontal dilation (e).

The morphological filters [5] of opening and closing contain the combination of dilation and erosion performed one after another: (9) ))(()( )()()( ff n

Bn

B

nB T εδγ = ; ))(()( )()()( ff n

Bn

B

nB T δεϕ =

where γ represents the operator of opening and φ – the operator of closing, }),(:),{( BjijiBT ∈−−= stand for

transposed structuring element B. Opening and closing removes from the image its elements (object, noise) respectively lighter and darker then the background. The side-effect of these filters is however the disturbance of the shape of objects which are not removed. Contrary to linear filters (mean, Gaussian etc.) this disturbance is not blurring but the change of shape. By combining the opening and closing the alternating filters are defined. They are applied to remove both darker and lighter elements of the image. Examples of morphological filtering are given on Fig. 4.

Fig.4. Noisy input image (a), its closing (b), closing followed by opening (c), opening (d), opening followed by

closing (e). 2.3. Labeling algorithm As described previously, a binary image containing several object can de transformed into a labeled image by the operation of labeling. An appropriate algorithm is described in this section. It makes use of the structure of stack which is accessible through three procedures: stack_push, which puts the pixel coordinates into the stack; stack_pop, which removes the coordinates from the top of the stack; and logical function stack_empty, which have true value if there are no pixel coordinates on the stack, and false otherwise. The complete algorithm written in pseudo-code is shown below:

The image p is the input binary image, l – is the output labeled one, containing the particles of image p labeled

Labeling algorithm

for i :=0 to x : for j := 0 to y: if p(i,j)=0 then l(i,j)=0 else l(i,j)=-1 label := 0 for i :=0 to x : for j := 0 to y: if l(i,j)=-1 then: label := label+1 stack_push([i,j]) while not(stack_empty): [i,j] := stack_pop l(i,j) := label for m:=-1 to 1: for n:=-1 to 1: if l(i+m,j+n)=-1 then: stack_push([i+m,j+n])

beginning from the value 1. The final value of variable label indicates the number of labeled particles. 3. Feature extraction using the morphological image processing In this section the application of previously described morphological operators to the feature extraction is presented. It shows features which could be obtained by using the morphological image processing. Since the example of application described later concerns the recognition of printed characters, all the features described below deal with such objects. 3.1. Character area The morphological directional dilation has the property of deforming the object’s shape in the horizontal/vertical direction. Owing to that fact, the area of the character (the smallest rectangle, in which the character is entirely included) can be detected using the following equation: (11) )()()( )()( fffarea n

Vn

H δδ ∩=

where H and V are respectively horizontal (Fig.2c) and vertical (Fig.2d) structuring elements; n is the size of dilations which must be larger than the height of the character in pixels. The process of extracting the area is shown on Fig. 5.

Fig.5. Character area: input character (a), its vertical dilation (b),

horizontal one (c) and intersection of both (d). Character area has two important parameters which are used in the recognition process: its height h and width w. They are applied to compute the character proportions, which is the first feature of the character: (12) whsproportion /=

3.2. Homotopy (holes) An important feature of the character to be recognized is its homotopy. In case of letters, which are single connected components, the homotopy can be measure by the number of holes of the character. This measurement is performed on the inverted character (see Fig. 6). The inverted binary image is labeled. Depending on the homotopy of the character different number of labels can be determined. Since the background is classified as one labeled region, the number of holes is equal to the number of labels minus 1.

Fig.6. From left to right: character; its inversion, labeled image

Labeling is performed using the previously described algorithm. Fig.6 presents the labeling of the inverted character. By using the feature of homotopy one can distinguish three groups of characters: without holes (e.g. ‘1’,’T’,’I’, etc.), with one hole (e.g. ‘0’,’A’,’P’, etc.) and with two holes (‘8’ and ‘B’). Considering the fact that the input bitmap with the character can be corrupted by some noise (even after the filtering it could be possible), only the holes larger (by the number of pixels belonging to) than a given threshold value can be counted. 3.3. Homotopy of deformed character By using the number of holes of the character, only 3 groups of characters can be determined, which is not enough for the correct recognition. In order to generate more features the numbers of holes are computed also in the modified character. These modifications consist in superimposing the masks with the input image. The four masks are proposed. They equal four inner borders of character area of a given thickness. They allow ‘closing’ the shape of the character and modifying its homotopy, which results in the change of the number of holes. The examples of characters with mask superimposed are shown on Fig. 7.

Fig.7. Characters with masks superimposed.

The modifications are obtained by using the directional erosion of the character area (Eq.11) with the directional structuring elements L,R,T,D shown on Fig. 2e – 2h, respectively. The modified characters are obtained from the following equations:

(13)

( )( )

( )( ))(

)(

)(

)(

)(

)(

)(

)(

fareaff

fareaff

fareaff

fareaff

kwLright

kwRleft

khTdown

khDtop

⋅

⋅

⋅

⋅

∪=

∪=

∪=

∪=

ε

ε

ε

ε

Sizes of erosions depend on the sizes of character area w and h and constant coefficient k (e.g. k=0.12). It determined the thickness of the mask. Also the combinations of the above masks are possible. For example by adding both down and right borders:

(14) ( ) ( ))()( )()( 21 fareafareaff kwL

khTrightdown

⋅⋅− ∪∪= εε

In this case two parameters k1 and k2 are considered, their choice depends on the mask one wants to obtain. This issue is discussed later in this paper. 3.4. Horizontal and vertical lines Important topological feature of the character is the existence of the horizontal and vertical lines in the

character. They are detected by means of the opening with directional structuring element. This operation performed with relatively big size removes all those parts of the character which are not long enough in the particular direction. Only the longer parts rest. The size of the operation has to be chosen so that the lines obtained allow detecting a difference between various characters.

Fig. 8 Vertical lines of character (a) detected in (b), horizontal lines

of (c) detected in (d). Detected lines are superimposed on the contours of the characters.

The following equations describe detection of horizontal and vertical lines respectively:

(15) ))(()(

))(()()()(

)()(

ffV

ffHn

Vm

H

nH

mV

γγγγ

=

=

where H and V are respectively horizontal (Fig.2c) and vertical (Fig.2d) structuring elements. Each equation consists of two openings: first one, of size n is the main one performed in the same direction as the detected lines. The second one – of size m – is performed in a direction perpendicular to the first one. Its size is small nm << due to the fact that it is applied to filter the lines detected by the first opening. It can happen that the first operation detects thin (thinner than the lines being the part of characters) lines which are not the characteristic line of the character. To remove them the second opening is applied. Lines on the image modified as above have to be counted by labeling algorithm. After applying this algorithm, the variable label indicates the number of detected lines. An example of line detection is shown on Fig. 9. Another feature which could be produced in the similar manner is the number of right angles (crossing points) existing in the shape of the character. Since both mentioned above detections are performed along x- and y- axes of the character coordinates system, by its intersection allow detecting the crossing points of the horizontal and vertical lines in the character: (16) ))()(()( )( fVfHfCP s

B ∩= γ

In this case the parameter n from the Eq. 15 has to be smaller than in the previous case. An additional opening is performed in order to remover smaller objects, which are not referring to angles. Structuring element B is an elementary one (Fig.2b), size s should be set in such a way that only the correctly detected angles rest. 4. Car number plates identification The morphological operators described above can be efficiently applied in the process of identification of the characters of graytone image. The scheme of identification is presented on Fig. 9. It starts with a graytone input image, finish with a recognized string of characters and consists of 4 steps. At the beginning the input graytone image is filtered so that all the noise is removed and the image is prepared for the segmentation, which is the second step of treatment. During this step the graytone image is transformed into a labeled one where every character to be later recognized has its own label. In the third step every character is extracted and the feature vector is generated. This vector is

recognized in the fourth step. Two last steps are performed for every character independently.

Fig.9. Identification scheme

Following that scheme an example of car number plates recognition is presented in this section. Typical Polish car number plate consists of 7 characters. The characters are either black on white background (new plates), either white on black background (old ones).

Fig.10. Polish number plate (new)

In the paper the process of identification starts from the graytone image containing only the number plate as shown on Fig. 10. In the complete system an important this must be preceded by an important pre-processing step of localizing and extracting the number plate from bigger image containing either front or rear side of the car. This issue is not discussed in this paper. The characters on both types of plates, old and new, are the same. They are shown on Fig. 11.

Fig.11. Character set

The process of identification is described in four sections below. Each of them refers to one step of treatment as shown on Fig. 9. 4.1. Filtering At first the graytone image has to be filtered in order to remove the noise. It is obtained by the alternating filter consisting of opening and closing according to the Eq.9 with

Filtering Segmentation

Feature extraction

Recognition

Detected number

Input

elementary structuring elements (shown on Fig. 2b). The result of filtering is presented on Fig. 12.

Fig.12. Filtered image from Fig.10.

Filtering was necessary to remove the darker and lighter spots which are not a part of characters. Their existence would disturb the result of segmentation. 4.2. Segmentation Filtered image must be later transformed into set of binary images containing single characters. This process consists of two steps - binarization and labeling. The first step is done by thresholding according to the Eq.2 with fixed threshold value (a=0, b=90). In order to remove some unnecessary noise and other small areas of value ‘1’ on the binarized image, which doesn’t belong to characters, an additional filtering by opening (Eq.9) with elementary structuring element (Fig. 2b) is applied. Results of thresholding and filtering are presented on Fig. 13a and 13b, respectively.

Fig.13. Segmentation: binarized image (a), filtered image (b),

labeled image (c) – graytones refer to labels. The next step of labeling of binary image is performed using the previously described algorithm. It results in the image with labeled particles (characters), so that the first character from the left gets label ‘1’, next one – label ‘2’ etc. To obtain single characters required by further steps, the extraction according to the Eq.3 is performed. The result of labeling is shown on Fig. 13c. 4.3. Feature extraction The binary character obtained in the preceding step is the input image for the feature extraction. As an example, the following 8 features are produced:

1. Proportions of the character (Eq. 12) – real values between 1.42 (character ‘D’) and 7.15 (character ‘I’).

2. Homotopy (number of holes) of the character. 3. Homotopy of the character with mask L. 4. Homotopy of the character with mask R. 5. Homotopy of the character with mask T. 6. Homotopy of the character with mask D. 7. Number of horizontal lines (Eq. 15). 8. Number of vertical lines (Eq.15).

Features 2-8 can have integer values 0, 1, 2. The features were calculated using previously described methods. Such a feature vector was sufficient to obtain a correct recognition.

Fig.14. Decision tree for character recognition

O

R V

R

D

D

T

P

O

T

6

A

P

U

P

T

H

W

H

M

M’

P W ‘

N

P B

K O

H

P P

4

P

G

C

8

F

D

O

0

T

L

P

R

E

Y

3

L

T

O 9

X

H

Z

P

S

2

5

J

1

I

T 7

V

<=1,48

1,48-1,7

>1,7

<=1,53

>1,53

<=1,57

>1,57

<=1,57

>1,57

<=1,64

>1,64

<=1,71

>1,71

<=1,9 2,5-3,5 >3,5

1,9-2,5

<=1,65

>1,65

<=1,56

1,56-1,77

>1,77

V

H

O

P

vertical

horizontal

holes proportions

L

R

T

D

left mask

right mask

top mask down mask

value = 0

value = 1

value = 2

4.4. Recognition The feature vector of recognized character has to be later classified into one of the classes related to recognized character. There are various approaches to the problem of classification. The possible approaches are various types of classifiers: statistical, neural, based on fuzzy logic etc. The simplest however classification scheme is based on decision trees. This solution is, without doubts, the fastest one. The recognition result is obtained after several comparisons of feature values. In the example presented in the paper this solution was applied. In the decision tree methods the main focus is put into the creation of the optimal decision tree. The method applied was based on the method of induction of decision trees proposed by Quinlan [6,7]. Beginning from the table of features of every character, the methods produces the decision tree. The decision tree for the car number plates recognition is shown on Fig. 14. It has been produced using the DeTreex knowledge acquisition system [8]. Every node of the tree represents a test of a particular feature. The number of tests necessary for the recognition vary from 2 (character ‘6’: no. of vertical lines = 0; number of holes with R mask = 2) to maximum 6 (e.g. characters ‘T’ and ‘L’) 4.5. Discussion The features described in section 4.3 and resulting decision tree described in section 4.4 was presented as an example. A complete recognition system, which is constantly being developed, must consider larger character set considering bigger variety of character types as well as more real examples of each character. The development requires exhaustive testing of the algorithm and tuning the parameters used in the feature extraction and – if necessary – adding new features or replacing the existing ones by other more robust. Robust means in this context less sensitive for character deformations. As the already made experiments have shown, some of the features are more fragile and sensitive while other ones are more robust. The most fragile feature was the proportions of the character, especially at those points of the decision tree where this feature is a decisive one. Watching on fig. 14 one can observe that, for example, character groups ‘4’ and ‘P’; ‘T’ and ‘L’; ‘2’,’5’ and ‘S’ etc. are differentiated only by analyzing their proportions. The problem is that the differences between the proportions of the characters in every group are small, and due to that fact, sensitive for distortions of the character shape. The solution is to replace at all those points the proportions feature by some other one. Two features described previously, but not yet applied, are helpful: homotopy of the character with down-right mask and the number of right angles. Two versions of down-right masking could be applied. Each of them has its own parameters k1 and k2. Depending on the k2 different shapes of a mask. For higher value of k2 the mask hide the lower half of the character which implies that after homotopy computation, the character ‘2’ can be easily differentiated from ‘5’ and ‘S’. For lower values one can get the mask able to detect characters ‘4’,’P’,’T’,’L’. Both mask superimposed on characters are shown on Fig. 15.

Fig.15. Two versions of down-right masking

Fig.16. Right angles

The right angles detected are shown on Fig. 16. This feature allow finding a difference between characters in pairs ‘O’,’D’; ‘C’,’G’; ‘5’,’S’; ‘B’,’8’. By considering the features described above three new features can be introduced:

9. Homotopy of the character with mask DR (Eq.14) with k1 = k2=0,12

10. Homotopy of the character with mask DR (Eq.14) with k1 =0,12 and k2=0.6.

11. Number of right angles (Eq. 16). Calculation of these features allows avoiding the use of most of the proportions features in the decision tree from fig. 14. 5. Conclusion The paper has presented an application of morphological image processing to character recognition. The non-linear morphological operators of erosion, dilation, opening and closing have been applied to a pre-processing step of filtering and to the main step of the feature extraction. Detected features were topological ones: the homotopy of characters and characters modified by mask superposition, as well as number of horizontal and vertical lines included in the character. To illustrate the applicability of the described features, the example of a recognition system has been presented. It allows identifying the car number on the number plate by analyzing the graytone image containing this plate. Apart from the morphological techniques, the supplementary ones have been used in the recognition process: the labeling algorithm and the decision tree classifier. The aim of the paper was to show the high potential of the morphological methods of image processing, to illustrate it by the practical example and therefore to proof that the described methods can be efficiently applied in the engineering practice.

REFERENCES [1] Ser ra J. , Image analysis and mathematical

morphology, vol.1, Academic Press, 1983. [2] Ser ra J. , Image analysis and mathematical

morphology, vol.2, Academic Press, 1988. [3] Nieniewski M. , Morfologia matematyczna w

przetwarzaniu obrazów, PLJ Warszawa, 1998. [4] Soi l le P. ,Morphological image analysis, Springer

Verlag, 2002 [5] Ser ra J. , Vincent L., An overview of morphological

filtering, Circuit systems Signal Processing, 11(1), 1992. [6] Quinlan J.R. , Induction of Decision Trees, Machine

Learning 1 (1986), pp.81-106 [7] Quinlan J.R. C4.5 Program for Machine Learning,

Morgan Kaufman, San Mateo, CA, 1993 [8] Michal i k K., DeTreex 3.0, Indukcyjny system

pozyskiwania wiedzy, Katowice 2000 . Author: Marcin Iwanowski,Ph.D., Politechnika Warszawska, Instytut Sterowania i Elektroniki Przemysłowej, ul.Koszykowa 75, 00-662 Warszawa, E-mail: iwanowski@isep.pw.edu.pl.