Sign Language Classification Process By neural Network
-
Upload
tunvir-rahman-tusher -
Category
Engineering
-
view
294 -
download
3
description
Transcript of Sign Language Classification Process By neural Network
A Contour Based Approach to Classify A Contour Based Approach to Classify Hand Posture using Neural NetworkHand Posture using Neural Network
Presented byMd.Tunvir Rahman
ID:0704026
Supervised byAnik Saha
Lecturer , CSE CUET
2
Department of CSE, CUET
Motivation
Touch less interaction with devices require fast, robust method to classify hand posture.
Autonomous driving require to classify the hand gesture shown by traffic police or passengers.
Home appliances like TV, Microwave oven etc. need posture classification.
Giving command to a robot can be done by hand gesture which needs a good gesture classifier.
3
Department of CSE, CUET
Previous Work and Limitation
In[6] template matching approach which require hand band in the hand to normalize the image.
In [1] orientation Histogram based approach some times map same posture in different class.
In [2] gesture classification by presence of number of fingers and their respective distance with palm center limit the number gesture to be classified
4
Department of CSE, CUET
Goal
Classify gesture in an dynamic background.
No special marker in the hand.
Noise reduction from image frame.
Implementation of neural network as classifier.
Implement this approach to classify Bangla Sign character.
5
Department of CSE, CUET
Our Proposed Methodology
Hand Region Segmentation
RGB Image
Preprocessing
Connected Component labeling and Noise Removal
Normalization and Contour Detection
Feature Extraction
Neural NetworkTraining Set
Classified Posture
Train
Test
6
Department of CSE, CUET
Recognition System
Detected sign
7
Department of CSE, CUET
ROI Detection
ROI Detection Based on Skin Color
Some unwanted region appears in the frame
8
Department of CSE, CUET
Preprocessing
Erosion and Dilation on Binary image to smooth the image contour and remove small holes.
255
( , )( , ) if
d I i j ifO i j at least one neighbor is 255
at least one neighbor is 255 Dilation
9
Department of CSE, CUET
Preprocessing
Erosion
255
( , )( , ) ife I i j ifO i j all 8 neighbors are 255
at least one neighbor is 0
10
Department of CSE, CUET
Noise Reduction
Label the Connected Component using Flood Fill and Consider two big region containing maximum binary data. Other will be considered as noise.
Color Segmented Image After Removing Noise
11
Department of CSE, CUET
Flood Fill Algorithm
2
P1 P2 p3 p4 p3 p4
2
p3 p4 P4 p5
2
p5 p6 p7
2 2
2 2
A Connected region label
by 2
12
Department of CSE, CUET
Normalization
Hand Forearm follow an Non-increasing radius shape up to wrist of the hand .
Forearm part is unwanted for classification.
Contour pixel of Hand shape
13
Department of CSE, CUET
Feature Extraction
90
0
Summing Up Number
of Pixel’s lie in this Angle
Total 19 histogram is extracted from the image
180
14
Department of CSE, CUET
Feature Extraction
Each Bin Contain count of Contour pixel .
Taking ratio of bins count and pass this ratio as the feature vector to the neural network.
First train the network by feature(input) and response (output).
Then test gesture with the trained network.
15
Department of CSE, CUET
F1
F2
1
2
3
N
5
2
1
Our Proposed Network
Hidden layer
F3
Fn
Input Output
16
Department of CSE, CUET
1
X1
x2
w11= 3
w21= 4
w12= 6
w22= 5
w10= 1
w20= -6
w21= -1
w22= 1
1
0
w01= -3.93Input
Target
Neural Network and Back propagation
1
w02= 1
17
Department of CSE, CUET
I1
1
I2
H1
H2
O1
X1
x2
-6
1
0
Input Target
Neural Network and Back propagation
Activation a=1*3+0*4+3*1*1=4
3
41
6
5
2
4
Output Y1=0.982
Activation a=0
Output Y2=0.50
-3.93
1 1
Output 0.51
Target-Output=0.49
O2 1
2
4-3.
Output 0.72
Target-Output=0.27
Total Error=0.49+0.27=0.76
18
Department of CSE, CUET
I1
1
I2
H1
H2
O1
X1
x2
-6
1
0
Input Target
Calculating the Delta values for Output and Hidden Neuron
3
41
6
5
2
4
-3.93
1 1
O2 1
2
4-3.
out= out*(1-out)*(target-out)
o1 =0.51*(1-0.51)*(1-0.51)=0.1225
o2 =0.27*(1-0.27)*(1-0.27)=0.14
= out*(1-out)*W*oi
h11 =0.982*(1-0.982)*2*0.1225=0.0043h12 =0.982*(1-
0.982)*2*0.14=0.0049
h21 =0.51*(1-0.51)*4*0.1225=0.1225
h22 =0.51*(1-0.51)*4*0.14=0.139
19
Department of CSE, CUET
I1
1
I2
H1
H2
O1
X1
x2
-6
1
0
Input Target
Neural Network and Back propagation
3
41
6
5
2
4
-3.93
1 1
O2 1
2
4-3.
Wij= η*Yi*jη= Learning Constant=0.1
o1 =0.1225
o1 =0.14
h11 =0.0043
h12 =0.0049
h21 =0.1225
h22 =0.139
w(03)= 0.1*1*0.0043=0.0004
1.0004
w (ij)new =wij(old)+w(ij)
20
Department of CSE, CUET
I1
1
I2
H1
H2
O1
X1
x2
-5.987
1
0
Input Target
Neural Network and Back propagation
3.0004
41
6.0123
5
2.012
4.006
-3.918
1 1
O2 1
2.013
4.012-2.98
1.0004
21
Department of CSE, CUET
I1
1
I2
H1
H2
O1
X1
x2
-5.987
1
0
InputTarget
Network response after Weight adjustment
3.0004
41
6.0123
5
2.012
4.006
-3.918
1 1
O2 1
2.013
4.012-2.98
1.0004
New OutH1=0.9820New OutH2=0.5063New OutO1=0.5214New OutO2=0.736
1-0.5214=0.4786
1-0.736=0.264
Total Error=0.4786+0.264
=0.7426In First Iteration Error reduced from 0.76 to 0.74
Iteration Continues until the desired error goal is achieved
22
Department of CSE, CUET
Experimental Analysis
• Performance depends on the no of training set
• Train: Test ratio significantly effects successful classification.
• Defining 100 neurons in
hidden layer
requires around
450 epochs to reach
the error goal.
23
Department of CSE, CUET
Performance Analysis
Train
SuccessfulClassification
Rate
10 Sample for each Sign character
24
Department of CSE, CUET
Limitation
Background fully skin-colored the classification system fail.
Noise component is larger than the hand ROI.
Angular distortion cause the system failure.
25
Department of CSE, CUET
Future Works
Shape based hand region segmentation can make the classification independent of background.
Dynamic hand gesture can be extracted from video and make the system user friendly.
26
Department of CSE, CUET
References
[1] William T. Freeman and Michel Roth, “Orientation Histograms for Hand Gesture Recognition ” IEEE Intl. Workshop on Automatic Face and Gesture Recognition, Zurich, June ,2006
[2] S.M Hassan Ahmed Todd C Alexender, “Real Time static and dynamic hand gesture recognition for human computer Interaction”-Electrical Engineering, University of Miami, FL.
[3]Priyanka Mekala, “Real-time Sign Language Recognition based on Neural Network Architecture”, Florida International University, FL, U.S.A
[4] Klimis Symeonidis “Hand Gesture Recognition Using Neural Networks”, School of Electronic and Electrical Engineering, August 23, 2009.
[5]Bowden & Sarhadi ” Building Temporal models for Gesture Recognition” in preceding British Machine Vision Conference, pages 32-41,2002.
[6] Dr. Kaushik deb, Helena Parveen Mony & Sujan Chowdhury “Two Handed Sign Language Recognition for Bangla Sign Character using Cross Correlation” Global journal of Computer Science and Technology, Volume 12, Issue 3, February 2012.
27
Department of CSE, CUET
Thanks