Applications of Image Processing to Sign Language Recognition

Byron HoodComputer Systems Lab Project 2007-08

Applications of Image Processing to Sign

Language Recognition


Overview Average person typing speed

Composing: 19 wpm Transcribing: 33 wpm

Sign speaker Full sign language: 200—220 wpm Spelling: est. 50 wpm


Purpose and Scope Native signers faster “typing”

Benefits: Hearing & speaking disabled Sign interpreters

Just letters & numbers


Research Original Related projects

Using mechanical gloves Tracking body parts

Image techniques Edge detection Line detection


Program Architecture (1)

Webcam interface Using Video 4 Linux 1 Saves images to disk

Edge detection Robert’s Cross

Line detection Hough transform


Program Architecture (2)

Line processing Image data dropped AI to find “best fit”

Letter matching Load hand data from XML files Try to match hand


Sample Code From edge_detect.c : int row,col;// loop through the imagefor(row=0; row<rows; row++) { for(col=0; col<cols; col++) { // avoid the problems from trying to calculate on an edge if(row==0 || row==rows-1 || col==0 || col==cols-1) { image2[row][col]=0; // so set value to 0 } else { int left = image1[row][col-1]; // some variables int right = image1[row][col+1]; // to make the final int top = image1[row-1][col]; // equation seem a int bottom = image1[row+1][col]; // little bit neater image2[row][col]=(int)(sqrt(pow(left – right , 2) + pow(top - bottom, 2))); } }}


Testing (1) General testing model

bhood@bulusan: ~/syslab-tech $ \> ./main images/hand.pgm in.pgm

[DEBUG] Edge detect time: 486 ms[DEBUG] Wrote image file to `in.pgm’

Errors: 0 Warnings: 0


Testing (2) Results:

Of edge detection

Similar for cropping or line finding

Automated testing difficult and impractical

Original image(800 x 703)

Final image(800 x 703)


The Mysterious Future Finish line detection

Detection and parameterization Build AI to interpret lines

Finish camera-computer interaction

OPTIMIZE!!!

Applications of Image Processing to Sign Language Recognition

Documents

Transcript of Applications of Image Processing to Sign Language Recognition