Image Processing Based Robotics.pdf

Karan Bajaj

[email protected] | karanhindustan.blogspot.com

IT-BHU

VARANASI

IMAGE PROCESSING BASED

ROBOTICS

Karan Bajaj IMAGE PROCESSING BASED ROBOTICS

2 Robotics Club, IT-BHU [email protected]

Getting Started

Image Processing comprises of any form of signal processing for which

the input is an image. Not very surprising!

How do we use image processing in Robotics?

The Robot we would be using has an eye. Now where is this eye

located? It‟s generally a webcam that may be permanently attached

at some specific location, or attached to the Robot itself.

This webcam inputs images, which are sent to the computer via the

webcam connecting wire.

The concept of Image processing now comes in, and the acquired

image is processed using MATLAB programming (introduced later in

the tutorial).

Now we need to inform the Robot about the processed details. For

this, we use a port of the computer, which sends information from the

computer to the Robot.

The Robot contains a microcontroller which would decode the

information send via the port. This information is worked upon, and

signals are sent to the motors.

We would work on understanding these aspects STEP by STEP as

follows –

1. Image acquisition

Image taken in from webcam to be brought into the environment

(MATLAB) where it will be processed

2. Image processing

Processing of the image using the environment (MATLAB)

3. Using the port

Interfacing using Serial ports

4. Motor control

Controlling DC motors and Stepper motors



This diagram roughly depicts the connections.

MATLAB

MATLAB stands for Matrix Laboratory. It is a numerical computation

environment and 4th generation programming language that uses

matrices to do hell lot of things!

As I mentioned we would be using MATLAB, it makes sense that the

work of processing the Image would be done in MATLAB.

MATLAB uses matrices to store multimedia entities. Hence, our images

would also be stored as matrices.



MATLAB environment

We would be using MATLAB 7.8.0 (R2009a) for the tutorial.

A typical MATLAB 2009 window would look like the image above.

It is divided into four major sections –

Command Window

We write all the commands in this section

Current Directory

Shows us the contents of the current directory

Workspace

Displays the details of variables being used

Command History

Shows the list of commands recently used

The toolbar gives option for setting a custom current directory.

You were informed that MATLAB does „hell lotta stuff‟ using matrices.

You can yourself find the tutorials to all that using the Help button



The Help menu looks like this –

We would be restricting ourselves to the Image Acquisition tool and

the Image Processing tool for this tutorial.

Now let‟s start working with MATLAB.

All the commands would be written on the „>>‟ symbol you can see in

the Command Window.

Type „k = 9‟ and Press ENTER



You can see that MATLAB automatically stores „k‟ as a matrix of size

1x1 with value 9. This action also reflects in the Command History.

1D and 2D matrices can be stored as follows –

We can use the „;‟ operator after the statement to avoid display.

To access any element in the matrix, we can use indices corresponding

to that location. Indices in MATLAB start from 1.

For instance –

The value stored at second row and first column is „5‟

We can do simple mathematical operations using the matrix variables -



ans is the default variable of MATLAB, and is used if the assignment

operator is not present.

We can assign the result of this operation using the „=‟ and a variable

on the left.

Result will be stored in „c‟ instead of „ans‟

Trigonometric functions

sin(1) – value in radians is passed

sind(45) – value in degrees is passed

asin(1) – inverse sin in radians

asind(1) – inverse sin in degrees

Similarly, we can use cos(), cosd(), tan(), tand(), etc

General commands

clc – Clear the Command Window

clear – Reset the Workspace and delete all the variables stored



The colon ‘:’ operator

The colon operator is used for specifying “throughout the range”.

Some illustrations of the use of colon operator are given above.

Relational Operators in MATLAB

Control statements in MATLAB

Conditions statement – „if‟

Iteration statement – „for‟, „while‟



Syntax –

if <condition>

statements

else

statements

end

while <condition>

statements

end

for <initialization and condition>

statements

end

zeros(k) : Returns matrix of dimension k-by-k filled with zeros

zeros(p,q) : Returns matrix of dimension p-by-q filled with zeros

ones(k) : Returns matrix of dimension k-by-k filled with ones

ones(p,q) : Returns matrix of dimension p-by-q filled with ones

dot(A,B) : Returns dot product of two vectors A and B



Working with Images

We can read images in MATLAB as follows –

imread() function

Now the variable „img‟ stores the image cameraman.tif as a matrix.

One would expect an error message on typing this directly, but

MATLAB has some default images present, among which

„cameraman.tif‟ is one.

If one wonders the value of the matrix formed, try NOT using the

semicolon once!

The image can be displayed using the imshow() function as shown.



MATLAB supports three types of images –

Binary Image (Black and White) – Consisting of only two colours

Grayscale – Black, White and shades of grey

Colour – Contains RED, GREEN and BLUE colours in varied intensities



RGB value

For each pixel in a colour image, we have a value of the RGB colour

triplet. This triplet indicates the intensity of Red, Green and Blue on a

scale of 0 to 255 (256 colours).

RGB value of an image can be very easily obtained using the Data

Cursor tool in the toolbox of the „Figure 1‟ window of MATLAB.

For instance, has RGB value as 255, 128 and 0.

You can verify this by using the data cursor tool, which shows -

Making m-files

It sounds quite sensible when I say that we do not write the complete

code in Command Window. MATLAB codes are stored in special files

called m files.

You can use the icon for opening the Editor –



Type in the code inside this editor –

Save the m file as „sample.m‟ –

For running the m file, write onto the Command Window, the name of

the m file without the extension –

And the output (display of image in our case) will appear –



FUNCTIONS in Matlab

Functions in MATLAB are also stored in an m-file, with an exception

that the first executable statement of the file must be of the form –

function [out1, out2, ...] = <function_name>(in1, in2, ...)

The out1, out2, etc are the return values of the function. Unlike

C,C++, etc, a MATLAB function can return multiple values.

in1, in2, etc are the input parameters required by the function.

The function is written in the m-file as follows –

Save the m-file with the name of the function – „stat.m‟ in our case

The function can be executed from the Command Window in the

following manner –

The vector „a‟ is passed to the function „stat‟ as parameter, and return

values „m‟ and „std‟ are obtained.



Coming on to the aspects of Image processing we would be primarily

using –

1. Obtaining the Image

2. Isolating the area of Interest

3. Obtaining the desired outputs

IMAGE ACQUISITION

Hardware Information

For obtaining an image, we would obviously be having an image

acquisition device (like a webcam) onto our PC/laptop.

MATLAB has software called adaptors for accessing these devices.

You can obtain this information by using this command –

>> imaqhwinfo

Can be visualized as im-aq-hw-info (image-acquisition-hardware-

information)

It produces the following output on my system –

The adapter that MATLAB indicates on my system is „winvideo‟ as

shown above.

We can now use this adapter to gallop into further details!

For getting some more information onto the default colour format,

device name, etc, use the following command -



In case you have multiple image acquisition devices, you may use

further DeviceIDs like 2, 3, etc in the statement –

Now we would see the formats that our adapter supports –

These are different formats of capturing the video. The 1024x768

indicates the resolution of the video in that particular mode.

Defining Video Input Object

„winvideo‟ is the adapter name, „1‟ is the DeviceID of the device we

would be using, and „YUY2_320x240‟ is one of the supported format.



On pressing ENTER, one would see the following output –

This gives all details of the Video Input Object.

Now, how to switch the PREVIEW ON?

And the Video Preview window opens with the resolution you specified

in the object creation statement -

Preview can also be viewed

using different formats (Resolutions) from the Supported Formats list.



Capturing Image

We use the getshapshot() function for capturing an image from our

image input object -

The object appears so weird, you think either your webcam has gone

berserk, or it‟s some evil leprechaun!

Nothing to worry, guys! This happened because we took picture in

YCbCr family of colour spaces, which is “YUY2_...” format.

You can grab some more details about YCbCr here - http://en.wikipedia.org/wiki/YCbCr

We would have to convert this to RGB as follows -

http://en.wikipedia.org/wiki/YCbCr



The new window shows the snapshot in RGB colour space.

Saving an Image file

Similar to imread() command, we have the imwrite() for writing an

image variable onto the disk in form of an image file.

Image file stored

IMPORTANT –

Getting a snapshot requires multiple steps – switching ON the camera,

sufficient light entering in it, capturing taking place.

So there should be a pause of some duration between both the

statements, when the code is written in m-file.



vid=videoinput('winvideo',1, 'YUY2_160x120');

preview(vid);

pause(3); % Provides the required pause

while(1)

img=getsnapshot(vid);

% Do all image processing and analysis here

end

We now have an idea about how to acquire an image. Let us work on

processing the acquired image now!

IMAGE PROCESSING

As the image is now with us as input, our main task is to work on it.

Typical Image processing problems include locating a particular object

with some colour, or counting objects with the same colour, etc.

Let us see a simple example – Suppose we have an image „shapes.tif‟

We can store this image in a variable img using the imread() function,

and check for the RGB values at different positions. For the image

taken, the RGB at Red region is (255, 0, 0), at Green region is (0, 255,

0), and at Blue region is (0, 0, 255).



We make an m-file as follows –

Save this file as BB.m

Running the program in the Command Window –

Figure Window –



What we did in the program was to declare a zero matrix with

dimensions same as that of the test image.

For every pixel location, we test if the region‟s RED value is 255 and

GREEN and BLUE values are 0. If yes, we mark that place as „1‟

(white), otherwise it remains „0‟ (black) in the resultant binary image.

A piece of cake!

But real time problems consist of not a uniform colour, but a

distribution of intensities! Let us take another example –

Suppose we have an image – „flower.tif‟ in our MATLAB directory



The image is processed using the above program. Lets name the file

BB2.m

On executing the program –

We have approximately localized the violet colour region from the real

world image.

This choice of intensity selection has to be done manually and with

great care, as the final binary image will be computed accordingly.

For - if(im(i,j,1)>50 && im(i,j,2)>50 && im(i,j,3)>120);



Noise Removal

As we saw in the first binary picture of the flower, the resultant image

is distorted, with presence of holes, irregularities, etc.

This can be treated using the imclose() function.

But for using the imclose() function, we need to define a structuring

element first, using the strel command as follows –

Combining this with the program –

Modified Output



Another important function used for filling all holes in a binary or

grayscale image is – imfill() function

Using the imfill() function along with the program –

Resultant binary image that is produced does not contain any hole –

Similar to imclose() function, we have the imopen() function, which

performs morphological opening of the grayscale or binary image.

Example –

I = imread('snowflakes.png'); imshow(I)



se = strel('disk',5);

I_opened = imopen(I,se);

imshow(I_opened)

What actually happens here is that all snowflakes with radius less than 5 pixels were removed by using the structuring element of type „disk‟

with size 5 pixel, which is used.

Image Noise removal is very essential for making the data to be computed error free, and hence, our program need to be efficient at it.



Some more important commands used for Image Processing are –

regionprops()

STATS = regionprops(BW, properties)

It measures a set of properties for each connected component (object)

in the binary image, BW. The image BW is a logical array; it can have

any dimension.

For example, if you want to locate centroid of any object in a picture

you can use the following command -

s = regionprops(BW, 'centroid');

Web Link: http://www.mathworks.in/help/toolbox/images/ref/regionprops.html

bwlabel()

[L, num] = bwlabel(BW, n)

It returns a matrix L, of the same size as BW, containing labels for the

connected objects in BW. The variable n can have a value of either 4

or 8, where 4 specifies 4-connected objects and 8 specifies 8-

connected objects. If the argument is omitted, it defaults to 8.

http://www.mathworks.in/help/toolbox/images/ref/regionprops.html



[L, num] = bwlabel(BW, n)

It returns in num the number of connected objects found in BW.

Web Link: http://www.mathworks.in/help/toolbox/images/ref/bwlabel.html

imcrop()

I = imcrop creates an interactive Crop Image tool associated with the

image displayed in the current figure, called the target image. The

Crop Image tool is a moveable, resizable rectangle that you can

position interactively using the mouse.

I2 = imcrop(I) displays the image I in a figure window and creates a

cropping tool associated with that image. I can be a grayscale image,

a truecolor image, or a logical array. The cropped image returned, I2,

is of the same type as I.

Web Link: http://www.mathworks.in/help/toolbox/images/ref/imcrop.html

bwboundaries()

B = bwboundaries(BW) traces the exterior boundaries of objects, as

well as boundaries of holes inside these objects, in the binary

image BW.

B = bwboundaries(BW,conn) specifies the connectivity to use when

tracing parent and child boundaries. conn can have either of the

following scalar values.

Web Link: http://www.mathworks.in/help/toolbox/images/ref/bwboundaries.html

http://www.mathworks.in/help/toolbox/images/ref/bwlabel.html

http://www.mathworks.in/help/toolbox/images/ref/imcrop.html

http://www.mathworks.in/help/toolbox/images/ref/bwboundaries.html



USING THE PORT

Now we have a scenario where we have to interface the computer with

the microcontroller.

When an image is processed, we have a signal for performing some

kind of work. This signal needs to reach the uC (short for

microcontroller), which is done by the port.

We will demonstrate Serial and Parallel ports in our tutorial.

PARALLEL PORT SERIAL PORT

In laptops/computers of today, we generally do not have parallel

ports, and some modern laptops (like mine!) do not have Serial port

also. But they do have multiple USB (Universal Serial BUS) ports.

So we use USB to Parallel, or USB to Serial port converters –

USB to Serial Port USB to Parallel Port



Parallel Port

The parallel port has 25 pins, which are numbered as follows –

The pinouts for a Parallel port are –

DB25 pinout



Before you being working with parallel port, do check if the parallel

port in your system is operational, by following these steps –

My Computer>System Properties>Device Manager>Ports

You will find the name of your parallel port from here.

Let us assume it is – „LPT1‟

>> parport = digitalio (‘parallel’, ‘LPT1’);

We have now defined an object named „parport‟.

Port address may be obtained using these statements –

>> get (parport, ‘PortAddress’)

>> daqhwinfo (‘parallel’);

Now we need to define PINS 2-9 as OUTPUT pins using the addline()

function –

>> addline(parport, 0:7, ‘out’)

Now, depending upon the motion of the robot desired, you can put the

data in a matrix using logical() function –

>> dout = logical([1 0 0 1 0 0 0 1]);

Using the putvalue() function, we can output this value –

>>putvalue(parport, dout);

We can also send the decimal (or octal/hexadecimal) value directly –

>>dat = 259;

>>putvalue(parport, dat);

While using the parallel port, we do not need any microcontroller, as

outputs can be directly received by the motors from the parallel port

using H-bridge (L293D or L298 IC – Motor driver ICs) for driving the

motor in both directions – clockwise and anticlockwise.



Serial Port

A Serial port has 9 pins, which are numbered as follows –

Serial transfer means that if you want to transfer a byte of

information, then Serial port will transfer the 8 bits bit-by-bit (one bit

at a time). As seen from the above diagram, the transmission happens

from PIN3 (Tx), and receiving happens at PIN2 (Rx).

Now, we would require a microcontroller with UART (Universal

Asynchronous Receiver and Transmitter) to be present.

Most of the uC that we use in Robotics (like ATMEGA 16L, PIC, etc)

have UART, which needs to be initialized for Serial data transfer.

The standard used for Serial communication is RS-232 (Recommended

Standard 232). This defines the voltage values that would be

equivalent to logic ZERO and logic ONE. Valid signals are positive and

negative 3 to 15 volts.



Now the problem that arises is that our microcontroller works on TTL

standards, and hence would not work on RS-232. So, here we

introduce IC MAX 232, which would help us in this situation.

The connection diagram of MAX 232 with Serial port, for our purpose is

given is follows –

PINs 11 and 12 can now be used for communication via pins of uC

meant for Serial transfer of data.

Before you start using the Serial Port, check in the Device Manager

(Port section), whether you can see COM ports available. If yes, then

your Serial port is ready for some action!

Let us assume your Serial port device name to be „COM1‟ –

We can create a MATLAB object using the following commands –

>> ser = serial (‘COM1’, ‘BaudRate’, 9600);

with Baud Rate = 9600 bps (bits per second)



Baud Rate - In telecommunications and electronics, baud is

synonymous to symbols per second or pulses per second.

While setting this value in the above command, we must make sure

that the value is the same as set while configuring the microcontroller

on-board the robot.

The Serial data format includes –

1 Start bit

5 to 8 Data bits

1 Stop bit

Parity bit (may be there)

Additional Stop bit (may be there)

Asynchronous transmission using the UART character

So the data that we are sending must be of 8bits.

Let us suppose that we keep codes in hexadecimal –

„A‟ – Forward

„B‟ – Backward

„C‟ – Turn left

„D‟ – Turn right

No we open the Serial port and start sending data through it –

>> fopen (ser);

>> fwrite (ser, ‘A’);

>> fwrite (ser, ‘C’);

Signal for moving the robot one unit FORWARD, then turning LEFT.



This data has to be sent to the microcontroller. So now we need to

program the microcontroller to „listen‟ to the COM1 port.

Let us take ATMEGA16L as our microcontroller.

We would generate the program to be burnt, using CodeVision AVR

software.

A typical window of CVAVR looks like this –

We need to go to File>New

This appears –

Select „Project‟ and click on OK.

From the new menu, select the first option, and click on OK



Now comes the most important part – configuring the microcontroller

for use.

The window that opens shows several tabs on the left –

Select „ATmega16L‟ (or any chip you are using) from the „Chip‟ tab.

We now need to configure the microcontroller for using its USART in

asynchronous mode, and along with that set the PORTs of the uC

which would be used for signaling the motors.



Select the „USART‟ tab from the same window –

Make the necessary settings in this submenu – putting a tick on the

Transmitter option, and Receiver also if it is being used in your case.

The Baud Rate is by default set to 9600. Notice that this is the same

as the rate set by us while defining the Serial port object in MATLAB.

Decide and set the communication parameters, and set the Mode as

„asynchronous‟.

PORTs can be set using the „Ports‟ tab –



Once, all settings are done, Click on -

Program>Generate, Save and Exit

Write the same names in the „Save‟ windows that come.

The program now appears in front of you with all configuration settings

done.

Browse down the window with the program code to find this –

You can now write your code here for accepting data from Serial port,

checking it across a set of operations (like moving left, forward,

blinking LED, etc), and perform the required action using the

respective PORTs that you already have set as input/output.

Output from here would now drive the motors.



MOTOR CONTROL

Now all the data acquisition and processing is complete. All we need to

do is to RESPOND to the instructions that the microcontroller gives.

We would do this by interfacing the Output PORT of the microcontroller

with the motors.

The problem that arises is – the output potential of the port of

microcontroller is not enough to drive a motor, hence, we need to use

a Motor driver IC (L293D, L298) for doing the same.

We would use these with both DC motors and Stepper motors as given

below.

DC Motors

As already mentioned above, the DC motor would require an IC for

working with the microcontroller output. We can use L293D for this

purpose –



Pin diagram of L293D

Each L293D has four Input-Output pairs. For making a motor run both

directions, we need to use two Input-Output pairs per motor.

Hence, we can use ONE L293D for controlling two DC motors –



Shown above is the connection diagram of the two DC motors with one

L293D IC.

Vs-pin stands for Motor Voltage and is the voltage with which the

motors would be driven. We can use a 12V DC connection on this pin.

Enable pins must be set HIGH for enabling the working of both motors.

These pins can also be used to DEACTIVATE the motor functionality

under certain circumstances as directed by the algorithm.

For various INPUT conditions at IN1, IN2 and IN3, IN4, we see the

following pattern per motor -

DC motors are simple, but they lack accuracy. In case we want to use

DC motors, we have to use an efficient feedback mechanism to

overcome and compensate the loss of accuracy.



Stepper Motors

It is a brushless, electric motor that can divide a full rotation into a

large number of steps. The motor's position can be

controlled precisely without any feedback mechanism.

How it works?

I II III IV

Hence, a complete rotation happens after four such steps.

The angle which the Stepper motor rotates with after completion of

any step above is called the Resolution of the Stepper motor.

Steppers are available in many Resolutions, ranging from 30 to 1.8

degrees per step.

Stepper motor can be either Unipolar or Bipolar.



As you can see, each stepper has four coils. Hence we have four

connections per stepper motor.

So, one L293D can drive one bipolar 2 phase stepper motor.

The connection diagram of the same is given as follows –

We have the four Controller Pins going to the OUTPUT port of the

microcontroller. And the data coming through these pins will be

controlling the operation of the Stepper.

We saw in a diagram above that there is a specific sequence in which

coils should get activated, so as to ensure proper completion of a

cycle.

The pulse sequence has to be in the following order –

Coil 1A Coil 2A Coil 1B Coil 2B

Step 1 High High Low Low

Step 2 Low High High Low

Step 3 Low Low High High

Step 4 High Low Low High

*For reversing the direction, follow the steps in opposite order.



Unipolar Stepper Motors

A Unipolar stepper motor has one winding with center tap per phase.

The special thing about Unipolar Steppers is that we can reverse the

direction of rotation of the motor without changing the direction of

current.

For Unipolar Stepper, we use IC ULN2003/2004 instead of L293D, as it

also has a COM port for serving as the Common connection of the

coils.



Thank you for going through the tutorial.

If you have any queries or suggestions that you wish to share, feel free

to contact me at -

[email protected]

[email protected]

mailto:[email protected]

mailto:[email protected]

Image Processing Based Robotics.pdf

Documents

Transcript of Image Processing Based Robotics.pdf