EN94941 Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate.

EN9494 1

Follow The Leader Using Vision OnlyBax Smith, BSc., BEng., MSc. Candidate

EN9494 2

Introduction

The purpose of this presentation is to introduce and give some background information on my project for this course

EN9494 3

Topics of Discussion

Project Description Project Background The Guts of the Project Current State of Things Literature Review Input From Audience

EN9494 4

Project Description

Implementation of a Real-Time follow-the-leader vehicle

Two vehicles: one leads, one follows

Has many of the elements for this course: real-time, discrete-event, continuous dynamics, multi-agent, hierarchical and time-critical

EN9494 5

Project Background – Why this project? A subset of my Masters Project:

Follow-the-Leader Using Vision under different weather conditions

Has aspects of Image Processing, A.I. and Control

It’s Cool!

EN9494 6

Project Background - Applications My purpose: Mining Vehicles Automated Highways – Smart Cars Automatic Target Recognition –

Like they use in Smart Missiles!! Security Systems – Identify the

intruder The Ultimate Paint-Ball Weapon!!

EN9494 7

The Guts of the Project

Hardware: Jagan Wagons, PIII 600 Laptop, Webcam

Software: Win2000, VC++6.0 Program Cycle

Grab an image from Webcam – Small Colored! Use LVQ Neural Net to isolate vehicle from other

objects using color Use Min Bounding Box to scale to classifier input size Use Support Vector Machines (SVM) to recognize

object – NECEC 2001 Plug! Call appropriate motor commands – Maybe PID, but I

doubt it

EN9494 8

Current State of Things

Term 8 Project was similar…sort of. Have developed/implemented LVQ

Nets for object detection based on color – worked very well

Have developed/implemented SVMs for 3D Object Recognition – NECEC 2001 Plug #2

EN9494 9

Term 8 Project

EN9494 10

LVQ Neural Net

Input

Map toSubclass

Map toClass Output

R

G

B

dR

mR

lR

dG

mG

lG

dB

mB

lB

Y

R

G

B

Y

EN9494 11

Why LVQ?

Each color is a 3D vector ie RGB All Reds are not the same LVQ stores general direction of a color

6080

100120

140160

50

100

150

20060

80

100

120

140

160

180

EN9494 12

Training

Extract color from sample images of environment

EN9494 13

Support Vector Machines

EN9494 14

Literature Review

General http://www-2.cs.cmu.edu/~cil/v-pubs.html http://www.ece.ubc.ca/~irenek/techpaps/

introip/title.html http://www.itsonline.com/index.htm

Neural Nets http://www.maths.uwa.edu.au/~rkealley/

ann_all/ann_all.html Support Vector Machines

http://www.kernel-machines.org/

EN9494 15

Input From Audience

Suggestions Comments Critisms

EN9494 16


Part II – Midterm Progress

EN9494 17

Review of Project




objects using color Use Min Bounding Box to scale to classifier input

size Use Support Vector Machines (SVM) to recognize

object Call appropriate motor commands

EN9494 18

The Setup

EN9494 19

Automatic Target Recognition

ImagingSensors

ImagePreprocessing

ObjectSegmentation

FeatureExtraction

ObjectClassification

EN9494 20

Imaging Sensor

ImagingSensors

ImagePreprocessing

ObjectSegmentation

FeatureExtraction


CCD TV Low-light Level TV (LLTV) Forward Looking Infrared (FLIR) Synthetic Aperture Radar (SAR) Moving Target Indicator Radar (MTI Radar) Laser Radar (LADAR)

EN9494 21

Webcam Pros:

Cheap Easy to Use Free Software!!!!!

Cons: Low Resolution Slow Software HARD to Use/Buggy

EN9494 22

Webcams - Software

OpenCV Video For Windows (VFW) API Ton of Wrappers for VFW Microsoft Vision SDK

Easy to use Low Level – Made for Realtime Apps

EN9494 23

Object SegmentationImagingSensors

ImagePreprocessing

ObjectSegmentation

FeatureExtraction


Preprocessing: Focus Contrast Enhancement

Detection/Segmentation: Foreground/Background Separation Silhouetting

EN9494 24

LVQ Neural Network

EN9494 25

Inside LVQ Net

EN9494 26

LVQ – Competitive Layer

6080

100120

140160

50

100

150

20060

80

100

120

140

160

180

EN9494 27

LVQ Competitive Layer - Training CL = - || w – p || If p classified correctly

w(q) = w(q-1) + a( p(q) – w(q-1) ) Else

w(q) = w(q-1) - a( p(q) – w(q-1) )

6080

100120

140160

50

100

150

20060

80

100

120

140

160

180

w1 w2

w3

EN9494 28

LVQ – Linear Layer

6080

100120

140160

50

100

150

20060

80

100

120

140

160

180

Non-Target

Target

EN9494 29

After LVQTarget

Non-Target

EN9494 30

Feature ExtractionImagingSensors

ImagePreprocessing

ObjectSegmentation

FeatureExtraction


Transforms the segmented image into a form that can be used as input to classifier – Feature Vector

EN9494 31

Minimum Bounding Box

EN9494 32

Object ClassificationImagingSensors

ImagePreprocessing

ObjectSegmentation

FeatureExtraction


Segmented Objects are classified based on their associated feature vector

Object Classification

EN9494 33

Support Vector Machines

Optimal Separating Hyperplane for linearly separable problems

Extending OSH to Nonlinear Using Kernels

EN9494 34

Optimal Separating Hyperplane

EN9494 35

Optimal Separating Hyperplane

EN9494 36

OSH - Mathematics

Given: (x1,y1) … (xn, yn) | xi is vector of

reals, y = {-1, +1} Construct Hyperplanes:

H1 : <w,x> + b = +1

H2 : <w,x> + b = -1 OSH: <w,x> + b = 0

EN9494 37

Support Vectors

EN9494 38

Formulating Constraints

From: H1 : <w,x> + b = +1

H2 : <w,x> + b = -1

We get: yi(<w,xi> + b) >= 1

EN9494 39

Distance Between Hyperplanes

Just subtract H2 from H1: <w,(x1-x2)> = 2

And normalize: <w/||w||, (x1 – x2)> = 2/||w||

EN9494 40

Bigger Margin = Better Classifier

Therefore: Maximize: 2/||w|| Constrained by: yi(<w,xi> + b) >= 1

Or equivalently: Minimize: ½ ||w||2

Constrained by: yi(<w,xi> + b) >= 1

EN9494 41

QP Problem - Lagrange

m

iii

m

iiiP bxwywL

11

2)(

2

1

Saddle Point Minimize wrt w and b Maximize wrt alphas

At Saddle Point dL/dw, dL/db = 0:

m

iiii xyw

1

m

iii y

1

0

m

iii y

1

0

EN9494 42

The Dual Problem

Substituting in the derivatives:

Maximize, subject to: alphas>0 and

Solve for alphas

m

ijijijiiD xxyyL

1

)(2

1

m

iii y

1

0

EN9494 43

Only Support Vectors Matter

Lagrange also gives us:

Which shows that only the data points that lie on the outer hyperplanes will have non-zero lagrange multipliers

01 bxwy iii

EN9494 44

Solution to QP

Classification of vector, v, is the sign of:

vxyb iii

EN9494 45

Extending OSH to SVM

OSH works for linearly separable data therefore transform data into a space where it is linearly separable

EN9494 46

Using a Kernel Function

Kernel Functions are easily added to the Lagrangian:

And classification function

jijijiiD xxyyL 2

1

vxyb iii

EN9494 47

Don’t Need to Compute Transform

We just need the Kernel Function:

For example: x = (x1, x2); z = (z1, z2);

<x,z>2 = (x1z1 + x2z2)2 = x12z12 + x22z22 +2x1z1x2z2 = <(x12,x22,sqrt(2)x1x2),(z12,z22,sqrt(2)z1z2)> = <phi(x), phi(z)>

yxyxK ,

EN9494 48

Examples of Kernels

K(x,z) = <x,z>d

K(x,z) = exp(-||x-z||2/2)

EN9494 49

Wheelchair Control

Send drive commands to Wheelchair

EN9494 50


Episode III – The Final Presentation

EN9494 51

Today

Quick Review Demo Setup Agents Supervisory Controller

EN9494 52

Review of Project




objects using color Use Min Bounding Box to scale to classifier input

size Use Support Vector Machines (SVM) to recognize

object Call appropriate motor commands

EN9494 53

The Setup

Supervisor

EN9494 54

The Agents

Robot1

Robot2

EN9494 55

Agents in MATLAB function state = getRobotState(state, sl, dsl, sr, dsr) idle = 1; fol_leader = 2; fol_r = 3;

switch state case idle, if sl == 1, state = fol_leader; end; case fol_leader, if sr == 1, state = fol_r; elseif dsl == 1, state = idle; end; case fol_r, if dsr == 1, state = idle; end; end;

EN9494 56

Object SegmentationImagingSensors

ImagePreprocessing

ObjectSegmentation

FeatureExtraction


Preprocessing: Focus Contrast Enhancement

Detection/Segmentation: Foreground/Background Separation Silhouetting

EN9494 57

Segmentation via LVQ

EN9494 58

Object ClassificationImagingSensors

ImagePreprocessing

ObjectSegmentation

FeatureExtraction


Segmented Objects are classified based on their associated feature vector

Object Classification

EN9494 59

Classification via SVM

EN9494 60

Creating the Plant

Sync

Robot1 Robot2

EN9494 61

Robert Plant

EN9494 62

Creating the Spec

EN9494 63

Creating the Supervisor

Sync

Plant Spec

EN9494 64

The Supervisor

EN9494 65

CONTROL.TXT c1

Control data are displayed by listing the plant state followed by the supervisor state (if different from the plant state) where disabling and/or forcing occur, together with the events which must be disabled or forced there.

Control Data:

PLANT: [fol_leader1,fol_r1] SUPER: [[fol_leader1,fol_r1],1] DELAY: sr2 PLANT: [fol_r2,fol_leader2] SUPER: [[fol_r2,fol_leader2],1] DELAY: sr1 PLANT: [fol_leader1,idle_r2] SUPER: [[fol_leader1,idle_r2],1] DELAY: sl2 PLANT: [idle_r1,fol_leader2] SUPER: [[idle_r1,fol_leader2],1] DELAY: sl1

c1 printed.

EN9494 66

Supervisor in MATLAB function [sl1, dsl1, sr2, dsr2, sl2, dsl2, sr1, dsr1] =

runSupervisor(stateOfR1, sl1, dsl1, sr2, dsr2, stateOfR2, sl2, dsl2, sr1, dsr1)

idle = 1; fol_leader = 2; fol_r = 3; if (stateOfR1 == fol_leader) & (stateOfR2 == idle), sl2 = 0; dsl2 = 1; end; if (stateOfR1 == idle) & (stateOfR2 == fol_leader), sl1 = 0; dsl1 = 1; end; if (stateOfR1 == fol_leader) & (stateOfR2 == fol_r), sr2 = 0; dsr2 = 1; end; if (stateOfR1 == fol_r) & (stateOfR2 == fol_leader), sr1 = 0; dsr1 = 1; end;

EN9494 67

Demos

W/O Supervisory Control With Supervisory Control

EN94941 Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate.

Documents

Transcript of EN94941 Follow The Leader Using Vision Only Bax Smith, BSc., BEng., MSc. Candidate.