SENSOR FUSION LABORATORY

Thad Roppel, Associate Professor AU Electrical and Computer Engineering Dept.

troppel@eng.auburn.edu

EXAMPLES

• Infrared / Millimeter wave radar for vehicle detection and identification

• Chemical sensor arrays – “artificial nose”

• Biomimetics – imitating animal sensorimotor behaviors

• Biomedical – using electrical and optical probes to study cardiac arrhythmias

MISSION: Study the benefits of using simultaneous information from multiple sensors to probe the environment.

SENSOR FUSION LABORATORY

Problem Complexity: Human vs. Machine

HUMAN

MA

CH

INE

EASY HARD

EASY

HARD Maximum

Potential Benefit

• Object recognition• Linguistics• Extraction of Relevant

Features from Sensor Arrays

• Arithmetic• Logic

• Thresholding• Tallying

• Judging

Personnel and PublicationsPERSONNEL

•Ting-To Lo (PhD): Molecular Switching in Biosensors

•Rama Narendran (PhD): Biomimetic Simulations of Organized Machine Behavior

•Jun Pan (PhD): Wireless Protocol for Electrical and Optical Cardiac Microprobes

•Aroldo Couto (MS): Flight Stabilization Using Adaptive Artificial Neural Networks

•Brian Wingfield (MS): Silicon Processing for Lateral Emission Fiber-Optic SensorsREPRESENTATIVE RECENT PUBLICATIONS

• D. M. Wilson, T. Roppel, and R. Kalim, "Aggregation of Sensory Input for Robust Performance in Chemical Sensing Microsystems," Sensors and Actuators B, 64(1–3), 107-117, June 2000.

• T. Roppel and D. M. Wilson, "Biologically-Inspired Pattern Recognition for Odor Detection," Pattern Recognition Letters, 21(3), 213–219, March 2000.

• D. M. Wilson, K. Dunman, T. Roppel, and R. Kalim, "Rank Extraction in Tin-Oxide Sensor Arrays," Sensors and Actuators B, 62(3), 199-210, April 2000.

• T. Roppel, R. Kalim, and D. Wilson, "Sensory Plane Analog-VLSI for Interfacing Sensor Arrays to Neural Networks, " Virtual Intelligence and Dynamic Neural Networks VI-DYNN '98, Stockholm, Sweden, June 22-26, 1998.

IR / MMW DATA FUSIONSupport: AFOSR 1992-93

Project Goal: Improved identification of military vehicles from aerial scenes.

LANCE Missile Launcher

T-62 Tank

M-113 Armored Personnel Carrier (APC)

IR / MMW Fusion, cont’d

APPROACH:

IR SCENE PIXELS

MMW RADAR DATA

NEURAL NETWORK

APCTANKLAUNCHER

PERFORMANCE ASSESSMENT: A T L

A + - -

T - + -

L - - +

•Multiple permutations

•Confusion matrix

•Average result

OVERALL RESULT: 14 % improvement with sensor fusion

Chemical Sensor Arrays

Support: DARPA 1997-99

PROJECT GOAL: Improved identification and detection of chemical plumes in non-laboratory conditions.

VEHICLE

SENSORS

PLUME COMMANDSTATION

RF LINK

ROAD

WIND

Canine Training at IBDSAuburn is world-renowned for training of detection dogs at the Institute for Biological Detection Systems.

Chemical Sensor Arrays, cont’d

Odor Sensor Array

0 100 200 300 400 5000

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Timestep

Sen

sor

Vol

tage

Sensor Outputs

Sensor Array Dynamic Response

Chemical Sensor Arrays, cont’d

0 100 200 300 400 5000

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Timestep

Sen

sor

Vol

tage

10 20 30 40 50

2

4

6

8

10

12

14

Sen

sor

Num

ber

Timestep

Sensors 1-15

Raw Output Thresholded Binary Output

Above ThresholdBelow ThresholdPreprocessing

Chemical Sensor Arrays, cont’d

ace

Sample 1 Sample 2 1

20

Sample 3 1

20

amm

dal

g87

g89

g93

oil

pth

Sensor #

xyl

5 10 15Sensor #5 10 15

Sensor #5 10 15

Chemical Sensor Arrays, cont’d

input categories

net

wo

rk r

esp

on

se 1 timestep

aceammdalg87g89g93oilpthxyl

5 timesteps 10 timestepsn

etw

ork

res

po

nse 20 timesteps

aceammdalg87g89g93oilpthxyl

50 timesteps Ideal Response

Time Evolution of Confusion Matrix: Forward SequenceTrained for 20 timesteps

00.10.20.30.40.50.60.70.80.91

Chemical Sensor Arrays, cont’d

00.10.20.30.40.50.60.70.80.91

Time Evolution of Confusion Matrix: Random SequenceTrained for 20 timesteps

1 timestep 5 timesteps 10 timesteps

20 timesteps 50 timesteps Ideal Response

net

wo

rk r

esp

on

se

aceammdalg87g89g93oilpthxyl

net

wo

rk r

esp

on

se

aceammdalg87g89g93oilpthxyl

input categories

Chemical Sensor Arrays - Summary

A recurrent neural network was trained to recognize 9 odors presented in an arbitrary time sequence.

Response time is reduced by an order of magnitude by threshold preprocessing.

Well-suited for use as a front-end for a hierarchical suite of NN’s in a portable, near-real time odor classification device.

BIOMIMETICSSupport: Under discussion with AF Advanced Guidance Division, Munitions Directorate at Eglin AFB

PROJECT GOAL: Learn sensor fusion from animals. Apply this to flying a drone to target using onboard video.

Flies land accurately

Bees find flowers

Bats catch evading insects in flight

BIOMIMETICS, cont’d

What do they “know” that we don’t?

One possibility is that they use variations of optic flow.

Represent sensory image field by motion vector field.

Image Sequence

Optic Flow Field

BIOMIMETICS, cont’d

EXAMPLES

A fly can land simply by maintaining constant optic flow.

A dog can track by maintaining constant sensory flow across olfactory epithelium and following the gradient (using sniffing as a form of “chopper amplifier.”

Questions to be answered: Can we guide a missile to target, orchestrate complex defense systems, identify faces in a crowd, or track contaminated food with similar approaches?

END OF PRESENTATION