Neuromorphic Object Detection Recognition and Tracking

Fopefolu FolowoseleTELL – Research Overview

September 17, 2009

Outline

Introduction to Neuromorphic Engineering Computational Sensory Motor Systems Lab Research Focus Approach Neural Array Transceivers HMAX Model of Object Recognition Neural Algorithms Conclusion

Neuromorphic Engineering

Neuromorphic was coined by Carver Mead to describe VLSI systems containing circuits that mimic neuro-biological architectures present in the nervous system

Neuromorphic Engineering involves designing artificial neural systems whose physical architecture and design principles are based on those of biological nervous systems

Computational Sensory Motor Systems Laboratory

Research Focus Object detection, recognition and

tracking are computationally difficult tasks

Primates excel at these tasks Engineered systems are unable to

match their level of proficiency, flexibility and speed

Intelligent robots need to be able to interact with their surroundings with limited human involvement

This interaction involves:› Detecting the presence of the object› Recognizing the object› Tracking the trajectory of the object

Hyperexperience 2008

Agile Systems, 2009

Approach Our overall goal is to work towards developing a real-time

autonomous intelligent system that can detect, recognize and track objects under various viewing conditions

Emulate cortical functions of primates to design more intelligent artificial systems› Mimic the visual information processing of the primate’s visual

system› Model computationally-intensive algorithms in neural hardware

Population Surveillance and Visual Search Engines

Visual Prosthesis and Ocular Implants

Research Tool for Neuroscientists

Techarena 2009; Future Predictions 2008; R. Friendman, Biomedical Computation Review 2009

Projected Contributions Develop a spike-based processing platform on which we

can demonstrate object detection, recognition and tracking› Design the next generation neural array transceiver › Realize silicon facsimiles of cortical simple cells, complex cells

and composite feature cells› Implement neural algorithms analogous to cross-correlation and

Kalman filtering for object detection and tracking respectively

Software vs. Hardware Models

Software models run slower than real time

and are unable to interact with the

environment

Silicon designs take a few months to be fabricated,

after which they are constrained by limited

flexibilityIBM 2004; Tenore 2008

Solution Reconfigurable Models Neural array transceivers are

reconfigurable systems consisting of large arrays of silicon neurons

Useful for studying real-time operations of cortical, large-scale neural networks› Able to leverage the known

fundamental blocks such as the operation of neurons and synapses

› Flexible enough for testing out unknowns

Neuro-Computational Spiking and Bursting Models

Izhikevich’s model seems most appropriate but› Result of highly nonlinear curve fitting offering little insight into the

underlying biological mechanism› Interference between the state variables› Parameters are 4-5 orders of magnitude apart

E.M. Izhikevich, Neural Networks, 2004

Mihalas-Niebur Neuron Model› Generalized version of the leaky integrate-and-fire model with

adaptive threshold› More biologically relevant

Suggested modifications to the threshold interpreted as nonlinear voltage dependent channels

Mihalas-Niebur Neuron Simulations

Circuit MatLab

F. Folowosele et al., ISCAS, 2009

3D Design

In collaboration with the Sensory Communication and Microsystems Lab

Visual Pathways Primary Visual Cortex V1

transmits information to two primary pathways› Dorsal stream› Ventral stream

Dorsal pathway is associated with motion

Ventral pathway mediates the visual identification of objects

T. Poggio, NIPS, 2007Wikipedia, The Free Encyclopedia

HMAX Summarizes and

integrates large amount of data from different levels of understanding (from biophysics to physiology to behavior)

Two main operations occur in the model› Gaussian-like tuning

operation in the S layers› Nonlinear MAX-like

operation in the C layers

M. Riesenhuber & T. Poggio, Nature Neuroscience 1999

Preliminary Results: S1 and C1 Stages S1 neurons are oriented spatial filters that detect local

changes in contrast S1 cell integrates inputs from a 4x1 retinal receptive field C1 neurons take the MAX of similarly-oriented simple cells

over a region of space C1 cell integrates inputs from an array of 5x5 similarly-

oriented S1 cells

F. Folowosele et al., BioCAS 2008

MAX Operation Nonlinear saturating pooling function on a set of inputs, such

that the output codes the amplitude of the largest input regardless of the strength and number of the other inputs

Set of input neurons {X} causes the output Z to generate spikes at a rate proportional to the input with the fastest firing rate

R.J. Vogelstein et. al, NIPS 2007

Test1: Test Images and Resulting Simple Cells

(A1-4) Generated test images

(B1-4) Horizontally-oriented simple cells that respond to light-to-dark transitions

(C1-4) Vertically-oriented simple cells that respond to dark-to-light transitions

F. Folowosele et al., ISCAS 2007

Test 1: MAX Network Computation Results

The ratio k obtained is approximately constant among all the simple cells, with a mean of 0.068 and a standard deviation of 0.0006

F. Folowosele et al., ISCAS 2007

Neural Algorithms

In computer vision, object detection and tracking algorithms are computationally-intensive processes› Cross-correlation for object and pattern detection› Kalman and particle filtering for object tracking

Neural-based algorithms are potentially more flexible and less computationally-intensive than their traditional counterparts

Object Detection Computation of cross correlation is utilized for

pattern and object detection Basis for neural cross-correlation is the

autocorrelative nature of the interspike interval histogram (ISIH) in spiking neurons› First proposed in 1951 (Licklider 1951) › ISIH for the auditory nerve ensemble response has

same shape as the autocorrelation function (Cariani 1996)

› Autocorrelation observed in accumulated output from a single integrate-and-fire neuron (Tapson 1998)

Neural Cross-Correlation The neural cross-correlation engine was proposed by

Jonathan Tapson in 2007 It utilizes integrate-and-fire neurons to produce cross-

correlation information in a novel way

J. Tapson & R. Etienne-Cummings ISCAS 2007

MatLab Simulation Results

ISIH as a proxy for correlation

Mathematical Computation

Interspike Interval (ISIH)

F. Folowosele et al., SPIE 2007

MatLab Simulation Results

Extracting the phase information of a signal

Mathematical Computation

Interspike Interval (ISIH)

F. Folowosele et al., SPIE 2007

Object Tracking Optimal solution to tracking tasks is Kalman filtering Core function of cerebral cortex hypothesized to

involve some mechanism of Kalman filtering Kalman filtering type algorithm used in hierarchical

generative model for visual recognition (Rao & Ballard 1997)

At each hierarchical level › Predicts current visual state at a lower level› Adapts own recognition state using the residual error

between the prediction and the actual lower-level state

Kalman Filtering Models natural processes in the external world

using a stochastic linear differential equation External system is described by a state vector

Each measurement vector satisfies

Organisms do not have access to the internal states of the world causing their sensory experiences

R. Rao, Uni. Of Rochester 1996T. Lacey, Georgia Tech. 1998

Neural Kalman Algorithm

Ralph Linsker’s Algorithm› Utilizes recurrent neural network composed of

linear-response nodes› Requires noisy measurement data as only input

Classical Kalman Approach› Estimate state vector

Neural Kalman Approach› Estimate measurement vector

Simple Tracking Example: 2D Plant State

0 10 20 30 40 50 60 70 80 90 100-2

-1.5

-1

-0.5

0

0.5

1

1.5Measurement Plots

Measurement

NeuralClassical

0 10 20 30 40 50 60 70 80 90 100-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Measurement

NeuralClassical

Conclusion

We are designing a neural array transceiver on which we intend to implement › Stages of the HMAX hierarchical model of object

recognition› Neural algorithms for object detection and

tracking Our overall goal is to work towards developing

a real-time autonomous intelligent system with an artificial visual cortex

Acknowledgments Prof. Ralph Etienne-Cummings Computational Sensory Motor Systems Lab Members Sensory Communications and Microsystems Lab

Members Collaborators

› Jonathan Tapson (University of Cape Town)› Tara Hamilton (University of Queensland)› Ernst Niebur & Stefan Mihalas (JHU Mind-Brain Institute)

UNCF-Merck

References R.R. Murphy and E. Rogers, “Cooperative assistance for remote robot supervision,” Presence: Teleoperators and Virtual Environments Journal, vol. 5,

no. 2, pp. 224-240, 1996. T. Serre, M. Kouh, C. Cadieu, U. Knoblich, G. Kreiman, and T. Poggio, “A theory of object recognition: computations and circuits in the feedforward

path of the ventral stream in primate visual cortex,” AI Memo, MIT, Cambridge 2005. M. Riesenhuber, and T. Poggio, “Computational models of object recognition in cortex: a review,” Technical Report Artificial Intelligence Laboratory

and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 2000b. R.J. Vogelstein, U. Mallik, E. Culurciello, G. Cauwenberghs, R. Etienne-Cummings, “A multichip neuromorphic system for spike-based visual

information processing,” Neural Computation, vol. 19, pp. 2281-2300, 2007a. D.H. Goldberg, G. Cauwenberghs, and A.G. Andreou, “Probabilistic synaptic weighting in a reconfigurable network of VLSI integrate-and-fire

neurons,” Neural Networks, vol. 14, pp. 781-793, 2001. T.Y.W. Choi, P.A. Merolla, J.V. Arthur, K.A. Boahen, and B.E. Shi, “Neuromorphic implementation of orientation hypercolumns,” IEEE ISCAS 2005. R.J. Vogelstein, U. Mallik, J.T. Vogelstein, G. Cauwenberghs, “Dynamically reconfigurable silicon array of spiking neurons with conductance-based

synapses,” IEEE Transactions on Neural Networks, 2007b. A. Cassidy, S. Denham, P. Kanold, and A.G. Andreou, “FPGA-based silicon spiking neural array,” IEEE BioCAS 2007. B. E. Shi, E. K. C. Tsang, S. Y. M. Lam and Y. Meng, "Expandable hardware for computing cortical maps," IEEE ISCAS 2006. D.H. Hubel and T.N. Wiesel, “Receptive fields, binocular interaction and functional architecture in the cat's visual cortex,” Journal of Physiology, vol.

160, no. 1, 1962. L.G. Ungerleider, and J.V. Haxby, “What and where in the human brain,” Curr. Opin. Neurobiol., pp. 157-165, 1994. E. Rolls and T. Milward, “A model of invariant object recognition in the visual system: Learning rules, activation functions, lateral inhibition, and

information-based performance measures, Neural Computation, vol. 12, pp. 2547-2572, 2000. P Merolla and K Boahen, “A recurrent model of orientation maps with simple and complex cells,” Advances in Neural Information Processing Systems

(NIPS) 16, S Thrun and L Saul, Eds, MIT Press, pp 995-1002, 2004. R.P.N. Rao, “Robut Kalman filters for prediction, recognition, and learning,” Technical Report 645, Computer Science Department, University of

Rochester, 1996. J. Licklider, “A duplex theory of pitch perception,” Cellular and Molecular Life Sciences (CMLS), vol. 7, no. 4, pp. 128-134, 1951 J. Tapson, “Autocorrelation properties of single neurons,” Proceedings of the 1998 South African Symposium on Communication and Signal

Processing, 1998. J. Tapson, C. Jin, A. van Schaik and R. Etienne-Cummings, “A First-Order Nonhomogeneous Markov Model for the Response of Spiking Neurons

Stimulated by Small Phase-Continuous Signals,” Neural Computation, vol. 21, no. 6, pp. 1554-1588, June 2009. T. Lacey, “Tutorial: The Kalman filter,” Lecure Notes, Department of Computer Science, Georgia Institute of Technology, 1998. R. Linsker, “Neural network learning of optimal Kalman prediction and control,” Neural Networks, vol. 21, no. 9, pp. 1328-1343, 2008. R.E. Kalman, “A new approach to linear filtering and prediction problems,” Transactions of the ASME–Journal of Basic Engineering (Series D), pp. 35-

45, 1960. S. Mihalas and E. Niebur, “A generalized linear integrate-and-fire neural model produces diverse spiking behaviors,” Neural Computation, 2008 in Press.

C. Cadieu, M. Kouh, A. Pasupathy, C.E. Connor, M. Riesenhuber, T. Poggio, “A model of V4 shape selectivity and invariance,” J. Neurophysiol., 2007.

Figure References Self-driving Car:

http://hyperexperience.com/wp-content/uploads/2008/01/nissanpivot.jpg Space-station Builder: http://www.agilesystems.com/images/happy%20space

%20rigger2.jpg Population Surveillance & Visual Search Engine:

http://gallery.techarena.in/data/516/New-visual-search-engine-TinEye.jpg Visual Prosthesis & Ocular Implants:

http://futurepredictions.files.wordpress.com/2008/12/080129-bionic-eye_big.jpg Research Tool:

R. Friedman, “Reverse engineering the brain,” Biomedical Computation Review, vol. 5, no. 2, pp. 10-17, Spring 2009

Blue Gene: http://agix.files.wordpress.com/2008/11/blue_gene.jpg

Hardware Chip http://etienne.ece.jhu.edu/ftenore/research/publications/tenore_biocas08.pdf

Relevant Publications

F. Folowosele, T.J. Hamilton, A. Harrison, A. Cassidy, A.G. Andreou, S. Mihalas, E. Niebur and R. Etienne-Cummings, “A Switched Capacitor Implementation of the Generalized Linear Integrate-and-Fire Neuron,” Submitted to Proceedings of IEEE International Symposium on Circuits and Systems (ISCAS), 2009

F. Folowosele, R.J. Vogelstein, and R. Etienne-Cummings, “Real-Time Silicon Implementation of V1 in Hierarchical Visual Information Processing,” Proceedings of IEEE Biomedical Circuits and Systems Conference (BioCAS), Baltimore, Maryland, November 2008.

S. Chen, F. Folowosele, D. Kim, R.J. Vogelstein, E. Culurciello and R. Etienne-Cummings, “Size and Position Invariant Human Posture Recognition Algorithm with Spike-Based Image Sensor” Proceedings of IEEE Biomedical Circuits and Systems Conference (BioCAS), Baltimore, Maryland, November 2008.

F. Folowosele, F. Tenore, A. Russell, G. Orchard, M. Vismer, J. Tapson, and R. Etienne-Cummings, “Implementing a Neuromorphic Cross-Correlation Engine with Silicon Neurons,” Proceedings of IEEE International Symposium on Circuits and Systems (ISCAS), Seattle, Washington, May 2008.

J. Tapson, M.P. Vismer, C. Jin, A van Schaik, F. Folowosele, and R. Etienne-Cummings, “A Two-Neuron Cross-Correlation Circuit with a Wide and Continuous Range of Time Delay,” Proceedings of IEEE International Symposium on Circuits and Systems (ISCAS), Seattle, Washington, May 2008.

F. Folowosele, R.J. Vogelstein, R. Etienne-Cummings, “Spike-Based MAX Network for Nonlinear Pooling in Hierarchical Vision Processing,” Proceedings of IEEE Biomedical Circuits and Systems Conference (BioCAS), Montreal, Canada, November 2007.

F. Folowosele, J. Tapson, R. Etienne-Cummings, “A Wireless Address Event Representation System for Biological Sensor Networks,” Proc. SPIE (Bioengineered and Bioinspired Systems), 2007.