The Time Dimension for Scene Analysis

DeLiang Wang

Perception & Neurodynamics LabThe Ohio State University, USA

Presentation outline

Introduction Scene analysis and temporal correlation theory

Oscillatory Correlation LEGION network

Oscillatory Correlation Approach to Scene Analysis Image segmentation Object selection Cocktail party problem

Concluding remarks

Scene analysis problem

Binding problem

Feature binding (integration) is a fundamental problem in neuroscience and perception (and perceptrons)

Binding problemin Rosenblatt’sperceptrons

Temporal correlation theory

Temporal correlation theory proposes a solution to the nervous integration problem (von der Malsburg’81; also Milnor’74)

Application to cocktail party processing (von der Malsburg & Schneider’86)

Physiological evidence (Gray et al.’89)

Oscillatory correlation theory

• Oscillators represent feature detectors

• Binding is encoded by synchrony within an oscillator assembly and desynchrony between different assemblies

Computational requirements Need to synchronize locally coupled oscillator

population Need to desynchronize different populations, when

facing multiple objects Synchrony and desynchrony

must be achieved rapidly

LEGION architecture

LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95)

Relaxation oscillator as building block

Typical x trace (membrane potential)

With stimulus

Without stimulus

Analytical results

Theorem 1. (Synchronization). The oscillators in a connected block synchronize at an exponential rate

Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize and different blocks desynchronize, then synchrony within each block and the ordering of activations among different blocks are maintained

Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of blocks

Connectedness problem

Minsky-Papert connectedness problem is a long-standing problem in perceptron learning

The problem exposes fundamental limitations of supervised learning, and illustrates the importance of proper representations

Connectedness problem: LEGION solution Basic idea:

Synchronization within a connected pattern and desynchronization between different ones

Presentation outline

Introduction Scene analysis and temporal correlation theory

Oscillatory Correlation LEGION network

Oscillatory Correlation Approach to Scene Analysis Image segmentation Object selection Cocktail party problem

Concluding remarks

Oscillatory correlation approach to scene segmentation

Feature extraction first takes place An visual feature can be pixel intensity, depth, local image patch,

texture element, optic flow, etc. An auditory feature can be a pure tone, amplitude and frequency

modulation, onset, harmonicity, etc.

Connection weights between neighboring oscillators are set to be proportional to feature similarity

Global inhibitor controls granularity of segmentation Larger inhibition results in more and smaller regions

Segments pop out from LEGION in time

Image segmentation example: Demo

Input image

Image segmentation example

Input image Segmentation result

Object selection The slow inhibitor keeps

trace of each pattern, which can be overcome by only more salient (larger) patterns

Unlike traditional winner-take-all dynamics, selection (competition) takes place at the object level Consistent with object-

based attention theory Binding precedes attention,

rather than attention precedes binding (Treisman & Gelade’80)

Results of object selection

Input image LEGION output Selection outputInput LEGION segmentation Selection

Cocktail party problem

• In a natural environment, target speech is usually corrupted by acoustic interference, creating a speech segregation problem Popularly known as cocktail-party problem (Cherry’53); also

ball-room problem (Helmholtz, 1863)

• Human listeners organize sound in a perceptual process called auditory scene analysis (Bregman’90)

Auditory scene analysis (ASA) takes place in two conceptual stages: Segmentation. Decompose the acoustic signal into ‘sensory

elements’ (segments) Grouping. Combine segments into groups, so that segments in

the same group likely originate from the same sound source

NeuralOscillatorNetwork

Correlogram

Cross-channelCorrelation

ResynthesisHairCells

CochlearFiltering

Speechand Noise

ResynthesizedSpeech

ResynthesizedNoise

Correlogram (detail)

TimeLag

Neural Oscillator Network (detail)

GroupingLayer

SegmentationLayer

GlobalInhibitor

TimeFrequency

Oscillatory correlation for ASA (Wang & Brown’99)

Fre

quen

cy

Auditory periphery: Cochleagram

Cochleagram representation of the utterance: “Why were you all weary?” mixed with phone ringing

Time (seconds)0.0 1.5

5000

2741

1457

729

315

80

Grouping layer: Example

Two streams emerge from the group layer Foreground: left (original mixture ) Background: right

More recent results (Hu & Wang’04):

Time (seconds)0.0 1.5

5000

2741

1457

729

315

80

Time (seconds)0.0 1.5

5000

2741

1457

729

315

80

Back to physiology

Chattering cells recorded by Gray & McCormick’96

Burst oscillations are best modeled by relaxation oscillators

Versatility and time dimension

The principle of universality: “Give me a concrete problem and I will devise a network that solves it.” (von der Malsburg’99) It characterizes artificial intelligence

The principle of versatility: “Given the network, learn to cope with situations and problems as they arise.” (von der Malsburg’99) It characterizes natural intelligence

Time dimension is necessary for versatility Flexible and infinitely extensible Irreplaceable by spatial organization

Conclusion

Advances in dynamical analysis overcome computational obstacles of oscillatory correlation theory

Major progress is made towards solving the scene analysis problem

From Hebb’s cell assemblies to von der Malsburg’s correlation theory, time is an indispensable dimension for scene analysis