Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological...

41
Lecture 7 The Eye and Neuromorphic Vision 1

Transcript of Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological...

Page 1: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Lecture 7

The Eye and Neuromorphic Vision

1

Page 2: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Outline

• The eye and the retina• Artificial Vs biological vision systems• Fundamentals of photo receptors • Read-out strategies• Examples of neuromorphic vision systems• Further processing

2

Page 3: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

• Eyeball Cross Section and Retina

3

Page 4: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Retina Cells

4

Page 5: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Schematized Retinal Cells

5

Page 6: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Outline

• The eye and the retina• Artificial Vs biological vision systems• Fundamentals of photo receptors • Read-out strategies• Examples of neuromorphic vision systems• Further processing

6

Page 7: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Frame-based Sensors Vs Biological Vision

Artificial Vision Biological Systems

Frame-based Continuous Detection

Global AGC Local AGC

Redundancy Low redundancy

High power consumption Low power consumption

7

Page 8: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Notion of Frame

Tframe

TintegrationTime

x

y

Tintegration<Tframe

8

Page 9: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Dynamic Range

Rüedi et al, JSSC 2003 ”A 128x128 Pixel 120dB Dynamic-Range Vision-Sensor Chip for Image Contrast and Orientation Extraction”

Bad choice of the integration time

The human eye has a dynamic range higher than 10 decades

9

Page 10: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Outline

• The eye and the retina• Artificial Vs biological vision systems• Fundamentals of photo receptors • Read-out strategies• Examples of neuromorphic vision systems• Further processing

10

Page 11: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Depletion Region and Phototransduction

11

Page 12: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Sili

con

Electromagnetic Spectrum

12

Page 13: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Some Possible Photodetectors in CMOS Technology

13

A. Moini. ‘Vision Chips’. Kluwer Academic Publishers, 2000.

Page 14: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Photogates

14

Page 15: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Logarithmic Photoreceptors

(a) (b)

15

Page 16: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Photoreceptors with Negative Feedback

(a)

(b)16

Page 17: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Classic Active Pixel Sensor (APS)

17

Page 18: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Outline

• The eye and the retina• Artificial vision Vs biological systems• Fundamentals of photo receptors • Read-out strategies• Examples of neuromorphic vision systems• Further processing

18

Page 19: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Read-out Strategies:

Addressing/Scanning Charge Coupled Devices (CCD) Address Event Representation (AER)

19

Page 20: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Addressing/Scanning

Token System20

Page 21: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Charge Coupled Devices (CCD)

21

Page 22: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Address Event Representation (AER) (I)

22

Page 23: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Address Event Representation (AER) (II)

23

Page 24: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Outline

• The eye and the retina• Artificial Vs vision biological systems• Fundamentals of photo receptors • Read-out strategies• Examples of neuromorphic vision systems• Further processing

24

Page 25: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Delbrück’s Adaptative Photo Cell (I)

25

Page 26: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Delbrück’s Adaptative Photo Cell (II)

Non-linear element26

Page 27: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Mahowald’s Pixel

27

Page 28: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Retina Pixel Boahen

28

Page 29: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Dynamic Vision Sensor (DVS)

P. Lichtsteiner, C. Posch, and T. Delbrück, “A 128 128 120 dB 15 µs latency asynchronous temporal

contrast vision sensor,” IEEE J. Solid-State Circuits, vol. 43, no. 2, pp. 566–576, Feb. 2008.

29

Page 30: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Common Source Amplifier

30

Can used as comparator

Page 31: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Examples of Biomorphic Vision Sensors

Spatio-Temporal Contrast Detector:http://folk.uio.no/juanle/Projects/Spatial_contrast_retina.html

Dynamic Vision Sensor:http://folk.uio.no/juanle/Projects/DVS.html

31

Page 32: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Outline

• The eye and the retina• Artificial vision Vs biological systems• Fundamentals of photo receptors • Read-out strategies• Examples of neuromorphic vision systems• Further processing

32

Page 33: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Further Image Processing

• Motion Detection• Features extraction

33

Page 34: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Token Based Motion Processing: Reichardt Detector

34

Page 35: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Intensity Based

35

Page 36: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Features Extraction

36

Page 37: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

’Difference of Gaussians’ Kernel (I)

37

Page 38: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

’Difference of Gaussians’ Kernel (II)

38

Page 39: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

Result of Edge Detection

39

Page 40: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

45 Degrees Edge Kernel (I)

40

Page 41: Lecture 7 The Eye and Neuromorphic Vision 1. Outline The eye and the retina Artificial Vs biological vision systems Fundamentals of photo receptors Read-out.

45 Degrees Edge Kernel (II)

41