Mark Buckler, Cornell Suren Jayasuriya, CMU Adrian Sampson, Cornell

Rethinking the Camera Pipeline for Computer Visionor, Building an Approximate Camera

Mobile vision is a pretty cool idea

…always on.…on your smartphone.…in real time.

object recognition object localization

image segmentation 3D structure reconstruction

localization & mapping optical character recognition

face recognition activity recognition

human pose estimation

hardware efficiency

algorithmic efficiency

images vision results

ISCA 2016

images vision results

hardware efficiency

algorithmic efficiency

imagesphotonsraw sensor data

ISP

This project: a programmable camera pipeline.

Let’s approximate a camera pipeline

Design approximation into the camera sensor and the ISP

Show how to retrain vision modelsto work on the cheaper, raw data

Measure energy-accuracy trade-offslatent in real-world vision applications

Camera Sensor

Image Signal Processor

Photodiode

Amplifier

ADC

Demosaicing

Denoising

White Balance

Gamut Mapping

JPEG Compression

Tone Mapping

Vision Application CPU/GPU/“VPU”

RAW image

JPEG image

Photodiode

Amplifier

ADC

Demosaicing

Denoising

White Balance

Gamut Mapping

JPEG Compression

Tone Mapping

Vision Application

RAW image

JPEG image

photography mode

vision mode

Vision Application

Photodiode

Amplifier

ADC

approximate RAW image

power gated

5-bit logarithmic

Reversing the pipeline

norm

alize

d er

ror

0

0.25

0.5

0.75

1

1.25

1.5

1.75

2

ISP pipeline stages

original demosaic+ gamma compress

+ denoise all off

LeNet3 ResNet20 ResNet44 Farneback SGBM OpenMVG RCNN OpenFace

Sensitivity to ISP stages

cras

h

Demosaicing

Denoising

White Balance

Gamut Mapping

JPEG Compression

Tone Mapping

“True” demosaicing.

Demosaicing

Denoising

White Balance

Gamut Mapping

JPEG Compression

Tone Mapping

Subsampling.

Demosaicing

Denoising

White Balance

Gamut Mapping

JPEG Compression

Tone Mapping

Demosaicing

Denoising

White Balance

Gamut Mapping

JPEG Compression

Tone Mapping

quantize

Demosaicing

Denoising

White Balance

Gamut Mapping

JPEG Compression

Tone Mapping

replace both within-sensor trickery!

Sensitivity to ADC quantizationno

rmal

ized

erro

r

0.0

1.0

2.0

3.0

4.0

bits (linear quantization)8 7 6 5 4 3 2 1

LeNet3 ResNet20 ResNet44 Farneback SGBM OpenMVG RCNN OpenFace

Sensitivity to ADC quantizationno

rmal

ized

erro

r

0.0

1.0

2.0

3.0

4.0

bits (logarithmic quantization)8 7 6 5 4 3 2 1

LeNet3 ResNet20 ResNet44 Farneback SGBM OpenMVG RCNN OpenFace

How much energy can vision mode save?

sensor ISP vision ASIC

137.1–338.6 mW [LiKamWa]

130–185 mW [ON Semiconductor] 250 mW [Hegarty]

204 mW [TrueNorth] 590 mW [EIE]

How much energy can vision mode save?

sensor ISP vision ASIC

How much energy can vision mode save?

sensor ISP vision ASIC

readout (ADCs)

How much energy can vision mode save?

sensor ISP vision ASIC

Unresolved questions

Dynamic feedback loop

New signal processing toimprove learnability

Incremental costfor incremental scene changes

Data movement between sensor, ISP, and application