CMOS BASED HYPERSPECTRAL IMAGINGspie.org/Documents/Industry relations/HSI imec for SPIE...IMEC’s...

CMOS BASED HYPERSPECTRAL IMAGINGFOR COMPACT / LOW-COST / HIGH-VOLUME IMAGING SPECTROSCOPY

SPIE Baltimore - April 2016

public © imec 20162

WHY DO WE NEED HYPERSPECTRAL IMAGING?

1

2

n

to improve vision

and discrimination

power ...

... by using

spectral

dimension of

objects being

imaged


SPECTRAL IMAGING OPEN ONE NEW DIMENSION

POINT SPECTROSCOPY

xy

λ

Accurate spectral analysis

of one spatial pixel only

HYPERSPECTRAL IMAGING

spectral signature images revealing

objects chemical composition

1

n

Seeing RGB colors

of one image only

COLOR IMAGING

xy

R

G

B1

2

3

one wavelength dimension for each pixel ...

x

y

ʎ


CONVERGENCE OF IMAGING & SPECTROSCOPY

Hig

h d

iscr

imin

atio

n p

ow

er

Single point

with

highest

spectral

resolution

SPECTROSCOPY

High speed, high spatial resolutionIMAGING

RGB filters

on multiple pixelsIMAGING

SPECTROSCOPY

IMEC approach: spectral filters on multiple pixels

Tiled patternMosaic

pattern

OR OR Line pattern


SPECTRAL FILTERS APPLIED TO IMAGE SENSOR WAFERS BY

CMOS SEMICONDUCTOR PROCESS

IMEC hyperspectral filter structures processed at wafer-level on top of commercial CMOS

image sensor wafer (here on CMOSIS’s CMV2000 & CMV4000 sensors)


474

nm

465

nm

630

nm

624

nm

608

nm

485

nm

4x4 mosaic = 16 spectral bands

~ x16 cameras on-chip!

496

nm

510

nm

522

nm

534

nm

546

nm

586

nm

578

nm

562

nm

548

nm

600

nm

we design & manufacture spectral filters directly

onto image sensor pixels ...

... based on CMOS compatible

semiconductor process


NANO-DEPOSITED FILTERS PATTERNED @ PIXEL LEVEL!

down to 2um pixels!

1 to 100+ colors


• R&D to production environment

• 1B€ pilot-line factories on-site

• Class 1 ‘clean’ processing

• Advanced CMOS processes (22nn 7nm)


E.G. OF POSSIBLE APPROACH FABRY-PEROT SPECTRAL FILTERS

θ

θ

LWavelength selection depends on cavity length L

cos2nLk

Tx (%)

FWHM ~

5-20nm

Narrow-band & high transmission

efficiencies spectral filters

Different cavity heights = different

spectral wavelengths captured!


CUSTOM HYPERSPECTRAL IMAGING SOLUTIONS

Snapshot

Bayer Mosaic

Snapshot

TiledContinuous

Line-scan

Custom

Line-scan

Fill the sensor ONLY with

selected bands of interest

Optical filters & imager can

be customized to match final application

requirements

1 n2

Filt

er

resp

onse

FWHM

OD

Spectral filters can be integrated onto

commercial image sensor wafers


OFF-THE-SHELF SPECTRAL IMAGING SENSORS

RGB + NIR

R G B NIR

(in development)

4 bands

R G B

RGB

70+ commercial

vendors

450nm 1000nm

Multi-spectral

Snapshot mosaic & tiled

16 / 25 / 32 / 40+ bands

Linescan

100 / 150+ bands

450nm 1000nm

Hyperspectral


LINESCAN 100+BANDS HSI SENSOR DESIGN (GEN1)

Key specifications

– Spectral resolution: 100+ bands in 600-1000nm

– FWHM: ~ 10-15nm

– Spatial resolution: 2048 pixels x length of scan

– Speed: up to 340 fps (full sensor frame)

Linescan GEN1 sensor with 100+bands (from CMOSIS CMV2000, without micro-lenses)


LINESCAN 150+BANDS HSI SENSOR DESIGN (GEN 2)

Key specifications

– Spectral resolution: 150+ bands in 470-925nm

– FWHM: ~ 10-15nm

– Spatial resolution: 2048 pixels x length of scan


Wavelength (nm)

QE (

%)

Linescan GEN2 sensor with 150+bands(from CMOSIS CMV2000, without micro-lenses)


SNAPSHOT TILED 32 BANDS HSI SENSOR DESIGN

Key specifications

– Spectral resolution: 32 bands in 600-940nm

– FWHM: ~ 10-15nm

– Spatial resolution: 256x256 RAW per band


Snapshot tiled sensor with 32bands (from CMOSIS CMV2000 - without microlenses)


SNAPSHOT MOSAIC 4X4 HSI SENSOR DESIGN

Key specifications


– FWHM: ~ 10-15nm



Snapshot mosaic sensor with 4x4 =16bands(from CMOSIS CMV2000 - without microlenses)


SNAPSHOT MOSAIC 5X5 HSI SENSOR DESIGNSnapshot mosaic sensor with 5x5 =25bands

(from CMOSIS CMV2000 - without microlenses)

0

2

4

6

8

10

12

14

600 650 700 750 800 850 900 950 1000

QE

(%

)

Wavelength (nm)

Quantum Efficiencyof hyperspectral 5x5 mosaic sensor = 25 bands sensor #XXX

band 1 band 2 band 3 band 4 band 5 band 6 band 7



band 22 band 23 band 24 band 25

Key specifications


– FWHM: ~ 10-15nm




ROADMAP FOR OFF-THE-SHELF HSI SENSORS

2013 2014 2015 2016 2017+

‘RGB+NIR’

LINESCAN

‘Wedge’

GEN1 600-975nm

(100+bands)

NIR

GEN2 470-925nm

(150+bands)

VNIR

Snapshot

‘TILED’

GEN1 600-975nm (32bands)

NIR

Snapshot

‘MOSAIC’

VIS+NIR 450-875nm

(40+bands, dual-sensor)

VNIR

VIS 460-640nm

(4x4 =16bands)

VIS

NIR 600-975nm

(5x5 = 25bands)

NIR

GEN1 1-1.7um

(100+ bands)

SWIR

... to machine vision!

Bringing hyperspectral

imaging...


ECOSYSTEM PARTNERS FOR HSI SENSOR CAMERA INTEGRATION

Adimec

Tattile

PhotonFocus

Ximea

Bayspec

3D-One

VRMAGIC

Cubert

Flexible camera platform Giga/Ethernet interface

Focus on remote sensing, UAV

agriculture & machine vision

High speed camera Cameralink or CoaXPress interface

Compact optical design with ZIFF socket to

swap different HSI sensors

industrial, medical and global security

Smart camera Giga/Ethernet interface

Reprogrammable FPGA pre-processing

hypercube data inside camera

Focus on industrial machine vision

Handheld mobile platform First proof-of-concept of

IMEC’s HSI sensors integration

to mobile

Android OS open platform

USB2.0 interfaceUSB 3.0 vision camera High-speed USB3.0 camera

Ultra compact camera: 26.4 x 26.4 x 21.2 mm

27grams weight only!

Embedded vision module Giga/Ethernet interface

Motorized optics and Multi-camera head

option

Embedded HSI data processing include

storage

Smart camera GigeVision and Genicam compliant

A/B shaft encoder interface (RS-422 or HTL levels).

FPGA pre-processing

Agriculture, food sorting and industrial applications


HSI DEMO-KITS FOR EVALUATION

compact | pc-powered | ready-to-go!


HSI SOFTWARE TO SUPPORT APPLICATION DEVELOPMENT

License available to all

demo-kit customers & partners

Support including installation,

training and Q&A (remotely

and on-site)

Source-code available for

OEM integration for the

following blocks:

– Cube registration

– Reflectance calculation

– Spectral corrections


LINESCAN 100+BANDS HSI SENSOR (GEN1) CUBE EXAMPLE

HSI data-cube (after scan)

1024 pixels

2048 p

ixels

x 100+ spectral

band images!

Classified image(extracting leafs only)Sample spectra

(x3 pixels extracted from leafs)


DRIVING APPLICATIONS FOR IMEC HSI

Imaging spectroscopy analyzers

DNA sequencers / flow cytometers

Water monitoring analyzers

Blood / urine analyzers

Life-science instrumentation

Food sorting / quality grading

Pharmaceutical defect inspection

Industrial process control

Bank-note inspection

Print quality inspection

Machine vision / Optical sorting

Surgery-guided imaging

Endoscopy

Ophthalmology / retina imaging

Wounds imaging

Fluorescence microscopy

Medical imaging

Night vision systems

Fuel monitoring systems

Driver drowsiness

Automotive & Transport

Industrial gas leaks monitoring

Intrusion detection / authentication

Forensics

Anti-counterfeiting

Security / Surveillance

Remote sensing

UAVs & cube-sattelites for:

Precision agriculture

Environment monitoring

Terrestrial / maritime earth observation

... integration path into UAVs

for precision agriculture

Soil analysis

Nutrient deficiency diagnostics

Grows monitoring

Disease and weeds alerts

Yield prediction


LINESCAN SENSOR IN PRECISION AGRICULTURE

862 nm 2.5cm ground

resolution!

NDVI (vegetation index)

management

zone map

vegetation

index mapspectral imageRAW image


snapshot hyperspectral

imaging in medical


REAL-TIME OXYGENATION MAPS

View live video here https://vimeo.com/132097972

Reconstructed Hb/HbO/HBT maps extracted from HSI snapshot mosaic

4x4 = 16bands HSI sensor on epileptic patient during brain surgery

Blood Spectral SignatureSpectrometer As seem by 4x4 HSI camera

https://vimeo.com/132097972


Courtesy of Perception

Park Studio

ATM / BANK NOTE INSPECTION

Hyperspectral (5x5 mosaic NIR)

RGB___________


LINESCAN HSI SENSOR FOR RICE INSPECTION

600 650 700 750 800 850 900 950 10000

50

100

150

200

250

300

350

400

450

500

wavelength (nm)

black_sample

dark_sample

background_dish

good_wholegrain

Training Set

of Hyperspectral signatures

1800

2000

2200

2400

2600

2800

3000 600

800

1000

1200

1400

1600

black spot dark

background

good

Principal Component Analysis

background_dish

black_sample

dark_sample

good_wholegrain

Object–level classification

+ standard vision software processing

dark

good

black

black

dark

good

samples

Classification Training Set


HIGH SPEED ANOMALIES DETECTION WITH

25+ BANDS SNAPSHOT MOSAIC SENSOR

View live video here https://vimeo.com/132098267

real leaf

plastic

leaf

https://vimeo.com/132098267


FOOD FRESHNESS MONITORING FOR FOOD

QUALITY GRADING APPLICATIONS

100% inspection via gill’s blood spectrum

Determine food quality freshness within few hours till 4 days


SAME TECHNO INDUSTRIAL TO CONSUMER?

Life-science instrumentation

Machine vision /

Optical sorting

Medical imaging

Automotive & Transport

Security / Surveillance

Remote sensing Consumer

Consumer

ConsumerConsumer

Consumer

Consumerconnected health


HSI SOFTWARE IMAGE PROCESSING PIPELINE

Camera/Sensor

SpecificApplication Specific

Pre-Processing

Camera calibration,

registration, synchronization,

reflectance calculation...etc

Feature

Extraction &

Selection

Feature

Classification/

Recognition

Spectral un-mixing,

smoothening, PCA, SVM, k-

means clustering, ...etc

Post-Processing

Visualization, GUI... etc

HSI

‘cube’

Application Specific (to be developed

for each platform)

IMEC can support these

SOFTWARE / Application layers


IMEC COLLABORATION MODEL FOR DEVELOPING HSI

SOLUTIONS, APPLICATIONS & MARKETS

Large

data-sets

&

services

Spectral

filter

materials

& Fab

operations

Sensor

&

Filter

co-

Design

Low

level

image

proc.

Firmware

HSI

image

proc.

SW

Full

system

feasibility

study

Appli.

SW

Camera

&

System

Design

CMOS BASED HYPERSPECTRAL IMAGINGspie.org/Documents/Industry relations/HSI imec for SPIE...IMEC’s...

Documents

Transcript of CMOS BASED HYPERSPECTRAL IMAGINGspie.org/Documents/Industry relations/HSI imec for SPIE...IMEC’s...