Dermoscopic Interest Point Detector and Descriptor

Howard Zhou1, Mei Chen2, James M. Rehg1

1School of Interactive Computing, Georgia Tech2Intel Research Pittsburgh

22009-07-01

Skin cancer Skin cancer : most common type of cancer ( > 1 million )

[ Top 5 categories of estimated annual cancer incidence for 2009 from National Cancer Institute ]

0 200,000 400,000 600,000 800,000 1,000,000 1,200,000

Bladder

Prostate

Breast

Lung

Skin

32009-07-01

Skin cancer Skin cancer : most common type of cancer ( > 1 million ) forms in tissues of the skin

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Skin lesions

42009-07-01

Skin cancer Skin cancer : most common type of cancer ( > 1 million ) forms in tissues of the skin

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Benign lesions

Skin cancer

52009-07-01

Skin cancer Skin cancer : most common type of cancer ( > 1 million ) forms in tissues of the skin

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Benign lesions

Skin cancer

Squamous cell carcinoma

Basal cell carcinoma

Melanoma

62009-07-01

Dermoscopy Non-invasive imaging technique Improve diagnostic accuracy by 30%

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Skin cancer

Squamous cell carcinoma

Basal cell carcinoma

Melanoma

72009-07-01

Dermoscopy

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Clinical view

Non-invasive imaging technique Improve diagnostic accuracy by 30%

82009-07-01

Dermoscopy

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermatoscope

Non-invasive imaging technique Improve diagnostic accuracy by 30% Microscope + light + liquid medium

92009-07-01

Dermoscopy Non-invasive imaging technique Improve diagnostic accuracy by 30% Microscope + light + liquid medium Reveal pigmented structures

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermatoscope

Dermoscopy view

102009-07-01

Dermoscopic features Pigmented structures revealed by dermoscopy

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

112009-07-01

Dermoscopic features Pigmented structures revealed by dermoscopy

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

122009-07-01

Dermoscopic features Pigmented structures revealed by dermoscopy

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

132009-07-01

Dermoscopic features Pigmented structures revealed by dermoscopy

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

142009-07-01

Dermoscopic features Pigmented structures revealed by dermoscopy

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

Negative network

152009-07-01

Dermoscopic features Pigmented structures revealed by dermoscopy [Betta et al. 2006], [Grana et al. 2006], [Iyatomi et al. 2007],

…

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

Negative network

162009-07-01

Dermoscopic features Over 100 dermoscopic features

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

Negative network

…

172009-07-01

Dermoscopic features

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

Negative network

…BW classifier

SLD classifier

BG classifier

NN classifier

… Over 100 dermoscopic features Multiple binary classifiers for each image

182009-07-01

Dermoscopic features

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

Negative network

…

General detector?

Generalized detector

192009-07-01

Dermoscopic features

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

Negative network

…

General detector?

Dermoscopic features consist of low level image characteristics (ridges, blobs, streaks, pigmentation,…)

Generalized detector

202009-07-01

Dermoscopic features

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

Negative network

…

General detector?

Dermoscopic features consist of low level image characteristics (ridges, blobs, streaks, pigmentation,…)

interest points

Generalized detector

212009-07-01

Dermoscopic Interest Point (DIP)

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Dermoscopy view

Blue-white veil

Scar-like depigmentation

Brown globules

Negative network

…

General detector: concentration/configuration of interest points

bag-of-visual-words approach

Dermoscopic features consist of low level image characteristics (ridges, blobs, streaks, pigmentation,…)

interest points

Generalized detector

222009-07-01

Dermoscopic Interest Point (DIP) Inspired by general interest point detector and descriptors

(SIFT & SURF) We propose Dermoscopic Interest Point (DIP)

detector - to extract these low level building blocks descriptor – for constructing a general visual vocabulary for

dermoscopic features

232009-07-01

Dermoscopic Interest Point (DIP) Compared to the general interest point detector and

descriptors (SIFT & SURF) Same key issues

Repeatable Distinctive Robust to noise and deformation (geometric and photometric)

Similar to SIFT & SURF Corners and blobs Scale and rotation invariant

In addition Curvilinear features (fibrillar pattern and radial streaming) Color component

242009-07-01

Detector Corners and blobs

Fast-Hessian detector [Bay, et al. 2006]

Hessian matrix

252009-07-01

Detector Corners and blobs

Fast-Hessian detector [Bay, et al. 2006]

Box filter approximation to replace Gaussian derivatives Fast using Integral image

Hessian matrix

262009-07-01

Detector Corners and blobs

Fast-Hessian detector [Bay, et al. 2006]

Curvilinear structures Curvilinear detector [Steger, 1996]

Hessian matrix

272009-07-01

Detector Corners and blobs

Fast-Hessian detector [Bay, et al. 2006]

Curvilinear structures Curvilinear detector [Steger, 1996]

Hessian matrix

282009-07-01

Detector Corners and blobs

Fast-Hessian detector [Bay, et al. 2006]

Curvilinear structures Curvilinear detector [Steger, 1996]

Hessian matrix

292009-07-01

Detector Corners and blobs

Fast-Hessian detector [Bay, et al. 2006]

Curvilinear structures Curvilinear detector [Steger, 1996]

Hessian matrix

302009-07-01

Descriptor Distinctiveness

Spatially localized information Distribution of gradient-related features Dermscopic: color features

Invariance (Repeatability) Relative strength to reduce the effect of photometric changes Relative orientation for rotation invariance

312009-07-01

Descriptor Distinctiveness

Spatially localized information Distribution of gradient-related features Dermscopic: color features

Invariance (Repeatability) Relative strength to reduce the effect of photometric changes Relative orientation for rotation invariance

To construct Reproducible orientation

322009-07-01

Descriptor Distinctiveness

Spatially localized information Distribution of gradient-related features Dermscopic: color features

Invariance (Repeatability) Relative strength to reduce the effect of photometric changes Relative orientation for rotation invariance

To construct Reproducible orientation

332009-07-01

Descriptor Distinctiveness

Spatially localized information Distribution of gradient-related features Dermscopic: color features

Invariance (Repeatability) Relative strength to reduce the effect of photometric changes Relative orientation for rotation invariance

To construct Reproducible orientation Feature vector

342009-07-01

Descriptor Orientation

For rotation invariance

Haar-wavelet responses in x and y direction (in a circular neighborhood)

352009-07-01

Descriptor Orientation

For rotation invariance

Haar-wavelet responses in x and y direction (in a circular neighborhood)

Reponses represented as 2D vectors dy

dx

362009-07-01

Descriptor Orientation

For rotation invariance

Haar-wavelet responses in x and y direction (in a circular neighborhood)

Reponses represented as 2D vectors

Average responses in a sliding window of 60 degree

dy

dx

372009-07-01

Descriptor Orientation

For rotation invariance

Haar-wavelet responses in x and y direction (in a circular neighborhood)

Reponses represented as 2D vectors

Average responses in a sliding window of 60 degree

The longest vector indicates the orientation

dy

dx

382009-07-01

Descriptor Descriptor components

Context of the descriptor: a square region oriented along the orientation (centered around the interest point)

Local statistics Uniform 4 x 4 subregions Intensity gradients (I): Sum of Haar-wavelet responses: dx, dy, |dx|,

|dy| Color statistics (C): Coarse color histogram of the region (alpha &

beta channels in L*a*b space)

[ Image courtesy of Bay et al. 2006]

392009-07-01

Dermoscopy Interest Point

402009-07-01

Dermoscopy specific Common interest point descriptor ignores linear

features

SURF DIP

412009-07-01

Experiment

422009-07-01

Conclusion A generalized framework for characterizing dermoscopic

features using Dermoscopic Interest Point (DIP) A feature detector and a descriptor specifically designed for

this purpose Initial experiments showed that our scheme achieves a

comparable level of invariance to lighting, scale, and rotation changes

432009-07-01

Future work Build a vocabulary of dermoscopic features using DIP Explore the possibility of using DIP in skin CAD related

applications: Dermoscopic feature extraction and classification Dermoscopy image registration Dermoscopy image search and retrieval via

dermoscopic features

442009-07-01

Acknowledgement Collaborators (in alphabetical order)

Dr. Laura K. Ferris M.D. Ph.D. UPMC Richard Gass, Intel Research Pittsburgh Casey Helfrich, Intel Research Pittsburgh

Many thanks to our anonymous reviewers for their helpful comments and suggestion

452009-07-01

Thank you

Thank you !

462009-07-01

Related publications Interest pointer detector and

descriptors Distinctive image features from scale-invariant

keypoints David G. LoweIntl. J. of Computer Vision (IJCV), 2004

Surf: Speeded up robust featuresHerbert Bay, Tinne tuytelaars, and Luc Van Gool,in Eur. Conf. on Computer Vision (ECCV), 2006

An unbiased detector of curvilinear structuresCarsten Steger,IEEE Trans. Pattern Anal. Machine Intell.(PAMI) 1996

472009-07-01

Outline Introduction Detector

Corners and blobs Curvilinear structures

Descriptor Orientation Descriptor components

Validation Conclusion

482009-07-01

Dermoscopic features A Pigmented Skin Lesion (PSL) typically has several

dermoscopic features Over 100 of these features

492009-07-01

Detecting line points

[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

n(x)

Cross section

n(x)

L(x)

L’ = 0L’’ large

Curve

502009-07-01

Experiment