FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
Click here to load reader
-
Upload
institute-of-information-systems-hes-so -
Category
Technology
-
view
1.363 -
download
2
Transcript of FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
Adrien Depeursinge, PhD MICCAI 2015 Tutorial on Biomedical Texture Analysis (BTA), Munich, Oct 5 2015
sampled lattices) is not straightforward and raises several chal-lenges related to translation, scaling and rotation invariances andcovariances that are becoming more complex in 3-D.
1.2. Related work and scope of this survey
Depending on research communities, various taxonomies areused to refer to 3-D texture information. A clarification of the tax-onomy is proposed in this section to accurately define the scope ofthis survey. It is partly based on Toriwaki and Yoshida, 2009. Three-dimensional texture and volumetric texture are both general andidentical terms designing a texture defined in R3 and include:
(1) Volumetric textures existing in ‘‘filled’’ objectsfV : x; y; z 2 Vx;y;z ! R3g that are generated by a volumetricdata acquisition device (e.g., tomography, confocal imaging).
(2) 2.5D textures existing on surfaces of ‘‘hollow’’ objects asfC : x; y; z 2 Cu;v ! R3g,
(3) Dynamic textures in two–dimensional time sequences asfS : x; y; t 2 Sx;y;t ! R3g,
Solid texture refers to category (1) and accounts for textures de-fined in a volume Vx,y,z indexed by three coordinates. Solid textureshave an intrinsic dimension of 3, which means that a number ofvariables equal to the dimensionality of the Euclidean space isneeded to represent the signal (Bennett, 1965; Foncubierta-Rodrí-guez et al., 2013a). Category (2) is designed as textured surface inDana and Nayar (1999) and Cula and Dana (2004), or 2.5-dimen-sional textures in Lu et al. (2006) and Aguet et al. (2008), wheretextures C are existing on the surface of 3-D objects and can be in-dexed uniquely by two coordinates (u,v). (2) is also used in Kajiyaand Kay (1989), Neyret (1995), and Filip and Haindl (2009), where3-D geometries are added onto the surface of objects to create real-istic rendering of virtual scenes. Motion analysis in videos can alsobe considered a multi-dimensional texture analysis problembelonging to category (3) and is designed by ‘‘dynamic texture’’in Bouthemy and Fablet (1998), Chomat and Crowley (1999), andSchödl et al. (2000).
In this survey, a comprehensive review of the literature pub-lished on classification and retrieval of biomedical solid textures(i.e., category (1)) is carried out. The focus of this text is on the fea-ture extraction and not machine learning techniques, since onlyfeature extraction is specific to 3-D solid texture analysis.
1.3. Structure of this article
This survey is structured as follows: The fundamentals of 3-Ddigital texture processing are defined in Section 2. Section 3 de-scribes the reviewmethodology used to systematically retrieve pa-pers dealing with 3-D solid texture classification and retrieval. Theimaging modalities and organs studied in the literature are re-viewed in Sections 4 and 5, respectively to list the various expecta-tions and needs of 3-D image processing. The resulting application-driven techniques are described, organized and grouped togetherin Section 6. A synthesis of the trends and gaps of the various ap-proaches, conclusions and opportunities are given in Section 7,respectively.
2. Fundamentals of solid texture processing
Although several researchers attempted to establish a generalmodel of texture description (Haralick, 1979; Julesz, 1981), it isgenerally recognized that no general mathematical model of tex-ture can be used to solve every image analysis problem (Mallat,1999). In this survey, we compare the various approaches based
on the 3-D geometrical properties of the primitives used, i.e., theelementary building block considered. The set of primitives usedand their assumed interactions define the properties of the textureanalysis approaches, from statistical to structural methods.
In Section 2.1, we define the mathematical framework andnotations considered to describe the content of 3-D digital images.The notion of texture primitives as well as their scales and direc-tions are defined in Section 2.2.
2.1. 3-D digitized images and sampling
In Cartesian coordinates, a generic 3-D continuous image is de-fined by a function of three variables f(x,y,z), where f represents ascalar at a point ðx; y; zÞ 2 R3. A 3-D digital image F(i, j,k) of dimen-sions M $ N $ O is obtained from sampling f at points ði; j; kÞ 2 Z3
of a 3-D ordered array (see Fig. 2). Increments in (i, j,k), correspondto physical displacements in R3 parametrized by the respectivespacings (Dx,Dy,Dz). For every cell of the digitized array, the valueof F(i, j,k) is typically obtained by averaging f in the cuboid domaindefined by (x,y,z) 2 [iDx, (i + 1)Dx]; [jDy, (j + 1)Dy]; [kDz, (k +1)Dz]) (Toriwaki and Yoshida, 2009). This cuboid is called a voxel.The three spherical coordinates (r,h,/) are unevenly sampled to(R,H,U) as shown in Fig. 3.
2.2. Texture primitives
The notion of texture primitive has been widely used in 2-D tex-ture analysis and defines the elementary building block of a giventexture class (Haralick, 1979; Jain et al., 1995; Lin et al., 1999). Alltexture processing approaches aim at modeling a given textureusing sets of prototype primitives. The concept of texture primitiveis naturally extended in 3-D as the geometry of the voxel sequenceused by a given texture analysis method. We consider a primitiveC(i, j,k) centered at a point (i, j,k) that lives on a neighborhood ofthis point. The primitive is constituted by a set of voxels with graytone values that forms a 3-D structure. Typical C neighborhoodsare voxel pairs, linear, planar, spherical or unconstrained. Signalassignment to the primitive can be either binary, categorical orcontinuous. Two example texture primitives are shown in Fig. 4.Texture primitives refer to local processing of 3-D images and localpatterns (see Toriwaki and Yoshida, 2009).
Fig. 2. 3-D digitized images and sampling in Cartesian coordinates.
Fig. 3. 3-D digitized images and sampling in spherical coordinates.
178 A. Depeursinge et al. /Medical Image Analysis 18 (2014) 176–196
sampled lattices) is not straightforward and raises several chal-lenges related to translation, scaling and rotation invariances andcovariances that are becoming more complex in 3-D.
1.2. Related work and scope of this survey
Depending on research communities, various taxonomies areused to refer to 3-D texture information. A clarification of the tax-onomy is proposed in this section to accurately define the scope ofthis survey. It is partly based on Toriwaki and Yoshida, 2009. Three-dimensional texture and volumetric texture are both general andidentical terms designing a texture defined in R3 and include:
(1) Volumetric textures existing in ‘‘filled’’ objectsfV : x; y; z 2 Vx;y;z ! R3g that are generated by a volumetricdata acquisition device (e.g., tomography, confocal imaging).
(2) 2.5D textures existing on surfaces of ‘‘hollow’’ objects asfC : x; y; z 2 Cu;v ! R3g,
(3) Dynamic textures in two–dimensional time sequences asfS : x; y; t 2 Sx;y;t ! R3g,
Solid texture refers to category (1) and accounts for textures de-fined in a volume Vx,y,z indexed by three coordinates. Solid textureshave an intrinsic dimension of 3, which means that a number ofvariables equal to the dimensionality of the Euclidean space isneeded to represent the signal (Bennett, 1965; Foncubierta-Rodrí-guez et al., 2013a). Category (2) is designed as textured surface inDana and Nayar (1999) and Cula and Dana (2004), or 2.5-dimen-sional textures in Lu et al. (2006) and Aguet et al. (2008), wheretextures C are existing on the surface of 3-D objects and can be in-dexed uniquely by two coordinates (u,v). (2) is also used in Kajiyaand Kay (1989), Neyret (1995), and Filip and Haindl (2009), where3-D geometries are added onto the surface of objects to create real-istic rendering of virtual scenes. Motion analysis in videos can alsobe considered a multi-dimensional texture analysis problembelonging to category (3) and is designed by ‘‘dynamic texture’’in Bouthemy and Fablet (1998), Chomat and Crowley (1999), andSchödl et al. (2000).
In this survey, a comprehensive review of the literature pub-lished on classification and retrieval of biomedical solid textures(i.e., category (1)) is carried out. The focus of this text is on the fea-ture extraction and not machine learning techniques, since onlyfeature extraction is specific to 3-D solid texture analysis.
1.3. Structure of this article
This survey is structured as follows: The fundamentals of 3-Ddigital texture processing are defined in Section 2. Section 3 de-scribes the reviewmethodology used to systematically retrieve pa-pers dealing with 3-D solid texture classification and retrieval. Theimaging modalities and organs studied in the literature are re-viewed in Sections 4 and 5, respectively to list the various expecta-tions and needs of 3-D image processing. The resulting application-driven techniques are described, organized and grouped togetherin Section 6. A synthesis of the trends and gaps of the various ap-proaches, conclusions and opportunities are given in Section 7,respectively.
2. Fundamentals of solid texture processing
Although several researchers attempted to establish a generalmodel of texture description (Haralick, 1979; Julesz, 1981), it isgenerally recognized that no general mathematical model of tex-ture can be used to solve every image analysis problem (Mallat,1999). In this survey, we compare the various approaches based
on the 3-D geometrical properties of the primitives used, i.e., theelementary building block considered. The set of primitives usedand their assumed interactions define the properties of the textureanalysis approaches, from statistical to structural methods.
In Section 2.1, we define the mathematical framework andnotations considered to describe the content of 3-D digital images.The notion of texture primitives as well as their scales and direc-tions are defined in Section 2.2.
2.1. 3-D digitized images and sampling
In Cartesian coordinates, a generic 3-D continuous image is de-fined by a function of three variables f(x,y,z), where f represents ascalar at a point ðx; y; zÞ 2 R3. A 3-D digital image F(i, j,k) of dimen-sions M $ N $ O is obtained from sampling f at points ði; j; kÞ 2 Z3
of a 3-D ordered array (see Fig. 2). Increments in (i, j,k), correspondto physical displacements in R3 parametrized by the respectivespacings (Dx,Dy,Dz). For every cell of the digitized array, the valueof F(i, j,k) is typically obtained by averaging f in the cuboid domaindefined by (x,y,z) 2 [iDx, (i + 1)Dx]; [jDy, (j + 1)Dy]; [kDz, (k +1)Dz]) (Toriwaki and Yoshida, 2009). This cuboid is called a voxel.The three spherical coordinates (r,h,/) are unevenly sampled to(R,H,U) as shown in Fig. 3.
2.2. Texture primitives
The notion of texture primitive has been widely used in 2-D tex-ture analysis and defines the elementary building block of a giventexture class (Haralick, 1979; Jain et al., 1995; Lin et al., 1999). Alltexture processing approaches aim at modeling a given textureusing sets of prototype primitives. The concept of texture primitiveis naturally extended in 3-D as the geometry of the voxel sequenceused by a given texture analysis method. We consider a primitiveC(i, j,k) centered at a point (i, j,k) that lives on a neighborhood ofthis point. The primitive is constituted by a set of voxels with graytone values that forms a 3-D structure. Typical C neighborhoodsare voxel pairs, linear, planar, spherical or unconstrained. Signalassignment to the primitive can be either binary, categorical orcontinuous. Two example texture primitives are shown in Fig. 4.Texture primitives refer to local processing of 3-D images and localpatterns (see Toriwaki and Yoshida, 2009).
Fig. 2. 3-D digitized images and sampling in Cartesian coordinates.
Fig. 3. 3-D digitized images and sampling in spherical coordinates.
178 A. Depeursinge et al. /Medical Image Analysis 18 (2014) 176–196
sampled
lattices)is
notstraightforw
ardand
raisesseveral
chal-lenges
relatedto
translation,scalingand
rotationinvariances
andcovariances
thatare
becoming
more
complex
in3-D
.
1.2.Relatedwork
andscope
ofthis
survey
Depending
onresearch
communities,
varioustaxonom
iesare
usedto
referto
3-Dtexture
information.A
clarificationof
thetax-
onomyis
proposedin
thissection
toaccurately
definethe
scopeof
thissurvey.Itis
partlybased
onToriw
akiandYoshida,2009.Three-
dimensional
textureand
volumetric
textureare
bothgeneral
andidenticalterm
sdesigning
atexture
definedin
R3and
include:
(1)Volum
etrictextures
existingin
‘‘filled’’objects
fV:x;y
;z2V
x;y;z !
R3g
thatare
generatedby
avolum
etricdata
acquisitiondevice
(e.g.,tomography,confocalim
aging).(2)
2.5Dtextures
existingon
surfacesof
‘‘hollow’’objects
asfC
:x;y;z
2C
u;v!
R3g,
(3)Dynam
ictextures
intw
o–dimensional
time
sequencesas
fS:x;y
;t2Sx;y
;t !R
3g,
Solidtexture
refersto
category(1)
andaccounts
fortextures
de-fined
inavolum
eVx,y,z indexed
bythree
coordinates.Solidtextures
havean
intrinsicdim
ensionof
3,which
means
thatanum
berof
variablesequal
tothe
dimensionality
ofthe
Euclideanspace
isneeded
torepresent
thesignal(Bennett,1965;
Foncubierta-Rodrí-guez
etal.,2013a).Category
(2)is
designedas
texturedsurface
inDana
andNayar
(1999)and
Culaand
Dana
(2004),or2.5-dim
en-sional
texturesin
Luet
al.(2006)and
Aguet
etal.(2008),w
heretextures
Care
existingon
thesurface
of3-Dobjects
andcan
bein-
dexeduniquely
bytw
ocoordinates
(u,v).(2)is
alsoused
inKajiya
andKay
(1989),Neyret
(1995),andFilip
andHaindl(2009),w
here3-D
geometries
areadded
ontothe
surfaceofobjects
tocreate
real-istic
renderingofvirtualscenes.M
otionanalysis
invideos
canalso
beconsidered
amulti-dim
ensionaltexture
analysisproblem
belongingto
category(3)
andis
designedby
‘‘dynamic
texture’’in
Bouthemyand
Fablet(1998),Chom
atand
Crowley
(1999),andSchödlet
al.(2000).In
thissurvey,
acom
prehensivereview
ofthe
literaturepub-
lishedon
classificationand
retrievalof
biomedical
solidtextures
(i.e.,category(1))is
carriedout.The
focusofthis
textis
onthe
fea-ture
extractionand
notmachine
learningtechniques,
sinceonly
featureextraction
isspecific
to3-D
solidtexture
analysis.
1.3.Structureof
thisarticle
Thissurvey
isstructured
asfollow
s:The
fundamentals
of3-D
digitaltexture
processingare
definedin
Section2.
Section3de-
scribesthe
reviewmethodology
usedto
systematically
retrievepa-
persdealing
with
3-Dsolid
textureclassification
andretrieval.The
imaging
modalities
andorgans
studiedin
theliterature
arere-
viewed
inSections
4and
5,respectivelyto
listthe
variousexpecta-
tionsand
needsof3-D
image
processing.Theresulting
application-driven
techniquesare
described,organizedand
groupedtogether
inSection
6.Asynthesis
ofthe
trendsand
gapsof
thevarious
ap-proaches,
conclusionsand
opportunitiesare
givenin
Section7,
respectively.
2.Fundam
entals
ofsolid
texture
processing
Although
severalresearchers
attempted
toestablish
ageneral
model
oftexture
description(H
aralick,1979;
Julesz,1981),
itis
generallyrecognized
thatno
generalmathem
aticalmodel
oftex-
turecan
beused
tosolve
everyim
ageanalysis
problem(M
allat,1999).
Inthis
survey,wecom
parethe
variousapproaches
based
onthe
3-Dgeom
etricalproperties
ofthe
primitives
used,i.e.,theelem
entarybuilding
blockconsidered.The
setof
primitives
usedand
theirassum
edinteractions
definethe
propertiesofthe
textureanalysis
approaches,fromstatisticalto
structuralmethods.
InSection
2.1,we
definethe
mathem
aticalfram
ework
andnotations
consideredto
describethe
contentof3-D
digitalimages.
Thenotion
oftexture
primitives
aswell
astheir
scalesand
direc-tions
aredefined
inSection
2.2.
2.1.3-Ddigitized
images
andsam
pling
InCartesian
coordinates,ageneric
3-Dcontinuous
image
isde-
finedby
afunction
ofthree
variablesf(x,y,z),w
herefrepresents
ascalar
atapointðx;y
;zÞ2R
3.A3-D
digitalimage
F(i,j,k)ofdim
en-sions
M$
N$
Ois
obtainedfrom
sampling
fat
pointsði;j;kÞ
2Z
3
ofa3-D
orderedarray
(seeFig.2).Increm
entsin
(i,j,k),correspondto
physicaldisplacem
entsin
R3param
etrizedby
therespective
spacings(D
x,Dy,D
z).Forevery
cellofthedigitized
array,thevalue
ofF(i,j,k)is
typicallyobtained
byaveraging
finthe
cuboiddom
aindefined
by(x,y,z)2
[iDx,(i+
1)Dx];
[jDy,(j+
1)Dy];
[kDz,(k
+1)D
z])(Toriw
akiandYoshida,2009).This
cuboidis
calledavoxel.
Thethree
sphericalcoordinates
(r,h,/)are
unevenlysam
pledto
(R,H,U
)as
shownin
Fig.3.
2.2.Textureprim
itives
Thenotion
oftextureprim
itivehas
beenwidely
usedin
2-Dtex-
tureanalysis
anddefines
theelem
entarybuilding
blockof
agiven
textureclass
(Haralick,1979;
Jainet
al.,1995;Lin
etal.,1999).A
lltexture
processingapproaches
aimat
modeling
agiven
textureusing
setsofprototype
primitives.The
conceptoftextureprim
itiveis
naturallyextended
in3-D
asthe
geometry
ofthevoxelsequence
usedby
agiven
textureanalysis
method.W
econsider
aprim
itiveC(i,j,k)
centeredat
apoint
(i,j,k)that
liveson
aneighborhood
ofthis
point.Theprim
itiveis
constitutedby
aset
ofvoxelswith
graytone
valuesthat
formsa3-D
structure.TypicalC
neighborhoodsare
voxelpairs,
linear,planar,
sphericalor
unconstrained.Signal
assignment
tothe
primitive
canbe
eitherbinary,
categoricalor
continuous.Tw
oexam
pletexture
primitives
areshow
nin
Fig.4.
Textureprim
itivesrefer
tolocalprocessing
of3-Dim
agesand
localpatterns
(seeToriw
akiandYoshida,2009).
Fig.2.3-D
digitizedim
agesand
sampling
inCartesian
coordinates.
Fig.3.3-D
digitizedim
agesand
sampling
insphericalcoordinates.
178A.D
epeursingeet
al./MedicalIm
ageAnalysis
18(2014)
176–196
OUTLINE
• Biomedical texture analysis: background
• Defining texture processes
• Notations, sampling and texture functions
• Texture operators, primitives and invariances
• Multiscale analysis
• Operator scale and uncertainty principle
• Region of interest and operator aggregation
• Multidirectional analysis
• Isotropic versus directional operators
• Importance of the local organization of image directions
• Conclusions
• References
REFERENCES
66
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2015 1
Steerable Wavelet Machines (SWM): LearningMoving Frames for Texture Classification
Adrien Depeursinge, Zsuzsanna Püspöki, John Paul Ward, and Michael Unser
Abstract—We present texture operators encoding class-specific local organizations of image directions (LOID) in arotation-invariant fashion. The operators are based on steerablecircular harmonic wavelets (CHW), offering a rich and yet com-pact initial representation for characterizing natural textures.We enforce the preservation of the joint location and orientationimage structure by building Cartan moving frames (MF) fromlocally-steered multi-order CHWs. In a second step, we usesupport vector machines (SVM) to learn a multi-class shapingmatrix of the initial CHW representations, yielding data-drivenMFs. Intuitively, the learned MFs can be seen as class-specificforged detectors that are applied and rotated at each point ofthe image to evaluate the magnitude of their responses, i.e., thepresence of the corresponding texture class. We experimentallydemonstrate the effectiveness of the proposed operators forclassifying natural textures, which outperformed the reportedperformance of recent approaches on several test suites of theOutex database.
Index Terms—Texture classification, feature learning, mov-ing frames, support vector machines, steerability, rotation-invariance, illumination-invariance, wavelet analysis.
I. INTRODUCTION
THe wealth of digital texture patterns is tightly relatedto the size of the observation window on a discrete
lattice. In an extreme case, an image region composedof one pixel cannot form geometrical structures. There-fore, the characterization of textural patterns often requiresintegrating image operators g (x), x 2 R2 over an imagedomain X . The latter raises two major challenges. First,the action of the integrated operators becomes diffuseover X , which hinders the spatial precision of texturesegmentation approaches. Second, the effect of integrationbecomes even more destructive when unidirectional opera-tors are jointly used to characterize the local organization ofimage directions (LOID) (e.g., directional filterbanks [1, 2],curvelets [3], histogram of gradients (HOG) [4], Haralick [5]).When separately integrated, the responses of unidirectionalindividual operators are not local anymore and their jointresponses become only sensitive to the global amount ofimage directions in the region X . For instance, the joint
responses of image gradients g1,2(x) =≥ØØ @ f (x)
@x1
ØØ,ØØ @ f (x)@x2
ØØ¥
arenot able to discriminate between the two textures classesf1(x) and f2(x) shown in Fig. 1 when integrated over theimage domain X .
This loss of information is detailed by Sifre et al. interms of separable group invariants [6]. When integrated
The authors are with the Biomedical Imaging Group, École Poly-technique Fédérale de Lausanne, Lausanne 1015, Switzerland (e-mail:[email protected]; [email protected]). A. Depeursinge isalso with the MedGIFT group, Institute of Information Systems, Universityof Applied Sciences Western Switzerland, Sierre (HES-SO).
of1(x) : of2(x) :
RX
ØØ @ f (x)@x1
ØØdx
R XØ Ø@
f(x)
@x 2
Ø Ø dx
image gradients
Fig. 1. The joint responses of image gradients≥| @ f (x)@x1
|, | @ f (x)@x2
|¥
are notable to discriminate between the two textures classes f1(x) and f2(x)when integrated over the image domain X . One circle in the gradientrepresentation corresponds to one realization (i.e., full image) of f1,2.
separately, the responses of unidirectional operators be-come invariant to a larger family of roto-translations wheredifferent orientations are translated by different values. Forinstance, it can be observed in Fig. 1 that f2 can be obtainedfrom f1 by vertically translating horizontal bars only andhorizontally translating vertical bars only.
The LOID are key for visual understanding [7] and is theonly property able to differentiate between f1 and f2 inFig. 1. It relates to the joint information between positionsand orientations in images, which is discarded by operatorsthat are invariant to separable groups. The LOID has beenleveraged in the literature to define [8] and discriminatetexture classes [9–12]. It is also at the origin of the successof methods based on local binary patterns (LBP) [9] andits extensions [13–19], which specifically encodes the LOIDwith uniform circular pixel sequences. Although classicaldeep learning approaches are not enforcing the characteri-zation of the LOID [20], deep convolutional networks wererecently proposed by Oyallon et al. to specifically preservethe structure of the roto-translation group [21].
An even bigger challenge is to design texture operatorsthat can characterize the LOID in a rotation-invariantfashion [9, 11]. The latter is very important to recognizeimage structures independently from both their local ori-entations and the global orientation of the image. Exam-ples of such structures are vascular, bronchial or dendritictrees in biomedical images, river deltas or urban areas insatellite images, or even more complex biomedical tissuearchitectures or crystals in petrographic analysis. Since mostof the above-mentioned imaging modalities yield imageswith pixel sizes defined in physical dimensions, we arenot interested in invariance to image scale. We aim todesign texture operators that are invariant to the family ofEuclidean transforms (also called rigid), i.e.,
g (x) = g (Rµ0 x °x0), 8x , x0,µ0 2R2,
·m
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Personalized medicine aims at enhancing the patient’s quality of life and prognosis
• Tailored treatment and medical decisions based on the molecular composition of diseased tissue
• Current limitations [Gerlinger2012]
• Molecular analysis of tissue composition is invasive (biopsy), slow and costly
• Cannot capture molecular heterogeneity
3
Intr atumor Heterogeneity Revealed by multiregion Sequencing
n engl j med 366;10 nejm.org march 8, 2012 887
tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.
Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense
and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).
mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had
B Regional Distribution of Mutations
C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling
A Biopsy Sites
R2 R4
DI=1.43
DI=1.81
M2bR9
Tetraploid
R4b
R9 R8R5
R4a
R1R3R2
M1M2b
M2a
VHL
KDM5C (missense and frameshift)mTOR (missense)
SETD2 (missense)KDM5C (splice site)
SETD2 (splice site)
?
SETD2 (frameshift)
PreP
PreM
Normal tissue
PrePPreMR1R2R3R5R8R9R4M1M2aM2b
C2o
rf85
WD
R7SU
PT6H
CD
H19
LAM
A3
DIX
DC
1H
PS5
NRA
PKI
AA
1524
SETD
2PL
CL1
BCL1
1AIF
NA
R1$
DA
MTS
10
C3
KIA
A12
67.
RT4
CD
44A
NKR
D26
TM7S
F4SL
C2A
1D
AC
H2
MM
AB
ZN
F521
HM
G20
AD
NM
T3A
RLF
MA
MLD
1M
AP3
K6H
DA
C6
PHF2
1BFA
M12
9BRP
S8C
IB2
RAB2
7ASL
C2A
12D
USP
12A
DA
MTS
L4N
AP1
L3U
SP51
KDM
5CSB
F1TO
M1
MYH
8W
DR2
4IT
IH5
AKA
P9FB
XO1
LIA
STN
IKSE
TD2
C3o
rf20
MR1
PIA
S3D
IO1
ERC
C5
KLALK
BH8
DA
PK1
DD
X58
SPA
TA21
ZN
F493
NG
EFD
IRA
S3LA
TS2
ITG
B3FL
NA
SATL
1KD
M5C
KDM
5CRB
FOX2
NPH
S1SO
X9C
ENPN
PSM
D7
RIM
BP2
GA
LNT1
1A
BHD
11U
GT2
A1
MTO
RPP
P6R2
ZN
F780
AW
SCD
2C
DKN
1BPP
FIA
1THSS
NA
1C
ASP
2PL
RG1
SETD
2C
CBL
2SE
SN2
MA
GEB
16N
LRP7
IGLO
N5
KLK4
WD
R62
KIA
A03
55C
YP4F
3A
KAP8
ZN
F519
DD
X52
ZC
3H18
TCF1
2N
USA
P172
X4KD
M2B
MRP
L51
C11
orf6
8A
NO
5EI
F4G
2M
SRB2
RALG
DS
EXT1
ZC
3HC
1PT
PRZ
1IN
TS1
CC
R6D
OPE
Y1A
TXN
1W
HSC
1C
LCN
2SS
R3KL
HL1
8SG
OL1
VHL
C2o
rf21
ALS
2CR1
2PL
B1FC
AM
RIF
I16
BCA
S2IL
12RB
2
PrivateUbiquitous Shared primary Shared metastasis
Private
Ubiquitous
Lungmetastases
Chest-wallmetastasis
Perinephricmetastasis
M110 cm
R7 (G4)
R5 (G4)
R9
R3 (G4)
R1 (G3) R2 (G3)
R4 (G1)
R6 (G1)
Hilu
m
R8 (G4)
Primarytumor
Shared primaryShared metastasis
M2b
M2a
Propidium Iodide Staining
No.
of C
ells
The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.
Copyright © 2012 Massachusetts Medical Society. All rights reserved.
Intr atumor Heterogeneity Revealed by multiregion Sequencing
n engl j med 366;10 nejm.org march 8, 2012 887
tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.
Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense
and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).
mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had
B Regional Distribution of Mutations
C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling
A Biopsy Sites
R2 R4
DI=1.43
DI=1.81
M2bR9
Tetraploid
R4b
R9 R8R5
R4a
R1R3R2
M1M2b
M2a
VHL
KDM5C (missense and frameshift)mTOR (missense)
SETD2 (missense)KDM5C (splice site)
SETD2 (splice site)
?
SETD2 (frameshift)
PreP
PreM
Normal tissue
PrePPreMR1R2R3R5R8R9R4M1M2aM2b
C2o
rf85
WD
R7SU
PT6H
CD
H19
LAM
A3
DIX
DC
1H
PS5
NRA
PKI
AA
1524
SETD
2PL
CL1
BCL1
1AIF
NA
R1$
DA
MTS
10
C3
KIA
A12
67.
RT4
CD
44A
NKR
D26
TM7S
F4SL
C2A
1D
AC
H2
MM
AB
ZN
F521
HM
G20
AD
NM
T3A
RLF
MA
MLD
1M
AP3
K6H
DA
C6
PHF2
1BFA
M12
9BRP
S8C
IB2
RAB2
7ASL
C2A
12D
USP
12A
DA
MTS
L4N
AP1
L3U
SP51
KDM
5CSB
F1TO
M1
MYH
8W
DR2
4IT
IH5
AKA
P9FB
XO1
LIA
STN
IKSE
TD2
C3o
rf20
MR1
PIA
S3D
IO1
ERC
C5
KLALK
BH8
DA
PK1
DD
X58
SPA
TA21
ZN
F493
NG
EFD
IRA
S3LA
TS2
ITG
B3FL
NA
SATL
1KD
M5C
KDM
5CRB
FOX2
NPH
S1SO
X9C
ENPN
PSM
D7
RIM
BP2
GA
LNT1
1A
BHD
11U
GT2
A1
MTO
RPP
P6R2
ZN
F780
AW
SCD
2C
DKN
1BPP
FIA
1THSS
NA
1C
ASP
2PL
RG1
SETD
2C
CBL
2SE
SN2
MA
GEB
16N
LRP7
IGLO
N5
KLK4
WD
R62
KIA
A03
55C
YP4F
3A
KAP8
ZN
F519
DD
X52
ZC
3H18
TCF1
2N
USA
P172
X4KD
M2B
MRP
L51
C11
orf6
8A
NO
5EI
F4G
2M
SRB2
RALG
DS
EXT1
ZC
3HC
1PT
PRZ
1IN
TS1
CC
R6D
OPE
Y1A
TXN
1W
HSC
1C
LCN
2SS
R3KL
HL1
8SG
OL1
VHL
C2o
rf21
ALS
2CR1
2PL
B1FC
AM
RIF
I16
BCA
S2IL
12RB
2
PrivateUbiquitous Shared primary Shared metastasis
Private
Ubiquitous
Lungmetastases
Chest-wallmetastasis
Perinephricmetastasis
M110 cm
R7 (G4)
R5 (G4)
R9
R3 (G4)
R1 (G3) R2 (G3)
R4 (G1)
R6 (G1)
Hilu
mR8 (G4)
Primarytumor
Shared primaryShared metastasis
M2b
M2a
Propidium Iodide Staining
No.
of C
ells
The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.
Copyright © 2012 Massachusetts Medical Society. All rights reserved.
Intr atumor Heterogeneity Revealed by multiregion Sequencing
n engl j med 366;10 nejm.org march 8, 2012 887
tion through loss of SETD2 methyltransferase func-tion driven by three distinct, regionally separated mutations on a background of ubiquitous loss of the other SETD2 allele on chromosome 3p.
Convergent evolution was observed for the X-chromosome–encoded histone H3K4 demeth-ylase KDM5C, harboring disruptive mutations in R1 through R3, R5, and R8 through R9 (missense
and frameshift deletion) and a splice-site mutation in the metastases (Fig. 2B and 2C).
mTOR Functional Intratumor HeterogeneityThe mammalian target of rapamycin (mTOR) ki-nase carried a kinase-domain missense mutation (L2431P) in all primary tumor regions except R4. All tumor regions harboring mTOR (L2431P) had
B Regional Distribution of Mutations
C Phylogenetic Relationships of Tumor Regions D Ploidy Profiling
A Biopsy Sites
R2 R4
DI=1.43
DI=1.81
M2bR9
Tetraploid
R4b
R9 R8R5
R4a
R1R3R2
M1M2b
M2a
VHL
KDM5C (missense and frameshift)mTOR (missense)
SETD2 (missense)KDM5C (splice site)
SETD2 (splice site)
?
SETD2 (frameshift)
PreP
PreM
Normal tissue
PrePPreMR1R2R3R5R8R9R4M1M2aM2b
C2o
rf85
WD
R7SU
PT6H
CD
H19
LAM
A3
DIX
DC
1H
PS5
NRA
PKI
AA
1524
SETD
2PL
CL1
BCL1
1AIF
NA
R1$
DA
MTS
10
C3
KIA
A12
67.
RT4
CD
44A
NKR
D26
TM7S
F4SL
C2A
1D
AC
H2
MM
AB
ZN
F521
HM
G20
AD
NM
T3A
RLF
MA
MLD
1M
AP3
K6H
DA
C6
PHF2
1BFA
M12
9BRP
S8C
IB2
RAB2
7ASL
C2A
12D
USP
12A
DA
MTS
L4N
AP1
L3U
SP51
KDM
5CSB
F1TO
M1
MYH
8W
DR2
4IT
IH5
AKA
P9FB
XO1
LIA
STN
IKSE
TD2
C3o
rf20
MR1
PIA
S3D
IO1
ERC
C5
KLALK
BH8
DA
PK1
DD
X58
SPA
TA21
ZN
F493
NG
EFD
IRA
S3LA
TS2
ITG
B3FL
NA
SATL
1KD
M5C
KDM
5CRB
FOX2
NPH
S1SO
X9C
ENPN
PSM
D7
RIM
BP2
GA
LNT1
1A
BHD
11U
GT2
A1
MTO
RPP
P6R2
ZN
F780
AW
SCD
2C
DKN
1BPP
FIA
1THSS
NA
1C
ASP
2PL
RG1
SETD
2C
CBL
2SE
SN2
MA
GEB
16N
LRP7
IGLO
N5
KLK4
WD
R62
KIA
A03
55C
YP4F
3A
KAP8
ZN
F519
DD
X52
ZC
3H18
TCF1
2N
USA
P172
X4KD
M2B
MRP
L51
C11
orf6
8A
NO
5EI
F4G
2M
SRB2
RALG
DS
EXT1
ZC
3HC
1PT
PRZ
1IN
TS1
CC
R6D
OPE
Y1A
TXN
1W
HSC
1C
LCN
2SS
R3KL
HL1
8SG
OL1
VHL
C2o
rf21
ALS
2CR1
2PL
B1FC
AM
RIF
I16
BCA
S2IL
12RB
2
PrivateUbiquitous Shared primary Shared metastasis
Private
Ubiquitous
Lungmetastases
Chest-wallmetastasis
Perinephricmetastasis
M110 cm
R7 (G4)
R5 (G4)
R9
R3 (G4)
R1 (G3) R2 (G3)
R4 (G1)
R6 (G1)H
ilum
R8 (G4)
Primarytumor
Shared primaryShared metastasis
M2b
M2a
Propidium Iodide Staining
No.
of C
ells
The New England Journal of Medicine Downloaded from nejm.org at UNIVERSITE DE GENEVE on June 2, 2014. For personal use only. No other uses without permission.
Copyright © 2012 Massachusetts Medical Society. All rights reserved.
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Huge potential for computerized medical image analysis
• Explore and reveal tissue structures related to tissue composition, function, ….
• Local quantitative image feature extraction
• Supervised and unsupervised machine learning
4
malignant, nonresponder
malignant, responder
benign
pre-malignant
undefined
quant. feat. #1
quan
t. fe
at. #
2
Supervised learning, big data
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Huge potential for computerized medical image analysis
• Create imaging biomarkers to predict diagnosis, prognosis, treatment response [Aerts2014]
5
Radiomics [Kumar2012] “Histopatholomics” [Gurcan2009]
Reuse existing diagnostic images ✓ radiology data1 ✓ digital pathology
Capture tissue heterogeneity
✓ 3D neighborhoods(e.g., 512x512x512)
✓ large 2D regions(e.g., 15,000x15,000)
Analytic power beyond naked eyes
✓ complex 3D tissue morphology
✓exhaustive characterization of 2D tissue structures
Non-invasive ✓ x
1e.g., X-ray, Ultrasound, CT, MRI, PET, OCT, …
BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Huge potential for computerized medical image analysis
• Explore and reveal tissue structures related to tissue composition, function, ….
• Local quantitative image feature extraction
• Supervised and unsupervised machine learning
6
malignant, nonresponder
malignant, responder
benign
pre-malignant
undefined
quant. feat. #1
quan
t. fe
at. #
2
Supervised learning, big data
Specific to texture!
OUTLINE
• Biomedical texture analysis: background
• Defining texture processes
• Notations, sampling and texture functions
• Texture operators, primitives and invariances
• Multiscale analysis
• Operator scale and uncertainty principle
• Region of interest and operator aggregation
• Multidirectional analysis
• Isotropic versus directional operators
• Importance of the local organization of image directions
• Conclusions
• References
REFERENCES
66
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2015 1
Steerable Wavelet Machines (SWM): LearningMoving Frames for Texture Classification
Adrien Depeursinge, Zsuzsanna Püspöki, John Paul Ward, and Michael Unser
Abstract—We present texture operators encoding class-specific local organizations of image directions (LOID) in arotation-invariant fashion. The operators are based on steerablecircular harmonic wavelets (CHW), offering a rich and yet com-pact initial representation for characterizing natural textures.We enforce the preservation of the joint location and orientationimage structure by building Cartan moving frames (MF) fromlocally-steered multi-order CHWs. In a second step, we usesupport vector machines (SVM) to learn a multi-class shapingmatrix of the initial CHW representations, yielding data-drivenMFs. Intuitively, the learned MFs can be seen as class-specificforged detectors that are applied and rotated at each point ofthe image to evaluate the magnitude of their responses, i.e., thepresence of the corresponding texture class. We experimentallydemonstrate the effectiveness of the proposed operators forclassifying natural textures, which outperformed the reportedperformance of recent approaches on several test suites of theOutex database.
Index Terms—Texture classification, feature learning, mov-ing frames, support vector machines, steerability, rotation-invariance, illumination-invariance, wavelet analysis.
I. INTRODUCTION
THe wealth of digital texture patterns is tightly relatedto the size of the observation window on a discrete
lattice. In an extreme case, an image region composedof one pixel cannot form geometrical structures. There-fore, the characterization of textural patterns often requiresintegrating image operators g (x), x 2 R2 over an imagedomain X . The latter raises two major challenges. First,the action of the integrated operators becomes diffuseover X , which hinders the spatial precision of texturesegmentation approaches. Second, the effect of integrationbecomes even more destructive when unidirectional opera-tors are jointly used to characterize the local organization ofimage directions (LOID) (e.g., directional filterbanks [1, 2],curvelets [3], histogram of gradients (HOG) [4], Haralick [5]).When separately integrated, the responses of unidirectionalindividual operators are not local anymore and their jointresponses become only sensitive to the global amount ofimage directions in the region X . For instance, the joint
responses of image gradients g1,2(x) =≥ØØ @ f (x)
@x1
ØØ,ØØ @ f (x)@x2
ØØ¥
arenot able to discriminate between the two textures classesf1(x) and f2(x) shown in Fig. 1 when integrated over theimage domain X .
This loss of information is detailed by Sifre et al. interms of separable group invariants [6]. When integrated
The authors are with the Biomedical Imaging Group, École Poly-technique Fédérale de Lausanne, Lausanne 1015, Switzerland (e-mail:[email protected]; [email protected]). A. Depeursinge isalso with the MedGIFT group, Institute of Information Systems, Universityof Applied Sciences Western Switzerland, Sierre (HES-SO).
of1(x) : of2(x) :
RX
ØØ @ f (x)@x1
ØØdx
R XØ Ø@
f(x)
@x 2
Ø Ø dx
image gradients
Fig. 1. The joint responses of image gradients≥| @ f (x)@x1
|, | @ f (x)@x2
|¥
are notable to discriminate between the two textures classes f1(x) and f2(x)when integrated over the image domain X . One circle in the gradientrepresentation corresponds to one realization (i.e., full image) of f1,2.
separately, the responses of unidirectional operators be-come invariant to a larger family of roto-translations wheredifferent orientations are translated by different values. Forinstance, it can be observed in Fig. 1 that f2 can be obtainedfrom f1 by vertically translating horizontal bars only andhorizontally translating vertical bars only.
The LOID are key for visual understanding [7] and is theonly property able to differentiate between f1 and f2 inFig. 1. It relates to the joint information between positionsand orientations in images, which is discarded by operatorsthat are invariant to separable groups. The LOID has beenleveraged in the literature to define [8] and discriminatetexture classes [9–12]. It is also at the origin of the successof methods based on local binary patterns (LBP) [9] andits extensions [13–19], which specifically encodes the LOIDwith uniform circular pixel sequences. Although classicaldeep learning approaches are not enforcing the characteri-zation of the LOID [20], deep convolutional networks wererecently proposed by Oyallon et al. to specifically preservethe structure of the roto-translation group [21].
An even bigger challenge is to design texture operatorsthat can characterize the LOID in a rotation-invariantfashion [9, 11]. The latter is very important to recognizeimage structures independently from both their local ori-entations and the global orientation of the image. Exam-ples of such structures are vascular, bronchial or dendritictrees in biomedical images, river deltas or urban areas insatellite images, or even more complex biomedical tissuearchitectures or crystals in petrographic analysis. Since mostof the above-mentioned imaging modalities yield imageswith pixel sizes defined in physical dimensions, we arenot interested in invariance to image scale. We aim todesign texture operators that are invariant to the family ofEuclidean transforms (also called rigid), i.e.,
g (x) = g (Rµ0 x °x0), 8x , x0,µ0 2R2,
·m
• Definition of texture
• Everybody agrees that nobody agrees on the definition of “texture” (context-dependent)
• “coarse”, “edgy”, “directional”, “repetitive”, “random”, …
• Oxford dictionary: “the feel, appearance, or consistency of a surface or a substance”
• [Haidekker2011]: “Texture is a systematic local variation of the image values”
• [Petrou2011]: “The most important characteristic of texture is that it is scale dependent. Different types of texture are visible at different scales”
COMPUTERIZED TEXTURE ANALYSIS
8
resolution, allowing to characterize structural properties of bio-medical tissue.1 Tissue anomalies are well characterized by localizedtexture properties in most imaging modalities (Tourassi, 1999;Kovalev and Petrou, 2000; Castellano et al., 2004; Depeursinge andMüller, 2011). This calls for scientific contributions on computerizedanalysis of 3-D texture in biomedical images, which engendered ma-jor scientific breakthroughs in 3-D solid texture analysis during thepast 20 years (Blot and Zwiggelaar, 2002; Kovalev and Petrou,2009; Foncubierta-Rodríguez et al., 2013a).
1.1. Biomedical volumetric solid texture
A uniform textured volume in a 3-D biomedical image is consid-ered to be composed of homogeneous tissue properties. The con-cept of organs or organelles was invented by human observersfor efficient understanding of anatomy. The latter can be definedas an organized cluster of one or several tissue types (i.e., definingsolid textures). Fig. 1 illustrates that, at various scales, everything istexture in biomedical images starting from the cell level to the or-gan level. The scaling parameter of textures is thus fundamentaland it is often used in computerized texture analysis approaches(Yeshurun and Carrasco, 2000).
According to the Oxford Dictionaries,2 texture is defined as ‘‘thefeel, appearance, or consistency of a surface or a substance’’, which re-lates to the surface structureor the internal structureof the consideredmatter in the context of 2-D or 3-D textures, respectively. The defini-tion of 3-D texture is not equivalent to 2-D surface texture since opa-que 3-D textures cannot be described in terms of reflectivity or albedoof a matter, which are often used to characterize textured surfaces(Dana et al., 1999). Haralick et al. (1973) also define texture as being‘‘an innate property of virtually all surfaces’’ and stipulates thattexture ‘‘contains important informationabout the structural arrange-ment of surfaces and their relationship to the surrounding environ-ment’’, which is also formally limited to textured surfaces.
Textured surfaces are central to human vision, because they areimportant visual cues about surface property, scenic depth, surfaceorientation, and texture information is used in pre-attentive visionfor identifying objects and understanding scenes (Julesz, 1962).The human visual cortex is sensitive to the orientation and spatialfrequencies (i.e., repetitiveness) of patterns (Blakemore and
Campbell, 1969; Maffei and Fiorentini, 1973), which relates to tex-ture properties. It is only in a second step that regions of homoge-neous textures are aggregated to constitute objects (e.g., organs) ata higher level of scene interpretation. However, the human com-prehension of the three-dimensional environment relies on ob-jects. The concept of three-dimensional texture is little used,because texture existing in more than two dimensions cannot befully visualized by humans (Toriwaki and Yoshida, 2009). Only vir-tual navigation in MPR or semi-transparent visualizations aremade available by computer graphics and allow observing 2-D pro-jections of opaque textures.
In a concern of sparsity and synthesis, 3-D computer graphicshave been focusing on objects. Shape-based methods allow encap-sulating essential properties of objects and thus provide approxi-mations of the real world that are corresponding to humanunderstanding and abstraction level. Recently, data acquisitiontechniques in medical imaging (e.g., tomographic, confocal, echo-graphic) as well as recent computing and storage infrastructuresallow computer vision and graphics to go beyond shape-basedmethods and towards three-dimensional solid texture-baseddescription of the visual information. 3-D solid textures encompassrich information of the internal structures of objects because theyare defined for each coordinate x; y; z 2 Vx;y;z ! R3, whereas shape-based descriptions are defined on surfaces x; y; z 2 Cu;v ! R3.jVj" jCj because every point of C can be uniquely indexed by onlytwo coordinates (u,v). Texture- and shape-based approaches arecomplementary and their success depends on the applicationneeds. While several papers on shape-based methods for classifica-tion and retrieval of organs and biomedical structures have beenpublished during the past 15 years (McInerney and Terzopoulos,1996; Metaxas, 1996; Beichel et al., 2001; Heimann and Meinzer,2009), 3-D biomedical solid texture analysis is still an emerging re-search field (Blot and Zwiggelaar, 2002). The most common ap-proach to 3-D solid texture analysis is to use 2-D texture in slices(Castellano et al., 2004; Depeursinge et al., 2007; Sørensen et al.,2010) or by projecting volumetric data on a plane (Chan et al.,2008), which does not allow exploiting the wealth of 3-D textureinformation. Based on the success and attention that 2-D textureanalysis obtained in the biomedical computer vision communityas well as the observed improved performance of 3-D techniquesover 2-D approaches in several application domains (Ranguelovaand Quinn, 1999; Mahmoud-Ghoneim et al., 2003; Xu et al.,2006b; Chen et al., 2007), 3-D biomedical solid texture analysisis expected to be a major research field in computer vision in thecoming years. The extension of 2-D approaches to R3 (or Z3 for
Fig. 1. 3-D biomedical tissue defines solid texture at multiple scales.
1 Biomedical tissue is considered in a broad meaning including connective, muscle,and nervous tissue.
2 http://oxforddictionaries.com/definition/texture, as of 9 October2013.
A. Depeursinge et al. /Medical Image Analysis 18 (2014) 176–196 177
[Depeursinge2014a]
COMPUTERIZED TEXTURE ANALYSIS
directions
9
scales
• Spatial scales and directions in images are fundamental for visual texture discrimination [Blakemore1969, Romeny2011]
• Relating to directional frequencies (shown in Fourier)
COMPUTERIZED TEXTURE ANALYSIS
10
directionsscales
• Spatial scales and directions in images are fundamental for visual texture discrimination [Blakemore1969, Romeny2011]
• Most approaches are leveraging these two properties
• Explicitly: Gray-level co-occurrence matrices (GLCM), run-length matrices (RLE), directional filterbanks and wavelets, Fourier, histograms of gradients (HOG), local binary patterns (LBP)
• Implicitly: Convolutional neural networks (CNN), scattering transform, topographic independant component analysis (TICA)
NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS
11
• 2-D continuous texture functions in space and Fourier
• Cartesian coordinates:
• Polar coordinates:
• 2-D digital texture functions
• Cartesian coordinates:
• Polar coordinates:
• Sampling (Cartesian)
• Increments in corresponds to physical displacements in as
f(k), k =
✓k1k2
◆2 Z2
f(R,⇥), R 2 Z+,⇥ 2 [0, 2⇡)
(k1, k2) R2
✓x1
x2
◆=
✓�x1 · k1�x2 · k2
◆
x1
x2 k2
k1
)
�x1
�x2
R2 Z2
·f(x) f(k)
f(x), x =
✓x1
x2
◆2 R2
, f(x)F�! f(!) =
Z
R2
f(x)e�jh!,xidx, ! 2 R2
f(r, ✓), r 2 R+, ✓ 2 [0, 2⇡), f(r, ✓)F�! f(⇢,�), ⇢ 2 R+,� 2 [0, 2⇡)
NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS
12
• 3-D continuous texture functions in space and Fourier
• Cartesian coordinates:
• Polar coordinates:
• 3-D digital texture functions
• Cartesian coordinates:
• Polar coordinates:
• Sampling (Cartesian)
• Increments in corresponds to physical displacements in as
x1
x2
k2k1
)�x1�x2
·f(x) f(k)
f(k), k =
0
@k1k2k3
1
A 2 Z3
f(r, ✓,�), r 2 R+, ✓ 2 [0, 2⇡),� 2 [0, 2⇡)
f(R,⇥,�), R 2 Z+,⇥ 2 [0, 2⇡),� 2 [0, 2⇡)
(k1, k2, k3) R3
0
@x1
x2
x3
1
A =
0
@�x1 · k1�x2 · k2�x3 · k3
1
A �x3
x3 k3R3 Z3
f(x), x =
0
@x1
x2
x3
1
A 2 R3, f(x)
F�! f(!) =
Z
R3
f(x)e�jh!,xidx, ! 2 R3
• We consider a texture function as a realization of a spatial stochastic process of where is the value at the spatial position indexed by
• The values of follow one or several probability density functions
• Examples
NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS
13
f(x)
Rd
{Xm,m 2 RM1⇥···⇥Md}
Xm m
moving average Gaussian pointwise Poisson biomedical: lung fibrosis in CT
m 2 R128⇥128 m 2 R32⇥32 m 2 R84⇥84
Xm fXm(q)
NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS
14
• Stationarity of spatial stochastic processes
• A spatial process is stationary if the probability density functions are equivalent for all
• Example: heteroscedastic moving average Gaussian process
{Xm,m 2 RM1⇥···⇥Md}mfXm(q)
stationarynon-stationary (strict sense)
fb,Xm(q) =1
3p2⇡
e�(q�0)2
2 32
fa fb
fa,Xm(q) =1
1p2⇡
e�(q�0)2
2 12
NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS
15
• Stationarity of textures and human perception / tissue biology
• Strict/weak process stationarity and texture class definition is not equivalent
• Image analysis tasks when textures are considered as
• Stationary (wide sense): texture classification
• Non-stationary: texture segmentation
Outex “canvas039”: stationary? brain glioblastoma in T1-MRI: stationary?
Fig. 3: Top: Texture mosaics. Bottom: Our method using γ = 0.14, 0.13, 0.12, and 0.8, respectively. Our method segments theleft and the right image almost perfectly. Even on the challenging central images, the major structures are segmented well.
ages2. We observe that the differently textured regions arenicely separated.
5. CONCLUSION
We have presented a novel approach to the segmentation oftextured images. We used feature vectors based on the am-plitude of monogenic curvelets. For the segmentation of thehigh-dimensional feature images, we used a fast computa-tional strategy for the Potts model. Tests carried out on syn-thetic texture images as well as on real color images show thepotential of our approach.
6. REFERENCES
[1] M. Yang, W. Moon, Y. Wang, M. Bae, C. Huang, J. Chen,and R. Chang, “Robust texture analysis using multi-resolutiongray-scale invariant features for breast sonographic tumor di-agnosis,” IEEE Transactions on Medical Imaging, vol. 32, no.12, pp. 2262–2273, 2013.
[2] A. Depeursinge, A. Foncubierta-Rodriguez, D. Van De Ville,and H. Müller, “Three-dimensional solid texture analysis inbiomedical imaging: Review and opportunities,” Medical Im-age Analysis, vol. 18, no. 1, pp. 176 – 196, 2014.
[3] J. Shotton, J. Winn, C. Rother, and A. Criminisi, “Textonboostfor image understanding: Multi-class object recognition andsegmentation by jointly modeling texture, layout, and context,”
2The images were taken from titanic-magazin.de, zastavki.com,123rf.com.
International Journal of Computer Vision, vol. 81, no. 1, pp. 2–23, 2009.
[4] M. Pietikäinen, Computer vision using local binary patterns,Springer London, 2011.
[5] R. Haralick, “Statistical and structural approaches to texture,”Proceedings of the IEEE, vol. 67, no. 5, pp. 786–804, 1979.
[6] A. Jain and F. Farrokhnia, “Unsupervised texture segmentationusing Gabor filters,” Pattern Recognition, vol. 24, no. 12, pp.1167–1186, 1991.
[7] M. Unser, “Texture classification and segmentation usingwavelet frames,” IEEE Transactions on Image Processing, vol.4, no. 11, pp. 1549–1560, 1995.
[8] T. Randen and J. Husoy, “Filtering for texture classification:A comparative study,” IEEE Transactions on Pattern Analysisand Machine Intelligence, vol. 21, no. 4, pp. 291–310, 1999.
[9] M. Rousson, T. Brox, and R. Deriche, “Active unsupervisedtexture segmentation on a diffusion based feature space,” inProceedings of the IEEE Conference on Computer Vision andPattern Recognition, Madison, 2003, vol. 2, pp. II–699.
[10] M. Ozden and E. Polat, “Image segmentation using color andtexture features,” in Proceedings of the 13th European SignalProcessing Conference, Antalya, 2005, pp. 2226–2229.
[11] J. Santner, M. Unger, T. Pock, C. Leistner, A. Saffari, andH. Bischof, “Interactive texture segmentation using randomforests and total variation,” in British Machine Vision Confer-ence, 2009, pp. 1–12.
[12] S. Geman and D. Geman, “Stochastic relaxation, Gibbs distri-butions, and the Bayesian restoration of images,” IEEE Trans-actions on Pattern Analysis and Machine Intelligence, vol. 6,no. 6, pp. 721–741, 1984.
ICIP 20144351
Fig. 3: Top: Texture mosaics. Bottom: Our method using γ = 0.14, 0.13, 0.12, and 0.8, respectively. Our method segments theleft and the right image almost perfectly. Even on the challenging central images, the major structures are segmented well.
ages2. We observe that the differently textured regions arenicely separated.
5. CONCLUSION
We have presented a novel approach to the segmentation oftextured images. We used feature vectors based on the am-plitude of monogenic curvelets. For the segmentation of thehigh-dimensional feature images, we used a fast computa-tional strategy for the Potts model. Tests carried out on syn-thetic texture images as well as on real color images show thepotential of our approach.
6. REFERENCES
[1] M. Yang, W. Moon, Y. Wang, M. Bae, C. Huang, J. Chen,and R. Chang, “Robust texture analysis using multi-resolutiongray-scale invariant features for breast sonographic tumor di-agnosis,” IEEE Transactions on Medical Imaging, vol. 32, no.12, pp. 2262–2273, 2013.
[2] A. Depeursinge, A. Foncubierta-Rodriguez, D. Van De Ville,and H. Müller, “Three-dimensional solid texture analysis inbiomedical imaging: Review and opportunities,” Medical Im-age Analysis, vol. 18, no. 1, pp. 176 – 196, 2014.
[3] J. Shotton, J. Winn, C. Rother, and A. Criminisi, “Textonboostfor image understanding: Multi-class object recognition andsegmentation by jointly modeling texture, layout, and context,”
2The images were taken from titanic-magazin.de, zastavki.com,123rf.com.
International Journal of Computer Vision, vol. 81, no. 1, pp. 2–23, 2009.
[4] M. Pietikäinen, Computer vision using local binary patterns,Springer London, 2011.
[5] R. Haralick, “Statistical and structural approaches to texture,”Proceedings of the IEEE, vol. 67, no. 5, pp. 786–804, 1979.
[6] A. Jain and F. Farrokhnia, “Unsupervised texture segmentationusing Gabor filters,” Pattern Recognition, vol. 24, no. 12, pp.1167–1186, 1991.
[7] M. Unser, “Texture classification and segmentation usingwavelet frames,” IEEE Transactions on Image Processing, vol.4, no. 11, pp. 1549–1560, 1995.
[8] T. Randen and J. Husoy, “Filtering for texture classification:A comparative study,” IEEE Transactions on Pattern Analysisand Machine Intelligence, vol. 21, no. 4, pp. 291–310, 1999.
[9] M. Rousson, T. Brox, and R. Deriche, “Active unsupervisedtexture segmentation on a diffusion based feature space,” inProceedings of the IEEE Conference on Computer Vision andPattern Recognition, Madison, 2003, vol. 2, pp. II–699.
[10] M. Ozden and E. Polat, “Image segmentation using color andtexture features,” in Proceedings of the 13th European SignalProcessing Conference, Antalya, 2005, pp. 2226–2229.
[11] J. Santner, M. Unger, T. Pock, C. Leistner, A. Saffari, andH. Bischof, “Interactive texture segmentation using randomforests and total variation,” in British Machine Vision Confer-ence, 2009, pp. 1–12.
[12] S. Geman and D. Geman, “Stochastic relaxation, Gibbs distri-butions, and the Bayesian restoration of images,” IEEE Trans-actions on Pattern Analysis and Machine Intelligence, vol. 6,no. 6, pp. 721–741, 1984.
ICIP 20144351
)[Storath2014]
OUTLINE
• Biomedical texture analysis: background
• Defining texture processes
• Notations, sampling and texture functions
• Texture operators, primitives and invariances
• Multiscale analysis
• Operator scale and uncertainty principle
• Region of interest and operator aggregation
• Multidirectional analysis
• Isotropic versus directional operators
• Importance of the local organization of image directions
• Conclusions
• References
REFERENCES
66
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2015 1
Steerable Wavelet Machines (SWM): LearningMoving Frames for Texture Classification
Adrien Depeursinge, Zsuzsanna Püspöki, John Paul Ward, and Michael Unser
Abstract—We present texture operators encoding class-specific local organizations of image directions (LOID) in arotation-invariant fashion. The operators are based on steerablecircular harmonic wavelets (CHW), offering a rich and yet com-pact initial representation for characterizing natural textures.We enforce the preservation of the joint location and orientationimage structure by building Cartan moving frames (MF) fromlocally-steered multi-order CHWs. In a second step, we usesupport vector machines (SVM) to learn a multi-class shapingmatrix of the initial CHW representations, yielding data-drivenMFs. Intuitively, the learned MFs can be seen as class-specificforged detectors that are applied and rotated at each point ofthe image to evaluate the magnitude of their responses, i.e., thepresence of the corresponding texture class. We experimentallydemonstrate the effectiveness of the proposed operators forclassifying natural textures, which outperformed the reportedperformance of recent approaches on several test suites of theOutex database.
Index Terms—Texture classification, feature learning, mov-ing frames, support vector machines, steerability, rotation-invariance, illumination-invariance, wavelet analysis.
I. INTRODUCTION
THe wealth of digital texture patterns is tightly relatedto the size of the observation window on a discrete
lattice. In an extreme case, an image region composedof one pixel cannot form geometrical structures. There-fore, the characterization of textural patterns often requiresintegrating image operators g (x), x 2 R2 over an imagedomain X . The latter raises two major challenges. First,the action of the integrated operators becomes diffuseover X , which hinders the spatial precision of texturesegmentation approaches. Second, the effect of integrationbecomes even more destructive when unidirectional opera-tors are jointly used to characterize the local organization ofimage directions (LOID) (e.g., directional filterbanks [1, 2],curvelets [3], histogram of gradients (HOG) [4], Haralick [5]).When separately integrated, the responses of unidirectionalindividual operators are not local anymore and their jointresponses become only sensitive to the global amount ofimage directions in the region X . For instance, the joint
responses of image gradients g1,2(x) =≥ØØ @ f (x)
@x1
ØØ,ØØ @ f (x)@x2
ØØ¥
arenot able to discriminate between the two textures classesf1(x) and f2(x) shown in Fig. 1 when integrated over theimage domain X .
This loss of information is detailed by Sifre et al. interms of separable group invariants [6]. When integrated
The authors are with the Biomedical Imaging Group, École Poly-technique Fédérale de Lausanne, Lausanne 1015, Switzerland (e-mail:[email protected]; [email protected]). A. Depeursinge isalso with the MedGIFT group, Institute of Information Systems, Universityof Applied Sciences Western Switzerland, Sierre (HES-SO).
of1(x) : of2(x) :
RX
ØØ @ f (x)@x1
ØØdx
R XØ Ø@
f(x)
@x 2
Ø Ø dx
image gradients
Fig. 1. The joint responses of image gradients≥| @ f (x)@x1
|, | @ f (x)@x2
|¥
are notable to discriminate between the two textures classes f1(x) and f2(x)when integrated over the image domain X . One circle in the gradientrepresentation corresponds to one realization (i.e., full image) of f1,2.
separately, the responses of unidirectional operators be-come invariant to a larger family of roto-translations wheredifferent orientations are translated by different values. Forinstance, it can be observed in Fig. 1 that f2 can be obtainedfrom f1 by vertically translating horizontal bars only andhorizontally translating vertical bars only.
The LOID are key for visual understanding [7] and is theonly property able to differentiate between f1 and f2 inFig. 1. It relates to the joint information between positionsand orientations in images, which is discarded by operatorsthat are invariant to separable groups. The LOID has beenleveraged in the literature to define [8] and discriminatetexture classes [9–12]. It is also at the origin of the successof methods based on local binary patterns (LBP) [9] andits extensions [13–19], which specifically encodes the LOIDwith uniform circular pixel sequences. Although classicaldeep learning approaches are not enforcing the characteri-zation of the LOID [20], deep convolutional networks wererecently proposed by Oyallon et al. to specifically preservethe structure of the roto-translation group [21].
An even bigger challenge is to design texture operatorsthat can characterize the LOID in a rotation-invariantfashion [9, 11]. The latter is very important to recognizeimage structures independently from both their local ori-entations and the global orientation of the image. Exam-ples of such structures are vascular, bronchial or dendritictrees in biomedical images, river deltas or urban areas insatellite images, or even more complex biomedical tissuearchitectures or crystals in petrographic analysis. Since mostof the above-mentioned imaging modalities yield imageswith pixel sizes defined in physical dimensions, we arenot interested in invariance to image scale. We aim todesign texture operators that are invariant to the family ofEuclidean transforms (also called rigid), i.e.,
g (x) = g (Rµ0 x °x0), 8x , x0,µ0 2R2,
·m
TEXTURE OPERATORS AND PRIMITIVES
17
• Texture operators
• A -dimensional texture analysis approach is characterized by a set of local operators centered at the position
• is local in the sense that each element only depends on a subregion of
• The subregion is the support of
• can be linear (e.g., wavelets) or non-linear (e.g., median, GLCMs, LBPs)
• For each position , maps the texture function into a -dimensional space
Nd
f(x)
xL1 ⇥ · · ·⇥ Ld
L1
L2
M1
M2
·
m
gn(x,m) : RM1⇥···⇥Md 7! RP , n = 1, . . . , N�gn(x,m)
�p=1,...,P
gn
m
gn
m gnP
gn(f(x),m) : RM1⇥···⇥Md 7! RP
L1 ⇥ · · ·⇥ Ld gn
TEXTURE OPERATORS AND PRIMITIVES
18
• From texture operators to texture measurements (i.e., features)
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region of interest
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
TEXTURE OPERATORS AND PRIMITIVES
• Texture primitives
• A “texture primitive” (also called “texel”) is a fundamental elementary unit (i.e., a building block) of a texture class [Haralick1979, Petrou2006]
• Intuitively, given a collection of texture functions , an appropriate set of texture operators must be able to:
(i) Detect and quantify the presence of all distinct primitives in
(ii) Characterize the spatial relationships between the primitives (e.g., geometric transformations, density) when aggregated
texture primitives
,
19
texture primitive
,
fj
�
f1 f2 primitive
,
primitive
,
fj=1,...,J
gn
TEXTURE OPERATORS AND PRIMITIVES
• General-purpose texture operators
• In general, the texture primitives are neither well-defined, nor known in advance (e.g., biomedical tissue)
• General-purpose operator sets are useful to estimate the primitives
• How to build such operator sets?
20
tissue in HRCT data with optimized SVMs. Lung tissue texture classificationusing co-occurence matrices, Gabor filters and Tamura texture features was in-vestigated in [15]. The classification of regions of interest (ROIs) delineated bythe user consitutes the intial steps towards automatic detection of abnormal lungtissue patterns in the whole HRCT volume.
2 Methods
The dataset used is part of an internal multimedia database of ILD cases con-taining HRCT images with annotated ROIs created in the Talisman project1.843 ROIs from healthy and five pathologic lung tissue patterns are selected fortraining and testing the classifiers selecting classes with sufficiently high repre-sentation (see Table 1).
The wavelet frame decompositions with dyadic and quincunx subsamplingare implemented in Java [11, 16] as well as optimization of SVMs. The basicimplementation of the SVMs is taken from the open source Java library Weka2.
Table 1. Visual aspect and distribution of the ROIs per class of lung tissue pattern.
visualaspect
class healthy emphysema ground glass fibrosis micronodules macronodules
# of ROIs 113 93 148 312 155 22# of patients 11 6 14 28 5 5
3 Results
3.1 Isotropic Polyharmonic B–Spline Wavelets
As mentioned in Section 1.1, isotropic analysis is preferable for lung texturecharacterization. The Laplacian operator plays an important role in image pro-cessing and is clearly isotropic. Indeed, ∆ = ∂2
∂x2
1
+ ∂2
∂x2
2
, is rotationally invariant.
The polyharmonic B–spline wavelets implement a multiscale smoothed versionof the Laplacian [16]. This wavelet, at the first decomposition level, can be char-acterized as
ψγ(D−1x) = ∆γ
2 {φ} (x), (1)
1 TALISMAN: Texture Analysis of Lung ImageS for Medical diagnostic AssistaNce,http://www.sim.hcuge.ch/medgift/01 Talisman EN.htm
2 http://www.cs.waikato.ac.nz/ml/weka/
2-D lung tissue in CT images [Depeursinge2012a]
3-D normal and osteoporotic bone in CT [Dumas2009]µ
• General-purpose texture operators
• The exhaustive analysis of spatial scales and directions is computationally expensive when the support of the operators is large: choices are required
• Directions (e.g., GLCMs, RLE, HOG)
• Scales
the classification accuracy of RLE versus GLCMs for categorizinglung tissue patterns associated with diffuse lung disease in HRCT.Using identical choices of the directions for RLE and GLCMs, theyfound found no statistical differences between the classificationperformance. Gao et al. (2010) and Qian et al. (2011) comparedthe performance of three-dimensional GLCM, LBP, Gabor filtersand WT in retrieving similar MR images of the brain. They ob-served a small increase in retrieval performance for LBP and GLCMwhen compared to Gabor filters and WT. However, the databaseused is rather small and the results might not be statisticallysignificant.
Several papers compared the performance of texture analysisalgorithms in their 2-D versus 3-D forms. As expected, 2-D textureanalysis is most often less discriminative than 3-D, which was ob-served for various applications and techniques, such as:
! GLCMs, RLE and fractal dimension for the classification of lungtissue types in HRCT in Xu et al. (2005, 2006b).
! GLCMs for the classification of brain tumors in MRI in Mah-moud-Ghoneim et al. (2003) and Allin Christe et al. (2012).
! GLCMs for the classification of breast in contrast–enhancedMRI, where statistical significance was assessed in Chen et al.(2007).
! GMRF for the segmentation of gray matter in MRI in Ranguelovaand Quinn (1999).
! LBP for synthetic texture classification in Paulhac et al. (2008).
This demonstrates that 2-D slice-based discrimination of 3-Dnative texture does not allow fully exploiting the informationavailable in 3-D datasets. An exception was observed with 2-D ver-sus 3-D WTs in Jafari-Khouzani et al. (2004), where the 2-D ap-proach showed a small increase in classification performance ofabnormal regions responsible for temporal lobe epilepsy. A separa-ble 3-D WT was used, which did not allow to adequately exploitthe 3-D texture information available and may explain the ob-served results.
7. Discussion
In the preceding sections, we have reviewed the current state-of-the-art in 3-D biomedical texture analysis. The papers were cat-egorized in terms of imaging modality used, organ studied and im-age processing techniques. The increasing number of papers overthe past 32 years clearly shows a growing interest in computerizedcharacterization of three-dimensional texture information (seeFig. 5). This is a consequence of increasingly available 3-D dataacquisition devices that are reaching high spatial resolutionsallowing to capture tissue properties in its natural space.
The analysis of the medical applications in 100 papers in Sec-tion 5 shows the diversity of 3-D biomedical textures. The variousgeometrical properties of the textures are summarized in Table 2,which defines the multiple challenges of 3-D texture analysis.The need for methods able to characterize structural scales and
Fig. 15. Amounts of pixels/voxels covered by the main 13 directions. The number N of discarded pixels/voxels (gray regions in (a) and (b)) according to the neighborhood sizeR is shown in (c).
A. Depeursinge et al. /Medical Image Analysis 18 (2014) 176–196 191
the classification accuracy of RLE versus GLCMs for categorizinglung tissue patterns associated with diffuse lung disease in HRCT.Using identical choices of the directions for RLE and GLCMs, theyfound found no statistical differences between the classificationperformance. Gao et al. (2010) and Qian et al. (2011) comparedthe performance of three-dimensional GLCM, LBP, Gabor filtersand WT in retrieving similar MR images of the brain. They ob-served a small increase in retrieval performance for LBP and GLCMwhen compared to Gabor filters and WT. However, the databaseused is rather small and the results might not be statisticallysignificant.
Several papers compared the performance of texture analysisalgorithms in their 2-D versus 3-D forms. As expected, 2-D textureanalysis is most often less discriminative than 3-D, which was ob-served for various applications and techniques, such as:
! GLCMs, RLE and fractal dimension for the classification of lungtissue types in HRCT in Xu et al. (2005, 2006b).
! GLCMs for the classification of brain tumors in MRI in Mah-moud-Ghoneim et al. (2003) and Allin Christe et al. (2012).
! GLCMs for the classification of breast in contrast–enhancedMRI, where statistical significance was assessed in Chen et al.(2007).
! GMRF for the segmentation of gray matter in MRI in Ranguelovaand Quinn (1999).
! LBP for synthetic texture classification in Paulhac et al. (2008).
This demonstrates that 2-D slice-based discrimination of 3-Dnative texture does not allow fully exploiting the informationavailable in 3-D datasets. An exception was observed with 2-D ver-sus 3-D WTs in Jafari-Khouzani et al. (2004), where the 2-D ap-proach showed a small increase in classification performance ofabnormal regions responsible for temporal lobe epilepsy. A separa-ble 3-D WT was used, which did not allow to adequately exploitthe 3-D texture information available and may explain the ob-served results.
7. Discussion
In the preceding sections, we have reviewed the current state-of-the-art in 3-D biomedical texture analysis. The papers were cat-egorized in terms of imaging modality used, organ studied and im-age processing techniques. The increasing number of papers overthe past 32 years clearly shows a growing interest in computerizedcharacterization of three-dimensional texture information (seeFig. 5). This is a consequence of increasingly available 3-D dataacquisition devices that are reaching high spatial resolutionsallowing to capture tissue properties in its natural space.
The analysis of the medical applications in 100 papers in Sec-tion 5 shows the diversity of 3-D biomedical textures. The variousgeometrical properties of the textures are summarized in Table 2,which defines the multiple challenges of 3-D texture analysis.The need for methods able to characterize structural scales and
Fig. 15. Amounts of pixels/voxels covered by the main 13 directions. The number N of discarded pixels/voxels (gray regions in (a) and (b)) according to the neighborhood sizeR is shown in (c).
A. Depeursinge et al. /Medical Image Analysis 18 (2014) 176–196 191
TEXTURE OPERATORS AND PRIMITIVES
21
2-D 3-D (one quadrant) such that
num
ber o
f dis
card
ed
pixe
ls/v
oxel
sof order −1/2 (an isotropic smoothing operator) of f : Rf =−∇∆−1/2f . Let’s indeed recall the Fourier-domain definition ofthese operators: ∇ F←→ jω and ∆−1/2 F←→ ||ω||−1. Unlike theusual gradient ∇, the Riesz transform is self-reversible
!R⋆Rf(ω) =(jω)∗(jω)
||ω||2 f(ω) = f(ω).
This allows us to define a self-invertible wavelet frame of L2(R3)(tight frame). We however see that there exists a singularity for thefrequency (0, 0, 0). This issue will be fixed later, thanks to the van-ishing moments of the primary wavelet transform.
2.2. Steerability
The interpretation of the Riesz transform as being a directionalderivative filterbank makes its steerability easy to understand: itbehaves similarly to a steerable gradient filterbank, with the addedcrucial property of perfect reconstruction. We parameterize any ro-tation in 3D with a real and unique 3 by 3 matrix U which is unitary(UTU = I). Let us consider the Fourier transform of the impulseresponse of the Riesz transform after a rotation by U as
!R{δ}(Ux)(ω) = −jUω
||Uω|| = U
!−j
ω||ω||
"= U !R{δ}(x)(ω),
with δ the Dirac distribution. The rotated Riesz transform of f there-fore corresponds to the multiplication by U of the non-rotated Rieszcoefficients
RUf(x) = URf(x), (2)
which demonstrates the 3D steerability of the Riesz transform.
2.3. Riesz-Wavelet Pyramid
One crucial property of the Riesz transform is its ability to map anyframe of L2(R3) (in particular wavelet frames) into L2(R3) since itpreserves the inner product of L2(R3) [3, 4]. Following the previousRiesz-wavelet constructions [3, 4], we propose to apply the 3D Riesztransform to the coefficients of a wavelet pyramid to build a steerablewavelet transform in 3D.
2.3.1. Primary Wavelet pyramid
A primary isotropic wavelet pyramid is required in order to pre-serve the relevance of the directional analysis performed by the Riesztransform. Moreover, the bandlimitedness of the wavelet bands mustbe enforced to ensure the isotropy of the primary wavelet togetherwith the possibility of down-sampling [1, 8]. A conventional or-thogonal and separable wavelet transform fulfills none of these con-ditions. In [3], a 2D spline-based wavelet transform was used asthe primary transform. However, while low-order spline waveletsare fast to compute, they are not truly isotropic. We thus proposeinstead a 3D non-separable wavelet with an isotropic wavelet func-tion, as done in 2D in [2]. To achieve bandlimitedness of the waveletbands it is more convenient to design the wavelet transform directlyin the 3D Fourier domain. Moreover, the isotropy constraint im-poses a purely radial wavelet function (i.e., it depends on ||ω|| andnot on the individual frequency components ωi in the Fourier do-main). Among all possible wavelet functions, two are of particularinterest: the Shannon’s wavelet
ψsha(ω) =
#1, π
2 ≤ ||ω|| ≤ π0, otherwise
Fig. 1. Frequency tiling with the Shannon’s wavelet. Each waveletscale is obtained by a bandpass filter of support [π/2k+1,π/2k]. Thespace-domain subsampling operations, which restrict the frequencyplane to the support of each wavelet function, are shown with boxes.
� �
(a) Filterbank implementation of the isotropic wavelet transform. A cas-cade of low-pass filters (Li(ω)) and high-pass filters (H0(ω) and B(ω))is applied. The filterbank is self-reversible.
(b) Self-reversible Riesz-wavelet filterbank.
Fig. 2. The Riesz-wavelet transform filterbank implementation.
and the Simoncelli’s wavelet used for the 2D steerable pyramid
ψsim(ω) =
$cos
%π2 log2
%2||ω||
π
&&, π
4 < ||ω|| ≤ π
0, otherwise.
The Shannon’s wavelet function is a radial step function which corre-sponds to the frequency-domain tiling shown in Fig. 1. This wavelettransform decomposes the signal spectrum with isotropic and non-overlapping tiles. Using the Simoncelli’s wavelet function would re-sult in a smooth frequency partitioning with overlapping tiles, whichis less prone to reconstruction artifacts after coefficient processing.The decomposition shown in Figure 1 can be efficiently achievedby a succession of filtering and downsampling operations, the high-pass coefficient remaining non-subsampled to alleviate aliasing, asopposed to the orthogonal wavelet transform. The wavelet decom-position cascade is illustrated in Figure 2(a).
2.3.2. Riesz-Wavelet Pyramid
We build a Riesz-wavelet frame by applying the Riesz transform toeach scale of the isotropic pyramid defined by the wavelet functionand its dual {ψ, ψ}. The continuous version of the Riesz-wavelettransform prior subsampling is
qk(x) = R{ψk ∗ f}(x)
2133
2-D GLCMs with various spatial distances [Haralick1979]2-D isotropic dyadic wavelets
in Fourier [Chenouard2011]
d = 1 (�k1 = 1,�k2 = 0) d = 2 (�k1 = 2,�k2 = 0) d = 2p2 (�k1 = 2,�k2 = 2)
r
L1 ⇥ · · ·⇥ Ld
L1 = L2 = L3 = 2r + 1
d
g(f(x),m) = g(f(R✓0x� x0),m), 8x0 2 R2, ✓0 2 [0, 2⇡)
• Invariances of the texture operators to geometric transformations can be desirable
• E.g., scaling, rotations and translations
• Invariances of texture operators can be enforced
• Example with 2-D Euclidean transforms (i.e., rotation and translation)with the rotation matrix
INVARIANCE OF TEXTURE OPERATORS
22
R✓0 =
✓cos ✓0 � sin ✓0sin ✓0 cos ✓0
◆
INVARIANCE OF TEXTURE OPERATORS
23
• Computer vision versus biomedical imaging
Computer vision Biomedical image analysis
translation translation-invariant translation-invariant
rotation rotation-invariant rotation-invariant
scale scale-invariant multi-scale
160 IEEE REVIEWS IN BIOMEDICAL ENGINEERING, VOL. 2, 2009
Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representations of tissue architecture via Delaunay Triangulation, VoronoiDiagram, and Minimum Spanning tree.
Fig. 11. Digitized histological image at successively higher scales (magnifica-tions) yields incrementally more discriminatory information in order to detectsuspicious regions.
or resolution. For instance at low or coarse scales color or tex-ture cues are commonly used and at medium scales architec-tural arrangement of individual histological structures (glandsand nuclei) start to become resolvable. It is only at higher res-olutions that morphology of specific histological structures canbe discerned.
In [93], [94], a multiresolution approach has been used for theclassification of high-resolution whole-slide histopathology im-ages. The proposed multiresolution approach mimics the eval-uation of a pathologist such that image analysis starts from thelowest resolution, which corresponds to the lower magnificationlevels in a microscope and uses the higher resolution represen-tations for the regions requiring more detailed information fora classification decision. To achieve this, images were decom-posed into multiresolution representations using the Gaussianpyramid approach [95]. This is followed by color space con-version and feature construction followed by feature extractionand feature selection at each resolution level. Once the classifieris confident enough at a particular resolution level, the systemassigns a classification label (e.g., stroma-rich, stroma-poor orundifferentiated, poorly differentiating, differentiating) to theimage tile. The resulting classification map from all image tilesforms the final classification map. The classification of a whole-slide image is achieved by dividing into smaller image tiles andprocessing each image tile independently in parallel on a clusterof computer nodes.
As an example, refer to Fig. 11, showing a hierarchicalcascaded scheme for detecting suspicious areas on digitizedprostate histopathology slides as presented in [96].
Fig. 12 shows the results of a hierarchical classifier for detec-tion of prostate cancer from digitized histopathology. Fig. 12(a)
Fig. 12. Results from the hierarchical machine learning classifier. (a) Originalimage with the tumor region (ground truth) in black contour, (b) results at scale1, (c) results at scale 2, and (d) results at scale 3. Note that only areas determinedas suspicious at lower scales are considered for further analysis at higher scales.
shows the original image with tumor outlined in black. The nextthree columns show the classifier results at increasing analysisscales. Pixels classified as “nontumor” at a lower magnification(scale) are discarded at the subsequent higher scale, reducingthe number of pixels needed for analysis at higher scales. Ad-ditionally, the presence of more discriminating information athigher scales allows the classifier to better distinguish betweentumor and nontumor pixels.
At lower resolutions of histological imagery, textural analysisis commonly used to capture tissue architecture, i.e., the overallpattern of glands, stroma and organ organization. For each digi-tized histological image several hundred corresponding featurescenes can be generated. Texture feature values are assignedto every pixel in the corresponding image. 3-D statistical, gra-dient, and Gabor filters can be extracted in order to analyzethe scale, orientation, and anisotropic information of the re-gion of interest. Filter operators are applied in order to extractfeatures within local neighborhoods centered at every spatiallocation. At medium resolution, architectural arrangement ofnuclei within each cancer grade can be described via severalgraph-based algorithms. At higher resolutions, nuclei and themargin and boundary appearance of ductal and glandular struc-tures have proved to be of discriminatory importance. Many ofthese features are summarized in Tables I and II.
D. Feature Selection, Dimensionality Reduction,and Manifold Learning
1) Feature Selection: While humans have innate abilities toprocess and understand imagery, they do not tend to excel at
COMPUTERIZED TEXTURE ANALYSIS
7
• Invariances: computer vision versus biomedical imaging
Computer vision Biomedical image analysis
scale scale-invariant multi-scale
rotation rotation-invariant rotation-invariant
[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
160 IEEE REVIEWS IN BIOMEDICAL ENGINEERING, VOL. 2, 2009
Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representations of tissue architecture via Delaunay Triangulation, VoronoiDiagram, and Minimum Spanning tree.
Fig. 11. Digitized histological image at successively higher scales (magnifica-tions) yields incrementally more discriminatory information in order to detectsuspicious regions.
or resolution. For instance at low or coarse scales color or tex-ture cues are commonly used and at medium scales architec-tural arrangement of individual histological structures (glandsand nuclei) start to become resolvable. It is only at higher res-olutions that morphology of specific histological structures canbe discerned.
In [93], [94], a multiresolution approach has been used for theclassification of high-resolution whole-slide histopathology im-ages. The proposed multiresolution approach mimics the eval-uation of a pathologist such that image analysis starts from thelowest resolution, which corresponds to the lower magnificationlevels in a microscope and uses the higher resolution represen-tations for the regions requiring more detailed information fora classification decision. To achieve this, images were decom-posed into multiresolution representations using the Gaussianpyramid approach [95]. This is followed by color space con-version and feature construction followed by feature extractionand feature selection at each resolution level. Once the classifieris confident enough at a particular resolution level, the systemassigns a classification label (e.g., stroma-rich, stroma-poor orundifferentiated, poorly differentiating, differentiating) to theimage tile. The resulting classification map from all image tilesforms the final classification map. The classification of a whole-slide image is achieved by dividing into smaller image tiles andprocessing each image tile independently in parallel on a clusterof computer nodes.
As an example, refer to Fig. 11, showing a hierarchicalcascaded scheme for detecting suspicious areas on digitizedprostate histopathology slides as presented in [96].
Fig. 12 shows the results of a hierarchical classifier for detec-tion of prostate cancer from digitized histopathology. Fig. 12(a)
Fig. 12. Results from the hierarchical machine learning classifier. (a) Originalimage with the tumor region (ground truth) in black contour, (b) results at scale1, (c) results at scale 2, and (d) results at scale 3. Note that only areas determinedas suspicious at lower scales are considered for further analysis at higher scales.
shows the original image with tumor outlined in black. The nextthree columns show the classifier results at increasing analysisscales. Pixels classified as “nontumor” at a lower magnification(scale) are discarded at the subsequent higher scale, reducingthe number of pixels needed for analysis at higher scales. Ad-ditionally, the presence of more discriminating information athigher scales allows the classifier to better distinguish betweentumor and nontumor pixels.
At lower resolutions of histological imagery, textural analysisis commonly used to capture tissue architecture, i.e., the overallpattern of glands, stroma and organ organization. For each digi-tized histological image several hundred corresponding featurescenes can be generated. Texture feature values are assignedto every pixel in the corresponding image. 3-D statistical, gra-dient, and Gabor filters can be extracted in order to analyzethe scale, orientation, and anisotropic information of the re-gion of interest. Filter operators are applied in order to extractfeatures within local neighborhoods centered at every spatiallocation. At medium resolution, architectural arrangement ofnuclei within each cancer grade can be described via severalgraph-based algorithms. At higher resolutions, nuclei and themargin and boundary appearance of ductal and glandular struc-tures have proved to be of discriminatory importance. Many ofthese features are summarized in Tables I and II.
D. Feature Selection, Dimensionality Reduction,and Manifold Learning
1) Feature Selection: While humans have innate abilities toprocess and understand imagery, they do not tend to excel at
[Gurcan2009][Lazebnik2005]
OUTLINE
• Biomedical texture analysis: background
• Defining texture processes
• Notations, sampling and texture functions
• Texture operators, primitives and invariances
• Multiscale analysis
• Operator scale and uncertainty principle
• Region of interest and operator aggregation
• Multidirectional analysis
• Isotropic versus directional operators
• Importance of the local organization of image directions
• Conclusions
• References
REFERENCES
66
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2015 1
Steerable Wavelet Machines (SWM): LearningMoving Frames for Texture Classification
Adrien Depeursinge, Zsuzsanna Püspöki, John Paul Ward, and Michael Unser
Abstract—We present texture operators encoding class-specific local organizations of image directions (LOID) in arotation-invariant fashion. The operators are based on steerablecircular harmonic wavelets (CHW), offering a rich and yet com-pact initial representation for characterizing natural textures.We enforce the preservation of the joint location and orientationimage structure by building Cartan moving frames (MF) fromlocally-steered multi-order CHWs. In a second step, we usesupport vector machines (SVM) to learn a multi-class shapingmatrix of the initial CHW representations, yielding data-drivenMFs. Intuitively, the learned MFs can be seen as class-specificforged detectors that are applied and rotated at each point ofthe image to evaluate the magnitude of their responses, i.e., thepresence of the corresponding texture class. We experimentallydemonstrate the effectiveness of the proposed operators forclassifying natural textures, which outperformed the reportedperformance of recent approaches on several test suites of theOutex database.
Index Terms—Texture classification, feature learning, mov-ing frames, support vector machines, steerability, rotation-invariance, illumination-invariance, wavelet analysis.
I. INTRODUCTION
THe wealth of digital texture patterns is tightly relatedto the size of the observation window on a discrete
lattice. In an extreme case, an image region composedof one pixel cannot form geometrical structures. There-fore, the characterization of textural patterns often requiresintegrating image operators g (x), x 2 R2 over an imagedomain X . The latter raises two major challenges. First,the action of the integrated operators becomes diffuseover X , which hinders the spatial precision of texturesegmentation approaches. Second, the effect of integrationbecomes even more destructive when unidirectional opera-tors are jointly used to characterize the local organization ofimage directions (LOID) (e.g., directional filterbanks [1, 2],curvelets [3], histogram of gradients (HOG) [4], Haralick [5]).When separately integrated, the responses of unidirectionalindividual operators are not local anymore and their jointresponses become only sensitive to the global amount ofimage directions in the region X . For instance, the joint
responses of image gradients g1,2(x) =≥ØØ @ f (x)
@x1
ØØ,ØØ @ f (x)@x2
ØØ¥
arenot able to discriminate between the two textures classesf1(x) and f2(x) shown in Fig. 1 when integrated over theimage domain X .
This loss of information is detailed by Sifre et al. interms of separable group invariants [6]. When integrated
The authors are with the Biomedical Imaging Group, École Poly-technique Fédérale de Lausanne, Lausanne 1015, Switzerland (e-mail:[email protected]; [email protected]). A. Depeursinge isalso with the MedGIFT group, Institute of Information Systems, Universityof Applied Sciences Western Switzerland, Sierre (HES-SO).
of1(x) : of2(x) :
RX
ØØ @ f (x)@x1
ØØdx
R XØ Ø@
f(x)
@x 2
Ø Ø dx
image gradients
Fig. 1. The joint responses of image gradients≥| @ f (x)@x1
|, | @ f (x)@x2
|¥
are notable to discriminate between the two textures classes f1(x) and f2(x)when integrated over the image domain X . One circle in the gradientrepresentation corresponds to one realization (i.e., full image) of f1,2.
separately, the responses of unidirectional operators be-come invariant to a larger family of roto-translations wheredifferent orientations are translated by different values. Forinstance, it can be observed in Fig. 1 that f2 can be obtainedfrom f1 by vertically translating horizontal bars only andhorizontally translating vertical bars only.
The LOID are key for visual understanding [7] and is theonly property able to differentiate between f1 and f2 inFig. 1. It relates to the joint information between positionsand orientations in images, which is discarded by operatorsthat are invariant to separable groups. The LOID has beenleveraged in the literature to define [8] and discriminatetexture classes [9–12]. It is also at the origin of the successof methods based on local binary patterns (LBP) [9] andits extensions [13–19], which specifically encodes the LOIDwith uniform circular pixel sequences. Although classicaldeep learning approaches are not enforcing the characteri-zation of the LOID [20], deep convolutional networks wererecently proposed by Oyallon et al. to specifically preservethe structure of the roto-translation group [21].
An even bigger challenge is to design texture operatorsthat can characterize the LOID in a rotation-invariantfashion [9, 11]. The latter is very important to recognizeimage structures independently from both their local ori-entations and the global orientation of the image. Exam-ples of such structures are vascular, bronchial or dendritictrees in biomedical images, river deltas or urban areas insatellite images, or even more complex biomedical tissuearchitectures or crystals in petrographic analysis. Since mostof the above-mentioned imaging modalities yield imageswith pixel sizes defined in physical dimensions, we arenot interested in invariance to image scale. We aim todesign texture operators that are invariant to the family ofEuclidean transforms (also called rigid), i.e.,
g (x) = g (Rµ0 x °x0), 8x , x0,µ0 2R2,
·m
k2k1
�x1�x2f(k)�x3
k3
MULTISCALE TEXTURE OPERATORS
25
• Inter-patient and inter-dimension scale normalization
• Most medical imaging protocols yield images with various sampling steps
• Inter-patient scale normalization is required to ensure the correspondance of spatial frequencies
• Inter-dimension scale normalization is required to ensure isotropic scale/directions definition
(�x1,�x2,�x3)
�x1 = �x2 = 0.4mm �x1 = �x2 = 1.6mm
spatial Fourier Fourierspatial
0 0
0 0
⇡ ⇡�⇡ �⇡
�⇡ �⇡
�x1�x2
�x3 )�x
01�x
02
�x
03
d = 1 (�k1 = 1,�k2 = 0)
GLCMs
0.4mm
1.6mm
MULTISCALE TEXTURE OPERATORS
26
• Which scales for texture measurements?
• Two aspects: A. How to define the size(s) of the operator(s) ?
B. How to define the size of the region of interest ?
TEXTURE OPERATORS AND PRIMITIVES
19
• From texture operators to texture measurements
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
M
original image d
increasingly small
operator sizeL1 > L2 > · · · > LN
0
BBBBBBBBBBBBBBBBBBBBBBBB@
µ11...µ1P
µ21...µ2P
...
µN1...
µNP
1
CCCCCCCCCCCCCCCCCCCCCCCCA
= µM: concatenated measurements from multiscale operators aggregated over
Ln1 ⇥ · · ·⇥ Ln
d
gn=1,...,N (f(x),m)
M
f(x)
M
[Depeursinge2012b]
MULTISCALE TEXTURE OPERATORS
27
• Which scales for texture measurements?
• Two aspects: A. How to define the size(s) of the operator(s) ?
B. How to define the size of the region of interest ?
TEXTURE OPERATORS AND PRIMITIVES
19
• From texture operators to texture measurements
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
M
original image d
increasingly small
operator sizeL1 > L2 > · · · > LN
0
BBBBBBBBBBBBBBBBBBBBBBBB@
µ11...µ1P
µ21...µ2P
...
µN1...
µNP
1
CCCCCCCCCCCCCCCCCCCCCCCCA
= µM: concatenated measurements from multiscale operators aggregated over
Ln1 ⇥ · · ·⇥ Ln
d
gn=1,...,N (f(x),m)
M
f(x)
M
[Depeursinge2012b]
MULTISCALE TEXTURE OPERATORS
28
• Size and spectral coverage of operators
• Uncertainty principle: operators cannot be well located both in space and Fourier [Petrou2006]
• In 2D, the trade-off between the spatial support (i.e., ) and frequency support (i.e., ) of the operators is given by
L1 ⇥ L2
⌦1 ⇥ ⌦2
F�!
f(!)f(x)
L21 ⌦
21 L
22 ⌦
22 � 1
16
MULTISCALE TEXTURE OPERATORS
29
• Size and spectral coverage of operators
• Becomes a problem in the case of non-stationary texture:
,accurate spatial localization
poor spectrum characterization
Gaussian window:
� = 3.2mm
0 ⇡|!1|
F�!
MULTISCALE TEXTURE OPERATORS
30
• Size and spectral coverage of operators
• Becomes a problem in the case of non-stationary texture:
,accurate spatial localization
poor spectrum characterization
Gaussian window:
� = 3.2mm
Gaussian window:
� = 38.4mm
0 ⇡|!1|
F�!
MULTISCALE TEXTURE OPERATORS
31
• Size and spectral coverage of operators
• Becomes a problem in the case of non-stationary texture:
,accurate spatial localization
poor spectrum characterization
Gaussian window:
� = 3.2mm
Gaussian window:
� = 38.4mm
0 ⇡|!1|
F�!
The spatial support should have the minimum size that allows rich enough texture-specific spectral characterization
MULTISCALE TEXTURE OPERATORS
32
• Other consequence:
• Large influence of proximal objects when the support of operators is larger than the region of interest
• Example with band-limited operators (2D isotropic wavelets) and lung boundary [Ward2015, Depeursinge2015a]
• Tuning the shape/bandwidth was found to have a strong influence on lung tissue classification accuracy
M
0
0.05
0.1
0.15
0.2
0.25
0.3
0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
0
5
10
15
20
25
30
35
0
0.05
0.1
0.15
0.2
0
0.5
1
1.5
2
0 ⇡|!1|0 0 ⇡|!1||!1| ⇡x1 x1 x1
f(!)f(x) f(x) f(x)f(!) f(!)
COMPUTERIZED TEXTURE ANALYSIS
31
• Other consequence:
• Large influence of proximal objects when the support of operators is larger than the region of interest:
• Example with band-limited operators (2D isotropic wavelets) and lung boundary [DPC2015,WPU2015]
• Tuning the shape/bandwidth was found to have a strong influence on lung tissue classification accuracy
L1
L2M1
M2
·
M
m
M
0
0.05
0.1
0.15
0.2
0.25
0.3
0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
0
5
10
15
20
25
30
35
0
0.05
0.1
0.15
0.2
0
0.5
1
1.5
2
0 ⇡|!1|0 0 ⇡|!1||!1| ⇡x1 x1 x1
f(!)f(x) f(x) f(x)f(!) f(!)
better spatial localizationworse spectral localization
TEXTURE OPERATORS AND PRIMITIVES
19
• From texture operators to texture measurements
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region of interest
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
MULTISCALE TEXTURE OPERATORS
33
• Which scales for texture measurements?
• Two aspects: A. How to define the size(s) of the operator(s) ?
B. How to define the size of the region of interest ? M
original image d
increasingly small
operator sizeL1 > L2 > · · · > LN
0
BBBBBBBBBBBBBBBBBBBBBBBB@
µ11...µ1P
µ21...µ2P
...
µN1...
µNP
1
CCCCCCCCCCCCCCCCCCCCCCCCA
= µM: concatenated measurements from multiscale operators aggregated over
Ln1 ⇥ · · ·⇥ Ln
d
gn=1,...,N (f(x),m)
M
f(x)
M
M
[Depeursinge2012b]
MULTISCALE TEXTURE OPERATORS
• How large must be the region of interest ?
• No more than enough to evaluate texture stationarity in terms of human perception / tissue biology
• Example with operator: undecimated isotropic Simoncelli’s dyadic wavelets [Portilla2000] applied to all image positions
• Operators’ responses are averaged over
M
TEXTURE OPERATORS AND PRIMITIVES
19
• From texture operators to texture measurements
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region of interest
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
M
f(x) g1(f(x),m) g2(f(x),m)
original image with regions I
1
|M |
Z
M|g1(f(x),m)|dm
M
feature space
1 |M|Z M
|g2(f(x
),m
)|dm
f(x)
Ma,M b,M c
The averaged responses over the entire image does not correspond to anything visually!
g1(⇢) =
⇢cos
�⇡2 log2
� 2⇢⇡
��, ⇡
4 < ⇢ ⇡0, otherwise.
g2(⇢) =
⇢cos
�⇡2 log2
� 4⇢⇡
��, ⇡
8 < ⇢ ⇡2
0, otherwise.
\g1,2�f(⇢,�)
�= g1,2(⇢,�) · f(⇢,�)
m 2 RM1⇥M2
MULTISCALE TEXTURE OPERATORS
• How large must be the region of interest ?
• No more than enough to evaluate texture stationarity in terms of human perception / tissue biology
• Example with operator: undecimated isotropic Simoncelli’s dyadic wavelets [Portilla2000] applied to all image positions
• Operators’ responses are averaged over
M
TEXTURE OPERATORS AND PRIMITIVES
19
• From texture operators to texture measurements
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region of interest
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
M
f(x) g1(f(x),m)
m 2 RM1⇥M2
g2(f(x),m)
original image with regions I
1
|M |
Z
M|g1(f(x),m)|dm
M
feature space
1 |M|Z M
|g2(f(x
),m
)|dm
f(x)
Ma,M b,M c
The averaged responses over the entire image does not correspond to anything visually!
g1(⇢) =
⇢cos
�⇡2 log2
� 2⇢⇡
��, ⇡
4 < ⇢ ⇡0, otherwise.
g2(⇢) =
⇢cos
�⇡2 log2
� 4⇢⇡
��, ⇡
8 < ⇢ ⇡2
0, otherwise.
\g1,2�f(⇢,�)
�= g1,2(⇢,�) · f(⇢,�)
Nor biologically!
MULTISCALE TEXTURE OPERATORS
• How large must be the region of interest ?
• No more than enough to evaluate texture stationarity in terms of human perception / tissue biology
• Example with operator: undecimated isotropic Simoncelli’s dyadic wavelets [Portilla2000] applied to all image positions
• Operators’ responses are averaged over
M
TEXTURE OPERATORS AND PRIMITIVES
19
• From texture operators to texture measurements
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region of interest
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
M
f(x) g1(f(x),m)
m 2 RM1⇥M2
g2(f(x),m)
original image with regions I
1
|M |
Z
M|g1(f(x),m)|dm
M
feature space
1 |M|Z M
|g2(f(x
),m
)|dm
f(x)
Ma,M b,M c
The averaged responses over the entire image does not correspond to anything visually!
g1(⇢) =
⇢cos
�⇡2 log2
� 2⇢⇡
��, ⇡
4 < ⇢ ⇡0, otherwise.
g2(⇢) =
⇢cos
�⇡2 log2
� 4⇢⇡
��, ⇡
8 < ⇢ ⇡2
0, otherwise.
\g1,2�f(⇢,�)
�= g1,2(⇢,�) · f(⇢,�)
Nor biologically!
Define regions that are homogeneous in terms of operators’ responses
(e.g., pixelwise clustering, graph cuts [Malik2001],
Pott’s model [Storath2014])
MULTISCALE TEXTURE OPERATORS
37
• How large must be the region of interest ?
• Example: supervised texture segmentation with undecimated isotropic Simoncelli’s wavelets and linear support vector machines (SVM)
M
feature space (training)
predicted labels
1
|M |
Z
M|g1(f(x),m)|dm
1 |M|Z M
|g2(f(x
),m
)|dm
decision values
-10
-8
-6
-4
-2
0
2
4
6
8
train class 1 (128x128) test image (256x256)train class 2 (128x128)
segmentation error=0.05127
MULTISCALE TEXTURE OPERATORS
38
• How large must be the region of interest ?
• Example: supervised texture segmentation with undecimated isotropic Simoncelli’s wavelets and linear support vector machines (SVM)
M
predicted labels
decision values
-10
-8
-6
-4
-2
0
2
4
6
8
train class 1 (128x128) test image (256x256)train class 2 (128x128)
segmentation error=0.05127
patch radius0 20 40 60 80 100 120
segm
enta
tion
erro
r
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
0.22
0.24
: circular patch with Mr = 30
: circular patch with
MULTISCALE TEXTURE OPERATORS
39
• How large must be the region of interest ?
• Example: supervised texture segmentation with undecimated isotropic Simoncelli’s wavelets and linear support vector machines (SVM)
M
patch radius0 20 40 60 80 100 120
segm
enta
tion
erro
r
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
0.22
0.24
train class 1 (128x128) train class 2 (128x128) test image (256x256)
r = 8
decision values
-2
-1
0
1
2
3
predicted labels
segmentation error=0.23853
M
MULTISCALE TEXTURE OPERATORS
40
• How large must be the region of interest ?
• Example: supervised texture segmentation with undecimated isotropic Simoncelli’s wavelets and linear support vector machines (SVM)
M
patch radius0 20 40 60 80 100 120
segm
enta
tion
erro
r
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
0.22
0.24
: circular patch with r = 128
test image (256x256)train class 1 (128x128) train class 2 (128x128) predicted labels
decision values
-1.5
-1
-0.5
0
0.5
1
segmentation error=0.22839
M
MULTISCALE TEXTURE OPERATORS
41
• How large must be the region of interest ?
• Tissue properties are not homogeneous (i.e., non-stationary) over the entire organ
• Importance of building tissue atlases and digital phenotypes [Depeursinge2015b]
M
the maximum of the score provided by the SVMs. A maximum areaunder the ROC curve (AUC) of 0.81 was obtained with the regionalRiesz attributes, which suggests that prediction was correct for morethan 4 of 5 patients. The performance of HU and GLCM attributeswas close to random (0.54 and 0.6 for HU and GLCMs, respectively).On the other hand, predictive SVM models based on the responses ofthe Riesz filters, averaged over the entire lungs, had an AUC of 0.72.
Our system's performance was also compared with the interpreta-tions of 2 fellowship-trained cardiothoracic fellows, each having 1 yearof experience. Interobserver agreement was assessed with the Cohenκ statistics30 and the percentage of agreement (ie, number of times the2 observer agreed). The comparisons are detailed in Tables 3 and 4.The operating points of the 2 independent observers are reported inFigure 4 (top right). A detailed analysis of the 6 cases that were mis-classified by our system is shown in Table 5 with representative CTimages, including predictions from the computer and the 2 fellows com-pared with the consensus classification. The system predicted 2 classicUIP cases as atypical UIP and 3 atypical UIP cases as classic UIP. A com-prehensive analysis of all 33 cases is illustrated in the SupplementalTable, Supplemental Digital Content 1, http://links.lww.com/RLI/A189.
Overall, 7 incorrect predictions were made by the fellows and 6 incor-rect predictions by the computer. The fellows and the computer madeonly 2 common errors (cases 1 and 13).
DISCUSSIONWe developed a novel computational method for the automated
classification of classic versus atypical UIP based on regional volumet-ric texture analysis. This constitutes, to the best of our knowledge, a firstattempt to automatically differentiate the UIP subtypes with computa-tional methods. An SVM classifier yielded a score that predicts if theUIP is classic or atypical. The classifier was based on a group of attri-butes that characterize the radiological phenotype of the lung paren-chyma, specifically the morphological properties (ie, texture) of theparenchyma. Because diffuse lung diseases can vary in the distributionand severity of abnormalities throughout the lungs, we extracted ourquantiative image features from 36 anatomical regions of the lung. Toour knowledge, adding this spatial characterization to the computationalmodel is also innovative, and it is particularly relevant for assessingdiffuse lung disease.
FIGURE 3. The 36 subregions of the lungs localized the prototype regional distributions of the texture properties. Figure 3 can be viewed online in colorat www.investigativeradiology.com.
FIGURE 4. The ROC analysis of the system's performance. Classic UIP is the positive class. Left, Comparison of various feature groups using the digitallung tissue atlas. Three-dimensional Riesz wavelets provide a superior AUC of 0.81. Right, Importance of the anatomical atlas when comparedwith anapproach based on the global tissue properties and comparison of the computer's and cardiothoracic fellows' performance. Bottom, Probability densityfunctions of the computer score for classic (red) and atypical UIP (blue) based on regional Riesz texture analysis and the computer's operating pointhighlighted in the upper right subfigure. Atypical UIP is associated with a negative score, which implies that positive scores predict classic UIPs with highspecificity. Figure 4 can be viewed online in color at www.investigativeradiology.com.
Depeursinge et al Investigative Radiology • Volume 00, Number 00, Month 2015
4 www.investigativeradiology.com © 2014 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2014 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
M1
M2
= µM1,...,S
OUTLINE
• Biomedical texture analysis: background
• Defining texture processes
• Notations, sampling and texture functions
• Texture operators, primitives and invariances
• Multiscale analysis
• Operator scale and uncertainty principle
• Region of interest and operator aggregation
• Multidirectional analysis
• Isotropic versus directional operators
• Importance of the local organization of image directions
• Conclusions
• References
REFERENCES
66
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2015 1
Steerable Wavelet Machines (SWM): LearningMoving Frames for Texture Classification
Adrien Depeursinge, Zsuzsanna Püspöki, John Paul Ward, and Michael Unser
Abstract—We present texture operators encoding class-specific local organizations of image directions (LOID) in arotation-invariant fashion. The operators are based on steerablecircular harmonic wavelets (CHW), offering a rich and yet com-pact initial representation for characterizing natural textures.We enforce the preservation of the joint location and orientationimage structure by building Cartan moving frames (MF) fromlocally-steered multi-order CHWs. In a second step, we usesupport vector machines (SVM) to learn a multi-class shapingmatrix of the initial CHW representations, yielding data-drivenMFs. Intuitively, the learned MFs can be seen as class-specificforged detectors that are applied and rotated at each point ofthe image to evaluate the magnitude of their responses, i.e., thepresence of the corresponding texture class. We experimentallydemonstrate the effectiveness of the proposed operators forclassifying natural textures, which outperformed the reportedperformance of recent approaches on several test suites of theOutex database.
Index Terms—Texture classification, feature learning, mov-ing frames, support vector machines, steerability, rotation-invariance, illumination-invariance, wavelet analysis.
I. INTRODUCTION
THe wealth of digital texture patterns is tightly relatedto the size of the observation window on a discrete
lattice. In an extreme case, an image region composedof one pixel cannot form geometrical structures. There-fore, the characterization of textural patterns often requiresintegrating image operators g (x), x 2 R2 over an imagedomain X . The latter raises two major challenges. First,the action of the integrated operators becomes diffuseover X , which hinders the spatial precision of texturesegmentation approaches. Second, the effect of integrationbecomes even more destructive when unidirectional opera-tors are jointly used to characterize the local organization ofimage directions (LOID) (e.g., directional filterbanks [1, 2],curvelets [3], histogram of gradients (HOG) [4], Haralick [5]).When separately integrated, the responses of unidirectionalindividual operators are not local anymore and their jointresponses become only sensitive to the global amount ofimage directions in the region X . For instance, the joint
responses of image gradients g1,2(x) =≥ØØ @ f (x)
@x1
ØØ,ØØ @ f (x)@x2
ØØ¥
arenot able to discriminate between the two textures classesf1(x) and f2(x) shown in Fig. 1 when integrated over theimage domain X .
This loss of information is detailed by Sifre et al. interms of separable group invariants [6]. When integrated
The authors are with the Biomedical Imaging Group, École Poly-technique Fédérale de Lausanne, Lausanne 1015, Switzerland (e-mail:[email protected]; [email protected]). A. Depeursinge isalso with the MedGIFT group, Institute of Information Systems, Universityof Applied Sciences Western Switzerland, Sierre (HES-SO).
of1(x) : of2(x) :
RX
ØØ @ f (x)@x1
ØØdx
R XØ Ø@
f(x)
@x 2
Ø Ø dx
image gradients
Fig. 1. The joint responses of image gradients≥| @ f (x)@x1
|, | @ f (x)@x2
|¥
are notable to discriminate between the two textures classes f1(x) and f2(x)when integrated over the image domain X . One circle in the gradientrepresentation corresponds to one realization (i.e., full image) of f1,2.
separately, the responses of unidirectional operators be-come invariant to a larger family of roto-translations wheredifferent orientations are translated by different values. Forinstance, it can be observed in Fig. 1 that f2 can be obtainedfrom f1 by vertically translating horizontal bars only andhorizontally translating vertical bars only.
The LOID are key for visual understanding [7] and is theonly property able to differentiate between f1 and f2 inFig. 1. It relates to the joint information between positionsand orientations in images, which is discarded by operatorsthat are invariant to separable groups. The LOID has beenleveraged in the literature to define [8] and discriminatetexture classes [9–12]. It is also at the origin of the successof methods based on local binary patterns (LBP) [9] andits extensions [13–19], which specifically encodes the LOIDwith uniform circular pixel sequences. Although classicaldeep learning approaches are not enforcing the characteri-zation of the LOID [20], deep convolutional networks wererecently proposed by Oyallon et al. to specifically preservethe structure of the roto-translation group [21].
An even bigger challenge is to design texture operatorsthat can characterize the LOID in a rotation-invariantfashion [9, 11]. The latter is very important to recognizeimage structures independently from both their local ori-entations and the global orientation of the image. Exam-ples of such structures are vascular, bronchial or dendritictrees in biomedical images, river deltas or urban areas insatellite images, or even more complex biomedical tissuearchitectures or crystals in petrographic analysis. Since mostof the above-mentioned imaging modalities yield imageswith pixel sizes defined in physical dimensions, we arenot interested in invariance to image scale. We aim todesign texture operators that are invariant to the family ofEuclidean transforms (also called rigid), i.e.,
g (x) = g (Rµ0 x °x0), 8x , x0,µ0 2R2,
·m
43
• Which directions for texture measurements?
• Isotropic operators: insensitive to image directions
• Linear:
• Non-linear: e.g., median filter
• Directional operators
• Linear:
Other: e.g., Fourier, circular and spherical harmonics [Unser2013, Ward2014], MR8 [Varma2005], HOG [Dalal2005], Simoncelli’s pyramid [Simoncelli1995], curvelets [Candes2000]
• Non-linear:
Other: e.g., RLE [Galloway1975]
2D Gaussian filter 2D isotropic wavelets [Portilla2000]
MULTIDIRECTIONAL TEXTURE OPERATORS
2D Gaussian derivatives (e.g., Riesz wavelets [Unser2011])@
@x1
@
@x2
@
2
@x
22
@
2
@x
21
@
2
@x1@x2
d = 2 (�k1 = 2,�k2 = 0)
2D GLCMs
d = 2p2 (�k1 = 2,�k2 = 2)
2D LBPs [Ojala2002]
!"#$%& "'(%)! *+ # $',$)% *- ,#&'.! ! !! " !" /0#/ -*,1 #$',$.)#,)2 !211%/,'$ +%'304*, !%/5
6- /0% $**,&'+#/%! *- #$ #,% 7!% !89 /0%+ /0% $**,&'+#/%!*- #& #,% 3':%+ 42 !#! "#$!%'&() "% ! &'"!%'&() ""5 ;'35 <')).!/,#/%! $',$.)#,)2 !211%/,'$ +%'304*, !%/! -*, :#,'*.!7)%!85 =0% 3,#2 :#).%! *- +%'304*,! >0'$0 &* +*/ -#))%(#$/)2 '+ /0% $%+/%, *- "'(%)! #,% %!/'1#/%& 42'+/%,"*)#/'*+5
!"# $%&'()'*+ ,-./01%.2( 3*).-'.*%(
?! /0% -',!/ !/%" /*>#,& 3,#2@!$#)% '+:#,'#+$%9 >% !.4/,#$/9>'/0*./ )*!'+3 '+-*,1#/'*+9 /0% 3,#2 :#).% *- /0% $%+/%,"'(%) 7#$8 -,*1 /0% 3,#2 :#).%! *- /0% $',$.)#,)2 !211%/,'$+%'304*,0**& #& !& $ !% ( ( ( % ) # )"9 3':'+3A
* $ +!#$% #! # #$% #) # #$% ( ( ( % #)#) # #$", !%"
B%(/9 >% #!!.1% /0#/ &'--%,%+$%! #& # #$ #,% '+&%"%+&%+/*- #$9 >0'$0 #))*>! .! /* -#$/*,'C% 7D8A
* % +!#$"+!#! # #$% #) # #$% ( ( ( % #)#) # #$", !*"
6+ ",#$/'$%9 #+ %(#$/ '+&%"%+&%+$% '! +*/ >#,,#+/%&E0%+$%9 /0% -#$/*,'C%& &'!/,'4./'*+ '! *+)2 #+ #"",*('1#/'*+*- /0% F*'+/ &'!/,'4./'*+5 G*>%:%,9 >% #,% >'))'+3 /* #$$%"//0% "*!!'4)% !1#)) )*!! '+ '+-*,1#/'*+ #! '/ #))*>! .! /*#$0'%:% '+:#,'#+$% >'/0 ,%!"%$/ /* !0'-/! '+ 3,#2 !$#)%5B#1%)29 /0% &'!/,'4./'*+ +!#$" '+ 7H8 &%!$,'4%! /0% *:%,#))).1'+#+$% *- /0% '1#3%9 >0'$0 '! .+,%)#/%& /* )*$#) '1#3%/%(/.,% #+&9 $*+!%I.%+/)29 &*%! +*/ ",*:'&% .!%-.) '+@-*,1#/'*+ -*, /%(/.,% #+#)2!'!5 G%+$%9 1.$0 *- /0% '+-*,1#@/'*+ '+ /0% *,'3'+#) F*'+/ 3,#2 )%:%) &'!/,'4./'*+ 7<8 #4*./ /0%/%(/.,#) $0#,#$/%,'!/'$! '! $*+:%2%& 42 /0% F*'+/ &'--%,%+$%&'!/,'4./'*+ JDKLA
* % +!#! # #$% #) # #$% ( ( ( % #&#) # #$", !+"
=0'! '! # 0'30)2 &'!$,'1'+#/':% /%(/.,% *"%,#/*,5 6/,%$*,&! /0% *$$.,,%+$%! *- :#,'*.! "#//%,+! '+ /0% +%'304*,@0**& *- %#$0 "'(%) '+ # ) ,-#./$"#'$01 0'!/*3,#15 ;*,$*+!/#+/ ,%3'*+!9 /0% &'--%,%+$%! #,% C%,* '+ #)) &',%$/'*+!5M+ # !)*>)2 !)*"%& %&3%9 /0% *"%,#/*, ,%$*,&! /0% 0'30%!/&'--%,%+$% '+ /0% 3,#&'%+/ &',%$/'*+ #+& C%,* :#).%! #)*+3/0% %&3% #+&9 -*, # !"*/9 /0% &'--%,%+$%! #,% 0'30 '+ #))&',%$/'*+!5
N'3+%& &'--%,%+$%! #& # #$ #,% +*/ #--%$/%& 42 $0#+3%! '+1%#+ ).1'+#+$%E 0%+$%9 /0% F*'+/ &'--%,%+$% &'!/,'4./'*+ '!'+:#,'#+/ #3#'+!/ 3,#2@!$#)% !0'-/!5 O% #$0'%:% '+:#,'#+$%>'/0 ,%!"%$/ /* /0% !$#)'+3 *- /0% 3,#2 !$#)% 42 $*+!'&%,'+3F.!/ /0% !'3+! *- /0% &'--%,%+$%! '+!/%#& *- /0%', %(#$/ :#).%!A
* % +!-!#! # #$"% -!#) # #$"% ( ( ( % -!#)#) # #$""% !2"
>0%,%
-!." $ )% . & !!% . / !,
!!3"
P2 #!!'3+'+3 # 4'+*1'#) -#$/*, %& -*, %#$0 !'3+ -!#& # #$"9>% /,#+!-*,1 7Q8 '+/* # .+'I.% 01))%! +.14%, /0#/$0#,#$/%,'C%! /0% !"#/'#) !/,.$/.,% *- /0% )*$#) '1#3% /%(/.,%A
01))%! $")#)
&$!
-!#& # #$"%&, !4"
=0% +#1% RS*$#) P'+#,2 T#//%,+U ,%-)%$/! /0% -.+$/'*+@#)'/2 *- /0% *"%,#/*,9 '5%59 # )*$#) +%'304*,0**& '!/0,%!0*)&%& #/ /0% 3,#2 :#).% *- /0% $%+/%, "'(%) '+/* #4'+#,2 "#//%,+5 01))%! *"%,#/*, '! 42 &%-'+'/'*+ '+:#,'#+/#3#'+!/ #+2 1*+*/*+'$ /,#+!-*,1#/'*+ *- /0% 3,#2 !$#)%9'5%59 #! )*+3 #! /0% *,&%, *- /0% 3,#2 :#).%! '+ /0% '1#3%!/#2! /0% !#1%9 /0% *./"./ *- /0% 01))%! *"%,#/*,,%1#'+! $*+!/#+/5
6- >% !%/ 7) $ 5% ! $ )89 >% *4/#'+ 01)5%)9 >0'$0 '!!'1')#, /* /0% 01) *"%,#/*, >% ",*"*!%& '+ JDVL5 =0% />*&'--%,%+$%! 4%/>%%+ 01)5%) #+& 01) #,%A <8 =0% "'(%)! '+/0% +%'304*, !%/ #,% '+&%(%& !* /0#/ /0%2 -*,1 # $',$.)#,$0#'+ #+& D8 /0% 3,#2 :#).%! *- /0% &'#3*+#) "'(%)! #,%&%/%,1'+%& 42 '+/%,"*)#/'*+5 P*/0 1*&'-'$#/'*+! #,% +%$%!@!#,2 /* *4/#'+ /0% $',$.)#,)2 !211%/,'$ +%'304*, !%/9 >0'$0#))*>! -*, &%,':'+3 # ,*/#/'*+ '+:#,'#+/ :%,!'*+ *- 01))%!5
!"! $%&'()'*+ 456.6'5* 3*).-'.*%(
=0% 01))%! *"%,#/*, ",*&.$%! %) &'--%,%+/ *./"./ :#).%!9$*,,%!"*+&'+3 /* /0% %) &'--%,%+/ 4'+#,2 "#//%,+! /0#/ $#+ 4%-*,1%& 42 /0% ) "'(%)! '+ /0% +%'304*, !%/5 O0%+ /0% '1#3%'! ,*/#/%&9 /0% 3,#2 :#).%! #& >')) $*,,%!"*+&'+3)2 1*:%#)*+3 /0% "%,'1%/%, *- /0% $',$)% #,*.+& #!5 N'+$% #! '!#)>#2! #!!'3+%& /* 4% /0% 3,#2 :#).% *- %)%1%+/ 7!% !8 /* /0%,'30/ *- #$ ,*/#/'+3 # "#,/'$.)#, 4'+#,2 "#//%,+ +#/.,#))2,%!.)/! '+ # &'--%,%+/ 01))%! :#).%5 =0'! &*%! +*/ #"")2 /*"#//%,+! $*1",'!'+3 *- *+)2 W! 7*, <!8 >0'$0 ,%1#'+$*+!/#+/ #/ #)) ,*/#/'*+ #+3)%!5 =* ,%1*:% /0% %--%$/ *-,*/#/'*+9 '5%59 /* #!!'3+ # .+'I.% '&%+/'-'%, /* %#$0 ,*/#/'*+'+:#,'#+/ )*$#) 4'+#,2 "#//%,+ >% &%-'+%A
01)23)%! $ 435'!6!!01))%!% 3" ( 3 $ !% )% ( ( ( %6# ))%
!5"
>0%,% !6!!.% 3" "%,-*,1! # $',$.)#, 4'/@>'!% ,'30/ !0'-/ *+/0% ) ,7#8 +.14%, . 3 /'1%!5 6+ /%,1! *- '1#3% "'(%)!9 7K8
!"#$# %& #$'( )*$&+,%-!$*&+!. /,#01-2#$% #.3 ,!&#&+!. +.4#,+#.& &%5&*,% 2$#--+6+2#&+!. 7+&8 $!2#$ 9+.#,0 :#&&%,.- ;<=
6>?' @' 2>ABCDEADF GFHHIJA>B KI>?LMNA GIJG ONA P>OOIAIKJ Q)%!R'
!"#$%& "'(%)! *+ # $',$)% *- ,#&'.! ! !! " !" /0#/ -*,1 #$',$.)#,)2 !211%/,'$ +%'304*, !%/5
6- /0% $**,&'+#/%! *- #$ #,% 7!% !89 /0%+ /0% $**,&'+#/%!*- #& #,% 3':%+ 42 !#! "#$!%'&() "% ! &'"!%'&() ""5 ;'35 <')).!/,#/%! $',$.)#,)2 !211%/,'$ +%'304*, !%/! -*, :#,'*.!7)%!85 =0% 3,#2 :#).%! *- +%'304*,! >0'$0 &* +*/ -#))%(#$/)2 '+ /0% $%+/%, *- "'(%)! #,% %!/'1#/%& 42'+/%,"*)#/'*+5
!"# $%&'()'*+ ,-./01%.2( 3*).-'.*%(
?! /0% -',!/ !/%" /*>#,& 3,#2@!$#)% '+:#,'#+$%9 >% !.4/,#$/9>'/0*./ )*!'+3 '+-*,1#/'*+9 /0% 3,#2 :#).% *- /0% $%+/%,"'(%) 7#$8 -,*1 /0% 3,#2 :#).%! *- /0% $',$.)#,)2 !211%/,'$+%'304*,0**& #& !& $ !% ( ( ( % ) # )"9 3':'+3A
* $ +!#$% #! # #$% #) # #$% ( ( ( % #)#) # #$", !%"
B%(/9 >% #!!.1% /0#/ &'--%,%+$%! #& # #$ #,% '+&%"%+&%+/*- #$9 >0'$0 #))*>! .! /* -#$/*,'C% 7D8A
* % +!#$"+!#! # #$% #) # #$% ( ( ( % #)#) # #$", !*"
6+ ",#$/'$%9 #+ %(#$/ '+&%"%+&%+$% '! +*/ >#,,#+/%&E0%+$%9 /0% -#$/*,'C%& &'!/,'4./'*+ '! *+)2 #+ #"",*('1#/'*+*- /0% F*'+/ &'!/,'4./'*+5 G*>%:%,9 >% #,% >'))'+3 /* #$$%"//0% "*!!'4)% !1#)) )*!! '+ '+-*,1#/'*+ #! '/ #))*>! .! /*#$0'%:% '+:#,'#+$% >'/0 ,%!"%$/ /* !0'-/! '+ 3,#2 !$#)%5B#1%)29 /0% &'!/,'4./'*+ +!#$" '+ 7H8 &%!$,'4%! /0% *:%,#))).1'+#+$% *- /0% '1#3%9 >0'$0 '! .+,%)#/%& /* )*$#) '1#3%/%(/.,% #+&9 $*+!%I.%+/)29 &*%! +*/ ",*:'&% .!%-.) '+@-*,1#/'*+ -*, /%(/.,% #+#)2!'!5 G%+$%9 1.$0 *- /0% '+-*,1#@/'*+ '+ /0% *,'3'+#) F*'+/ 3,#2 )%:%) &'!/,'4./'*+ 7<8 #4*./ /0%/%(/.,#) $0#,#$/%,'!/'$! '! $*+:%2%& 42 /0% F*'+/ &'--%,%+$%&'!/,'4./'*+ JDKLA
* % +!#! # #$% #) # #$% ( ( ( % #&#) # #$", !+"
=0'! '! # 0'30)2 &'!$,'1'+#/':% /%(/.,% *"%,#/*,5 6/,%$*,&! /0% *$$.,,%+$%! *- :#,'*.! "#//%,+! '+ /0% +%'304*,@0**& *- %#$0 "'(%) '+ # ) ,-#./$"#'$01 0'!/*3,#15 ;*,$*+!/#+/ ,%3'*+!9 /0% &'--%,%+$%! #,% C%,* '+ #)) &',%$/'*+!5M+ # !)*>)2 !)*"%& %&3%9 /0% *"%,#/*, ,%$*,&! /0% 0'30%!/&'--%,%+$% '+ /0% 3,#&'%+/ &',%$/'*+ #+& C%,* :#).%! #)*+3/0% %&3% #+&9 -*, # !"*/9 /0% &'--%,%+$%! #,% 0'30 '+ #))&',%$/'*+!5
N'3+%& &'--%,%+$%! #& # #$ #,% +*/ #--%$/%& 42 $0#+3%! '+1%#+ ).1'+#+$%E 0%+$%9 /0% F*'+/ &'--%,%+$% &'!/,'4./'*+ '!'+:#,'#+/ #3#'+!/ 3,#2@!$#)% !0'-/!5 O% #$0'%:% '+:#,'#+$%>'/0 ,%!"%$/ /* /0% !$#)'+3 *- /0% 3,#2 !$#)% 42 $*+!'&%,'+3F.!/ /0% !'3+! *- /0% &'--%,%+$%! '+!/%#& *- /0%', %(#$/ :#).%!A
* % +!-!#! # #$"% -!#) # #$"% ( ( ( % -!#)#) # #$""% !2"
>0%,%
-!." $ )% . & !!% . / !,
!!3"
P2 #!!'3+'+3 # 4'+*1'#) -#$/*, %& -*, %#$0 !'3+ -!#& # #$"9>% /,#+!-*,1 7Q8 '+/* # .+'I.% 01))%! +.14%, /0#/$0#,#$/%,'C%! /0% !"#/'#) !/,.$/.,% *- /0% )*$#) '1#3% /%(/.,%A
01))%! $")#)
&$!
-!#& # #$"%&, !4"
=0% +#1% RS*$#) P'+#,2 T#//%,+U ,%-)%$/! /0% -.+$/'*+@#)'/2 *- /0% *"%,#/*,9 '5%59 # )*$#) +%'304*,0**& '!/0,%!0*)&%& #/ /0% 3,#2 :#).% *- /0% $%+/%, "'(%) '+/* #4'+#,2 "#//%,+5 01))%! *"%,#/*, '! 42 &%-'+'/'*+ '+:#,'#+/#3#'+!/ #+2 1*+*/*+'$ /,#+!-*,1#/'*+ *- /0% 3,#2 !$#)%9'5%59 #! )*+3 #! /0% *,&%, *- /0% 3,#2 :#).%! '+ /0% '1#3%!/#2! /0% !#1%9 /0% *./"./ *- /0% 01))%! *"%,#/*,,%1#'+! $*+!/#+/5
6- >% !%/ 7) $ 5% ! $ )89 >% *4/#'+ 01)5%)9 >0'$0 '!!'1')#, /* /0% 01) *"%,#/*, >% ",*"*!%& '+ JDVL5 =0% />*&'--%,%+$%! 4%/>%%+ 01)5%) #+& 01) #,%A <8 =0% "'(%)! '+/0% +%'304*, !%/ #,% '+&%(%& !* /0#/ /0%2 -*,1 # $',$.)#,$0#'+ #+& D8 /0% 3,#2 :#).%! *- /0% &'#3*+#) "'(%)! #,%&%/%,1'+%& 42 '+/%,"*)#/'*+5 P*/0 1*&'-'$#/'*+! #,% +%$%!@!#,2 /* *4/#'+ /0% $',$.)#,)2 !211%/,'$ +%'304*, !%/9 >0'$0#))*>! -*, &%,':'+3 # ,*/#/'*+ '+:#,'#+/ :%,!'*+ *- 01))%!5
!"! $%&'()'*+ 456.6'5* 3*).-'.*%(
=0% 01))%! *"%,#/*, ",*&.$%! %) &'--%,%+/ *./"./ :#).%!9$*,,%!"*+&'+3 /* /0% %) &'--%,%+/ 4'+#,2 "#//%,+! /0#/ $#+ 4%-*,1%& 42 /0% ) "'(%)! '+ /0% +%'304*, !%/5 O0%+ /0% '1#3%'! ,*/#/%&9 /0% 3,#2 :#).%! #& >')) $*,,%!"*+&'+3)2 1*:%#)*+3 /0% "%,'1%/%, *- /0% $',$)% #,*.+& #!5 N'+$% #! '!#)>#2! #!!'3+%& /* 4% /0% 3,#2 :#).% *- %)%1%+/ 7!% !8 /* /0%,'30/ *- #$ ,*/#/'+3 # "#,/'$.)#, 4'+#,2 "#//%,+ +#/.,#))2,%!.)/! '+ # &'--%,%+/ 01))%! :#).%5 =0'! &*%! +*/ #"")2 /*"#//%,+! $*1",'!'+3 *- *+)2 W! 7*, <!8 >0'$0 ,%1#'+$*+!/#+/ #/ #)) ,*/#/'*+ #+3)%!5 =* ,%1*:% /0% %--%$/ *-,*/#/'*+9 '5%59 /* #!!'3+ # .+'I.% '&%+/'-'%, /* %#$0 ,*/#/'*+'+:#,'#+/ )*$#) 4'+#,2 "#//%,+ >% &%-'+%A
01)23)%! $ 435'!6!!01))%!% 3" ( 3 $ !% )% ( ( ( %6# ))%
!5"
>0%,% !6!!.% 3" "%,-*,1! # $',$.)#, 4'/@>'!% ,'30/ !0'-/ *+/0% ) ,7#8 +.14%, . 3 /'1%!5 6+ /%,1! *- '1#3% "'(%)!9 7K8
!"#$# %& #$'( )*$&+,%-!$*&+!. /,#01-2#$% #.3 ,!&#&+!. +.4#,+#.& &%5&*,% 2$#--+6+2#&+!. 7+&8 $!2#$ 9+.#,0 :#&&%,.- ;<=
6>?' @' 2>ABCDEADF GFHHIJA>B KI>?LMNA GIJG ONA P>OOIAIKJ Q)%!R'
REFERENCES
66
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2015 1
Steerable Wavelet Machines (SWM): LearningMoving Frames for Texture Classification
Adrien Depeursinge, Zsuzsanna Püspöki, John Paul Ward, and Michael Unser
Abstract—We present texture operators encoding class-specific local organizations of image directions (LOID) in arotation-invariant fashion. The operators are based on steerablecircular harmonic wavelets (CHW), offering a rich and yet com-pact initial representation for characterizing natural textures.We enforce the preservation of the joint location and orientationimage structure by building Cartan moving frames (MF) fromlocally-steered multi-order CHWs. In a second step, we usesupport vector machines (SVM) to learn a multi-class shapingmatrix of the initial CHW representations, yielding data-drivenMFs. Intuitively, the learned MFs can be seen as class-specificforged detectors that are applied and rotated at each point ofthe image to evaluate the magnitude of their responses, i.e., thepresence of the corresponding texture class. We experimentallydemonstrate the effectiveness of the proposed operators forclassifying natural textures, which outperformed the reportedperformance of recent approaches on several test suites of theOutex database.
Index Terms—Texture classification, feature learning, mov-ing frames, support vector machines, steerability, rotation-invariance, illumination-invariance, wavelet analysis.
I. INTRODUCTION
THe wealth of digital texture patterns is tightly relatedto the size of the observation window on a discrete
lattice. In an extreme case, an image region composedof one pixel cannot form geometrical structures. There-fore, the characterization of textural patterns often requiresintegrating image operators g (x), x 2 R2 over an imagedomain X . The latter raises two major challenges. First,the action of the integrated operators becomes diffuseover X , which hinders the spatial precision of texturesegmentation approaches. Second, the effect of integrationbecomes even more destructive when unidirectional opera-tors are jointly used to characterize the local organization ofimage directions (LOID) (e.g., directional filterbanks [1, 2],curvelets [3], histogram of gradients (HOG) [4], Haralick [5]).When separately integrated, the responses of unidirectionalindividual operators are not local anymore and their jointresponses become only sensitive to the global amount ofimage directions in the region X . For instance, the joint
responses of image gradients g1,2(x) =≥ØØ @ f (x)
@x1
ØØ,ØØ @ f (x)@x2
ØØ¥
arenot able to discriminate between the two textures classesf1(x) and f2(x) shown in Fig. 1 when integrated over theimage domain X .
This loss of information is detailed by Sifre et al. interms of separable group invariants [6]. When integrated
The authors are with the Biomedical Imaging Group, École Poly-technique Fédérale de Lausanne, Lausanne 1015, Switzerland (e-mail:[email protected]; [email protected]). A. Depeursinge isalso with the MedGIFT group, Institute of Information Systems, Universityof Applied Sciences Western Switzerland, Sierre (HES-SO).
of1(x) : of2(x) :
RX
ØØ @ f (x)@x1
ØØdx
R XØ Ø@
f(x)
@x 2
Ø Ø dx
image gradients
Fig. 1. The joint responses of image gradients≥| @ f (x)@x1
|, | @ f (x)@x2
|¥
are notable to discriminate between the two textures classes f1(x) and f2(x)when integrated over the image domain X . One circle in the gradientrepresentation corresponds to one realization (i.e., full image) of f1,2.
separately, the responses of unidirectional operators be-come invariant to a larger family of roto-translations wheredifferent orientations are translated by different values. Forinstance, it can be observed in Fig. 1 that f2 can be obtainedfrom f1 by vertically translating horizontal bars only andhorizontally translating vertical bars only.
The LOID are key for visual understanding [7] and is theonly property able to differentiate between f1 and f2 inFig. 1. It relates to the joint information between positionsand orientations in images, which is discarded by operatorsthat are invariant to separable groups. The LOID has beenleveraged in the literature to define [8] and discriminatetexture classes [9–12]. It is also at the origin of the successof methods based on local binary patterns (LBP) [9] andits extensions [13–19], which specifically encodes the LOIDwith uniform circular pixel sequences. Although classicaldeep learning approaches are not enforcing the characteri-zation of the LOID [20], deep convolutional networks wererecently proposed by Oyallon et al. to specifically preservethe structure of the roto-translation group [21].
An even bigger challenge is to design texture operatorsthat can characterize the LOID in a rotation-invariantfashion [9, 11]. The latter is very important to recognizeimage structures independently from both their local ori-entations and the global orientation of the image. Exam-ples of such structures are vascular, bronchial or dendritictrees in biomedical images, river deltas or urban areas insatellite images, or even more complex biomedical tissuearchitectures or crystals in petrographic analysis. Since mostof the above-mentioned imaging modalities yield imageswith pixel sizes defined in physical dimensions, we arenot interested in invariance to image scale. We aim todesign texture operators that are invariant to the family ofEuclidean transforms (also called rigid), i.e.,
g (x) = g (Rµ0 x °x0), 8x , x0,µ0 2R2,
·m
gn(r, ✓,m) 7! gn(r,m)x
MULTIDIRECTIONAL TEXTURE OPERATORS
44
• Which directions for texture measurements?
• Is directional information important for texture discrimination?
F�!
f(!)f(x)
F�!
f(!)f(x)
MULTIDIRECTIONAL TEXTURE OPERATORS
45
• Which directions for texture measurements?
• Importance of the local organization of image directions (LOID)
• i.e., how directional structures intersect
�
MULTIDIRECTIONAL TEXTURE OPERATORS
46
• Which directions for texture measurements?
• Isotropic and unidirectional operators can hardly characterize the LOIDs, especially when aggregated over a region [Sifre2014, Depeursinge2014b]
• Example of feature representation when integrated over entire image
M
isotropic Simoncelli wavelets
scale 1
scal
e 2
o o
GLCM contrast
GLC
M c
ontra
st
d = 1 (�k1 = 1,�k2 = 0)
d=
1(�
k1=
0,�
k2=
1)
GLCMsgradients along and
1
|M |
Z
M
✓@f(x)
@x1
◆2
dx
1 |M|Z M
✓@f(x
)
@x
2
◆2
dx
x1 x2
M ⌘
L1
L2M1
M2
·
M
m L1
L2M1
M2
·
M
m
MULTIDIRECTIONAL TEXTURE OPERATORS
47
• Which directions for texture measurements?
• Isotropic and unidirectional operators can hardly characterize the LOIDs, especially when aggregated over a region [Sifre2014, Depeursinge2014b]
• Example of feature representation when integrated over entire image
M
isotropic Simoncelli wavelets
scale 1
scal
e 2
o o
GLCM contrast
GLC
M c
ontra
st
d = 1 (�k1 = 1,�k2 = 0)
d=
1(�
k1=
0,�
k2=
1)
GLCMsgradients along and
1
|M |
Z
M
✓@f(x)
@x1
◆2
dx
1 |M|Z M
✓@f(x
)
@x
2
◆2
dx
x1 x2
M ⌘
L1
L2M1
M2
·
M
m L1
L2M1
M2
·
M
m
Very poor discrimination! Solutions proposed in a few slides…
MULTIDIRECTIONAL TEXTURE OPERATORS
48
• Locally rotation-invariant operators over
• Isotropic operators:
• By definition
• Directional:
• Averaging operators’ responses over all directions:
2D GLCMs
µ1 (e.g., GLCM contrast)
No characterization of image directions!
�µ⇡/21µ⇡/4
1 µ3⇡/41µ0
1
L1 ⇥ L2
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
REFERENCES
68
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
MULTIDIRECTIONAL TEXTURE OPERATORS
49
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• MR8 filterbank [Varma2005]
• Rotation-invariant LBP [Ojala2002, Ahonen2009]
• Steerable Riesz wavelets [Unser2013, Depeursinge2014b]
L1 ⇥ L2
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
50
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Maximum response 8 (MR8) filterbank [Varma2005]
• Filter responses are obtained for each pixel from the convolution of the filter and the image
• For each position , only the maximum responses among gradient and Laplacian filters are kept
isotropic mutliscale oriented gradients multiscale oriented Laplacians
m
L1 ⇥ L2
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
51
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Maximum response 8 (MR8) filterbank [Varma2005]
• Filter responses are obtained for each pixel from the convolution of the filter and the image
• For each position , only the maximum responses among gradient and Laplacian filters are kept
isotropic mutliscale oriented gradients multiscale oriented Laplacians
m
L1 ⇥ L2
Yields approximate local rotation invariance
Poor characterization of the LOIDs
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
52
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Local binary patterns (LBP) [Ojala2002]
• Rotation-invariant LBP [Ahonen2009]
L1 ⇥ L2
1) define a circular neighborhood 2) Binarize and build a number that encode the LOIDs
)
3) Aggregate over the entire image and count code occurrences
0 170
⌘
4) make codes invariant to circular shifts
U8(1, 0) = 10101010 = 170
U8(1, 0) = 10101010U8(1, 1) = 01010101
m
rotation r
discrete Fourier transform
The new measures are independent of the rotation ) r
H8(1, u) =7X
r=0
hI(U8(1, r))e�j2⇡ur/8
µp = |H8(1, u)|
µ0,p = hI(U8(1, 0))
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
53
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Local binary patterns (LBP) [Ojala2002]
• Rotation-invariant LBP [Ahonen2009]
L1 ⇥ L2
1) define a circular neighborhood 2) Binarize and build a number that encode the LOIDs
)
3) Aggregate over the entire image and count code occurrences
0 170
⌘
4) make codes invariant to circular shifts
U8(1, 0) = 10101010 = 170
U8(1, 0) = 10101010U8(1, 1) = 01010101
m
rotation r
discrete Fourier transform
The new measures are independent of the rotation ) r
H8(1, u) =7X
r=0
hI(U8(1, r))e�j2⇡ur/8
µp = |H8(1, u)|
µ0,p = hI(U8(1, 0))
Encodes the LOIDs independently from their local orientations!
Requires binarization…
Spherical sequences are undefined in 3D…
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
54
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Steerable Riesz wavelets [Unser2013, Depeursinge2014b]
• Operators: th-order multi-scale image derivatives
L1 ⇥ L2
input imageN4 A. Depeursinge et al.
N = 1 N = 2
N = 3
Fig. 1. Templates corresponding to the Riesz kernels convolved with a Gaussiansmoother for N=1,2,3.
is determined by the partial derivatives in Eq. (1). Whereas 2N Riesz filtersare generated by (1), only N + 1 components have distinct properties due tocommutativity of the convolution operators in (2) (e.g., ∂2/∂x∂y is equivalentto ∂2/∂y∂x). The Riesz components yield a steerable filterbank [15] allowingto analyze textures in any direction, which is an advantage when compared toclassical Gaussian derivatives or Gabor filters. Qualitatively, the first Riesz com-ponent of even order corresponds to a ridge profile whereas for odd ones we obtainan edge profile, but much richer profiles can be obtained by linear combinationsof the different components. The templates of h1,2(x) convolved with Gaussiankernels for N=1,2,3 are depicted in Fig. 1. The Nth–order Riesz transform canbe coupled with an isotropic multiresolution decomposition (e.g., Laplacian ofGaussian (LoG)) to obtain rotation–covariant (steerable) basis functions [15].
The main idea of the proposed approach is to derive texture signatures frommultiscale Riesz coefficients. An example showing healthy and fibrosis tissuerepresented in terms of their Riesz components with N=2 is depicted in Fig. 2 a).In order to provide a local categorization of the lung parenchyma, lung regionsin 2D axial slices are divided into 32×32 overlapping blocks with a distancebetween contiguous block centers of 16. The Riesz transform is applied to eachblock, and every Riesz component n = 1, . . . , N+1 is mapped to a multiscalerepresentation by convolving them with four LoG filters of scales s = 1, . . . , 4with a dyadic scale progression. In a total of (N+1)×4 subbands, the variancesσn,s of the coefficients are used as texture features along with 22 grey levelhistogram (GLH) bins in [-1050;600] Hounsfield Units (HU). The percentageof air pixels with values ≤ −1000 HU completes the feature space learned bysupport vector machines (SVM) with a Gaussian kernel.
The local dominant texture orientations have an influence on the reparti-tion of respective responses of the Riesz components, which is not desirable forcreating robust features with well–defined clusters of instances. For example, arotation of π/2 will switch the responses of h1 and h2 for N=1. To ensure thatthe repartitions of σn,s are comparable for two similar textures having distinct
N = 1g(1,0)(x,m)
f(x)
g(1,0)(f(x),m)
g(0,1)(x,m)
g(0,1)(f(x),m)
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
55
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Steerable Riesz wavelets [Unser2013, Depeursinge2014b]
• Operators: th-order multi-scale image derivatives
L1 ⇥ L2
N4 A. Depeursinge et al.
N = 1 N = 2
N = 3
Fig. 1. Templates corresponding to the Riesz kernels convolved with a Gaussiansmoother for N=1,2,3.
is determined by the partial derivatives in Eq. (1). Whereas 2N Riesz filtersare generated by (1), only N + 1 components have distinct properties due tocommutativity of the convolution operators in (2) (e.g., ∂2/∂x∂y is equivalentto ∂2/∂y∂x). The Riesz components yield a steerable filterbank [15] allowingto analyze textures in any direction, which is an advantage when compared toclassical Gaussian derivatives or Gabor filters. Qualitatively, the first Riesz com-ponent of even order corresponds to a ridge profile whereas for odd ones we obtainan edge profile, but much richer profiles can be obtained by linear combinationsof the different components. The templates of h1,2(x) convolved with Gaussiankernels for N=1,2,3 are depicted in Fig. 1. The Nth–order Riesz transform canbe coupled with an isotropic multiresolution decomposition (e.g., Laplacian ofGaussian (LoG)) to obtain rotation–covariant (steerable) basis functions [15].
The main idea of the proposed approach is to derive texture signatures frommultiscale Riesz coefficients. An example showing healthy and fibrosis tissuerepresented in terms of their Riesz components with N=2 is depicted in Fig. 2 a).In order to provide a local categorization of the lung parenchyma, lung regionsin 2D axial slices are divided into 32×32 overlapping blocks with a distancebetween contiguous block centers of 16. The Riesz transform is applied to eachblock, and every Riesz component n = 1, . . . , N+1 is mapped to a multiscalerepresentation by convolving them with four LoG filters of scales s = 1, . . . , 4with a dyadic scale progression. In a total of (N+1)×4 subbands, the variancesσn,s of the coefficients are used as texture features along with 22 grey levelhistogram (GLH) bins in [-1050;600] Hounsfield Units (HU). The percentageof air pixels with values ≤ −1000 HU completes the feature space learned bysupport vector machines (SVM) with a Gaussian kernel.
The local dominant texture orientations have an influence on the reparti-tion of respective responses of the Riesz components, which is not desirable forcreating robust features with well–defined clusters of instances. For example, arotation of π/2 will switch the responses of h1 and h2 for N=1. To ensure thatthe repartitions of σn,s are comparable for two similar textures having distinct
N = 1 N = 2
N = 3
g(1,0)(x,m) g(2,0)(x,m)g(0,1)(x,m) g(0,2)(x,m)
g(0,3)(x,m)
g(1,1)(x,m)
g(3,0)(x,m) g(2,1)(x,m) g(1,2)(x,m)
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
56
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Steerable Riesz wavelets [Unser2013, Depeursinge2014b]
• Steerability:
L1 ⇥ L2
g(1,0)(R✓0x,0) = cos ✓0 g(1,0)(x,0) + sin ✓0 g(0,1)(x,0)
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
57
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Steerable Riesz wavelets [Unser2013, Depeursinge2014b]
• Local rotation-invariance:
L1 ⇥ L2
✓max
(m) := argmax
✓02[0,2⇡)
✓cos ✓
0
g(1,0)(f(x),m) + sin ✓
0
g(0,1)(f(x),m)
◆
µM =1
|M |
Z
M
✓g(1,0)(f(R✓
max
(m) x),m)
◆2
dm)
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
MULTIDIRECTIONAL TEXTURE OPERATORS
58
• Locally rotation-invariant operators over
• Locally “aligning” directional operators
• Steerable Riesz wavelets [Depeursinge2014b, Unser2013]
• Local rotation-invariance:
L1 ⇥ L2
✓max
(m) := argmax
✓02[0,2⇡)
✓cos ✓
0
g(1,0)(f(x),m) + sin ✓
0
g(0,1)(f(x),m)
◆
µM =1
|M |
Z
M
✓g(1,0)(f(R✓
max
(m) x),m)
◆2
dm)
Encodes the LOIDs independently from their local orientations!
No binarization required!
Available in 3D [Chenouard2012, Depeursinge2015a], and combined with feature learning [Depeursinge2014b].
g(f(x),m) = g(f(R✓0x),m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
• Operators characterizing the LOIDs
MULTIDIRECTIONAL TEXTURE OPERATORS
GLCMs Riesz wavelets ( )
59
GLCM contrast
GLC
M c
ontra
st
d = 1 (�k1 = 1,�k2 = 0)
d=
1(�
k1=
0,�
k2=
1)
N = 2
1 |M|Z M
✓g (
2,0)(f(x
),m
)◆2
dm
1
|M |
Z
M
✓g(0,2)(f(x),m)
◆2
dm
aligned Riesz wavelets ( )N = 2
1
|M |
Z
M
✓g(0,2)(f(R✓
max
(m) x),m)
◆2
dm1 |M|Z M
✓g (
2,0)(f(R
✓m
ax
(m)x),m
)◆2
dm
• Operators characterizing the LOIDs
MULTIDIRECTIONAL TEXTURE OPERATORS
60
GLCMs
GLCM contrast
GLC
M c
ontra
st
d = 1 (�k1 = 1,�k2 = 0)
d=
1(�
k1=
0,�
k2=
1)
Riesz wavelets ( )
1 |M|Z M
✓g (
2,0)(f(x
),m
)◆2
dm
1
|M |
Z
M
✓g(0,2)(f(x),m)
◆2
dm
N = 2
aligned Riesz wavelets ( )N = 2
1
|M |
Z
M
✓g(0,2)(f(R✓
max
(m) x),m)
◆2
dm1 |M|Z M
✓g (
2,0)(f(R
✓m
ax
(m)x),m
)◆2
dm
MULTIDIRECTIONAL TEXTURE OPERATORS
61
• Isotropic or directional analysis? [Depeursinge2014b]
• Outex [Ojala2002]: 24 classes, 180 images/class, 9 rotation angles in
• Texture classification: linear SVMs trained with unrotated images only
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5
1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021
9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035
17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009
Fig. 5. 128⇥ 128 blocks from the 24 texture classes of the Outex database.
1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin
9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool
Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.
180 ⇥ 180 images from rotation angles 20
�, 70�, 90�, 120�,135
� and 150
� of the other seven Brodatz images for eachclass. The total number of images in the test set is 672.
G. Experimental setupOVA SVM models using Gaussian kernels as K(x
i
,x
j
) =
exp(
�||xi�xj ||22�
2k
) are used both to learn texture signatures andto classify the texture instances in the final feature spaceobtained after k iterations. A number of scales J = 6
was used to cover the whole spectrum of the 128 ⇥ 128
subimages in Outex and J = 3 for covering the spectrum of16 ⇥ 16 subimages in Contrib TC 00000. The angle matrixthat maximizes the response of the texture signature at thesmallest scale ⇥
1
(x) (see Eq. (11)) is used to steer Riesztemplates from all scales. The dimensionality of the initialfeature space is J(N + 1). Every texture signature �
N
c,K
iscomputed using the texture instances from the training set.The coefficients from all instances are rotated to locally aligneach signature �N
c,K
and are concatenated to constitute the finalfeature space. The dimensionality of the final feature space isJ ⇥ (N + 1) ⇥ N
c
. OVA SVM models are trained in thisfinal feature space using the training instances. The remainingtest instances obtained are used to evaluate the generalizationperformance. All data processing was performed using MAT-LAB R2012b (8.0.0.783) 64–bit (glnxa64), The MathWorks
Inc., 2012. The computational complexity is dominated by thelocal orientation of �N
c
in Eq. 11, which consists of finding theroots of the polynomials defined by the steering matrix A
✓.It is therefore NP–hard (Non–deterministic Polynomial–timehard), where the order of the polynomials is controlled by theorder of the Riesz transform N .
III. RESULTS
The performance of our approach is demonstrated withthe Outex and the Brodatz databases. The performance oftexture classification is first investigated in Section III-A.The evolution and the convergence of the texture signatures�
N
c,k
through iterations k = 1, . . . , 10 is then studied inSection III-B for the Outex TC 00010 test suite.
A. Rotation–covariant texture classification
The rotation–covariant properties of our approach are eval-uated using Outex TC 00010, Outex TC 00012 and Con-trib TC 00000 test suites. The classification performanceof the proposed approach after the initial iteration (k=1)is compared with two other approaches that are based onmultiscale Riesz filterbanks. As a baseline, the classificationperformance using the energy of the coefficients of the initialRiesz templates was evaluated. Since the cardinality of the
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10"
ini/al"coeffs"
aligned"1st"template"
order of the Riesz transformN
clas
sific
atio
n ac
cura
cy
Riesz waveletsaligned Riesz wavelets
MULTIDIRECTIONAL TEXTURE OPERATORS
62
• Isotropic or directional analysis? [Depeursinge2014b]
• Outex [Ojala2002]: 24 classes, 180 images/class, 9 rotation angles in
• Texture classification: linear SVMs trained with unrotated images only
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5
1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021
9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035
17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009
Fig. 5. 128⇥ 128 blocks from the 24 texture classes of the Outex database.
1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin
9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool
Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.
180 ⇥ 180 images from rotation angles 20
�, 70�, 90�, 120�,135
� and 150
� of the other seven Brodatz images for eachclass. The total number of images in the test set is 672.
G. Experimental setupOVA SVM models using Gaussian kernels as K(x
i
,x
j
) =
exp(
�||xi�xj ||22�
2k
) are used both to learn texture signatures andto classify the texture instances in the final feature spaceobtained after k iterations. A number of scales J = 6
was used to cover the whole spectrum of the 128 ⇥ 128
subimages in Outex and J = 3 for covering the spectrum of16 ⇥ 16 subimages in Contrib TC 00000. The angle matrixthat maximizes the response of the texture signature at thesmallest scale ⇥
1
(x) (see Eq. (11)) is used to steer Riesztemplates from all scales. The dimensionality of the initialfeature space is J(N + 1). Every texture signature �
N
c,K
iscomputed using the texture instances from the training set.The coefficients from all instances are rotated to locally aligneach signature �N
c,K
and are concatenated to constitute the finalfeature space. The dimensionality of the final feature space isJ ⇥ (N + 1) ⇥ N
c
. OVA SVM models are trained in thisfinal feature space using the training instances. The remainingtest instances obtained are used to evaluate the generalizationperformance. All data processing was performed using MAT-LAB R2012b (8.0.0.783) 64–bit (glnxa64), The MathWorks
Inc., 2012. The computational complexity is dominated by thelocal orientation of �N
c
in Eq. 11, which consists of finding theroots of the polynomials defined by the steering matrix A
✓.It is therefore NP–hard (Non–deterministic Polynomial–timehard), where the order of the polynomials is controlled by theorder of the Riesz transform N .
III. RESULTS
The performance of our approach is demonstrated withthe Outex and the Brodatz databases. The performance oftexture classification is first investigated in Section III-A.The evolution and the convergence of the texture signatures�
N
c,k
through iterations k = 1, . . . , 10 is then studied inSection III-B for the Outex TC 00010 test suite.
A. Rotation–covariant texture classification
The rotation–covariant properties of our approach are eval-uated using Outex TC 00010, Outex TC 00012 and Con-trib TC 00000 test suites. The classification performanceof the proposed approach after the initial iteration (k=1)is compared with two other approaches that are based onmultiscale Riesz filterbanks. As a baseline, the classificationperformance using the energy of the coefficients of the initialRiesz templates was evaluated. Since the cardinality of the
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10"
ini/al"coeffs"
aligned"1st"template"
order of the Riesz transformN
clas
sific
atio
n ac
cura
cy
Riesz waveletsaligned Riesz wavelets
Isotropic operators (i.e., ) perform best when not aligned!
N = 0
OUTLINE
• Biomedical texture analysis: background
• Defining texture processes
• Notations, sampling and texture functions
• Texture operators, primitives and invariances
• Multiscale analysis
• Operator scale and uncertainty principle
• Region of interest and operator aggregation
• Multidirectional analysis
• Isotropic versus directional operators
• Importance of the local organization of image directions
• Conclusions
• References
REFERENCES
66
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2015 1
Steerable Wavelet Machines (SWM): LearningMoving Frames for Texture Classification
Adrien Depeursinge, Zsuzsanna Püspöki, John Paul Ward, and Michael Unser
Abstract—We present texture operators encoding class-specific local organizations of image directions (LOID) in arotation-invariant fashion. The operators are based on steerablecircular harmonic wavelets (CHW), offering a rich and yet com-pact initial representation for characterizing natural textures.We enforce the preservation of the joint location and orientationimage structure by building Cartan moving frames (MF) fromlocally-steered multi-order CHWs. In a second step, we usesupport vector machines (SVM) to learn a multi-class shapingmatrix of the initial CHW representations, yielding data-drivenMFs. Intuitively, the learned MFs can be seen as class-specificforged detectors that are applied and rotated at each point ofthe image to evaluate the magnitude of their responses, i.e., thepresence of the corresponding texture class. We experimentallydemonstrate the effectiveness of the proposed operators forclassifying natural textures, which outperformed the reportedperformance of recent approaches on several test suites of theOutex database.
Index Terms—Texture classification, feature learning, mov-ing frames, support vector machines, steerability, rotation-invariance, illumination-invariance, wavelet analysis.
I. INTRODUCTION
THe wealth of digital texture patterns is tightly relatedto the size of the observation window on a discrete
lattice. In an extreme case, an image region composedof one pixel cannot form geometrical structures. There-fore, the characterization of textural patterns often requiresintegrating image operators g (x), x 2 R2 over an imagedomain X . The latter raises two major challenges. First,the action of the integrated operators becomes diffuseover X , which hinders the spatial precision of texturesegmentation approaches. Second, the effect of integrationbecomes even more destructive when unidirectional opera-tors are jointly used to characterize the local organization ofimage directions (LOID) (e.g., directional filterbanks [1, 2],curvelets [3], histogram of gradients (HOG) [4], Haralick [5]).When separately integrated, the responses of unidirectionalindividual operators are not local anymore and their jointresponses become only sensitive to the global amount ofimage directions in the region X . For instance, the joint
responses of image gradients g1,2(x) =≥ØØ @ f (x)
@x1
ØØ,ØØ @ f (x)@x2
ØØ¥
arenot able to discriminate between the two textures classesf1(x) and f2(x) shown in Fig. 1 when integrated over theimage domain X .
This loss of information is detailed by Sifre et al. interms of separable group invariants [6]. When integrated
The authors are with the Biomedical Imaging Group, École Poly-technique Fédérale de Lausanne, Lausanne 1015, Switzerland (e-mail:[email protected]; [email protected]). A. Depeursinge isalso with the MedGIFT group, Institute of Information Systems, Universityof Applied Sciences Western Switzerland, Sierre (HES-SO).
of1(x) : of2(x) :
RX
ØØ @ f (x)@x1
ØØdx
R XØ Ø@
f(x)
@x 2
Ø Ø dx
image gradients
Fig. 1. The joint responses of image gradients≥| @ f (x)@x1
|, | @ f (x)@x2
|¥
are notable to discriminate between the two textures classes f1(x) and f2(x)when integrated over the image domain X . One circle in the gradientrepresentation corresponds to one realization (i.e., full image) of f1,2.
separately, the responses of unidirectional operators be-come invariant to a larger family of roto-translations wheredifferent orientations are translated by different values. Forinstance, it can be observed in Fig. 1 that f2 can be obtainedfrom f1 by vertically translating horizontal bars only andhorizontally translating vertical bars only.
The LOID are key for visual understanding [7] and is theonly property able to differentiate between f1 and f2 inFig. 1. It relates to the joint information between positionsand orientations in images, which is discarded by operatorsthat are invariant to separable groups. The LOID has beenleveraged in the literature to define [8] and discriminatetexture classes [9–12]. It is also at the origin of the successof methods based on local binary patterns (LBP) [9] andits extensions [13–19], which specifically encodes the LOIDwith uniform circular pixel sequences. Although classicaldeep learning approaches are not enforcing the characteri-zation of the LOID [20], deep convolutional networks wererecently proposed by Oyallon et al. to specifically preservethe structure of the roto-translation group [21].
An even bigger challenge is to design texture operatorsthat can characterize the LOID in a rotation-invariantfashion [9, 11]. The latter is very important to recognizeimage structures independently from both their local ori-entations and the global orientation of the image. Exam-ples of such structures are vascular, bronchial or dendritictrees in biomedical images, river deltas or urban areas insatellite images, or even more complex biomedical tissuearchitectures or crystals in petrographic analysis. Since mostof the above-mentioned imaging modalities yield imageswith pixel sizes defined in physical dimensions, we arenot interested in invariance to image scale. We aim todesign texture operators that are invariant to the family ofEuclidean transforms (also called rigid), i.e.,
g (x) = g (Rµ0 x °x0), 8x , x0,µ0 2R2,
·m
CONCLUSIONS
• We presented a general framework to describe and analyse texture information in 2D and 3D
• Tissue structures in 2D/3D medical images contain extremely rich and valuable information to optimize personalized medicine in a non-invasive way
• Invisible to the naked eye!
64
REFERENCES
66
[?] Texture in Biomedical Images, Petrou M.,
L1
L2
M1
M2
M
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2015 1
Steerable Wavelet Machines (SWM): LearningMoving Frames for Texture Classification
Adrien Depeursinge, Zsuzsanna Püspöki, John Paul Ward, and Michael Unser
Abstract—We present texture operators encoding class-specific local organizations of image directions (LOID) in arotation-invariant fashion. The operators are based on steerablecircular harmonic wavelets (CHW), offering a rich and yet com-pact initial representation for characterizing natural textures.We enforce the preservation of the joint location and orientationimage structure by building Cartan moving frames (MF) fromlocally-steered multi-order CHWs. In a second step, we usesupport vector machines (SVM) to learn a multi-class shapingmatrix of the initial CHW representations, yielding data-drivenMFs. Intuitively, the learned MFs can be seen as class-specificforged detectors that are applied and rotated at each point ofthe image to evaluate the magnitude of their responses, i.e., thepresence of the corresponding texture class. We experimentallydemonstrate the effectiveness of the proposed operators forclassifying natural textures, which outperformed the reportedperformance of recent approaches on several test suites of theOutex database.
Index Terms—Texture classification, feature learning, mov-ing frames, support vector machines, steerability, rotation-invariance, illumination-invariance, wavelet analysis.
I. INTRODUCTION
THe wealth of digital texture patterns is tightly relatedto the size of the observation window on a discrete
lattice. In an extreme case, an image region composedof one pixel cannot form geometrical structures. There-fore, the characterization of textural patterns often requiresintegrating image operators g (x), x 2 R2 over an imagedomain X . The latter raises two major challenges. First,the action of the integrated operators becomes diffuseover X , which hinders the spatial precision of texturesegmentation approaches. Second, the effect of integrationbecomes even more destructive when unidirectional opera-tors are jointly used to characterize the local organization ofimage directions (LOID) (e.g., directional filterbanks [1, 2],curvelets [3], histogram of gradients (HOG) [4], Haralick [5]).When separately integrated, the responses of unidirectionalindividual operators are not local anymore and their jointresponses become only sensitive to the global amount ofimage directions in the region X . For instance, the joint
responses of image gradients g1,2(x) =≥ØØ @ f (x)
@x1
ØØ,ØØ @ f (x)@x2
ØØ¥
arenot able to discriminate between the two textures classesf1(x) and f2(x) shown in Fig. 1 when integrated over theimage domain X .
This loss of information is detailed by Sifre et al. interms of separable group invariants [6]. When integrated
The authors are with the Biomedical Imaging Group, École Poly-technique Fédérale de Lausanne, Lausanne 1015, Switzerland (e-mail:[email protected]; [email protected]). A. Depeursinge isalso with the MedGIFT group, Institute of Information Systems, Universityof Applied Sciences Western Switzerland, Sierre (HES-SO).
of1(x) : of2(x) :
RX
ØØ @ f (x)@x1
ØØdx
R XØ Ø@
f(x)
@x 2
Ø Ø dx
image gradients
Fig. 1. The joint responses of image gradients≥| @ f (x)@x1
|, | @ f (x)@x2
|¥
are notable to discriminate between the two textures classes f1(x) and f2(x)when integrated over the image domain X . One circle in the gradientrepresentation corresponds to one realization (i.e., full image) of f1,2.
separately, the responses of unidirectional operators be-come invariant to a larger family of roto-translations wheredifferent orientations are translated by different values. Forinstance, it can be observed in Fig. 1 that f2 can be obtainedfrom f1 by vertically translating horizontal bars only andhorizontally translating vertical bars only.
The LOID are key for visual understanding [7] and is theonly property able to differentiate between f1 and f2 inFig. 1. It relates to the joint information between positionsand orientations in images, which is discarded by operatorsthat are invariant to separable groups. The LOID has beenleveraged in the literature to define [8] and discriminatetexture classes [9–12]. It is also at the origin of the successof methods based on local binary patterns (LBP) [9] andits extensions [13–19], which specifically encodes the LOIDwith uniform circular pixel sequences. Although classicaldeep learning approaches are not enforcing the characteri-zation of the LOID [20], deep convolutional networks wererecently proposed by Oyallon et al. to specifically preservethe structure of the roto-translation group [21].
An even bigger challenge is to design texture operatorsthat can characterize the LOID in a rotation-invariantfashion [9, 11]. The latter is very important to recognizeimage structures independently from both their local ori-entations and the global orientation of the image. Exam-ples of such structures are vascular, bronchial or dendritictrees in biomedical images, river deltas or urban areas insatellite images, or even more complex biomedical tissuearchitectures or crystals in petrographic analysis. Since mostof the above-mentioned imaging modalities yield imageswith pixel sizes defined in physical dimensions, we arenot interested in invariance to image scale. We aim todesign texture operators that are invariant to the family ofEuclidean transforms (also called rigid), i.e.,
g (x) = g (Rµ0 x °x0), 8x , x0,µ0 2R2,
·m
malignant, nonresponder
malignant, responder
benign
pre-malignant
undefined
quant. feat. #1
quan
t. fe
at. #
2
MULTISCALE TEXTURE OPERATORS
• How large must be the region of interest ?
• Enough to evaluate texture stationarity in terms of human perception / tissue biology
• Example with operator: undecimated isotropic Simoncelli’s dyadic wavelets [PoS2000] applied to all image positions
• Operators’ responses are averaged over
M
TEXTURE OPERATORS AND PRIMITIVES
19
• From texture operators to texture measurements
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region of interest
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
M
f(x) g1(f(x),m)
m 2 RM1⇥M2
g2(f(x),m)
original image with regions I
1
|M |
Z
M|g1(f(x),m)|dm
M
feature space
1 |M|Z M
|g2(f(x
),m
)|dm
f(x)
Ma,M b,M c
The averaged responses over the entire image does not correspond to anything visually!
g1(⇢) =
⇢cos
�⇡2 log2
� 2⇢⇡
��, ⇡
4 < ⇢ ⇡0, otherwise.
g2(⇢) =
⇢cos
�⇡2 log2
� 4⇢⇡
��, ⇡
8 < ⇢ ⇡2
0, otherwise.
\g1,2�f(⇢,�)
�= g1,2(⇢,�) · f(⇢,�)
Nor biologically!
BoVW can be used to first reveal the intra-class visual diversity
texture operators region of interestand aggregation
scales
uncertainty principle averaging operators’ responses
directions
isotropic versus directional importance of LOIDs
CONCLUSIONS
• Biomedical textures are realizations of complex non-stationary spatial stochastic processes
• General-purpose image operators are necessary to identify data-specific discriminative scales and directions
65
MULTISCALE TEXTURE OPERATORS
• How large must be the region of interest ?
• Enough to evaluate texture stationarity in terms of human perception / tissue biology
• Example with operator: undecimated isotropic Simoncelli’s dyadic wavelets [PoS2000] applied to all image positions
• Operators’ responses are averaged over
M
TEXTURE OPERATORS AND PRIMITIVES
19
• From texture operators to texture measurements
• The operator is typically applied to all positions of the image by “sliding” its window over the image
• Regional texture measurements can be obtained from the aggregation of over a region of interest
• For instance, integration can be used to aggregate over
• e.g., average:
L1
L2M1
M2
L1 ⇥ · · ·⇥ Ld
·
gn(x,m) m
µ 2 RP
gn(f(x),m) M
m
gn(f(x),m) M
µ =
0
B@µ1...µP
1
CA =1
|M |
Z
M
�gn(f(x),m)
�p=1,...,P
dm
M
f(x) g1(f(x),m)
m 2 RM1⇥M2
g2(f(x),m)
original image with regions I
1
|M |
Z
M|g1(f(x),m)|dm
M
feature space
1 |M|Z M
|g2(f(x
),m
)|dm
f(x)
Ma,M b,M c
The averaged responses over the entire image does not correspond to anything visually!
g1(⇢) =
⇢cos
�⇡2 log2
� 2⇢⇡
��, ⇡
4 < ⇢ ⇡0, otherwise.
g2(⇢) =
⇢cos
�⇡2 log2
� 4⇢⇡
��, ⇡
8 < ⇢ ⇡2
0, otherwise.
\g1,2�f(⇢,�)
�= g1,2(⇢,�) · f(⇢,�)
Nor biologically!
BoVW can be used to first reveal the intra-class visual diversity
F�!
f(!)f(x)
)TextureQbased'biomarkers:'current'limitaGons'
x Assume'homogeneous'texture'properGes'over'the'enGre'lesion'[5]'
'
x NonQspecific'features'x Global'vs'local'characterizaGon'of'image'direcGons'[6]'
Radiology: Volume 269: Number 1—October 2013 n radiology.rsna.org 13
REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al
with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver-sion-recovery sequence, we can define four habitats: high or low postgadolini-um T1 divided into high or low fluid-at-tenuated inversion recovery. When these voxel habitats are projected into the tu-mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5).
Analysis of spatial patterns in cross-sectional images will ultimately re-quire methods that bridge spatial scales from microns to millimeters. One possi-ble method is a general class of numeric tools that is already widely used in ter-restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ-mental selection forces in MR imaging-defined habitats to the evolutionary dy-namics of cancer cells.
Summary
Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re-place the current subjective qualitative assessments of tumor characteristics. The need for quantitative imaging has been clearly recognized by the National Cancer Institute and has resulted in for-mation of the Quantitative Imaging Net-work. A critical objective of this imaging consortium is to use objective, repro-ducible, and quantitative feature metrics extracted from clinical images to develop patient-specific imaging-based prog-nostic models and personalized cancer therapies.
rise to local-regional phenotypic adap-tations. Phenotypic alterations can re-sult from epigenetic, genetic, or chro-mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations.
Emerging Strategies for Tumor Habitat Characterization
A method for converting images to spa-tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is defined by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a fluid-attenuated inversion-recov-ery (or FLAIR) sequence. Voxels in each sequence can be defined as high or low based on their value compared
microenvironment can be rewarded by increased proliferation. This evolution-ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec-tional images (Fig 5).
Interpretation of the subsegmenta-tion of tumors will require computa-tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex-ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character-istics. Conceptually, this approach as-sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to first order, by variations in vascular density and blood flow. The environmental conditions that result from alterations in blood flow, such as hypoxia, acidosis, immune response, growth factors, and glucose, represent evolutionary selection forces that give
Figure 4
Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.
Radiology: Volume 269: Number 1—October 2013 n radiology.rsna.org 13
REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al
with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver-sion-recovery sequence, we can define four habitats: high or low postgadolini-um T1 divided into high or low fluid-at-tenuated inversion recovery. When these voxel habitats are projected into the tu-mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5).
Analysis of spatial patterns in cross-sectional images will ultimately re-quire methods that bridge spatial scales from microns to millimeters. One possi-ble method is a general class of numeric tools that is already widely used in ter-restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ-mental selection forces in MR imaging-defined habitats to the evolutionary dy-namics of cancer cells.
Summary
Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re-place the current subjective qualitative assessments of tumor characteristics. The need for quantitative imaging has been clearly recognized by the National Cancer Institute and has resulted in for-mation of the Quantitative Imaging Net-work. A critical objective of this imaging consortium is to use objective, repro-ducible, and quantitative feature metrics extracted from clinical images to develop patient-specific imaging-based prog-nostic models and personalized cancer therapies.
rise to local-regional phenotypic adap-tations. Phenotypic alterations can re-sult from epigenetic, genetic, or chro-mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations.
Emerging Strategies for Tumor Habitat Characterization
A method for converting images to spa-tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is defined by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a fluid-attenuated inversion-recov-ery (or FLAIR) sequence. Voxels in each sequence can be defined as high or low based on their value compared
microenvironment can be rewarded by increased proliferation. This evolution-ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec-tional images (Fig 5).
Interpretation of the subsegmenta-tion of tumors will require computa-tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex-ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character-istics. Conceptually, this approach as-sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to first order, by variations in vascular density and blood flow. The environmental conditions that result from alterations in blood flow, such as hypoxia, acidosis, immune response, growth factors, and glucose, represent evolutionary selection forces that give
Figure 4
Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.
[5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013'
5'
global'direcGonal'operators:' local'grouped'steering:'
[6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898Q908,'2014.'
TextureQbased'biomarkers:'current'limitaGons'
x Assume'homogeneous'texture'properGes'over'the'enGre'lesion'[5]'
'
x NonQspecific'features'x Global'vs'local'characterizaGon'of'image'direcGons'[6]'
Radiology: Volume 269: Number 1—October 2013 n radiology.rsna.org 13
REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al
with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver-sion-recovery sequence, we can define four habitats: high or low postgadolini-um T1 divided into high or low fluid-at-tenuated inversion recovery. When these voxel habitats are projected into the tu-mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5).
Analysis of spatial patterns in cross-sectional images will ultimately re-quire methods that bridge spatial scales from microns to millimeters. One possi-ble method is a general class of numeric tools that is already widely used in ter-restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ-mental selection forces in MR imaging-defined habitats to the evolutionary dy-namics of cancer cells.
Summary
Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re-place the current subjective qualitative assessments of tumor characteristics. The need for quantitative imaging has been clearly recognized by the National Cancer Institute and has resulted in for-mation of the Quantitative Imaging Net-work. A critical objective of this imaging consortium is to use objective, repro-ducible, and quantitative feature metrics extracted from clinical images to develop patient-specific imaging-based prog-nostic models and personalized cancer therapies.
rise to local-regional phenotypic adap-tations. Phenotypic alterations can re-sult from epigenetic, genetic, or chro-mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations.
Emerging Strategies for Tumor Habitat Characterization
A method for converting images to spa-tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is defined by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a fluid-attenuated inversion-recov-ery (or FLAIR) sequence. Voxels in each sequence can be defined as high or low based on their value compared
microenvironment can be rewarded by increased proliferation. This evolution-ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec-tional images (Fig 5).
Interpretation of the subsegmenta-tion of tumors will require computa-tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex-ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character-istics. Conceptually, this approach as-sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to first order, by variations in vascular density and blood flow. The environmental conditions that result from alterations in blood flow, such as hypoxia, acidosis, immune response, growth factors, and glucose, represent evolutionary selection forces that give
Figure 4
Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.
Radiology: Volume 269: Number 1—October 2013 n radiology.rsna.org 13
REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al
with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a fluid-attenuated inver-sion-recovery sequence, we can define four habitats: high or low postgadolini-um T1 divided into high or low fluid-at-tenuated inversion recovery. When these voxel habitats are projected into the tu-mor volume, we find they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5).
Analysis of spatial patterns in cross-sectional images will ultimately re-quire methods that bridge spatial scales from microns to millimeters. One possi-ble method is a general class of numeric tools that is already widely used in ter-restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ-mental selection forces in MR imaging-defined habitats to the evolutionary dy-namics of cancer cells.
Summary
Imaging can have an enormous role in the development and implementation of patient-specific therapies in cancer. The achievement of this goal will require new methods that expand and ultimately re-place the current subjective qualitative assessments of tumor characteristics. The need for quantitative imaging has been clearly recognized by the National Cancer Institute and has resulted in for-mation of the Quantitative Imaging Net-work. A critical objective of this imaging consortium is to use objective, repro-ducible, and quantitative feature metrics extracted from clinical images to develop patient-specific imaging-based prog-nostic models and personalized cancer therapies.
rise to local-regional phenotypic adap-tations. Phenotypic alterations can re-sult from epigenetic, genetic, or chro-mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of specific ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efficacy and emergence of resistant populations.
Emerging Strategies for Tumor Habitat Characterization
A method for converting images to spa-tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is defined by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a fluid-attenuated inversion-recov-ery (or FLAIR) sequence. Voxels in each sequence can be defined as high or low based on their value compared
microenvironment can be rewarded by increased proliferation. This evolution-ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec-tional images (Fig 5).
Interpretation of the subsegmenta-tion of tumors will require computa-tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex-ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to specific imaging character-istics. Conceptually, this approach as-sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to first order, by variations in vascular density and blood flow. The environmental conditions that result from alterations in blood flow, such as hypoxia, acidosis, immune response, growth factors, and glucose, represent evolutionary selection forces that give
Figure 4
Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.
[5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013'
5'
global'direcGonal'operators:' local'grouped'steering:'
[6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898Q908,'2014.'
MULTIDIRECTIONAL TEXTURE OPERATORS
58
• Isotropic or directional analysis? [DFV2014]
• Outex [OPM2002]: 24 classes, 180 images/class, 9 rotation angles in
• Texture classification: linear SVMs trained with unrotated images only
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5
1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021
9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035
17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009
Fig. 5. 128⇥ 128 blocks from the 24 texture classes of the Outex database.
1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin
9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool
Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.
180 ⇥ 180 images from rotation angles 20
�, 70�, 90�, 120�,135
� and 150
� of the other seven Brodatz images for eachclass. The total number of images in the test set is 672.
G. Experimental setupOVA SVM models using Gaussian kernels as K(x
i
,x
j
) =
exp(
�||xi�xj ||22�
2k
) are used both to learn texture signatures andto classify the texture instances in the final feature spaceobtained after k iterations. A number of scales J = 6
was used to cover the whole spectrum of the 128 ⇥ 128
subimages in Outex and J = 3 for covering the spectrum of16 ⇥ 16 subimages in Contrib TC 00000. The angle matrixthat maximizes the response of the texture signature at thesmallest scale ⇥
1
(x) (see Eq. (11)) is used to steer Riesztemplates from all scales. The dimensionality of the initialfeature space is J(N + 1). Every texture signature �
N
c,K
iscomputed using the texture instances from the training set.The coefficients from all instances are rotated to locally aligneach signature �N
c,K
and are concatenated to constitute the finalfeature space. The dimensionality of the final feature space isJ ⇥ (N + 1) ⇥ N
c
. OVA SVM models are trained in thisfinal feature space using the training instances. The remainingtest instances obtained are used to evaluate the generalizationperformance. All data processing was performed using MAT-LAB R2012b (8.0.0.783) 64–bit (glnxa64), The MathWorks
Inc., 2012. The computational complexity is dominated by thelocal orientation of �N
c
in Eq. 11, which consists of finding theroots of the polynomials defined by the steering matrix A
✓.It is therefore NP–hard (Non–deterministic Polynomial–timehard), where the order of the polynomials is controlled by theorder of the Riesz transform N .
III. RESULTS
The performance of our approach is demonstrated withthe Outex and the Brodatz databases. The performance oftexture classification is first investigated in Section III-A.The evolution and the convergence of the texture signatures�
N
c,k
through iterations k = 1, . . . , 10 is then studied inSection III-B for the Outex TC 00010 test suite.
A. Rotation–covariant texture classification
The rotation–covariant properties of our approach are eval-uated using Outex TC 00010, Outex TC 00012 and Con-trib TC 00000 test suites. The classification performanceof the proposed approach after the initial iteration (k=1)is compared with two other approaches that are based onmultiscale Riesz filterbanks. As a baseline, the classificationperformance using the energy of the coefficients of the initialRiesz templates was evaluated. Since the cardinality of the
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10"
ini/al"coeffs"
aligned"1st"template"
order of the Riesz transformN
clas
sifica
tion
accu
racy
Riesz waveletsaligned Riesz wavelets
Isotropic operators (i.e., ) perform best when not aligned!
N = 0
MULTIDIRECTIONAL TEXTURE OPERATORS
58
• Isotropic or directional analysis? [DFV2014]
• Outex [OPM2002]: 24 classes, 180 images/class, 9 rotation angles in
• Texture classification: linear SVMs trained with unrotated images only
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5
1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021
9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035
17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009
Fig. 5. 128⇥ 128 blocks from the 24 texture classes of the Outex database.
1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin
9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool
Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.
180 ⇥ 180 images from rotation angles 20
�, 70�, 90�, 120�,135
� and 150
� of the other seven Brodatz images for eachclass. The total number of images in the test set is 672.
G. Experimental setupOVA SVM models using Gaussian kernels as K(x
i
,x
j
) =
exp(
�||xi�xj ||22�
2k
) are used both to learn texture signatures andto classify the texture instances in the final feature spaceobtained after k iterations. A number of scales J = 6
was used to cover the whole spectrum of the 128 ⇥ 128
subimages in Outex and J = 3 for covering the spectrum of16 ⇥ 16 subimages in Contrib TC 00000. The angle matrixthat maximizes the response of the texture signature at thesmallest scale ⇥
1
(x) (see Eq. (11)) is used to steer Riesztemplates from all scales. The dimensionality of the initialfeature space is J(N + 1). Every texture signature �
N
c,K
iscomputed using the texture instances from the training set.The coefficients from all instances are rotated to locally aligneach signature �N
c,K
and are concatenated to constitute the finalfeature space. The dimensionality of the final feature space isJ ⇥ (N + 1) ⇥ N
c
. OVA SVM models are trained in thisfinal feature space using the training instances. The remainingtest instances obtained are used to evaluate the generalizationperformance. All data processing was performed using MAT-LAB R2012b (8.0.0.783) 64–bit (glnxa64), The MathWorks
Inc., 2012. The computational complexity is dominated by thelocal orientation of �N
c
in Eq. 11, which consists of finding theroots of the polynomials defined by the steering matrix A
✓.It is therefore NP–hard (Non–deterministic Polynomial–timehard), where the order of the polynomials is controlled by theorder of the Riesz transform N .
III. RESULTS
The performance of our approach is demonstrated withthe Outex and the Brodatz databases. The performance oftexture classification is first investigated in Section III-A.The evolution and the convergence of the texture signatures�
N
c,k
through iterations k = 1, . . . , 10 is then studied inSection III-B for the Outex TC 00010 test suite.
A. Rotation–covariant texture classification
The rotation–covariant properties of our approach are eval-uated using Outex TC 00010, Outex TC 00012 and Con-trib TC 00000 test suites. The classification performanceof the proposed approach after the initial iteration (k=1)is compared with two other approaches that are based onmultiscale Riesz filterbanks. As a baseline, the classificationperformance using the energy of the coefficients of the initialRiesz templates was evaluated. Since the cardinality of the
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10"
ini/al"coeffs"
aligned"1st"template"
order of the Riesz transformN
clas
sifica
tion
accu
racy
Riesz waveletsaligned Riesz wavelets
Isotropic operators (i.e., ) perform best when not aligned!
N = 0
THANKS ! BIG @ EPFLMichael UnserJulien FageotArash Aminiand all members
MedGIFT @ HES-SO Henning MüllerYashin DicenteRoger SchaerRanveer Joyseree Oscar JimenezManfredo Atzori
66
Source code and data available!https://sites.google.com/site/btamiccai2015/
FUNDAMENTALS OF DIGITAL TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS
Adrien Depeursinge, PhD MICCAI 2015 Tutorial on Biomedical Texture Analysis (BTA), Munich, Oct 5 2015
sampled lattices) is not straightforward and raises several chal-lenges related to translation, scaling and rotation invariances andcovariances that are becoming more complex in 3-D.
1.2. Related work and scope of this survey
Depending on research communities, various taxonomies areused to refer to 3-D texture information. A clarification of the tax-onomy is proposed in this section to accurately define the scope ofthis survey. It is partly based on Toriwaki and Yoshida, 2009. Three-dimensional texture and volumetric texture are both general andidentical terms designing a texture defined in R3 and include:
(1) Volumetric textures existing in ‘‘filled’’ objectsfV : x; y; z 2 Vx;y;z ! R3g that are generated by a volumetricdata acquisition device (e.g., tomography, confocal imaging).
(2) 2.5D textures existing on surfaces of ‘‘hollow’’ objects asfC : x; y; z 2 Cu;v ! R3g,
(3) Dynamic textures in two–dimensional time sequences asfS : x; y; t 2 Sx;y;t ! R3g,
Solid texture refers to category (1) and accounts for textures de-fined in a volume Vx,y,z indexed by three coordinates. Solid textureshave an intrinsic dimension of 3, which means that a number ofvariables equal to the dimensionality of the Euclidean space isneeded to represent the signal (Bennett, 1965; Foncubierta-Rodrí-guez et al., 2013a). Category (2) is designed as textured surface inDana and Nayar (1999) and Cula and Dana (2004), or 2.5-dimen-sional textures in Lu et al. (2006) and Aguet et al. (2008), wheretextures C are existing on the surface of 3-D objects and can be in-dexed uniquely by two coordinates (u,v). (2) is also used in Kajiyaand Kay (1989), Neyret (1995), and Filip and Haindl (2009), where3-D geometries are added onto the surface of objects to create real-istic rendering of virtual scenes. Motion analysis in videos can alsobe considered a multi-dimensional texture analysis problembelonging to category (3) and is designed by ‘‘dynamic texture’’in Bouthemy and Fablet (1998), Chomat and Crowley (1999), andSchödl et al. (2000).
In this survey, a comprehensive review of the literature pub-lished on classification and retrieval of biomedical solid textures(i.e., category (1)) is carried out. The focus of this text is on the fea-ture extraction and not machine learning techniques, since onlyfeature extraction is specific to 3-D solid texture analysis.
1.3. Structure of this article
This survey is structured as follows: The fundamentals of 3-Ddigital texture processing are defined in Section 2. Section 3 de-scribes the reviewmethodology used to systematically retrieve pa-pers dealing with 3-D solid texture classification and retrieval. Theimaging modalities and organs studied in the literature are re-viewed in Sections 4 and 5, respectively to list the various expecta-tions and needs of 3-D image processing. The resulting application-driven techniques are described, organized and grouped togetherin Section 6. A synthesis of the trends and gaps of the various ap-proaches, conclusions and opportunities are given in Section 7,respectively.
2. Fundamentals of solid texture processing
Although several researchers attempted to establish a generalmodel of texture description (Haralick, 1979; Julesz, 1981), it isgenerally recognized that no general mathematical model of tex-ture can be used to solve every image analysis problem (Mallat,1999). In this survey, we compare the various approaches based
on the 3-D geometrical properties of the primitives used, i.e., theelementary building block considered. The set of primitives usedand their assumed interactions define the properties of the textureanalysis approaches, from statistical to structural methods.
In Section 2.1, we define the mathematical framework andnotations considered to describe the content of 3-D digital images.The notion of texture primitives as well as their scales and direc-tions are defined in Section 2.2.
2.1. 3-D digitized images and sampling
In Cartesian coordinates, a generic 3-D continuous image is de-fined by a function of three variables f(x,y,z), where f represents ascalar at a point ðx; y; zÞ 2 R3. A 3-D digital image F(i, j,k) of dimen-sions M $ N $ O is obtained from sampling f at points ði; j; kÞ 2 Z3
of a 3-D ordered array (see Fig. 2). Increments in (i, j,k), correspondto physical displacements in R3 parametrized by the respectivespacings (Dx,Dy,Dz). For every cell of the digitized array, the valueof F(i, j,k) is typically obtained by averaging f in the cuboid domaindefined by (x,y,z) 2 [iDx, (i + 1)Dx]; [jDy, (j + 1)Dy]; [kDz, (k +1)Dz]) (Toriwaki and Yoshida, 2009). This cuboid is called a voxel.The three spherical coordinates (r,h,/) are unevenly sampled to(R,H,U) as shown in Fig. 3.
2.2. Texture primitives
The notion of texture primitive has been widely used in 2-D tex-ture analysis and defines the elementary building block of a giventexture class (Haralick, 1979; Jain et al., 1995; Lin et al., 1999). Alltexture processing approaches aim at modeling a given textureusing sets of prototype primitives. The concept of texture primitiveis naturally extended in 3-D as the geometry of the voxel sequenceused by a given texture analysis method. We consider a primitiveC(i, j,k) centered at a point (i, j,k) that lives on a neighborhood ofthis point. The primitive is constituted by a set of voxels with graytone values that forms a 3-D structure. Typical C neighborhoodsare voxel pairs, linear, planar, spherical or unconstrained. Signalassignment to the primitive can be either binary, categorical orcontinuous. Two example texture primitives are shown in Fig. 4.Texture primitives refer to local processing of 3-D images and localpatterns (see Toriwaki and Yoshida, 2009).
Fig. 2. 3-D digitized images and sampling in Cartesian coordinates.
Fig. 3. 3-D digitized images and sampling in spherical coordinates.
178 A. Depeursinge et al. /Medical Image Analysis 18 (2014) 176–196
sampled lattices) is not straightforward and raises several chal-lenges related to translation, scaling and rotation invariances andcovariances that are becoming more complex in 3-D.
1.2. Related work and scope of this survey
Depending on research communities, various taxonomies areused to refer to 3-D texture information. A clarification of the tax-onomy is proposed in this section to accurately define the scope ofthis survey. It is partly based on Toriwaki and Yoshida, 2009. Three-dimensional texture and volumetric texture are both general andidentical terms designing a texture defined in R3 and include:
(1) Volumetric textures existing in ‘‘filled’’ objectsfV : x; y; z 2 Vx;y;z ! R3g that are generated by a volumetricdata acquisition device (e.g., tomography, confocal imaging).
(2) 2.5D textures existing on surfaces of ‘‘hollow’’ objects asfC : x; y; z 2 Cu;v ! R3g,
(3) Dynamic textures in two–dimensional time sequences asfS : x; y; t 2 Sx;y;t ! R3g,
Solid texture refers to category (1) and accounts for textures de-fined in a volume Vx,y,z indexed by three coordinates. Solid textureshave an intrinsic dimension of 3, which means that a number ofvariables equal to the dimensionality of the Euclidean space isneeded to represent the signal (Bennett, 1965; Foncubierta-Rodrí-guez et al., 2013a). Category (2) is designed as textured surface inDana and Nayar (1999) and Cula and Dana (2004), or 2.5-dimen-sional textures in Lu et al. (2006) and Aguet et al. (2008), wheretextures C are existing on the surface of 3-D objects and can be in-dexed uniquely by two coordinates (u,v). (2) is also used in Kajiyaand Kay (1989), Neyret (1995), and Filip and Haindl (2009), where3-D geometries are added onto the surface of objects to create real-istic rendering of virtual scenes. Motion analysis in videos can alsobe considered a multi-dimensional texture analysis problembelonging to category (3) and is designed by ‘‘dynamic texture’’in Bouthemy and Fablet (1998), Chomat and Crowley (1999), andSchödl et al. (2000).
In this survey, a comprehensive review of the literature pub-lished on classification and retrieval of biomedical solid textures(i.e., category (1)) is carried out. The focus of this text is on the fea-ture extraction and not machine learning techniques, since onlyfeature extraction is specific to 3-D solid texture analysis.
1.3. Structure of this article
This survey is structured as follows: The fundamentals of 3-Ddigital texture processing are defined in Section 2. Section 3 de-scribes the reviewmethodology used to systematically retrieve pa-pers dealing with 3-D solid texture classification and retrieval. Theimaging modalities and organs studied in the literature are re-viewed in Sections 4 and 5, respectively to list the various expecta-tions and needs of 3-D image processing. The resulting application-driven techniques are described, organized and grouped togetherin Section 6. A synthesis of the trends and gaps of the various ap-proaches, conclusions and opportunities are given in Section 7,respectively.
2. Fundamentals of solid texture processing
Although several researchers attempted to establish a generalmodel of texture description (Haralick, 1979; Julesz, 1981), it isgenerally recognized that no general mathematical model of tex-ture can be used to solve every image analysis problem (Mallat,1999). In this survey, we compare the various approaches based
on the 3-D geometrical properties of the primitives used, i.e., theelementary building block considered. The set of primitives usedand their assumed interactions define the properties of the textureanalysis approaches, from statistical to structural methods.
In Section 2.1, we define the mathematical framework andnotations considered to describe the content of 3-D digital images.The notion of texture primitives as well as their scales and direc-tions are defined in Section 2.2.
2.1. 3-D digitized images and sampling
In Cartesian coordinates, a generic 3-D continuous image is de-fined by a function of three variables f(x,y,z), where f represents ascalar at a point ðx; y; zÞ 2 R3. A 3-D digital image F(i, j,k) of dimen-sions M $ N $ O is obtained from sampling f at points ði; j; kÞ 2 Z3
of a 3-D ordered array (see Fig. 2). Increments in (i, j,k), correspondto physical displacements in R3 parametrized by the respectivespacings (Dx,Dy,Dz). For every cell of the digitized array, the valueof F(i, j,k) is typically obtained by averaging f in the cuboid domaindefined by (x,y,z) 2 [iDx, (i + 1)Dx]; [jDy, (j + 1)Dy]; [kDz, (k +1)Dz]) (Toriwaki and Yoshida, 2009). This cuboid is called a voxel.The three spherical coordinates (r,h,/) are unevenly sampled to(R,H,U) as shown in Fig. 3.
2.2. Texture primitives
The notion of texture primitive has been widely used in 2-D tex-ture analysis and defines the elementary building block of a giventexture class (Haralick, 1979; Jain et al., 1995; Lin et al., 1999). Alltexture processing approaches aim at modeling a given textureusing sets of prototype primitives. The concept of texture primitiveis naturally extended in 3-D as the geometry of the voxel sequenceused by a given texture analysis method. We consider a primitiveC(i, j,k) centered at a point (i, j,k) that lives on a neighborhood ofthis point. The primitive is constituted by a set of voxels with graytone values that forms a 3-D structure. Typical C neighborhoodsare voxel pairs, linear, planar, spherical or unconstrained. Signalassignment to the primitive can be either binary, categorical orcontinuous. Two example texture primitives are shown in Fig. 4.Texture primitives refer to local processing of 3-D images and localpatterns (see Toriwaki and Yoshida, 2009).
Fig. 2. 3-D digitized images and sampling in Cartesian coordinates.
Fig. 3. 3-D digitized images and sampling in spherical coordinates.
178 A. Depeursinge et al. /Medical Image Analysis 18 (2014) 176–196
sampled
lattices)is
notstraightforw
ardand
raisesseveral
chal-lenges
relatedto
translation,scalingand
rotationinvariances
andcovariances
thatare
becoming
more
complex
in3-D
.
1.2.Relatedwork
andscope
ofthis
survey
Depending
onresearch
communities,
varioustaxonom
iesare
usedto
referto
3-Dtexture
information.A
clarificationof
thetax-
onomyis
proposedin
thissection
toaccurately
definethe
scopeof
thissurvey.Itis
partlybased
onToriw
akiandYoshida,2009.Three-
dimensional
textureand
volumetric
textureare
bothgeneral
andidenticalterm
sdesigning
atexture
definedin
R3and
include:
(1)Volum
etrictextures
existingin
‘‘filled’’objects
fV:x;y
;z2V
x;y;z !
R3g
thatare
generatedby
avolum
etricdata
acquisitiondevice
(e.g.,tomography,confocalim
aging).(2)
2.5Dtextures
existingon
surfacesof
‘‘hollow’’objects
asfC
:x;y;z
2C
u;v!
R3g,
(3)Dynam
ictextures
intw
o–dimensional
time
sequencesas
fS:x;y
;t2Sx;y
;t !R
3g,
Solidtexture
refersto
category(1)
andaccounts
fortextures
de-fined
inavolum
eVx,y,z indexed
bythree
coordinates.Solidtextures
havean
intrinsicdim
ensionof
3,which
means
thatanum
berof
variablesequal
tothe
dimensionality
ofthe
Euclideanspace
isneeded
torepresent
thesignal(Bennett,1965;
Foncubierta-Rodrí-guez
etal.,2013a).Category
(2)is
designedas
texturedsurface
inDana
andNayar
(1999)and
Culaand
Dana
(2004),or2.5-dim
en-sional
texturesin
Luet
al.(2006)and
Aguet
etal.(2008),w
heretextures
Care
existingon
thesurface
of3-Dobjects
andcan
bein-
dexeduniquely
bytw
ocoordinates
(u,v).(2)is
alsoused
inKajiya
andKay
(1989),Neyret
(1995),andFilip
andHaindl(2009),w
here3-D
geometries
areadded
ontothe
surfaceofobjects
tocreate
real-istic
renderingofvirtualscenes.M
otionanalysis
invideos
canalso
beconsidered
amulti-dim
ensionaltexture
analysisproblem
belongingto
category(3)
andis
designedby
‘‘dynamic
texture’’in
Bouthemyand
Fablet(1998),Chom
atand
Crowley
(1999),andSchödlet
al.(2000).In
thissurvey,
acom
prehensivereview
ofthe
literaturepub-
lishedon
classificationand
retrievalof
biomedical
solidtextures
(i.e.,category(1))is
carriedout.The
focusofthis
textis
onthe
fea-ture
extractionand
notmachine
learningtechniques,
sinceonly
featureextraction
isspecific
to3-D
solidtexture
analysis.
1.3.Structureof
thisarticle
Thissurvey
isstructured
asfollow
s:The
fundamentals
of3-D
digitaltexture
processingare
definedin
Section2.
Section3de-
scribesthe
reviewmethodology
usedto
systematically
retrievepa-
persdealing
with
3-Dsolid
textureclassification
andretrieval.The
imaging
modalities
andorgans
studiedin
theliterature
arere-
viewed
inSections
4and
5,respectivelyto
listthe
variousexpecta-
tionsand
needsof3-D
image
processing.Theresulting
application-driven
techniquesare
described,organizedand
groupedtogether
inSection
6.Asynthesis
ofthe
trendsand
gapsof
thevarious
ap-proaches,
conclusionsand
opportunitiesare
givenin
Section7,
respectively.
2.Fundam
entals
ofsolid
texture
processing
Although
severalresearchers
attempted
toestablish
ageneral
model
oftexture
description(H
aralick,1979;
Julesz,1981),
itis
generallyrecognized
thatno
generalmathem
aticalmodel
oftex-
turecan
beused
tosolve
everyim
ageanalysis
problem(M
allat,1999).
Inthis
survey,wecom
parethe
variousapproaches
based
onthe
3-Dgeom
etricalproperties
ofthe
primitives
used,i.e.,theelem
entarybuilding
blockconsidered.The
setof
primitives
usedand
theirassum
edinteractions
definethe
propertiesofthe
textureanalysis
approaches,fromstatisticalto
structuralmethods.
InSection
2.1,we
definethe
mathem
aticalfram
ework
andnotations
consideredto
describethe
contentof3-D
digitalimages.
Thenotion
oftexture
primitives
aswell
astheir
scalesand
direc-tions
aredefined
inSection
2.2.
2.1.3-Ddigitized
images
andsam
pling
InCartesian
coordinates,ageneric
3-Dcontinuous
image
isde-
finedby
afunction
ofthree
variablesf(x,y,z),w
herefrepresents
ascalar
atapointðx;y
;zÞ2R
3.A3-D
digitalimage
F(i,j,k)ofdim
en-sions
M$
N$
Ois
obtainedfrom
sampling
fat
pointsði;j;kÞ2
Z3
ofa3-D
orderedarray
(seeFig.2).Increm
entsin
(i,j,k),correspondto
physicaldisplacem
entsin
R3param
etrizedby
therespective
spacings(D
x,Dy,D
z).Forevery
cellofthedigitized
array,thevalue
ofF(i,j,k)is
typicallyobtained
byaveraging
finthe
cuboiddom
aindefined
by(x,y,z)2
[iDx,(i+
1)Dx];
[jDy,(j+
1)Dy];
[kDz,(k
+1)D
z])(Toriw
akiandYoshida,2009).This
cuboidis
calledavoxel.
Thethree
sphericalcoordinates
(r,h,/)are
unevenlysam
pledto
(R,H,U
)as
shownin
Fig.3.
2.2.Textureprim
itives
Thenotion
oftextureprim
itivehas
beenwidely
usedin
2-Dtex-
tureanalysis
anddefines
theelem
entarybuilding
blockof
agiven
textureclass
(Haralick,1979;
Jainet
al.,1995;Lin
etal.,1999).A
lltexture
processingapproaches
aimat
modeling
agiven
textureusing
setsofprototype
primitives.The
conceptoftextureprim
itiveis
naturallyextended
in3-D
asthe
geometry
ofthevoxelsequence
usedby
agiven
textureanalysis
method.W
econsider
aprim
itiveC(i,j,k)
centeredat
apoint
(i,j,k)that
liveson
aneighborhood
ofthis
point.Theprim
itiveis
constitutedby
aset
ofvoxelswith
graytone
valuesthat
formsa3-D
structure.TypicalC
neighborhoodsare
voxelpairs,
linear,planar,
sphericalor
unconstrained.Signal
assignment
tothe
primitive
canbe
eitherbinary,
categoricalor
continuous.Tw
oexam
pletexture
primitives
areshow
nin
Fig.4.Texture
primitives
referto
localprocessingof3-D
images
andlocal
patterns(see
Toriwakiand
Yoshida,2009).
Fig.2.3-D
digitizedim
agesand
sampling
inCartesian
coordinates.
Fig.3.3-D
digitizedim
agesand
sampling
insphericalcoordinates.
178A.D
epeursingeet
al./MedicalIm
ageAnalysis
18(2014)
176–196
Stanford UniversityDaniel RubinOlivier GevaertAnn Leung
Dimitri Van de Ville, UNIGECamille Kurtz, Paris Descartes Pierre-Alexandre Poletti, HUGJohn-Paul Ward, UCF
REFERENCES (SORTING IN ALPHABETICAL ORDER)
67
[Ahonen2009] Ahonen, T.; Matas, J.; He, C. & Pietikäinen, M. Salberg, A.-B.; Hardeberg, J. & Jenssen, R. (Eds.)Rotation Invariant Image Description with Local Binary Pattern Histogram Fourier Features Image Analysis, Springer Berlin Heidelberg, 2009, 5575, 61-70
[Aerts2014] Aerts, H. J. W. L.; Velazquez, E. R.; Leijenaar, R. T. H.; Parmar, C.; Grossmann, P.; Carvalho, S.; Bussink, J.; Monshouwer, R.; Haibe-Kains, B.; Rietveld, D.; Hoebers, F.; Rietbergen, M. M.; Leemans, C. R.; Dekker, A.; Quackenbush, J.; Gillies, R. J. & Lambin, P. Decoding Tumour Phenotype by Noninvasive Imaging Using a Quantitative Radiomics Approach Nature Communications, Nature Publishing Group, a division of Macmillan Publishers Limited. All Rights Reserved, 2014, 5
[Blakemore1969] Blakemore, C. & Campbell, F. W. On the Existence of Neurones in the Human Visual System Selectively Sensitive to the Orientation and Size of Retinal ImagesThe Journal of Physiology, 1969, 203, 237-260
[Candes2000] Candès, E. J. & Donoho, D. L. Curvelets - A Surprisingly Effective Nonadaptive Representation For Objects with Edges Curves and Surface Fitting, Vanderbilt University Press, 2000, 105-120
[Chenouard2011] Chenouard, N. & Unser, M. 3D Steerable Wavelets and Monogenic Analysis for Bioimaging 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2011, 2132-2135
[Chenouard2012] Chenouard, N. & Unser, M. 3D Steerable Wavelets in Practice IEEE Transactions on Image Processing, 2012, 21, 4522-4533
REFERENCES (SORTING IN ALPHABETICAL ORDER)
68
[Dalal2005] Dalal, N. & Triggs, B. Histograms of Oriented Gradients for Human Detection Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), IEEE Computer Society, 2005, 1, 886-893
[Depeursinge2012a] Depeursinge, A.; Foncubierta-Rodrguez, A.; Van De Ville, D. & Müller, H. Multiscale Lung Texture Signature Learning Using The Riesz Transform Medical Image Computing and Computer-Assisted Intervention MICCAI 2012, Springer Berlin / Heidelberg, 2012, 7512, 517-524
[Depeursinge2012b] Depeursinge, A.; Van De Ville, D.; Platon, A.; Geissbuhler, A.; Poletti, P.-A. & Müller, H. Near-Affine-Invariant Texture Learning for Lung Tissue Analysis Using Isotropic Wavelet Frames IEEE Transactions on Information Technology in BioMedicine, 2012, 16, 665-675
[Depeursinge2014a] Depeursinge, A.; Foncubierta-Rodrguez, A.; Van De Ville, D. & Müller, H. Three-Dimensional Solid Texture Analysis and Retrieval in Biomedical Imaging: Review and Opportunities Medical Image Analysis, 2014, 18, 176-196
[Depeursinge2014b] Depeursinge, A.; Foncubierta-Rodrguez, A.; Van De Ville, D. & Müller, H. Rotation-Covariant Texture Learning Using Steerable Riesz Wavelets IEEE Transactions on Image Processing, 2014, 23, 898-908
[Depeursinge2015a] Depeursinge, A.; Pad, P.; Chin, A. C.; Leung, A. N.; Rubin, D. L.; Müller, H. & Unser, M. Optimized Steerable Wavelets for Texture Analysis of Lung Tissue in 3-D CT: Classification of Usual Interstitial PneumoniaIEEE 12th International Symposium on Biomedical Imaging, IEEE, 2015, 403-406
[Depeursinge2015b] Depeursinge, A.; Chin, A. C.; Leung, A. N.; Terrone, D.; Bristow, M.; Rosen, G. & Rubin, D. L. Automated Classification of Usual Interstitial Pneumonia Using Regional Volumetric Texture Analysis in High-Resolution CTInvestigative Radiology, 2015, 50, 261-267
REFERENCES (SORTING IN ALPHABETICAL ORDER)
69
[Dumas2009] Dumas, A.; Brigitte, M.; Moreau, M.; Chrétien, F.; Baslé, M. & Chappard, D.Bone Mass and Microarchitecture of Irradiated and Bone Marrow-Transplanted Mice: Influences of the Donor StrainOsteoporosis International, Springer-Verlag, 2009, 20, 435-443
[Galloway1975] Galloway, M. M.Texture Analysis Using Gray Level Run LengthsComputer Graphics and Image Processing, 1975, 4, 172-179 (Gal1975)
[Gerlinger2012] Gerlinger, M.; Rowan, A. J.; Horswell, S.; Larkin, J.; Endesfelder, D.; Gronroos, E.; Martinez, P.; Matthews, N.; Stewart, A.; Tarpey, P.; Varela, I.; Phillimore, B.; Begum, S.; McDonald, N. Q.; Butler, A.; Jones, D.; Raine, K.; Latimer, C.; Santos, C. R.; Nohadani, M.; Eklund, A. C.; Spencer-Dene, B.; Clark, G.; Pickering, L.; Stamp, G.; Gore, M.; Szallasi, Z.; Downward, J.; Futreal, P. A. & Swanton, C.Intratumor Heterogeneity and Branched Evolution Revealed by Multiregion Sequencing New England Journal of Medicine, 2012, 366, 883-892
[Gurcan2009] Gurcan, M. N.; Boucheron, L. E.; Can, A.; Madabhushi, A.; Rajpoot, N. M. & Yener, B. Histopathological Image Analysis: A Review IEEE Reviews in Biomedical Engineering, 2009, 2, 147-171
[Haidekker2011] Haidekker, M. A. Texture AnalysisAdvanced Biomedical Image Analysis, John Wiley & Sons, Inc., 2010, 236-275
[Haralick1979] Haralick, R. M.Statistical and Structural Approaches to Texture Proceedings of the IEEE, 1979, 67, 786-804
REFERENCES (SORTING IN ALPHABETICAL ORDER)
70
[Khan2013] Khan, A. M.; El-Daly, H.; Simmons, E. & Rajpoot, N. M.HyMaP: A Hybrid Magnitude-Phase Approach to Unsupervised Segmentation of Tumor Areas in Breast Cancer Histology ImagesJournal of pathology informatics, Medknow Publications, 2013, 4
[Kumar2012] Kumar, V.; Gu, Y.; Basu, S.; Berglund, A.; Eschrich, S. A.; Schabath, M. B.; Forster, K.; Aerts, H. J. W. L.; Dekker, A.; Fenstermacher, D.; Goldgof, D. B.; Hall, L. O.; Lambin, P.; Balagurunathan, Y.; Gatenby, R. A. & Gillies, R. J. Radiomics: The Process and the ChallengesMagnetic Resonance Imaging, 2012, 30, 1234-1248
[Lazebnik2005] Lazebnik, S.; Schmid, C. & Ponce, J. A Sparse Texture Representation Using Local Affine Regions IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Computer Society, 2005, 27, 1265-1278
[Malik2001] Malik, J.; Belongie, S.; Leung, T. & Shi, J. Contour and Texture Analysis for Image Segmentation International Journal of Computer Vision, Kluwer Academic Publishers, 2001, 43, 7-27
[Mosquera2015] Mosquera-Lopez, C.; Agaian, S.; Velez-Hoyos, A. & Thompson, I. Computer-Aided Prostate Cancer Diagnosis From Digitized Histopathology: A Review on Texture-Based SystemsIEEE Reviews in Biomedical Engineering, 2015, 8, 98-113
[Ojala2002] Ojala, T.; Pietikäinen, M. & Mäenpää, T. Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24, 971-987
REFERENCES (SORTING IN ALPHABETICAL ORDER)
71
[Petrou2006] Petrou, M. & Garcia Sevilla, P.Image Processing: Dealing with TextureWiley, 2006
[Petrou2011] Petrou, M.Deserno, T. M. (Ed.) Texture in Biomedical Images Biomedical Image Processing, Springer-Verlag Berlin Heidelberg, 2011, 157-176
[Portilla2000] Portilla, J. & Simoncelli, E. P.A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients International Journal of Computer Vision, Kluwer Academic Publishers, 2000, 40, 49-70
[Romeny2011] ter Haar Romeny, B. M.Deserno, T. M. (Ed.) Multi-Scale and Multi-Orientation Medical Image Analysis Biomedical Image Processing, Springer-Verlag Berlin Heidelberg, 2011, 177-196
[Sifre2014] Sifre, L. & Mallat, S.Rigid-Motion Scattering for Texture Classification Submitted to International Journal of Computer Vision, 2014, abs/1403.1687, 1-19
[Simoncelli1995] Simoncelli, E. P. & Freeman, W. T. The Steerable Pyramid: A Flexible Architecture for Multi-Scale Derivative ComputationProceedings of International Conference on Image Processing, 1995, 3, 444-447
[Storath2014] Storath, M.; Weinmann, A. & Unser, M. Unsupervised Texture Segmentation Using Monogenic Curvelets and the Potts Model IEEE International Conference on Image Processing (ICIP), 2014, 4348-4352
REFERENCES (SORTING IN ALPHABETICAL ORDER)
72
[Unser2011] Unser, M.; Chenouard, N. & Van De Ville, D.Steerable Pyramids and Tight Wavelet Frames in IEEE Transactions on Image Processing, 2011, 20, 2705-2721
[Unser2013] Unser, M. & Chenouard, N. A Unifying Parametric Framework for 2D Steerable Wavelet Transforms SIAM Journal on Imaging Sciences, 2013, 6, 102-135
[Varma2005] Varma, M. & Zisserman, A. A Statistical Approach to Texture Classification from Single ImagesInternational Journal of Computer Vision, Kluwer Academic Publishers, 2005, 62, 61-81
[Ward2014] Ward, J. & Unser, M.Harmonic Singular Integrals and Steerable Wavelets in Applied and Computational Harmonic Analysis, 2014, 36, 183-197
[Ward2015] Ward, J.; Pad, P. & Unser, M. Optimal Isotropic Wavelets for Localized Tight Frame Representations IEEE Signal Processing Letters, 2015, 22, 1918-1921
[Xu2012] Xu, J.; Napel, S.; Greenspan, H.; Beaulieu, C. F.; Agrawal, N. & Rubin, D. Quantifying the Margin Sharpness of Lesions on Radiological Images for Content-Based Image Retrieval Medical Physics, 2012, 39, 5405-5418
L2(Rd)
L2(Rd)