Image semantic coding using OTB
-
Upload
melaneum -
Category
Technology
-
view
1.047 -
download
2
description
Transcript of Image semantic coding using OTB
1
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
Image Semantic Coding using OTB
Marie Liénou - Marine CampedelTélécom ParisTech
July 2009
2
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
SUMMARY
Semantic CodingOTB tool
COCNotion of
semantic CodingA promising
approach
Development of an OTB toolConclusion and
perspectives
3
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
COC = COmpetence Centre…
Tripartite agreement between CNES – DLR and Télécom ParisTech
Signed in June 2005
Goal : joint action on image understanding SAR/Optical, HR and VHR, temporal series Feature extraction, modeling, indexing, compression,
(interactive) classification, interpretation, knowledge representation, reasoning, …
Means ~ 4 new phds / year ~10 permanent researchers partially involved financial support for specific actions (studentships, engineers,
post-docs)
4
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
Image Semantic coding
Semantic Coding
Compression Reduce data while
ensuring informational content
MeaningUnderstandingInterpretationImage to text?
[Barnard et al., 2003 ; Jeon et al., 2003][Li et Bretschneider, 2006]
Goal: find an image representation able to capture the contained semantics Idea: use text indexing approach + active learning
5
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
Image Semantic coding
Feature extraction
Quantization
« visual words »
Indexing Mining
Active learning
Visual interaction Manual annotation
Where is semantics?
Automatic annotation
6
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
Image Semantic coding vs KIM
« Design and evaluation of HMC for Image Information Mining » Daschiel and Datcu IEEE transaction on multimedia, vol 7, no6, dec. 2005
7
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
A promising approach
Feature extraction Segmentation, arbitrar regions “Classical” signature: color, texture, shape descriptors Experiments: intensity mean and variance in each spectral band
Quantization K-Means: each estimated cluster corresponds to one “visual word” K estimated using MDL (Minimum Description Length) descriptor
Bag-of-words signature for semantics identification Count visual words on image regions which will be annotated Normalize (tf-idf)
Exploitation using machine learning (SVM, LDA)
8
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
Marie Lienou PhD work (march 2009) Tested on several VHR (multispectral) images Compared to other classification approachs (GMM, SVM)
Recognition accuracy demonstrated for “semantically complex” classes Ex: “urban area”
LDA = fast + does not need negative examples
A promising approach
Feature extraction
QuantizationClassificationSVM, LDA
Count words
Feature extraction
ClassificationGMM, SVM
Majority rule
Annotations
Low level annotations
Visual word production
9
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
OTB tool: cocSemanticCoding
Feature extraction Vectorial image with as many components as feature dimension Exploitation of OTB extractors at each pixel
Quantization Use of K-Means filter
Bag-of-words signature Count visual words on image regions which will be annotated Normalization (tf-idf)
Learning from manual annotation Fluid interface facilities Learn LDA from only target samples Learn SVM from target samples and counter examples Classify the whole image Iterate
10
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
OTB tool: cocSemanticCoding
11
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
OTB tool: cocSemanticCoding
12
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
OTB tool: cocSemanticCoding
13
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
14
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
15
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
16
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
Learning and classification toolsLDA on occurrence dataSVM on TFiDF data (features)Both results can be obtained with same labeling for comparisonDifficulty for the user : compute features adapted to the underlying semantics
17
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
Conclusion
OTB useful features Vectorial image representation Great diversity of available filters (extractors, classifiers)
New = LDA classifier + estimator Visualization tools
cocSemanticCoding tool availability www.tsi.enst.fr/~campedel/ will be updated
Necessity to valorize research results Engineering process (C++ programming) Not easy but OTB is a nice initiative to help researchers In the future: centralize processing tools (in OTB) + easy their
exploitation (documentations, interfaces)
18
Com
pe t
ence
Cen
tre
on I
nfo
rmat
ion
Ext
r act
ion
an
d I
mag
e U
nd
erst
a nd
ing
for
Ear
th O
bs e
rvat
ion
Perspectives
Other COC tools should be integrated in cocSemanticCoding MDL to estimate of visual words number new feature extractors (QMF-based texture descriptors) Feature selection Complete relevance feedback framework
New approaches for image interpretation From semantics to knowledge? Knowledge engineering: modeling (ontologies) + reasoning Several works on characterizing relations between identified
concepts and/or image objects