1 Evaluation 2001-2005 14 November 2005 INRIA Rocquencourt IMEDIA Image and Multimedia Indexing,...
-
Upload
lee-cannon -
Category
Documents
-
view
220 -
download
2
Transcript of 1 Evaluation 2001-2005 14 November 2005 INRIA Rocquencourt IMEDIA Image and Multimedia Indexing,...
1
Evaluation 2001-2005
14 November 2005
INRIA Rocquencourt
http://www-rocq.inria.fr/imedia/
IMEDIAImage and Multimedia Indexing,
Browsing and Retrieval
214 November 2005 IMEDIA
The Team (November 2005)
Senior members INRIA personnel
Nozha Boujemaa (DR2) Anne Verroust-Blondet (CR1)
Jean-Paul Chièze Research Engineer [part-time] Laurence Bourcier Team Assistant
Scientific Adviser Donald Geman (1/2 time, Pr. Johns Hopkins)
External collaborators Michel Crucianu (Pr. CNAM) [3 years mob. IMEDIA] Valérie Gouet-Brunet (MdC CNAM) [2 years mob. IMEDIA] Jean-Philippe Tarel (CR1 LCPC) [2 years mob. IMEDIA] Olivier Buisson INA Researcher (Institut National de l’Audiovisuel)
Marie-Luce Viaud INA Researcher
314 November 2005 IMEDIA
Post-docs /Expert engineers
Sabri Boughorbel Marin Ferecatu Alexis Joly Itheri YahiaouiPhD students Olfa Besbes Mohamed Chaouch Nizar Grira Nicolas Hervé Hichem Houissa Julien Law-To
The TeamNon permanent members
Former team members
Peter Belhumeur (Sab. visit) Prof. Columbia Univ. NY
François Fleuret (CR) EPFL researcher
Yuchun Fang (Post-doc) Assistant Prof. Shanghai Univ.
Andreas Rauber (Post-doc) Assoc. Prof. Vienna Univ. of Technology
Sylvain Bernard (PhD) Research Engineer (GE Health Care)
Julien Fauqueur (PhD) Research Assoc. Cambridge
Hichem Sahbi (PhD) Research Assoc. Cambridge
Bertrand Le Saux (PhD) Research Assoc. CMLA - ENS
Present team members
414 November 2005 IMEDIA
Overview
Objectives
Results and Contributions
Applications and Grants
Positioning
Future objectives
514 November 2005 IMEDIA
Objectives
Design and Develop new Methods for Visual Information Retrieval by Content
Visual content indexing Visual appearance modeling
Constructing efficient indexes for minimizing query cost
Interactive browsing, querying and retrieval Similarity learning
Clustering techniques
Relevance feedback: learning from user interaction
Combine keyword annotation (when available) search with visual-content search
614 November 2005 IMEDIA
Key Issues Fidelity of physical-content descriptors to visual
appearance Numerical gap vs. Semantic gap
Rich user expression : Partial visual query formulation focused on user interest
(region-based or point-based)
Subjective preference by relevance feedback mechanism
Mental image search and “page zero” problem
Smart navigation
Cross-media indexing and retrieval
714 November 2005 IMEDIA
General Methodological Issues Image content description:
analysis, segmentation;
considering specific and generic content
Learning from few examples:
Active learning for efficient personalization mechanism
Semi-supervised clustering
Adaptive Clustering (interactive SVM-based refinement)
Information theory: Mental Image search
814 November 2005 IMEDIA
Overview Objectives
Results and Contributions Visual Content Description
Clustering Methods
Relevance Feedback Mechanism
Mental Image Search
Applications and Grants
Positioning
Future Objectives
914 November 2005 IMEDIA
Visual Content Description Generic content:
Global image signature: combined color-structure signature (MMCBIR 01, LNCS 05), shape signature (ICIP 05), 3D signature,
Local image description: region-based (JVLC 04), color point-based (CBAIVL/CVPR 01)
Specific content: Face detection (IJCV 01, JMLR 05) Face recognition (Biometric WS/ECCV 02) Fingerprints recognition (ACCV 02)
IKONA search engine demo availablehttp://www-rocq.inria.fr/imedia/ikona.html
1014 November 2005 IMEDIA
Basic color histogram
Local Color activity descriptor (before combination with shape and texture descrip.)
Numerical Gap / Fidelity vs Weakness of signatures
1114 November 2005 IMEDIA
FaceRecognition
Dynamic
programming on local
entropy map features
WBA/ ECCV 2002 (LNCS)
Specific Content Image Database
1214 November 2005 IMEDIA
Coarse-to-Fine Strategy forFace Detection
(Nested partitions of the set of possible poses– IJCV01)( Hierarchy of SVM-classifiers - JMLR 05)
1314 November 2005 IMEDIA
Local Description of the Image
R
p
Region-based query Points-based query
Region SegmentationPoint of interest
extraction
1414 November 2005 IMEDIA
Region-based Indexing and Retrieval
User interest selection (Visual query):
Lavender regions regardless the
background information
X
Yj
Xi
Y
Yj
Yi
Y
Xj
Xi
X n
jicc
n
jijicc
n
jijicc
n
jijiquad ayxayyaxxYXd
1, 1,1,1,
2),(
New Coarse Segmentation +Fine Region Description
Introduction of ADCS Signature+Generalized Quadratic Distance
JVLC 2004
1514 November 2005 IMEDIA
Precise Search by Local Color Invariants Descriptors
CVPR/CBAIVL 01
Optimal order of color differential invariantRobustness to JPEG coding
Color constancy
1614 November 2005 IMEDIA
Overview Objectives Results and Contributions
Visual Content Description Clustering Methods Relevance Feedback Mechanism Mental Image Search
Applications and Grants Positioning Future Objectives
1714 November 2005 IMEDIA
Clustering Methods
Context: unknown number of clusters, competitive agglomeration approaches
Application: image database categorization, image segmentation
Contributions: Adaptive robust clustering (ICPR02) : Noise cluster and
cluster density/shape adapting Entropy regularization and extension to non linearly
separable data (IEEE Fuz.Sys05) Active semi-supervised learning (MIR05, IEE VISP 05)
1814 November 2005 IMEDIA
Active Semi-Supervised Categorization
Learning from few examples:Fully automatic categories could do not reflect user expectations
User constraints indicate how similarity space is different from feature spaceNew clustering
objective function that takes into account
violation cost of “must-link” and “can-not-link” constraints
IEE Vision, Image & Signal Processing, to appear
1914 November 2005 IMEDIA
Active Semi-Supervised Categorization
Active selection of constraints:Identifying the ambiguous data items with weak membership
Supervision effort Identifying non compact and less separated clusters from their neighbors
Identify the frontier of the least well separated cluster using the fuzzy hypervolume:
Ck is the covariance matrix
2014 November 2005 IMEDIA
Illustration
Scientific databases: Gene Expression Studies
Plants with long stems and round leaves
Textured plants, …
must-link
Can not-link
Generalist databases applicable to video-keyframes for smart video abstract
Class1
Class2
Class3
Class4
2114 November 2005 IMEDIA
Overview Objectives Results and Contributions
Visual Content Description Clustering Methods Relevance Feedback Mechanism Mental Image Search
Applications and Grants Positioning Future Objectives
2214 November 2005 IMEDIA
Relevance Feedback Mechanism
Example: search for Cézanne Paintings
Positive ExamplesNegative Examples
Selection strategy?
Most informative images Most similar images
Online Personalization of Retrieval Results
2314 November 2005 IMEDIA
Contribution to Components of RF Mechanism :
Learner: kernels inducing insensitivity to the scale of the data in the feature vector space
Selector: active learning selection criterion that minimizes the redundancy between the samples
SVM-based decision function select least redundant (orthogonal) items among most ambiguous
items
User: consistent annotation?
Extensive study of user strategies
[MIR04, MIR05, AVIVDiLib'05 ][ACM Multimedia journal (under revision)]
Active Relevance Feedback Framework
2414 November 2005 IMEDIA
Overview Objectives Results and Contributions
Visual Content Description Clustering Methods Relevance Feedback Mechanism Mental Image Search
Applications and Grants Positioning Future Objectives
2514 November 2005 IMEDIA
Mental Picture Retrieval
Context: No starting image example or keyword
A person has a picture “in mind”, e.g., a face painting Scene
Problem: How to reach the target? Bayesian framework
Composition from Visual Thesaurus
2614 November 2005 IMEDIA
Bayesian Framework Components:
Answer Model: Discover answer models which match human behavior
Display Model: (Optimization Problem)
Discover approximations to the optimal display
Each display should catch as much as possible information about target from user.
=> The idea is to maximize mutual information between
target and answer. )|()();( XYHYHYXI Reduction in uncertainty of r.v. Y due to r.v. X
2714 November 2005 IMEDIA
Mental face retrieval: Complications Mental matching involves human memory,
perception and opinions. Images are not indexed by semantic content,
but rather by low-level features (“semantic gap”).
Face recognition is easier, yet unsolved. Sparse literature.
Best Paper Award A-V-based Biometric Person Authentication (AVBA'2005)
Joint work with Sagem Corp.
2814 November 2005 IMEDIA
Query by “Visual Words” Composition
Rejected images
Landscapes
Visual Thesaurus:set of similar regions categories“Cityscape”
? Retrieved images
2914 November 2005 IMEDIA
Query composition interface => The Visual Thesaurus = summary of region categories (cluster prototypes set)
Category 23
Category 48
[MTAP 05]
Query by “Visual Words” Composition
3014 November 2005 IMEDIA
Symbolic Indexing
“Inverted visual files”in MTAP 05
3114 November 2005 IMEDIA
Additional Results Cross-modal Indexing and Retrieval
Copy detection and more generally semantic behavior of local descriptors for selective video content retrieval
Kernels for similarity learning
Extensive study of user strategies in relevance feedback.
3D model indexing and retrieval, 2D shape descriptors
3214 November 2005 IMEDIA
3D model retrieval
3314 November 2005 IMEDIA
Overview
Objectives Results and Contributions Applications and Grants Positioning Future objectives
3414 November 2005 IMEDIA
Applications and Grants Scientific content collections:
Remote sensing images (ACI QuerySat – CNES, IGN)
Biodiversity images (ACI Biotim – INRA/NASC, IRD)
Audio-visual content: TV news (RIAM Mediaworks – TF1 Tv; INA)
Personal and prof. content (IP-FP6 AceMedia)
Art and Design: Alinari collection
Security application: Pedophilia images (Central Judiciary Police Dep. Europ. STOP)
Biometry (Face - Sagem, fingerprints – Thales)
3514 November 2005 IMEDIA
Other Grants NoE-FP6 Muscle
Important involvement (WP leader, NoE deputy scientific
coordinator, steering committee)
NoE-FP6 Delos
PAI Galilée (recognition for video-surveillance with Modena Univ.)
Associated-Team ViMining with NII
RNRT - RECIS (FT R&D, INSA, NF)
3614 November 2005 IMEDIA
IKONA Search Engine
Images courtesy of Alinari (Oldest private European art photo
archive)
3714 November 2005 IMEDIA
Relevance of hybrid signatures: visual + semantic
information
keyword: “building”
[MIR05]
3814 November 2005 IMEDIAStarting point for RF
Costal area with visible boats
3914 November 2005 IMEDIA
Gene expression studies on “Arabidopsis”
Images courtesy of NASC (Nottingham Arabidopsis Stock Centre)
Jointly with INRA
4014 November 2005 IMEDIA
Leaf IdentificationSmithsonian databaseShape descriptor [ICIP05]
Images courtesy of Peter Belhumeur (Columbia Univ. NY)
4114 November 2005 IMEDIA
Copy detection
False Alarm
Detected copy
4214 November 2005 IMEDIA
Security ApplicationCriminal Investigation within Pedophilia Images
Central Judiciary Police Department within EC « STOP »
Ikona prototype for “Ministère de l’Intérieur”
4314 November 2005 IMEDIA
USER INTERFACE
Annotate display given a target face
4414 November 2005 IMEDIA
Overview
Objectives Results and Contributions Applications and Grants Positioning Future objectives
4514 November 2005 IMEDIA
INRIA PositioningWrt. INRIA’s strategic goals (2nd): Developing multimedia data and information processing
INRIA projects: ARIANA: probabilistic and variational image analysis for earth
observation, joint ACI QuerySat on remote sensing image indexing, Muscle NoE
LEAR: focus on object recognition involving offline learning methods (learning datasets) while we work on information retrieval and develop different learning methods from few examples (on-line) for image clustering and search personalization - complementary, joint AceMedia FP6
VISTA: Video indexing – complementary, NoE Muscle, MediaWorks,
TEXMEX (SymC): Pluri-disciplinary project (NLP, ImageP.,DB), we have joint interest to feature space structuring and hybrid indexing. (Texmex: audio, video, NLP, visual…); AceMedia and NoE Muscle
4714 November 2005 IMEDIA
National Positioning Telecom Paris – SIP: Remote sensing indexing,
partner within ACI QuerySat, 3D indexing
INT ARTEMIS: 2D and 3D indexing
Ecole Centrale Lyon (L. Chen): face detection recognition, TechnoVision IV2.
INSA Lyon IRIS (J-M Jolion): local descriptors
ENSEA ETIS : Relevance feedback, Muscle NoE
Ecole des Mines (JP. Vert): kernel design
4814 November 2005 IMEDIA
International PositioningVery active domain, below non-exhaustive list T.Huang (Urbana-Champaign), Ed. Chang (U.Cal.Santa-Barbara),
Relevance feedback, A. Smeulders (ISIS group U. Amsterdam), D. Lowe (Univ. BC), A.
Zisserman (Oxford), H. Bishof (Tech. Univ Graz); point-based features
J. Wang (Penn State Univ.), region-based retrieval P. Belhumeur (Columbia Univ.), Leaf species identification and shape
descriptors S. Satoh (NII – Japan) Associated-team “ViMining”, saliency
detection, face detection, image and text–based retrieval R. Cucchiara (Univ. Modena) PAI Gallileo, biometry and video
surveillance, 3D indexing A. Delbimbo (Univ. Florence) NoEDelos, 3D indexing H. Frigui (Univ. NSF-INRIA), semi-supervised clustering T. Tan (CASIA) Liama project
4914 November 2005 IMEDIA
Overview
Objectives Results and Contributions Applications and Grants Positioning Future Objectives
5014 November 2005 IMEDIA
Future Scientific Objectives
Visual content description Saliency investigation for selective content
retrieval Geometric consistency of local descriptors Specific content: 2D/3D shape (biodiversity),
extension of face detection methods to be invariant to view point
Efficient search in large collections of imagesMultidimensional data structure indexing (example: multiple queries processing)
5114 November 2005 IMEDIA
Future Scientific Objectives (cont.)
Mental image search: improved models for perceptual similarity for a higher
degree of coherence between system models and actual human behavior
More efficient visual thesaurus construction methods (hierarchical description with relational clustering)
Toward scalable methods: semi-supervised clustering, Relevance Feedback
Hybrid image and text indexing and retrieval: extension to semi-annotated databases,
dynamic weighting of text and visual rankings
5214 November 2005 IMEDIA
Future Applications
Biodiversity: Pollen database indexing and retrieval (INRA)
Remote sensing image collection - QuerySat
Design Trends (FP6 Strep – TREND, start January 2006)
Audi-visual: INFOMAGIC (“Pôle de compétitivité” IdF IMVN)
SIGMUND (RIAM with INA)
Security IRFACE: : jointly with Liama and INT on Iris-face biometry,
Information filtering with “Ministère de l’Intérieur”
5314 November 2005 IMEDIA
Future Plan
A common project between IMEDIA and the Database Research Group VERTIGO of the Cedric/CNAM Lab is planned
5414 November 2005 IMEDIA
Planned Joint IMEDIA Project INRIA/CNAM composition
INRIA personnel Nozha Boujemaa (DR2) Anne Verroust-Blondet (CR1)
Scientific Adviser Donald Geman (1/2 time, Pr. Johns Hopkins)
CNAM personnel Michel Crucianu (Pr. CNAM) Valérie Gouet-Brunet (MdC CNAM) Michel Scholl (Pr. CNAM) [part-time]
External collaborators Jean-Philippe Tarel (CR1 LCPC) Olivier Buisson INA Researcher (National Institute of Audiovisual)
Marie-Luce Viaud INA Researcher
Research engineer Jean-Paul Chièze (part-time)
Post-Doc and Engineer (4)
PhD (9)Team Assistant: Laurence Bourcier
5514 November 2005 IMEDIA
Summary
Promising scientific results
Smooth evolution of current research directions
Important application impact
Highly competitive context
Support for INRIA research scientist hiring highly
appreciated (major risk)
5614 November 2005 IMEDIA
Thanks for your attention
http://www-rocq.inria.fr/imedia/