Post on 20-Dec-2015
RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR
SEGMENTATION AND DIMENSION REDUCTION OF FEATURES
Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu
Department of Computer Science
Kent State University
September, 2009 Kent State University 2
Data Mining and Knowledge Management
• Processing multimedia objects
• Defining and extracting features
• Feature dimension reduction
• Multimedia data retrieval
• Knowledge representation and management
September, 2009 Kent State University 3
Current Tasks
• Off-line data training– Segment images – batch mode– Find region of interest (ROI)– Interface with feature extraction and analysis– Feature domain processing
September, 2009 Kent State University 4
Current Tasks (cont.)
• Users Interfaces– Reading user-input images– Segmentation– Find ROI– Feature extraction of ROI– Compare with trained data in repository– Return data (images) satisfying certain criteria
September, 2009 Kent State University 5
Data Training
Image &Feature DataRepository
Segmentation Finding ROI Interface Feature ExtractDimension Reduction
Sending Images for Processing Store Feature Data back
Image Domain Feature Domain
September, 2009 Kent State University 6
Image Domain Procesisng
• Segmentation – Color VQ, Texture based image segmentation
• Find ROI – ROI occupies large area– ROI locates near the image center– ROI contains homogenous texture
September, 2009 Kent State University 7
Color-Texture SegmentationApplications
• Identify Regions of Interest (ROI) in a scene
• Image classification
• Image annotation
• Object based image and video coding
September, 2009 Kent State University 8
Color-Texture SegmentationCurrent Limitations
• Many existing techniques work well on homogeneous color regions, while natural scenes are rich in color and texture.
• Many texture segmentation algorithms require the estimation of texture model parameters, which is a difficult problem and often requires a good homogeneous region for robust estimation.
September, 2009 Kent State University 9
Color-Texture SegmentationAdvantage of Color VQ and Texture
based segmentation • Does not attempt to estimate a specific
model for a texture region.
• Tests for the homogeneity of a given color-texture pattern, which is computationally more feasible than estimation of model parameters.
September, 2009 Kent State University 10
Color-Texture SegmentationTwo-Step Process
• Color Quantization– Performed in the color space without
consideration of spatial distribution of colors.– Label each pixel with a quantized color to form
a class-map.
• Spatial Segmentation– Performed on the class-map
September, 2009 Kent State University 11
Color-Texture SegmentationColor Quantization
• Use Peer Group Filtering
• As a result, coarse quantization can be obtained while preserving the color information in the original images.
• Usually 10-20 colors are needed in the images of natural scenes.
September, 2009 Kent State University 12
Color-Texture SegmentationCriteria for Good Segmentation
WWT
W
C
i
C
i ZziiW
ZzT
SSSJ
S
mzSS
mzS
i
/)(
Define
class. same the tobelonging points of variance total theis
and
Let
1 1
2
2
September, 2009 Kent State University 13
Color-Texture Segmentation-A Criterion for Good Segmentation
• When the color classes are more separated from each other, J is getting larger.
• If all color classes are uniformly distributed over the entire image, J tends to be small.
September, 2009 Kent State University 14
Color-Texture SegmentationA Criterion for Good Segmentation
• Now let us recalculate J over each segmented region instead of the entire class-map and define the average by
• A segmentation which can minimize J is considered a good segmentation.
k
Kk JMN
J1
September, 2009 Kent State University 15
Color-Texture Segmentation-Spatial Segmentation
• Seed Determination
• Seed Growing
• Region Merge
September, 2009 Kent State University 17
ROI Determination
• Find ROI – Mechanism– Pixel closer to the center contributes more
weight to the region it belongs to.
– Region with more pixels tends to get higher weight
dw
1
September, 2009 Kent State University 18
Results of Image Domain Processing
• Results of Color Quantization
• Results of Finding ROI
Results of Image Domain Processing
March, 2004 Kent State University 19
V1 = 500, V2 = 1, V3 = 0.5 AutoV1 = 500, V2 = 1, V3 = 0.5
Results of Image Domain Processing
March, 2004 Kent State University 20
V1 = 500, V2 = 1, V3 = 0.5 Auto
September, 2009 Kent State University 22
Interface with Feature Domain
• Find the rectangle circumscribing the ROI
• Store its coordinate information into to a temporary file for feature domain’s use.
September, 2009 Kent State University 23
Feature Domain(Overview)• Two Stages:
– Feature Extraction
– Dimension Reduction (DR)
Image &Feature DataRepository
Interface
Store Feature Data back
Image Domain Feature ExtractDimension Reduction
Feature Domain
September, 2009 Kent State University 24
Implementations
• Acquire ROI information from the image domain
• Extract features based on Gabor Filter and color histogram on HSV space
• Integrate two feature spaces
• Reduce the high feature dimensions to a very low number
September, 2009 Kent State University 25
Implementations (cont.)
• Calculate the similarity measurement between the query object and the objects in the image repository
• Search the similar images in the repository based on similarity index
• Output the corresponding retrieval images
• Knowledge extraction
September, 2009 Kent State University 26
Feature Extraction Algorithm
• Gabor Filter Feature– One of the most important wavelets with multi-
scale and multi-resolution – Mainly reflect texture information
• Color histogram on HSV space– Provide color features
September, 2009 Kent State University 27
Gabor Filter Concept
• A complete but non-orthogonal basis wavelet set
• A significant aspect: localized frequency description – composed of space information
September, 2009 Kent State University 28
Gabor(cont.)
• A two dimensional Gabor function g(x, y) and its Fourier transform G(u, v) can be written as:
jWx
yxyxg
yxyx
22
1exp
2
1),(
2
2
2
2
2
2
2
2)(
2
1exp),(
vu
vWuvuG
yvxu andwhere 2/1,2/1
September, 2009 Kent State University 29
Gabor(cont.)
• Let g(x, y) be the mother Gabor wavelet, then this self-similar filter dictionary can be obtained by appropriate dilations and rotations of g(x, y) through the generating function
number. scale themeans
ns.orientatio ofnumber total theis and , where
)cossin(),sincos(
int,,1),,(),(
m
Knππ/θ
yxayandyxax
nmayxGayxgmm
mmn
September, 2009 Kent State University 30
Color Histogram in HSV Space
• HSV color space includes – Hue (H)– Saturation (S)– Value (V or Lightness)
• Only consider Hue and saturation information, since the lightness of pictures is very sensitive to the surrounding conditions.
September, 2009 Kent State University 32
HSV space bands
• Design bands in the HSV space– 8 hue bands – 4 saturation bands, – Total 32 sub-spaces
• Compute color histogram feature in each sub-space to form 32 feature dimensions eventually
September, 2009 Kent State University 33
Feature Integration
• Normalize both Gabor filter and HSV color histogram features
• Set a weight factor to balance two feature spaces. Usually Gabor filter features will have the bigger weight value.
September, 2009 Kent State University 34
DR Algorithm
• Disadvantages in the high dimension space– The computational complexity arise sharply– The database indexing becomes difficult
• Principal Component Analysis (PCA) – PCA seeks to reduce the dimension of the data
by finding a few orthogonal linear combinations (Principal Component “PC”)
September, 2009 Kent State University 35
DR implementation
• Original feature dimensions– Gabor filter features: 6*5*2 = 60– HSV color histogram features: 4*8 = 32– Total dimensions: 92
• Feature dimensions after DR– 10 ~15 dimensions
September, 2009 Kent State University 36
Simulation Results in the Feature Domain
• We randomly select 11 query pictures as the test samples in this report.
• At each query time, at most 14 retrieval pictures are retrieved.
• The minimum square error method is served as the similarity measurement.
• The value in the tables as below means the positive pictures out of the 14 retrieval pictures.
September, 2009 Kent State University 37
Performance between different feature extraction techniques
• the integration of Gabor Filter and HSV color Histogram
gains the better performance. • See pictures in detail. Click here
Query pic#
1 2 3 4 5 6 7 8 9 10 11
Gabor 6 7 7 4 12 1 1 2 4 3 2
HSV 8 2 9 1 2 3 1 1 4 2 3
Integrated 10 5 11 4 12 3 3 2 5 2 4
September, 2009 Kent State University 38
Performance between with and without DR applied
• The performance after DR applied slightly degrades on average in comparison to the results before DR takes on
stage • See pictures in detail. Click here
Query pic#
1 2 3 4 5 6 7 8 9 10
11
Integrated
10 5 11 4 12 3 3 2 5 2 4
DR 9 6 5 5 12 2 1 1 4 3 2
September, 2009 Kent State University 39
More Simulations
• Performance between different weight used
• Performance between different dimensions retained after DR
September, 2009 Kent State University 40
Final Integration Results
• Simulation results when both the image domain and the feature domain are used
• See the detail pictures, Click here
September, 2009 Kent State University 41
Integration
• UAV media capture and analysis
• WWW based media analysis
• Vehicle based media capture and analysis
September, 2009 Kent State University 42
Future ResearchExtended to video objects
• Object based video coding
• Non-object based video coding
• Video indexing
• Knowledge extraction and management