11. Image Data Analytics - Jacobs University Bremen...Data Analytics 405 Visualization and Computer...
Transcript of 11. Image Data Analytics - Jacobs University Bremen...Data Analytics 405 Visualization and Computer...
Visualization and Computer Graphics LabJacobs University
11. Image Data Analytics
Data Analytics 377
Visualization and Computer Graphics LabJacobs University
Motivation
• Images (and even videos) have become a popular data format for storing information digitally.
Data Analytics 378
Visualization and Computer Graphics LabJacobs University
Motivation
• Traditionally, scientific and medical imaging techniques were considered.
Data Analytics 379
Visualization and Computer Graphics LabJacobs University
Motivation
• Nowadays, photographs form the majority.
1 pixel
Data Analytics 380
Visualization and Computer Graphics LabJacobs University
Image Analytics?
• Similarity computation is difficult.• Similar images may look very different.
– E.g., two photographs of cars may have completely different colors and shapes.
• Digital image processing allows for the detection of features in the images.
• Semantic queries on images is difficult without metadata.
Data Analytics 381
Visualization and Computer Graphics LabJacobs University
Semantic gap
• Image processing allows for the detection of regions of certain properties (typically homogeneous regions).
• Heuristics can be used to combine homogeneous regions to larger structures.
• Larger structures can be considered as low-level semantics.– E.g., detect human hands.
• The problem is to bridge the low-level semanstics to high-level semantics as in queries:– E.g., show me all images, where presidents are shaking hands.
• The gap between the low-level and high-level semantics is called the semantic gap.
Visualization and Computer Graphics LabJacobs University
11.1 Digital Image Processing
Data Analytics 383
Visualization and Computer Graphics LabJacobs University
Digital Image Processing
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 384
Visualization and Computer Graphics LabJacobs University
Image Aquisition
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 385
Visualization and Computer Graphics LabJacobs University
Image Enhancement
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 386
Visualization and Computer Graphics LabJacobs University
Image Restoration
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 387
Visualization and Computer Graphics LabJacobs University
Morphological Processing
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 388
Visualization and Computer Graphics LabJacobs University
Segmentation
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 389
Visualization and Computer Graphics LabJacobs University
Object Recognition
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 390
Visualization and Computer Graphics LabJacobs University
Representation & Description
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 391
Visualization and Computer Graphics LabJacobs University
Image Compression
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 392
Visualization and Computer Graphics LabJacobs University
Colour Image Processing
Image Acquisition
Image Restoration
Morphological Processing
Segmentation
Representation & Description
Image Enhancement
Object Recognition
Problem Domain
Colour Image Processing
Image Compression
Data Analytics 393
Visualization and Computer Graphics LabJacobs University
11.2 Image Segmentation
Data Analytics 394
Visualization and Computer Graphics LabJacobs University
Definition
• Image segmentation is the operation of partitioning an image into a collection of connected sets of pixels.
Data Analytics 395
Visualization and Computer Graphics LabJacobs University
Definition
A segmentation is a partition of an image I into a set ofregions S satisfying the following conditions:1. Partition covers the whole image.2. No regions intersect.3. Each region is homogeneous within itself.4. Adjacent regions form no homogeneous region when
united.
Data Analytics 396
Visualization and Computer Graphics LabJacobs University
Region Growing• Region growing techniques start with one pixel of a
potential region and try to grow it by adding adjacent pixels till the pixels being compared are too disimilar.
• The first pixel selected can be just the first unlabeled pixel in the image or a set of seed pixels can be chosen from the image.
• Usually a statistical test is used to decide which pixels can be added to a region, e.g., threshold on
to decide whether pixel with intensity y is added, where X denotes the mean, S standard deviation, and N is the number of pixels in the region.
(N-1) * NT = -------------- (y - X) / S
(N+1)
221/2
Data Analytics 397
Visualization and Computer Graphics LabJacobs University
Region Growing
• Result:
image
segmentation
Data Analytics 398
Visualization and Computer Graphics LabJacobs University
Clustering• Apply a clustering approach on the pixel intensities.• Histogram-based approaches try to automatically
split the range of the intensities by looking into minima of the histogram.
Data Analytics 399
Visualization and Computer Graphics LabJacobs University
Clustering
• We can use the clustering methods we have been looking into.
• For example, we can use k-means with a guessed number of segments k.
• There exist modifications to k-means that look into local statistics to consider spatial distribution of pixels.
Data Analytics 400
Visualization and Computer Graphics LabJacobs University
Clustering• Results for k-means:
Data Analytics 401
Visualization and Computer Graphics LabJacobs University
Clustering• Graph cut algorithms are widely used.• Let G = (V,E) be a graph. • The vertices V represent the pixels.• Each edge (u,v) has a weight w(u,v) that represents
the similarity between u and v.• The goal is to partition the vertices into disjoint sets
with high similarity within each set and low similarity across sets.
• Graph G can be broken into 2 disjoint graphs by removing edges that connect these sets.
• The segmentation is obtained by finding minimal cuts of G.
Data Analytics 402
Visualization and Computer Graphics LabJacobs University
Clustering
• Graph cuts with normalized cut:
2
2 2
2 2
41 3
2
2 2
3
22
2
1
3 3Ncut(A,B) = ------- + ------
21 16
A B
cut(A,B) = ∑ w(u,v).uεA, vεB
cut(A, B) cut(A,B)Ncut(A,B) = ------------- + -------------
asso(A,V) asso(B,V)
asso(A,V) = ∑ w(u,t)u∈A, t∈V
Data Analytics 403
Visualization and Computer Graphics LabJacobs University
Clustering
• Graph cut results:
Data Analytics 404
Visualization and Computer Graphics LabJacobs University
11.3 Image Collection Data Analytics
Data Analytics 405
Visualization and Computer Graphics LabJacobs University
Image collections
• Image segmentation helps to identify certain features in the image.
• They do not work perfectly.• However, if we are trying to perform a data analytics
approach on a collection of images, the data segmentation results are only of use for low-level semantics queries.
• Is there a way to analyze a collection of images on a higher level?
Data Analytics 406
Visualization and Computer Graphics LabJacobs University
Data analytics
• Assume that we can characterize an image by a number of image descriptors, we can try to compute similarities based on those descriptors.
• Then, we can build a similarity (or distance) matrix of pairwise (dis-)similarities of images.
• Based on the similarity (or distance) matrix, we can apply the data analytics approaches:– Cluster approaches,– Classification approaches,– Interactive visual analysis based on MDS projections.
Data Analytics 407
Visualization and Computer Graphics LabJacobs University
Data analytics example
• Corel data set includes 1,000 photographs on 10 different themes, described by 150 dimensions (SIFT descriptors).
• The Medical data set is of magnetic resonance (MRI) images and has 540 objects and 28 dimensions (Fourier descriptors and energies derived from histograms, plus mean intensity and standard deviation).
Data Analytics 408
Visualization and Computer Graphics LabJacobs University
Projection-based visualization of labeled data
Data Analytics 409
Visualization and Computer Graphics LabJacobs University
11.4 Image Descriptors
Data Analytics 410
Visualization and Computer Graphics LabJacobs University
Image descriptors
• Fourier analysis• Wavelet analysis• SIFT features• Color statistics
Data Analytics 411
Visualization and Computer Graphics LabJacobs University
Image Transforms
• Many times, image processing tasks are best performed in a domain other than the spatial domain.
• Key steps(1) Transform the image(2) Carry the task(s) in the transformed domain.(3) Apply inverse transform to return to the spatial domain.
Data Analytics 412
Visualization and Computer Graphics LabJacobs University
Fourier Series Theorem
• Any periodic function f(t) can be expressed as a weighted sum (infinite) of sine and cosine functions of varying frequency:
is called the “fundamental frequency”
Data Analytics 413
Visualization and Computer Graphics LabJacobs University
Fourier Series
α1
α2
α3
Data Analytics 414
Visualization and Computer Graphics LabJacobs University
Continuous Fourier Transform (FT)
• Transforms a signal (i.e., function) from the spatial (x) domain to the frequency (u) domain.
where
Data Analytics 415
Visualization and Computer Graphics LabJacobs University
Example: Removing undesirable frequencies
remove highfrequencies
reconstructedsignal
frequenciesnoisy signal
To remove certainfrequencies, set theircorresponding F(u)coefficients to zero!
Data Analytics 416
Visualization and Computer Graphics LabJacobs University
How do frequencies show up in an image?
• Low frequencies correspond to slowly varying pixel intensities (e.g., continuous surface).
• High frequencies correspond to quickly varying pixel intensities (e.g., edges)
Original Image Low-passed
Data Analytics 417
Visualization and Computer Graphics LabJacobs University
Example of noise reduction using FT
Input image
Output image
Spectrum (frequency domain)
Band-reject filter
Data Analytics 418
Visualization and Computer Graphics LabJacobs University
Extending FT in 2D
• Forward FT
• Inverse FT
Data Analytics 419
Visualization and Computer Graphics LabJacobs University
Discrete Fourier Transform
• Assume that f(x,y) is M x N.
• Forward DFT
• Inverse DFT:
Data Analytics 420
Visualization and Computer Graphics LabJacobs University
Extending DFT to 2D2D cos/sin functions
Interpretation:
Data Analytics 421
Visualization and Computer Graphics LabJacobs University
Magnitude and Phase of DFT
only phase
only magnitude phase (woman)magnitude (rectangle)
phase (rectangle)magnitude (woman)
Data Analytics 422
Visualization and Computer Graphics LabJacobs University
Wavelet transform
• An alternative to Fourier transforms are wavelet transforms.
• They have the advantage that they represent the image at multiple levels of details.
Data Analytics 423
Visualization and Computer Graphics LabJacobs University
B-spline representation
coarsening
details
Data Analytics 424
Visualization and Computer Graphics LabJacobs University
B-spline representation
Haar wavelet transform
Data Analytics 425
Visualization and Computer Graphics LabJacobs University
Haar wavelets
Basis function
Wavelet function
Data Analytics 426
Visualization and Computer Graphics LabJacobs University
Haar wavelets
Data Analytics 427
Visualization and Computer Graphics LabJacobs University
Multiresolution representation
Data Analytics 428
Visualization and Computer Graphics LabJacobs University
Multiresolution representation
• Object is represented as a sequence of resolutions.• The resolutions are called levels (levels of detail,
LOD)• The differences are called detail coefficients.• The levels build a multiresolution hierarchy:
• The level is the base level.• The base level does not need to be represented by a
regular mesh. All levels use then semi-regular meshes.
Data Analytics 429
Visualization and Computer Graphics LabJacobs University
Multiresolution representation with Haar wavelets
Data Analytics 430
Visualization and Computer Graphics LabJacobs University
2D Haar wavelet transform
• 2D basis and wavelet functions are tensor productsof 1D basis and wavelet functions.
Data Analytics 431
Visualization and Computer Graphics LabJacobs University
2D Haar wavelet transformBasis:
Data Analytics 432
Visualization and Computer Graphics LabJacobs University
2D Haar wavelet transform
Data Analytics 433
Visualization and Computer Graphics LabJacobs University
2D Haar wavelet transformAlternative construction:
Use 2D basis function
and three 2D wavelet functions
Data Analytics 434
Visualization and Computer Graphics LabJacobs University
2D Haar wavelet transformBasis:
Data Analytics 435
Visualization and Computer Graphics LabJacobs University
2D Haar wavelet transform
Advantage: One obtains undistorteddownscaled versions of the 2D image.
Data Analytics 436
Visualization and Computer Graphics LabJacobs University
2D wavelet transform in RGB space
Data Analytics 437
Visualization and Computer Graphics LabJacobs University
Image compression
100% 21% 4% 1%Error: 0% 5% 10% 15%
Haar wavelets:
Data Analytics 438
Visualization and Computer Graphics LabJacobs University
Image compressionJPEG 2000: Cohen-Daubechies-Feauveau wavelets
Data Analytics 439
Visualization and Computer Graphics LabJacobs University
Image compressionJPEG 2000: lossy compression leads to blurring.
Data Analytics 440
Visualization and Computer Graphics LabJacobs University
SIFT features
• The scale-invariant feature transform (SIFT) is another transform that can be used to describe image characteristics.
Data Analytics 441
Visualization and Computer Graphics LabJacobs University
Canonical Frames
Data Analytics 442
Visualization and Computer Graphics LabJacobs University
Multi-Scale Oriented Patches
Extract oriented patches at multiple scales
Data Analytics 443
Visualization and Computer Graphics LabJacobs University
Application: Image Stitching
Data Analytics 444
Visualization and Computer Graphics LabJacobs University
Multi-Scale Oriented Patches
1. Detect an interesting patch with an interest operator. Patches are translation invariant.
2. Determine its dominant orientation.3. Rotate the patch so that the dominant orientation
points upward. This makes the patches rotation invariant.
4. Do this at multiple scales, converting them all to one scale through sampling.
5. Convert to illumination “invariant” form
Data Analytics 445
Visualization and Computer Graphics LabJacobs University
Idea of SIFT
Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters
Data Analytics 446
Visualization and Computer Graphics LabJacobs University
Claimed Advantages of SIFT
• Locality: features are local, so robust to occlusion and clutter (no prior segmentation)
• Distinctiveness: individual features can be matched to a large database of objects
• Quantity: many features can be generated for even small objects
• Efficiency: close to real-time performance• Extensibility: can easily be extended to wide range of
differing feature types, with each adding robustness
Data Analytics 447
Visualization and Computer Graphics LabJacobs University
Overall Procedure at a High Level1. Scale-space extrema detection
2. Keypoint localization
3. Orientation assignment
4. Keypoint description
Search over multiple scales and image locations.
Fit a model to determine location and scale.Select keypoints based on a measure of stability.
Compute best orientation(s) for each keypoint region.
Use local image gradients at selected scale and rotationto describe each keypoint region.
Data Analytics 448
Visualization and Computer Graphics LabJacobs University
11/15/2016 448
Using SIFT for Matching “Objects”
Data Analytics 449
Visualization and Computer Graphics LabJacobs University
Color statistics
• We have been looking at greyscale images (application: medical imaging data).
• We have been using histograms on color distribution.• However, photographs are typically color images.• How can we process color statistics?
Data Analytics 450
Visualization and Computer Graphics LabJacobs University
11.5 Color Models
Data Analytics 451
Visualization and Computer Graphics LabJacobs University
Electromagnetic spectrum
purple blue green yellow orange red
Data Analytics 452
Visualization and Computer Graphics LabJacobs University
Visible light spectrum
Data Analytics 453
Visualization and Computer Graphics LabJacobs University
Relative sensitivity of human eye
• The ability of the human eye to distinguish colors isbased on the sensitivity in the retina to light of different wavelength.
Data Analytics 454
Visualization and Computer Graphics LabJacobs University
Tristimulus
• The retina contains 3 types of receptor cells, which iscalled tristimulus.
• The receptors are most responsive to light of wavelengths 420nm, 534nm, and 564nm.
Data Analytics 455
Visualization and Computer Graphics LabJacobs University
RGB color model
Idea: • Use three wavelengths R, G, and B that reflect
monochromatic light and represent a tristimulus.• Other colors are obtained by combining/mixing the
three components R, G, and B.
Data Analytics 456
Visualization and Computer Graphics LabJacobs University
RGB color model
Implementation: • Choose three colors:
– R = red (700 nm)– G = green (546.1 nm)– B = blue (435.8 nm)
• Arrange them in a 3D Cartesian coordinate systemsuch that
Data Analytics 457
Visualization and Computer Graphics LabJacobs University
RGB color model
• This model allows the generation of colors c with
where .
Data Analytics 458
Visualization and Computer Graphics LabJacobs University
RGB color cube
• All colors c that can be generated are represented bythe unit cube in the 3D Cartesian coordinate system.
red
green
blue
black
white
grey
yellow
magenta
cyan
Data Analytics 459
Visualization and Computer Graphics LabJacobs University
Additive color scheme
• RGB color model is additive, i.e., adding colors makesthe resulting color brighter.
• Application: color monitors.
Data Analytics 460
Visualization and Computer Graphics LabJacobs University
Composition example
Data Analytics 461
Visualization and Computer Graphics LabJacobs University
Compositing in the RGB color cube
Data Analytics 462
Visualization and Computer Graphics LabJacobs University
Caveat
• It is often believed that the RGB color modelreflects the tristimulus of the human eye.
• This is wrong.• In particular, the large wavelength of the human eye‘s
tristimulus is 565 nm, which is not red but ratheryellow-green.
Data Analytics 463
Visualization and Computer Graphics LabJacobs University
Choice of RGB wavelength
• The choice of the wavelength in the RGB model has historical and practical reasons.
• When first monitors were developed, generatingmonochromatic light was a difficult task.
• The chosen wavelength were those that could begenerated most easily.
Data Analytics 464
Visualization and Computer Graphics LabJacobs University
Reconstruction of visible spectral light• In order to reconstruct all wavelengths of the visible
spectral light, we have to add the R, G, and B components with the following weighting factors:
Data Analytics 465
Visualization and Computer Graphics LabJacobs University
Reconstruction of visible spectral light
• The negative values indicate that not all visible colorscan be produced with the RGB color model.
• Nevertheless, close approximations can be achieved.
Visualization and Computer Graphics LabJacobs University
11.6 Assignment
Data Analytics 467
Visualization and Computer Graphics LabJacobs University
Assignment 9• Download and extract the archive of an image collection data set from
http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/oxbuild_images.tgz. The data set consists of around 5,000 images of different buildings in Oxford. If processing of the entire data set takes too long, you may use a subset, but make sure the subset contains images of different buildings. Then, convert the color images to grayscale images.
• Approach 1: Interpreting the grayscale images as 1D vectors of pixel intensities, run a clustering algorithm with an appropriate distance metric. What do the clusters represent?
• Approach 2: Transform the greyscale images into the space of SIFT descriptors. How many dimensions does this space have?Hint: You may use the Python bindings of OpenCV. Here is a tutorial: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/ py_feature2d/py_sift_intro/py_sift_intro.html.Apply a clustering approach to the SIFT descriptors and visualise the result in an MDS projection (in 2D). What do the clusters represent?p.t.o
Data Analytics 468
Visualization and Computer Graphics LabJacobs University
Assignment 9• Approach 3: Exchange the order of the two analysis steps in
Approach 2, i.e., first project the SIFT descriptors to a 2D space using an MDS approach and then cluster the 2D points using the same clustering approach as above. Again, visualise the result. What do the discovered clusters represent?
• How do the three approaches compare?