Feature Extraction

Feature ExtractionLecture 9Image SegmentationCSC 59866CDFall 2004

Zhigang Zhu, NAC 8/203Ahttp://www-cs.engr.ccny.cuny.edu/~zhu/Capstone2004/Capstone_Sequence2004.html

Finding CirclesIf we dont know r, accumulator array is 3-dimensionalIf edge directions are known, computational complexity if reducedSuppose there is a known error limit on the edge direction (say +/- 10o) - how does this affect the search?Hough can be extended in many ways.see, for example:Ballard, D. H. Generalizing the Hough Transform to Detect Arbitrary Shapes, Pattern Recognition 13:111-122, 1981.Illingworth, J. and J. Kittler, Survey of the Hough Transform, Computer Vision, Graphics, and Image Processing, 44(1):87-116, 1988

Region SegmentationPartitioning of an image into different regions (connected components), each having uniform properties in some (set of) image feature(s):gray valuecolor value(s)textural qualitieslocal gradientmotionshape info..... etc.

Goal of SegmentationSegment a scene into image elements which may correspond to meaningful scene elements

Primary Goal of SegmentationSegmenting an image into image elements which may correspond to meaningful scene elements

What sort of image elements may correspond to meaningful scene elements?

Answer depends on type and complexity of images: Less constrained scenes must be segmented more conservatively.

Segmentation is not a well defined problem.

Region Segmentation Example

Color Image SegmentationGiven a grayscale image, how do we generate a region segmentation?In general, regions can be formed from the original image data or from 'derived' images:- color images from R, G, B- textural images- displacement images from motion analyses- 3D depth images?

Problems with SegmentationIn general, high level contextual knowledge is required for successful segmentation

Formal Definition of RegionsA region segmentation of an image, I, is a partition of the set of pixels of I into a set of K regions {Rj}, 1jK, such that:1. I =i=1K Rj2. Ri Rj = for i j3. p connected to p for all p, p in Rj4. For some predicate PP(Ri) is TRUE for I = 1,2,,KP(Ri Rj) is FALSE for Ri, Rj adjacent and ijEvery pixel belongs to a regionNo pixel belongs to more than one regionSpatial coherenceFeature coherence

Representing RegionsRegion Occupancy MapA set of region labels in registration with image I specifying the region association for each pixelImageOccupancy Map or Label Plane

Contour RepresentationImage

Chain CodeThe chain code representation of a boundary is found by 'walking' counterclockwise around the boundary and recording the direction to turn to stay on the border:Direction Code

Chain CodeCC = (i,j) {5 5 6 6 6 0 0 0 0 0 0 0 1 1 2 2 2 4 4 5 4 3 4 4}

Region SegmentationBasic ApproachesGeneralized thresholdingRegion growingRegion mergingRegion splittingSplit and MergeExtensions to split and mergeK-means clusteringWatershed algorithmsPartitioning methodsGrouping methods

Partitioning MethodsPartitioning: Given: a large data set. Goal: carve it up according to some notion of the association between items inside the set. We would like to decompose it into pieces that are good according to our model. For example, we might:decompose an image into regions which have coherent color and/or texture inside them;take a video sequence and decompose it into shots segments of video showing about the same stuff from about the same view point;decompose a video sequence into motion blobs, consisting of regions thatwhave coherent color, texture and motion.

Grouping MethodsGrouping: Given: a set of distinct data itemsGoal: we wish to collect together sets of data items that look similar according to our model. Effects like occlusion mean that image components that belong to the same object are often separated. Examples of grouping include:collecting together tokens that, taken together, form an interesting objectcollecting together tokens that seem to be moving together.Collecting together regions that have similar color and/or texture

Region SegmentationTwo Basic TechniquesRegion MergingSTART with many trivial regions (each pixel?)MERGE regions into larger regions based on some similarity criteriaCONTINUE merging until no further merging is possible

Region SplittingSTART with a single large region (entire image?)SPLIT into several smaller regions based on a 'splitting' criterionCONTINUE until no further splitting is possible (regions are 'uniform')

Region SegmentationBasic ApproachesGeneralized thresholdingRegion growingRegion mergingRegion splittingSplit and MergeExtensions to split and mergeK-means clusteringWatershed algorithms

Generalized ThresholdingRLP(i,j) = k if Tk-1

Threshold SelectionManual: try one and see if it looks goodHistogram analysis

StrategiesSearch for minimum between P1 and P2(search for minima between several peaks - multi-thresholding)Fit second order equationDifferentiate to find minimumSmooth image and/or histogram firstHistogram only points not on edges in histogramOr unweight contribution of pixels having high gradient magnitude

Threshold SelectionGaussian fittingIntensity distribution for objects assumed to be normally distributedMinimize false positives/false negativesMatch properties of binary and gray level imagee.g. momentsChoose threshold to maximize/minimize some function of the total edge gradientMaximize property of image or histograme.g. entropy

and probably lots of others

Optimal ThresholdingApproximate the histogram using a weighted sum of two or more probability densities with normal distributionThreshold set at the closest gray level corresponding to the minimum probability between the maxima of two or more normal distributions.Results in minimum error segmentation

Need to estimate parameters of the density functionsImplies optimizationProbability distributions of backgrounds and objectCorresponding histograms and optimal thresholds

Local or Adaptive ThresholdingIntensity values can vary as a function of lighting (for example).e.g.:

Very difficult to threshold using conventional methods.

Local or Adaptive ThresholdingEach pixel in image needs a unique thresholdTwo basic approachesChow and Kaneka Adaptive Thresholding (1972)Compute thresholds in a local window at sampled locations using histogram techniqueInterpolate local thresholds across imageLocal ThresholdingExamine statistically pixel values in local neighborhood around pixel to be thresholdingUse local statistic as thresholdPossibilities include mean, median, or mean of max and min value

Mean of 7x7 neighborhood

Improved ResultsResults can be improved if the threshold employed is not the mean, but (mean-C), where C is a constant.Using this statistic, all pixels which exist in a uniform neighborhood (e.g. along the margins) are set to background.

Local Histogram Thresholding

Region GrowingGoal: Segment the image my repeatedly starting from a particular pixel, called a "seed" point, growing it into a region by iteratively adding neighboring points while some similarity criterion is met.Is a set of algorithms to group pixels with similar attributes togetherGENERAL IDEAA pixel is added to a partially grown region if two conditions are satisfied:The pixel must be adjacent to the region, and,The pixel must be "similar enough" to pixels already in the region.The process continues until no further points can be added.Some other seed point, not already in any region, is chosen and the process is repeated until the entire image is segmented.

Region GrowingUse a region label plane RLPsame size as the original imagecontains a 1 if the corresponding image point is in the region, 0 otherwiseinitially contains a 1 corresponding to the seed pointImageRegion Label Plane

Similarity CriteriaWhen is a pixel "similar enough" to be added to the region?different choices:statistical population testssimple feature differencesGeneral IdeaApply iteratively to all pixels until no more pixels are added.IFRLP(i,j) = 0AND RLP(i-n, j-m) = 1 for some n,m = -1 to 1AND |I(i-n, j-m) - I(i,j)| < T (a predefined threshold)

THEN RMP(i,j) = 1

Example: Simple Region GrowingSeed PointT=16T=32T=64T=128

ProblemsWhats the similarity criterion?How do I select the threshold?Why does the algorithm leak?

Because of leaks, features in a region can vary arbitrarily.Fixed thresholds do not take into account characteristics of global spatial distribution.pixel sequence: p1,..., pkpj, pj+1neighbors| pj - pj+1| < T but | p1 - pk| > T p1pk

Region MergingDefine a distance function between regions. general form:

dij = D(Ri, Rj) > 0

Typically D is a distance measure in feature space and is a function only of the feature vectors associated with regions Ri and Rj:

dij = D(fi, fj) > 0

Merge regions with minimum distanceNeed to define some kind of termination criteria

Example Distance MeasuresDistance between color means

Distance between region centers

Weighted combination of the two

Where Nm is the number of pixels in Rmcij =NiNnewNjNnew|mi - mnew|2|mj - mnew|2+dij = a cij + b rij

Region MergingDefine a distance measure dij = D(f(Ri), f(Rj)) >0Algorithm: { While termination condition false Determine the minimum distance regions {i*, j*} = arg min dij

Merge the minimum distance regions Ri* Ri* U Rj* Remove merged region from region list L L-{Rj* Compute termination condition }

i,jRi, Rj L

Merging HierarchyAlgorithm generates a binary tree

Merging can be terminated when the minimum distance exceeds a threshold d i*,j* > T stop mergingDifferent thresholds produce different segmentations

Fishers CriterionMean and variance are good features to use for mergingEspecially if data is distributed normallye.g. is modeled by the Gaussian distributionPeak occurs at m (the mean) and has a value aa is (1 / 2ps2) and ensures that the area under the curve = 1

For modeling histogramsCompute area under the histogram and divide by (2p n)Parameter n is called the variances = sqrt (n) is the standard deviationn measures the flatness of the distribution

1/21/2

Histogram Model

Fishers CriterionDiscrimination between regions of different means and standard deviations can be done using Fischers criterion:

l is a thresholdIf two regions have good separation in the means and low variance, then we can separate them.

UniformityThus, the merging threshold for the mean intensity for two adjacent regions, l, should vary depending on the expected uniformity of the merged regionLess uniform regions will require a lower threshold to prevent under merging Uniformity a function of both intensity mean and variance of the regionCombine them (heuristic) asUniformity = 1 - n / mIn range 0-1 for case where the samples are all positiveThe threshold value l decreases with the decrease in uniformity asl = (1 - n / m ) l0User need supply only one threshold l0

22

Region SplittingRegion Splitting:(1) Start with a single large region (initially, entire image).(2) Recursively split it into smaller regions.(3) Continue splitting until each region is uniform (no further splits are possible).

A simple approach: Global ThresholdingDefine a global threshold TApply to every pixel in the image I: RLP(i,j) = 0 if I(i,j) < T RLP(i,j) = 1 if I(i,j) T

Region splitting: Multiple FeaturesOld algorithm: Ohlander-PriceBasically a sequential histogram-based multiple threshold algorithm.Features used RGB, HSI, and YIQ (9 images)General Idea:Start with the entire image as the initial region.Get the next region to be segmentedif none - the segmentation is complete.Compute the set of one-dimensional histograms.Select the "best" peak and find valleys on either sideif none, the region is "done" - put on finished list.Apply the threshold to the region and determine connected components.Add these regions to the list of regions to be further segmented and go to step 1.

Ohlander-Price Data

Ohlander-Price Result

Region Splitting

Peak Selection Criteria0Intensity peak in 0-60 or 200-255 ranges Best is closest to end1Both minima < 10% highest value max/min ratio > 4 Another peak exists with max/min ratio > 22Both minima < 25% of the peak value max/min ratio > 2 Another peak with max/min ratio > 23Max/min ratio > 2 Another peak with max/min ratio > 2 If maxima are within 10%, then both are acceptable (a bimodal distribution)4(Saturation only) Minima in 0-200 (lowest 20%)< 25% of the peak value max/min ratio > 2 Specified minima must separate peak with max/min ratio > 1.25Minima < 10% of highest value 10% of all points must be outside the peak6Minima < 70% of highest value max/min ratio > 1.7

Hybrid TechniqueSplit and Mergecombination: splits followed by merges (or vice-versa) split and merge decisions can be eitherlocal: a pixel and its immediate neighbors a region and its immediate neighborsglobal: on the basis of a large number of pixels scattered throughout the image

Split and MergeGeneral idea:Begin with an arbitrary region decomposition in a quadtree planeInitial decomposition = entire image?Split each region which violates a uniformity predicate into its 4 quadtree childrenMerge (recursively) all regions which jointly satisfy a uniformity criterionSupporting data structure: region adjacency graph

Split and Merge Example

Segmentation by k-means clusteringAssume we know the data has k clusters (k known)Each cluster is assumed to have a center ciThe jth element to be cluster is described by a feature vector xjFor scattered points, x would be coordinate(s)For an intensity image, x might be the intensity at a pixelDefine an objective function that measures how good the clustering result is.Develop an algorithm to maximize the objective function.

Objective FunctionAssume that element are close to the center of their clusterOne possible objective function is

Note that if allocation of points to clusters is known, we can compute the best center easilyFar too many associations of points to clusters to search this space for a minimumInstead, define an algorithm that alternatesAssume centers are known, allocate pointsAssume allocation is known, compute centers

k-means algorithmChoose k data points to act as cluster centersRandom selectionFirst k data pointsUntil the cluster centers are unchangedAllocate each data point to cluster whose center is nearestNow ensure that every cluster has at least one data point; possible techniques for doing this include:supplying empty clusters with a point chosen at random from points far from their cluster center.Replace the cluster centers with the mean of the elements in their clusters.endApply connected components algorithm to generate regions

k-means Clustering

ExampleAssume k=5

Each pixel is represented by the mean value of the cluster to which the pixel belongsA connected components algorithm must be applied to make these true region segementationsOriginal Imagek-means on intensityk-means on colorK=5

Results, k=11Sample clusters with k-means clustering based on color

Other Distance MeasuresSuppose we want to have compact regionsNew feature space: 5D(2 spatial coordinates, 3 color components)Points close in this space are close both in color and in actual proximityProblem with simple Euclidean distance: what if coordinates range from 0-1000 but colors only range from 0-255?Depending on how things are scaled, gives different weight to different kinds of dataWeighted Euclidean distance: adjust weights to emphasize different dimensionsS

Hybrid Edge-Region ApproachesSame idea as region mergingstart with oversegmented image.Use edge detection information as well.Merge not based on region statistics but on weak boundaries.Usually use fraction of edge pixels along shared boundary that are below some threshold.Often used as post-processing for edge-based segmentation.

Watershed AlgorithmsA gray scale image can be viewed as a topographic relief map where the intensity function of the image represents the altitude.A watershed region or catchment basin is defined as the region over which all points flow downhill to a common point.Points at which water would be equally likely to fall to more than one such minimum: watersheds or watershed linesWatersheds of gradient magnitude make useful region-based segmentation primitives.Boundaries of watersheds are one way to define ridgesthis is basically the same idea as finding a ridge of gradient magnitude.

Watersheds

Example on GrayScale Image

Immersion AlgorithmStart with all pixels with the lowest possible value (grad. mag.)these form the basis for initial watershedsFor each intensity level k:For each group of pixels of intensity kIf adjacent to exactly one existing region, add these pixels to that regionElse if adjacent to more than one existing regions, mark as boundaryElse start a new regionCan use a histogram-like structure to keep lists of all pixels with each intensity level k.

Watershed ProblemsOversegmentationwatershed from markersComputationally intensive computation intensive graph algorithm and appropriate data structures graph Graylevel might not be the optimal choice as the local similarity measure similarity measureother local features Statisticaledge enhanced imagedistance transformed imagee transformed image)

Over-segmentation ProblemOversegmentation due to noise and other local irregularities of the gradient

Solution: markers (of object locations)

TobogganingAgain work on the gradient magnitude image.Link each pixel to the smallest of its neighbors.If no smaller neighbors, become a are region seed.All pixels that flow downhill (smallest neighbor) to the same point form a single region.

Multispectral SegmentationGiven N multi-spectral images:1. Independently segment each multi-spectral feature image using the localized histogram algorithm.

2. Intersect the N segmentation to create a new, combined segmentation.

3. Merge using a region merging algorithm.

See J. Beveridge, J. Griffith, R. Kohler, A. Hanson, and E. Riseman, Segmenting Images Using Localized Histograms and Region Merging, IJCV 2(3), January 1989, pp. 311-347

Color ImageIntensityRedGreenBlue

SegmentationsIntensityRedGreenBlue

Three Color UnionNote: This result was obtained by taking the union of three color segmentations obtained with a more sensitive set of parameters than those shown on the previous page and consequently has more boundaries.

Final Segmentation: After MergingSegmentation obtained from over-segmented image on previous slide after applying a rule-based merging strategy using regions similarity, size, and connectivity.

House ResultsThree Color UnionIntensity Only

Region FeaturesMany features can be use to characterize a region and its propertiesSupports many tasks, including object recognition.Example features:area, height, and widthperimeter, bounding box, area of bounding boxcentroidorientationcompactnessmoments.....etc.

Region FeaturesBasicsPerimeter AreaOrientation

Rotation/translation/scale invariantCompactness = perimeter2/areaRectangularity = AreaRect/AreaObject (next slide)Euler number = #regions - #holesConvexity = AreaConvexHull/AreaObject

RectangularityW and L are a function of qRectangularity = WxL/A Choose q to get WxL minimumCalled the minimum bounding rectangleMinimized for rectangular objects

PerimeterRegion planeboundaryor

MomentsCentroid: center of mass

Higher order moments

Note: A = m00 ; r = m10 / m00 ; c = m01 / m00 ;

Central MomentsMoments around the center of mass

Note that A = u00 ; u01 = 0; u10 = 0

Higher order (2nd order and higher) moments are used heavily in oject recognition

OrientationLet q be the region orientation with respect to the r axis

Can be shown that

SummaryCovered only basic approachessimple region growingsplit, merge, and split and merge algorithmsgeneralized thresholds.Many approaches to region segmentationstatistical techniques: parameter estimation, mode estimation, clustering, decision theoretic methods...surface fits to the intensity surface: constant, plane, or bivariate polynomial,...relaxation: traditional, stochastic (simulated annealing)markov random fields methodscontext sensitive and knowledge-driven methodscombinations with edge detection techniquesoptimization and learning methodsmulti-resolutionLiterature is enormous

Next TopicNext: Camera Model

Watershed DescriptionConsider a rain drop falling onto a topological surface and watch as it follows a line of steepest descent toward some local surface minimum. The set of all points attracted to a particular minimum defines the catchment basin for that minimum. Adjacent catchment basins are separated by watershed lines or divide lines. Watershed lines generally occur along the peaks of ridge-like structures (`mountain ranges'). It can easily be seen that raindrops falling on opposite sides of a divide line flow into different catchment basins. In mathematical morphology, greyscale images are often interpreted as topological surfaces where the intensity value of a pixel represents the elevation of the surface at the pixel's position. Due to this interpretation the concept of watersheds can be applied to greyscale images. However, the task of extracting watershed information from digital images is not a simple one. Most algorithms to date have suffered from being either very computationally expensive or highly inaccurate. The idea of rain falling on a topographic surface does not lend itself well to an efficient algorithm. In recent years a fast, accurate method for watershed analysis that is based on what they refer to as a flooding scheme or immersion simulation has been proposed.

Watershed Immersion Simulation MethodIn this approach, the surface is thought of as being slowly lowered into a pool of water at a fixed speed. The water initially seeps through the surface at the global minimum and continues to penetrate the local minima as the surface is lowered further. As water from one catchment basin comes into contact with water from an adjacent basin, a dam is constructed which actually prevents contact of the water from the different basins. The immersion is continued until the surface is completely covered. At this point the dams that have been constructed are labelled as watershed points.

Feature Extraction

Documents

Transcript of Feature Extraction