Sample Segmentation Results Use of Image...
Transcript of Sample Segmentation Results Use of Image...
Lecture14:ImageSegmentation
Bohyung HanCSE,POSTECH
CSED441:IntroductiontoComputerVision(2017F)
ImageSegmentation
• Whatisimagesegmentation?§ Processofpartitioninganimageintomultiplehomogeneoussegments§ Processofassigningalabeltoeverypixelinanimagesuchthatpixels
withthesamelabelsharecertainvisualcharacteristics§ Compactrepresentationforimagedataintermsofasetofcomponents
• Generalframework§ Tokens:whateverweneedtogroup(pixels,points,surfaceelements)§ Bottomupsegmentation:Tokensbelongtogetherbecausetheyare
locallycoherent§ Topdownsegmentation:Tokensbelongtogetherbecausetheylieon
thesameobject
2
SampleSegmentationResults
3http://www.cs.berkeley.edu/~fowlkes/BSE/
UseofImageSegmentation
• Primitivesforothertasks§ Grouptogethersimilar-lookingpixelsforefficiencyoffurther
processing§ Unsupervisedandbottom-upprocess
4
UseofImageSegmentation
• High-levelunderstandingofobjectorscene§ Separateanimageintocoherent“objects”or“region”§ Semanticsegmentation§ Objectdetection
5
UseofImageSegmentation
• Imagemanipulation(incomputergraphics)
6
Challenges
• Imagesegmentationisaninherentlydifficulttaskdueto§ Subjectivityofproblem§ Weakandinconsistentfeatures§ Requirementofhigh-levelinferences
7
Challenges
• Subjectivityofproblem§ Eachpersonhasadifferentconceptofsegmentinanimage.§ Evenapersonhasdifferentconceptsofsegmentdependingonthe
situation
8
Inputimage
Annotationsbyhuman
Challenges
• Weakandinconsistentfeatures§ Unclearobjectboundaries§ Imagenoisesandartifacts§ Backgroundclutter
9
Challenges
• Requirementofhigh-levelinferences§ Recoveringmissingboundaries§ Consideringscenecontext§ Handlingrepetitivepatterns
10
Evaluation
• Howtoevaluateimagesegmentationalgorithm?§ Weneedtoconstructdatasetsandevaluationprotocols.
• Datasets§ Berkeleysegmentationdataset§ COCO
• Evaluationmethods§ Precisionandrecallofregionboundaries§ Othermeasuresforclusteringalgorithmevaluation
• (Adjusted)RANDindex• Normalizedmutualinformation
1112
DatasetsandBenchmarks
BerkeleySegmentationDatasetandBenchmark
• BSD300§ Originaldataset§ 200trainingand100testingimages§ 12,000hand-labeledsegmentationsof1,000Coreldatasetimages
from30humansubjects.§ http://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/§ Relatedpaper
• D.MartinandC.FowlkesandD.TalandJ.Malik.ADatabaseofHumanSegmentedNaturalImagesanditsApplicationtoEvaluatingSegmentationAlgorithmsandMeasuringEcologicalStatistics.ICCV2001
13
BerkeleySegmentationDatasetandBenchmark
• Anexample§ Trainingimage#159029§ 6annotations
14
#1107:20segments #1109:13segments
#1113:12segments #1119:10segments
#1121:6segments #1129:7segments
BerkeleySegmentationDatasetandBenchmark
• BSD500§ Anextendedversion:200additionaltestingimages§ Newimagesweresegmentedbyfivedifferentsubjectsonaverage.§ Performanceisevaluatedbymeasuringprecision/recallondetected
boundariesandthreeadditionalregion-basedmetrics.§ http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/reso
urces.html§ Relatedpaper
• P.Arbelaez,M.Maire,C.FowlkesandJ.Malik.ContourDetectionandHierarchicalImageSegmentation. TPAMI2011.
§ Downloads• Pre-compiledMatlab package• Sourcecode(forLinux/Mac,32/64bits)• Pre-computedresultsontheBSD500andPASCALVOC2012
15
• WhatisCOCO?
§ CommonObjectsinCOntext§ Anewimagerecognitionandsegmentationdatasetthatwasreleasedin
Summer2014.
• Resources§ http://cocodataset.org/§ Paper
• Tsung-YiLin,MichaelMaire,SergeBelongie,JamesHays,PietroPerona,DevaRamanan,PiotrDollár,C.LawrenceZitnick:MicrosoftCOCO:CommonObjectsinContext. ECCV(5)2014:740-755
• arXiv link:http://arxiv.org/abs/1405.0312
COCO
16
COCO
• Characteristics§ Morethan70categories§ Objectsegmentation§ Recognitionincontext§ Multipleobjectsperimage§ Morethan300,000images§ Morethan2Millioninstances
17
COCO
• Tools§ InPythonandMatlab (notyetreadythough)§ Downloadandset-uppackage§ CommonAPIinPythonandMatlab withminordifferencesforreadingand
visualizingCOCO
• Annotation§ Instanceannotation
• Instances:storinganarrayofinstanceobjectthatcontainscategoryidandsegmentationofaninstance
• Categories:mappingofcategoryidtocategoryname§ Sentenceannotation
• Anarrayofsentenceannotationthatdescribesanimage• Eachimagehasatleastfivesentences(fewhasmorethanfive).
18
SomeExampleImages
19
COCO
20
http://cocodataset.org/
21
ImageSegmentationAlgorithms
ImageSegmentationAlgorithms
• Segmentationasgrouping§ k-meansclustering§ Mean-shift
• Segmentationasgraphpartitioning§ Spectralclustering§ Normalizedcut
• Segmentationbyothermethods§ Boundarydetection:Watershed§ Labeling:Graph-cut
22
WatershedAlgorithm
• Mainidea§ Imageasatopographicrelief:thegreylevelofapixelisinterpretedasits
altitudeintherelief.§ Thewatershedofareliefcorrespondtothelimitsoftheadjacent
catchmentbasinsofthedropsofwater.
23
WatershedSegmentation
24
Image Gradient Watershedboundaries
seg = watershed(bnd_im)
Implementation
• Procedure§ Chooselocalminimaasregionseeds§ Addneighborstopriorityqueue,sortedbyvalue§ Taketopprioritypixelfromqueue
• Ifalllabeledneighborshavethesamelabel,assignittothepixel• Addallnon-markedneighbors
§ Repeatstep3untilfinished
25
Thenon-labeledpixelsarethewatershedlines.
Results
26
Results
27
Characteristics
• Pros§ Fast(<1secfor512x512image)§ Amongthebestmethodsforhierarchicalsegmentation
• Cons§ Onlyasgoodasthesoftboundaries§ Noteasytogetavarietyofregionsformultiplesegmentations§ Notop-downinformation
• Usage§ Preferredalgorithmforhierarchicalsegmentation
28
Graph-Cut
• Averygeneraltoolthatcanbeappliedto§ Stereodepthreconstruction§ Texturesynthesis§ Videosynthesis§ Imagedenoising§ Foreground/backgroundsegmentation§ Matching
• Generalcharacteristics§ Formulatedusingagraph§ Usedforbinarylabelingproblem§ Solvedbyenergyminimization
29
[Boykov04]Y.Boykov,V.Kolmogorov:AnExperimentalComparisonofMin-Cut/Max-FlowAlgorithmsforEnergyMinimizationinVision.TPAMI26(9),2004
ImageRepresentationbyGraph-Cut
• Representationasagraph§ Anodeiscreatedforeachpixel.§ Anedgerepresentsthe
relationshipbetweenadjacentpixels.
§ Twospecialnodes:source(!)andsink(")
• Largesearchspace§ #ofnodesforlabeling:#§ thenumberofallpossible
binarysegmentations:2%
§ Thisisintractable.
30
t
s
[Boykov04]Y.Boykov,V.Kolmogorov:AnExperimentalComparisonofMin-Cut/Max-FlowAlgorithmsforEnergyMinimizationinVision.TPAMI26(9),2004
ImageRepresentationbyGraph-Cut
• Representationasagraph§ Anodeiscreatedforeachpixel.§ Anedgerepresentsthe
relationshipbetweenadjacentpixels.
§ Twospecialnodes:source(!)andsink(")
• Largesearchspace§ #ofnodesforlabeling:#§ thenumberofallpossible
binarysegmentations:2%
§ Thisisintractable.
31
B
F
[Boykov04]Y.Boykov,V.Kolmogorov:AnExperimentalComparisonofMin-Cut/Max-FlowAlgorithmsforEnergyMinimizationinVision.TPAMI26(9),2004
EnergyMinimizationProblem
• Energyfunction
• Dataterm§ Definedforeachnode§ Howsimilariseachlabelednodetotheforegroundorbackground?§ Probabilitythatthisfeaturebelongstoforeground(resp.background)
• Smoothnessterm(a.k.a.regularizationterm)§ Definedforeachedge§ Penaltyforhavingdifferentlabels§ Encouragingspatiallycoherentsegments§ Penaltyisdown-weightedifthetwopixelcolorsareverydifferent.
32
& ' = &) ' + +&, '
dataterm smoothnessterm
[Boykov04]Y.Boykov,V.Kolmogorov:AnExperimentalComparisonofMin-Cut/Max-FlowAlgorithmsforEnergyMinimizationinVision.TPAMI26(9),2004
TwoTerms
• Dataterm
§ Definedforeachnode
• Smoothnessterm
§ Definedforeachedge§ -./:visualsimilaritybetweentwonodes
(pixels),0 and1
33
&, ' = 2 -./ ' 0 − ' 14
.,/ ∈ℇ
&) ' =
289(;)4
=∈>
if' ; = B, FGenergy
28J(;)4
=∈>
if' ; = K, (BGenergy)
MN(O)
MP(O)
[Boykov04]Y.Boykov,V.Kolmogorov:AnExperimentalComparisonofMin-Cut/Max-FlowAlgorithmsforEnergyMinimizationinVision.TPAMI26(9),2004
Solution
• Approach§ Weshouldminimizeenergy
bycuttingthegraphintotwopartitions.
§ Solvedbymax-flow/min-cutalgorithm
• Max-flow/min-cutalgorithm§ Energyoptimizationisequivalent
tographmincut.§ Cut:removeedgestodisconnect
tworegions§ Minimum:minimizesumof
cutedgeweight
34
B
F
[Boykov04]Y.Boykov,V.Kolmogorov:AnExperimentalComparisonofMin-Cut/Max-FlowAlgorithmsforEnergyMinimizationinVision.TPAMI26(9),2004
35