Research Article Patch-Based Segmentation with Spatial ... · Research Article Patch-Based...

Research ArticlePatch-Based Segmentation with Spatial ConsistencyApplication to MS Lesions in Brain MRI

Roey Mechrez1 Jacob Goldberger2 and Hayit Greenspan1

1Biomedical Engineering Department Tel-Aviv University 69978 Tel Aviv Israel2Engineering Faculty Bar-Ilan University 52900 Ramat Gan Israel

Correspondence should be addressed to Roey Mechrez roimehrezgmailcom

Received 29 October 2015 Revised 24 December 2015 Accepted 31 December 2015

Academic Editor Yantian Zhang

Copyright copy 2016 Roey Mechrez et al This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

This paper presents an automatic lesion segmentationmethod based on similarities betweenmultichannel patches A patch databaseis built using training images for which the label maps are known For each patch in the testing image 119896 similar patches areretrieved from the database The matching labels for these 119896 patches are then combined to produce an initial segmentation mapfor the test case Finally an iterative patch-based label refinement process based on the initial segmentation map is performed toensure the spatial consistency of the detected lesions The method was evaluated in experiments on multiple sclerosis (MS) lesionsegmentation in magnetic resonance images (MRI) of the brain An evaluation was done for each image in the MICCAI 2008MS lesion segmentation challenge Results are shown to compete with the state of the art in the challenge We conclude that theproposed algorithm for segmentation of lesions provides a promising new approach for local segmentation and global detection inmedical images

1 Introduction

Patch-based methods have been shown to be an effectiveapproach for labeling brain structures (and other bodystructures) as shown for example in [1 2] In general theseapproaches label each voxel of a target image by comparingthe image patch centered on the voxel with patches froman atlas library and assigning the most probable labelaccording to the closest matches Often a localized searchwindow centered around the target voxel is used Variouspatch-based label fusion procedures have been proposed andwere shown to produce accurate and robust segmentationUsing affine registration comparable results to works thatuse nonrigid registration have been reported Patch-basedtechniques have recently demonstrated high performance invarious computer vision tasks including texture synthesis[3] inpainting [4] and super resolution [5] Nonlocal meansdenoising [6] has helped advance the field and has led to thedevelopment of various patch-based segmentation tools formedical imaging applications methods [1 2 7 8]

The focus of segmentation tasks in existing works is onregions of interest which are substantial in size andor are

anatomically localized (eg brain tumors knee brain tissuesthe hippocampus cortical parcellation and brain structures)In these cases the anatomical context provides labelingsupport and a good approximate alignment of the image toan atlas (expert priors) is needed and is a key componentFor example in the hippocampus or the knee the algorithmis designed to differentiate between the surrounding tissuesand the convex target region thus the inner voxels are easyto label Similarly in the case of brain tissues anatomicalconstraints facilitate labeling most of the voxels and theerrors are mainly in the border voxels

In the current work we focus on developing a patch-based segmentation for small nonlocalized regions Theregions of interest examined here such as lesions do notentail anatomical constraints and hence the location ofthese lesions is subject to inconsistency in terms of bothneighboring tissues and the absolute position These regionsare not contained within other regions as is the case forbrain tumor or the knee We aim to overcome the lack ofspatial constraints and anatomical context by a novel spatialconsistency step as discussed in Section 26 We deal withMS lesion detection and segmentation in MR images of

Hindawi Publishing CorporationInternational Journal of Biomedical ImagingVolume 2016 Article ID 7952541 13 pageshttpdxdoiorg10115520167952541

2 International Journal of Biomedical Imaging

T1 T2 FLAIR Ground truth

Figure 1 Three axial slices of MS lesions on MRI FLAIR T1w T2w and the associated ground truth lesion map

the brain which involve a number of challenges in particularMS lesions are very small they can appear anywhere in thebrain image and they are relatively similar to other braintissues Examples of MS lesions are presented in Figures 1 2and 3

MS is the most common nontraumatic neurologicaldisease in young adults It is an inflammatory demyelinatingdisease that is primarily associated with axonal loss andformation of lesions in the central nervous system whichare characterized by demyelination axonal injury and axonalconduction block These classically described white matter(WM) lesions are visible in conventional magnetic resonanceimaging (MRI) appearing as hyperintense in T2-weighted(T2w) images and as hypointense in T1-weighted (T1w)images (see Figure 1) Fluid attenuated inversion recovery(FLAIR) images have been shown to be the most sensitiveto WM lesions but can also present other hyperintensityartifacts [9]

MRI is currently used to diagnose of MS assess diseaseprogression and evaluate the efficiency of drug therapy[10] The most common quantitative parameter is the lesionload of the disease expressed in terms of the number andvolume of brain lesions The MRI measured lesion load ishighly correlated with clinical findings [11 12] An accuratesegmentation at the voxels level is a necessary step to calculatethe lesion load or any other measure

Manual segmentation ofWMlesions is a time-consumingprocess Furthermore the 3D data of an MRI scan requiresmultislice segmentation which makes the manual segmen-tation a laborious task [13] Fully automated algorithms forMS lesion detection and segmentation have been the focusof research for many years Algorithms to date have failed torespond to the complexity of the task and new approacheshave recently been introduced [14ndash17]

Pattern recognition and machine learning techniqueshave been widely investigated to identify the patterns of MSlesions by making use of the neuroimaging data [18ndash21]These segmentation methods can be classified as supervisedor unsupervised as a function of themathematical algorithmsthey implement

Unsupervised methods [23 24] are designed to solvethe segmentation task without needing labeled training dataThese methods attempt to formalize the definition of alesion and differentiate lesions from other tissues on thebasis of anatomical knowledge For example in [21] the MS

Figure 2 ROI (blue) on a FLAIR image

Figure 3 Candidate lesion regionsmask (blue) and the lesions (red)on a FLAIR image

lesions were identified as outlier from a constrained Gaussianmixture model (CGMM) Likewise in [16] the lesions areidentified as outliers using dictionary learning and sparsecoding Some works [15 25] use an anatomical atlas as theprior anatomical knowledge but since these works requiredaccurate 3D registration step this step may be misdirected inthe presence of lesions

International Journal of Biomedical Imaging 3

Supervised methods are a useful way to recognize apattern in a new test sample based on information learnedfrom the training samples Typically the source of the traininglabels is manual segmentation Typically studies use one ofthemany available supervised learningmethods for example119896-nearest neighbor (119896-NN) [19] support vector machine(SVM) [26 27] cellular neural networks (CNN) [28] ora Bayesian framework [29] For example one of the out-standing works [14] used a discriminative random decisionforest to provide a voxel-wise probabilistic classification of thevolume

Since the detection of white matter lesions (WML) is achallenging task extraction of discriminative features playsan important role in lesion segmentation Voxel-related fea-tures such as the probabilities of different tissues (graymatter(GM) white matter (WM) and cerebrospinal fluid (CSF))and the multichannel intensity (ie T1 T2 and FLAIR)play an important role in neuroimaging studies Cruciallyalthough some authors use only one sequence (T1 or FLAIR)[15 30] the lesion should be confirmed in other sequencesto avoid false positives (FPs) [13] Nevertheless voxel-relatedfeatures are not sufficient for good segmentation since othertissues have a similar appearance and therefore spatial infor-mation is necessary [13] Specifically the spatial informationis usually included in the local mean using the neighborhoodinformation as in many imaging applications [1 2 5 19]The local spatial information reduces the impact of noise andimproves coherence of the results To use the spatial infor-mation mathematical techniques as Markov random fields(MRF) graph cut and kernel features are employed [31]

In this study we propose a patch-based method fordetection and segmentation of MS lesions in brain MRI byutilizing multimodal spatial information The segmentationis first obtained based on intensity patch similarity and thenfurther iteratively refined with the spatial label informationThe proposed framework segments the MS image withoutrequiring registration or an atlas This makes the frameworkrobust to registration errors which are likely to occur if thereis a high degree of anatomical variability

The main contributions of this paper can be summarizedas follows

(1) Adding a spatial consistency refinement step to thepatch-based approach using a novel label propagationbased metric

(2) Creating a patch database for the MS task In MSthe lesion anatomical positions differ significantlybetween subjects We therefore cannot use the sameanatomical volumes of interest as in classic patch-based segmentation

(3) Providing a framework for segmentation of MSlesions that does not require registration with noneed for an atlas

The rest of this paper is organized as follows Theproposed method is described in the next section Then inSection 3 extensive experiments and comparisons with othersegmentation methods on the MICCAI08 grand challengedatasets are presented to demonstrate the segmentation

accuracy of the proposed method Finally in Section 4 wediscuss parameters influences on the proposed frameworkand possible future directions

2 Patch-Based Segmentation andLabel Refinement

21 System Overview Our method is based on labeling thetest image voxels (as lesion or nonlesion) by finding similarpatches in a database ofmanually labeled imagesThe trainingstep involves constructing a patch database using expert-marked lesion regions which provide voxel-level labelingGiven a test image we first detect candidate lesion regions forfurther analysis For all candidate voxels we find the 119896-nearestneighbor patches from the patch database Labels of theselected patches are used to determine the current patch labelusing a voting scheme thus generating the test image labelmap Finally to enforce spatial consistency we iterativelyincorporate a label decision from the neighboring voxelsobtained during the previous iteration by adding them to the119896-NN metric Before we applied our method preprocessingsteps subject selection step and candidate region detectionare done In the following subsections we provide a detaileddescription of the algorithm

22 Preprocessing Before starting the segmentation process(in the training phase as well as for a new test image) a fewpreprocessing steps are taken During these steps variabilitycaused by image formation is minimized by inhomogene-ity correction and intersubject intensity normalization Thebrain is extracted to avoid nonrelevant tissues (ie the skulland scalp)

221 Subsampling We subsample the images so that they allare the same size 169 times 129 times 130 and the same isotropicresolution 1 times 1 times 1mm3 The subsampling is designed toreduce the computational time and the background as muchas possible From this step on voxels outside the brain maskare ignored

222 Normalization The most crucial step in the prepro-cessing is the normalization of the intensities in the imageThis is mandatory to ensure that all brain tissues in differentsubjects have the same contrast and luminance We use thenormalization procedure of Nyul et al [32] to make patchmatching between different subjects possible Some authorsuse normalization to zero mean and unit standard deviationIn the datasets we experimented with we found that usingthe intensity as is provided better results Note that for theanalysis we need very good intensity normalizations acrossbrains In order to support the normalization procedure weadd an additional step of subject selection (Section 233) toselect the most similar brains in the training set

223 Inhomogeneity Correction To ensure that each tissuetype has the same intensity within a single image the well-known N4ITK intensity nonuniformity correction in [33] isused on all three channels


224 Brain Extraction and Tissue Classification A brainmask was created using the BrainSuite13 toolbox [34] asdescribed in [35] Classification to WM GM and CSFwas also applied using [36] In this classification processvoxels in an intensity-normalized image are classified usinga maximum a posteriori (MAP) classifier This classifiercombines the partial volume tissue measurement model witha Gibbs prior that models the spatial properties of the brain

23 Creation of a Labeled Patch Database Recent patch-based segmentation works are based on the nonlocal means(NLM) idea [6 37] where similar patches are searched ina cubic region around the location under study A patch-to-patch similarity in specific anatomical regions is assumedto hold true and the segmentation tasks are considered tohave spatial consistency (eg the hippocampus in differentbrains appears in the same region) MS lesions howevercan be found in different brain regions and local regionpatch matching cannot be implemented To overcome thisshortcoming we suggest a labeled patch database creationstep

231 Patch Sources In the training step we extract regions-of-interest (ROIs) around allmanuallymarked lesions AROIis defined as the bounding box of the lesion (see Figure 2)Note that we expand the borders of the bounding box by afew voxels (three) in order to capture the surrounding tissueas well ROIs therefore represent the lesion and nonlesionaround the lesions in a roughly balanced way To betterrepresent the entire brain we add randomly selected patchesfrom all brain regions This way we represent all brain tissuesbut give special attention to complex lesion regions and theldquonormal appearing white matterrdquo From each ROI and foreach input channel we then extract 3 times 3 times 3 patches as in[16 38]

232 Augmented Patch Patches are concatenated acrossthe input channels (in our case T1 T2 and FLAIR) toform augmented patches 119875 isin 119877

81 (referred to hereafter asconcatenated patches) This set of patches forms the trainingpatch database D We further examine the use of differentweights for the three modalities using multiplication by threeconstants see Section 253 for details

233 Subject Selection and Patch Selection In many seg-mentations tasks multiple training subjects (or atlases) areused and a process of label fusion is needed to combine thedifferent segmentations A subject selection step is neededin order to ensure fusion of information from most similarsources For example in [1 39 40] the nearest subjects areselected for each region that requires labeling by comparingthe Euclidean distance between corresponding regions Incontrast in our method we do not use the concept ofsimilar anatomical regions for patch retrieval rather weuse the labeled patch database originating from multibrainregions We define a global subject selection step to facilitatea selection of similar brains to increase the accuracy ofsimilarity comparisons between patches Given a set of 119872training images 119879

119899| 119899 = 1 119872 we select a subset of 119873

training images 119879119899| 119899 = 1 119873 whose Kullback-Leibler

(KL) divergence [41] from the test image is minimal Let 119877and 119876 be the multimodal histograms (ie three histogramsone for each modality concatenated) of the given test imageand a train image respectively The KL divergence is definedto be

119863KL (119877 119876) = sum

119894

119877119894sdot log

119877119894

119876119894

(1)

The subject selection leads to a labeled patch subset 119863119868sub 119863

denoted by 119863119868= 119875

119863isin 119879119899

| 119899 = 1 119873 For thetraining images the label maps 119871

119899| 119899 = 1 119873 are

known Following the subject selection step we select a subsetof patches1198631015840

119868 out of119863

119868 by subsampling in fixed increments

such that the number of lesion and nonlesion patches is equaland the total number of patches reaches a fixed number Thesubsampling is done using the index numbers of the extractedpatches Thus the subsampling is in a sense in space recallthat patches from the same ROI have close indexes From ourempirical experimentation we found an appropriate numberto be between 100 and 150 thousand patches

24 Detection of Candidate Lesion Regions Next we focus onautomatically detecting candidate lesion regions in a giventest image The detection is based on two clinical rules

(i) The lesions appear as hyperintense in FLAIR imagesthus they can be roughly identified using threshold-ing It is well known that the FLAIR input channelis a good source for analyzing MS lesions [13]Using a global threshold (TH) on a FLAIR imageprovides high sensitivity in the detection Since sucha threshold results in poor specificity we can use theFLAIR input to provide an initial rough delineationof candidate lesion regions (see [26] for a detaileddescription)

(ii) The lesions are characterized by demyelination thusthey are part of the WM tissue Lesions tend to befound in the WM or on the border between WM andGM We classify the brain tissues using a maximuma posteriori probability classifier (MAP) [34] (seepreprocessing Section 224) and extracted the WMregion (dilated) as a second mask The dilation stepis done in order to guarantee that lesions surroundedby WM WM boundaries and peripheral lesions willbe in the mask The structure element used is a ldquoballrdquowith 10 pixels radius

The final set of lesion candidate regions consists of all thevoxels above the FLAIR TH intensity which are in the WMdilated region as follows

LesionMask (119909) =

1 FLAIR (119909) gt TH cap 119909 isin WM

0 otherwise(2)


where 119909 is a voxel in the test image and TH is defined similarto [26]

TH = 120583GM + 120582 sdot 120590GM (3)

120583GM and 120590GM are themean and standard deviation of the GMtissue in the FLAIR test image respectively The parameter 120582is an empirical parameter selected experimentally as 05 Theextracted LesionMask which includes the set of candidatelesion regions contains around 20 of the brain voxels butmore than 95 of the lesionsThis stage significantly reducesthe number of voxels that need to be further analyzed forlesions It was found in [26] to decrease false positives (FPs)almost without any degradation of the true positives (TPs)that is less than 5 loss The parameter 120582 was chosen asthe maximum value that maintains 95 of the lesions inthe candidate lesion regions We optimized this parameterbetween 0 and 1 with increments of 01 Note that for 120582 = 0

(ie TH = 120583GM) the results were almost identical

25 Generation of an Initial Label Map After the proceduredescribed above the voxels marked by the mask are furtheranalyzed as lesion or nonlesion using a patch-based decisionmethod This patch-based segmentation strategy is based onthe NLM estimator [6] that has been tested on a variety oftasks [1 2 26] Likewise in our work given an augmentedpatch from a test image (combining several MR channels inthe patch definition) similar patches are found in the labeleddatabase described above

Given input image 119868 and the selected patch database1198631015840

119868 the similar patch labels are combined to yield a lesion

segmentation map as follows

251 Patch Matching For each voxel 119909 in the test imagewe extract a corresponding patch 119875

119909centered at 119909 We then

retrieve 119896 similar patches 119875119863119894(119909) 119894 = 1 119896 from the

database1198631015840119868 Each selected patch is weighted as follows

119908(119909 119875119863

119894) = exp(minus

119889 (119875119909 119875119863

119894(119909))

1205902) (4)

where 119889(sdot sdot) is the square Euclidean distance and 1205902 is the

maximum distance measure obtained for all the voxels 119909 andfor all the patches in the database 119863

1015840

119868 Equations (6) and

(7) below are alternative metric definitions for modalitiesbalancing and spatial refinement respectively

252 Vote Aggregation To obtain a label for a test voxel 119909 avote aggregation method is defined using all labeled patchesextracted via the 119896-nearest neighbors (NN) patch matchingprocedureThe voxel 119909 appears in 119897 = 27 test patches centeredon 119909 and its neighboring voxels For each test patch 119875

119909we

extract 119896-nearest neighbor labeled patches 119875119863119894(119909) | 119894 =

1 119896 from the dataset 1198631015840119868 Each retrieved similar patch

along with its labels 119897119863119894(119909) | 119894 = 1 119896 and its weight

contributes a vote Thus for a given voxel 119909 we have 119897 sdot 119896

votes that are averaged The aggregation of votes results in aprobabilistic lesion decision as follows

119901 (lesion | 119909) =sum119910isin119875119909

sum119896

119894=1119908(119910 119875

119863

119894(119910)) sdot 119897

119863

119894(119910 119909)

sum119910isin119875119909

sum119896

119894=1119908 (119910 119875119863

119894(119910))

(5)

where 119910 goes over the 27 neighboring voxels of 119909 and 119897119863119894(119910 119909)

is the label that the training patch 119875119863

119894(119910) (which is similar

to the patch centered at 119910) assigns to 119909 The final lesion label119871(119909) for each voxel 119909 is defined as 1 if 119901(lesion | 119909) gt 05 and0 otherwiseThis content-based segmentation process resultsin an initial label map

253 Modalities Balancing We examined the use of differentweights for the three modalities119872 = T1T2 FLAIR usingmultiplication by three constants 119862

119872= 119862T1 119862T2 119862FLAIR

This can be achieved by reformulation of the metric for thepatch-based voting aggregation algorithm described aboveThe augmented patch (119875

119909) is a concatenation of the T1 patch

(119875T1119909) the T2 patch (119875T2

119909) and the FLAIR patch (119875FLAIR

119909) thus

the balanced metric between a test patch 119875119909and a training

patch 119875119863is defined as follows

119889119868(119875119909 119875119863) = sum

119872

119862119872sdot 119889 (119875119872

119909 119875119872

119863) (6)

Thebalancedmetric has the opportunity to treat each channelof the MRI acquisition differently and hence give moreweight to the more incriminating channel This metric alsoenables examination of the patch source that is using onetwo or three modalities For example by using 119862

119872= 119862T1 =

1 119862T2 = 0 119862FLAIR = 1 T2 is ignored

26 Patch-Based Label Refinement The above proceduretreats each voxel independently and ignores spatial consis-tency constraints For example it may be the case that a singlevoxel is labeled as nonlesion even though all its neighboringvoxels are labeled as lesions (see eg the segmentationresults shown in Figure 4(c)) In the following we focus onincorporating spatial consistency into the lesion map resultAs noted in [13] a segmentation method that relies solelyon voxel intensity is unlikely to produce sufficient results Assuch many attempts have been made to incorporate spatialor anatomical information within the segmentation process[8 42] In other patch-based segmentation algorithms [1 2]a search volume is defined around the voxel under studyHowever this cannot be applied in a MS lesion task giventhe lack of clear anatomical location in addition to thepotential weakness of reliance on good affine registration inthe presence of lesions

In Wang et al (2013) [40] an alternative framework thatcombines the use of 119896-NN ball trees and a spatial weight labelfusion scheme to search for patches in large regional areaswassuggested This Spatially Aware Patch-Based Segmentation(SAPS) is designed to overcome the problem of limitedsearch windows and combine spatial information by usingthe anatomical location of the patch Similarly in Wanget al (2014) [42] an iterative refinement step using a sparse


(a) (b) (c) (d) (e) (f)

Figure 4 Two segmentation examples (top UNC06 bottom UNC02) (a) FLAIR image (b) first iteration (cndashe) iterations 2 3 and 5 (f)ground truth Proposed method in blue reference segmentation in red

representation was implemented to correct the anatomicalerrors introduced in the segmentation Our goal is to use thelabel information to enforce a consistency of the decisionTheterm spatial consistency is adapted from Freeman et al [5]where the authors proposed a Markov random field solutionfor spatial consistencyThey introduced a one-pass algorithmthat used the neighbors relation in a similar way to the onereflected in the MRF pixel grid

In our method we enforce spatial consistency using anewmetric for the patch-based voting aggregation algorithmdescribed aboveThenewmetric uses the current labelmap inaddition to the intensity information Aweighting parameter120572 is used to determine the relative importance of the intensitysimilarity and the spatial consistency constraint This newmetric is used in the patch matching step and in the voteweighing step The metric between a test patch 119875

119909and a

training patch 119875119863is defined as follows

119889 (119875119909 119875119863) = 119889119868(119875119909 119875119863) + 120572 sdot 119889

119871(119875119909 119875119863) (7)

where 119889119868(sdot sdot) is the intensity-based metric described above

and 119889119871(119875119909 119875119863) is the square Euclidean distance between the

current labels of the patch 119875119909and the labels of training patch

119875119863 A larger weight parameter 120572 tends to favor the current

labeling decision of the algorithm We use the followingdefinition for the weight

120572119905= 1205720lowast (119905 minus 1) (8)

with constant 1205720ensuring that the intensity metric and label

metric are on the same scale and 119905 is the iteration numberIn the first iteration 120572 = 0 thus we start with the patch-based method as described in Section 23 As 120572 is increasedfromone iteration to the next it givesmore weight to labels ofthe nearest-neighbor training patches in making the current

label decision A large 120572 ensures that the labels of the nearest-neighbor training patches coincide with the current labeldecision and therefore the algorithm converges We havefound that in practice there is no need for more than 4-5iterations until convergence

Our method imposes a global consistency constraintbetween the labels of neighboring voxels in an efficient wayFor each patch we find themost similar patch based on inten-sity while also taking into account spatial compatibility withthe neighboring voxelsThe initial patch-based segmentationresults are sensitive to small hyperintense regions that arecaused by noise and inhomogeneities These regions containpatches that are similar to lesion patches The majorityof these voxels are correctly labeled as nonlesions duringthe refinement stage Algorithm 1 summarizes the proposedsegmentation algorithm

27 Implementation Details The proposed method wasimplemented in MATLAB 80 using CMEX code and theexperiments were conducted using an Intel Core i7-3770processor at 34GHz with 8GB RAM The preprocessingsteps were carried out with using BrainSuite13 [34] 3DSlicer[43] and MATLAB The average overall runtime for a singleimage is less than 4 minutes

3 Results

This section reports on the segmentation results and com-pares the patch-based approach to other top-rankedmethods[14ndash16]

31 Datasets We evaluated our framework using clinicalpublic data provided by theMS lesion segmentation challengewhich was introduced at the MICCAI MS lesion segmenta-tion workshop 2008 [22] This is the largest dataset publicly


Input An MR brain imageDetermine a subset of training brains that are most similar to the given testfor 119905 = 1 119879 do

(i) Set 120572119905= 1205720lowast (119905 minus 1)

(ii) For each patch 119875119909centered at pixel 119909 in the test brain image find 119896 similar patches 119875119863

119894(119909) in the training set using

the distance measure defined in (7)(iii) Compute the weight of each selected patched using (4) and compute the weighted average lesion label decision using (5)

end forOutput A voxel-labeled lesion map of the brain image

Algorithm 1 The lesion detection and segmentation algorithm

available andmost recent works in the field use this dataset asit provides a benchmark for algorithm comparisons The MSlesion grand challenge offers two datasets a labeled dataset(originally designed for training) and an unlabeled dataset(used for testing) We will hereon term them the public setand private set respectivelyThe public set (train set) contains20 cases 10 from theUniversity ofNorthCarolina (UNC) and10 from the Childrenrsquos Hospital Boston (CHB) datasets Theprivate dataset (testing set) contains 15 cases from CHB and10 from UNC For the private dataset two lesion markingsare available segmented by UNC and CHB rater The datasetshows high variability in intensity contrast image noise andbias field The dataset contains highly heterogeneous casesand can thus be considered as a realistic test case In all casesthe data resolution is 05mm3 isotropic in three differentmodalities (T1w T2w and FLAIR) All data were rigidly reg-istered to a common reference frame and resliced to isotropicvoxel spacing using b-spline based interpolation by thechallenge organizers Full documentation is available in [22]

32 Evaluation

321 MICCAI2008 Public Dataset (Training Set) Our pro-cedure was evaluated in a leave-one-out framework for eachmedical center (CHB andUNC)We computed true positives(TPs) true negatives (TNs) false positives (FPs) and falsenegatives (FNs) and used the following validation measuresTrue Positive Rate (TPR) defined as TP(TP + FN) PositivePredictive Value (PPV) defined as TP(TP + FP) and DiceSimilarity Coefficient (DSC) defined as (2TP)(FP + FN +2TP) The measures were computed using the expert labelmap provided in the dataset The scores appear in Table 1along with a comparison to three state-of-the-art works usingthe same dataset The three selected works are Souplet et al[15] the winner of the MICCAI MS lesion grand challenge[22] Weiss et al [16] the latest work published using thisdatabase and Geremia et al [14] one of the outstandingsupervisedmethods published Note that DSCmeasures werenot provided by the authors in the first two methods

The proposed method achieved a mean TPR of 40 amean PPV of 29 and a mean DSC of 31 These resultsare comparable with the other state-of-the-art algorithmsusing the same data Two successful segmentation examplesare shown in Figure 4 The input image is presented inFigure 4(a) and the final segmentation map is shown in

Figure 4(e) The similarity to the reference segmentationshown in Figure 4(f) is evident There was a substantialdecrease in FPs as part of the algorithm iterative refinementprocedure in both cases The reduction in FPs is moresignificant in the second iteration In addition observe anincreased number of border voxels being classified as lesionsin iterations 3 to 5 Thus overall there was a significantdecrease in the quantity of the FPs and an increase in all themeasured indices The result analysis in this section is basedon the public dataset

322 MICCAI2008 Private Dataset (Testing Set) Quantita-tive evaluation was also carried out on the private datasetusing a set of known metrics defined in [22] In orderto allow quantitative comparison between methods and tohuman export the organizers define average score Eachmetric is related to the result that could be expected ifan independent human observer would perform the seg-mentation manually Thus 100 points mean a perfect result(the best value that could be obtained for a metric) and apredefined amount (see [22] for details) of 90 for a scorethat is typical for an independent human observer Thealgorithm results on the MICCAI2008 private dataset asprovided by the MS-lesion challenge organizers are availablein the challenge website (httpwwwiaunceduMSseg)Theproposed method achieved an average score of 72 wherethe average scores on UNC and CHB databases were 68and 74 respectively Generally a significant difference isreflected when comparing the method results on the UNC(68) and CHB (74) datasets It is hard to understand thespecific reason for that difference in the absence of the GTWe believe that this difference could be explained by changesin the MRI acquisition parameters which the normalizationsteps failed to bridge

33 Computational Time Due to the high dimension ofthe search space finding the absolute best match wouldbe computationally prohibitive Instead we used FLANN(Fast Library for Approximate Nearest Neighbors) [44 45]FLANN is a library for performing fast approximate nearestneighbor searches in high dimensional spaces It containsa collection of algorithms found to work best for nearestneighbor searches and a system for automatically choosingthe best algorithm and optimum parameters depending onthe dataset Using FLANN each NN iteration takes less than


Table 1 Comparison of current method to three state-of-the-art methods on clinical brain image data from the public part of MICCAI 08challenge [22] TPR PPV and DSC are in where 100 is perfect segmentation

Patient Souplet et al [15] Geremia et al [14] Weiss et al [16] Current methodTPR PPV TPR PPV TPR PPV DSC TPR PPV DSC

UNC01 1 1 2 1 33 29 31 0 0 0UNC02 37 39 48 36 54 51 53 67 34 45UNC03 12 16 24 35 64 27 38 57 25 35UNC04 38 54 54 38 40 51 45 65 17 27UNC05 38 8 56 19 25 10 16 36 9 14UNC06 57 9 15 8 13 55 20 37 69 48UNC07 27 18 76 16 44 23 30 51 48 49UNC08 27 20 52 32 43 13 20 24 11 15UNC09 16 43 67 36 69 6 11 29 35 32UNC10 22 28 53 34 43 23 30 44 45 45Average 28 24 45 26 43 29 29 41 29 31CHB01 22 41 49 64 60 58 59 40 33 36CHB02 18 29 44 63 27 45 34 41 11 17CHB03 17 21 22 57 24 56 34 47 19 27CHB04 12 55 31 78 27 66 38 37 7 12CHB05 22 42 40 52 29 33 31 58 25 35CHB06 13 46 32 52 10 36 16 43 38 41CHB07 13 39 40 54 14 48 22 34 50 41CHB08 13 55 46 65 21 73 32 48 52 50CHB09 3 18 23 28 5 22 8 31 23 26CHB10 5 18 23 39 15 12 13 13 28 18Average 14 36 35 55 23 45 29 39 29 30

1 minute (asymp40 sec) and results in a total time of less than4 minutes Note that when we compared the approximatedresult to an accurate result achieved by brute force NNimplementation the results were similar without a significantdisparity Computational times in other methods were notspecified in a way that enables comparison In Souplet et al[15] an example of execution time on one case was given andthe computational time reported (without the preprocessing)was 34min Weiss et al [16] reported 5min which is similarto the patch-based method Geremia et al [14] did not reportthe computational time for a single test case but rather asignificant training time of 8 hours to train the random forestclassifier they used All frameworks use similar preprocessingsteps thus the comparison of preprocessing time is irrelevant

34 Visualization of the Lesion Data in Intensity Space Thedriving force behind the patch-based approach is the intensitysimilarity between the patches We compared the meanintensity of the lesions detected by the algorithm to thosemarked by the expert Figure 5 shows the mean intensity ofthe lesion across the multimodal images Generally the fitbetween the lesion dots is high suggesting that the intensityrange of our method is compatible with the ground truthEach sample in Figure 5 represents one lesion some of thelesions in the ground truth (the full blue dots) appear tobe significant outliers for example the two lesions withthe lowest FLAIR and T1 intensities and the highest T2-intensity (FLAIR(T1) graph upper left corner T2(T1) graph

bottom left corner) The sources of these lesions are patientsUNC01 and CHB05 Furthermore this figure emphasizes theimportance of the subject selection step for outlier handling

35 Influence of the Patch Database Definition One of thegreatest challenges in the system is defining the patchdatabase It is clear that if the database characterizationcontains too few patches or represents the brain tissue and thelesions poorly this can lead to misclassification FP errors areinstances of retrieved patches which are similar to lesions butoriginate from healthy tissue and therefore their labels leadto misclassification We also tried to use only the patches inthe ROI marked as example in Figure 2 (Section 23) and theresults were slightly worse Furthermore the WM segmenta-tionmaskmay contain classification errors or adjacent tissueswhich may cause the need to decide whether patches fromother tissues are lesion or not

In classical machine learning systems the training dataand the testing data are usually identical in terms of theirproperties That is to classify all the brain voxels it makessense to train all of them However creating a huge databasecontaining millions of patches would cause the classifier tofavor the healthy class Therefore we chose to build thedatabase with patches extracted from the lesion regions Theparameter that determines the amount of patches that areproximal to the lesions is the size of the bounding box aroundthe lesion As the bounding box is larger it will containmore healthy patches and the database size will increase


20 40 60 80 100

T1-intensity

50

100

150

200

FLA

IR-in

tens

ity

Ground truthPatch-based

(a)

20 40 60 80 100

T1-intensity


50

100

150

200

T2-in

tens

ity(b)

Figure 5 Gray level feature space for the lesions in all images (a) FLAIR and T1 feature space (b) T2 and T1 feature space The blue dotsrepresent the mean intensity of the lesions as marked by the expert The red dots represent the mean intensity of the lesions as detected bythe algorithm

0

20

40

60

DSC

()

0 8642

Size of ROI expansion (voxels)

(a)

50 100 150 200

Database size (k patches)

0

20

40

60

DSC

()

(b)

Figure 6 (a) DSC () as function of the number of voxels expansion per ROI Recall that we expand the borders of a lesion bounding boxby a few voxels in order to capture the surrounding tissue as well The 119909-axis represents the number of voxels in this expansion (b) DSC ()as function of the number of patches in the database which was obtained from the ROI expansion size

Figure 6 shows the influence of this parameter on the results(DSC) The results in Figure 6(a) support our selection of abounding box expansion of 3 voxels From Figure 6(b) we seethat the best DSC result was obtained for a patch numberbetween 100000 and 150000 patches

36 Influence of Number of Nearest Patches (119896) and SubjectSelection (119873) Figure 7 depicts performance change as afunction of the number of nearest patches 119896 and for differentnumber of selected subject 119873 It shows that TPR increasesand PPV decreases as 119896 increases which causes the DSCto remain almost unchanged We chose an optimal workingpoint where the PPV and TPR are similar that is 119896 = 30

Figure 7 shows that the optimal 119896 value is similar fordifferent 119873 values A slight improvement was observed for119873 = 5 When the number of training images is very largeimages that are not very similar (large KL divergence in thesubject selection stage) are also used and thus the patch

matching is biased Conversely for119873 = 1 the patch selectionis too small and thus for some of the patches similar patchesare not found

Generally we found that the True Positive Rate increasesas 119896 increases this could be explained by the fact that in (4)we calculate the weights for each patch vote For dissimilarpatches this weight is practically zero thus for 119896 gt 100

the results are with small change In practice we selected119896 = 30 because when using more patches the algorithmtends to classify the patches as lesions This can be explainedby the balanced labeled database we used As explained inSection 231 the selection of the patches represented thelesion and nonlesion in a roughly balanced way thus when119896 gt 30 the algorithm finds patches that are quite similar tothe target patch with high probability to be lesion Moreoverwe found that the patches retrieved after the 30th patch aredifferent one from the other mainly in the noise level thiscauses the classification probability to be as in the database


0 10 20 30 40 50 60 70 80 90 100

Number of nearest patches selected k

0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 6070 80 90 100


20

25

30

35

DSC

()

30

40

50

TPR

()

20

30

40

PPV

()

N = 3

N = 5

N = 9

Figure 7 DSC () for a range of number of nearest patches 119896 withdifferent number of selected subjects119873

roughly balanced and thus decreases the overall performance(while the TPR is increasing)

37 Influence of the Spatial Refinement Figure 8 shows theeffect of the iterative refinement step An increase is seen inthe DSC from one iteration to the next This increase wasobserved consistently for all subjects A mean DSC value of21 was achieved before the patch-based label refinementprocess a meanDSC of 25was obtained after two iterationsand in the last iterationwe achieved ameanDSCof 31whichshows a substantial improvement

The weighting for the spatial information is determinedby 1205720 Specifically we selected a value of 20 which results

in parameter 120572 to range between 0 and 80 (from the initialiteration to the fifth) The value of 120572 in the fifth iterationis similar to the average intensity of a lesion (across all MRchannels)

38 Influence of Modality Combinations In this work aug-mented patches are used that is we concatenate the patchesacross the different MRI modalities Using the augmentedpatches in this way ensures that all three modalities areapproximately equally treated Using the Euclidean distancewhich is sensitive to the intensity of the patch may infact result in the fact that the brightest modality (FLAIR)has the greatest effect Still the effect of the differencesin intensity is relatively random making it legitimate toassume that the overall effect on patch matching is negligibleThe segmentation performance was evaluated for variouscombinations of modalities Figure 9 shows DSC TPR andPPV values for three single-modality three double-modalityand one multimodal patch sources When using a singlemodality T2 resulted in the best performance Combining

DSC TPR PPV0

20

40

60

()

Initial IT1st IT2nd IT

3rd IT4th IT

Figure 8 TPRPPVDSC () with respect to iteration (IT) number(using 119896 = 30 and119873 = 5) The first iteration is the initialization stepand the remainder are label refinement steps

DSC TPR PPV

T1+ T2 + FLAIRT1+ T2T1+ FLAIRT2 + FLAIR

T1T2FLAIR

0

10

20

30

40

50

60

70

()

Figure 9 Segmentation performance with different modality com-binations (using 119896 = 30 and 119873 = 5) The results reported inSection 32 and Table 1 are with all three modalities

across several modalities reduced the FPs and increasedthe TPs On the entire dataset the combination T1w +T2w + FLAIR performed statistically better than all othercombinations (aside from T1w + T2w in the TPR measure)

Figure 10 shows DSC TPR and PPV values for fivedifferent modality weights 119862

119872= 119862T1 119862T2 119862FLAIR In this

experiment we did not perform an accurate optimization ofthe three weight parameters Our goal was to identify theinfluence of the parameter set and its possible future use inpatch-based segmentation approaches Slightly better resultswere achieved using 119862T2 = 2 and 119862FLAIR = 2 This result iscompatible with the fact that themain characteristic of aWMlesion is that its intensity is brighter than its surroundings onT2w and FLAIR

4 Discussion and Conclusion

In this paper we proposed a novel patch-based method fordetection and segmentation of MS lesions in brain MRI byutilizing multimodal spatial information The segmentationis first obtained based on intensity patch similarity and thenfurther iteratively refined with the spatial label informationThe proposed framework segments the MS image without


DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()

Figure 10 Segmentation performance with different modalityweights (using 119896 = 30 and 119873 = 5) The numbers in the legendfrom left to right are themodalityweights119862T1 119862T2 and119862FLAIR in themetric respectively (The results reported in Section 32 and Table 1are with 119862T1 = 119862T2 = 119862FLAIR = 1)

requiring registration or an atlas This makes the frameworkrobust to registration errors which are likely to occur if thereis a high degree of anatomical variability

The relationships between patches can be modeled asa Markov Network (or random field) [46] We let the testbrain image patches be observation nodes and selected the119896 closest examples to each input patch as the differentstates of the hidden nodes that we sought to estimate In[5] a good approximate solution for this relationship wasobtained by running a belief propagation These authorsalso introduced a new one-pass algorithm that computesthe patch compatibilities for previously selected neighboringpatches In our algorithmwe use this idea in order to take therelationships between the labels of neighboring patches intoaccount

A few main characteristics of the presented approachinclude the following (1) The patch-based segmentationmethod is nonparametric We do not assume any intensitymodels and rely only on intensity information In otherparametric models the amount of lesion load needs to bedetermined a priori (2) We integrate the spatial informationthrough the patch-based metric by incorporating the labelterm The spatial refinement step adds both the neighbor-hood relations and introduces spatial consistency (3) Theproposed framework is supervised and thus aims to representthe lesion and nonlesion tissues using labeled data In otherunsupervised methods finding outliers in the model is theguiding principle

In this paper we compared the proposed method againstthree othermethods In Souplet et al [15] the authors showedthat a global threshold on the FLAIR MR sequence inferredusing EM brain tissue classification suffices to detect mostMS lesions The final segmentation is then constrained toappear in the white matter by applying morphological oper-ations The method reported here achieves better accuracythan the Souplet et al method In Geremia et al [14] theauthors proposed a discriminative random decision forestframework to provide a voxel-wise probabilistic classificationof the volumeThemethod usesmultichannelMR intensities

knowledge about tissue classes and long-range spatial con-text to discriminate lesions from background A symmetryfeature is introduced to account for the fact that some MSlesions tend to develop in an asymmetric way

When compared to this method our findings are mixedThis may be due to the fact that in the current work we areusing intensity only features This representation is sensitiveto normalization In their work Geremia et al indicatedthat sophisticated features such as context-rich and symmet-ric features reduce the strong normalization preprocessingrequirements We therefore believe that combining some ofthe context-rich and symmetry features may enable us toreach better results In Weiss et al [16] an unsupervisedapproach addressing the problem with dictionary learningand sparse coding was used This method is the most recentpublished work addressing the MS lesion segmentation taskusing the MICCAI08 datasets

It is interesting to note the low results for subject UNC01for both our algorithm and Souplet et al [15] and Geremiaet al [14] algorithm This specific case contains two lesionswhere the first is relatively small and the second is very similarto theCSF tissue and touches it (Figure 5 shows the significantintensity outlier) In addition when comparing the averageintensity of the second lesion in all three channels it appearsas a significant outlier

One of themajor drawbacks ofMRI is the lack of standardand quantifiable interpretation of image intensities Unlikeother modalities such as X-ray computerized tomographyMR images of the same patient taken on the same scannerat different times may appear different from each other dueto a variety of scanner-dependent variations and thereforethe absolute intensity values do not have a fixed mean-ing [32] The key step in overcoming this drawback is anormalization step during preprocessing (Section 22) thisenables intensity-based patch comparisons We use state-of-the-art intensity normalization schemes Still we have foundhigh sensitivity to any intensity differences For this reasonwe added a subject selection stage Only the more similarsubjects have a patch-comparison conducted Out of the 20subject MR images in the datasets a few of them (eg CHB2and CHB5) differ markedly The impact was considerable onthese subjectrsquos segmentation results and in fact went fromtotal segmentation failure to reasonable results In the patchselection stage (Section 23) a fixed number of patches wereselected (150K) In our experimentations we have found thatthis fixed number of patches is satisfactory for any trainingdata size as long as the patch database formed containssufficient diversity The fixed number of patches selectedprovides invariance to data size and any scalability concerns

In future research we plan to explore various additionalrepresentations for the patches such as using the Histogramof Oriented Gradients (HoG [47]) One possible explanationfor the lower results we achieved on the private dataset is thedifficulty of the subject selection step to find sufficiently goodmatches for the test cases In other words the advantage ofcomparing patches from a variety of brains and tissues causedmisclassification of lesions Evidence for that is the relativehigh score on the volume difference as compared with theFPR and TPR scores


This framework could be coupled with dictionary learn-ing as well as additional pre- and postprocessing to improveperformance In future work we aim to improve the resultsusing better database characterization and advanced metriclearning Evaluation of our algorithm on larger datasets fromvarying input sources will allow us to test the robustnessof the algorithm to noise and other variance We are nowworking on a system for screening of highly noisy patchesor patches which are extracted from fundamentally differentbrains We believe that better learning of the database willlead to a considerable improvement in the results

To conclude we presented a novel framework for patch-based segmentation that integrates intensity informationwith a patch-based label refinementThemethoddoes not callfor nonlinear alignment of the training images onto the spaceof the testing image Many other works use this step which isamajor source of error and is computationally expensiveThedatabase concept as the novel refinement step can be easilyapplied in variety of patch-based segmentation frameworksAlthough the patch-based algorithm is based on a 119896-NNsearch a good approximation for the search was found toresult in less than 5min The total segmentation time (notincluding preprocessing) makes themethod one of the fastestproposed for the MS lesion task The method proposed is ageneral one and as we believe can be generalized to othersmall lesion tasks such as liver metastasis and chest X-raypathologies

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

References

[1] P Coupe J V Manjon V Fonov J Pruessner M Robles andD L Collins ldquoPatch-based segmentation using expert priorsapplication to hippocampus and ventricle segmentationrdquo Neu-roImage vol 54 no 2 pp 940ndash954 2011

[2] F Rousseau P A Habas and C Studholme ldquoA supervisedpatch-based approach for human brain labelingrdquo IEEE Trans-actions on Medical Imaging vol 30 no 10 pp 1852ndash1862 2011

[3] A A Efros and W T Freeman ldquoImage quilting for texturesynthesis and transferrdquo in Proceedings of the 28th AnnualConference on Computer Graphics and Interactive Techniques(SIGGRAPH rsquo01) pp 341ndash346 Los Angeles Calif USA August2001

[4] A Criminisi P Perez and K Toyama ldquoRegion filling andobject removal by exemplar-based image inpaintingrdquo IEEETransactions on Image Processing vol 13 no 9 pp 1200ndash12122004

[5] W T Freeman T R Jones and E C Pasztor ldquoExample-basedsuper-resolutionrdquo IEEE Computer Graphics and Applicationsvol 22 no 2 pp 56ndash65 2002

[6] A Buades B Coll and J-M Morel ldquoA non-local algorithm forimage denoisingrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) vol 2 pp 60ndash65 IEEE San Diego Calif USA June 2005

[7] N Cordier B Menze H Delingette and N Ayache ldquoPatch-based segmentation of brain tissuesrdquo in Proceedings of the MIC-CAI Challenge onMultimodal Brain Tumor Image Segmentation(BRATS rsquo13) pp 6ndash17 Nagoya Japan September 2013

[8] Z Wang C Donoghue and D Rueckert ldquoPatch-based seg-mentation without registration application to knee MRIrdquo inMachine Learning in Medical Imaging 4th International Work-shop MLMI 2013 Held in Conjunction with MICCAI 2013Nagoya Japan September 22 2013 Proceedings vol 8184 ofLecture Notes in Computer Science pp 98ndash105 Springer BaselSwitzerland 2013

[9] L S Aıt-Ali S Prima P Hellier B Carsin G Edan andC Barillot ldquoSTREM a robust multidimensional parametricmethod to segment MS lesions in MRIrdquo in Medical ImageComputing and Computer-Assisted Intervention MICCAI vol8 part 1 of Lecture Notes in Computer Science pp 409ndash416Springer Berlin Germany 2005

[10] D H Miller R I Grossman S C Reingold and H FMcFarland ldquoThe role of magnetic resonance techniques inunderstanding andmanagingmultiple sclerosisrdquo Brain vol 121no 1 pp 3ndash24 1998

[11] M Filippi M Rovaris and M A Rocca ldquoImaging primaryprogressive multiple sclerosis the contribution of structuralmetabolic and functional MRI techniquesrdquo Multiple Sclerosisvol 10 supplement 3 pp S36ndashS45 2004

[12] R Zivadinov R De Masi D Nasuelli et al ldquoMRI techniquesand cognitive impairment in the early phase of relapsing-remitting multiple sclerosisrdquo Neuroradiology vol 43 no 4 pp272ndash278 2001

[13] D Garcıa-Lorenzo S Francis S Narayanan D L Arnoldand D L Collins ldquoReview of automatic segmentation methodsof multiple sclerosis white matter lesions on conventionalmagnetic resonance imagingrdquo Medical Image Analysis vol 17no 1 pp 1ndash18 2013

[14] E Geremia O Clatz B H Menze E Konukoglu A CriminisiandNAyache ldquoSpatial decision forests forMS lesion segmenta-tion inmulti-channelmagnetic resonance imagesrdquoNeuroImagevol 57 no 2 pp 378ndash390 2011

[15] J C Souplet C Lebrun N Ayache and G MalandainldquoAn automatic segmentation of T2-FLAIR multiple sclerosislesionsrdquo inProceedings of theMultiple Sclerosis Lesion Segmenta-tion Challenge Workshop (MICCAI rsquo08) vol 613 of Proceedingsin The MIDAS Journal New York NY USA September 2008

[16] N Weiss D Rueckert and A Rao ldquoMultiple sclerosislesion segmentation using dictionary learning and sparsecodingrdquo in Medical Image Computing and Computer-AssistedInterventionmdashMICCAI 2013 vol 8149 of Lecture Notes inComputer Science pp 735ndash742 Springer Berlin Germany 2013

[17] C Elliott D L Arnold D L Collins and T Arbel ldquoTemporallyconsistent probabilistic detection of new multiple sclerosislesions in brain MRIrdquo IEEE Transactions on Medical Imagingvol 32 no 8 pp 1490ndash1503 2013

[18] K Van Leemput F Maes D Vandermeulen A Colchesterand P Suetens ldquoAutomated segmentation of multiple sclerosislesions by model outlier detectionrdquo IEEE Transactions onMedical Imaging vol 20 no 8 pp 677ndash688 2001

[19] P Anbeek K L Vincken M J P van Osch R H C Bisschopsand J van der Grond ldquoProbabilistic segmentation of whitematter lesions in MR imagingrdquo NeuroImage vol 21 no 3 pp1037ndash1044 2004


[20] Y Wu S K Warfield I L Tan et al ldquoAutomated segmentationof multiple sclerosis lesion subtypes with multichannel MRIrdquoNeuroImage vol 32 no 3 pp 1205ndash1215 2006

[21] O Freifeld H Greenspan and J Goldberger ldquoMultiple sclerosislesion detection using constrained GMM and curve evolutionrdquoInternational Journal of Biomedical Imaging vol 2009 ArticleID 715124 13 pages 2009

[22] M Styner J Lee B Chin et al ldquo3D segmentation in the clinica grand challenge II MS lesion segmentationrdquoMIDAS Journalpp 1ndash5 2008

[23] N Shiee P-L Bazin A Ozturk D S Reich P A Calabresiand D L Pham ldquoA topology-preserving approach to thesegmentation of brain images with multiple sclerosis lesionsrdquoNeuroImage vol 49 no 2 pp 1524ndash1535 2010

[24] A Shahar andHGreenspan ldquoA probabilistic framework for thedetection and tracking in time of multiple sclerosis lesionsrdquo inProceedings of the IEEE International Symposium on BiomedicalImaging Nano to Macro vol 1 pp 440ndash443 IEEE ArlingtonVa USA April 2004

[25] P-Y Bondiau G Malandain S Chanalet et al ldquoAtlas-basedautomatic segmentation of MR images validation study onthe brainstem in radiotherapy contextrdquo International Journal ofRadiation Oncology Biology Physics vol 61 no 1 pp 289ndash2982005

[26] J-B Fiot L D Cohen P Raniga and J Fripp ldquoEfficient brainlesion segmentation using multi-modality tissue-based featureselection and support vector machinesrdquo International Journalfor Numerical Methods in Biomedical Engineering vol 29 no 9pp 905ndash915 2013

[27] R J Ferrari X Wei Y Zhang J N Scott and J R MitchellldquoSegmentation ofmultiple sclerosis lesions using support vectormachinesrdquo in Medical Imaging Image Processing vol 5032 ofProceedings of SPIE pp 16ndash26 International Society for Opticsand Photonics San Diego Calif USA February 2003

[28] A Cerasa E Bilotta A Augimeri et al ldquoA cellular neural net-work methodology for the automated segmentation of multiplesclerosis lesionsrdquo Journal of Neuroscience Methods vol 203 no1 pp 193ndash199 2012

[29] X Tomas-Fernandez and SKWarfield ldquoAnew classifier featurespace for an improved multiple sclerosis lesion segmentationrdquoin Proceedings of the 8th IEEE International Symposium onBiomedical Imaging From Nano to Macro (ISBI rsquo11) pp 1492ndash1495 Chicago Ill USA April 2011

[30] P Schmidt C Gaser M Arsic et al ldquoAn automated toolfor detection of FLAIR-hyperintense white-matter lesions inmultiple sclerosisrdquo NeuroImage vol 59 no 4 pp 3774ndash37832012

[31] Z Karimaghaloo M Shah S J Francis D L Arnold D LCollins and T Arbel ldquoAutomatic detection of gadolinium-enhancing multiple sclerosis lesions in brain MRI using con-ditional random fieldsrdquo IEEE Transactions on Medical Imagingvol 31 no 6 pp 1181ndash1194 2012

[32] L G Nyul J K Udupa and X Zhang ldquoNew variants of amethod of MRI scale standardizationrdquo IEEE Transactions onMedical Imaging vol 19 no 2 pp 143ndash150 2000

[33] N J Tustison B B Avants P A Cook et al ldquoN4ITK improvedN3 bias correctionrdquo IEEE Transactions on Medical Imaging vol29 no 6 pp 1310ndash1320 2010

[34] D W Shattuck and R M Leahy ldquoBrainsuite an automatedcortical surface identification toolrdquoMedical Image Analysis vol6 no 2 pp 129ndash142 2002

[35] B Dogdas D W Shattuck and R M Leahy ldquoSegmentationof skull and scalp in 3-D human MRI using mathematicalmorphologyrdquo Human Brain Mapping vol 26 no 4 pp 273ndash285 2005

[36] D W Shattuck S R Sandor-Leahy K A Schaper D ARottenberg and RM Leahy ldquoMagnetic resonance image tissueclassification using a partial volume modelrdquo NeuroImage vol13 no 5 pp 856ndash876 2001

[37] P Coupe P Yger S Prima P Hellier C Kervrann and CBarillot ldquoAn optimized blockwise nonlocal means denoisingfilter for 3-Dmagnetic resonance imagesrdquo IEEE Transactions onMedical Imaging vol 27 no 4 pp 425ndash441 2008

[38] S Roy Q He A Carass et al ldquoExample based lesion seg-mentationrdquo inMedical Imaging 2014 Image Processing 90341YProceedings of SPIE International Society for Optics andPhotonics March 2014

[39] P Aljabar R A Heckemann A Hammers J V Hajnal and DRueckert ldquoMulti-atlas based segmentation of brain images atlasselection and its effect on accuracyrdquo NeuroImage vol 46 no 3pp 726ndash738 2009

[40] Z Wang R Wolz T Tong and D Rueckert ldquoSpatially awarepatch-based segmentation (SAPS) an alternative patch-basedsegmentation frameworkrdquo in Medical Computer Vision Recog-nition Techniques and Applications inMedical Imaging vol 7766of Lecture Notes in Computer Science pp 93ndash103 SpringerBerlin Germany 2013

[41] S Kullback and R A Leibler ldquoOn information and sufficiencyrdquoAnnals of Mathematical Statistics vol 22 pp 79ndash86 1951

[42] L Wang F Shi Y Gao et al ldquoIntegration of sparse multi-modality representation and anatomical constraint for isoin-tense infant brain MR image segmentationrdquo NeuroImage vol89 pp 152ndash164 2014

[43] A Fedorov R Beichel J Kalpathy-Cramer et al ldquo3D sliceras an image computing platform for the quantitative imagingnetworkrdquoMagnetic Resonance Imaging vol 30 no 9 pp 1323ndash1341 2012

[44] M Muja and D G Lowe ldquoFast matching of binary featuresrdquo inProceedings of the 9th Conference on Computer and Robot Vision(CRV rsquo12) pp 404ndash410 IEEE Toronto Canada May 2012

[45] M Muja and D G Lowe ldquoFast approximate nearest neighborswith automatic algorithm configurationrdquo in Proceedings of theInternational Conference on Computer VisionTheory and Appli-cations (VISAPP rsquo09) pp 331ndash340 Springer Lisbon PortugalFebruary 2009

[46] S Geman and D Geman ldquoStochastic relaxation Gibbs distri-butions and the Bayesian restoration of imagesrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 6 no 6pp 721ndash741 1984

[47] N Dalal and B Triggs ldquoHistograms of oriented gradients forhuman detectionrdquo in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPRrsquo05) vol 1 pp 886ndash893 IEEE SanDiego Calif USA June 2005

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014


Active and Passive Electronic Components

Control Scienceand Engineering

Journal of



RotatingMachinery


Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design



Shock and Vibration


Civil EngineeringAdvances in

Acoustics and VibrationAdvances in



Electrical and Computer Engineering

Journal of

Advances inOptoElectronics


Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of


Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014


Chemical EngineeringInternational Journal of Antennas and

Propagation




Navigation and Observation



DistributedSensor Networks



T1 T2 FLAIR Ground truth

Figure 1 Three axial slices of MS lesions on MRI FLAIR T1w T2w and the associated ground truth lesion map

the brain which involve a number of challenges in particularMS lesions are very small they can appear anywhere in thebrain image and they are relatively similar to other braintissues Examples of MS lesions are presented in Figures 1 2and 3

MS is the most common nontraumatic neurologicaldisease in young adults It is an inflammatory demyelinatingdisease that is primarily associated with axonal loss andformation of lesions in the central nervous system whichare characterized by demyelination axonal injury and axonalconduction block These classically described white matter(WM) lesions are visible in conventional magnetic resonanceimaging (MRI) appearing as hyperintense in T2-weighted(T2w) images and as hypointense in T1-weighted (T1w)images (see Figure 1) Fluid attenuated inversion recovery(FLAIR) images have been shown to be the most sensitiveto WM lesions but can also present other hyperintensityartifacts [9]

MRI is currently used to diagnose of MS assess diseaseprogression and evaluate the efficiency of drug therapy[10] The most common quantitative parameter is the lesionload of the disease expressed in terms of the number andvolume of brain lesions The MRI measured lesion load ishighly correlated with clinical findings [11 12] An accuratesegmentation at the voxels level is a necessary step to calculatethe lesion load or any other measure

Manual segmentation ofWMlesions is a time-consumingprocess Furthermore the 3D data of an MRI scan requiresmultislice segmentation which makes the manual segmen-tation a laborious task [13] Fully automated algorithms forMS lesion detection and segmentation have been the focusof research for many years Algorithms to date have failed torespond to the complexity of the task and new approacheshave recently been introduced [14ndash17]

Pattern recognition and machine learning techniqueshave been widely investigated to identify the patterns of MSlesions by making use of the neuroimaging data [18ndash21]These segmentation methods can be classified as supervisedor unsupervised as a function of themathematical algorithmsthey implement

Unsupervised methods [23 24] are designed to solvethe segmentation task without needing labeled training dataThese methods attempt to formalize the definition of alesion and differentiate lesions from other tissues on thebasis of anatomical knowledge For example in [21] the MS

Figure 2 ROI (blue) on a FLAIR image

Figure 3 Candidate lesion regionsmask (blue) and the lesions (red)on a FLAIR image

lesions were identified as outlier from a constrained Gaussianmixture model (CGMM) Likewise in [16] the lesions areidentified as outliers using dictionary learning and sparsecoding Some works [15 25] use an anatomical atlas as theprior anatomical knowledge but since these works requiredaccurate 3D registration step this step may be misdirected inthe presence of lesions



























119863KL (119877 119876) = sum

119894

119877119894sdot log

119877119894

119876119894

(1)


denoted by 119863119868= 119875

119863isin 119879119899


119899| 119899 = 1 119873 are


119868 out of119863









0 otherwise(2)



TH = 120583GM + 120582 sdot 120590GM (3)











119908(119909 119875119863

119894) = exp(minus

119889 (119875119909 119875119863

119894(119909))

1205902) (4)



1015840




119909we






119901 (lesion | 119909) =sum119910isin119875119909

sum119896

119894=1119908(119910 119875

119863

119894(119910)) sdot 119897

119863

119894(119910 119909)

sum119910isin119875119909

sum119896

119894=1119908 (119910 119875119863

119894(119910))

(5)






119872= 119862T1 119862T2 119862FLAIR



(119875T1119909) the T2 patch (119875T2


119909) thus



119889119868(119875119909 119875119863) = sum

119872

119862119872sdot 119889 (119875119872

119909 119875119872

119863) (6)


119872= 119862T1 =





(a) (b) (c) (d) (e) (f)




119909and a


119889 (119875119909 119875119863) = 119889119868(119875119909 119875119863) + 120572 sdot 119889

119871(119875119909 119875119863) (7)






120572119905= 1205720lowast (119905 minus 1) (8)






3 Results












32 Evaluation
















20 40 60 80 100

T1-intensity

50

100

150

200

FLA

IR-in

tens

ity


(a)

20 40 60 80 100

T1-intensity


50

100

150

200

T2-in

tens

ity(b)


0

20

40

60

DSC

()

0 8642


(a)

50 100 150 200


0

20

40

60

DSC

()

(b)









0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 6070 80 90 100


20

25

30

35

DSC

()

30

40

50

TPR

()

20

30

40

PPV

()

N = 3

N = 5

N = 9







DSC TPR PPV0

20

40

60

()


3rd IT4th IT


DSC TPR PPV


T1T2FLAIR

0

10

20

30

40

50

60

70

()




119872= 119862T1 119862T2 119862FLAIR In this





DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()
















References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation



































119863KL (119877 119876) = sum

119894

119877119894sdot log

119877119894

119876119894

(1)


denoted by 119863119868= 119875

119863isin 119879119899


119899| 119899 = 1 119873 are


119868 out of119863









0 otherwise(2)



TH = 120583GM + 120582 sdot 120590GM (3)











119908(119909 119875119863

119894) = exp(minus

119889 (119875119909 119875119863

119894(119909))

1205902) (4)



1015840




119909we






119901 (lesion | 119909) =sum119910isin119875119909

sum119896

119894=1119908(119910 119875

119863

119894(119910)) sdot 119897

119863

119894(119910 119909)

sum119910isin119875119909

sum119896

119894=1119908 (119910 119875119863

119894(119910))

(5)






119872= 119862T1 119862T2 119862FLAIR



(119875T1119909) the T2 patch (119875T2


119909) thus



119889119868(119875119909 119875119863) = sum

119872

119862119872sdot 119889 (119875119872

119909 119875119872

119863) (6)


119872= 119862T1 =





(a) (b) (c) (d) (e) (f)




119909and a


119889 (119875119909 119875119863) = 119889119868(119875119909 119875119863) + 120572 sdot 119889

119871(119875119909 119875119863) (7)






120572119905= 1205720lowast (119905 minus 1) (8)






3 Results












32 Evaluation
















20 40 60 80 100

T1-intensity

50

100

150

200

FLA

IR-in

tens

ity


(a)

20 40 60 80 100

T1-intensity


50

100

150

200

T2-in

tens

ity(b)


0

20

40

60

DSC

()

0 8642


(a)

50 100 150 200


0

20

40

60

DSC

()

(b)









0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 6070 80 90 100


20

25

30

35

DSC

()

30

40

50

TPR

()

20

30

40

PPV

()

N = 3

N = 5

N = 9







DSC TPR PPV0

20

40

60

()


3rd IT4th IT


DSC TPR PPV


T1T2FLAIR

0

10

20

30

40

50

60

70

()




119872= 119862T1 119862T2 119862FLAIR In this





DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()
















References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation











TH = 120583GM + 120582 sdot 120590GM (3)











119908(119909 119875119863

119894) = exp(minus

119889 (119875119909 119875119863

119894(119909))

1205902) (4)



1015840




119909we






119901 (lesion | 119909) =sum119910isin119875119909

sum119896

119894=1119908(119910 119875

119863

119894(119910)) sdot 119897

119863

119894(119910 119909)

sum119910isin119875119909

sum119896

119894=1119908 (119910 119875119863

119894(119910))

(5)






119872= 119862T1 119862T2 119862FLAIR



(119875T1119909) the T2 patch (119875T2


119909) thus



119889119868(119875119909 119875119863) = sum

119872

119862119872sdot 119889 (119875119872

119909 119875119872

119863) (6)


119872= 119862T1 =





(a) (b) (c) (d) (e) (f)




119909and a


119889 (119875119909 119875119863) = 119889119868(119875119909 119875119863) + 120572 sdot 119889

119871(119875119909 119875119863) (7)






120572119905= 1205720lowast (119905 minus 1) (8)






3 Results












32 Evaluation
















20 40 60 80 100

T1-intensity

50

100

150

200

FLA

IR-in

tens

ity


(a)

20 40 60 80 100

T1-intensity


50

100

150

200

T2-in

tens

ity(b)


0

20

40

60

DSC

()

0 8642


(a)

50 100 150 200


0

20

40

60

DSC

()

(b)









0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 6070 80 90 100


20

25

30

35

DSC

()

30

40

50

TPR

()

20

30

40

PPV

()

N = 3

N = 5

N = 9







DSC TPR PPV0

20

40

60

()


3rd IT4th IT


DSC TPR PPV


T1T2FLAIR

0

10

20

30

40

50

60

70

()




119872= 119862T1 119862T2 119862FLAIR In this





DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()
















References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










(a) (b) (c) (d) (e) (f)




119909and a


119889 (119875119909 119875119863) = 119889119868(119875119909 119875119863) + 120572 sdot 119889

119871(119875119909 119875119863) (7)






120572119905= 1205720lowast (119905 minus 1) (8)






3 Results












32 Evaluation
















20 40 60 80 100

T1-intensity

50

100

150

200

FLA

IR-in

tens

ity


(a)

20 40 60 80 100

T1-intensity


50

100

150

200

T2-in

tens

ity(b)


0

20

40

60

DSC

()

0 8642


(a)

50 100 150 200


0

20

40

60

DSC

()

(b)









0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 6070 80 90 100


20

25

30

35

DSC

()

30

40

50

TPR

()

20

30

40

PPV

()

N = 3

N = 5

N = 9







DSC TPR PPV0

20

40

60

()


3rd IT4th IT


DSC TPR PPV


T1T2FLAIR

0

10

20

30

40

50

60

70

()




119872= 119862T1 119862T2 119862FLAIR In this





DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()
















References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation


















32 Evaluation
















20 40 60 80 100

T1-intensity

50

100

150

200

FLA

IR-in

tens

ity


(a)

20 40 60 80 100

T1-intensity


50

100

150

200

T2-in

tens

ity(b)


0

20

40

60

DSC

()

0 8642


(a)

50 100 150 200


0

20

40

60

DSC

()

(b)









0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 6070 80 90 100


20

25

30

35

DSC

()

30

40

50

TPR

()

20

30

40

PPV

()

N = 3

N = 5

N = 9







DSC TPR PPV0

20

40

60

()


3rd IT4th IT


DSC TPR PPV


T1T2FLAIR

0

10

20

30

40

50

60

70

()




119872= 119862T1 119862T2 119862FLAIR In this





DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()
















References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation



















20 40 60 80 100

T1-intensity

50

100

150

200

FLA

IR-in

tens

ity


(a)

20 40 60 80 100

T1-intensity


50

100

150

200

T2-in

tens

ity(b)


0

20

40

60

DSC

()

0 8642


(a)

50 100 150 200


0

20

40

60

DSC

()

(b)









0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 6070 80 90 100


20

25

30

35

DSC

()

30

40

50

TPR

()

20

30

40

PPV

()

N = 3

N = 5

N = 9







DSC TPR PPV0

20

40

60

()


3rd IT4th IT


DSC TPR PPV


T1T2FLAIR

0

10

20

30

40

50

60

70

()




119872= 119862T1 119862T2 119862FLAIR In this





DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()
















References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 60 70 80 90 100


0 10 20 30 40 50 6070 80 90 100


20

25

30

35

DSC

()

30

40

50

TPR

()

20

30

40

PPV

()

N = 3

N = 5

N = 9







DSC TPR PPV0

20

40

60

()


3rd IT4th IT


DSC TPR PPV


T1T2FLAIR

0

10

20

30

40

50

60

70

()




119872= 119862T1 119862T2 119862FLAIR In this





DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()
















References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










DSC TPR PPV

112

121

211

122

212

221

111

0

10

20

30

40

50

60

70

()
















References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation














References



















































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation








































RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation









Research Article Patch-Based Segmentation with Spatial ... · Research Article Patch-Based...

Documents

Transcript of Research Article Patch-Based Segmentation with Spatial ... · Research Article Patch-Based...