Eurographics Conference on Visualization (EuroVis) 2020M. Gleicher, T. Landesberger von Antburg, and I. Viola(Guest Editors)

Volume 39 (2020), Number 3

Knowledge-Assisted Comparative Assessment of Breast Cancerusing Dynamic Contrast-Enhanced Magnetic Resonance Imaging

K. Nie1 , P. Baltzer2 , B. Preim1 , and G. Mistelbauer1

1Department of Simulation and Graphics, Otto-von-Guericke University Magdeburg, Germany2Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria

AbstractBreast perfusion data are dynamic medical image data that depict perfusion characteristics of the investigated tissue. These dataconsist of a series of static datasets that are acquired at different time points and aggregated into time intensity curves (TICs) foreach voxel. The characteristics of these TICs provide important information about a lesion’s composition, but their analysis istime-consuming due to their large number. Subsequently, these TICs are used to classify a lesion as benign or malignant. Thislesion scoring is commonly done manually by physicians and may therefore be subject to bias. We propose an approach thataddresses both of these problems by combining an automated lesion classification with a visual confirmatory analysis, especiallyfor uncertain cases. Firstly, we cluster the TICs of a lesion using ordering points to identify the clustering structure (OPTICS) andthen visualize these clusters. Together with their relative size, they are added to a library. We then model fuzzy inference rules byusing the lesion’s TIC clusters as antecedents and its score as consequent. Using a fuzzy scoring system, we can suggest a scorefor a new lesion. Secondly, to allow physicians to confirm the suggestion in uncertain cases, we display the TIC clusters togetherwith their spatial distribution and allow them to compare two lesions side by side. With our knowledge-assisted comparativevisual analysis, physicians can explore and classify breast lesions. The true positive prediction accuracy of our scoring systemachieved 71.4 % in one-fold cross-validation using 14 lesions.

CCS Concepts• Human-centered computing → Graph drawings; Visual analytics; Information visualization; • Computing methodologies→ Vagueness and fuzzy logic; • Information systems → Clustering; Digital libraries and archives;

1. Introduction

Breast cancer is the most common female’s invasive cancer, account-ing for almost a quarter of cancer cases among women globally, andit is also the leading cause of cancer death among women in over 100countries [BFS∗18]. Early detection of breast cancer by screeningprograms is supported by most expert societies [SAea17]. Com-monly used diagnostic imaging modalities are mammography andbreast ultrasound [WAJ∗18, Nat18]. Conventional imaging is ham-pered by limited sensitivity and specificity. While cancers are regu-larly missed in women with higher mammographic breast density,positive imaging findings require image-guided biopsy to establisha final diagnosis. Biopsies, however, are invasive, and in case ofstereotactic guidance costly procedures, can further cause physicalpain and psychological stress to the patient [HKvH08]. The major-ity of these biopsies yield benign results and are thus potentiallyavoidable. Therefore, an additional non-invasive imaging-based di-agnostic test method, such as dynamic contrast-enhanced magneticresonance imaging (DCE-MRI), would be welcome.

For diagnosis, radiologists need to analyze the features of le-sions for understanding the severity of the pathological processes

and the grade of the disease. According to the breast imaging re-porting and data system (BI-RADS) [BSB∗09] or Göttinger Score(GS) [FKG99], the disease grade can be expressed as a score, whichis conducive to the understanding of physicians. With the com-mon static imaging modalities, such as mammography and breastultrasound, radiologists can only use the morphological character-istics of the lesion for analysis. DCE-MRI, as a dynamic modal-ity, supplies the morphological characteristics together with thefunctional features of breast lesions, which not only illustrate thespatial information, such as tissue structure and vascularity, butalso physiological information on perfusion and permeability char-acteristics [MMSJP99]. Due to characteristic differences in thenet capillary diameter, vessel permeability and extracellular ex-travascular distribution space, DCE-MRI is able to distinguish be-nign from malignant lesions. Based on the characteristics of DCE-MRI, the analysis of these data is mainly focused on morpholog-ical [DGV∗16, MCW∗16, DVK∗18], texture [ASL∗11, MDvP17]and temporal features [BLA∗07, BFB∗09, MBB∗17].

Using the temporal features to analyze breast lesions is impor-tant [RPP∗20], since the perfusion of contrast agent (CA) variesdepending on the tissue types contained in a lesion, the temporal dy-

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and JohnWiley & Sons Ltd. Published by John Wiley & Sons Ltd.

DOI: 10.1111/cgf.13959

https://diglib.eg.orghttps://www.eg.org

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

Time

Inte

nsity

Baseline

Integral

DownSlope

PE

CA Arrival

TTPMTT

Acquisition Time End

Figure 1: A typical TIC with its parameters. PE means peakenhancement, TTP is time to peak, MTT is mean transit time,which represents the half position of the integral area. (Adaptedfrom [POM∗09])

namics of the acquired signal, which reflects the absorption, and therelease of the CA can provide a good basis for lesion classification.This temporal information can directly be expressed as the intensitychange of a lesion’s voxel over time, referred to as time intensitycurve (TIC), which is acquired by a series of magnetic resonanceimaging (MRI) scans before and after injection of CA (see Figure 1).Meanwhile, the signal intensity change percentage of a TIC can bequantified by the relative enhancement (RE) as

RE = 100 · (Ipost − Ipre)/Ipre, (1)

with Ipre being the precontrast (before CA arrival) and Ipost beingthe postcontrast (after CA arrival) signal intensity [KMK∗99]. Fig-ure 2 shows the nine typical categories of RE curves [GPTP10].Analyzing TICs and RE of a lesion allows physicians to efficientlyclassify it, or parts of it, as benign or malignant. Though rapid initialenhancement and distinct washout in the late phase are typical ofmalignant lesions, both benign and malignant lesions show variableenhancement characteristics [FGKD01, BFB∗09]. To account fornoise in DCE-MRI data, physicians usually select a small region ofinterest (ROI) instead of a single voxel and then inspect the averageTIC to analyze and classify the lesion [Kuh07].

However, this approach has some limitations. It is very time-consuming and error-prone, since a vast amount of data has to beanalyzed [ODH∗07]. Moreover, displaying only the average TICdiminishes the dynamic characteristics of the TICs when the lesionis strongly heterogeneous, but drawing all TICs into a single plotresults in overplotting. By only investigating the characteristics ofTICs, the spatial information of the lesion is ignored. Recently, ma-chine learning approaches aim to support physicians in performingthis task. However, these methods are still not sufficiently accurateto cope with the large variability of the data and to compete with thediagnosis of experienced physicians [MPF∗17, ZLD∗19].

In this paper, we propose a knowledge-assisted visual analysisapproach that allows physicians to analyze breast lesions based onthe classification results of a fuzzy inference system (FIS) and then

Time

Rel

ativ

e E

nhan

cem

ent

Type I

Type II

Type III

Type IV

Type V

Type VI

Type VII

Type VIII

Type IX

Early post-contrast phase Late postcontrast phase

Figure 2: Nine typical REs. At the late postcontrast phase, thecurves can be divided into three categories: type I, IV, VII (persistent)show continuous increase, type II, V, VIII (plateau) reach a plateau,and type III, VI, IX (washout) exhibit a decreasing pattern after aninitial increase. At the early postcontrast phase, the curves can alsobe divided into three categories: type I, II, III (rapid) have a rapidincrease, type VI, V, VI (normal) increase moderately and type VII,VIII, IX (slow) increase slowly. (Adapted from [GPTP10])

to visually confirm or correct the results. Our contributions can besummarized as:

• a visualization of TIC clusters (temporal analysis),• a spherical histogram of TIC clusters around the lesion centroid

(spatial analysis),• a visualization of two lesions’ spatio-temporal information side

by side (comparative analysis),• a fuzzy inference system for lesion scoring (classification), and• a confirmatory visual analysis approach.

To demonstrate if the classification results of our FIS can providephysicians with a useful reference, we evaluated the classificationaccuracy by using one-fold cross-validation. In addition, we showedour visual analysis approach to our collaborating clinical radiologistsfor evaluation and feedback.

2. Related Work

Analyzing TICs to comprehend the temporal features of DCE-MRIdata is widely adopted by physicians to distinguish between benignand malignant lesions [POM∗09]. In order to assess breast lesionsby using TICs, physicians have to detect or segment the lesionsfirst. Traditional approaches include manual selection [ARE∗14]or semi-automated [PMW∗14] selection of ROIs, which containeither a part or the whole lesion. Graph cut algorithms are com-monly used for segmenting lesions in medical imaging [CNZ∗12].An improved version of graph cuts is GrabCut [TGVB13], whichintroduces a global optimization term to calculate the link strengthbetween vertices to improve computation efficiency and simplifyimplementation.

Lesions, especially malignant ones, usually contain a large num-ber of voxels [HHHV01]. Displaying the TICs of each voxel of

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

14

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

such a lesion results in overplotting. To assess the major character-istics of many TICs, functional boxplots [SG11] abstract a seriesof curves into a representative curve and their band depth. Clus-tering may restrict the analysis of TICs to few cluster representa-tives. Iorio et al. [IFDS16] proposed a parsimonious time seriesclustering method with a penalized spline (P-spline) and k-meansalgorithm, which shows a good performance on multidimensionaldata measurements, while the cluster number needs to be set manu-ally. Unlike methods that require specifying the number of clusters,such as k-means, density-based clustering methods determine thenumber of clusters automatically. Density-based spatial clusteringof applications with noise (DBSCAN) is one of the most commondensity-based clustering algorithms [BK07]. Ordering points to iden-tify the clustering structure (OPTICS) is more robust against noisecompared to DBSCAN, but is more time-consuming [KKSZ11]. Pat-wary et al. [PPA∗13] proposed a parallel implementation of OPTICS,which significantly improved the computing time.

Since spatial information is not considered when analyzing TICs,Glaßer et al. [GPTP10] proposed a visual analytics approach thatmerged the voxels with similar TIC perfusion parameters into thesame color and glyph. This approach offers an improved TIC anal-ysis since the spatial and temporal features are simultaneously ex-pressed in a slice view. In a follow-up work, Glaßer et al. [GNPS13]applied density-connected subspace clustering (SUB-CLU), DB-SCAN and OPTICS to cluster TICs of the most suspicious regionsand displayed TICs characteristics in a slice view. This approachimproves the merging accuracy of the voxels, which have similarTIC perfusion parameters. However, it cannot represent details ofthe dynamic characteristics of voxels in a slice view.

Machine learning approaches have recently been employed tosupport physicians in lesion classification. Rasti et al. [RTP17]designed a mixture ensemble of convolutional neural network mod-els to discriminate between benign and malignant breast tumors.Zhou et al. [ZLD∗19] employed 3D deep learning to classify breastlesions. Although this approach is highly accurate, it is still notcomparable to experienced physicians. Besides, this approach cannot provide a clear physiological interpretation of its classificationmodel, which makes it difficult to share and reproduce the experi-ence of physicians in this model.

Instead of a binary lesion classification, fuzzy logic can be usedto model various degrees of malignancy. Furthermore, in a FIS,knowledge is represented in the form of linguistic rules. This al-lows physicians to share their expertise. Miranda et al. [MF15]created a breast cancer diagnosis system that is based on fuzzylogic and provides a suggestion using the BI-RADS classification.Ahadi et al. [ADL∗17] introduced a fuzzy inference system to pre-dict a benign or malignant state of a lesion, based on its physicalcharacteristics. While these fuzzy systems employ physicians’ expe-rience and provide good classification results, they lack a feedbackmechanism to take advantage of the initiative of physicians.

3. Methodology

Diagnostic visualizations play an important role in the analysis andclassification of breast lesions. Manual analysis is a time-consumingprocess and medical visualization approaches support physicians in

– Draw TIC cluster plot– Draw TIC cluster

histogram plot– Compare two lesions

side by side

Visualization

– Compare TIC clusters– Add new TIC clusters– Generate rules– Assign antecedents– Derive score

Fuzzy Inference System

– Load DCE-MRI data– Segment lesions– Cluster TICs

Feature Extraction

– Get info from FIS– View predicted score– Confirm/correct score

Visual Confirmation

Figure 3: Workflow of our visual analytics approach for breastlesion assessment. This approach includes four parts: feature extrac-tion, visualization, fuzzy inference system and visual confirmation.

aggregating low-level information. Visualization and analysis tech-niques require additional information about the imaging data, suchas the position of the lesions, their temporal intensity behavior, andthe spatial distribution of TICs. Moreover, providing an estimatedmalignancy score based on the knowledge of experienced physicianssupports guided as well as confirmatory analysis.

The workflow of our approach is outlined in Figure 3 and consistsof the following four steps: feature extraction, fuzzy inference sys-tem, visualization and visual confirmation. We first load a DCE-MRIdataset. Then, the lesions are automatically located and segmented.The TICs of the selected lesion are clustered using a parallel imple-mentation of OPTICS clustering [PPA∗13]. The feature extractionprocess is illustrated in Figure 4. In order to describe the spatialdistribution of this lesion, we create spherical bins around it andcompute a histogram of the TIC clusters. Meanwhile, a FIS is cre-ated on the basis of the TIC clusters and expert knowledge. It is usedto classify lesions and predict a score. Finally, the TICs clusters andthe histogram are shown to support physicians analyzing the lesionand evaluating the classification score. We also arrange both visual-izations in such a way that two lesions can be compared side by side,to directly compare the current lesion with others. After the analysis,physicians can confirm the FIS score or correct it. Subsequently, wedescribe each step of our workflow in detail.

3.1. Feature Extraction

To extract the TICs (temporal feature) from a lesion, we establish athree-step approach (see Figure 4). All datasets were acquired overan average of 400 seconds within 11 time points and were motion-

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

15

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

2D segmentation results

3D segmentation results

– 7 datasets (DCE-MRI)– 11 time points– Acquired over ≈ 400 s

Extract TICs(of selected lesion)

Cluster TICs(parallel OPTICS)

First time point

Merge

Load one dataset

Identify left & right breast

Select lesion

Load

DC

E-M

RId

ata

Seg

men

tles

ions

Clu

ster

TIC

s

Figure 4: Extraction of lesion features. After a DCE-MRI dataset isloaded, the images at the first time point are used to separate breastregions. With the selected dataset and the regions, suspicious objectsare segmented at each slice, and then these objects are merged into3D objects. When an object (mostly a lesion) is selected, all TICs inthis object are extracted and computed to get their clusters.

compensated afterwards. They are of resolution 384× 384× 144voxels. We then separate the breast regions and segment suspiciousobjects (including lesions and their surrounding blood vessels). TheTICs of a user-selected lesion are clustered with a density-basedmethod, which detects the number of clusters automatically.

Lesion segmentation. To facilitate physicians locating lesionsand reducing the search space for subsequent image processing op-erations, we use the dataset of the first time point to find the breastboundary. The left and right breast are then identified by this bound-ary with the vertical centerline of the axial slice images. Lesionsare then determined and segmented using the GrabCut [TGVB13]method. We automatically define the fore- and background seeds forGrabCut using the RE of a lesion (recall Figure 2). Since benign andmalignant tumors have a rapid increase in RE at the early postcon-trast phase [FGKD01], we mark such pixels inside the breast regionsas foreground and pixels with slow increase in RE as background.

Compared with the conventional graph cut method [CNZ∗12],GrabCut uses a new energy term to measure the L1 distance (Man-hattan distance) between the foreground and background appear-ance. The segmentation efficiency is optimized via this new energyterm [TGVB13]. Let S⊂Ω be a segment, where S is the segmentedobject and Ω is the set of all image pixels. Let θ

S and θS be the un-

normalized intensity histograms for the foreground and backgroundappearance, respectively. The energy function with the new term toconsider the L1 distance is written as follows [TGVB13]:

EL1(θS,θS) =−‖θS−θ

S‖L1 . (2)

Let nk be the number of pixels in the image that belongs to bin k andlet nS

k and nSk be the number of foreground and background pixels in

bin k, then Equation 2 can be rewritten as:

EL1(θS,θS) =

K

∑k=1

min(nSk ,n

Sk)−

|Ω|2

. (3)

Then, K auxiliary nodes A1,A2, . . . ,AK are added to the graph andall pixels of bin k are connected to auxiliary node Ak. Hence, anycut separating the fore- and background pixels must either cut nS

k or

nSk number of links that connect the pixels in bin k to the auxiliary

node Ak [TGVB13].

Because GrabCut is a 2D segmentation algorithm, we obtainmany ROIs in each axial slice of the input dataset. By comparingthe area coverage of each ROI between adjacent slices, we mergeoverlapping ROIs into a 3D object and then get the volume of interest(VOI). Each VOI (object, which is a lesion or its surrounding bloodvessels) is separated from each other by the connectivity.

TIC clustering. By observing all VOIs (objects) in the imag-ing data, the user can select one that is most suspected to be alesion. After the lesion is selected, the temporal information of itsvoxels in DCE-MRI breast data is extracted. We fit cubic splinesto the discrete time points to obtain a continuous time representa-tion [BJGG∗03]. This allows us to model even time points betweenthe observed data.

To cluster the TICs, we use their spline coefficients and a parallelimplementation of OPTICS [PPA∗13]. This method requires twoparameters: ε, which describes the maximum distance (radius) toconsider, and MinPts, the number of points required to form a cluster.The parameter MinPts is set to 4 in our experiments, because thesmallest possible region that can be segmented by our method is4 pixels. The other parameter ε has to be manually specified, becauseit controls the sensitivity to noise.

The parallel implementation of OPTICS [PPA∗13] is describedas follows. Let D be a high-dimensional data point set that containsM points. The neighborhood of a point p ∈ D within a given radiusε is defined as:

Nε(p) = q ∈ D : d(q, p)≤ ε , q 6= p, (4)

with d(q, p) being the Euclidean distance between points p and q.If |Nε(p)| ≥ MinPts, the point p ∈ D is considered a core point.A point q ∈ D is directly density-reachable with p ∈ D, if q ∈Nε(p). The point q ∈ D is density-reachable if there is a chain ofpoints p1, p2, . . . , pn with p1 = p, pn = q, and every pi+1 is directly

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

16

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

5mm 10mm

(a)

(b) (c)

(d)

(e)

Figure 5: Overview of our visual analysis approach. (a) shows a 3D visualization of a malignant tumor (red), whereas (b) displays an axialslice view of this tumor. The left column of (c) shows TIC clusters and their spatial distribution of a selected lesion (which is displayedin (a) and (b)). The TIC clusters and their spatial distribution in the right column of (c) are belonging to a lesion to be compared with theselected one. The middle column of (c) (between the left and right cluster plot areas) shows the statistical distribution of the selected clusters.(d) illustrates the spherical histogram bins. The center of the region is the centroid of the tumor and the size of each spherical bin is fivemillimeters. An input cluster (orange), which is selected in the left column of (c), and its most similar cluster (blue) in the lesion and TICcluster library (LC library) are illustrated in (e). The relative sizes of the selected lesion are RS(C_1) = 0.939 and RS(D) = 0.061.

density-reachable from pi, where 1≤ i < n and pi ∈ D. The user-specified initial distance ε is called generating distance (GD). Thesmallest distance δ that satisfies Nδ(p)≥MinPts is referred to ascore distance (CD). IfNε(p)< MinPts, the CD does not exist. Thereachability distance (RD) between two points is defined as:

RD(p,q) =

0, if Nε(p)< MinPts,max(CD(p),d(p,q)), otherwise.

(5)

Unreachable voxels (RD(p,q) = 0) are the noise and are excludedfrom the clustering. The clustering is started from an unprocessedpoint p that satisfies Nε(p) ≥ MinPts. We then store the pair ofpoints (q, p) with q ∈ Nε(p) that have the smallest RD(p,q). Forany δ with RD(p,q)≤ δ, the points p and q are in the same clusteras long asNδ(p)≥MinPts. If RD(p,q)> δ, p and q are in differentclusters. Repeatedly, a point k that is closest to the previously pickedcore points is selected and stored until traversing all points in space.

A minimum spanning tree (MST) [Pri57] T = V,E is em-ployed, where V and E denote the set of vertices and edges ofgraph T , respectively. The weight in T is minimal and it representsthe RD between two points in D. A MST on its local dataset iscomputed on a single CPU core, without any communication toother MSTs. Once all local MSTs are computed, they are mergedinto a global MST [PPA∗13]. The final result is obtained by re-moving any edge (q, p), where RD(p,q) > δ, from the MST andreturning the points in the remaining connected components as theclusters. The MSTs and connected components can be computed inparallel, increasing the clustering performance.

To obtain a robust result, we set ε in such a way that 65-70% of alesion’s TICs are clustered based on our research. At last, we use apointwise boxplot method [SG11], which is similar to a functionalboxplot but requires less computing time to calculate the medianintensity values, 50% regions and an envelope which completelyencloses the values represented by a cluster.

3.2. Visualization

In order to demonstrate how our system visualizes the temporal andspatial characteristics of a lesion, we selected a malignant tumor asexample. Figures 5a and 5b show this tumor highlighted in red in a3D and 2D view, respectively. Figure 5c shows the clustered TICsof the selected tumor, and the spatial distribution of these clusters. Aspherical bin, which is used to calculate the spatial distribution of theTIC clusters, is shown in Figure 5d. A selected cluster (see Figure 5c,left column) and its most similar cluster in the lesion and TIC clusterlibrary (LC library) (see Section 3.3) are shown in Figure 5e. Theselected cluster is colored in orange, while its most similar clusteris colored in blue.

When a lesion is selected, all its TICs are clustered, and the rep-resentative curves of each cluster are plotted. If a representativecurve is selected, its 50% region and envelope region will beshown (see Figure 5c). In this way, it is possible to prevent dif-ferent clusters from visually overlapping each other. All clusters arecolored based on a scheme determined with color brewer [BHH03]to prevent the adjacent colors being similar and difficult to distin-

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

17

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

guish. The legend in Figure 5c describes the following attributes ofa cluster. The first entry shows the corresponding color of the repre-sentative curve. Secondly, the name of the cluster is given, whereC stands for a normal cluster and D for a cluster that representsa necrotic region (dead tissue). The third entry is the number ofvoxels of the cluster and the last entry describes the relative sizein comparison to the size of all TIC clusters. The TICs that wereclassified as noise are not included in the plot. Whenever a clusteris selected, its statistical distribution is shown in the middle plotarea (see Figure 5c).

To display the correlation between the TIC clusters (temporalfeatures) and their spatial distribution (spatial features) of a lesion,we create a sphere with several bins to cover the entire lesion. Thesize of these spherical bins (see Figure 5d), which are constantlyfive millimeters wide, is plotted on the horizontal axis, and thenumber of the clustered voxels in each bin is plotted on the verticalaxis (see Figure 5c, bottom). As shown in Figure 5c, we arrangethe spherical histogram window below the TIC cluster window inorder to provide users with comprehensive means to observe andanalyze the TIC clusters. When comparing two lesions, the rightside is vertically flipped, i.e., the x-axis goes from right to left. Byarranging the windows side by side, differences and similaritiesbetween different tumors can be explored.

3.3. Fuzzy Inference System

We create an LC library (see Figure 6, left column) that stores thecontent (TICs, relative sizes, score) of each lesion. This libraryprovides reference data for predicting a score and for supportingphysicians to confirm the accuracy of the classification results ina knowledge-assisted analysis way (see Section 3.4). This libraryconsists of two parts: a set of TIC clusters and a set of lesions.The contents of each cluster and lesion are saved in these two sets.The cluster content includes a unique cluster name, color and itsintensity value at a given time point. The lesion content includes thelesion name, score (given by domain experts) and a set of its TICclusters, which is a subset of the library’s set of TIC clusters. Thissubset stores the cluster index (lesion’s cluster name) and each TICcluster’s relative size.

Meanwhile, similar TIC clusters in the library’s set of clusters willbe merged to clearly reflect the associations between various clusters.If two lesions link with a large number of the same unique clusters,these lesions are considered as similar. To merge similar clusters,we employed the overlap coefficient between two clusters [MK16].It is defined as:

VS =|F(Ci)∩F(C j)|

min(|F(Ci)|, |F(C j)|), (6)

where i 6= j and F(·) indicates the 50% central region of a cluster. IfVS ≥ 0.8, the cluster with the narrower central region will be mergedwith the other one.

When a new lesion (without a score) is put into the LC library, itsscore can be predicted via a FIS (see Figure 6). In the following, weexplain the construction of our fuzzy model by linguistic translation,knowledge generation and prediction.

Linguistic translation. In order to use human semantic expres-sions to build expert knowledge in a FIS, the cluster’s relative size

Predicted Score & Certainty:SP = 8.0, SC = 50%

Fuzzy Inference System (FIS)

Lesions In Library:

L1 = <C1,RS1 >,<C2,RS2 >,SL1 = 4.0 ,...... ,L14 = < C37,RS1 >,<C38,RS2 >,< C10,RS3 >,SL14 = 4.0

New Lesion:

Lnew = < C_1,RS1 >,<D,RS2 >

Clusters In Library:

GC = C1,C2, ...,C38

......Input & Output Variables:

Rules:

L1: if RS(C1) is Large andRS(C2) is Small then Scoreis Intermediate,

L2: if RS(C3) is Small andRS(C4) is Large and RS(C5)is Small then Score isIntermediate, ...,L14: if RS(C37) is Small and RS(C38) is Large andRS(C10) is Small then Score is Intermediate

Map of Clusters:

C_1 is most similar with C33

D is merged with C21

Input Values:RS(C33) = 0.939, RS(C21) =0.061, others = 0

New Input Lesion New Input Lesion

Figure 6: Illustration of our fuzzy inference system (FIS) workflow.Ln are the unique lesion names, GC is a set of clusters, Cn are theunique cluster names, RS is the relative size and Ln is the identifierof the rules. The left column describes the structure of the LC libraryand the right column shows the score prediction process of a newlesion. The predicted score of the new lesion is SP and the certaintyscore is SC.

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

18

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

and the lesion’s score in the library should be translated into a natu-ral language representation. The conversion of numbers to linguisticvariables is defined as follows:

KRS =

Small, if 0.0≤ RS(Cn)< 0.4,Medium, if 0.4≤ RS(Cn)≤ 0.6,Large, if 0.6 < RS(Cn)≤ 1.0.

(7)

KS =

Benign, if 0≤ S(Ln)≤ 2,ProbablyBenign, if 2 < S(Ln)≤ 3,Intermediate, if 3 < S(Ln)< 5,ProbablyMalignant, if 5≤ S(Ln)< 6,Malignant, if 6≤ S(Ln)≤ 8,

(8)

where KRS is the knowledge of relative size, KS is the knowledge oflesion score, RS(Cn) is the relative size of cluster Cn and S(Ln) isthe score of lesion Ln. The score of a lesion is from 0 to 8, basedon GS [FB14], where 0 means benign and 8 malignant.

Knowledge generation. The generation of explicit knowledge isbased on the lesions’ contents in the library and the way humansthink [FWR∗17]. Knowledge is modeled in the form of linguisticrules and terms and not in the form of exact numbers when fuzzylogic is involved. These rules usually have the structure of IF-THENclauses with their antecedents and consequents using linguistic vari-ables and terms. The linguistic input and output fuzzy variables andtheir linguistic terms are shown in Figure 6. The linguistic termsSmall, Large, Medium of the linguistic input variables are basedon the sigmoid and Gaussian kernel functions, respectively. Thelinguistic terms Benign and Malignant of the linguistic output vari-ables are based on the sigmoid functions, while ProbablyBenign,Intermediate and ProbablyMalignant use Gaussian kernel functions.The fuzzy rules Kε represent the domain knowledge in a humanreadable form, which can be saved in an external file. This allowsphysicians to share and adapt their knowledge with other medicalcenters, which possibly increases the reproducibility of the scoringprocedure. For this reason we chose a FIS, a design choice based onseveral discussions with one of our collaborating physicians. Someof the rules are displayed in Figure 6, where RS(Cn) is the relativesize of the cluster Cn in the lesion Ln, and Score is the GS of thislesions, provided by domain experts.

Prediction. The TIC clusters C_n of the input lesion will bemerged with the clusters Cn in the LC library. If such an input clustercannot be merged, it is substituted with the most similar cluster fromthe LC library. After merging and substitution, the map of clustersand lesions will be established (see Figure 6), i. e., C_1←C33 andD← C21 with C_1,D ∈ Lnew and C33,C21 ∈ LC library. Weuse the Mamdani method as inference model and the relative size ofthe input lesion’s clusters as input values [MF15]. The schematicrepresentation of a FIS is shown in Figure 7. For each lesion in theLC library we generate an IF-THEN rule. The antecedents of theserules consist of the LC library’s clusters with their relative size andthe consequents are the lesions’ score. The input variables of therules are the LC library’s clusters. For each input variable, we usethe relative size of the input lesion’s clusters if the cluster in the LClibrary is merged or substituted with the one of the input lesion’sclusters. If the cluster is not included in the input lesion, its relativesize is zero. All implications (IF-THEN rules) are subsequentlyevaluated, their results aggregated using the maximum operator andthen defuzzified with the centroid method [ADL∗17]. The certainty

InputValues

Fuzzi-fication

InferenceDefuzzi-fication

Score

FuzzyRules

Figure 7: Illustration of our fuzzy inference system for breast lesionclassification.

score SC is defined as SC = 100 ·Nm/Nt , where Nm is the numberof the merged clusters and Nt is the total number of TIC clustersof the input lesion. When SC = 100%, all clusters were mergedwith the clusters in the LC library. This represents a certain result,because the predicted score is derived from existing knowledge. Ifno cluster could be merged, i.e., all were substituted, SC = 0%. It isespecially important to show such results to the physicians, as theyare reminded to validate the result by comparing the input lesionwith several known lesions. This can be done by visually comparinglesions side by side.

3.4. Visual Confirmation

After getting the classification result (score) of a lesion from ourFIS, physicians can confirm this result or correct it. If one inputTIC cluster is selected, its merged or substituted cluster in the LClibrary is automatically found and highlighted. Meanwhile, these twoclusters will be plotted simultaneously in the same area for visualverification (recall Figure 5e). If the input cluster is substituted thenit will be colored in orange, otherwise, the input cluster will bered if it is merged. All clusters in the LC library can be selectedfor visual comparison with the input TIC cluster, which supportsthis verification step. The library also assists physicians in theirdecision-making by allowing them to examine the fuzzy rule of alesion together with their relative sizes of the TIC clusters (IF-THENantecedent). By comparing the spatio-temporal information of theselected lesion with a known lesion, the classification result (score,IF-THEN consequent) can be further confirmed. If the physiciansthink the result is wrong, they can replace the suggested score by theresult obtained from analyzing the lesion using the visual analysisapproach that we provide.

4. Implementation

We implemented the segmentation method, clustering, visualiza-tion, fuzzy inference system and the knowledge-assisted analysisapproach in C++ using OpenMP, OpenGL, FuzzyLite, Qt andQCustomPlot. We used Qt5 for the user interface and QCustomPlotfor drawing the TIC plots and histograms. The parallel OPTICSclustering was implemented in C++ and parallelized with OpenMP.Volume and slice rendering was implemented in OpenGL. We usedthe C++ library FuzzyLite [RV18] for the FIS. All results in thiswork have been created on an Intel Core i7 CPU at 2.8 GHz and32 GB system memory.

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

19

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

5. Results and Discussions

In this section, we demonstrate example lesions with their TICclusters and spatial distributions, also involved with their scoreprediction via FIS (see Figures 8 to 10). These lesions encompassthree benign and three malignant tumors, highlighted in red in theslice views. We separately specify the ε of OPTICS (see Table 1)for each lesion such that the clustered voxels cover about 65-70%of the total number of voxels in each lesion (recall Section 3).

Table 1 shows the test results of each lesion, where ID is thelesion’s identifier, DataSet is the dataset that lesion belongs to, ε isthe parameter for OPTICS clustering, TS is the segmentation time,TC is the clustering time, SR is the real score which is given by thedomain experts, SP is the prediction score via our FIS using one-foldcross-validation, and SC is the certainty score (recall Section 3.3).

Figure 8 presents two breast tumors, a malignant (left) and abenign (right) one. The top images show axial slices of the DCE-MRI data, with the corresponding lesion identification at the top.The malignant tumor (left) exhibits necrosis (dead tissue), as shownby its cluster representative curve (black) and its spatial distribu-tion. Necrosis is an important indicator of malignancy. Since weuse the alpha channel when filling colors, transition colors appearwhen histograms with different colors are overlapping, e.g., theblack histogram in the left that relates to the black cluster curveis shown as ink-blue. Analyzing clusters C_1 to D of the right le-sion (see Figure 8, right column), we can observe a clear upwardtrend throughout the entire course of time, while the trend of theleft lesion’s cluster C_1 remains flat after 200 seconds. The size ofeach cluster or lesion can be directly compared by examining thespherical histogram of both lesions. Both lesions are removed fromthe LC library to predict their score. The suggested GS are 8.0 (left)and 4.0 (right). The cluster D of the left lesion is merged with C21in the LC library, and cluster C_1 is substituted (cannot be mergedthrough the similarity matching, recall Section 3.3) by C11 in the LClibrary, so the certainty score of the left lesion is 50%. For the right

Table 1: For each lesion we show its identifier (ID), the dataset(DataSet) it belongs to, the ε of the OPTICS clustering, the segmen-tation time (TS), the clustering time (TC), the real score (SR), thepredicted score (SP) and the certainty score (SC).

ID DataSet ε TS[s] TC[s] SR SP SC

L1 D01 6.20 13.2 0.14 2 4.0 0%L2 D02 0.93 66.5 2.17 4 4.0 100%L3 D02 1.83 66.5 2.02 5 5.6 33%L4 D02 0.92 66.5 2.84 5 5.6 50%L5 D02 2.51 66.5 1.62 7 7.4 0%L6 D03 0.92 97.3 2.43 4 4.0 50%L7 D04 0.90 102.6 21.94 4 4.0 80%L8 D04 0.93 102.6 39.51 7 5.0 100%L9 D04 2.05 102.6 2.74 4 4.0 50%L10 D04 4.09 102.6 5.32 4 6.4 33%L11 D05 1.40 75.8 37.28 6 4.0 25%L12 D06 0.72 131.4 510.34 8 4.0 33%L13 D07 0.74 115.1 53.98 7 8.0 50%L14 D07 0.50 115.1 78.73 4 4.0 33%

FIS

L13 L7

SP = 8.0, SC = 50% SP = 4.0, SC = 75%

Figure 8: Comparison of a malignant (left) and a benign (right)tumor. An axial slice view of both tumors is shown at the top. Thecentral images show the TIC clusters and their spatial distributionis displayed below. The scores suggested by our FIS are given at thebottom together with their certainty.

lesion, its clusters C_1, C_2 and C_3 are merged with C7, C19 andC22 in the LC library, respectively, and its cluster D is substituted byC12 in the LC library, so its certainty score is 75%.

Two malignant lesions are compared in Figure 9. The left lesionhas 2 TIC clusters of which C1 increases slowly and then changes toa plateau. The right lesion has 4 TIC clusters with a rapid increasebefore 200 seconds and a slow increase thereafter. The left lesionreceives a correct predicted score (SR : SP = 7.0 : 7.4) from the FIS,but the score for the right lesion is lower than the actual score givenby the domain experts (SR : SP = 6.0 : 4.0). The reason for this isthat the clusters C_1 to C_3 of the right lesion show a moderate torapid increase and an increasing trend, making it more similar tothe lower scored lesions in the LC library when used by the FISfor prediction. The rather low certainty score of the left (0%) andright (25%) lesion advises physicians to compare these lesions withseveral known lesions to verify their predicted score.

Two benign lesions are compared in Figure 10. TIC cluster C_1of the lesion shows a rapid and steady increase, whereas cluster C_1of the right lesion exhibits similar characteristics, but cluster C_2 ofthe right lesion increases slowly and then changes to a plateau. The

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

20

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

FIS

L5 L11

SP = 7.4, SC = 0% SP = 4.0, SC = 25%

Figure 9: Comparison of two malignant tumors. The left one gets acorrect prediction score while the right one gets a wrong score.

left lesion receives a correct predicted score (SR : SP = 4.0 : 4.0)from the FIS, but the score for the right lesion is higher than theactual score given by the domain experts (SR : SP = 4.0 : 6.4). Thereason for this is that the clusters C_2 of the right lesion show a slowincrease and a decrease trend after 350 seconds, making it moresimilar to the higher scored lesions in the LC library when usedby the FIS for prediction. The certainty score of the right lesion islow (33%), physicians are advised to compare it with other knownlesions to make a confident decision on the predicted score.

6. Evaluation

As shown in Table 1, lesions L2, L3, L4, L5, L6, L7, L9, L13 andL14 are accurately predicted. L1 has the lowest score and it is theonly one with such a low score in our LC library. If L1 is removedfrom the library and a new lesion is scored by our FIS, the predictedscore cannot be lower than the minimum score of the current lesionsin the library, which is 4.0. Hence, we consider the predicted scoreof L1 to be correct, but plan to add further lesions in the futureto solve this problem. The remaining incorrectly classified lesionsare L8, L10, L11 and L12. Hence, our approach has a predictionaccuracy of 71.4%, with 10 of 14 lesions being correctly classified.

The segmentation time TS (see Table 1) is the time required tosegment all suspicious objects (lesions and their surrounding blood

FIS

L9 L10

SP = 4.0, SC = 50% SP = 6.4, SC = 33%

Figure 10: Comparison of two benign tumors. The left one gets acorrect prediction score while the right one gets a wrong score.

vessels) in both breasts of a single dataset, and TC is the clusteringtime used by the parallel OPTICS implementation. The segmen-tation time is mainly affected by the breast size and number ofsuspicious voxels, whereas the clustering time is only affected bythe number of TICs in the lesion. The average segmentation andclustering time is 87.4 seconds and 54.36 seconds, respectively.

To obtain a qualitative feedback on the applicability of our visualconfirmation analysis, we consulted a radiologist with three yearsof experience. In her opinion, the initial goal of such a tool is tospeed up the workflow, reduce the interobserver variability andimprove reproducibility. She also stated that the tool seems to bequickly understood and learned by radiologists and it seems to bebetter suited for uncertain cases. Both statements confirm our designdecisions, as our tool is tailored to radiologists and uncertain cases.On the one hand, such cases are characterized by low certaintyscores, and on the other hand we provide visual means to confirm orcorrect predicted scores. The radiologist also commented that thelesion scoring is rather subjective and our tool seems to improve thereproducibility of the scoring. This confirms the design choice of theFIS, but the radiologist mentioned that this tool does not provide anobjective score. For this we would have to conduct a large, preferablymulticenter, study. In summary, we received indicative feedback thatshows the potential of our tool to deliver reproducible results. To

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

21

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

obtain a profound assessment of applicability and accuracy, largerand more comprehensive studies would have to be carried out.

Limitations. Although our approach automatically identifies theleft and right breasts, segments and cluster lesions, and predicts thescore based on FIS, it has some limitations. Firstly, the parameter ε

of the clustering method has to be set manually, which means theusers must have a certain understanding of the spatial distributionof the data. Secondly, our segmentation algorithm potentially re-moves small objects that are even difficult for experienced users tosegment, especially if the lesion has only subtle changes over time.Thirdly, our analysis is currently limited to TICs and their spatialdistribution, which does not involve additional spatial information,such as morphology and texture. Fourth, the number of datasets weused to construct the FIS was limited.

7. Conclusions and Future Work

We proposed a knowledge-assisted visual analysis approach thatarranges the temporal and spatial characteristics of TICs side byside in order to facilitate the classification of breast lesions. Byusing our visual analysis approach and FIS classification result,physicians can make effective use of expert knowledge to provide areproducible lesion score. The FIS demonstrated a good predictionaccuracy, even though the number of our datasets was limited. Thevisual comparison of two lesions allows physicians to understandand analyze TIC clusters, which is helpful for tumor assessment.The 50% central region and the cluster envelope is only displayed fora selected cluster to avoid the mutual interference between differentclusters in the same plotting area.

Future avenues are an automatic estimation of the parallel OP-TICS parameter ε and the investigation of the FIS predictive powerby using a larger cohort. We also plan to combine the TIC clusterswith texture as well as morphological features and a profound analy-sis of a lesion’s boundary. As suggested by one of our collaboratingradiologists, we will extend the temporal and spatial TIC clusterviews with a third histopathological view of a lesion and we planto support other imaging modalities such as mammography, normalmagnetic resonance imaging and ultrasound.

Acknowledgments

The authors would like to thank Joanna Rabczak, University Hospi-tal Magdeburg, Germany, for her indispensable help and valuablefeedback. The clinical data were acquired at the university hospitalof Jena, Germany in a prospective, ethical review board approved(approval number 2576-06/09) study. All patients gave informedconsent to the use of their anonymized MR imaging data and finaldiagnosis (i.e. benign vs. malignant).

References[ADL∗17] AHADI F. S., DESAI M. R., LEI C., LI Y., JIA R.: Feature-

Based classification and diagnosis of breast cancer using fuzzy inferencesystem. In Proc. of IEEE International Conference on Information andAutomation (ICIA) (July 2017), pp. 517–522. 3, 7

[ARE∗14] AGNER S. C., ROSEN M. A., ENGLANDER S.,TOMASZEWSKI J. E., FELDMAN M. D., ZHANG P., MIES C.,SCHNALL M. D., MADABHUSHI A.: Computerized image analysis for

identifying triple-negative breast cancers and differentiating them fromother molecular subtypes of breast cancer on dynamic contrast-enhancedMR images: A feasibility study. Radiology 272, 1 (2014), 91–99.doi:10.1148/radiol.14121031. 2

[ASL∗11] AGNER S. C., SOMAN S., LIBFELD E., MCDONALD M.,THOMAS K., ENGLANDER S., ROSEN M. A., CHIN D., NOSHER J.,MADABHUSHI A.: Textural kinetics: A novel dynamic contrast-enhanced(DCE)-MRI feature for breast lesion classification. J Digit Imaging 24, 3(June 2011), 446–463. doi:10.1007/s10278-010-9298-1. 1

[BFB∗09] BALTZER P. A. T., FREIBERG C., BEGER S., VAG T., DI-ETZEL M., HERZOG A. B., GAJDA M., CAMARA O., KAISER W. A.:Clinical MR-mammography: are computer-assisted methods superiorto visual or manual measurements for curve type analysis? a system-atic approach. Acad Radiol 16, 9 (Sept. 2009), 1070–1076. doi:10.1016/j.acra.2009.03.017. 1, 2

[BFS∗18] BRAY F., FERLAY J., SOERJOMATARAM I., SIEGEL R. L.,TORRE L. A., JEMAL A.: Global cancer statistics 2018: GLOBOCANestimates of incidence and mortality worldwide for 36 cancers in 185countries. CA: A Cancer Journal for Clinicians 68, 6 (2018), 394–424.doi:10.3322/caac.21492. 1

[BHH03] BREWER C. A., HATCHARD G. W., HARROWER M. A.: Color-Brewer in Print: A Catalog of Color Schemes for Maps. Cartogr Geogr InfSci 30, 1 (Jan. 2003), 5–32. doi:10.1559/152304003100010929.5

[BJGG∗03] BAR-JOSEPH Z., GERBER G. K., GIFFORD D. K.,JAAKKOLA T. S., SIMON I.: Continuous representations of time-seriesgene expression data. J Comput Biol 10, 3 (June 2003), 341–356.doi:10.1089/10665270360688057. 4

[BK07] BIRANT D., KUT A.: ST-DBSCAN: An algorithm for clusteringspatial–temporal data. Data & Knowledge Engineering 60, 1 (Jan. 2007),208–221. doi:10.1016/j.datak.2006.01.013. 3

[BLA∗07] BEHRENS S., LAUE H., ALTHAUS M., BOEHLER T., KUEM-MERLEN B., HAHN H. K., PEITGEN H.-O.: Computer assistancefor MR based diagnosis of breast cancer: Present and future chal-lenges. Comput Med Imaging Graph 31, 4 (June 2007), 236–247.doi:10.1016/j.compmedimag.2007.02.007. 1

[BSB∗09] BURNSIDE E. S., SICKLES E. A., BASSETT L. W., RUBIND. L., LEE C. H., IKEDA D. M., MENDELSON E. B., WILCOX P. A.,BUTLER P. F., D’ORSI C. J.: The ACR BI-RADS Experience: LearningFrom History. J Am Coll Radiol 6, 12 (Dec. 2009), 851–860. doi:10.1016/j.jacr.2009.07.023. 1

[CNZ∗12] CHEN X., NIEMEIJER M., ZHANG L., LEE K., ABRAMOFFM. D., SONKA M.: Three-Dimensional Segmentation of Fluid-Associated Abnormalities in Retinal OCT: Probability Constrained Graph-Search-Graph-Cut. IEEE Trans Med Imaging 31, 8 (Aug. 2012), 1521–1531. doi:10.1109/TMI.2012.2191302. 2, 4

[DGV∗16] DALMIS M. U., GUBERN-MÉRIDA A., VREEMANN S.,KARSSEMEIJER N., MANN R., PLATEL B.: A computer-aided diagnosissystem for breast DCE-MRI at high spatiotemporal resolution. Med Phys43, 1 (Jan. 2016), 84–94. doi:10.1118/1.4937787. 1

[DVK∗18] DALMIS M. U., VREEMANN S., KOOI T., MANN R. M.,KARSSEMEIJER N., GUBERN-MÉRIDA A.: Fully automated detectionof breast cancer in screening MRI using convolutional neural networks. JMed Imaging 5, 1 (Jan. 2018). doi:10.1117/1.JMI.5.1.014502.1

[FB14] FISCHER U., BAUM F.: Diagnostik und Therapie des Mam-makarzinoms. Georg Thieme Verlag, Jan. 2014. 7

[FGKD01] FURMAN-HARAN E., GROBGELD D., KELCZ F., DEGANIH.: Critical role of spatial resolution in dynamic contrast-enhancedbreast MRI. J Magn Reson Imaging 13, 6 (June 2001), 862–867. doi:10.1002/jmri.1123. 2, 4

[FKG99] FISCHER U., KOPKA L., GRABBE E.: Breast Carcinoma:Effect of Preoperative Contrast-enhanced MR Imaging on the Ther-apeutic Approach. Radiology 213, 3 (Dec. 1999), 881–888. doi:10.1148/radiology.213.3.r99dc01881. 1

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

22

K. Nie et al. / Knowledge-Assisted Comparative Assessment of Breast Cancer using Dynamic Contrast-Enhanced Magnetic Resonance Imaging

[FWR∗17] FEDERICO P., WAGNER M., RIND A., AMOR-AMORÓS A.,MIKSCH S., AIGNER W.: The role of explicit knowledge: A conceptualmodel of knowledge-assisted visual analytics. In Proc. of IEEE VAST(Oct. 2017), pp. 92–103. doi:10.1109/VAST.2017.8585498. 7

[GNPS13] GLASSER S., NIEMANN U., PREIM B., SPILIOPOULOU M.:Can we distinguish between benign and malignant breast tumors in DCE-MRI by studying a tumor’s most suspect region only? In Proc. of Interna-tional Symposium on Computer-Based Medical Systems (2013), pp. 77–82.doi:10.1109/CBMS.2013.6627768. 3

[GPTP10] GLASSER S., PREIM U., TÖNNIES K., PREIM B.: A visualanalytics approach to diagnosis of breast DCE-MRI data. Comput Graph34, 5 (Oct. 2010), 602–611. doi:10.1016/j.cag.2010.05.016.2, 3

[HHHV01] HIEKEN T. J., HARRISON J., HERREROS J., VELASCO J. M.:Correlating sonography, mammography, and pathology in the assessmentof breast cancer size. The American Journal of Surgery 182, 4 (Oct. 2001),351–354. doi:10.1016/S0002-9610(01)00726-7. 2

[HKvH08] HEMMER J. M., KELDER J. C., VAN HEESEWIJK H. P. M.:Stereotactic large-core needle breast biopsy: analysis of pain and dis-comfort related to the biopsy procedure. Eur Radiol 18, 2 (Feb. 2008),351–354. doi:10.1007/s00330-007-0762-3. 1

[IFDS16] IORIO C., FRASSO G., D’AMBROSIO A., SICILIANO R.: Par-simonious time series clustering using P-splines. Expert Syst Appl 52,Supplement C (June 2016), 26–38. doi:10.1016/j.eswa.2016.01.004. 3

[KKSZ11] KRIEGEL H.-P., KRÖGER P., SANDER J., ZIMEK A.: Density-based clustering. Data Min Knowl Discov 1, 3 (2011), 231–240. doi:10.1002/widm.30. 3

[KMK∗99] KUHL C. K., MIELCARECK P., KLASCHIK S., LEUTNERC., WARDELMANN E., GIESEKE J., SCHILD H. H.: Dynamic breastMR imaging: Are signal intensity time course data useful for differentialdiagnosis of enhancing lesions? Radiology 211, 1 (Apr. 1999), 101–110.doi:10.1148/radiology.211.1.r99ap38101. 2

[Kuh07] KUHL C.: The Current Status of Breast MR Imaging Part I.Choice of Technique, Image Interpretation, Diagnostic Accuracy, andTransfer to Clinical Practice. Radiology 244, 2 (2007), 356–378. doi:10.1148/radiol.2442051620. 2

[MBB∗17] MUS R. D., BORELLI C., BULT P., WEILAND E., KARSSE-MEIJER N., BARENTSZ J. O., GUBERN-MÉRIDA A., PLATEL B.,MANN R. M.: Time to enhancement derived from ultrafast breast MRI asa novel parameter to discriminate benign from malignant breast lesions.Eur J Radiol 89 (Apr. 2017), 90–96. doi:10.1016/j.ejrad.2017.01.020. 1

[MCW∗16] MARINO M. A., CLAUSER P., WOITEK R., WENGERT G. J.,KAPETAS P., BERNATHOVA M., PINKER-DOMENIG K., HELBICHT. H., PREIDLER K., BALTZER P. A. T.: A simple scoring systemfor breast MRI interpretation: does it compensate for reader experi-ence? Eur Radiol 26, 8 (Aug. 2016), 2529–2537. doi:10.1007/s00330-015-4075-7. 1

[MDvP17] MILENKOVIC J., DALMIS M. U., ŽGAJNAR J., PLATEL B.:Textural analysis of early-phase spatiotemporal changes in contrast en-hancement of breast lesions imaged with an ultrafast DCE-MRI protocol.Med Phys 44, 9 (Sept. 2017), 4652–4664. doi:10.1002/mp.12408.1

[MF15] MIRANDA G. H. B., FELIPE J. C.: Computer-aided diagnosissystem based on fuzzy logic for breast cancer categorization. Computersin Biology and Medicine 64 (Sept. 2015), 334–346. doi:10.1016/j.compbiomed.2014.10.006. 3, 7

[MK16] M.K V., K K.: A Survey on Similarity Measures in Text Mining.Machine Learning and Applications: An International Journal 3, 1 (Mar.2016), 19–28. doi:10.5121/mlaij.2016.3103. 6

[MMSJP99] MEYER S., MÜLLER-SCHIMPFLE M., JÜRGENS H., PEIT-GEN H.: MT-DYNA: Computer assistance for the evaluation of dynamicMR and CT data in a clinical environment. In Proc. of Computer AssistedRadiology and Surgery (1999), pp. 331–334. 1

[MPF∗17] MARRONE S., PIANTADOSI G., FUSCO R., PETRILLO A.,SANSONE M., SANSONE C.: An Investigation of Deep Learningfor Lesions Malignancy Classification in Breast DCE-MRI. In Im-age Analysis and Processing (2017), Lecture Notes in Computer Sci-ence, Springer International Publishing, pp. 479–489. doi:10.1007/978-3-319-68548-9_44. 2

[Nat18] NATIONAL GUIDELINE ALLIANCE (UK): Early and locallyadvanced breast cancer: diagnosis and management. National Institutefor Health and Care Excellence: Clinical Guidelines. National Institutefor Health and Care Excellence (UK), London, 2018. 1

[ODH∗07] OELTZE S., DOLEISCH H., HAUSER H., MUIGG P., PREIMB.: Interactive Visual Analysis of Perfusion Data. IEEE Trans Vis ComputGraph 13, 6 (Nov. 2007), 1392–1399. doi:10.1109/TVCG.2007.70569. 2

[PMW∗14] PLATEL B., MUS R., WELTE T., KARSSEMEIJER N., MANNR.: Automated characterization of breast lesions imaged with an ultrafastDCE-MR protocol. IEEE Trans Med Imaging 33, 2 (Feb. 2014), 225–232.doi:10.1109/TMI.2013.2281984. 2

[POM∗09] PREIM B., OELTZE S., MLEJNEK M., GROLLER E., HEN-NEMUTH A., BEHRENS S.: Survey of the Visual Exploration and Analysisof Perfusion Data. IEEE Trans Vis Comput Graph 15, 2 (Mar. 2009),205–220. doi:10.1109/TVCG.2008.95. 2

[PPA∗13] PATWARY M. A., PALSETIA D., AGRAWAL A., LIAO W.-K.,MANNE F., CHOUDHARY A.: Scalable parallel OPTICS data clusteringusing graph algorithmic techniques. In Proc. SC: The International Con-ference for High Performance Computing, Networking, Storage, and Anal-ysis (2013), pp. 49:1–49:12. doi:10.1145/2503210.2503255. 3,4, 5

[Pri57] PRIM R. C.: Shortest connection networks and some general-izations. Bell System Technical Journal 36, 6 (Nov. 1957), 1389–1401.5

[RPP∗20] ROMEO V., PICARIELLO V., PIGNATA A., MANCUSI V.,STANZIONE A., CUOCOLO R., DI CRESCENZO R., ACCURSO A.,STAIBANO S., IMBRIACO M.: Influence of different post-contrast timepoints on dynamic contrast-enhanced (DCE) MRI T staging in breast can-cer. Eur J Radiol 124 (Mar. 2020), 108819. doi:10.1016/j.ejrad.2020.108819. 1

[RTP17] RASTI R., TESHNEHLAB M., PHUNG S. L.: Breast cancerdiagnosis in DCE-MRI using mixture ensemble of convolutional neuralnetworks. Pattern Recognition 72 (Dec. 2017), 381–390. doi:10.1016/j.patcog.2017.08.004. 3

[RV18] RADA-VILELA J.: The fuzzylite libraries for fuzzy logic control,2018. Accessed 2020/01/10. URL: https://fuzzylite.com/. 7

[SAea17] SARDANELLI F., AASE H. S., ET AL.: Position paper onscreening for breast cancer by the European Society of Breast Imaging(EUSOBI) and 30 national breast radiology bodies. Eur Radiol 27, 7 (July2017). doi:10.1007/s00330-016-4612-z. 1

[SG11] SUN Y., GENTON M. G.: Functional boxplots. J Comput GraphStat 20, 2 (Jan. 2011), 316–334. doi:10.1198/jcgs.2011.09224.3, 5

[TGVB13] TANG M., GORELICK L., VEKSLER O., BOYKOV Y.: Grab-Cut in one cut. In Proc. of ICCV (Dec. 2013), pp. 1769–1776. doi:10.1109/ICCV.2013.222. 2, 4

[WAJ∗18] WÖCKEL A., ALBERT U.-S., JANNI W., SCHARL A.,KREIENBERG R., STÜBER T.: The Screening, Diagnosis, Treatment, andFollow-Up of Breast Cancer. Deutsches Ärzteblatt International 115, 18(May 2018), 316–323. doi:10.3238/arztebl.2018.0316. 1

[ZLD∗19] ZHOU J., LUO L.-Y., DOU Q., CHEN H., CHEN C., LI G.-J., JIANG Z.-F., HENG P.-A.: Weakly supervised 3d deep learningfor breast cancer classification and localization of the lesions in MRimages. Journal of Magnetic Resonance Imaging 50, 4 (2019), 1144–1151. doi:10.1002/jmri.26721. 2, 3

c© 2020 The Author(s)Computer Graphics Forum c© 2020 The Eurographics Association and John Wiley & Sons Ltd.

23