Clinical Assessment for Deep Vein Thrombosis using Support...

IN DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING 300, SECOND CYCLECREDITS

, STOCKHOLM SWEDEN 2015

Clinical Assessment for DeepVein Thrombosis using SupportVector MachinesA DESCRIPTION OF A CLINICALASSESSMENT AND COMPRESSIONULTRASONOGRAPHY JOURNALINGSYSTEM FOR DEEP VEIN THROMBOSISUSING SUPPORT VECTOR MACHINES

DANIEL ÖBERG

KTH ROYAL INSTITUTE OF TECHNOLOGY

SCHOOL OF COMPUTER SCIENCE AND COMMUNICATION

Clinical Assessment for Deep Vein Thrombosisusing Support Vector Machines

A description of a clinical assessment and compression ultrasonography journaling

system for deep vein thrombosis using support vector machines

D. ÖBERG

Master’s Thesis at NADA

Supervisor: J. Lagergren

Examiner: J. Lagergren

Abstract

This master thesis describes a journaling system for com-pression ultrasonography and a clinical assessment systemfor deep vein thrombosis (DVT). We evaluate Support Vec-tor Machines (SVM) models with linear- and radial basisfunction-kernels for predicting deep vein thrombosis, andfor facilitating creation of new clinical DVT assessment.

Data from 159 patients where analysed, with our dataset,Wells Score with a high clinical probability have an accuracyof 58%, sensitivity 60% and specificity of 57% these figuredshould be compared to those of our base models accuracy of81%, sensitivity 66% and specificity 84%. A 23 percentagepoint increase in accuracy. The diagnostic odds ratio wentfrom 2.12 to 11.26. However a larger dataset is required toreport anything conclusive.

As our system is both a journaling and prediction system,every patient examined helps the accuracy of the assessment.

Referat

Klinisk bedömning av djup ventrombos

genom SVMs

I denna rapport beskrivs ett journalsystem samt ett systemför klinisk bedömning av djupvenstromboser. Vår modellbaserar sig på en stödvektormaskin (eng. Support VectorMachine) med linjär och radial basfunktion för att fastställaförekomsten av djupa ventromboser samt att hjälpa till iskapandet av nya modeller för bedömning.

159 patientjournaler användes för att fastställa att WellsScore har en klinisk precision på 58%, 60% sensitivitet ochspecificitet på 57% som kan jämföras med våran modell somhar en precision på 81%, 66% sensitivitet och specificitet på84%. En 23 procentenheters ökning i precision. Den diagnos-tiska oddskvoten gick från 2.12 till 11.26. Det behövs docken större datamängd för att rapportera något avgörande.

Då vårt system både är för journalskapande och kliniskbedömning så kommer varje undersökt patient att bidra tillhögre precision i modellen.

Contents

1 Introduction 1

2 Medical Background 3

3 Technical Background 73.1 Slack variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Di�erent error costs . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.3 Radial Basis Function . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Implementation 154.1 Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.2 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.4 Data format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5 Results 215.1 Test set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.2 Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.3 Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6 Discussion 296.1 Class Weight (Aka. How much is a life worth?) . . . . . . . . . . . . 296.2 What is a question worth? . . . . . . . . . . . . . . . . . . . . . . . . 29

7 Conclusion 33

8 Future Work 358.1 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358.2 Network Synchronisation & Security . . . . . . . . . . . . . . . . . . 358.3 Statistical Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358.4 Non Negative Matrix Factorisation . . . . . . . . . . . . . . . . . . . 36

9 Abbreviations 37

10 Appendix 39

Bibliography 45

Chapter 1

Introduction

Every year 1.6 per 1000 inhabitants su�er from venous thrombosis, blood cloths intheir veins (Nordström et al. 1992). Venous thrombosis formed in the deep veins aremost often found in the legs. Multiple ways to asses deep venous thrombosis existsbut diagnostics for this dangerous condition range from the accurate but expensivecontrast venography to the cheap but unreliable clinical assessment (Schumann andEwigman 2007).

With this in mind we can assess deep vein thrombosis that has a high specificity witha D-Dimer test which is a small protein fragment present in the blood after a bloodclot is degraded. High concentration of D-Dimer correlates with thrombosis, but falsepositive readings can occur from liver disease, high rheumatoid factor, inflammation,and many other factors (Kabrhel et al. 2010) which makes the sensitivity low.

Another way is the gold standard for DVT clinical assessment, named Wells Score,that was introduced with the paper “Accuracy of clinical assessment of deep-veinthrombosis”. This assessment is performed by a simple scoring system and a yes orno questionnaire regarding the patients medical history. Created by univariate, andstepwise logistic regression analysis of 529 patients’ clinical data. Wells Score is avery quick assessment as it only has 9 significant variables (Gao and Yang 2008).A combination of Wells Score and a D-dimer test reliably excludes DVT in adultswithout the need for imaging studies (Ho and others 2010) , such as compressionultrasonography (CUS), which can be both painful and expensive.

In the case that DVT cannot be reliably ruled out a compression ultrasonographycan be used to find a thrombus or the lack of such. In a compression ultrasonographythe examiner will start compressing the veins (usually beginning with the Femoralvein as far proximally as possible) and work distally towards the feet. Using aprobe containing transducers to send pulses of sound into the leg. When the soundis hitting a material with di�erent density, part of the sound wave is reflectedback to the probe. The compression is done to check for a reduced coe�cient ofcompressibility as the cloths are blocking the veins from compressing normally. The

1

CHAPTER 1. INTRODUCTION

examiner typically uses this to check for thrombosis each 2-5 cm noting the absenceor presence of occluded veins.

This process is often noted down on either paper or in a system which cannotintelligently be used for anything other than storage and reference. We hope thatwith our software both the assessment and the compression ultrasonography findingscould help train a new kind of improved assessment.

2

Chapter 2

Medical Background

Although this is a thesis in the area of Computer Science some knowledge of deepvein thrombosis is necessary.The most important thing to understand is what a deep vein thrombosis actually is.The blood’s unique ability to coagulate and therefore stop bleedings and start thehealing of wound is o� vital importance. Every day small wounds are healed in theblood vessel. In this process more than 50 di�erent substances is working together(D. Bergqvist et al. 2002) to make sure that the vessels are not coagulating in thevessels in other cases than when wounded. Some substances even has the role ofdissolution of coagulated blood.This balance between the stimulating and inhibitory is sensitive. Coagulated bloodcould produce blood clots, thrombosis, that leads to degraded circulation on the otherhand if the inhibitory substances outbalance the stimulating factors the result will bedangerous internal bleeding.Blood cloths are usually stuck in a vein in the calf of the leg or the femur in bigveins were the blood flow is slower. Often one side of the cloth is stuck to the veinwall were the other side is free and continuously built upon, sometimes measuringseveral decimeters long (D. Bergqvist et al. 2002).If part of a cloth breaks free and is allowed to travel to the lung , we call it apulmonary embolism . A serious condition that 1000 patients a year die from justin Sweden. These figures should be compared to the ~4000 who gets diagnosedwith pulmonary embolism or ~8000 patients who gets diagnosed with deep veinthrombosis.If we instead focus on US, a study the average yearly incidence of first lifetimevenous thromoboembolism among people between 1966 and 1990 was 117 per 100000 people, a similar study for that for Europe in general the rate is as high as 183per 100 000 people (Antovic and Blombäck 2010).The rate of occurrence might be even higher than that, one study concluded that,quote (Sandler and Martin 1989):

3

CHAPTER 2. MEDICAL BACKGROUND

Pulmonary embolism was thought to be the cause of death in 239 of2388 autopsies performed (10%): 15% of these patients were aged lessthan 60 years and 68% did not have cancer. Of these patients, 83% haddeep-vein thrombosis (DVT) in the legs at autopsy, of whom only 19%had symptoms of DVT before death. Only 3% of patients who had DVTat autopsy had undergone an investigation for such before death.

The Swedish hospitals costs of venous thromboembolism alone was estimated to0.375 billion SEK in 1999 (D. Bergqvist et al. 2002). Part of the high cost is due tothe di�culty to confirm diagnosis without either a thrombosonography, a D-dimertest or in rare cases intravenous venography; indeed some forms of DVT remainsclinically inapparent.

The decision to order thrombosonography is, in several guidelines, in large part donebased entirely on pretest risk assessment like Wells Score . The patients with lowrisk get D-dimer blood test and only go on to ultrasonography if the test is positive.The ones with high risk goes straight to ultrasonography without getting a D-dimertest.

Thrombosonography, the use of high frequency sound to visualise body tissue, ispreferred over other methods of proximal DVT assessment as it is a non-invasiveprocedure with very high sensitivity and specificity (96% and 98%, respectively)(Gaitini 2006). The main criteria for diagnosing DVT is to find a reduced coe�cientof compressibility. This is done by compressing the vein under observation andchecking for relatively low compressibility.

Large thrombus often become pronounced after a couple of days, but even a normalvein can produce an echo that looks similar to a thrombus.

As doing ultrasonograhies for every patient would be far to expensive and time-consuming a blood sample test, namely D-dimer, with a clinical assessment is toprefer.

Clinical assessment was long considered unreliable. P.S. Wells and his colleagueschallenged this dogma in 1995 with the publication of his now well known paper“Accuracy of clinical assessment of deep-vein thrombosis”.

The use of their clinical model is now standard practice in DVT diagnoses.

Table 2.1. Wells score (Bounameaux, Perrier, and Righini 2010)

Variable PointsCancer treatment during the past 6 months +1Lower leg paralysis or plastering +1Bed rest > 3 days or surgery < 4 weeks +1Pain on palpation of deep veins +1Swelling of entire leg +1

4

Variable PointsDiameter di�erence on a�ected calf > 3 cm +1Pitting oedema (a�ected side only) +1Dilated superficial veins (a�ected side) +1Alternative diagnosis at least as probable as DVT -2

A score is given from analysing the patients medical history were each criteria isincreasing the score by one, except if an alternative diagnosis is possible whichdecreases the score by two.

The old variant of Wells Score divided the probability into three classes Low,Intermediate and High.

Table 2.2. Clinical probability for Wells score

Low 0 totalIntermediate 1-2 totalHigh > 2 total

While a recent modification of the score only have two groups, namely likely orunlikely DVT (Le Gal, Carrier, and Rodger 2012).

5

Chapter 3

Technical Background

With machine learning algorithms we can figure out how to assess deep vein throm-bosis by generalising from examples. As more data would become available, thebetter the assessment would become. To be more specific, we are looking for a binaryclassification algorithm to improve on the modified Wells Score.In other words, estimate a function f : RN æ {±1} were RN represents the inputspace with N number of features/dimensions.We focus on generalisation when creating a classifier. A classifier that cannotgeneralise might still have a low training error and this does not imply a lowexpected test error.Given that we have training samples in the form of T = {(x1, y1), ..., (x

¸

, y¸

)} ™(X ◊ Y )¸ were the output domain is Y = {≠1, +1} and the input space X ™ RN .

f(x)I

yÕ {yÕ | ÷(xÕ, yÕ) œ T · xÕ = x} ”= ÿ≠1 otherwise

(3.1)

The classifier shown in equation 3.1 is an example of a function that does notgeneralise and therefore does not learn. One should note that generalisation is adouble edged sword and that finding the proper capacity is an research area itself inmachine learning.Support Vector Machines were developed by Cortes & Vapnik (Cortes and Vapnik1995) for classification. The basic gist is that classification is done by finding themaximal margin of separation between two classes (optimal hyperplane also knownas widest street), which can be seen as the generalisation, for linearly separablepatterns.With support vector classifiers the hyperplane can be described by the unknown, u,and the margin-vector, w (see figure 3.1):

w · u Ø C (3.2)

7

CHAPTER 3. TECHNICAL BACKGROUND

Figure 3.1. Optimal hyperplane

We are interested in which side the u vector is in so we project the u onto w and ifthis is bigger than some constant C then we say that u a positive sample.

By setting b = ≠C we get our decision rule:

(w · u) + b = 0, w œ RN , u œ RN , b œ R (3.3)

which corresponds to the decision function:

f(x) = sign((w · u) + b) (3.4)

The problem is that we do not know neither the w nor the b. However addingadditional constraints we can calculate them.

Taking a positive and negative sample and arbitrary setting it to bigger and smallerthan one respectively:

w · x+ Ø 1 (3.5)w · x≠ Æ 1 (3.6)

Then introducing a variable yi

that is +1 for positive samples and ≠1 for negativesamples we can combine these into:

yi

ú (xi

· w + b) Ø 1 (3.7)

8

3.1. SLACK VARIABLES

And with that we can add the extra constraint that:

yi

ú (xi

· w + b) ≠ 1 = 0 (3.8)

should be were xi

is on the margin.

Now if we want the widest margin possible we could take the di�erence of a negativeand positive sample on the margins and project it onto a unit normal.

WIDTH = (x+ ≠ x≠) · w

ÎwÎ (3.9)

Which can be simplified further to

(x+ ≠ x≠) · w

ÎwÎ = 2ÎwÎ (3.10)

The way one maximize this in the support vector machines algorithm is to useLagrange multipliers but to do that we need a constraint, which we have happenedto already have mentioned y

i

ú (xi

·w +b)≠1 = 0 . In this case it’s easier to minimize12 ú ÎwÎ2 than to maximize 2

ÎwÎ .

The Lagrange multiplier equation we try to optimize become:

L = 12 ú ÎwÎ2 ≠

ÿ

i

–i

(yi

ú (w · xi

+ b) ≠ 1) (3.11)

Which is something we can put into quadratic programming solvers to get the w, band –

i

parameters.

The result is one unique solution independent on whether we add non-support vectorpoints, as the Lagrange multipliers ( a

i

) becomes zero for these points.

But there is a problem with this solution. If we try it against our dataset we get anaccuracy of 20.75%! What is happening? Well the problem is that we do not have acleanly linearly separable dataset. Using the dataset we see that it has given up withfinding a solution and is classifing all of the input vectors into +1 (existing deepvein thrombosis) which gives a sensitivity of 100% and a specificity of 0%.

3.1 Slack variablesWhat we then need then is to relax the constraints to allow for a so called softmargin by using a slack variables, ›

i

:

yi

(w · xi

≠ b) Ø 1 ≠ ›i

, ›i

Ø 0 (3.12)

9


arg minw,x

i

,b

I12ÎwÎ2 + C

nÿ

i=1›

i

J

(3.13)

wereq

i

›i

is an upper bound on the number of training errors and C is a constantfor assigning higher penalty to errors.

With this we note a 26 percentage point increase in accuracy compared to ourbaseline (figure 3.2).

Figure 3.2. Linear SVM compared to Wells Score

Even though the accuracy is better, the sensitivity, which is arguably more importantin our case (the thrombosonography is relatively cheap), has a 24 percentage pointdecrease.

3.2 Di�erent error costsAs with most machine learning algorithms we always have the probability to trainthe model in a way to counteract an unbalanced dataset and tweak the sensitivityor specificity for a given purpose.

10

3.2. DIFFERENT ERROR COSTS

H0 is actually true H0 is actually falseWe conclude H0 is true correct conclusion Type II errorWe conclude H0 is false Type I error correct conclusion

H0 is called the null hypothesis, and H1 is called the alternative hypothesis.

Or put in another way:

H0 is actually true H0 is actually falseWe conclude H0 is true True Positives (TP) False Positives (FP)We conclude H0 is false False Negatives (FN) True Negatives (TN)

Higher sensitivity corresponds to increased likelihood of the SVM recommending thatthe doctor go through with a sonography. Doctors might strive for 100% sensitivitybut this impacts the accuracy negatively. Higher sensitivity often leads to lowerspecificity which if it would be 0% would just make every doctor recommending toalways take a sonography no matter what. This corresponds to zero Type 1 errors.

classification accuracy = TP + TN

TP + TN + FP + FN

Sensitivity: proportion of actual positives which are predicted positive

sensitivity = TP

TP + FN

Specificity: proportion of actual negative which are predicted negative

specificity = TN

TN + FP

So how do we get the SVM to prioritize a high sensitivity? Some researchers haveproposed to use di�erent penalty parameters to handle unbalanced data (Osuna,Freund, and Girosi 1997). We can show that this works very well even for balanceddata that needs a higher sensitivity.

11


If we go back to the formula we need to minimize:

yi

(w · xi

≠ b) Ø 1 ≠ ›i

, ›i

Ø 0 (3.14)

arg minw,b,›

i

I12ÎwÎ2 + C

nÿ

i=1›

i

J

(3.15)

and instead of C use di�erent misclassification costs for the positive- and negativeclass examples, C+ and C≠.

arg minw,b,›

i

I12ÎwÎ2 + C+

nÿ

i=1›

i

+ C≠

nÿ

i=1›

i

J

(3.16)

This goes by the name di�erent error costs (DEC). By assigning a higher misclassifi-cation costs for the positive samples (C+ Ø C≠) we get a higher sensitivity. In otherwords we skew the separating hyperplane towards the positive set.

With this we get a sensitivity of 84,90% when we change the misclassification penaltyratio to 0,9 & 0,1 which is a 24.2 percentage point increase while still having a 3.1increase in accuracy compared to our baseline.

3.3 Radial Basis FunctionBut there are still hurdles to overcome as the questions asked by the doctor might notbe statistically independent from another. For example cancer correlates heavily withage, and it is not a linear correlation (Ukraintseva and Yashin 2003). This rules outsimpler classification algorithms like Naïve Bayes but not Support Vector Machines.Even though we have made strides so far, what we have shown has only have beenable to account for linearly separable points. But with the so called kernel-trick ,were we map data into a richer feature space then construct a hyperplane in thatspace, we are able to classify points that were not linearly separable in its previousspace.

We call the function that maps from the vector x to another input space „(x).

By doing this simple transformation we know need to maximize:

K(x, y) = „(x) · „(y) (3.17)

But here we see that we do not really need „(x) on its own but can instead focus onK(x, y) which we call our kernel function.

By using the radial basis kernel (RBF):

12

3.3. RADIAL BASIS FUNCTION

K(x, y) = e≠“Îx≠yÎ2 (3.18)

were “ is a chosen constant, we get a great and fast nonlinear kernel.

With the similar accuracy as our baseline and linear svm we can get 100% sensitivitywith the RBF kernel.

Specifically we are using SVM with C-classification with an RBF-kernel because ofits good general performance and the few number of parameters (Meyer and Wien2014).

13

Chapter 4

Implementation

4.1 EnvironmentOne of the requirements was that the implementation should be able to run on AppleiOS platforms. This made it natural to develop the software in a combination ofC, C++ and Objective-C. For the machine learning we choose the OpenCV library,developed by Intel Russia research center in Nizhny Novgorod for realtime computervision. This library contains implementations for both RBF and linear kernels as itadopted the SVM/C++ library libsvm. E�ort has gone into making sure the SVMtheory presented is the same as the implementation. Note that details have beenglossed over, for example an explanation of Lagrange.

4.2 PreprocessingThe first step was preprocessing of data to use as a base for support vector machineswere access was given to 159 anonymous patients DVT journals. The journals werewithout identifiable information. From the journals we extracted the Wells scoreinformation and whether a DVT or occlusion were found. Note that from our pointof view the occlusion and a DVT is the same.

Table 4.1. Count of each label in dataset

DVT Nothing found33 126

As we can see we have a heavy bias in the nothing-found-category, even thoughwe are merging the DVT and occlusion columns. In a way this is good for us as itmeans our assessment system has actual value. It is also surprising as many of thepatients from the dataset has already gone through a Wells score assessment whichshould make the dataset have a heavy bias in the other direction, with very few in

15

CHAPTER 4. IMPLEMENTATION

the nothing found category. The binary yes or no questions was converted to 1.0and -1.0 respectively and stored in memory as a matrix.

4.3 ArchitectureAfter the preproccessing step the actual implementation began. We used the typicalModel-View-Controller concept and ended up with six controllers:

FeaturesStatistics Settings

PatientJounal

RiskAssessment

Sonography

One starts at the controller Patient Journal were one is required to write down, avalid, social security number of the patient, the family name and the examiner toproceed to the risk assessment. On the risk assessment the examiner is presentedwith questions fetched from a JSON-file. If no such file exists the JSON-file will becreated with the assessment questions corresponding to Wells Score.

4.4 Data formatOne of the goals of the assessment was not only to use the features of Wells Score butalso to be able to find new features that is better for assessing deep vein thrombosis.We would prefer to have doctors try and collaborate with di�erent clinical assessmentand share the data but we could not find any existing open format for sharing clinicalassessments and compression ultrasonographies therefore we had a need to createone from scratch.

The format is based upon A. Thurins DVT-journals and conforms to the JSON-spec(Bray 2014) with RFC 3339 (Klyne and Newman 2002).

Within this format the risk of name clashing between features needed to be addressed.For that purpose we assign a universally unique identifier (more commonly known

16

4.4. DATA FORMAT

as its abbreviation: UUID) to each feature, even the wells score features. A UUID issimply a 128-bit value commonly used in distributed systems to identify informationwithout significant central coordination. The probability of a feature id clash is withthe UUID:

p(n) ¥ 1 ≠ e≠ n

22x .

Were n is the number of features in our case.

With this we can train values with just sending a list of UUIDs and the system willfind the patients with this set of features and return a SVM. We hope that makingit easy to create a disjoint set of features will encourage experimentation.

The UUID must be represented by 32 uppercase hexadecimal digits, displayed in fivegroups separated by hyphens (e.g. B6C7A40E-6FA3-4C91-B31B-918C8776D474).

The JSON-keys are not optional and all the corresponding values must be non-empty.

JSON-Key JSON-Valuegroup String, name of a group the feature belongs tostandardValue Float or boolean of start valueriskAssesmentItemsModelId String, UUIDtimeCreated String, yyyy-MM-dd’T’HH:mm:ssZZZZZdescriptionText String, Long descriptionshortName String, Short description

JSON-Key Examplegroup “wells_score”standardValue falseriskAssesmentItemsModelId “52004621-75CB-422F-9FBE-EC0D77C3E4A8”timeCreated “2015-01-11T14:16:04+01:00”descriptionText “Paralysis or recent plaster cast”shortName “Paralysis”

The thing to note are:

• leftLeg & rightLeg contains examination points. The valid values for thechildnodes are anyone of the set:

– “T1”, Thrombosis found level 1 - Biggest– “T2”, Thrombosis found level 2– “T3”, Thrombosis found level 3 - Smallest– “Tr”, Thrombosis remnant– “x”, Removed thrombosis via surgery

17

CHAPTER 4. IMPLEMENTATION

– “#”, Missing– “?”, Not visible– “-”, Not surveyed– “0”, Normal

• riskAssessments children has the features with UUID as keys and floats orbooleans as values. The UUID keys are represented by 32 uppercase hexadec-imal digits, displayed in five groups separated by hyphens (e.g. B6C7A40E-6FA3-4C91-B31B-918C8776D474).

Key ValuepatientsSocialSecurityNumber String, Number without spaces and dashespatientsName StringexaminersName String, Person responsible for assessmentriskAssessment Node, See notessonographyExamination Nodeexaminer String, Person responsible for sonographyidCheck Boolean, Patients ID has been checkedadditionalInformation String, Sonography informationanamnesis Stringcomplications BooleannormalRepositoryVariance BooleanrightLeg Node, See notesleftLeg Node See notesv-fem-communis-inguen Stringfem-sup-dist Stringtibialis-post Strings-magna-prox-femur Stringiliaca-ext Stringpoplitea-prox Stringtibialis-ant Stringperonea Stringgastrocnemius Stringfem-com Strings-parva-prox Stringpoplitea-dist Stringsoleus Stringfem-profunda Stringv-fem-superficialis String

Key ExamplepatientsSocialSecurityNumber “9912290104”

18

4.4. DATA FORMAT

Key ExamplepatientsName “Svensson”examinersName “Andersson”riskAssessment —sonographyExamination —examiner “Andersson”idCheck trueadditionalInformation “Patient became sick”anamnesis —complications falsenormalRepositoryVariance —rightLeg —leftLeg —v-fem-communis-inguen “T1”fem-sup-dist “Tr”tibialis-post “T3”s-magna-prox-femur “Tr”iliaca-ext “T2”poplitea-prox “x”tibialis-ant “Tr”peronea “Tr”gastrocnemius “#”fem-com “Tr”s-parva-prox “?”poplitea-dist “?”soleus “—”fem-profunda “Tr”v-fem-superficialis “Tr”

19

Chapter 5

Results

5.1 Test setAccess was given to 159 patients DVT journals without identifiable information.The information was manually extracted. It contained the Wells score questions,answers and whether a DVT or occlusion were found.

Table 5.1. Count of each label in dataset

DVT Nothing found33 126

All SVMs were trained with optimized C and gamma values which is consideredoptimal when the cross-validation estimate of the test set error is minimal using 5folds, looking for C-values between 2≠5 and 215, gamma-values between 2≠15 and 23

. This should help against overfitting.

One should note that this data set does not say anything about the DVT rateamongst the general population as there is a heavy bias towards DVT as the patientsthat come to Klinisk Fysiologi to be examined in most cases already have beenexamined by doctors and is thought to have DVT.

5.2 BaselineAs a baseline we use Wells Score. Philip S. Wells et al. modeled Wells Score usingunivariate and stepwise logistic analysis (see Wells et al. 1997). This is the currentgold standard when it comes to DVT assessment.

Table 5.2. Wells score (Bounameaux, Perrier, and Righini 2010)

Variable PointsCancer treatment during the past 6 months +1

21

CHAPTER 5. RESULTS

Variable PointsLower leg paralysis or plastering +1Bed rest > 3 days or surgery < 4 weeks +1Pain on palpation of deep veins +1Swelling of entire leg +1Diameter di�erence on a�ected calf > 3 cm +1Pitting oedema (a�ected side only) +1Dilated superficial veins (a�ected side) +1Alternative diagnosis at least as probable as DVT -2

A score is given from analysing the patients medical history were each criteria isincreasing the score by one, except if an alternative diagnosis is possible whichdecreases the score by two.

The old variant of Wells Score divided the probability into three classes Low,Intermediate and High.

Table 5.3. Clinical probability for Wells score

Low 0 totalIntermediate 1-2 totalHigh > 2 total

5.3 BenchmarkingWe knew that SVM with RBF kernel was fast but we were interested in just how fasttraining and assesment could be done with our very modest dataset. We were out toprove that the SVM prediction with RBF kernel would be able to run a predictionon every single change of the assessment questionnaire.

The hardware used was an iPad Air 2 (model A1566). Our tests show that this isvery reasonable as the median of the time for prediction is in the sub millisecondrange with a median of 0.11 ms. Even the training of the SVM model has a medianof 1.79 ms. The benchmark was done with the Wells Score features and our existingdataset previously mentioned in this paper.

The following version of Clang was used

Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)Target: x86_64-apple-darwin14.0.0Thread model: posix

With the compiler directives (warning directives removed):

22

5.3. BENCHMARKING

clang -x objective-c -arch arm64 -fmessage-length=0-fdiagnostics-show-note-include-stack-fmacro-backtrace-limit=0 -std=c11 -fobjc-arc -fmodules-fmodules-prune-interval=86400-fmodules-prune-after=345600 -fpascal-strings -O0

With the training model params:

svm_type = CvSVM::C_SVC;kernel_type = CvSVM::RBF;gamma = 0.033750;C = 12.500000;class_weights = {0.167914, 0.832086};term_crit = cvTermCriteria(CV_TERMCRIT_ITER, 1000, 1e-6);

As the performance was more than enough with our case we never benchmarkedwith release flags (-O3 or -Os).

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40

time (ms)

0

5

10

15

20

25

dens

ity

DVT Prediction (SVM - RBF)µ = 0.12 , median = 0.11 , � = 0.04 , sample size = 364

Figure 5.1. Time for classification

23

CHAPTER 5. RESULTS

1 2 3 4 5 6

time (ms)

0.0

0.2

0.4

0.6

0.8

1.0

dens

ityDVT Training (SVM - RBF)

gaussian kernel density estimation, µ = 1.87 , median = 1.79 , � = 0.48 , sample size = 2261

Figure 5.2. Time for training

5.4 Results

In this section we report results obtained by applying support vector machines tothe patient data. The baseline is Wells Score with intermediate probability if nototherwise stated.

We started with a linear SVM and got a accuracy of 84.91% but it only had asensitivity of 36.36% so it became evident that we needed to approach the problemwith di�erent error costs (DEC) in mind.

For comparison we have included both Wells Score with intermediate and highprobability as defined by the table 5.3.

C-values are 312.50 for the linear SVM with DEC and SVM with 100% sensitivityboth and 2.5 for SVM RBF with class weight 0.81. “ -values are 0.50 for the SVMswith RBF kernel.

24

5.4. RESULTS

Figure 5.3. Linear SVM with soft margins and di�erent error costs compared toWells Score. C : 12.5

With our dataset the RBF kernel has a better performance than the linear kernelfigure 5.5.

In theory a SVM with a RBF kernel is going to out perform Wells Score with a gooddataset as long as we have nonlinear data. The ROC curve in figure 5.5 hints thatthis is indeed the case for our test set.

Class Weight Accuracy Sensitivity Specificity BCR DORSVM RBF 0.9226 58.49% 100.00% 35.71% 67.85% ŒWells Score - MEDIUM N/A 23.12% 97.05% 3.17% 50.11% 01.08SVM RBF 0.8196 81.11% 66.66% 84.92% 75.79% 11.26SVM Linear 0.9193 65.40% 63.63% 65.87% 64.75% 03.37Wells Score - HIGH N/A 58.49% 60.60% 57.93% 59.27% 02.11

25

CHAPTER 5. RESULTS

Figure 5.4. SVM with RBF kernel compared to Wells Score.

26

5.4. RESULTS

Figure 5.5. Comparsions of receiver operating characteristics (ROC) with varyingclass weights. C : 2.5; “ : 0.50625; 100 iterations, 0.01

27

Chapter 6

Discussion

6.1 Class Weight (Aka. How much is a life worth?)We have shown that we could counteract an unbalanced dataset and tweak thesensitivity or specificity by choosing the class weight. Wells Score on the other handhas three risk classes for DVT, low, intermediate and high. In our tests Wells Scorepatient data with a high risk just has a sensitivity of 60.60% with an accuracy of58.49%. As sensitivity rate is complementary to false negative rate it means that39.40 percent would be wrongly classified as not having deep vein thrombosis. Moresobering is looking at the medium risk class which has the false negative rate at2.95% but keep in mind that its specificity is only 3.17%, a 96.83% false positiverate. As seen, we can either choose high accuracy or high sensitivity. In one wetweak the class weight value to get an sensitivity value similar to Wells Score with ahigh risk and get 26.99 percentage point increase in specificity. The resulting classweight then is 0.9226/0.0774. If we instead maximise the sensitivity to be 100% wesee an 35.27 percentage point increase in accuracy compared to Wells Score withmedium risk.

6.2 What is a question worth?Taking a subset with only the more relevant features is called feature selection.Feature selection is important, in our case not for the performance of SVMs but forthe limited time of doctors, asking the patients thousands of questions would not befeasible. As the median of training with our dataset is 1.79 ms it opens up for thepossibility to train SVMs without a feature and check the di�erence of the balancederror rate (BER) which is the average of both the error rate of the positive classand the error rate of the negative class.

BER = FP/(TN + FP ) + FN/(FN + TP )2

29

CHAPTER 6. DISCUSSION

But only looking at the BER does not give us the whole picture. A feature is notworth much if very few people have had the symptoms. Because of this we show theamount of positive features to show how common they are in the dataset.

We also show the rate of deep vein thrombosis given that symptom, number ofconfirmed dvts/total number for the subset that had the symptom.

Balanced Error Rate (BER)

Dilated superficial veins (affected side)Cancer

Alternative diagnosis at least as probable as DVTBed rest > 3 days or surgery < 4 weeks

Pitting oedema (affected side only)Diameter difference on affected calf > 3 cm

Swelling of entire legPrevious DVT diagnostic

Pain on palpation of deep veinsParalysis or recent plaster cast

0,00 11,00 22,00 33,00 44,00

Count






0 35 70 105 140

Probability






0,00 12,50 25,00 37,50 50,00

From our data we can see that if you are previously diagnosed with DVT being themost important of all the features. Removing it will increase the balanced errorrate to 42% from 29%, a 13 percentage point increase. We can also see that themost common active features are: “Pain on palpation of deep veins” and “Diameter

30

6.2. WHAT IS A QUESTION WORTH?

di�erence on a�ected calf > 3 cm”

A previous DVT diagnostic is the strongest indicator of a future DVT diagnostic forboth the probability and the SVM while the SVMs next best indicator is a pittingoedema whilst for the independent probability is cancer.

31

Chapter 7

Conclusion

In this paper we have shown an implementation of thrombus reporting and how wecan do deep vein thrombosis (DVT) assessment using Support Vector Machines. Alsodescribed is an open JSON format for deep vein thrombus findings, based on AndersThurins work. We compared our assassement method for deep vein thrombosis withWells Score which is the current gold standard. The C and “ values used wherebased on cross-validation training and were able to get 100% specificity with 58%accuracy. Balanced Classification Rate for Well Score was at most 02.11 whilst ourhighest benchmarked was 11.26 . Using di�erent error costs (DEC) we can tweakthe sensitivity and specificity. As Wells Score is extremely simplistic it is easilyreplaced by even a linear SVM as our tests show. Our results although shows thatusing support vector machines with an RBF kernel is to prefer.

33

Chapter 8

Future Work

8.1 Regression

In this thesis we have been focused on checking the likelihood of a DVT using a pretestjust as Wells Score. A major di�erence is that Wells Score divides the probabilityinto three classes instead of two. As we are using (C-)Support Vector Classificationinstead of Support Vector Regression (with ‘- and ‹-Support Vector Regressionbeing popular) we become limited in its ability to check use the probability. Ofcourse one solution would be to train a SVM per model with di�erent class weightswhich the system allows for but this is a sub par solution.

Even so further research would be necessary to check if Support Vector Regressionwould be a better tool for the pretest.

8.2 Network Synchronisation & Security

As we had limited time no network synchronisation was implemented, but one couldimagine sharing between doctors form di�erent hospitals making a large Swedishdatabase of features and trained SVM models, specifically trained for the diet andsedentary life style living in Sweden brings.

8.3 Statistical Power

Our current dataset had a modest 159 patient records. It would be wise to automat-ically switch between di�erent feature sets, or warn the doctors when the dataset isnot enough for using for prediction, depending on the statistical power of the SVMmodel. As a comparison the number of patients of the original Wells Score paperevaluated 529 patients instead of our more modest set.

35

CHAPTER 8. FUTURE WORK

8.4 Non Negative Matrix FactorisationThe most common way to minimize the amount of features necessary is to usethe dimensionality reduction technique that factorizes a matrix into a product ofmatrices. This could be used on our feature set to reduce the amount of questionsthe doctors should ask. Simply check for redundant assessment questions.

36

Chapter 9

Abbreviations

CUS (Compression Ultrasonography)

A technique for diagnosing deep vein thrombosis in which ultrasonography is com-bined with venous compression of the deep veins.

DVT (Deep Vein Thrombosis)

The formation of a blood clot in a deep vein. Commonly a�ects the veins in the legor pelvis. The symptoms commonly include local pain, redness and swelling.

In 1992 it was reported that 1.6 per 1000 inhabitants had confirmed venous throm-bosis in Malmö (Nordström et al. 1992) per year and the number of deaths causedby DVT per year exceeds that of womens breast cancer in America (Hirsh and Hoak1996).

PE (Pulmonary embolism)

The obstruction of the main artery of the lung or one of its branches were theblockage must be caused by a substance that has travelled from elsewhere.

Typically this occurs when a blood clot forms and eventually manages to break free.

VTE (Venous Thromboembolism)

The formation of a blood clot in a vein. This makes it a collective term for deepvein thrombosis and pulminary embolism.

JSON (JavaScript Object Notation)

An open standard (T. Bray 2014) for encoding documents in a both human- andmachine-readable form.

In short it is a widely used metalanguage based on a subset of the JavaScriptProgramming Language, Standard ECMA-262 3rd Edition - December 1999.

37

CHAPTER 9. ABBREVIATIONS

SVM (Support Vector Machine)

Support Vector Machines were developed by Cortes & Vapnik (Cortes and Vapnik1995) for binary classification. The basic gist is that classification is done by findingthe optimal hyperplane for linearly seperable patterns, using a kernel function onnot linear patterns to map the dataset to a new space were it hopefully is.

38

Chapter 10

Appendix

[{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "52004621-75CB-422F-9FBE-EC0D77C3E4A8","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Cancer","shortName" : "Cancer"

},{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "A4E845EB-A40A-40CF-86D0-409F952F46CD","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Paralysis","shortName" : "Paralysis or recent plaster cast"

},{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "F785B94A-910A-4A88-92D8-0F0C9679FA10","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Bed rest or surgery","shortName" : "Bed rest > 3 days or surgery < 4 weeks"

},{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "273D54E1-E1EA-4AC2-8315-4D3AB9029D82","timeCreated" : "2015-01-11T14:16:04+01:00",

39

CHAPTER 10. APPENDIX

"descriptionText" : "Pain on palpation of deep veins","shortName" : "Pain on palpation of deep veins"

},{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "5CB3761B-1C68-4657-A4EA-6CE855E8D1F7","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Swelling","shortName" : "Swelling of entire leg"

},{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "FBEC8420-7520-4A57-BF9C-13FC0FB8D0B8","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Diameter difference","shortName" : "Diameter difference on affected calf > 3 cm"

},{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "3E15FF7D-2AA6-429E-BCFC-EE9D38D693E1","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Pitting oedema","shortName" : "Pitting oedema (affected side only)"

},{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "1F2F0426-CE21-4CFB-9AC6-6F45222134E7","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Dilated superficial veins","shortName" : "Dilated superficial veins (affected side)"

},{

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "19734A6C-89E7-4747-A3A2-35AB2D6DA914","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Previous DVT diagnostic","shortName" : "Previous DVT diagnostic"

},{

40

"group" : "wells_score","standardValue" : false,"riskAssesmentItemsModelId" : "87979A87-3510-4023-9ED2-1478B51503B6","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Alternative diagnosis","shortName" : "Alternative diagnosis at least as probable as DVT"

},{

"group" : "sahlgrenska_extension","standardValue" : false,"riskAssesmentItemsModelId" : "61EE0825-66A6-4A99-B583-99545F2BE05F","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Pregnancy","shortName" : "Pregnancy"

},{

"group" : "sahlgrenska_extension","standardValue" : false,"riskAssesmentItemsModelId" : "CCF86FC4-3EC2-4470-A88C-1750AC9CFA7A","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Malignity","shortName" : "Malignity"

},{

"group" : "sahlgrenska_extension","standardValue" : false,"riskAssesmentItemsModelId" : "B6C7A40E-6FA3-4C91-B31B-918C8776D474","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Fracture","shortName" : "Fracture"

},{

"group" : "sahlgrenska_extension","standardValue" : false,"riskAssesmentItemsModelId" : "3FB9FB95-1E22-4CCF-B73E-DAB0F3727AF2","timeCreated" : "2015-01-11T14:16:04+01:00","descriptionText" : "Surgery","shortName" : "Surgery"

},{

"group" : "sahlgrenska_extension","standardValue" : false,"riskAssesmentItemsModelId" : "3DDC6D15-E6CD-40B3-BD98-3C2AB2C44FF3","timeCreated" : "2015-01-11T14:16:04+01:00",

41


"descriptionText" : "Immobility","shortName" : "Immobility"

}]

42

{"patient" : {

"patientsSocialSecurityNumber" : "9912290104","patientsName" : "Svensson","examinersName" : "Andersson"

},"riskassessment" : {

"B6C7A40E-6FA3-4C91-B31B-918C8776D474" : false,"F785B94A-910A-4A88-92D8-0F0C9679FA10" : true,"3FB9FB95-1E22-4CCF-B73E-DAB0F3727AF2" : false,"1F2F0426-CE21-4CFB-9AC6-6F45222134E7" : false,"87979A87-3510-4023-9ED2-1478B51503B6" : false,"FBEC8420-7520-4A57-BF9C-13FC0FB8D0B8" : true,"19734A6C-89E7-4747-A3A2-35AB2D6DA914" : false,"A4E845EB-A40A-40CF-86D0-409F952F46CD" : false,"52004621-75CB-422F-9FBE-EC0D77C3E4A8" : false,"CCF86FC4-3EC2-4470-A88C-1750AC9CFA7A" : false,"3DDC6D15-E6CD-40B3-BD98-3C2AB2C44FF3" : false,"3E15FF7D-2AA6-429E-BCFC-EE9D38D693E1" : true,"273D54E1-E1EA-4AC2-8315-4D3AB9029D82" : true,"61EE0825-66A6-4A99-B583-99545F2BE05F" : false,"5CB3761B-1C68-4657-A4EA-6CE855E8D1F7" : true

},"sonographyExamination" : {

"examiner" : "Andersson","idCheck" : true,"additionalInformation" : "This is a fake patient","leftLeg" : {

"v-fem-communis-inguen" : "-","fem-sup-dist" : "-","tibialis-post" : "-","s-magna-prox-femur" : "-","iliaca-ext" : "Tr","poplitea-prox" : "-","tibialis-ant" : "-","peronea" : "-","gastrocnemius" : "-","fem-sup-prox" : "Tr","fem-com" : "0","s-parva-prox" : "-","poplitea-dist" : "-","v-fem-poplitea" : "-","soleus" : "-","fem-profunda" : "-",

43


"v-fem-superficialis" : "-"},"anamnesis" : "","complications" : false,"normalRepositoryVariance" : false,"rightLeg" : {

"v-fem-communis-inguen" : "-","fem-sup-dist" : "-","tibialis-post" : "-","s-magna-prox-femur" : "-","iliaca-ext" : "-","poplitea-prox" : "-","tibialis-ant" : "-","peronea" : "-","gastrocnemius" : "-","fem-sup-prox" : "-","fem-com" : "-","s-parva-prox" : "-","poplitea-dist" : "-","v-fem-poplitea" : "-","soleus" : "-","fem-profunda" : "-","v-fem-superficialis" : "-"

}}

}

44

Bibliography

Antovic, Jovan P, and Margareta Blombäck. 2010. Essential Guide to BloodCoagulation. Wiley Online Library.

Bergqvist, D, P Blomqvist, M Eliasson, H Eriksson, M Hellgren-Wångdahl, LarsHolmberg, J Janson, et al. 2002. “Blodpropp-Förebyggande, Diagnostik OchBehandling Av Venös Tromboembolism. En Systematisk Kunskapssammanställning.”Linköpings universitet.

Bounameaux, H., A. Perrier, and M. Righini. 2010. “Diagnosis of Venous Throm-boembolism: An Update.” Vascular Medicine 15 (5). SAGE Publications: 399.

Bray, Tim. 2014. “The JavaScript Object Notation (JSON) Data InterchangeFormat.”

Cortes, Corinna, and Vladimir Vapnik. 1995. “Support-Vector Networks.” MachineLearning 20 (3). Springer: 273–97.

Gaitini, D. 2006. “Current Approaches and Controversial Issues in the Diagnosisof Deep Vein Thrombosis via Duplex Doppler Ultrasound.” Journal of ClinicalUltrasound 34 (6). Wiley Online Library: 289–97.

Gao, Bao-An, and Jun Yang. 2008. “Wells Score for Venous Thromboembolism.Basic Diagnostic Algorithm for Venous Thromboembolism.” Saudi Medical Journal29 (9): 1229–34.

Hirsh, J., and J. Hoak. 1996. “Management of Deep Vein Thrombosis and PulmonaryEmbolism: A Statement for Healthcare Professionals from the Council on Thrombosis(in Consultation with the Council on Cardiovascular Radiology), American HeartAssociation.” Circulation 93 (12). Am Heart Assoc: 2212–45.

Ho, Wai Khoon, and others. 2010. “Deep Vein Thrombosis: Risks and Diagnosis.”Australian Family Physician 39 (7): 468.

Kabrhel, Christopher, D Mark Courtney, Carlos A Camargo, Michael C Plewa,Kristen E Nordenholz, Christopher L Moore, Peter B Richman, Howard A Smithline,Daren M Beam, and Je�rey A Kline. 2010. “Factors Associated With Positive D-Dimer Results in Patients Evaluated for Pulmonary Embolism.” Academic EmergencyMedicine 17 (6). Wiley Online Library: 589–97.

45


Klyne, Graham, and Chris Newman. 2002. “Date and Time on the Internet:Timestamps.” RFC 3339, July.

Le Gal, Grégoire, Marc Carrier, and Marc Rodger. 2012. “Clinical Decision Rulesin Venous Thromboembolism.” Best Practice & Research Clinical Haematology 25(3). Elsevier: 303–17.

Meyer, David, and FH Technikum Wien. 2014. “Support Vector Machines.” TheInterface to Libsvm in Package E1071.

Nordström, M., B. Lindblad, D. Bergqvist, and T. Kjellström. 1992. “A ProspectiveStudy of the Incidence of Deep-Vein Thrombosis Within a Defined Urban Population.”Journal of Internal Medicine 232 (2). Wiley Online Library: 155–60.

Osuna, Edgar, Robert Freund, and Federico Girosi. 1997. “Training SupportVector Machines: An Application to Face Detection.” In Computer Vision andPattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conferenceon, 130–36. IEEE.

Sandler, DA, and JF Martin. 1989. “Autopsy Proven Pulmonary Embolism inHospital Patients: Are We Detecting Enough Deep Vein Thrombosis?” Journal ofthe Royal Society of Medicine 82 (4). Royal Society of Medicine Press: 203.

Schumann, Sarah-Anne, and Bernard Ewigman. 2007. “Is It DVT? Wells Scoreand D-Dimer May Avert Costly Workup.” The Journal of Family Practice 56 (12).Quadrant HealthCom Inc.: 1010.

T. Bray, Ed. 2014. “RFC 7159 - The JavaScript Object Notation (JSON) DataInterchange Format.”

Ukraintseva, Svetlana V, and Anatoli I Yashin. 2003. “Individual Aging and CancerRisk: How Are They Related.” Demographic Research 9 (8). Max Planck Institutefor Demographic Research, Rostock, Germany: 163–96.

Wells, Philip S, David R Anderson, Janis Bormanis, Fred Guy, Michael Mitchell, LisaGray, Cathy Clement, K Sue Robinson, and Bernard Lewandowski. 1997. “Value ofAssessment of Pretest Probability of Deep-Vein Thrombosis in Clinical Management.”The Lancet 350 (9094). Elsevier: 1795–98.

46

www.kth.se

Clinical Assessment for Deep Vein Thrombosis using Support...

Documents

Transcript of Clinical Assessment for Deep Vein Thrombosis using Support...