MENTIS - Microscope Embedded Neurosurgical Training and ...

MENTIS - Microscope Embedded Neurosurgical Training and Intraoperative System

Microscope Embedded Neurosurgical

Training and Intraoperative System

M. Eng. Alessandro De Mauro Institute for Process Control and Robotics

Karlsruhe Institute of Technology

Microscope Embedded Neurosurgical

Training and Intraoperative System

Zur Erlangung des akademischen Grades eines

Doktors der Ingenieurwissenschaften von der Fakultät für Informatik

der Universität Fridericiana zu Karlsruhe (TH) - Karlsruhe Institute of Technology

Deutschland

genehmigte

Dissertation

von

Alessandro De Mauro aus Lecce

Tag der mündlichen Prüfung: 23.10.2009

Erster Gutachter: Prof. Dr.-Ing. Heinz Wörn

Zweiter Gutachter: Prof. Dr. med. Christian Rainer Wirtz


© Copyright 2010

by

Alessandro De Mauro

All Rights Reserved

ACKNOWLEDGMENT

I would like to thank my first advisor Prof. H. Wörn and my co-advisor Prof. Dr. Christian Rainer Wirtz for their

guidance and support professionally in the technical and medical part of this work. I sincerely appreciate them

for giving me the great opportunity to learn so much while working at the Institute for Process Control and

Robotics of the University of Karlsruhe (TH) and in the Neurosurgical Department of the University Hospital of

Heidelberg. I would also like to express my gratitude to Dr. Jörg Raczkowsky, who has always been enthusiastic

to help and advise me with professional, personal suggestions and motivating ideas. I wish to extend my

gratefulness to Dr. Marc-Eric Halatsch for the support in developing and testing the final prototype, Prof. Dr.

Rüdiger Marmulla, Dr. Robert Boesecke, Dr. Georg Eggers for giving me technical feedback and thoughts from

different perspectives during the periodically project meetings.

I would like to thank the Director of the Surgical Planning Laboratory (SPL) of the Department of Radiology,

Brigham and Women's Hospital and Harvard Medical School, Boston Dr. Kikinis for showing interest in this

project.

Special thanks to the associate Professor of Radiology at Harvard Medical School Prof. Hata for the hospitality,

technical support and suggestions during the programming and useful period spent to develop the 3DSlicer

module in Boston.

Thanks of course to my colleagues of the Medical Group (MeGI), Matteo, Matthias, Daniel, Markus, Holger,

Lüder, Jessica, Mathew, Christophe, Oliver. Special thanks to Dr. Michael Aschke to review my ideas, giving

suggestion and teaching me about augmented reality in medicine: his scientific contribute and personal advices

were very important for this work.

I wish to thank all the Heidelberg people: Vitor, Gavin, Monica, Raskie and Horia.

Finally, I would like to thanks all the colleagues that support me continuously and with kind patience during my

work life at the institute: Elke, Nina, Friederike for the burocratic activities and Margit for the technical part.

Thanks to all my friends in Italy and in Germany.

Special thanks to Miss Lenka D. who gave to me, with an immense love an infinite energy to proceed.

Last but not least, thanks to my family who I truly miss and love, for their encouragement during all these years

in Germany: this thesis is for you.

This work has been funded by the Marie Curie action (FP7) and it is part of the CompuSurge Project.

http://www.precisis.de/contact.asp

KURZFASSUNG

In den letzten Jahren hielten in der Neurochirurgie zahlreiche neue Technologien Einzug. Die

Computerassistierte Chirurgie (CAS) verringert die Patientenrisiken spürbar aber gleichzeitig besteht der Bedarf

für minimalinvasive bzw minimaltraumatische Techniken, da falsche Bewegungen des Chirurgen gefährliche

Auswirkungen haben können und im schlimmsten Falle zum Tode des Patienten führen. Die Präzision eines

chirurgischen Eingriffs hängt einerseits von der Genauigkeit der verwendeten Instrumente, andererseits auch

von der Erfahrung des Arztes ab. Aus diesem Grund ist das Training der Eingriffe besonders wichtig. Unter

technologischen Aspekten verspricht der Einsatz von Virtueller Realität (VR) und Erweiterter Realität (ER/AR)

für die intraoperative Unterstützung die besten Ergebnisse.

Traditionelle chirurgische Trainingsprogramme beinhalten Übungen an Tieren, Phantomen und Kadavern.

Problematisch an diesem Ansatz ist die Tatsache, daß lebendiges Gewebe andere Eigenschaften aufweist als

totes Gewebe und die Anatomie von Tieren signifikant vom Menschen abweicht. Auf der medizinischen Seite

sind niedriggradige Gliome (engl. Low-Grade Gliomas, LGG) intrinsische Hirntumore, die typischerweise bei

jungen Erwachsenen auftreten. Eine Behandlung zielt darauf ab, so viel Tumorgewebe wie möglich zu

entfernen und dabei den Schaden zum Gehirn zu minimieren. Beim Blick durch ein neurochirurgisches

Mikroskop ähnelt das pathologische dem gesunden Gewebe sehr stark, durch Abtasten kann es aber von einem

erfahrenen Chirurgen wesentlich zuverlässiger identifiziert werden.

Im ersten Teil dieser Doktorarbeit wird ein System zur visuellen und haptischen Simulation der Palpatierung der

Spatula von niedriggradigen Gliomen beschrieben. Dies stellt den bisher ersten Ansatz dar, ein Trainingssystem

für die Neurochirurgie zu entwickeln, das auf virtueller Realität, Haptik und einem echten Operationsmikroskop

basiert.

Die vorgestellte Architektur kann aber auch für intraoperative Anwendungen angepasst werden. Beispielsweise

kann das System für die Bildgestützte Therapie (engl. Image Guided Therapy, IGT) eingesetzt werden:

Mikroskop, Bildschirme und navigierte chirurgische Werkzeuge. Dieselbe virtuelle Umgebung kann als

Erweiterte Realität in die Optik des Operationsmikroskops eingekoppelt werden. Das Ziel ist es, die

intraoperative Orientierung des Chirurgen durch eine dreidimensionale Sicht und zusätzliche Informationen,

die er zur sicheren Navigation im Inneren des Patienten benötigt, zu verbessern.

Der zweite Teil dieser Arbeit ist dieser intraoperativen Anwendung gewidmet. Das Ziel ist die Verbesserung des

Prototypen eines stereoskopischen Mikroskops mit Erweiterter Realität für neurochirurgische Eingriffe, der am

gleichen Institut in den vergangenen Jahren entwickelt worden ist. Es wurde eine völlig neue Software unter

Beibehaltung der vorhandenen Hardware entwickelt und sowohl die Darstellungsperformanz als auch die

Usability gesteigert.

Da sich Erweiterte und Virtuelle Realität die gleiche Plattform teilen, wird das System als Mixed Reality System

für die Neurochirurgie bezeichnet.

Alle Kompononten sind “open source” oder zumindest GPL lizensiert.

ABSTRACT

In the recent years, neurosurgery has been strongly influenced by new technologies. Computer Aided Surgery

(CAS) offers several benefits for patients’ safety but fine techniques targeted to obtain minimally invasive and

traumatic treatments are required, since intra-operative false movements can be devastating, resulting in

patients deaths. The precision of the surgical gesture is related both to accuracy of the available technological

instruments and surgeon’s experience. In this frame, medical training is particularly important. From a

technological point of view, the use of Virtual Reality (VR) for surgeon training and Augmented Reality (AR) for

intra-operative treatments offer the best results.

In addition, traditional techniques for training in surgery include the use of animals, phantoms and cadavers.

The main limitation of these approaches is that live tissue has different properties from dead tissue and that

animal anatomy is significantly different from the human. From the medical point of view, Low-Grade Gliomas

(LGGs) are intrinsic brain tumours that typically occur in younger adults. The objective of related treatment is

to remove as much of the tumour as possible while minimizing damage to the healthy brain. Pathological tissue

may closely resemble normal brain parenchyma when looked at through the neurosurgical microscope. The

tactile appreciation of the different consistency of the tumour compared to normal brain requires considerable

experience on the part of the neurosurgeon and it is a vital point.

The first part of this PhD thesis presents a system for realistic simulation (visual and haptic) of the spatula

palpation of the LGG. This is the first prototype of a training system using VR, haptics and a real microscope for

neurosurgery.

This architecture can be also adapted for intra-operative purposes. In this instance, a surgeon needs the basic

setup for the Image Guided Therapy (IGT) interventions: microscope, monitors and navigated surgical

instruments. The same virtual environment can be AR rendered onto the microscope optics. The objective is to

enhance the surgeon’s ability for a better intra-operative orientation by giving him a three-dimensional view

and other information necessary for a safe navigation inside the patient.

The last considerations have served as motivation for the second part of this work which has been devoted to

improving a prototype of an AR stereoscopic microscope for neurosurgical interventions, developed in our

institute in a previous work. A completely new software has been developed in order to reuse the microscope

hardware, enhancing both rendering performances and usability.

Since both AR and VR share the same platform, the system can be referred to as Mixed Reality System for

neurosurgery.

All the components are open source or at least based on a GPL license.


Co

nte

nts

1

CONTENTS

CHAPTER 1 0

1.1 Training and Simulation in Medicine 1

1.2 Medical Motivations and Targets 4

1.2.1 Concept Extension to Mixed Reality System 6

CHAPTER 2 7

2.1 Introduction 8

2.2 Low Grade Glioma 8

2.3 Glioma Classification 9

2.4 LGG Treatment 10

2.4.1 Radiotherapy 10

2.4.2 Chemotherapy 11

2.4.3 Surgery 11

2.5 Image Guided Therapy and Interventions 14

2.6 Neurosurgical Workflow 15

2.7 Medical Imaging for LGG diagnosis and treatment 16

2.7.1 Computer Tomography 16

2.7.2 Magnetic resonance imaging 18

2.8 Neurosurgical Microscopy 20

2.9 Image Processing and 3D Patient Reconstruction 20

2.9.1 Segmentation 21

2.9.4 Delaunay Triangulation 25

2.10 Tracking Systems for neurosurgical navigation 28

2.10.1 Optical Tracking 28

2.10.2 Electromagnetic 29

2.11 Patient Tracking 30

2.12 Patient registration 31

2.12.1 Iterative Closest Point 31


Co

nte

nts

2

2.12.2 Patient Registration Error Analysis 32

CHAPTER 3 33

3.14 Main concepts and classification 34

3.15 Simulator Design 35

3.16 Visual Feedback 37

3.17 Haptic Feedback 37

3.18 Implementing a Simulator 37

3.19 Virtual and Augmented Reality concepts 39

CHAPTER 4 40

4.1 Haptic interfaces 41

4.2 State-of-the-art in surgical simulation with haptics 44

4.2.1 Surgicalscience: LapSim® System 44

4.2.2 Simbionix Simulators 46

4.2.3 KISMET (Kinematic Simulation, Monitoring and Off-Line Programming Environment for

Telerobotics). 48

4.3 Virtual training systems in Neurosurgery 49

4.3.1 Web-based Neurosurgical Training Tools 49

4.3.2 Virtual environment-based endoscopic third ventriculostomy simulator 49

4.3.3 Human Ventricle Puncture and Interaction between Spatula and Brain Tissue 50

CHAPTER 5 51

5.1 Introduction 52

5.2 Physical behaviours of soft tissues 52

5.3 Mass spring-damper model 53

5.3.1 Mathematical solution 55

5.4 Finite Elements Method 56

5.4.1 Brain Tissue Modelling using FEM 58

5.5 Models Comparison 59

5.6 Collision Detection 60

5.6.1 Introduction 60

5.6.2 Bounding Spheres 61


Co

nte

nts

3

5.6.3 Axis Aligned Bounding Boxes 61

5.6.4 Object Oriented Bounding Boxes 62

CHAPTER 6 63

6.1 System overview 64

6.2 Software Architecture 66

6.3 Scene graph API: H3D 66

6.4 3D MENTIS environment reconstruction 68

6.5 Physical modelling and haptics 69

6.6 Building a new physical model 71

6.7 Collision Detection 74

6.8 User Interface 75

6.9 Integration of 3DSlicer 76

6.10 Integration OpenIGTLink-TrackerBase 77

6.11 Web portability 79

6.12 Performances 80

6.13 Preliminary results in the validation of the simulation 82

CHAPTER 7 84

7.1 Introduction to the Intra-operative Augmented Reality 85

7.2 Patient Registration 87

7.2.1 Results 87

7.3 Microscope Camera Calibration 88

7.4 Single ocular calibration 90

7.4.1 Methods 90

7.4.2 Results 92

7.5 Stereo Camera calibration 96

7.5.1 Methods 96

7.5.2 Results 97

7.6 Augmented Reality Scenario Validation 98


Co

nte

nts

4

CHAPTER 8 99

8.1 Summary 100

8.2 Task 1: Microscope Embedded Neurosurgical Training System 100

8.3 Task 2: Extension of the platform to Augmented Reality Microscope for Intraoperative Navigation 101

8.4 Other general considerations 101

8.5 Disciplines 102

8.6 Future work 103

APPENDIX A 104

ICP 105

ICP related problems 105

PUBLICATIONS RELATED TO THIS PHD 107

BIBLIOGRAPHY 108

Tab

le o

f Fi

gure

s

0

TABLE OF FIGURES

FIGURE 1: THE ANATOMY LECTURE OF DR. NICOLAES TULP (REMBRANDT, 1632 OIL ON CANVAS, 169.5 X 216.5

CM, MAURITSHUIS, THE HAGUE) 1

FIGURE 2: TRADITIONAL LEARNING METHODS IN SURGERY 2

FIGURE 3: MEDICAL IMPACT: STANDARD TRAINING VS. VR TRAINING. 2

FIGURE 4: TRAINING SESSION WITH A VR SYSTEM (LAPSIM) 3

FIGURE 5: SIMULATOR CONCEPTS. 5

FIGURE 6: NOWADAYS THE TECHNOLOGY FOR TRAINING AND INTRAOPERATIVE PURPOSES IS COMPLETE

DIFFERENT. ON THE LEFT TYPICAL TRAINING SYSTEM, ON THE RIGHT MODERN OPERATING THEATER FOR

NEUROSURGERY 6

FIGURE 7: LOW GRADE GLIOMA APPEARENCE: A) T2 WEIGHTED MRI IMAGE OF LOW GRADE GLIOMA; B) GROSS

APPEARANCE OF TUMOR AFTER SURGICAL EXPOSURE: NOTE THE LACK OF DISTINCTION

BETWEEN NEOPLASTIC TISSUE AND NORMAL BRAIN TISSUE; C) DELINEATION OF TUMOR BY IMAGE

PROCESSING. 8

FIGURE 8: A. BRAIN NORMAL VIEW. B, C, D. MICROSCOPIC (NOT SURGICAL) VIEW OF THE TISSUE. BLACK

ARROWS SHOWS TUMOR TISSUE. 9

FIGURE 9: RADIOTHERAPY DIAGRAM (LEFT) AND PATIENT ON A LINEAR ACCELERATOR (RIGHT) 10

FIGURE 10: SCALP INCISION (LEFT) AND DRILL BURR HOLES IN SKULL (RIGHT). COURTESY OF PAUL D’URSO. 12

FIGURE 11: DURA IS CUT AND REFLECTED BACK EXPOSING THE BRAIN 12

FIGURE 12: A) DURA IS SUTURED, BONE FLAP REPLACED. B) INCISION CLOSURE 13

FIGURE 13: LEFT THE IGT OPERATING THEATRE OF THE FUTURE (COURTESY OF THE NATIONAL CENTER FOR

IMAGE GUIDED THERAPY - NCIGT). RIGHT: COMMERCIAL SOFTWARE AVAILABLE ON THE MARKET FOR IGT

(COURTESY OF BRAINLAB). 14

FIGURE 14: NEUROSURGERY WORKFLOW 15

FIGURE 15: TWO COMMON VIEWS FOR THE NEUROSURGEON DURING THE INTERVENTION: MONITOR VIEW

(UP) AND MICROSCOPE VIEW (DOWN). IN YELLOW, IS VISIBLE THE TWO-DIMENSIONAL CONTOUR OF THE

REGION OF INTEREST (USUALLY TUMOUR). 16

FIGURE 16: CT EXAMINATION OF THE HEAD. THE PATIENT'S HEAD IS POSITIONED CENTRALLY WITHIN THE

GANTRY OF THE CT SCANNER AS HE LIES ON HIS BACK, AND THE TABLE MOVES HORIZONTALLY AS THE

IMAGES ARE RAPIDLY OBTAINED. 17

FIGURE 17: A SINGLE SLICE FROM A NORMAL HEAD CT AT BRAIN WINDOW (LEFT) AND BONE WINDOW (RIGHT)

SETTINGS. ARROWS INDICATE TISSUES OF DIFFERENT DENSITY, INCLUDING WATER, FAT, SOFT TISSUE

AND BONE. 17

FIGURE 18: MRI TECHNIQUE [15] (LEFT). A PATIENT HEAD EXAMINATION WITH MRI. THE PATIENT'S HEAD IS

POSITIONED IN THE CENTER OF THE SCANNER TUBE AS HE LIES ON HIS BACK (RIGHT). 18

FIGURE 19: MRI OBTAINED IN A NORMAL VOLUNTEER, ALL ACQUIRED AT THE SAME LEVEL THROUGH THE

HEAD. PROTON DENSITY (PD, TOP LEFT), T1-WEIGHTED (T1, BOTTOM LEFT), T2-WEIGHTED (T2, TOP

RIGHT), AND MR ANGIOGRAPHY (MRA, BOTTOM RIGHT). 19

FIGURE 20: ORIGINAL SCHEME OF THE MICROSCOPE (COURTESY OF ZEISS) AND DETAILS ABOUT THE SURGICAL

MICROSCOPE ADOPTED IN THIS SCIENTIFIC WORK. 20

FIGURE 21: MARCHING CUBES ALGORITHMS IN 2D. THE FINAL RESULTING RED SURFACE IT'S A FAIRLY DECENT

REPRESENTATION OF THE CIRCLE. THIS SHAPE SUFFERS FROM SOME KIND OF SPATIAL ALIASING. 23

FIGURE 22: MARCHING CUBES AFTER SEVERAL REFINEMENTS. THE SURFACE IS APPROXIMATED BY PLANES

WHICH RESULT FROM THE CUBES TRIANGULATION. 24


Tab

le o

f Fi

gure

s

1

FIGURE 23: AN EXAMPLE DATA SET COVERING ALL OF THE 15 POSSIBLE COMBINATIONS. THE BLUE SPHERES

DENOTE CORNERS THAT HAVE TESTED AS INSIDE THE SHAPE AND THE GREEN ARROWS DENOTE THE

SURFACE NORMALS OF THE RELEVANT TRIANGLES. COURTESY OF [18] 24

FIGURE 24: ON THE LEFT, HUMAN BRAIN SURFACE RENDERED AFTER RECONSTRUCTION BY USING MARCHING

CUBES (128984 VERTICES AND 258004 TRIANGLES). ON THE RIGHT, MAGNIFIED DISPLAY OF BRAIN

SURFACE CONSTRUCTED BY USING MARCHING CUBES. 24

FIGURE 25: A DELAUNAY TRIANGULATION WITH CIRCUMCIRCLES 25

FIGURE 26: ON THE LEFT, THE DELAUNAY TRIANGULATION WITH ALL THE CIRCUMCIRCLES AND THEIR CENTERS

(IN RED). ON THE RIGHT, CONNECTING THE CENTERS OF THE CIRCUMCIRCLES PRODUCES THE VORONOI

DIAGRAM (IN RED). 26

FIGURE 27: THE DELAUNAY TRIANGULATION OF A RANDOM SET OF 100 POINTS IN A PLANE (LEFT). 3D BRAIN

FINALLY OVER IMPOSED TO REAL CT SCAN IMAGE (RIGHT). 27

FIGURE 28: NDI POLARIS TRACKED TOOLS: ACTIVE (A) AND PASSIVE (B) WITH RETROREFLECTING SPHERES (C),

INFRARED TRACKING SCHEME (D). 29

FIGURE 29: TRACKING VOLUME (LEFT) AND NDI VICRA AND SPECTRA SYSTEMS FOR MEDICAL TRACKING

(RIGHT). 29

FIGURE 30: COORDINATE TRANSFORMS INVOLVED IN THE INTRA-OPERATIVE TRACKING 30

FIGURE 31: SCHEME OF ALL THE DISCIPLINES INVOLVED IN A SURGICAL SIMULATOR. 34

FIGURE 32: SIMULATORS CLASSIFICATION 34

FIGURE 33: SIMULATION STEPS 35

FIGURE 34: 3D SCANNING FOR REVERSE ENGINEERING (FARO ARM) 36

FIGURE 35: DIAGRAM OF A COMMON IMPLEMENTATION OF HAPTIC RENDERING SPECIFIED FOR SURGICAL

SIMULATION 38

FIGURE 36: DEFINITION OF MIXED REALITY WITHIN THE CONTEXT OF THE RV CONTINUINUUM (MILGRAM AND

KISCHINO 1994) 39

FIGURE 37: EXAMPLE OF FORCE-FEEDBACK GLOVE WITH PNEUMATIC PISTONS TO SIMULATE GRASPING

(HUMAN-MACHINE INTERFACE LABORATORY OF RUTGERS UNIVERSITY) 41

FIGURE 39: TWO TYPES OF HAPTIC INTERFACES: OMNI (LEFT) AND PHANTOMTM

DESKTOP (RIGHT). COURTESY

SENSABLE TECHNOLOGIES 42

FIGURE 40: THE CYBERGRASP® (LEFT) AND CYBERTOUCH® (RIGHT) FROM IMMERSION. 43

FIGURE 41: SURGICALSCIENCE: LAPSIM SYSTEM 45

FIGURE 42: DIFFERENT TRAINING PLATFORMS AND SCENARIOS FROM SIMBIONIX 47

FIGURE 43: KISMET FROM FZK KARLSRUHE: PROTOTYPE (LEFT), 3D ENVIRONMENT (RIGHT) 48

FIGURE 44: WEB-BASED NEUROSURGICAL TRAINING TOOLS 49

FIGURE 45: VENTRICULAR ANATOMY FROM A SIMULATION CREATED VIEW FROM THE LATERAL VENTRICLE.

THE BASILAR ARTERY IS VISIBLE THROUGH THE SEMI-TRANSPARENT MEMBRANE AT THE FLOOR OF THE

THIRD VENTRICLE. 50

FIGURE 46: VIRTUAL BRAIN INTERACTION BETWEEN SPATULA AND BRAIN TISSUE (LEFT) AND HUMAN

VENTRICLE PUNCTURE (RIGHT) 50

FIGURE 47: MASS-SPRING TISSUE MODEL 53

FIGURE 48: MASS SPRING SYSTEM (IDEALLY WITHOUT ANY FRICTION). 54

FIGURE 49: THE IDEAL MASS-SPRING-DAMPER MODEL (LEFT). A MASS ATTACHED TO A SPRING AND DAMPER.

THE DAMPING COEFFICIENT, IS REPRESENTED BY D AND THE ELASTICITY IS K IN THIS CASE. THE F IN THE

DIAGRAM DENOTES AN EXTERNAL FORCE. ON THE RIGHT, SCHEMATIC REPRESENTATION OF KELVIN-

VOIGT MODEL IN WHICH E IS A MODULUS OF ELASTICITY AND Η IS THE VISCOSITY. 54

FIGURE 50: FAST COMPUTATION VS. ACCURACY 59

FIGURE 51: MODEL PARTITIONING OF A BRAIN (BOUNDING BOXES STRATEGY). 60


Tab

le o

f Fi

gure

s

2

FIGURE 52: SPHERE COLLISIOND DETECTION 61

FIGURE 53: AABBS STRATEGY 61

FIGURE 54: OOB STRATEGY. THE INITIAL BOUNDING BOX IS TIGHT FIT AROUND THE MODEL IN LOCAL

COORDINATE SPACE AND THEN TRANSLATED AND ROTATED WITH THE MODEL. 62

FIGURE 55: MENTIS ARCHITECTURE 64

FIGURE 56: SIMULATOR. LEFT: BRAIN TISSUE DEFORMATIONS. RIGHT: COMPLETE PROTOTYPE. 65

FIGURE 57: SOFTWARE ARCHITECTURE OF MENTIS 66

FIGURE 58: 3D ENVIRONMENT DEVELOPMENT STEPS 68

FIGURE 59: DIFFERENT MASS SPRING TOPOLOGIES FOR DIFFERENT LAYERS OF TISSUE. 69

FIGURE 60: WORKLOW OF HAPTIC RENDERING IN HAPI (FOLLOWING THE SENSEGRAPHICS SPEC.) 70

FIGURE 61: THREAD COMMUNICATION. NOTE THAT HAPTIC FREQUENCY IS HIGHER THEN GRAPHICS ONE. 70

FIGURE 62: HAPTIC SURFACE RENDERING AND DIFFERENT LAYERS. 71

FIGURE 63: STRUCTURE DEFORMATION FOR THE 3-LEVEL MASS-SPRING-DAMPER (LEFT) AND SURFACE

DEFORMATION APPEARANCE (RIGHT) 72

FIGURE 64: MODEL OF THE BRAIN EXTERNAL SURFACE 73

FIGURE 65: BRAIN DEFORMATIONS AFTER A COLLISION WITH A SURGICAL TOOL 73

FIGURE 66: COLLISION DETECTION. BOUNDING OF THE PATIENT HEAD WITH OBB (TOP LEFT) AND AABB (TOP

RIGHT), DETAILS ABOUT VENTRICLES BOUNDING WITH OBB (DOWN) 74

FIGURE 67: TWO DIFFERENT VISUAL RENDERINGS: MONITOR OUTPUT (LEFT) AND STEREOSCOPIC VIEW INSIDE

THE MICROSCOPE OCULARS. 75

FIGURE 68: 3DSLICER. TYPICAL SCENARIO: 3D RECONSTRUCTED SURFACE OF ORGANS OVER IMPOSED ON

MEDICAL IMAGES. 76

FIGURE 69: OPENIGTLINK 77

FIGURE 70: DATA COORDINATES FLOW 78

FIGURE 71: 3D VENTRICLES RENDERED IN THE SIMULATOR CAN BE VIEWED AND NAVIGATED IN A NORMAL

WEB BROWSER THANKS TO THE X3D STANDARD. THAT CAN PERMIT THE SHARING OF THE VIRTUAL

PATIENT FOR MEDICAL CONSULTING OR FOR DISTRIBUTED TRAINING. 79

FIGURE 72: VIRTUAL SCENARIO COMPOSITION 80

FIGURE 73: PERFORMANCES 80

FIGURE 74: VIRTUAL BRAIN-TOOL INTERACTION 81

FIGURE 75: MEDICAL EVALUATION IN GÜNZBURG AND ULM HOSPITALS 82

FIGURE 76: RESULTS. ENOUGH REALISM OF THE DEFORMATIONS OF THE BRAIN TISSUE AND OF THE LGG. 15

SURGEONS DEFINED THE SYSTEM REALISTIC ENOUGH TO BE USED FOR TRAINING. 83

FIGURE 77: SYSTEM ARCHITECTURE PROTOTYPE. 85

FIGURE 78: VIDEO FEEDBACK FOR THE SURGEON: MICROSCOPE AND SCREEN. 86

FIGURE 79: OCULAR VIEW: REAL VIEW (LEFT) AND AR VIEW (RIGHT). THE YELLOW LINE IS THE CONTOUR OF

THE CRANIOTOMY AREA ON THE PHANTOM SKULL. 86

FIGURE 80: ICP REGISTRATION APPLIED TO A 3D MODEL OF THE PATIENT (LEFT, BEFORE AND RIGHT AFTER THE

ALIGNMENT). 87

FIGURE 81: TWO CAMERAS ATTACHED ON MENTIS (LEFT) AND TRACKED PATTERN (RIGHT) 88

FIGURE 82: MICROSCOPE CALIBRATION SCHEME OF MENTIS 89

FIGURE 83: CALIBRATION IMAGES. SEVERAL IMAGES FOR DIFFERENT ANGLES AND POSITIONS WERE ACQUIRED

FOR EACH OF THE OCULARS. 90

FIGURE 84: ON THE LEFT CALIBRATION PATTERN AND ON THE RIGHT WITH THE DETECTED CORNERS (RED

CROSSES) AND THE REPROJECTED GRID CORNERS (CIRCLES) 90

FIGURE 85: ERROR ANALYSIS: REPROJECTION ERROR (IN PIXEL) FOR THE LEFT OCULAR (UP) AND RIGHT

(DOWN). 93


Tab

le o

f Fi

gure

s

3

FIGURE 86: DISTORSION. RADIAL (UP) AND TANGENTIAL (DOWN) FOR THE RIGHT OCULAR (SIMILAR RESULTS

FOR THE LEFT OCULARS). 94

FIGURE 87: EXTRINSIC PARAMETERS FROM LEFT OCULAR (SIMILAR RESULTS FOR THE RIGHT ONE). 95

FIGURE 88: EXTRINSIC PARAMETERS FOR THE STEREO CALIBRATION. THE TWO OCULARS POSITION ARE

SHOWN REFERRING TO THE CALIBRATION PATTERN. 96

FIGURE 89: DIFFERENT DISCIPLINES ARE INVOLVED IN THE MENTIS PROTOTYPE DEVELOPMENT 102

Ch

apte

r 1

0

CHAPTER 1

INTRODUCTION


Ch

apte

r 1

1

1.1 TRAINING AND SIMULATION IN MEDICINE

Medical training has been needed particularly in recent years in which neurosurgery has been deeply

influenced by new technologies. Fine techniques targeted to obtain treatments minimally invasive and

traumatic are required. Intra-operative false movements can be devastating, leaving patients paralyzed,

comatose or dead. The precision of the surgical gesture is related both to the experience of the surgeon and

accuracy of the available technological instruments. Computer Aided Surgery (CAS) can offer several benefits

for the patient safety.

FIGURE 1: THE ANATOMY LECTURE OF DR. NICOLAES TULP (REMBRANDT, 1632

OIL ON CANVAS, 169.5 X 216.5 CM, MAURITSHUIS, THE HAGUE)

In one of the most famous of his paints (Fig. 1), Rembrandt depicted how a typical learning procedure in

anatomy looked like in the past (1632). Anatomy lessons were a social event in the 17th century, taking place in

lecture rooms that were actual theatres, with students, colleagues and the general public being permitted to

attend. To the observer’s eyes it is immediately evident that cadavers were used. In addition, one of the

students is taking notes about the lesson and which leads us to the conclusion that books were the first

learning method together with the use of autopsy.

Traditional techniques for training in surgery (Fig.2) include the use of animals, phantoms and cadavers. The

main limitation of these approaches is that live tissue has different properties from dead tissue and also that

animal anatomy is significantly different from the human. In other words, they lead the realism of the surgical

trained procedures.


Ch

apte

r 1

2

FIGURE 2: TRADITIONAL LEARNING METHODS IN SURGERY

Nowadays, this classical training is improved by the use of well-illustrated books and excellent training movies

recorded directly in the operating theatre but the main training for surgeons is still performed on the real

patient. From 1998 simulation was validated by international community [1] and it was shown in [2] that virtual

reality simulators can speed-up the learning process and improve the proficiency of surgeons prior to

performing surgery on a real patient. A Youngblood et al [3] comparison between computer-simulation-based

training and traditional mechanical simulator training for basic laparoscopic skills that trainees who trained on

the computer-based simulator performed better on subsequent porcine surgery.

Fig. 3 shows the specific results obtained using a virtual training system with haptic interface for laparoscopic

surgical procedures.

FIGURE 3: MEDICAL IMPACT: STANDARD TRAINING VS. VR TRAINING.

In addition, exploration of the human organs by going inside can be used as a didactic and educational tool that

helps one to understand the interrelation of anatomical structures. It is possible to develop many different

virtual reality models of organs, in normal or diseased states, and dynamic interaction with these can show

their responses to externally applied forces provided by medical instruments.


Ch

apte

r 1

3

FIGURE 4: TRAINING SESSION WITH A VR SYSTEM (LAPSIM)


Ch

apte

r 1

4

1.2 MEDICAL MOTIVATIONS AND TARGETS

Low-grade gliomas are intrinsic brain tumours that typically occur in younger adults. The objective of related

surgery is to remove as much of the tumour as possible while minimizing the damage to the normal brain. One

of the obstacles associated with the surgical resection of these tumours is that the pathological tissue may

closely resemble normal brain, parenchyma when looked at through the neurosurgical microscope.

As a result, efforts to remove all tumour cells inevitably remove some of the normal brain and can leave behind

small sections of tumorous cells. The remaining glioma cells continue to grow, eventually causing additional

damage to the remaining normal brain, and a recurrence of symptoms.

Neuronavigation can help only partially because the brain-shift phenomena effects the pre-operative patient

data after craniotomy and tissue removal. The tactile appreciation of this difference, in consistency of the

tumour compared to normal brain, requires considerable experience on the part of the neurosurgeon.

A virtual reality based training system can be used to learn human anatomy and to try surgical movements. In

this way it is possible to obtain, by touching and moving the organs, an interactive navigation and to see how

the organs behave in relation to contact with a navigational instrument and the neighbouring organs.

The aspect of anatomy knowledge is particularly important because demands that surgeons be proficient not

only with the tools but also with the complex anatomy to be negotiated. For example, the development of a

sense of the anatomic relationships between neural and vascular structures encased by bone is critical to avoid

damage to the underlying nerves or arteries which maybe hidden. Another task that can be naturally achieved

using a detailed organ anatomy reconstruction is the correct patient positioning (one of the key elements for a

perfect outcome of the intervention). An incorrect position of the patient could hamper the execution of some

surgeon movements and can even produce lesions to delicate brain structures and/or obstruct a correct view

of the operative field provided by the microscope.

Since in neurosurgery surgical microscopes are regularly used, the 3D virtual environment should be entirely

implemented in the oculars of the microscope. These previous considerations were the main reasons to

develop a neurosurgical simulator directed towards both educational and preoperative purposes based on a

virtual environment set up on reconstructed human organs from real patients’ images. It is the very first

prototype of neurosurgical simulator embedded on a real operating microscope. Its main purposes are: the

realistic simulation (visual and haptic) of the spatula palpation of low-grade glioma and the stereoscopic

visualization in augmented reality of relevant 3D data for safe surgical movements in the image guided

interventions.

The system could be also used for future use in a collaborative virtual environment, allowing two users to

independently observe and manipulate a common model, allows one user to experience the movements and

the forces generated by the other’s contact with the bone surface. This enables an instructor to remotely

observe a trainee and provide real-time feedback and demonstration.

This work is the first example of neurosurgical training system using a real surgical microscope used in the

operating room. It is a task of the interdisciplinary project COMPU SURGE in close collaboration with the

University Hospital of Heidelberg (Department of Neurosurgery).

In Figure 5: Simulator concepts.


Ch

apte

r 1

5

FIGURE 5: SIMULATOR CONCEPTS.


Ch

apte

r 1

6

1.2.1 CONCEPT EXTENSION TO MIXED REALITY SYSTEM However, the simulation is only a preoperative task

One way to improve patient safety is to provide the surgeon with intra-operative navigation thus comparing in

real-time against pre-operative images. This three dimensional information is produced well in advance of

surgery in a normal radiological practice.

In neurosurgery this methodology, called Image Guided Surgery (IGS), offer the best results in conjunction with

the revolutionary introduction of Augmented Reality (AR).

As described later, microscope, surgical tools and patient data are commonly used in the OR during the image

guided operation. The hardware (microscope, tracking system, tools) and the software (navigation system

based on the patient dataset) are both involved in the training and intra-operative activities.

Nowadays from the technology point of view there are very big distances between simulator systems and

intraoperative instruments. This influences negatively the outcome of the training.

This research thesis proposes the use of the same tools and technology (microscope and tracking system) for

pre- and intra-operative. This ideal and natural continuum line starts on the training phase (VR) and finish

directly in the operating theatre (AR) during the operation.

This extension to AR application is based on the hardware built up in previous works at the Karlsruhe Institute

of Technology (see [4]) and on which complete novel software architecture for 3D stereoscopic augmented and

virtual reality have been set up.

FIGURE 6: NOWADAYS THE TECHNOLOGY FOR TRAINING AND INTRAOPERATIVE PURPOSES IS COMPLETE DIFFERENT. ON THE LEFT

TYPICAL TRAINING SYSTEM, ON THE RIGHT MODERN OPERATING THEATER FOR NEUROSURGERY (BRAINLAB)


Ch

apte

r 2

7

CHAPTER 2

MEDICAL BACKGROUND AND TECHNOLOGY


Ch

apte

r 2

8

2.1 INTRODUCTION

In order to have a better understanding of this research study it is crucial to have the medical background and

related technology. In this chapter a short introduction to the medical problems and tools it will be provided in

order to facilitate the understanding of all the choices that we adopted to develop this system.

2.2 LOW GRADE GLIOMA

Brain tumours are a diverse group of neoplasms arising from different cells within the central nervous system

(CNS) or from systemic tumours that have metastasized to the CNS. Brain tumours include a number of

histologic types with markedly different tumour growth rates.

Brain tumours can produce symptoms and signs by local brain invasion, compression of adjacent structures,

and increased intracranial pressure (IIP). In addition to the histology of the tumour, the clinical manifestations

are determined by the function of the involved areas of the brain. The proper evaluation of the patient with a

suspected brain tumour requires a detailed history, comprehensive neurological examination, and appropriate

diagnostic neuroimaging studies [5]. Gliomas comprise a group of primary central nervous system neoplasms

with characteristics of neuroglial cells that show different degrees of aggressiveness. The slower growing

lesions are commonly referred to as low-grade gliomas (LGGs). These have fewer aggressive characteristics and

therefore are more likely to grow slowly, as opposed to high-grade gliomas, which show more aggressive

features and are more likely to grow rapidly. The distinction between low-grade or high-grade is an important

one, since both the prognosis and the treatments are different.

Another important characteristic is the appearance: to the naked eye, as well in surgical microscope view,

tumours so closely resemble healthy brain tissue [6] that even the most experienced neurosurgeons may have

difficulties knowing if they have removed all possible traces of the abnormal growth.

As shown in the following figure, LGG is clearly visible on MRI but not on a post-craniotomy real surgical view.

At this instance, only the tactile experience of the surgeons can help in the identification process.

A B C

FIGURE 7: LOW GRADE GLIOMA APPEARENCE: A) T2 WEIGHTED MRI IMAGE OF LOW GRADE GLIOMA; B) GROSS APPEARANCE OF

TUMOR AFTER SURGICAL EXPOSURE: NOTE THE LACK OF DISTINCTION BETWEEN NEOPLASTIC TISSUE AND NORMAL BRAIN TISSUE; C)

DELINEATION OF TUMOR BY IMAGE PROCESSING.


Ch

apte

r 2

9

Probably the most important concept in the pathology of diffuse LGGs is simply that they are diffuse (Fig. 7).

Instead of forming a solid mass which destroys or displaces non-neoplastic parenchyma, they are infiltrative, ill-

defined tumours. It is precisely this infiltrative growth pattern that accounts for the major therapeutic

challenges and surgically incurable nature of diffuse gliomas. A lot of LGG patients have harboured

asymptomatic tumours for many years prior to clinical detection.

FIGURE 8: A. BRAIN NORMAL VIEW. B, C, D. MICROSCOPIC (NOT SURGICAL) VIEW OF THE TISSUE. BLACK ARROWS SHOWS TUMOR

TISSUE.

2.3 GLIOMA CLASSIFICATION

LGGs can be divided into several distinct entities based upon their histopathologic appearance. These

differences correlate with important differences in biologic behaviour and thus have important implications for

patient management.

The classification of gliomas is usually based upon the presumed cell of origin and the degree of malignancy.

Two systems are used:

Bailey and Cushing originally proposed that gliomas are originated from transformation of normal glial

cells during their development [7]. Astrocytomas are tumours with the appearance of astrocytes, while

oligodendrogliomas had the appearance of oligodendrocytes. Grading based upon histologic features

was not incorporated into this system. This system forms the foundation for the WHO classification

schema that remains in widespread use [8];

Kernohan estimated the prognosis of glial tumours based upon the extent of observed anaplastic

features (ie, mitoses, endothelial proliferation, cellular atypia, necrosis) [9].


Ch

apte

r 2

10

Although the term LGG is widely used, it is not explicitly defined in either system. LGG describes a spectrum of

primary brain tumours composed of cells that histologically resemble one or more different types of macroglial

cells (extended and fibrillary astrocytes, oligodendrocytes, ependymal cells) without evidence of anaplasia. In

the Kernohan scheme, LGGs encompass grade I and II tumors.

2.4 LGG TREATMENT

2.4.1 RADIOTHERAPY Radiotherapy is used in oncology to threat malignant tumours with ionizing radiation to control malignant cells

for curative or adjuvant cancer treatment. If the cure is not possible it is adopted as palliative for survival

benefits.

FIGURE 9: RADIOTHERAPY DIAGRAM (LEFT) AND PATIENT ON A LINEAR ACCELERATOR (RIGHT)

Radiotherapy (Figure 9: Radiotherapy diagram (left) and patient on a linear accelerator (right)) is also used after

surgery to destroy any remaining tumour cells in children older than 8-10 years of age and it is usually directed

locally to where the tumour is or was.

Practically, it works by directly damaging the DNA of cells. In fact, photon, electron, proton, neutron, or ion

beam directly or indirectly ionizing the atoms and consequently the DNA chain. The indirect way of ionization

happens as a result of the ionization of water, create free radicals which damage the DNA.

http://en.wikipedia.org/wiki/Ionizing_radiation

http://en.wikipedia.org/wiki/Malignant

http://en.wikipedia.org/wiki/Cell_%28biology%29

http://en.wikipedia.org/w/index.php?title=Curative&action=edit&redlink=1

http://en.wikipedia.org/wiki/Adjuvant_chemotherapy

http://en.wikipedia.org/wiki/DNA

http://en.wikipedia.org/wiki/Photon

http://en.wikipedia.org/wiki/Electron

http://en.wikipedia.org/wiki/Proton

http://en.wikipedia.org/wiki/Neutron

http://en.wikipedia.org/wiki/Ion

http://en.wikipedia.org/wiki/Ionizing

http://en.wikipedia.org/wiki/DNA

http://en.wikipedia.org/wiki/Ionization

http://en.wikipedia.org/wiki/Radical_%28chemistry%29


Ch

apte

r 2

11

2.4.2 CHEMOTHERAPY Chemotherapy, in its most general sense, refers to tumour treatment of disease using chemicals that kill cells,

both good and bad, with prevalence of the second group. For this reason it is a controversial technique [11].

Chemotherapy is generally used in conjunction with surgery and or radiotherapy to treat the tumour.

Treatment with anti-cancer drugs is used to destroy the tumour cells. It is usually given by injections and drips

into a vein (intravenous infusion). Chemotherapy is usually outpatient based and lasts over a year but is quite

well tolerated and children can usually continue to attend.

2.4.3 SURGERY Most patients will undergo initial surgery to confirm the diagnosis having as much of the tumour removal as

possible.

The objective of surgery is to remove as much of the tumour as possible while minimizing damage to the

normal brain. It is followed by observation with brain scans. If the tumour is deep in the brain, surgery may be

limited to a biopsy to confirm the diagnosis, in order to prevent further damage to the brain.

One of the obstacles associated with the surgical resection of these tumours is that the pathological tissue may

closely resemble normal brain parenchyma when looked at through the neurosurgical microscope. It means

that there is often no distinct boundary between the tumour and normal brain. As a result, efforts to remove

all tumour cells inevitably remove some of the normal brain and can leave behind tumour cells. The remaining

glioma cells continue to grow, eventually causing additional damage to the remaining normal brain, and a

recurrence of symptoms.

Due to intra-operative brain shift, neuronavigation is unreliable for guiding the extent of resection and a better

guide is through the slightly increased consistency of the tumour compared to normal brain tissue. In some

hospital open MRI is used and the LGG resection procedure is presented in [12]. Without Open MRI (very

expensive and rare resource) tactile sensations are the only suitable targets for the neurosurgeon. The

appreciation of this difference in consistency requires considerable experience on the part of the

neurosurgeon. The aim of our current development is to provide a training device for this particular task of

haptic tissue differentiation to neurosurgeons in training.

The neurosurgical common tasks for a tumour resection will be now described [13].

2.4.3.1 CRANIOTOMY

Craniotomy is the surgical opening of the cranium, the bones of the skull. Crani refers to the cranium (skull),

and otomy means to cut into. A craniotomy is performed to treat various brain problems surgically, such as

tumours, aneurysms, blood clots, head injuries, and abscesses.

The goal of all brain tumour surgery is to take out as much of the tumour as can safely be removed with as little

injury to the brain as possible. This may be especially complicated if the boundaries of the tumour cannot easily

be identified. For malignant brain tumours, radiation therapy and/or chemotherapy after surgery may be

recommended.

Successful recovery from craniotomy for brain tumour requires that the patient and his family approach the

operation and recovery period with confidence based on a thorough understanding of the process. The

surgeon has the training and expertise to remove all or part of the tumour, if its removal is possible; however,

http://en.wikipedia.org/wiki/Chemicals


Ch

apte

r 2

12

recovery may at times be limited by the extent of damage already caused by the tumour and by the brain's

ability to heal. If a neurological deficit remains, a period of rehabilitation will be necessary to maximize

improvement. This process requires that the patient and his family maintain a strong, positive attitude, set

realistic goals for improvement, and work steadily to accomplish each goal.

After a general anaesthetic has been given, the patient is positioned according to the area of the brain that

must be reached. In this type of operations usually the patient lying on his back. The incision area is clipped and

shaved.

FIGURE 10: SCALP INCISION (LEFT) AND DRILL BURR HOLES IN SKULL (RIGHT). COURTESY OF PAUL D’URSO.

A curved incision is made in the scalp over the appropriate location. The second step is to laid back the scalp

flap in order to expose the skull. It is possible drill burr holes in the skull with a power drill. A surgical saw is

used to connect the holes and create a "window" in the skull through which surgery will take place. The

removed bone piece is kept sterile for replacement at the end of the operation.

2.4.3.2 EXPOSURE OF THE BRAIN

After the dura is exposed, ultrasounds are used to confirm the location and depth of the underlying tumour.

This helps the surgeon to plan the best approach. Then the dura is cut with a scalpel or scissors and is laid back

to uncover the brain.

Figure 11: Dura is cut and reflected back exposing the brain


Ch

apte

r 2

13

2.4.3.3 REMOVAL OF THE TUMOUR

Special microsurgical instruments are used to carefully dissect the tumour from the brain tissue. Surgeons do a

small incision through the surface of the brain and into brain tissue until the tumour is reached. Neurosurgeon

endoscopic instruments may be used by to visualize, cut into, and remove the tumour, including a

neurosurgical microscope, a surgical laser that vaporizes the tumour, and an ultrasonic tissue aspirator that

breaks apart and suctions up the abnormal tissue.

There many part of the body, where some extra tissue around a tumour may be surgically removed to be sure

of the tumour totally resection: brain is not one of them. This means that only tissue that can clearly be

identified as abnormal may be removed from the brain-and even then only if its removal is possible without

devastating consequences. With meningioma and metastatic tumours, usually easy to distinguish from healthy

dura and brain tissue around them, the surgeon is more likely to be able to "get it all" than in the case of LGGs,

where the boundaries of the tumour are unclear and may be impossible to identify.

2.4.3.4 BONE REPLACEMENT

After the dura has been closed, the piece of bone is replaced and sutured into place. The incision is completely

closed and the operation finishes. Unless dissolving suture material is used, the skin sutures (stitches or staples)

have to be removed after the incision has healed.

FIGURE 12: A) DURA IS SUTURED, BONE FLAP REPLACED. B) INCISION CLOSURE


Ch

apte

r 2

14

2.5 IMAGE GUIDED THERAPY AND INTERVENTIONS

Surgeons need the best information available to support their movements and decisions. Lapse of time,

changes in position, natural motion, difficulties in sensing, have an high impact on the procedure (tissue

displacement and removal) and other factors, pre-procedural images and models are not often correct after a

procedure has begun. Tools that help the application of surgery and real-time image-based information to both

processes of diagnosis and therapy are vital.

Image Guided Therapy (IGT) or image-guided interventions (IGI) techniques lead to improve outcomes, patient

safety, shorter hospitalizations, quality and speed of surgical procedures. IGT technologies are emerging and

growing rapidly. They are in routine clinical use, published in well regarded and journals and several small

companies have successfully commercialized sophisticated IGT systems. These products provides a precise

ways to “visualize” intra-procedural anatomical changes in real time, clarifying surgeons’ understanding of

patients’ anatomy and enables minimally invasive surgery to be performed.

Decisions on the basis of accurate data instead of conjectures sometimes represent the crucial difference

between life and death especially in the LGGs case. For this reason nowadays, all the neurosurgical workflow is

based on IGT pre- and intra-operative software and tools.

FIGURE 13: LEFT THE IGT OPERATING THEATRE OF THE FUTURE (COURTESY OF THE NATIONAL CENTER FOR IMAGE GUIDED THERAPY -

NCIGT). RIGHT: COMMERCIAL SOFTWARE AVAILABLE ON THE MARKET FOR IGT (COURTESY OF BRAINLAB).


Ch

apte

r 2

15

2.6 NEUROSURGICAL WORKFLOW

Previously, we have introduced before the LGGs and intra-operative treatment. At this point, it is particularly

important to give a brief overview about all the steps involved before and during the neurosurgical procedures

(Fig. 30). The geometric models of the organs or the region of interest (e.g. tumour) are reconstructed from

data acquired by CT, MRI or other means by a radiologist. This is the preoperative phase. In the intra-operative

phase a tracking system is used to track the relative positions of the patient, relevant tools and microscope.

Detailed steps of image processing will be described in the future chapter.

All data are shown on the screen using the normal three views (coronal, axial, sagittal).

In the OR, surgeon’s eyes are typically on the microscope oculars but occasionally they need to see the screen

in order to understand the correct position compared to the preoperative images (CT, MRI). The position and

orientation of an active tool tracked by the infrared tracking system and its relative position in the patient

images are shown on the monitors (Fig.31). The two-dimensional contour of the region of interest is recognized

as defined by the radiologist in the preoperative step. This two-dimensional shape is visible inside the

commercial microscopes overlaid to the oculars views. Nowadays, the three-dimensional environment

reconstruction from two dimensions is another difficult and critical mental work for the surgeon.

FIGURE 14: NEUROSURGERY WORKFLOW


Ch

apte

r 2

16

FIGURE 15: TWO COMMON VIEWS FOR THE NEUROSURGEON DURING THE INTERVENTION: MONITOR VIEW (UP) AND MICROSCOPE

VIEW (DOWN). IN YELLOW, IS VISIBLE THE TWO-DIMENSIONAL CONTOUR OF THE REGION OF INTEREST (USUALLY TUMOUR).

2.7 MEDICAL IMAGING FOR LGG DIAGNOSIS AND TREATMENT

If a brain tumour is suspected, the physician will want to obtain an imaging scan of the brain. This is done using

either magnetic resonance imaging (MRI) or computed tomography (CT or CAT scan) [14]. A major difference

between an MRI and a CT scan is that the MRI uses a magnet to image the brain, while CT uses x-rays. Both can

be used for the tumour presence and localization because they are able to give a detailed image of the brain's

structure.

CT scans are the first test is done because of economic reasons: they are less expensive than MRI scans. On the

other hand, MRI provides much more useful information when a brain tumour is suspected, and may be

recommended after a tumour is confirmed.

2.7.1 COMPUTER TOMOGRAPHY The “tome” in tomography is the Greek word for slice. Computed tomography technique is based on the

measurement of the amount of energy that the head absorbs as a beam of radiation passes through it from a

source to a detector. Within a CT scanner, the radiation source and detector are mounted opposite each other

along a circular track, allowing them to rotate rapidly and synchronously around the table on which the patient


Ch

apte

r 2

17

lies. As the x-ray source and detector move around the patient’s head, measurements consisting of many

projections through the head are obtained at prescribed angles and stored on a computer. The table moves

horizontally in and out of the scanner in order to cover the entire head. At the core of the scanner there is a

computer that not only controls the radiation source, the rotation of the x-ray tube and detector, and the

movement of the table, but also generates anatomical slices, or tomograms, from the measured projections.

The mathematical technique that allows an image of the head to be recovered from its projections is referred

to as the back projection algorithm.

FIGURE 16: CT EXAMINATION OF THE HEAD. THE PATIENT'S HEAD IS POSITIONED CENTRALLY WITHIN THE GANTRY OF THE CT SCANNER

AS HE LIES ON HIS BACK, AND THE TABLE MOVES HORIZONTALLY AS THE IMAGES ARE RAPIDLY OBTAINED.

FIGURE 17: A SINGLE SLICE FROM A NORMAL HEAD CT AT BRAIN WINDOW (LEFT) AND BONE WINDOW (RIGHT) SETTINGS. ARROWS

INDICATE TISSUES OF DIFFERENT DENSITY, INCLUDING WATER, FAT, SOFT TISSUE AND BONE.


Ch

apte

r 2

18

2.7.2 MAGNETIC RESONANCE IMAGING Magnetic resonance imaging relies upon signals derived from water molecules (almost the 80% of the human

brain composition). This ubiquitous biological molecule has two protons, which by virtue of their positive

charge act as small magnets on a subatomic scale. Positioned within the large magnetic field of an MR scanner,

typically 30 to 60 thousand times stronger than the magnetic field of the earth, these microscopic magnets

collectively produce a tiny net magnetization that can be measured outside of the body and used to generate

very high-resolution images that reveal information about water molecules in the brain and their local

environment.

Protons placed in a magnetic field have the interesting property to absorb energy at specific frequencies, and

then re-emit the energy at the same frequency. In order to measure the net magnetization, a coil placed

around the head is used for both, generating electromagnetic waves and measuring the electromagnetic waves

that are emitted from the head in response. MRI uses electromagnetic waves in the same portion of the

electromagnetic spectrum as the broadcast FM radio (CT uses x-rays with very high frequency energy).

MRI is also a tomographic imaging modality since it produces two-dimensional images that consist of individual

slices of the brain. The table and the scanner do not move to cover different slices in the brain. Rather, images

can be obtained in any plane through the head by electronically guide the plane of the scan.. The switching on

and off of these magnetic field gradients is the source of the loud clicking and whirring noises commonly heard

during an MRI scan section. This process requires more time than CT scan but imaging can be performed

relatively quickly using modern gradient systems.

FIGURE 18: MRI TECHNIQUE [15] (LEFT). A PATIENT HEAD EXAMINATION WITH MRI. THE PATIENT'S HEAD IS POSITIONED IN THE CENTER

OF THE SCANNER TUBE AS HE LIES ON HIS BACK (RIGHT).

Image intensity in MRI is a consequence of several parameters. These are proton density, which is determined

by the relative concentration of water molecules, and T1, T2, and T2* relaxation, which reflect different

features of the local environment of individual protons. The degree to which these parameters contribute to

overall image intensity is controlled by the application and timing of radiofrequency energy through different

pulse sequences. The most commonly used pulse sequences in brain imaging preferentially emphasize T1

relaxation, T2 relaxation, T2* relaxation or proton density. Specialized pulse sequences can sensitize images of

flowing blood, minute changes in local brain oxygen content, or even to the microscopic movement of water


Ch

apte

r 2

19

molecules within the brain. Each sequence of pulse is characterized by a different contrast weighting to the

image, such that when combined, the aggregate intensities from the different pulse sequences allow inference

about the properties and local environment of the brain tissue being studied. For example, using MRI, one can

infer the phase (solid or liquid), content (fat, water, air, blood) or movement (i.e. pulsation) of a given structure

in the brain.

FIGURE 19: MRI OBTAINED IN A NORMAL VOLUNTEER, ALL ACQUIRED AT THE SAME LEVEL THROUGH THE HEAD. PROTON DENSITY (PD,

TOP LEFT), T1-WEIGHTED (T1, BOTTOM LEFT), T2-WEIGHTED (T2, TOP RIGHT), AND MR ANGIOGRAPHY (MRA, BOTTOM RIGHT).


Ch

apte

r 2

20

2.8 NEUROSURGICAL MICROSCOPY

The operation microscope was developed in order to see the very small details of the human body. Surgeons

make used of the operational microscope in delicate procedures that requires them to be perform the

operation with precision especially in the field of neurosurgery. The stereo microscope serves different

purposes. It uses two separate optical paths with two objectives and two eyepieces to provide slightly different

viewing angles to the left and right eyes. In this way, it produces a three-dimensional visualization of the

sample being examined.

FIGURE 20: ORIGINAL SCHEME OF THE MICROSCOPE (COURTESY OF ZEISS) AND DETAILS ABOUT THE SURGICAL MICROSCOPE ADOPTED

IN THIS SCIENTIFIC WORK.

2.9 IMAGE PROCESSING AND 3D PATIENT RECONSTRUCTION

The geometric model of the organs or the region of interest (example tumour) is reconstructed using real

patients’ data acquired by CT, MRI or fMRI for convenient visualisation. The classification phase is a user-driven

process.

For instance, some very good open source project offer accurate medical image processing and the possibility

to create a complete and detailed geometric human model starting by the real patient images (i.e. Osirix, 3D

Slicer). Obviously, many commercial products are available and used in the radiological pre-operative step. In

addition, the availability from 1995 of the Visible Human dataset provided by the National Library of Medicine

has allowed the creation of 3D human structures for academic use when no other patient images are available.

The segmentation and organ classification phases are carried out in order to obtain information about the size

and the shape of the human organs. In the following paragraphs the most common methods are presented.


Ch

apte

r 2

21

2.9.1 SEGMENTATION Segmentation of medical images (MRI, CT, US) is the technique of partitioning the data into contiguous regions

representing individual anatomical objects. It is a prerequisite for further investigations in many computer-

assisted medical applications, e.g. individual therapy planning and evaluation, diagnosis, simulation and image

guided surgery. For example, it is necessary to segment the brain in an MR image, before it can be rendered in

3D for visualization purposes. The result of image segmentation is a set of segments that collectively cover the

entire image, or a set of contours extracted from the image. Each of the pixels in a region is similar with respect

to some characteristic or computed property, such as colour, intensity, or texture. Adjacent regions are

significantly different with respect to the same characteristics.

This step is a routine in a radiologist preoperative work. Segmentation can be a very complex and difficult task

since it is often very hard to separate the object from the image background. This is due to the characteristics

of the imaging process as well as the grey-value mappings of the objects themselves.

However, the automatic delineation of structures (automatic segmentation) from medical images is still

considered an unsolved problem. Therefore a lot of human interaction is usually required for reconstructing

the human anatomy.

Because of the nature of the image acquisition process noise is inherent in all medical data. The resolution of

each acquisition device is limited. Moreover, inhomogeneities in the data might lead to undesired boundaries

within the object to be segmented, while homogenous regions might conceal true boundaries between organs.

In general, segmentation is an application specific task.

These problems can be solved by the radiologist (thanks to their anatomy knowledge) identifying region of

interests in the data due to their knowledge about typical shape and image data characteristics. Manual

segmentation however is a very time-consuming process for large 3D image stacks. After the segmentation

step a surface mesh generation and simplification is carried out.

The number of segmentation algorithms found in the literature is very high but, unfortunately, an automatic

methods for image segmentation of organs it is still not realized.

There are many good papers in the literature reviewing the various segmentation algorithms for medical

volumes [16]. They can be so classified:

1. structural techniques;

2. stochastic techniques;

3. hybrid techniques.

Structural techniques utilize information concerning the structure of the region during the segmentation.

Stochastic techniques are the ones that are applied on discrete voxels without any consideration for the

structure of the region. Finally, the hybrid methods include both the previous techniques present all their

characteristics.

2.9.2 REGION EXTRACTION

This is probably the simplest among the hybrid techniques. Region growing is a technique to extract a

connected region from a 3D volume based on some pre-defined connecting criterion. In the simplest form,


Ch

apte

r 2

22

region growing requires a seed point to start with and from this, the algorithm grows until the connecting

criterion is satisfied.

Here is the basic formulation for Region-Based Segmentation:

(b) Ri is a connected region, i = 1, 2, ..., n

(d) P(Ri) = TRUE for i = 1,2,...,n.

P(Ri) is a logical predicate defined over the points in set P(Rk) and is the null set.

In the previous equations:

(a) indicates that the segmentation must be complete; that is, every pixel must be in a region; (b) requires that

points in a region must be connected in some predefined sense;

(c) indicates that the regions must be disjoint;

(d) concerns the properties that must be satisfied by the pixels in a segmented region-for example Ri = TRUE if

all pixels in Ri have the same gray level and the condition;

(e) indicates that region Ri and Rj are different in the sense of predicate P.

The primary disadvantage of this algorithm is that it requires seed points which generally means manual

interaction. Thus for each region to be segmented, a seed point is needed. Region growing can also be sensitive

to noise and partial volume effect causing the extracted region to have holes or disconnections.

2.9.3 CONTOUR EXTRACTION

After segmentation, a three-dimensional scalar field (called voxels) is obtained. At this point a surface

representation of the human organs or region of interest is required. Marching cubes is a computer graphics

algorithm very useful for this purpose, published in the by Lorensen and Cline [17] for extracting a polygonal

mesh of an isosurface.

Marching Cubes algorithm needs some basic information to reconstruct the final surface especially about the

point’s locations (has to be evaluated if the point is inside or outside of the object).

The basic principle behind the marching cubes algorithm is to subdivide space into a series of small cubes. The

algorithm then instructs us to 'march' through each of the cubes testing the corner points and replacing the

cube with an appropriate set of polygons. The sum total of all polygons generated will be a surface that

approximates the one the data set describes.

To explain the algorithm let us first look at a 2 dimensional equivalent. The diagram bellow on the left shows a

grid of squares equivalent to the cubes from the 3d algorithm (many people refer to the cubes as cells or

voxels). A solid circle has been drawn which is the shape we are going to approximate using lines (instead of

polygons).

http://en.wikipedia.org/w/index.php?title=Region-Based_Segmentation&action=edit&redlink=1


Ch

apte

r 2

23

The first step is to calculate the corners that are inside the shape (represented by the green dots). At this step

some vertices can be introduced, since is it possible to know which points are inside and which are outside and,

as a consequence, is it even possible to guess that a vertex should be positioned approximately halfway

between an inside corner and any outside corners that are connected by the edge of a cell. The central diagram

bellow shows the discussed vertices as small red dots and the diagram on the right shows the matching surface

formed by joining the vertices with lines.

FIGURE 21: MARCHING CUBES ALGORITHMS IN 2D. THE FINAL RESULTING RED SURFACE IT'S A FAIRLY DECENT REPRESENTATION OF THE

CIRCLE. THIS SHAPE SUFFERS FROM SOME KIND OF SPATIAL ALIASING.

From one hand, the resulting surface has given a fairly decent representation of the circle, but the shape

suffers from some kind of spatial aliasing.

In 3D the algorithm proceeds through the scalar field, taking eight neighbour locations at a time (thus forming

an imaginary cube), then determining the polygon(s) needed to represent the part of the isosurface that passes

through this cube. The individual polygons are then fused into the desired surface. This is done by creating an

index to a pre-calculated array of 256 possible polygon configurations (28 = 256) within the cube, by treating

each of the 8 scalar values as a bit in an 8-bit integer. If the scalar's value is higher than the iso-value (because it

is inside the surface) then the appropriate bit is set to one, while if it is lower (outside), it is set to zero. The

final value after all 8 scalars are checked is the actual index to the polygon configuration array.

Finally, each vertex of the generated polygons (usually triangles) is placed on the appropriate position along the

cube's edge by linearly interpolating the two scalar values that are connected by that edge. The pre-calculated

array of 256 cube configurations can be obtained by reflections and symmetrical rotations of 15 unique cases.

The gradient of the scalar field at each grid point is also the normal vector of a hypothetical isosurface passing

from that point. Finally, is it possible interpolating these normals along the edges of each cube to find the

normals of the generated vertices. These are essential for shading the resulting mesh with some illumination

model.


Ch

apte

r 2

24

FIGURE 22: MARCHING CUBES AFTER SEVERAL REFINEMENTS. THE SURFACE IS APPROXIMATED BY PLANES WHICH RESULT FROM THE

CUBES TRIANGULATION.

FIGURE 23: AN EXAMPLE DATA SET COVERING ALL OF THE 15 POSSIBLE COMBINATIONS. THE BLUE SPHERES DENOTE CORNERS THAT

HAVE TESTED AS INSIDE THE SHAPE AND THE GREEN ARROWS DENOTE THE SURFACE NORMALS OF THE RELEVANT TRIANGLES.

COURTESY OF [18]

FIGURE 24: ON THE LEFT, HUMAN BRAIN SURFACE RENDERED AFTER RECONSTRUCTION BY USING MARCHING CUBES (128984 VERTICES

AND 258004 TRIANGLES). ON THE RIGHT, MAGNIFIED DISPLAY OF BRAIN SURFACE CONSTRUCTED BY USING MARCHING CUBES.


Ch

apte

r 2

25

2.9.4 DELAUNAY TRIANGULATION In mathematics, and computational geometry, a Delaunay triangulation for a set P of points in the plane is a

triangulation DT(P) such that no point in P is inside the circumcircle of any triangle in DT(P). Delaunay

triangulations aim to maximize the minimum angle of all the angles of the triangles in the triangulation; they

tend to avoid skinny triangles. The triangulation was invented by Boris Delaunay in 1934. Based on Delaunay's

definition, the circumcircle of a triangle formed by three points from the original point set is empty if it does

not contain vertices other than the three that define it (other points are permitted only on the very perimeter,

not inside).

The Delaunay condition states that a triangle net is a Delaunay triangulation if all the circumcircles of all the

triangles in the net are empty. This is the original definition for two-dimensional spaces. It is possible to use it

in three-dimensional spaces by using a circumscribed sphere in place of the circumcircle.

For a set of points on the same line there is no Delaunay triangulation (in fact, the notion of triangulation is

undefined for this case).

For four points on the same circle (e.g., the vertices of a rectangle) the Delaunay triangulation is not unique:

clearly, the two possible triangulations that split the quadrangle into two triangles satisfy the Delaunay

condition.

FIGURE 25: A DELAUNAY TRIANGULATION WITH CIRCUMCIRCLES

The Delaunay triangulation of a discrete point set P in general position corresponds to the dual graph of the

Voronoi tessellation for P. Special cases include the existence of three points on a line and four points on circle.

http://en.wikipedia.org/wiki/Mathematics

http://en.wikipedia.org/wiki/Computational_geometry

http://en.wikipedia.org/wiki/Triangulation_%28advanced_geometry%29

http://en.wikipedia.org/wiki/Circumcircle#Circumcircles_of_triangles

http://en.wikipedia.org/wiki/Triangle_%28geometry%29

http://en.wikipedia.org/wiki/Boris_Delaunay

http://en.wikipedia.org/wiki/Triangulation

http://en.wikipedia.org/wiki/Discrete_space

http://en.wikipedia.org/wiki/General_position

http://en.wikipedia.org/wiki/Dual_graph

http://en.wikipedia.org/wiki/Voronoi_tessellation


Ch

apte

r 2

26

FIGURE 26: ON THE LEFT, THE DELAUNAY TRIANGULATION WITH ALL THE CIRCUMCIRCLES AND THEIR CENTERS (IN RED). ON THE RIGHT,

CONNECTING THE CENTERS OF THE CIRCUMCIRCLES PRODUCES THE VORONOI DIAGRAM (IN RED).

The Delaunay triangulation has the following properties. Let n be the number of points and d the number of

dimensions:

The union of all simplices in the triangulation is the convex hull of the points.

The Delaunay triangulation contains at most simplices.

In the plane (d = 2), if there are b vertices on the convex hull, then any triangulation of the points has

at most 2n − 2 − b triangles, plus one exterior face (see Euler characteristic).

In the plane, each vertex has on average six surrounding triangles.

In the plane, the Delaunay triangulation maximizes the minimum angle. Compared to any other

triangulation of the points, the smallest angle in the Delaunay triangulation is at least as large as the

smallest angle in any other. However, the Delaunay triangulation does not necessarily minimize the

maximum angle.

A circle circumscribing any Delaunay triangle does not contain any other input points in its interior.

If a circle passing through two of the input points doesn't contain any other of them in its interior,

then the segment connecting the two points is an edge of a Delaunay triangulation of the given points.

The Delaunay triangulation of a set of points in d-dimensional spaces is the projection of the convex

hull of the projections of the points onto a (d+1)-dimensional paraboloid.

http://en.wikipedia.org/wiki/Voronoi_diagram

http://en.wikipedia.org/wiki/Euler_characteristic

http://en.wikipedia.org/wiki/Convex_hull

http://en.wikipedia.org/wiki/Convex_hull

http://en.wikipedia.org/wiki/Paraboloid


Ch

apte

r 2

27

2.9.5 DIVIDE AND CONQUER

A divide and conquer algorithm for triangulations in two dimensions was presented by Lee and Schachter and

refined by Guibas and Stolfi [19]. This algorithm recursively draws a line to split the vertices into two sets. The

triangulation is computed for each set, and then the two sets are merged along the splitting line. The merge

operation can be done in time O(n). In this way, the total running time is O(n log n).

For some types of point sets, such as a uniform random distribution, the expected time can be reduced to

O(n log log n) by intelligently picking the splitting lines.

FIGURE 27: THE DELAUNAY TRIANGULATION OF A RANDOM SET OF 100 POINTS IN A PLANE (LEFT). 3D BRAIN FINALLY OVER IMPOSED

TO REAL CT SCAN IMAGE (RIGHT).

http://en.wikipedia.org/wiki/Divide_and_conquer_algorithm

http://en.wikipedia.org/wiki/Leonidas_J._Guibas

http://en.wikipedia.org/wiki/Jorge_Stolfi


Ch

apte

r 2

28

2.10 TRACKING SYSTEMS FOR NEUROSURGICAL NAVIGATION

One of the most important issues in intracranial surgery is to precisely localize targets within the brain and to

refer them to important anatomical structures. The first method used in this field was the craniometry, the

technique of measuring the bones of the skull. It was developed in the 19th century and it is considered the

first practical method of surgical navigation.

In 1908, Horsley and Clark introduced the first apparatus for performing the so-called stereotactic

neurosurgery, whereby a set of precise numerical coordinates are used to locate each brain structure. The

evolution of this technique it's related to the developing of advanced techniques in intracranial imaging, 3D

reconstruction and visualization and in general in computer science. In particular, the 3D patient data

reconstruction it was crucial for the developing of new methods of planning for the intervention based on

computed trajectories. The ideal approach it's the use of real-time imaging system updated constantly during

the various steps of the intervention. Unfortunately this approach (known as Open MRI) is expensive, requires

special sets of non-ferromagnetic surgical instruments and has several restrictions of the space for the optimal

patient positioning and surgeons’ movements. For this reason the most common approach is based on the

preoperative image dataset obtained in the preoperative step. Ideally a neuronavigation system should allow

the surgeon to know continuously the position of the instrument inside the intracranial anatomy and related to

the pathology.

During the navigation all the crucial instruments positions are given by a tracking system. Tracking methods

used in neurosurgery are either optical or electromagnetic with a large prevalence of the first category.

2.10.1 OPTICAL TRACKING Optical technologies are reasonably priced, high-resolution systems often used in medicine.

Two infrared emitting cameras are mounted on a holder at a fixed distance from each other (usually around

100cm). The angulations of the cameras can be arranged to achieve the best performances.

Tool markers are placed on each surgical tool and on the patient and each one is characterized by different

configurations. Markers (Figure 28) can be passive or active depending on their function (emitting infrared

light, or reflecting light emitted from the sensor). The most common types of passive markers are plastic

spheres with a glass-grain coating and can be sterilized by gas or plasma. They have limited lifetime of about

ten procedures. Active markers have been built in diodes emitting infrared light which is tracked by camera.

The position of the markers is calculated in real-time by processing the accumulated information from the

received infrared. Due to the nature of this tracking method, a direct line-of-sight needs to be maintained

between the sensor and the markers at all times and the markers have to be placed at a minimal and maximal

distance relating to the cameras. This gives a well-defined tracking range (volume, see Figure 29).

The infrared light tracked permits localization of tools or generic rigid bodies on which markers are placed. It's

described in terms of rotation and translation from the camera coordinate system into the marker coordinate

systems.


Ch

apte

r 2

29

FIGURE 28: NDI POLARIS TRACKED TOOLS: ACTIVE (A) AND PASSIVE (B) WITH RETROREFLECTING SPHERES (C), INFRARED TRACKING

SCHEME (D).

FIGURE 29: TRACKING VOLUME (LEFT) AND NDI VICRA AND SPECTRA SYSTEMS FOR MEDICAL TRACKING (RIGHT).

Navigation system transfers the transformation matrix from camera coordinate system into coordinate system

of a navigated Rigid Body. The transformation matrix, consisting of the translation vector (T) and the rotation

matrix (R), transforms the camera coordinate system into the Rigid Body coordinate system.

The elements, s1–s3 elements specify scaling and aberrations of camera.

The most used tracking devices used by neurosurgical department in the middle Europe are from Brainlab [21],

Stryker [22] and NDI [23]. The single marker accuracy is around 0.35mm and the updated rates depend by the

number of the tools that are tracked at the same time. One problem related to this technology is that the

functioning of optical localizer may be disturbed by reflective objects and external sources of IR light.

2.10.2 ELECTROMAGNETIC In electromagnetic tracking, sensor coils are embedded to the tools that are being tracked. Also, an apparatus

designed to emit magnetic fields is installed in the room that tracking is to be conducted. When a coil-


Ch

apte

r 2

30

embedded tool is placed inside these fluctuating magnetic fields, voltages are induced in the coils. The values

of those voltages are used for the computation of the position and orientation of the surgical tool.

Since the fields are magnetic and of low strength, thus harmless for living tissue, tracking can continue even

without direct line-of-sight, in contrast to optical tracking.

2.11 PATIENT TRACKING

Several transformation matrices must be computed to enable the tracking of an intra-operative tool, and relate

its position to the patient’s image (Figure 30): MTL represents the transform between the coordinate system of

the optical tracking device, and the LED infra-red emitters on the tracked probe. MLP related these emitters to

the probe tip (or other instrument being tracked); MWT is the transform between the tracking device and

“world” coordinates (the physical patient), and MPW the transform that maps the patient coordinate system to

the image. Each of these transformation matrices must be carefully determined from calibration procedures. If

a microscope is involved in the procedures then another transformation matrix has to be considered.

In order to define the position of one marker in the coordinate system of the “reference marker”, the

transformation matrix MWTL (Figure 30: Coordinate transforms involved in the intra-operative tracking) is

calculated according to the following formula:

MWT·MWTL=MTL or MWTL=MWT-1

·MTL

MWT and MTL define the transformation from the camera coordinate system into the rigid bodies of the surgical

tool and of the patient coordinate systems.

FIGURE 30: COORDINATE TRANSFORMS INVOLVED IN THE INTRA-OPERATIVE TRACKING

MWTLMWTL


Ch

apte

r 2

31

2.12 PATIENT REGISTRATION

Registration is a key problem in medical imaging because the surgeon must often compare or fuse different

images or, in the case of surgical navigation, they need to have the exact alignment between preoperative data

and real patient position. This perfect overlapping of virtual dataset and reality is a rigid registration problem.

There are different registration methods in literature but only two different registration techniques are

currently used in the existing neuro-navigation systems: point based (also called landmark based) registration

and surface-based methods.

Point-based registration correlate geometric and anatomical landmarks present in both the preoperative image

and the patient’s head. Few markers are placed on the patient’s skin (intrinsic markers) or bones (extrinsic

markers), which are then localized in the images together with additional anatomical landmarks. It is the

method adopted in commercial neurosurgery, ENT, and orthopaedics IGS systems.

During the intra-operative phase, the surgeon touches the markers and landmarks with a tracked probe and

pairs them to those present in the pre-operative images. The transformation that aligns the point pairs is then

computed, and the CT/MRI image is registered to the intra-operative coordinate system. This method assumes

that the markers remain on the patient skin between the imaging and surgery time and that the markers and

landmarks are reachable in the intra-operative phase.

Other constraints of this technique are that it requires an additional time-consuming and error-prone intra-

operative manual procedure and that its accuracy depends on the surgeon’s ability to localize the markers and

landmarks.

Surface-based registration uses surface data acquired from the patient anatomy in the operating theatre to

compute the alignment. Data are acquired with a tracked laser probe or with a 3D surface scanner [24].

Movements of the tracked probe around patient regions of interest permit to collect hundreds of points.

Surface scanners moving the probe around patient regions of interest permit the acquirement in a few seconds

a cloud of hundreds of thousands of points in a single scan. The method is applicable when the anatomy is

visible, such as in neurosurgery (face scan). It is marker-free, fast, intuitive, and easy to use and accuracy is

more dependent by the number of acquired points then by surgeon ability.

Unfortunately the state of the art reports only few clinical accuracy studies using this method and both for the

z-touch probe [25][26] and for surface scanners [27][28]. The studies include few cases and report the

registration error at the fiducials area or at the scanned surface points, which falls short of providing the

desired clinical target registration error. Some recent studies propose to use a neural network representation

of the surface for surface-based registration system to reduce the computational load improving performances

[29].

2.12.1 ITERATIVE CLOSEST POINT Mathematically speaking registration means to find the transformation matrix between two datasets. The ICP

algorithm [30] is an iterative alignment algorithm. It consists of three phases:

1. establishment of the correspondence between pairs of features in the two structures (points) that are

to be aligned based on proximity;

2. estimating of the rigid transformation that best maps the first member of the pair onto the second;

3. applying that transformation to all features in the first structure.


Ch

apte

r 2

32

These three steps are iterated until convergence is obtained. If the initial estimation is good this simple

algorithm works effectively in real-time applications.

More details about ICP can be found in Appendix A and in the MENTIS extension to AR implementation

(Chapter 7).

2.12.2 PATIENT REGISTRATION ERROR ANALYSIS Following the suggestions in [31] three useful measures of error for analyzing the accuracy of point-based

registration methods:

1. Fiducial localization error (FLE): the error in locating the fiducial points. It is the intrinsic error of the

tracking system.

2. Fiducial registration error (FRE): the root-mean-square distance between corresponding fiducial points

after registration.

3. Target registration error (TRE): the distance between corresponding points other than the fiducial

points after registration. It can be studied qualitatively using numerical simulations or estimated using

an expression that gives a good approximation to the distribution of TRE for any given target and

fiducial points configuration [32].


Ch

apte

r 3

33

CHAPTER 3

SIMULATION IN MEDICINE


Ch

apte

r 3

34

3.14 MAIN CONCEPTS AND CLASSIFICATION

Surgical simulation is a very complex field in which various topics and disciplines are involved and mixed

together. Normally it takes advantage from available knowledge in computer science, medicine, bioengineering

(see Figure 31).

FIGURE 31: SCHEME OF ALL THE DISCIPLINES INVOLVED IN A SURGICAL SIMULATOR.

Following the Satava [36] classification of surgical simulators there are three categories (see Figure 32:

Simulators classification)

FIGURE 32: SIMULATORS CLASSIFICATION

Topology

Deformable

objects

Graphics

Simulation

Haptic

sensations

Anatomy

Tissue

properties

Phisiology

Computer

science

Medicine

Bioengneering

Topology

Deformable

objects

Graphics

Simulation

Haptic

sensations

Anatomy

Tissue

properties

Phisiology

Computer

science

Medicine

Bioengneering


Ch

apte

r 3

35

The first generation simulators are based on anatomical information and they are only concerning the

geometry of the human structures for learning or pre-operative planning. The user interaction with the model

organs is limited to the navigation.

The second generation simulators add a description the physical properties of the human body (i.e.

biomechanical modelling of soft tissue, deformation, cutting etc.). The prototype describe in this work belongs

to this type of simulators. Third generation of surgical simulators provides a coupling of anatomy, physics and

physiological descriptions of the human body.

3.15 SIMULATOR DESIGN

Realism and real-time interactions are essential features for a simulator used as an educating system. The

realism of the simulation strictly depends on the accuracy of the human tissue modelling and on the use of a

force feedback device. Therefore, the most critical issues in designing simulators are both accuracy (the

simulator should generate visual and haptic sensations which are very close to reality) and efficiency

(deformations must be rendered in real-time). Accuracy and efficiency are two opposite requirements; in fact,

increased accuracy implies a higher computational time and vice versa. Thus it will necessary to find a trade-off

in terms of the application.

One of the essential requirements in order to have a realistic surgical simulator is real-time interaction by

means of a haptic interface. In fact, reproducing haptic sensations increases the realism of the simulation.

However, the interaction need to be performed in real-time, since a delay between the user action and the

system reaction reduces the user immersion.

The simulation process is divided into different phases needed to render graphically and haptically the virtual

scene (Figure 33: simulation steps).

FIGURE 33: SIMULATION STEPS

Segmentation Classification

3D ModelGeometrical/Physical

Collision

Detection

Collision

Response

Force Feedback

User

Pre-simulation

steps

Real-Time

Computation

Human-Machine

Interface

Visual Feedback



Collision

Detection

Collision

Response

Force FeedbackForce Feedback

User

Pre-simulation

steps

Real-Time

Computation

Human-Machine

Interface

Human-Machine

Interface

Visual FeedbackVisual Feedback



Collision

Detection

Collision

Response


User

Pre-simulation

steps

Real-Time

Computation

Human-Machine

Interface

Human-Machine

Interface




Collision

Detection

Collision

Response


User

Pre-simulation

steps

Real-Time

Computation

Human-Machine

Interface

Human-Machine

Interface



Ch

apte

r 3

36

From the real patients data it is possible to build a model of the diseased organ or area. Starting by the real

patient images, is it possible to apply the image processing techniques previously described in paragraph 2.8

obtaining a complete virtual patient. This is similar to the usual procedure in the IGT workflow where the

patient data set has been processed in the radiological phase in order to segment the regions of interest (i.e.

tumour or important anatomical structures). The organs, after classification, has been ready to be described by

3D volumetric data, muscles, wires and other tissue structures can be classified assigned different datasets to

organs. This approach is a bit different in the case of surgical tools reconstruction just because in this case, data

are acquired using 3D laser scan (reverse engineering) or using CAD software.

In order to carry out high quality recognition of the tissues is necessary to use the correlative information

obtained in the three acquisition phases. Due to the distortion produced by the movement of the brain, the

three images have to be aligned (i.e. using a morphing algorithm) and then recognized, using a clustering

algorithm. Last phase is the extraction of the triangulated model of the organs (as previously described in

paragraph 2.8.1).

In the case of a surgical simulation, interactions between virtual organs and surgical instruments have to be

considered. It means that these rigid objects have to be also modelled. There are different ways to do that. The

most common approach is to use CAD tools for modelling after accurate measurement. Another and faster way

is to use 3D scanning tool (see Figure 34).

FIGURE 34: 3D SCANNING FOR REVERSE ENGINEERING (FARO ARM)


Ch

apte

r 3

37

3.16 VISUAL FEEDBACK

A surgery simulator must provide a realistic visualization of the surgical procedure scene helping the surgeon to

have a 3D perception of the complete environment. In particular shading, shadows and textures are must be

reproduced in a simulator as key-factors for realism.

The quality of visual feedback is related directly to the availability and performances of graphics accelerators.

Nowadays, the market of graphics cards has evolved in several directions: improving price-performance ratio,

increasing geometric transformation and rasterization performances and parallelizing algorithms for rendering

in a GPU context. The realistic visual feedback for surgery simulation, achieved if the graphics rendering, is

focused on the three-dimensional clues used by surgeons to understand the surgical field.

3.17 HAPTIC FEEDBACK

Haptics serves at least two purposes in a surgical simulator: kinaesthetic and cognitive. These types of

interfaces provide from one hand the sensation of movement to the user and therefore it significantly

enhances it surgical performance. Second, it helps to distinguish between tissues by testing their mechanical

properties thanks to the haptic force feedback. The addition of a haptic feedback in a simulation system highly

increases the complexity and the required computational power [39]: it leads to an increase by a factor 10 of

the required bandwidth, synchronization between visual and haptic displays, force computation. Some

important features for a haptic interface are dynamic range, bandwidths, degrees of freedom, structural

friction and stiffness. Dynamic range is the maximum force divided by the interface friction. High bandwidths

are important for short time delays and overall system stability. Friction is the force resisting the relative lateral

(tangential) motion of haptic arm in contact. The degrees of freedom are the set of independent displacements

and/or rotations that specify completely the displaced position and orientation of the body or system.

Sustained force levels are usually much smaller than maximum output force produced by haptic interfaces.

Stiffness is the resistance of an elastic body to deformation by an applied force during the contact with the

haptic tip.

Only a few papers have assessed the importance of haptic feedback in surgery [40]. In general, it is accepted

that the combination of visual and haptics is optimal for surgery training or pre-planning.

3.18 IMPLEMENTING A SIMULATOR

The main problem encountered when implementing a surgical simulator originates from the trade-off that

must be found between real-time interaction and the necessary surgical realism of a simulator.

The first constraint indicates that a minimum bandwidth between the computer and the interface devices must

be available in order to provide a satisfactory visual and haptic feedback. If this bandwidth is too small, the user

http://en.wikipedia.org/wiki/Displacement_%28vector%29


Ch

apte

r 3

38

cannot properly interact with the simulator and it becomes useless for surgery gesture training. However, the

real-time constraint can be interpreted in different ways. Most of the time, it implies that the mean update

rate is high enough to allow a suitable interaction. However, it is possible that during the simulation, some

events (such as the collision with a new structure) may increase the computational load of the simulation

engine. This may result in a lack of synchronicity between the user gesture and the feedback the user gets from

the simulator. When the computation time is too irregular, the user may even not be able to use the simulator

properly. In order to guarantee a good user interaction, it is necessary to use dedicated real-time software that

supervises all tasks executed on the simulator. The simulation frame rate can be influenced by other

parameters coming from other not haptic devices (i.e. a tracking system).

The second constraint is related to the targeted application of a simulator: training surgeons to new gestures or

procedures. To reach this goal, the user must believe that the simulator environment corresponds to a real

procedure and the simulator ergonomics has to be realistic.

The level of realism of a simulator is related on the type of surgical procedures and it is also connected with

physio-psychological parameters. In any case, increasing the realism of a simulator requires an increase of

computational time which is contradictory with the constraint of real-time interaction.

The main key-factor in implementing a simulator is to optimize its credibility, given an amount of graphics and

computational resources.

Figure 35 shows a diagram of a common implementation of haptic rendering specified for surgical simulation.

The user (surgeon) has two types of feedback: tactile from the haptic interface and visual from various types of

displays. A good frame rate to feel the tactile sensations is around 1 KHz and the correspondent frequency to

see a continuous image flow with human eyes is around 30 KHz. Two different threads (haptic and visual) are

needed for the complete simulation. Tissue deformations are based on the geometrical and physical tissue

model. They must be rendered graphically and haptically.

FIGURE 35: DIAGRAM OF A COMMON IMPLEMENTATION OF HAPTIC RENDERING SPECIFIED FOR SURGICAL SIMULATION

Haptic

ThreadVisual

Thread

Haptic

Interface

Display

Tissue

Deformations

Thread

3D

Geometry,

Mechanical

Properties

Force Feedback

Hand Motion

Position

Information

Collision Detection

Haptic

Rendering

(~1KHz)

(~1KHz) (~30 Hz)

Graphic

Rendering

Visual Feedback

Trainee

Surgeon

Haptic

ThreadVisual

Thread

Haptic

Interface

Display

Tissue

Deformations

Thread

3D

Geometry,

Mechanical

Properties

Force Feedback

Hand Motion

Position

Information

Collision Detection

Haptic

Rendering

(~1KHz)

(~1KHz) (~30 Hz)

Graphic

Rendering

Visual Feedback

Haptic

ThreadVisual

Thread

Haptic

Interface

Display

Tissue

Deformations

Thread

3D

Geometry,

Mechanical

Properties

Force Feedback

Hand Motion

Position

Information

Collision Detection

Haptic

Rendering

(~1KHz)

(~1KHz) (~30 Hz)

Graphic

Rendering

Visual Feedback

Trainee

Surgeon


Ch

apte

r 3

39

3.19 VIRTUAL AND AUGMENTED REALITY CONCEPTS

It is particularly important to introduce now the concepts of virtual, augmented and mixed reality. There is

often confusion about these meanings because not many systems are nowadays in common use. In 1994 Paul

Milgram and Fumio Kishino defined a mixed reality as anywhere between the extremes of the virtuality

continuum (VC) [41], where the VC extends from the completely real through to the completely virtual

environment with augmented reality and augmented virtuality ranging between.

FIGURE 36: DEFINITION OF MIXED REALITY WITHIN THE CONTEXT OF THE RV CONTINUINUUM (MILGRAM AND KISCHINO 1994)

In the original paper, the two authors describe the continuum in this way:

"The concept of a virtuality continuum relates to the mixture of classes of objects presented in any particular

display situation, as illustrated in the previous diagram, where real environments, are shown at one end of the

continuum, and virtual environments, at the opposite extremes. The former case, at the left, defines

environments consisting solely of real objects (defined below), and includes for example what is observed via a

conventional video display of a real-world scene. An additional example includes direct viewing of the same

real scene, but not via any particular electronic display system. The latter case, at the right, defines

environments consisting solely of virtual objects (defined below), an example of which would be a conventional

computer graphic simulation. As indicated in the figure, the most straightforward way to view a Mixed Reality

environment, therefore, is one in which real world and virtual world objects are presented together within a

single display, that is, anywhere between the extremes of the virtuality continuum. "

Figure 36 shows these concepts. According to the scientific convention we are working about a Mixed Reality

system for neurosurgical pre- and intra- operative aid to the surgeons. A direct consequence is that if the same

system can be used as virtual and augmented system then it is a mixed reality system.

For this reason, the final prototype of this scientific work can be considered a mixed reality system.

http://en.wikipedia.org/wiki/Virtuality_Continuum

http://en.wikipedia.org/wiki/Augmented_reality

http://en.wikipedia.org/wiki/Augmented_virtuality


Ch

apte

r 4

40

CHAPTER 4

HAPTICS AND SURGERY


Ch

apte

r 4

41

4.1 HAPTIC INTERFACES

Haptic is from the Greek "haptesthai," meaning to touch. Usually used in a plural form (haptics), it means the

science and physiology of the sense of touch. It is related to technology that interfaces to the user via the

sense of touch by applying forces, vibrations, and/or motions to the user. The field of haptics is growing rapidly

and is inherently multidisciplinary and in many areas, including robotics, experimental psychology, biology,

computer science, systems and control, and others.

The main human structure associated with the sense of touch is the hand. It is extraordinarily complex: several

types of receptors are in the skin and send trough the nerves information back to the central nervous system

and the point of contact. The hand is composed of 27 bones and 40 muscles which offers a big dexterity. This

concept is quantified using the degrees of freedom (movement afforded by a single joint). Since the human

hand contains 22 joints, it allows movement with 22 degrees of freedom. The skin covering the hand is a rich of

receptors and nerves is a source of sensations for the brain and spinal cord.

The haptic feedback is usually a conjunction of two types: kinaesthetic and tactile.

Kinaesthetic information concerns physical forces applied to an object and returned from that object. It takes

advantage of human proprioception, the awareness of body position through the muscle and tendon positions.

It deals with contours, shapes and sensations like the resistance and weight of objects. Tactile sensations are

often included under the general term of haptics. These sensations incorporate vibrations and surface textures

and are detected by receptors close to the skin. It is related to roughness, friction, and somatic properties,

which includes changes perceived during static contact with the skin, such as temperature. In the specific,

known as proprioceptors, these receptors carry signals to the brain, where they are processed by the

somatosensory region of the cerebral cortex. The muscle spindle is one type of proprioceptor that provides

information about changes in muscle length. The Golgi tendon organ is another type of proprioceptor that

provides information about changes in muscle tension. The brain processes this kinesthetic information to

provide a sense of the baseball's gross size and shape, as well as its position relative to the hand, arm and body.

There are several commercial haptic interfaces characterized by software to determine the forces that result

when a user's virtual identity interacts with an object and a device through which those forces can be applied

to the user.

FIGURE 37: EXAMPLE OF FORCE-FEEDBACK GLOVE WITH PNEUMATIC PISTONS TO SIMULATE GRASPING (HUMAN-MACHINE INTERFACE

LABORATORY OF RUTGERS UNIVERSITY)

http://en.wikipedia.org/wiki/Forces

http://en.wikipedia.org/wiki/Vibrations


Ch

apte

r 4

42

The actual process used by the software to perform its calculations is called haptic rendering. A common

rendering method uses polyhedral models to represent objects in the virtual world. These 3-D modelscan

accurately portray a variety of shapes and can calculate touch data by evaluating how force lines interact with

the various faces of the object. Such 3-D objects can be made to feel solid and can have surface texture.

The PHANTOM® interface from SensAble Technologies (see Figure 38) was one of the first haptic systems to be

sold commercially. Its success lies in its simplicity. Instead of trying to display information from many different

points, this haptic device simulates touching at a single point of contact. It achieves this through a stylus which

is connected to a lamp-like arm. Three small motors give force feedback to the user by exerting pressure on the

stylus. Therefore, a user can feel the elasticity of a virtual balloon or the solidity of a brick wall. He or she can

also feel texture, temperature and weight.

The stylus can be customized so that it closely resembles just about any object. For example, it can be fitted

with a syringe attachment to simulate what it feels like to pierce skin and muscle when giving a shot.

FIGURE 38: TWO TYPES OF HAPTIC INTERFACES: OMNI (LEFT) AND PHANTOMTM DESKTOP (RIGHT). COURTESY SENSABLE TECHNOLOGIES

The CyberGrasp® system, another commercially available haptic interface from Immersion Corporation [42],

takes a different approach. This device fits over the user's entire hand like an exoskeleton and adds resistive

force feedback to each finger. Five actuators produce the forces, which are transmitted along tendons that

connect the fingertips to the exoskeleton. With the CyberGrasp® system, users are able to feel the size and

shape of virtual objects that only exist in a computer-generated world. To make sure a user's fingers do not

penetrate or crush a virtual solid object, the actuators can be individually programmed to match the object's

physical properties. The CyberTouch® (another Immersion Corporation product) uses six electromechanical

vibrators placed on the back of the fingers and in the palm. These actuators produce vibrations of 0- 125 Hz,

with a force amplitude of 1.2 N at 125 Hz.

Researchers at Carnegie Mellon University are experimenting with a haptic interface that does not rely on

actuated linkage or cable devices. Instead, their interface uses a powerful electromagnet to levitate a handle

that looks a bit like a joystick. The user manipulates the levitated tool handle to interact with computed

environments. As she moves and rotates the handle, she can feel the motion, shape, resistance and surface

texture of simulated objects. This is one of the big advantages of a levitation-based technology: It reduces

friction and other interference so the user experiences less distraction and remains immersed in the virtual

environment. It also allows constrained motion in six degrees of freedom (compared to the entry-level

Phantom interface, which only allows for three active degrees of freedom).

http://science.howstuffworks.com/exoskeleton.htm

http://science.howstuffworks.com/electromagnet.htm


Ch

apte

r 4

43

The one disadvantage of the magnetic levitation haptic interface is its footprint. An entire cabinet is required to

house the maglev device, power supplies, amplifiers and control processors. The user handle protrudes from a

bowl embedded in the cabinet top.

FIGURE 39: THE CYBERGRASP® (LEFT) AND CYBERTOUCH® (RIGHT) FROM IMMERSION.

http://computer.howstuffworks.com/power-supply.htm


Ch

apte

r 4

44

4.2 STATE-OF-THE-ART IN SURGICAL SIMULATION WITH HAPTICS

Previous studies show that the use of force feedback during the execution of surgical tasks can be a big help.

Wagner et al [43] asked subjects to dissect a physical model of an artery with and without force feedback, and

found that force feedback significantly reduced the number of errors and the overall level of applied force.

Tholey et al [44][45] asked subjects to perform a soft-tissue identification task in a physical model, and found

that haptic feedback significantly enhanced subjects’ ability to distinguish among tissue types. Kazi [46] founds

that force feedback reduces applied forces in a catheter insertion task. These results confirm the intuition that

haptic feedback is critical to the fine dexterous manipulation required for surgery.

Simulators are particularly focused on minimally invasive techniques especially video-surgery (endoscopy,

laparoscopy) in which particularly skill have to be developed. In laparoscopy for instance, the surgical

procedure is made more complex by the limited number of degrees of freedom of each surgical instrument.

Moreover, high level of hand-eye coordination is requested to use the fixed point on the patient’s abdomen

incisions especially considering that the surgeon cannot see his hand on the monitor. In addition, the

development of minimally invasive techniques has reduced the sense of touch compared to open surgery,

surgeons must rely more on the feeling of net forces resulting from tool-tissue interactions and need more

training to operate successfully on patients. Basdogan et al [47] show that haptics is a valuable tool especially

in minimally invasive surgical simulation and training. Such systems bring a greater flexibility by providing

scenarios including different types of pathologies. Furthermore, thanks to the development of medical image

reconstruction algorithms, surgery simulation allows surgeons to verify and optimize the surgical procedure

(gestures and strategy) on a specific patient case.

Webster et al. [48] present a haptic simulation environment for laparoscopic cholecystectomy, and

Montgomery et al. [49] present a simulation environment for laparoscopic hysteroscopy; both projects focus

on haptic interaction with deformable tissue. Cotin et al [50] present a haptic simulator for interventional

cardiology procedures, incorporating blood flow models and models of cardiopulmonary physiology. De et al

[51] apply the method of finite spheres to a haptic simulator for laparoscopic GI surgery.

The success of using haptic devices in medical training simulators has already been obtained by several

commercial companies working in this field (Immersion Medical, Surgical Science, Mentice, and Reachin

Technologies, for example) and other research works [52], [53], [54], [55], [56], [57], [58].

4.2.1 SURGICALSCIENCE: LAPSIM® SYSTEM The LapSim® System [59] from SurgicalScience [60] consists of an equipment and software (see Figure 40:

surgicalscience: LapSim System). It is designed like current surgery instruments. The contact point between

hand and interfaces looks like the handhold of a scissor. The software component provides the different

exercises. They are displayed as an interactive live video on a screen. The interaction takes place due to the

interface movement which is factored into the VR. The software allocates different levels of difficulty. By this, a

surgeon’s minimal-invasive competence is built and improved. A simple interface leads to an easy handling.

Each training session is recorded for post processing reasons.

The LapSim system is available in three different versions: Basic Skills, Dissection and Gyn. The first one

contains fundamental exercises for each surgeon. These are grasping, knotting, camera navigation, general

coordination or lifting elements. A medic needs to have a good command on the simple tools. There are two

upgrades for the Basic Skill software. The Dissection upgrade treats two additional exercises in the range of


Ch

apte

r 4

45

laparoscopic cholecystectomy procedures. The Gyn add-on supplies four training exercises: tubal occlusion,

salpingectomy, tubotomy and myoma suturing.

FIGURE 40: SURGICALSCIENCE: LAPSIM SYSTEM

There is a large number of studies treating short-term and long-term systematic training on this product. A few

compare traditional training methods with the new possibility of LapSim. In general, the regular trainings with

LapSim improve basic skills much better than conventional trainings. To measure the effort of LapSim it is

important to establish the transfer rate between the MVR and the original reality (OR). Studies show, that the

learned skills in MVR are quickly implemented in OR. Even the time consumption decreases in comparison to

non-trained subjects. LapSim R can be used for experts or advanced too.

Clinical background and understanding is important for the effort of the training. Advanced users show a faster

acquisition and foresight during a three days training. Novices benefit more of the first use and improve very

quickly.


Ch

apte

r 4

46

4.2.2 SIMBIONIX SIMULATORS

4.2.2.1 LAPAROSCOPYVR VIRTUAL-REALITY SYSTEM

The LapVR® Surgical Simulator uses interactive 3D models, haptic force feedback, and performance tracking

and evaluation to help decrease the learning curve in laparoscopic surgery. The simulation tasks are about:

Basic skills (camera navigation, peg transfer, cutting, knot tying, clipping)

Adhesiolysis

Cholecystectomy

Gynecological surgery (ectopic pregnancy intervention, tubal occlusion).

4.2.2.2 ENDOSCOPY ACCUTOUCH SYSTEM

The AccuTouch® endoscopy surgical simulator allows medical training of multiple disciplines on the same

platform:

Bronchoscopy

Upper gastrointestinal flexible endoscopy

Lower gastrointestinal flexible endoscopy.

The simulation system consists of a pc, an interface for the flexible tube; realistic endoscopes. The robotic

interface provides realistic forces, emulating the feel of the real procedure, virtual-reality patients respond in a

physiologically accurate manner, real-time graphics and audio feedback combine with haptic feedback,

anatomic models developed from actual patient data provide increasingly challenging anatomy and multimedia

didactic content supports independent learning.

4.2.2.3 ARTHROSCOPY SURGICAL SIMULATION: INSIGHTARTHROVR SYSTEM

The insightArthroVR® arthroscopy surgical simulator provides arthroscopy training on knees and shoulders in a

controlled, stress-free, and virtual-reality environment.

The system includes:

Realistic anatomical models validated by experts in arthroscopy and anatomy, including both healthy

joints and a variety of pathologies

A camera and multipurpose tool that adapts to different joints and arthroscopic techniques

Skill indicators that allow for practitioner skills evaluation through configurable key performance

indicators.

A training program in which the practitioner can advance through exercises of increasing difficulty.


Ch

apte

r 4

47

a)

c)

b)

d)

e)

f)

h)

g)

i)

l)

m)

n)

o)

FIGURE 41: DIFFERENT TRAINING PLATFORMS AND SCENARIOS FROM SIMBIONIX


Ch

apte

r 4

48

4.2.3 KISMET (Kinematic Simulation, Monitoring and Off-Line Programming

Environment for Telerobotics). Developed at Forschungszentrum Karlsruhe, it is not a simulator for neurosurgical interventions but it´s good to

be referred because it’s probably the best simulator for mini-invasive surgery. It’s a graphical monitoring tool

to provide on-line "synthetic viewing" during remote handling (RH) task execution, using interfaces to the tool

control systems or other means of position sensor acquisition. Since almost a decade it was used in numerous

robotics and teleoperation applications to give engineering support during RH-equipment and workcell design,

task planning, RH-operator training and task execution monitoring. Because of its high quality real-time

graphics capabilities and additional features like geometrical and kinematical modelling, multibody-dynamics,

and its database concept allowing for multiple detail-levels, it was found to be an ideal platform for computer

aided surgical simulation. Using the advanced capabilities of high-performance graphical workstations

combined with state-of-the-art simulation software, it is possible to generate virtual endoscopic views of

surgical scenarios with high realism. Excellent results have been obtained using 3D graphical simulations with

KISMET for instrument design, operation room simulation and the prototypic MIS training simulator "Karlsruhe

Endoscopic Surgery Trainer" for laparoscopic surgery (Figure 42).

FIGURE 42: KISMET FROM FZK KARLSRUHE: PROTOTYPE (LEFT), 3D ENVIRONMENT (RIGHT)


Ch

apte

r 4

49

4.3 VIRTUAL TRAINING SYSTEMS IN NEUROSURGERY

The state of the art about the training systems in neurosurgery is limited to few relevant works. This is probably

due to the complexity of the brain system or to the fact that almost all the operations in neurosurgery are

carried out using a microscope that cannot be simulated in a realistic way using conventional monitors, semi-

transparent mirror or head mounted displays.

4.3.1 Web-based Neurosurgical Training Tools The Manchester Computing Group has developed a web-based Neurosurgical Training Tools [61] in

collaboration with Neurosurgeon at Leeds General Infirmary.

A common procedure in neurosurgery (and one often done in an emergency) is to artificially drain fluid from

the ventricles of the brain when the pressure within them increases due to deficient drainage of the

cerebrospinal fluid (CSF) contained within them. This usually occurs in cases of meningitis. A cannula (silicon

tube) of diameter 2.5 mm is inserted into the ventricles after a burr hole is made in the skull. The trajectory

mostly employed is through the lateral aspect of the forebrain, aiming at the anterior horn of the ventricles.

Neurosurgical trainees early in training need to get an appreciation of the 3d anatomy of the brain in some

detail and of things they cannot see eg the ventricular system. The aim of this project is to generate a widely

available 3D VRML model of the ventricular system within the human brain and cranial vault to be used as a

teaching aid.

FIGURE 43: WEB-BASED NEUROSURGICAL TRAINING TOOLS

4.3.2 Virtual environment-based endoscopic third ventriculostomy simulator It is being developed by Dept. of Electrical Eng. and Computer Sc., Case Western Reserve University and Dept.

of Neurosurgery, Rainbow Babies’ and Children’s Hospital for training neurosurgeons as a standardized method

for evaluating competency [62]. Magnetic resonance (MR) images of a patient’s brain are used to construct the

geometry model, realistic behaviour in the surgical area is simulated by using physical modelling and surgical

instrument handling is replicate by a haptic interface. The completion of the proposed virtual training simulator


Ch

apte

r 4

50

will help the surgeon to practice the techniques repeatedly and effectively, serving as a powerful educational

tool.

FIGURE 44: VENTRICULAR ANATOMY FROM A SIMULATION CREATED VIEW FROM THE LATERAL VENTRICLE. THE BASILAR ARTERY IS

VISIBLE THROUGH THE SEMI-TRANSPARENT MEMBRANE AT THE FLOOR OF THE THIRD VENTRICLE.

4.3.3 Human Ventricle Puncture and Interaction between Spatula and Brain Tissue The department of Health Science and Technology (Aalborg University) has developed two lines of dedicated

simulators (Figure 45), one for training the insertion of a catheter in the human brain for ventricular puncture,

and one for training the interaction between a surgical spatula and brain tissue [63]. Both simulators are

focused on having realistic haptic sensations.

A specialized virtual reality system has been developed to allow for simulation of the insertion of a catheter in

the human ventricles. The user can rehearse the insertion location, angle, and speed. A force feedback system

gives the user a haptic sensation of the forces involved in penetrating the different layers of tissue. Different

normal and abnormal ventricles can be included in the simulation.

Brain tissue retraction is necessary for the majority of intracranial procedures. To allow for computer based

training of the use of a spatula, a virtual reality system has been developed, where the user can feel the forces

involved in the interaction between the spatula and the tissue (Figure 45).

FIGURE 45: VIRTUAL BRAIN INTERACTION BETWEEN SPATULA AND BRAIN TISSUE (LEFT) AND HUMAN VENTRICLE PUNCTURE (RIGHT)


Ch

apte

r 5

51

CHAPTER 5

SOFT TISSUE PHYSICAL MODELLING


Ch

apte

r 5

52

5.1 INTRODUCTION

Nowadays many things are computer aided like development or production. For example every up-to-date

enterprise uses computer aided design to optimize its production process. If an enterprise plans the

development of a new product, it uses CAD programs to visualize probable prototypes before production or

physical pre-development. Computer Aided Design is today used for nearly every new application such as

products and development but it is also very useful to give assistance to processes.

There are basically two types of modelling for simulation: physically-based or only geometry and cinematic

based. For a realistic and interactive simulation a first type of modelling is requested because it is based on

material properties, external forces (i.e. gravity) and environmental constraints. For these reasons we will

focus the attention on the physically-based modelling to simulate the tissue deformations of the organs.

The modelling can be based on the surfaces (requiring low computational load) or on the volumes (very

accurate but with slow rendering). The difference between this to types is due to the different (higher) number

of vertices used by the volumetric models. The surface modelling accuracy can be improved adding liquids or

gas effects and combining with different types of surface tensions, friction, etc. Another common way to

improve this method is to adopt a multilayer layer model composed of surfaces mesh of masses, linear springs

and dampers, and a set of nonlinear springs orthogonal to the surface to model volumetric effects by giving

normal support to the surface mesh [64]. The most common methods for surgical simulation are essentially:

mass-spring-damper, finite element and boundary element model (the last one will be not exploited in the

following paragraphs).

5.2 PHYSICAL BEHAVIOURS OF SOFT TISSUES

Soft tissues have a biomechanical behaviour very complex and characterized by very high deformations

(sometimes up to 40%). The relation between applied force and deformation is highly not linear and tissues are

viscoelastic and anisotropic (depending by direction). These properties are different according to age and sex of

the subject.

The measurement of the mechanical features of the tissue is based on the application of tension or

compression to animals or cadavers tissues. In addition, it is possible to attach sensors to the tip of the surgical

instruments in order to have data related to some specific task like tissue cut, removing, grabbing or needle

insertion.

In any case the measurement are corrupted by different factors the animal anatomy is different, the removed

tissue as different behaviour because of loos of blood, exsiccation, etc.

One novel type of biomechanical properties acquisition is the medical image processing. This is possible

because the elasticity is strictly related with the presence of the water(which absorb the X-ray). Nowadays

these methods are not yet optimized and precise.


Ch

apte

r 5

53

5.3 MASS SPRING-DAMPER MODEL

This Model can describe the different behaviour of an object under force just using particles (with mass)

connected by springs. Every particle is subject to a force generated by its interconnecting springs.

Mass-spring method is used in a wide range of computer graphics and virtual reality applications, e.g. in the

animation of facial expressions, the cloth motion and the modelling of inner organs in surgery simulations.

Cover et al. [65] were the first to present a real time model for gall bladder surgery simulation.

As mentioned in the state of art chapter, Çakmak et al. [54] used a mass-spring model to simulate a realistic

interaction between surgical tools and organs in the KISMET system, a virtual reality training system for

minimally invasive surgery.

An improvement to spring models has been proposed, specifically with regard to their dynamic behaviour [66].

FIGURE 46: MASS-SPRING TISSUE MODEL

The mathematical model for the ideal mass-spring-damper system is easy to describe. Starting with the second

Newton’s law:

“Mutationem motus proportionalem esse vi motrici impressae, et fieri secundum lineam rectam qua vis illa

imprimitur” or in other words:

In which f is the force vector, m is the mass of the particle (or body) and u is the acceleration vector.

It says that the change of momentum of a body is proportional to the impulse impressed on the body, and

happens along the straight line on which that impulse is impressed.

A specification of this concept is the Hooke’s law of elasticity:

“Ut tensio, sic vis”, that means “As the extension, so the force”:

In which f is the restoring force (opposite to the deformations) exerted by the material (newtons), k is the

force constant (or spring constant that has units of force per unit length, newtons per meter) and x is the

distance that the spring has been stretched or compressed away from the equilibrium position, which is the

position where the spring would naturally come to rest (meters).


Ch

apte

r 5

54

FIGURE 47: MASS SPRING SYSTEM (IDEALLY WITHOUT ANY FRICTION).

If we consider two particles i and j connected by one spring then we have the force along them ijf :

where ijl is the rest length of the spring, ijk is the spring constant that determinates the elasticity and

is the non-linear term.

In physics and engineering, damping may be mathematically modelled as a force synchronous with the velocity

of the object but opposite in direction to it. If such force is also proportional to the velocity, as for a simple

mechanical viscous damper (dashpot), the force vf may be related to the velocity u by

where d is the viscous damping coefficient, given in units of newton-seconds per meter. This law is perfectly

analogous to electrical resistance (Ohm's law) and can be expressed as the viscoelastic model of Kelvin-Voigt.

vf is an approximation to the friction caused by drag.

FIGURE 48: THE IDEAL MASS-SPRING-DAMPER MODEL (LEFT). A MASS ATTACHED TO A SPRING AND DAMPER. THE DAMPING

COEFFICIENT, IS REPRESENTED BY D AND THE ELASTICITY IS K IN THIS CASE. THE F IN THE DIAGRAM DENOTES AN EXTERNAL FORCE. ON

THE RIGHT, SCHEMATIC REPRESENTATION OF KELVIN-VOIGT MODEL IN WHICH E IS A MODULUS OF ELASTICITY AND Η IS THE VISCOSITY.


Ch

apte

r 5

55

The equation of the model mass-spring-damper can be formulated in the Lagrange equation:

(*)

In this equation in is the number of virtual springs connecting the particles of mass im , ijk is the elasticity

constant of the springs between the points of mass im and jm , extF is the external force active on the

particle of mass jm , id is the damping and g is the gravity constant.

This is an ordinary differential equation (ODE) of the second order that can be converted in two of the first

order and solved by numerical integration.

In the simulation field the most common methods for numerical integration are Newton-Euler and fourth order

Runge-Kutta.

5.3.1 MATHEMATICAL SOLUTION

5.3.1.1 NEWTON-EULER METHOD

If h is the “step size” the approximate numerical solution to the generic form of a first order ODE is:

1 0 0y = y + h f (y )

where: 1y is the following state and 0y is the current and f compute the derivative of a certain state y.

From the previous equation we can apply the method extracting the acceleration at the time t:

If the speed and acceleration are constant during the step then the new speed and position will be:


Ch

apte

r 5

56

5.3.1.2 FOURTH ORDER RUNGE-KUTTA (RK4) METHOD

The formulation of this method to compute slope at four places within each step is:

1 0

12 0

23 0

4 0 3

k = h f (y )

kk = h f (y + )

2

kk = h f (y + )

2

k = h f (y + k )

Use weighted average of slopes to obtain:

It requires four times the derivative for each step.

RK4 is much more accurate (smaller global discretization error) than Euler but takes more flops per step and it

can achieve comparable accuracy with much larger time steps. The net effect is that RK4 is more accurate and

more efficient.

5.4 FINITE ELEMENTS METHOD

FEM treats deformable objects as a continuum: solid bodies with mass and energies distributed throughout.

The model is continuous but the computational methods used for solving the models in computer simulations

are ultimately discrete. It must be parameterized by a finite state vector that comprises the positions and

velocities of representative points. Continuum models are derived from equations of continuum mechanics.

The full continuum model of a deformable object considers the equilibrium of a general body acted on by

external forces. The object reaches equilibrium when its potential energy is at a minimum.

The total potential energy of a deformable system is denoted by , and is given by:

where is the total strain energy of the deformable object (is the energy stored in the body as material

deformation). W is the work done by external loads on the deformable object (the sum of concentrated loads

applied at discrete points, loads distributed over the body, such as gravitational forces, and loads distributed

over the surface of the object, such as pressure forces).

In order to determine the equilibrium shape of the object, both and W are expressed in terms of the object

deformation, which is represented by a function of the material displacement over the object. The system

potential energy reaches a minimum when the derivative of with respect to the material displacement

function is zero. Because is not always possible to reach a closed-form analytic solution of this equation, a


Ch

apte

r 5

57

number of numerical methods are used to approximate the object deformation. As discussed previously, mass-

spring methods approximate the object as a finite mesh of points and discretisize the equilibrium equation at

the mesh points. Finite element methods, FEM, divide the object into a set of elements and approximate the

continuous equilibrium equation over each element.

The basic steps in using FEM to compute object deformations are:

1. Derive an equilibrium equation from the potential energy equation in terms of material displacement

over the continuum.

2. Select appropriate finite elements and corresponding interpolation functions for the problem.

Subdivide the object into elements.

3. For each element, re-express the components of the equilibrium equation in terms of the

interpolation functions and the element's node displacements.

4. Combine the set of equilibrium equations for all of the elements in the object into a single system.

Solve the system for the node displacements over the whole object.

5. Use the node displacements and the interpolation functions of a particular element to calculate

displacements or other quantities of interest (such as internal stress or strain) for points within the

element.

The strain energy is derived from an integral expression over the volume

of the material stress, , and strain, , components:

where D is the linear matrix which relates stress and strain components for

an elastic system (from the generalized Hooke's law), and and

are vectors of the stress and strain components.

In an elastic system, the material strain is related to the displacement vector u ( , , )Tu v w by a set of

differential equations:

The work done by an external force f (x; y; z) is computed as the dot product

of the applied force and the material displacement u integrated over the object

volume:


Ch

apte

r 5

58

where: fb(x,y,z) are body forces applied to the object volume V, fi(x,y,z) are surface forces applied to the

object surface and pi are concentrated loads acting at the points (xi yi ,zi).

5.4.1 BRAIN TISSUE MODELLING USING FEM In the past many researches attempted to develop of analytical models of the brain for the study of injury

subject to large deformations. This type of research field was motivated by the fact that the brain is the most

critical organ to protect from trauma, since injuries to its structures are currently non-reversible, and the

consequences of injury can be devastating.

A lack of knowledge regarding the deformation properties of the brain tissue has been an inherent weakness of

these models based on the Finite Element analysis.

The research presented in some papers develops constitutive relations for brain material subject to large

deformations for subsequent implementation in finite element models [67][68].

These researches on the material properties of brain tissue, have assumed that brain tissue behaves as an

isotropic viscoelastic material. The simplest constitutive equations are obtained by modelling brain tissue as an

isotropic linear viscoelastic material in which the stress is related to the strain by

In the previous equation, the function G is the stress relaxation function of the brain material.

If the material is subjected to the strain:

[69] gives an accurate overview about the state of art on this field in the special case of FE analysis.


Ch

apte

r 5

59

5.5 MODELS COMPARISON

The use of FEM in computer graphics has been limited because of the computational requirements. In

particular, it has proven difficult to apply FEM in real-time systems. Because the force vectors and the mass and

stiffness matrices are computed by integrating over the object, they must, in theory, be re-evaluated as the

object deforms. This re-evaluation is very costly and is frequently avoided by assuming that objects undergo

only small deformations.

The following table and figure provides an overview on the difference between the discrete and the continuous

models. Figure 49: Fast computation vs. accuracy resume the relation between speed and accuracy of the

computation for the modelling methods.

Discrete models:

(Mass Spring Damper)

Continuous models:

(FEM)

Advantages:

Less computable load

Simple

Good for dynamic problems

Advantages:

Mathematical robust

Good for static descriptions

Disadvantages:

Accuracy

Risk of instability

Disadvantages:

High computable load

Accuracy problems by large

deformations

FIGURE 49: FAST COMPUTATION VS. ACCURACY

SpeedAccuracy

Accuracy vs computation time


Ch

apte

r 5

60

5.6 COLLISION DETECTION

5.6.1 INTRODUCTION In the real world, bodies are controlled by nature’s laws that automatically avoid them to interpenetrate. They

are made of matter and matter is impenetrable. Just remember that elementary law of physics that says that

two bodies cannot occupy the same space at the same time. In Computer Graphics’ virtual environments,

however, bodies are not made of matter and consequently are not automatically subjected to nature’s laws. It

means that they can pass right through each other unless we create mechanisms to impose the same nature’s

constraints.

In virtual worlds as in the real world, interactions between objects and other environmental effects are

mediated by forces applied onto them. In particular, if we wish to influence the behaviour of objects we must

do so through application of forces. Thus, a computerized physical simulation must enforce non-penetration by

calculating appropriate forces between contacting objects and then use these forces to derive their actual

motion.

Over the last twenty years, a number of approaches to this problem have appeared in the Computer Graphics

literature. Basically there are two different ways to check the intersections: spatial and model partition. As

shown in [70], model partitioning is often the better choice since it does not suffer from the problem of having

multiple references to the same objects. It is a strategy of subdividing a set of objects into geometrically

coherent subsets and computing a bounding volume for each subset of objects. We present the currently most

widely used approaches for determining whether two objects penetrate each other (collision).

FIGURE 50: MODEL PARTITIONING OF A BRAIN (BOUNDING BOXES STRATEGY).


Ch

apte

r 5

61

5.6.2 BOUNDING SPHERES One of the most primitive ways of doing collision detection is to approximate each object or a part of the

object of the scene with a sphere, and then check whether spheres intersect each other. This method is widely

used because it is computationally inexpensive. The algorithm checks whether the distance between the

centres of two spheres is less than the sum of the two radii (which means that a collision has occurred).

The strategy can be resumed in few steps (see Figure 51: Sphere collisiond detection):

compute distance d between centres

if d < r1 + r2, colliding

If a collision is detected the precision is increased subdividing the big sphere into a set of smaller spheres and

checking each of them for collision. We continue to subdivide and check until we are satisfied with the

approximation.

FIGURE 51: SPHERE COLLISIOND DETECTION

5.6.3 AXIS ALIGNED BOUNDING BOXES Although, the previous described technique is very simple, it is not accurate especially because real objects can

be rarely well approximated by a sphere.

Axis-aligned bounding boxes (AABBs) technique refers to the fact that either the objects are divided in boxes

aligned with the world axes or each face of the box is perpendicular to one coordinate axis. Since AABBs always

have to be axis-aligned, they have to be recomputed for each frame.

The AABBs strategy steps are (see Figure 52: AABBs strategy):

compare x values in min,max vertices

if min2 > max1 or min1 > max2, no collision (separating plane) otherwise check y and z directions

FIGURE 52: AABBS STRATEGY


Ch

apte

r 5

62

5.6.4 OBJECT ORIENTED BOUNDING BOXES A more accurate approach that permits a tighter fitting around the model is the object oriented bounding

boxes [71]. The initial bounding box is tight fit around the model in local coordinate space and then translated

and rotated with the model. In this case, the advantage is that when the object moves, the recalculation of the

box every time is no more required and just a transformation of the initial one is sufficient.

AABBs are aligned to the axes of the model’s local coordinate system, whereas OBBs can be arbitrarily

oriented. However this freedom of an OBB is gained at a considerable cost in terms of storage space and

computation of the intersections.

FIGURE 53: OOB STRATEGY. THE INITIAL BOUNDING BOX IS TIGHT FIT AROUND THE MODEL IN LOCAL COORDINATE SPACE AND THEN

TRANSLATED AND ROTATED WITH THE MODEL.


Ch

apte

r 6

63

CHAPTER 6

MICROSCOPE EMBEDDED NEUROSURGICAL TRAINING

SYSTEM


Ch

apte

r 6

64

6.1 SYSTEM OVERVIEW

As mentioned before, in the operation theatre surgeon’s eyes are on the microscope oculars in order to

understand the right tumour position compared to the preoperative images (CT, MRI) and occasionally on the

screen to understand the position of the tool inside the patient anatomy (acquired using the preoperative

images).

For this reason, a complete training system is requested to:

simulate the virtual view directly inside the microscope oculars;

provide the user hand with a force feedback to feel rigid and soft tissue interactions;

simulate the navigational software actually used in OR (ex. BrainLab or Stryker).

The simulator has been setup basically as a sequence of three processes that run concurrently in three

different threads:

graphic rendering;

haptic rendering;

tracking.

FIGURE 54: MENTIS ARCHITECTURE


Ch

apte

r 6

65

The graphic and the haptic renderings have been developed on top of H3D, a new high-level 3D library for real-

time computer simulation. The graphics rendering process is in charge of the stereo visualization of the 3D

virtual scene (basically composed of brain organs, skull, and surgical instruments) inside both microscope

oculars and screen.

H3D has been developed on a top of different haptic library and offers the possibilities to use different

renderings: Openhaptics (commercialized by Sensable Technologies), Chai3D, Ruspini, God Object, etc.

The haptic rendering reads the current status of the haptic device, detects the collisions between the virtual

surgical instruments and the virtual patient, and computes the reaction forces to be applied by the haptic

device.

A Phantom Desktop haptic feedback device (described before) was used in order to provide the surgeon an

immersive experience during the interaction between the surgical tools and the brain or the skull of the virtual

patients. The force feedback workspace and other important properties (ex. stiffness range and nominal

position resolution) make it suitable to be used in an ergonomic way together with the microscope. The

architecture is shown in Figure 54.

FIGURE 55: SIMULATOR. LEFT: BRAIN TISSUE DEFORMATIONS. RIGHT: COMPLETE PROTOTYPE.


Ch

apte

r 6

66

6.2 SOFTWARE ARCHITECTURE

Figure 56 describes the simulation software architecture. The rendering software developed in C++ and Python

is built on the open source GPU licensed and cross-platform H3D [12], a scene-graph API based on OpenGL for

graphics rendering, OpenHaptics for haptic rendering and X3D [13] for the 3D environment description. All the

components are open source or at least based on a GPL license.

FIGURE 56: SOFTWARE ARCHITECTURE OF MENTIS

6.3 SCENE GRAPH API: H3D

H3D API is an open-source, cross-platform, scene-graph API. It’s written entirely in C++ and uses OpenGL for

graphics rendering and OpenHaptics for haptic rendering. H3D has been carefully designed to be a cross-

platform API. The currently supported operating systems are Windows XP, Linux and Mac OS X, though the

open-source nature of H3D means that it can be easily ported to other operating systems. It is designed to

support a special rapid development process and several specialised haptic interface and immersive displays.

By combining X3D, C++ and the scripting language Python, H3D offers three ways of programming applications.

Execution and development speed are critical aspects well supported by this library. It offers haptic extensions

to X3D to write hapto-visual applications for tactile and visual feedback. H3D is tighter with Chai3D and the

“defacto” industry standard haptic library OpenHaptics, developed and maintained by SensAble Technologies

Inc., one of the few relevant haptic library. Immersive workstations and a wide variety of VR display systems

are also supported.

H3D is built using many industry standards including:

X3D [72], the Extensible 3D file format that is the successful successor to the now outdated VRML

standard. X3D is an ISO open standard scene-graph design that is easily extended to offer new

functionality in a modular way. It’s an open software standard for defining and communicating real-

H3D API

OpenHaptics

OpenGL

PythonC ++

H3D API

OpenHaptics

OpenGL

PythonC ++

User

Hardware

X3D


Ch

apte

r 6

67

time, interactive 3D content for visual effects and behavioural modelling.

XML (Extensible Markup Language) is the standard markup language used in a wide variety of

applications. The X3D file format is based on XML and H3D comes with a full XML parser for loading

scene-graph definitions.

OpenGL (Open Graphics Library) is the cross-language, cross-platform standard for 3D graphics. Today,

all commercial graphics processors support OpenGL accelerated rendering and OpenGL rendering is

available on nearly every known operating system.

STL (Standard Template Library) is a large collection of C++ templates that support rapid development

of highly efficient applications.


Ch

apte

r 6

68

6.4 3D MENTIS ENVIRONMENT RECONSTRUCTION

For the image processing (segmentation/classification) of the region we have evaluated Osirix and 3DSlicer

obtaining similar results and we have decided to use the second because it is platform-independent and

versatile. The model data is represented initially in the vtkPolyData format provided by the Visualization Toolkit

(VTK). This is the output of the segmentation process and modelling from 3DSlicer using marching cubes and

Delaunay triangulation (described in chapter 2). At this instance the reference model as well as the MRI data

has to be matched onto the patient's anatomy. The model file is converted in X3D and imported in our

application. High-resolution texture is applied on the region of interest for the simulation. The two

environments (3DSlicer and Mentis) are registered after the patient registration procedure (see Chapter 7).

This step is required to understand the correct transformation matrix which has to be assigned to the 3D

environment. Figure 57 resumes the steps used to obtain a complete 3D environment for the surgical

simulation discussed in the present research work.

FIGURE 57: 3D ENVIRONMENT DEVELOPMENT STEPS

The scene, described using X3D. This introduces a structural division in the scene-graph concept: the use of

nodes and fields. Fields are data containers that know how to store and manipulate data properties. Nodes are

essentially containers and managers of fields and all node functionality is implemented in terms of fields. A

specific node has been created to support simulation needs in Mentis. It manages the data coming from the

tracking system and from the haptic interface in two separate threads. They are both actively used for the

simulation in real-time. Position and orientation of the microscope and haptic interface are updated as field of

the node and routed to the 3D model positions. The feedback from the haptic interface is deeply related to the

physical model and the collision detection strategy. This topic will be shown in the following paragraphs.


Ch

apte

r 6

69

6.5 PHYSICAL MODELLING AND HAPTICS

The physical model of the patient head can be realistic simulated by subdividing it into different layers. Each of

these can be described by different topologies of points and springs (as described in the previous chapter) that

give different deformations. Figure 58 shows several head layers and how, the anatomy, can be described in

different mass spring damper configurations depending by the tissue properties. This concept is valid for both

volumetric and surface modelling strategy.

FIGURE 58: DIFFERENT MASS SPRING TOPOLOGIES FOR DIFFERENT LAYERS OF TISSUE.

Force feedback interaction in the simulator was developed upon HAPI, an open-source, cross-platform, haptics

rendering engine written entirely in C++. Main features of this library are:

it is device-independent and supports multiple currently commercial haptic devices;

it is highly modular and easily extendable;

it is naturally useable in H3D;

it is extendable adding, substituting or modifying any component of the haptic rendering process;

it provides classes for collision handling (i.e. axis-aligned and oriented bounding box) used in the

haptics rendering algorithms.

HAPI is device independent and can be used with a wide variety of haptics devices such as Phantom devices

from Sensable Technologies, Delta and Omega device from ForceDimension, Falcon from Novint, HapticMaster

from Moog/FCS. The management of the main differences between haptic devices is almost transparent to the

user and this simulator can be simply adapted to all of the previous haptics.

Figure 59 shows the workflow generally used by HAPI implementations. Values of haptic device are updated by

effects and forces (or/and torques) specifications and used to compute the new forces to be sent to the haptic

interface.


Ch

apte

r 6

70

FIGURE 59: WORKLOW OF HAPTIC RENDERING IN HAPI (FOLLOWING THE SENSEGRAPHICS SPEC.)

Haptic and graphic threads have different frame rates. Please note that the haptic frame rate need 1000 FPS to

be realistic and in other hand the graphical FPS has a minimum requirement of 25-26 FPS in order to be

considered as a continuous flow by human eyes. This difference suggest using different threads and invites to

another important consideration: every system that involves haptics is naturally exposed to have bottleneck

related to the haptic frame rate (Figure 60). In this scientific work, haptic rendering is not the only source of

delay because the tracking system rate has to be carefully controlled during the simulation workflow.

FIGURE 60: THREAD COMMUNICATION. NOTE THAT HAPTIC FREQUENCY IS HIGHER THEN GRAPHICS ONE.

HAPI was used for rendering of different surface layers on one haptic device. Each layer has its own haptics

rendering algorithm and it is possible to specify shapes for different tissue layers in different haptic layers. For

example, in a simulation of the head, one layer can be for the skin, one layer for the bone and another for the

brain surface. This makes it possible to feel the hard bone through the soft skin. In addition, the use of different

layers accurate distributed in the space can improve the realism of the sense of touch. In this case the finally

rendered force will be the sum of all interacting surfaces (layers). Figure 61 shows (on the right) that the two

surfaces are both part of layer 0. The proxy is then stopped at the first surface, and the underlying surface will


Ch

apte

r 6

71

never be felt. On the left image, the two surfaces are put in two different layers, which each have its own

proxy. It means it perceptible.

FIGURE 61: HAPTIC SURFACE RENDERING AND DIFFERENT LAYERS.

6.6 BUILDING A NEW PHYSICAL MODEL

The physical model described before has some limitations. The most relevant of these is that the model does

not take in account the whole object but only the contact area on the surface. That is not so relevant if we

consider very local deformation which really the case of the brain during palpation or if we consider rigid

bodies.

Since we wanted to have a more realistic model then we expanded the possibilities offered by H3D applying a

modified version of mass spring damper model to the patient brain.

At this instance, the virtual environment has been also described using X3D.

In order to obtain deformations of the organ, which are correctly situated and react in a way similar as possible

to the real brain, a three tiered structure of springs has been built; each tier has been modelled using the mass-

spring method.

Together with the external layer of springs, two other layers (identical in shape and inner to the external

surface but reduced in size) have been modelled. All the nodes of each inner tier are connected to the

corresponding points of the tier immediately above by means of springs and dampers. This approach is similar

to that one described in [73] for real-time simulation of complex organ modelled with multiple surface mass

spring damper model. Using multiple surfaces and internal pressure improve the realism and simulate

volumetric modelling effects.

By adding these other inner surfaces within the first one, it is possible to obtain more accurate deformation

effects simulating the behaviour of the brain correctly. We obtained local and realistic deformations using an

ad-hoc point distribution in the volume where the contact between the brain surface and a surgical instrument

takes place.

The external layer is provided with geometrical and haptic rendering, the second one, without rendering, has

the same shape as the first one but is scaled down by a factor equal to 1,2; the third layer, with the same shape

but scaled down by a factor equal to 2, is made up of fixed and rigid nodes (Figure 62).

In addition, it is possible to modify at run-time the parameters of mass spring model (spring, mass and damper

coefficients) using a specific menu.


Ch

apte

r 6

72

Although the model ends up being slightly heavy in terms of computational time when using this type of

physical modelling, the level of realism increases when compared to a model with a single layer and the

dynamic behaviour is closer to the real one.

Some tests have been performed to estimate the computational time of the algorithm executed on inputs of

growing complexity.

A PC has been used with processor Intel Core2 CPU 6600 2.40 GHZ, 1Gbyte RAM, video card NVIDIA GeForce

8800 GTS, Windows XP Pro operating system; the haptic device used is the SensAble PHANTOM Desktop with

the OpenHaptics library.

The preliminary tests have been carried out using as a model a sphere made up of 1.926 nodes and 8.964

springs (Figure 62); the obtained frame rate was between 59.9 fps and 60.6 fps.

FIGURE 62: STRUCTURE DEFORMATION FOR THE 3-LEVEL MASS-SPRING-DAMPER (LEFT) AND SURFACE DEFORMATION APPEARANCE

(RIGHT)

Figure 62 shows the deformations using the haptic interface when a point is pulled.

The following tests have been carried out on the brain model made up of 27531 nodes and 128604 springs

(Figure 63); the obtained frame rate was between 6.9 fps and 7.4 fps. This frame rate is inclusive of both

microscope tracking and tool rs31position data sending to 3DSlicer on the local area network.


Ch

apte

r 6

73

FIGURE 63: MODEL OF THE BRAIN EXTERNAL SURFACE

FIGURE 64: BRAIN DEFORMATIONS AFTER A COLLISION WITH A SURGICAL TOOL

By increasing the number of points, the graphical realism of the simulation increases as well, but it is necessary

to find a trade-off with the requirements of real-time interactions. In order to obtain at the same time a very

high realism of the surface deformation and real-time interactions, it is necessary to increase the number of

nodes and springs and, consequently, the numerical time integration of spring displacements needs to be

accelerated. To fulfil this requirement, the exporting of the developed model onto a multi-processor

architecture or the exploitation of the features of recent graphics accelerators to simulate spring elongation

and compression on the CUDA [74] could be considered as future extension. Figure 64 shows the brain tissue

deformations after a collision with a surgical tool in the virtual environment. In this last case the brain model is

mapped with a realistic texture.


Ch

apte

r 6

74

6.7 COLLISION DETECTION

The collision between 3D object in the scene in inherited by the general strategy applied in H3D. MENTIS

graphics loop runs with a frame rate of ca. 100Hz using the haptic device position given by the haptic loop (ca.

1000Hz). For this reason is obvious to think to adopt a general strategy for collision detection in the graphical

loop to determinate, step by step, all the objects or primitives that can collide. This is just a rough estimation,

at graphics thread rate, on which primitives are in close proximity to the haptic device. All triangles, lines or

points far within a certain radius from the haptic proxy position (plus expected movement) are collected and

sent to HAPI for use in haptic thread (haptic loop). The collisions are detected at haptic rate in order to do

haptic rendering.

Two strategies are implemented AABB and OBB. They have been both described in chapter 5. It is possible to

specify which one has to be used just modify the X3D description of the field boundType in the simulator file

description. AABB is used for default because it has been provided better results.

FIGURE 65: COLLISION DETECTION. BOUNDING OF THE PATIENT HEAD WITH OBB (TOP LEFT) AND AABB (TOP RIGHT), DETAILS ABOUT

VENTRICLES BOUNDING WITH OBB (DOWN)


Ch

apte

r 6

75

6.8 USER INTERFACE

The user interface was designed to be simpler as possible. Basically is possible to select different training

environment. A different patients, anatomy, disease, organ tissue or surgical tools characterize each of this.

The 3D real time rendering is divided in three different views: one for the monitor and one for each microscope

oculars. Each organ or region of interest has different colour for a better anatomical learning process. Only the

difference between healthy brain tissue and low-grade glioma is not graphically evidenced because it is the

main part and aim of the training. The first step of the simulation is to choice a simulation scenario just loading

a X3D file. This file has a description of the virtual patient and tools (surgical instruments). When the file is

loaded the tracking system start to track the microscope position and the haptic rendering thread is turned on.

Several visualization options are available for the stereo rendering in the microscope oculars and at the screen

(vertical and horizontal split or red-blue, red-green, red-cyan stereo for 3D glasses). The haptic rendering

options are inherited from H3D. Is it possible use one of the following library or haptic method: Openhaptics

(default), Chai3D, Godobject and Ruspini and for each of them specify several parameters. The mouse is

basically used to navigate in the 3D world (rotation and translation) but it is also possible use the haptic

interface to rotate the patient. Bounded boxes tree used for the collision detection can be visualised to

facilitate a better understanding of the organs and tool positions. The user interface is completed by the 3D

slicer rendering of the same virtual patient (but without real-time deformation), tool and preoperative images

on a Macintosh connected by local area network.

FIGURE 66: TWO DIFFERENT VISUAL RENDERINGS: MONITOR OUTPUT (LEFT) AND STEREOSCOPIC VIEW INSIDE THE MICROSCOPE

OCULARS.


Ch

apte

r 6

76

6.9 INTEGRATION OF 3DSLICER

3D Slicer is free open-source and multi-platform software for visualization, registration, segmentation, and

quantification of medical data for diagnostic purposes. It provides different features including:

Sophisticated complex visualization capabilities

Multi-platform support: pre-compiled binaries for Windows, Mac OS X, and Linux

Extensive support for IGT and diffusion tensor imaging

Advanced registration / data fusion capabilities

Comprehensive I/O capabilities.

FIGURE 67: 3DSLICER. TYPICAL SCENARIO: 3D RECONSTRUCTED SURFACE OF ORGANS OVER IMPOSED ON MEDICAL IMAGES.

Slicer has been used in clinical research (after created appropriately validated clinical protocols). In image-

guided therapy research, it permits to construct and visualize collections of MRI, CT, fMRI datasets and for

ultrasound navigation. In the normal procedures all the data are available pre- and intra-operatively especially

in order to acquire spatial coordinates for tracking systems.

Standard image file formats are supported, and the application integrates interface capabilities to biomedical

research software and image informatics frameworks. The system is platform independent (Windows, Linux,

Macintosh are supported).

In particular, the Image Guided Therapy Toolkit developed at the Brigham and Women's Hospital [75], is a set

of open source software tools integrated with supported hardware devices for MR-guided therapy. This toolkit


Ch

apte

r 6

77

(see Figure 68 is based on a middleware (OpenIGTLink) on which several data type can flow (basically tool or

object position coordinates). It is able to connect 3DSlicer with several imaging devices, tracking devices and

medical robots.

FIGURE 68: OPENIGTLINK

6.10 INTEGRATION OPENIGTLINK-TRACKERBASE

TrackerBase is a library to interact with tracking system developed in a previous work at our laboratories

(Medical Group of the Institute for Process Control and Robotics, University of Karlsruhe, Germany). It supports

different tracking systems (Polaris from NDI in particular) under different operating systems. Basically it works

taking the data string from the tracker and giving back a transformation matrix describing the movement of all

the tracked tools in the space. A server module to take data from Polaris (using TrackerBase) and pass them to

OpenIGTLink has been developed in this scientific work and the client code was tested with 3DSlicer. In this

way, using this middleware and sending data using local area network or internet we obtained:

a connection between MENTIS and 3DSlicer which permit to share same geometries (only rigid

objects) and coordinates;

a distributed architecture with the separation between tracking, rendering and computation load

running on different PCs (on different IP address and ports). For instance, 3DSlicer runs on a

Macintosh sharing data (3D and tool position) with Mentis (running on Windows XP). This improves

performances and realism.

In addition, OpenIGTLink is built up IGSTK, which has support for practically all the common tracking systems.

This means that our (mixed reality) application it will work as client for all the trackers supported by

OpenIGTLink just changing few settings (few code lines).


Ch

apte

r 6

78

FIGURE 69: DATA COORDINATES FLOW


Ch

apte

r 6

79

6.11 WEB PORTABILITY

Portability is usually used to indicate the ability of a software system to execute properly on multiple hardware

platforms.

The 3D virtual environment rendered inside the simulator is completely described in X3D language. This is the

successful successor of VRML (Virtual Reality Modelling Language) and it is considered the “de facto” standard

for virtual web technology. It means that the complete 3D data set (patient and tools) can be naturally web

rendered using all the common web browser and plug-ins (see Figure 70). A consequence is that it is possible to

share the 3D data set simply embedding it in a web page. This is useful to study the patient doing a sort of

intuitive planning or anatomical studies. The application can be in a simple way extended to be a real

distributed training. A typical scenario is to have one expert surgeon in one location evaluating the training of

students located in different sites. Another can be the possibility to follow the surgical movements of the

surgeon rendered directly on the student hands. In this case, it is required to collect the complete workflow-

gesture of the surgeon and send this data to the student location to be rendered on-site. In this way it is

possible to share the 3D data set simply embedding it in a web page. This is useful to study the patient doing a

sort of intuitive planning or anatomical studies.

FIGURE 70: 3D VENTRICLES RENDERED IN THE SIMULATOR CAN BE VIEWED AND NAVIGATED IN A NORMAL WEB BROWSER THANKS TO

THE X3D STANDARD. THAT CAN PERMIT THE SHARING OF THE VIRTUAL PATIENT FOR MEDICAL CONSULTING OR FOR DISTRIBUTED

TRAINING.

http://en.wiktionary.org/wiki/software

http://en.wiktionary.org/wiki/system

http://en.wiktionary.org/wiki/execute

http://en.wiktionary.org/wiki/hardware

http://en.wiktionary.org/wiki/platform


Ch

apte

r 6

80

6.12 PERFORMANCES

A demonstration has been set up with a complex scenario (Fig. 72). Patient skin, brain, ventricles, vessels and

tumour are graphically rendered together with a 3D model of the tracked active tool (surgical instruments). The

tests have been run on Intel Core2 CPU 6600 2.40 GHZ, 1Gbyte RAM with a video card NVIDIA GeForce 8800

GTS and the SensAble PHANTOM Desktop.

The experimental data show the Fps are enough good to provide a realistic simulation (Fig. 73)

FIGURE 71: VIRTUAL SCENARIO COMPOSITION

FIGURE 72: PERFORMANCES

Performances

70 66

40 35 30 25

0

20

40

60

80

4689

5684

1433

9

5530

8

7433

6

1056

08

Complexity of the virtual

environment

Fp

s

Fps


Ch

apte

r 6

81

FIGURE 73: VIRTUAL BRAIN-TOOL INTERACTION


Ch

apte

r 6

82

6.13 PRELIMINARY RESULTS IN THE VALIDATION OF THE SIMULATION

As anticipated in paragraph 5.2 the physical description of the biomechanics of the soft tissues by on vivo

measurement is a difficult task. This limitation is even bigger in the specific case of the brain which is one of the

most complex organs in the human body. We have decided to define the tissue parameters, in an empiric way,

using directly the surgeon tactile experience in several tests of the simulator.

Simple customizable simulator scenarios were built for a first model validation and two evaluation sessions

have been organized in the hospitals of Günzburg and ULM (Germany). Several objects (segmented brain or

primitives) characterized by different tissue properties were collected in a functional demo. Fifteen, between

expert surgeons and assistants, were invited to test the haptic interaction between a virtual spatula and

different tissues. Their impression and suggestion were acquired and after some demo sessions, the suitable

simulation parameters were identified and isolated. In the final demo, they were able to recognize clearly and

to explore the different between normal brain and tissue affected by tumour. At this instance a 3D patient was

reconstructed by real patient images and a LGG was hidden inside the brain model. Three parameters were

studied and used to give different haptic feedback for each type of tissue: stiffness, static friction and dynamic

friction. The experiments show a realistic simulation can be obtained with a brain tissue with a stiffness of

0.002 and a tumour with stiffness of 0.03. The better values for dynamic and static friction are almost close to

zero for both tissues. In order to increase the realism of the simulation (Figure 73: Virtual brain-tool

interaction) a periodical movement was applied to the brain dynamic model, simulating the blood circulation.

The 3D visual scenarios of the virtual patient and instruments were considered highly realistic. In addition,

haptic feedback was enough realistic to distinguish LGGs and healthy brain parenchyma. The best good results

were obtained for the simulation of the interaction with the virtual skull. Results reported by the experiment

showed that the prototype haptics-based simulator was realistic enough to serve as a useful instruction tool

with high teaching potential on neurosurgery procedures (Figure 75: Results. Enough realism of the

deformations of the brain tissue and of the LGG. 15 surgeons defined the system realistic enough to be used

for training.

FIGURE 74: MEDICAL EVALUATION IN GÜNZBURG AND ULM HOSPITALS


Ch

apte

r 6

83

FIGURE 75: RESULTS. ENOUGH REALISM OF THE DEFORMATIONS OF THE BRAIN TISSUE AND OF THE LGG. 15 SURGEONS DEFINED THE

SYSTEM REALISTIC ENOUGH TO BE USED FOR TRAINING.

0123456789

Realism

1 3 5 7 9 11 13 15

Medical Doctors

Evaluation Results

Brain-LGG

palpation

Skull interaction


Ch

apte

r 7

84

CHAPTER 7

EXTENSION TO INTRA-OPERATIVE AUGMENTED

REALITY


Ch

apte

r 7

85

7.1 INTRODUCTION TO THE INTRA-OPERATIVE AUGMENTED REALITY

The best commercial systems (i.e. Brainlab and Stryker) provide the neurosurgeons with only a real two-

dimensional overlay of the region of interest (ex. tumour) inside the oculars of the operating microscope

related on the preoperative processed patient’s image data. The three-dimensional environment

reconstruction from 2D is another difficult and critical mental work for the surgeon. There were only two

working examples of AR 3D stereoscopic microscope for neurosurgery. The first was described in (Edwards, P.

et al., 2000) and the second was developed in our laboratories (Aschke et al. 1999). We improved this previous

work extending it with a higher-level graphic modality and enhancing its real-time performances. Actually, the

architecture described before, can be used for intra-operative purposes. In this instance, a surgeon needs to

use the microscope, monitors and surgical tools. This is the basic setup for an image guided therapy

interventions. The same virtual environment can be AR rendered in to the microscope optics with the

difference that now the complete anatomy is considered rigid (deformations are not requested in this step).

FIGURE 76: SYSTEM ARCHITECTURE PROTOTYPE.

The haptic interface is no longer required and is replaced by new navigated infrared active tools. The prototype

(Figure 76: System architecture prototype.) is capable of tracking, in real time, the microscope, the patient's

head and one or more surgical tools (pointers with active or passive markers).

It provides two different video renderings to support the surgeons during the intra-operative phase (Figure 77):

3D region of interests (tumours or organs) inside microscope oculars and the 3D view on the screen related

with patients’ real images (screen).


Ch

apte

r 7

86

FIGURE 77: VIDEO FEEDBACK FOR THE SURGEON: MICROSCOPE AND SCREEN.

FIGURE 78: OCULAR VIEW: REAL VIEW (LEFT) AND AR VIEW (RIGHT). THE YELLOW LINE IS THE CONTOUR OF THE CRANIOTOMY AREA ON

THE PHANTOM SKULL.

Figure 78 shows the augmented reality view inside the microscope oculars in which is possible identify 3D

region of interest (in this example the brain surface is rendered). The microscope hardware related part was

realized at our institute and described in the previous mentioned work [4]. Registration and camera calibration

are required for a perfect alignment between real and virtual world. Both are off-line steps with similar

approaches used in [4] but solved using Matlab.


Ch

apte

r 7

87

7.2 PATIENT REGISTRATION

Registration is required in order to alignment all the different coordinates systems involved in the augmented

reality world. The position of the phantom patient positioned under the microscope is given from the tracking

system. The ICP algorithm is adopted with four input points and solved in order to apply the transformation

and to align the real position to the 3D model. The implementation used in this application is an adaptation of a

C++/Matlab code found at [76]. The algorithm can work with a point clouds which makes it theoretically

possible to make it works with all the points of the 3D surface and other taken on the patient phantom,

thereby improving the accuracy. Points are acquired with the NDI active tool. The tip of the tool is computed as

offset performing the pivoting (for instance is 290mm from the centre of the transformation matrix of the

tool). The obtained roto-translation matrix is converted in Euler notation and applied to the rendered model.

Four points have been used for the registration algorithm and then rotation and translation matrix have been

acquired and applied to the 3D object for the alignment.

Patient registration step is repeated by default in the beginning on every patient navigation session in order to

provide the needed accuracy for intra-operative augmented reality.

Since the virtual cameras must be located in the right viewpoint position, registration step in not only required

for the patient but for the microscope too. This step has to be paired with the calibration process.

7.2.1 RESULTS The registration error analysis (following the nomenclature in Chapter 3) has shown a FLE of 0.35 mm (given by

the Polaris NDI tracking system) and a FRE of 0.10 mm given by the distances between original model points

and re-projected points after registration. In the following figure, we report the effect of ICP registration used

in MENTIS applied to the patient phantom.

FIGURE 79: ICP REGISTRATION APPLIED TO A 3D MODEL OF THE PATIENT (LEFT, BEFORE AND RIGHT AFTER THE ALIGNMENT).


Ch

apte

r 7

88

7.3 MICROSCOPE CAMERA CALIBRATION

Camera calibration is a necessary step in 3D computer vision in order to extract metric information from 2D

images. For an optical see-through AR system there is no way to access the augmented image on the user’s

retina, it is not possible to use traditional image-based measurement methods to determine system accuracy.

Thus far, there have been several approaches to accuracy verification for optical see-through displays.

The standard method uses a camera in place of the human eye and conducts image-based measurements to

assess the accuracy of the alignment [33] [34]. It is performed easily by mounting a camera to the HMD so that

it “sees” through the display panel. This is not very accurate anyway since a camera is only an approximation of

the human eye.

In the specific case of the intraoperative microscope, calibration procedure is needed in order to find the right

position and orientation of the semi-transparent mirror inside the microscope. It is performed attaching two

cameras on the microscope oculars (Figure 80) and taking different shots of a specific pattern (it will be

described in the following paragraphs). This specific hardware setup has been presented in [4].

FIGURE 80: TWO CAMERAS ATTACHED ON MENTIS (LEFT) AND TRACKED PATTERN (RIGHT)

For an accurate alignment of the 3D image inside the microscope view, several coordinates systems are

involved. The transformation matrix involved (Figure 81) are provided by tracking and camera calibration

procedure.

The coordinates systems used are:

wx is the system of the calibration pattern;

sx denotes the system of the tracker probe mounted to the pattern;

hx is the system of the tracker probe mounted to the microscope;


Ch

apte

r 7

89

dx is the system of the semi-transparent mirror of the microscope;

cx is the camera system is the coordinate system of a video camera located behind the eyepiece of

the oculars.

FIGURE 81: MICROSCOPE CALIBRATION SCHEME OF MENTIS

Denoting the rigid body transformation, consisting of a rotation and a translation from the system A to B by

ABT , we see (Figure 81) that for the calibration of the Microscope the transformation:

TMIC = TCDTWCTSWTHS

The effective focal length of the pinhole camera model projecting the camera system into the video camera has

to be determined. SWT is determined applying point to point registration (ICP). The point pairs were obtained

by measuring known pattern positions with the optical tracker (Polaris, Northern Digital Inc., Canada). HST is

given by the tracking system. WCT is provided by the camera calibration routine. The most common used

technique [35] only requires the camera to observe a planar pattern shown at a few (at least two) different

orientations. The pattern can be printed and attached to a planar surface.


Ch

apte

r 7

90

7.4 SINGLE OCULAR CALIBRATION

7.4.1 METHODS The camera calibration was carried out using the stereo camera calibration tool in Matlab [77] based on [35]. A

pattern (see Figure 82) has been used taking 35 screenshots from the microscope in different poses and from

each ocular. For this aim, two cameras have been attached to the oculars.

FIGURE 82: CALIBRATION IMAGES. SEVERAL IMAGES FOR DIFFERENT ANGLES AND POSITIONS WERE ACQUIRED FOR EACH OF THE

OCULARS.

Using the standard procedure, before running the camera calibration algorithm, a precisely localisation of each

square and corner of the pattern is requested.

This identification is semi-automatic. Figure 83 shows the accurate results.

FIGURE 83: ON THE LEFT CALIBRATION PATTERN AND ON THE RIGHT WITH THE DETECTED CORNERS (RED CROSSES) AND THE

REPROJECTED GRID CORNERS (CIRCLES)


Ch

apte

r 7

91

After this image processing step, the Zhang calibration method has been applied to obtain intrinsic (focal

length, principal point, skew, distortion, pixel error) and extrinsic parameters (ocular position related to the

pattern).


Ch

apte

r 7

92

7.4.2 RESULTS In the following table the results are reported for the left and right camera separately. The numerical errors are

approximately three times the standard deviations.

Left calibration results after optimization (intrinsic parameters with uncertainties):

Focal Length:

fc = [ 4203.97384 4337.54251 ] ± [ 78.48042 114.14546 ]

Principal point:

cc = [ 684.66643 899.86958 ] ± [ 13.01974 94.47382 ]

Skew:

alpha_c = [ 0.00000 ] ± [ 0.00000 ] => angle of pixel axes = 90.00000 ± 0.00000 degrees

Distortion:

kc = [ 1.10639 -1.65233 0.14500 0.00262 0.00000 ] ± [ 0.21054 2.83326 0.02980 0.00469 0.00000 ]

Pixel error:

err = [ 0.70659 0.57028 ]

Right calibration results after optimization (intrinsic parameters with uncertainties):

Focal Length:

fc = [ 4210.92373 4348.51217 ] ± [ 77.52672 113.49222 ]

Principal point:

cc = [ 627.54642 889.39840 ] ± [ 12.54792 92.40800 ]

Skew:

alpha_c = [ 0.00000 ] ± [ 0.00000 ] => angle of pixel axes = 90.00000 ± 0.00000 degrees

Distortion:

kc = [ 1.17955 -3.44078 0.14742 0.00183 0.00000 ] ± [ 0.20869 3.44340 0.02922 0.00416 0.00000 ]

Pixel error:

err = [ 0.71155 0.57007 ]


Ch

apte

r 7

93

The results show an acceptable pixel error for both oculars even with a presence of distortion components

(especially on the right ocular). A results refinement is always possible especially after a deep reprojection

error analysis (Figure 84). Since there are images responsible sources of calibration error (points far from the 0

components of x and y axis) is it possible to eliminate them increasing the calibration accuracy.

FIGURE 84: ERROR ANALYSIS: REPROJECTION ERROR (IN PIXEL) FOR THE LEFT OCULAR (UP) AND RIGHT (DOWN).


Ch

apte

r 7

94

Distortion coefficients (radial and tangential) are shown in Figure 85 for the right ocular (similarly for the left

one).

FIGURE 85: DISTORSION. RADIAL (UP) AND TANGENTIAL (DOWN) FOR THE RIGHT OCULAR (SIMILAR RESULTS FOR THE LEFT OCULARS).


Ch

apte

r 7

95

Microscope calibration gives an important output on the position of the two cameras. These are extrinsic

parameters shown in the following diagrams in camera- and pattern-centred views. This data have been used

to locate the virtual camera in the augmented reality scene.

FIGURE 86: EXTRINSIC PARAMETERS FROM LEFT OCULAR (SIMILAR RESULTS FOR THE RIGHT ONE).


Ch

apte

r 7

96

7.5 STEREO CAMERA CALIBRATION

7.5.1 METHODS Camera calibration results for single oculars must be refined and both used for the stereo calibration. This is

required since translations and rotations of the both oculars, one respect to the other have to be known. The

previous Zhang calibrations results have been used as inputs and the following table shows the relative position

of the right ocular respect to the left one (see Figure 87). Interocular and focal distances have been directly

imported in MENTIS for stereo rendering.

FIGURE 87: EXTRINSIC PARAMETERS FOR THE STEREO CALIBRATION. THE TWO OCULARS POSITION ARE SHOWN REFERRING TO THE

CALIBRATION PATTERN.


Ch

apte

r 7

97

7.5.2 RESULTS In this paragraph, results of the stereo calibration procedure are reported.

Stereo calibration parameters after optimization:

Intrinsic parameters of left camera:

Focal Length:

fc_left = [ 4322.26607 4496.30076 ] ± [ 51.46770 100.69633 ]

Principal point:

cc_left = [ 683.11133 955.36666 ] ± [ 8.35938 88.90927 ]

Skew:

alpha_c_left = [ 0.00000 ] ± [ 0.00000 ] => angle of pixel axes = 90.00000 ± 0.00000 degrees

Distortion:

kc_left = [ 1.09222 -0.31383 0.16064 0.00359 0.00000 ] ± [ 0.19059 2.03602 0.02737 0.00209 0.00000 ]

Intrinsic parameters of right camera:

Focal Length:

fc_right = [ 4305.59030 4425.51372 ] ± [ 49.67069 86.61071 ]

Principal point:

cc_right = [ 619.64537 854.29780 ] ± [ 9.24889 86.56662 ]

Skew:

alpha_c_right = [ 0.00000 ] ± [ 0.00000 ] => angle of pixel axes = 90.00000 ± 0.00000 degrees

Distortion:

kc_right = [ 1.17554 -2.77489 0.13590 -0.00040 0.00000 ] ± [ 0.19757 3.89041 0.02781

0.00234 0.00000 ]

Extrinsic parameters (relative position of right camera vs. left camera):

Rotation vector:

om = [ -0.02308 0.08588 0.01359 ] ± [ 0.02214 0.00247 0.00128 ]

Translation vector:

T = [ -25.99359 -0.37782 1.50767 ] ± [ 0.15459 0.19000 1.99171 ]


Ch

apte

r 7

98

7.6 AUGMENTED REALITY SCENARIO VALIDATION

A demonstration has been set up with a complex scenario (similarly to the previous section).

Patient skin, brain, ventricles, vessels and tumour are graphically rendered both with a 3D model of the active

tool (surgical instruments).

Results shown average frame rate of 30 Fps when the scene is rendered in a single view on one screen. This

value goes down to 15 Fps when the scene is rendered in stereo in the microscope oculars. This rendering time

computation includes the tracking system activity (to locate microscope, patient and active tool) and the delay

of data communication with 3DSlicer on the local area network.


Ch

apte

r 8

99

CHAPTER 8

CONCLUSION AND FUTURE WORK


Ch

apte

r 8

100

8.1 SUMMARY

This thesis has presented the development of the first mixed reality system for training and intra-operative

purposes in neurosurgery based on a real surgical microscope and on a haptic interface for a better visual and

ergonomic realism.

We began with a description of the medical background, workflows, and technologies and of course

motivations (for using both the virtual and the augmented reality in neurosurgery). After discussing the state of

art in training and augmented systems in medicine, we focused the dissertation on the methods that we used

for haptic and visual rendering, real-time tracking, patient registration and microscope calibration. In the

following paragraph, a conclusion shall be made on the basis of the results in chapters 6 and 7, while

differentiating them for the each task.

8.2 TASK 1: MICROSCOPE EMBEDDED NEUROSURGICAL TRAINING SYSTEM

The main training task is the advanced simulation of brain tissue palpation enabling the surgeon to distinguish

between normal brain tissue, and that tissue affected with a Low Grade Glioma (LGG). This task has been

motivated by the fact that the ability of a surgeon (to feel the difference in consistency between tumours and

normal brain parenchyma) requires considerable experience and it is a key factor for a successful intervention.

Force feedback interaction with soft and hard tissues is provided to the surgeon’s hands in order to provide a

complete immersive experience in our simulator prototype. To meet the neurosurgeon needs we have

developed a system, which includes:

1. simulation of the virtual patient inside the real microscope oculars;

2. force feedback rendering directly at the user’s hand;

3. parameterization of the tissue properties and physical modelling;

4. a module for the system connection with an high level intra-operative navigation software (3DSlicer).

Our approach to build and to locate of deformable objects emerged as our simulator progressed toward

incorporating soft tissues (organs) and surgical instruments that were both patient-specific and realistic. At

these considerations our prototype it is a simulator of the second generation (according with the definition in

3.1).

The virtual brain has been segmented from CT images of the real patient with a LGG with standard and

validated techniques and using the open source 3DSlicer. The 3D objects have been exported in X3D format

and loaded in the application. The microscope oculars positions (coming from the tracker system) have been

used as viewpoints of the stereo scene.

We registered a final graphical frame rate of ca. 26 Fps and a haptic frame rate of ca. 1000 Fps for a complex

3D scenario composed of brain and tumour as deformable objects (688634 polygons) and skin and surgical

instruments (8882 polygons) as rigid. Considering that 25 Fps for the human eye and 1000 Fps for the haptic

perceptions are the limits of feeling the continuity we are able to conclude that the performances of our


Ch

apte

r 8

101

prototype are suitable to be used in a realistic patient-specific simulator. Please note that these frame rates

includes also all the delays related to the tracking system data-flow and the time to send data using the LAN to

3DSlicer.

8.3 TASK 2: EXTENSION OF THE PLATFORM TO AUGMENTED REALITY MICROSCOPE FOR

INTRAOPERATIVE NAVIGATION

The architecture described above can be adapted for intra-operative purposes. In this instance, a surgeon

needs the basic setup for the IGT interventions: microscope, monitors and surgical tools. The same virtual

environment can be AR rendered onto the microscope optics with the difference that now the complete

anatomy is considered rigid (deformable organs are not requested in this frame since only geometrical

information are required). At this instance, new navigated infrared active tools replace the haptic interface.

The prototype is capable of tracking, in real time, the microscope, the patient's head and one or more surgical

instruments (pointers with active markers).

Inside the microscope oculars, it is possible to identify the 3D region of interest (the brain surface and

craniotomy area, tumour or important organs).

Registration with ICP and Zhang camera calibration are carried out with standard procedures. Both are off-line

steps and required for a perfect alignment between real and virtual world at the microscope view.

8.4 OTHER GENERAL CONSIDERATIONS

This research focused essentially on the software development of a common architecture for both AR and VR

systems directly on OpenGL and H3D to have high rendering performances.

The microscope hardware related part is not part of this dissertation because, as mentioned before, it was

realized at our institute, in a previous excellent work (Aschke et al. 1999).

A server module for transferring data from Polaris (using TrackerBase) to OpenIGTLink was developed and the

client code hes been tested with 3DSlicer. In this way, using this middleware and sending data using local area

network or internet we obtained:

a connection between MENTIS and 3DSlicer which permit to share same geometries (only rigid

objects) and coordinates;

a distributed architecture with the separation between tracking, rendering and computation load

running on different PCs (on different IP address and ports). For instance, 3DSlicer runs on a

Macintosh sharing data (3D and tool position) with Mentis (running on Windows XP). This improves

performances and realism.


Ch

apte

r 8

102

In addition, OpenIGTLink is built on IGSTK, which provide support for practically all the common tracking

systems. This means that our (mixed reality) application it will work as client for all the trackers supported by

OpenIGTLink just changing few settings (few code lines).

The previously described software architecture guarantees satisfactory performances and portability.

The architecture and the main features have been defined and tested in strongly collaboration with surgeons,

and that collaboration has revealed the essential technical problems whose solutions would contribute to

effective simulation. The “step by step” validation of the simulator, focused on the tissue realism was essential

for this research.

Last but not least: all the components are open source or at least based on a GPL license.

8.5 DISCIPLINES

Different disciplines are involved in the MENTIS prototype development (Figure 88). One of the main issue in

this work has been, obviously, the integration of different components taking in account the needs of real-time

and realism. Possible bottlenecks related to computational load or huge data flow have been considered and

managed.

FIGURE 88: DIFFERENT DISCIPLINES ARE INVOLVED IN THE MENTIS PROTOTYPE DEVELOPMENT


Ch

apte

r 8

103

8.6 FUTURE WORK

Possible future natural directions are here collected:

the 3D environment could be better produced achieving more surgeons’ needs;

simulation scenario could be more complex of the realized brain palpation for example including other

pathologies or adding the tissue resection;

physical modelling is a complex area and can be more explored and tested to have better realism. FEM

are methods very computational load consuming but could be implemented using parallel computing:

CUDA (running on Nvidia GPU) is very promising in this direction. This consideration can be also

extended to the collision detection computation;

the 3D environment description language used is X3D because it is the natural successor of the well-

known VRML. It means that the complete scene in the application can be rendered in a web browser

for future extension as web- collaborative and distributed training. Texturing is applied at this step in

order to improve realism;

an automated thread for evaluate or record surgeons training session would be a useful extension

giving an accurate feedback to the trainee.


Ap

pen

dix

A

104

APPENDIX A

ICP


Ap

pen

dix

A

105

ICP

Iterative Closest Point (ICP) is an algorithm employed to match two clouds of points. This matching is used to

reconstruct 3D surfaces from different scans, or, in augmented reality, to align a virtual object to the real one.

This is done by minimizing the sum of square of the Euclidean distance between the different sets of points.

The result will be a rotation matrix R and a translation matrix T that align the virtual object position to the

reality.

Original ICP algorithm [30] is very simple and relies on previous work [78] as the method to find the best rigid

body transformation. ICP for two free-form shapes (model X, object P) can be formulated in common words by

the next algorithm:

Estimate initial transformation

Iterate next steps to converge to local minima

1. For all the p ϵ P’ find closest point x ϵ X

2. Find transformation Q which minimizes distances between each p and x

3. Transform Pk+1 ← Q(Pk)

4. Terminate when change in the error falls below a preset threshold

Choose the best among found solutions for different

initial transformations

ICP RELATED PROBLEMS

While being very simple, ICP method has created a big wave of research to overcome its drawbacks:

it is quite computationally demanding to identify the closest point for each object point at each

iteration;

converges to local minima, which might be a global minimum under ’lucky’ initial transformation;

doesn’t converge very fast;

doesn’t work properly for some specific shapes, though it is a minor and rare problem;

can be sensitive to outliers.

Several papers (i.e. [79]), provided detailed overviews of existing approaches and usually present themselves a

new one on how to improve ICP. Most of them have done work in attempt

to speed-up closest point selection:

1. K-d trees, dynamic caching

2. sampling of model and object points

to avoid local minima:

1. removal of outliers


Ap

pen

dix

A

106

2. stochastic ICP, simulated annealing, weighting

3. use other metrics (point-to-surface vs -point)

4. use additional information besides geometry (color, curvature)


Pu

blic

atio

ns

rela

ted

to

th

is P

hD

107

PUBLICATIONS RELATED TO THIS PHD

INTERNATIONAL PUBLICATIONS

De Mauro, J.Raczkowsky, C.R. Wirtz, M. E. Halatsch , H. Wörn – “Augmented Microscope System for

Training and Intra-Operative purposes” – chapter for "Modelling, Simulation and Optimization", ISBN

978-953-7619-36-7, In-tech Web publications, June 2009 .

A. De Mauro, J.Raczkowsky, M. E. Halatsch , H. Wörn - “Virtual Reality Training Embedded in

Neurosurgical Microscope”, – Accepted paper for proceedings– IEEE VR 2009, March 14-18, 2009,

Lafayette, Louisiana, USA.

A. De Mauro, J.Raczkowsky, M. E. Halatsch, H. Wörn - “Mixed Reality Neurosurgical Microscope for

training and intra-operative purposes” - Accepted paper for proceedings - HCI International 2009, 19-

24 July 2009, San Diego, CA, USA.

A. De Mauro, J.Raczkowsky, R.Wirtz, H. Wörn “A neurosurgical Training System Embedded on an

Operating Microscope” - Applied Simulation and Modelling (ASM 2008) conference proceedings -

ISBN: 978-0-88986-748-2, June 23 – 25, 2008, Corfu, Greece.

A. De Mauro, J.Raczkowsky, R.Wirtz, H. Wörn “Development of a microscope embedded training

system for neurosurgery”, - Lecture Notes in Computer Science 5104 2008, ISBN 978-3-540-70520-8,

June 2008, London, UK.

A. De Mauro, J.Raczkowsky, R.Wirtz, H. Wörn “Neurosurgical Training System Based on an Operating

Microscope” MMVR-The 16TH Annual Medicine Meets Virtual Reality Conference, Digital Publication

on Proocedings, Long Beach (CA), 28th January – 1 February 2008.

A. De Mauro, M. Aschke, J.Raczkowsky, R.Wirtz, H. Wörn “Improvement of the Concept for an

Augmented Reality Neurosurgical Microscope”, MMVR-The 16TH Annual Medicine Meets Virtual

Reality Conference, Digital Publication on Proocedings, Long Beach (CA), 28th January – 1 February

2008.

NATIONAL PUBLICATIONS

A. De Mauro, M. Aschke, J.Raczkowsky, R.Wirtz, H. Wörn “Mixed Reality System for Neurosurgery

based on Operating Microscope”, Institute for Process Control and Robotics, University of Karlsruhe

(TH), Germany – Tagungsband der 6. Jahrestagung der Deutschen Gesellschaft für Computer- und

Roboterassistierte Chirurgie e.V., pp 29-32, ISBN 978-3-86805-008-0, 2007, Karlsruhe, Germany.

A. De Mauro, M. Aschke, J.Raczkowsky, R.Wirtz, H. Wörn “Tecniche di realta’ virtuale ed aumentata

applicate ad un microscopio operativo. Training e supporto per neurochirurghi”, Institute for Process

Control and Robotics, University of Karlsruhe (TH), Germany – Presented in MIMOS 2007 – Digital

Publication on MIMOS website, Torino – November 2007.

A. De Mauro, M. Aschke, J.Raczkowsky, R.Wirtz, H. Wörn “Realta’ Aumentata in Neurochirurgia”,

Institute for Process Control and Robotics, University of Karlsruhe (TH), Germany – Presented in

MIMOS 2007 – Digital Publication on MIMOS website. Torino – November 2006.


Bib

liogr

aph

y

108

BIBLIOGRAPHY

[1] S. Dawson e J. Kaufman, “The imperative for medical simulation,” Proceedings of the IEEE, vol. 86, 1998,

pp. 479-483.

[2] V. Tomulescu e I. Popescu, “*The use of LapSim virtual reality simulator in the evaluation of laparoscopic

surgery skill. Preliminary results+,” Chirurgia (Bucharest, Romania: 1990), vol. 99, pp. 523-7.

[3] P.L. Youngblood, S. Srivastava, M. Curet, W.L.R. Heinrichs, P. Dev, e S.M. Wren, “Comparison of training

on two laparoscopic simulators and assessment of skills transfer to surgical performance,” Journal of the

American College of Surgeons, vol. 200, 2005, pp. 546-551.

[4] M. Aschke, "Stereoskopische Navigation in 3D-Datensaetzen fuer die Erwaiterte Realitaet in der

Chirurgie", PhD Thesis, GCA-Verlag, 2007.

[5] P.H. Gutin e J.B. Posner, “Neuro-oncology: diagnosis and management of cerebral gliomas--past, present,

and future,” Neurosurgery, vol. 47, Lug. 2000, pp. 1-8.

[6] A. Perry, “Pathology of low-grade gliomas: An update of emerging concepts,” Neuro-oncol, vol. 5, Lug.

2003, pp. 168-178.

[7] F.C. Eastman, “A Classification of Tumors of the Glioma Group,” Annals of Surgery, vol. 84, Lug. 1926, pp.

143–144.

[8] P. Kleihues, P.C. Burger, e B.W. Scheithauer, “The new WHO classification of brain tumours,” Brain

Pathology (Zurich, Switzerland), vol. 3, Lug. 1993, pp. 255-68.

[9] J.W. Kernohan, R.F. Mabon, H.J. Swen, e A.W. Adson, “A simplified classification of gliomas,” J.

Neurosurg., vol. 74, 1991, pp. 27.

[10] C. Daumas-Duport, B.W. Scheithauer, e P.J. Kelly, “A histologic and cytologic method for the spatial

definition of gliomas,” Mayo Clinic Proceedings. Mayo Clinic, vol. 62, Giu. 1987, pp. 435-49.

[11] T. O'Shea, “Cancer Healing Solutions,” 2009.

[12] W.A. Hall, H. Liu, e C.L. Truwit, “Functional magnetic resonance imaging-guided resection of low-grade

gliomas,” Surgical Neurology, vol. 64, Lug. 2005, pp. 20-27.

[13] “Alabama Neurosurgeon: Brain Surgery”, http://www.brainspine.com/brain-surgery.html

[14] C. Hess, “BRAIN: CT, MRI Noninvasive imaging of the central nervous system.”

[15] K.E. Coyne, “MRI: A Guided Tour”, http://www.magnet.fsu.edu/education/tutorials/magnetacademy/mri

[16] S. Lakare e A. Kaufman, “3D segmentation techniques for medical volumes,” Center for Visual Computing,

Department of Computer Science, State University of New York, 2000.


Bib

liogr

aph

y

109

[17] W.E. Lorensen e H.E. Cline, “Marching cubes: A high resolution 3D surface construction algorithm,”

SIGGRAPH Comput. Graph., vol. 21, 1987, pp. 163-169.

[18] “The Marching Cubes”, http://users.polytech.unice.fr/~lingrand/MarchingCubes/algo.html

[19] L. Guibas e J. Stolfi, “Primitives for the manipulation of general subdivisions and the computation of

Voronoi diagrams,” Acm transactions on graphics, vol. 4, 1985, pp. 74-123.

[20] P. Cignoni, C. Montani, e R. Scopigno, “DeWall: A fast divide and conquer Delaunay triangulation

algorithm in Ed,” COMPUTER AIDED DESIGN-LONDON-BUTTERWORTH, vol. 30, 1998, pp. 333-342.

[21] “BrainLAB ”, http://www.brainlab.com

[22] “Stryker”, http://www.stryker.com,

[23] “NDI: Polaris Vicra & Spectra - Optical Medical Measurement Systems.”

[24] W. Grimson, G. Ettinger, S. White, T. Lozano-Perez, W. Wells, e R. Kikinis, “An automatic registration

method for frameless stereotaxy, image guided surgery, and enhanced reality visualization,” Medical

Imaging, IEEE Transactions on, vol. 15, 1996, pp. 140, 129.

[25] A. Raabe, R. Krishnan, R. Wolff, E. Hermann, M. Zimmermann, e V. Seifert, “Laser surface scanning for

patient registration in intracranial image-guided surgery,” Neurosurgery, vol. 50, Apr. 2002, pp. 797-801;

discussion 802-3.

[26] J. Hoffmann, C. Westendorff, C. Leitner, D. Bartz, e S. Reinert, “Validation of 3D-laser surface registration

for image-guided cranio-maxillofacial surgery,” Journal of Cranio-Maxillo-Facial Surgery: Official

Publication of the European Association for Cranio-Maxillo-Facial Surgery, vol. 33, Feb. 2005, pp. 13-8.

[27] M.I. Miga, T.K. Sinha, D.M. Cash, R.L. Galloway, e R.J. Weil, “Cortical surface registration for image-guided

neurosurgery using laser-range scanning,” IEEE Transactions on Medical Imaging, vol. 22, Ago. 2003, pp.

973-85.

[28] R. Marmulla, S. Hassfeld, T. Lüth, U. Mende, e J. Mühling, “Soft tissue scanning for patient registration in

image-guided surgery,” Computer Aided Surgery: Official Journal of the International Society for

Computer Aided Surgery, vol. 8, 2003, pp. 70-81.

[29] J. Zhang, Y. Ge, S. Ong, C. Chui, S. Teoh, e C. Yan, “Rapid surface registration of 3D volumes using a neural

network approach,” Image and Vision Computing, vol. 26, Feb. 2008, pp. 201-210.

[30] P.J. Besl e N.D. McKay, “A Method for Registration of 3-D Shapes,” IEEE Trans. Pattern Anal. Mach. Intell.,

vol. 14, 1992, pp. 239-256.

[31] C.R. Maurer Jr, J.J. McCrory, e J.M. Fitzpatrick, “Estimation of accuracy in localizing externally attached

markers in multimodal volume head images,” Proceedings of SPIE, 1898.

[32] J. West e J. Fitzpatrick, “The Distribution of Target Registration Error in Rigid-Body, Point-Based

Registration,” Information Processing in Medical Imaging, 1999, pp. 460-465.

[33] R. Holloway, “An analysis of registration errors in a see-through head-mounted display system for

craniofacial surgery planning,” Unpublished doctoral dissertation, University of North Carolina at Chapel

Hill, 1995.


Bib

liogr

aph

y

110

[34] E. McGarrity e M. Tuceryan, “A method for calibrating see-through head-mounted displays for AR,” 2nd

IEEE and ACM International Workshop on Augmented Reality, 1999.(IWAR'99) Proceedings, 1999, pp. 75-

84.

[35] Z. Zhang, “Flexible camera calibration by viewing a plane from unknownorientations,” Computer Vision,

1999. The Proceedings of the Seventh IEEE International Conference on, 1999.

[36] R.M. Satava, “Medical virtual reality. The current status of the future,” Studies in Health Technology and

Informatics, vol. 29, 1996, pp. 100-6.

[37] R. Adams e L. Bischof, “Seeded region growing,” Pattern Analysis and Machine Intelligence, IEEE

Transactions on, vol. 16, 1994, pp. 641-647.

[38] J. Duncan e N. Ayache, “Medical image analysis: progress over two decades and the challenges ahead,”

Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 22, 2000, pp. 85-106.

[39] W.R. Mark, S.C. Randolph, M. Finch, J.M.V. Verth, e I.I. Russell M. Taylor, “Adding force feedback to

graphics systems: issues and solutions,” Proceedings of the 23rd annual conference on Computer graphics

and interactive techniques, ACM, 1996, pp. 447-452.

[40] B. Marcus, “Hands on: Haptic feedback in surgical simulation,” Proc. of Medecine Meets Virtual Reality IV

(MMVR IV), pp. 134139.

[41] P. Milgram e F. Kishino, “A taxonomy of mixed reality visual displays,” IEICE TRANSACTIONS on

Information and Systems, vol. 77, 1994, pp. 1321-1329.

[42] “Immersion Corporation”, http://www.immersion.com/

[43] C. Wagner, N. Stylopoulos, e R. Howe, “The role of force feedback in surgery: analysis of blunt

dissection,” Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2002. HAPTICS 2002.

Proceedings. 10th Symposium on, 2002, pp. 68-74.

[44] G. Tholey, J.P. Desai, e A.E. Castellanos, “Force Feedback Plays a Significant Role in Minimally Invasive

Surgery: Results and Analysis,” Annals of Surgery, vol. 241, Gen. 2005, pp. 102–109.

[45] T. Hu, G. Tholey, J.P. Desai, e A.E. Castellanos, “Evaluation of a laparoscopic grasper with force feedback,”

Surgical endoscopy, vol. 18, 2004, pp. 863-867.

[46] A. Kazi, “Operator performance in surgical telemanipulation,” Presence: Teleoperators & Virtual

Environments, vol. 10, 2001, pp. 495-510.

[47] C. Basdogan, S. De, J. Kim, Manivannan Muniyandi, H. Kim, e M. Srinivasan, “Haptics in minimally invasive

surgical simulation and training,” Computer Graphics and Applications, IEEE, vol. 24, 2004, pp. 56-64.

[48] R. Webster, R.S. Haluck, G. Zoppetti, A. Benson, J. Boyd, N. Charles, J. Reeser, e S. Sampson, “A haptic

surgical simulator for laparoscopic cholecystectomy using real-time deformable organs,” Proceedings of

the IASTED International Conference on Biomedical Engineering, pp. 25-27.

[49] K. Montgomery, L.R. Heinrichs, C. Bruyns, S. Wildermuth, C. Hasser, S. Ozenne, e D. Bailey, “Surgical

simulator for hysteroscopy: A case study of visualization in surgical training,” Presented at IEEE

Visualization, 2001.


Bib

liogr

aph

y

111

[50] S. Cotin, S. Dawson, D. Meglan, D. Shaffer, M. Ferrell, R. Bardsley, F. Morgan, T. Nagano, J. Nikom, e P.

Sherman, “ICTS, an interventional cardiology training system,” Studies in Health Technology and

Informatics, 2000, pp. 59-65.

[51] S. De, Multimodal Simulation of Laparoscopic Heller Myotomy Using a Meshless Technique. Medicine

Meets Virtual Reality, Westwood et al, IOS Press, 2002.

[52] C. Basdogan, C.H. Ho, e M.A. Srinivasan, “Virtual environments for medical training: graphical and

hapticsimulation of laparoscopic common bile duct exploration,” IEEE/ASME Transactions On

Mechatronics, vol. 6, 2001, pp. 269-285.

[53] S. Cotin, H. Delingette, e N. Ayache, “Real-time elastic deformations of soft tissues for

surgerysimulation,” IEEE transactions on Visualization and Computer Graphics, vol. 5, 1999, pp. 62-73.

[54] H.K. Çakmak e U. Kühnapfel, “Animation and simulation techniques for vr-training systems in endoscopic

surgery,” Computer Animation and Simulation 2000: Proceedings of the Eurographics Workshop in

Interlaken, Switzerland, August 21-22, 2000, Springer, 2000, pp. 173.

[55] J. Brown, S. Sorkin, J.C. Latombe, K. Montgomery, e M. Stephanides, “Algorithmic tools for real-time

microsurgery simulation,” Medical Image Analysis, vol. 6, 2002, pp. 289-300.

[56] X. Wu, M.S. Downes, T. Goktekin, e F. Tendick, “Adaptive nonlinear finite elements for deformable body

simulation using dynamic progressive meshes,” Computer Graphics Forum, 2001.

[57] G. Szekely, C. Brechbuehler, R. Hutter, A. Rhomberg, N. Ironmonger, e P. Schmid, “Modelling of soft

tissue deformation for laparoscopic surgery simulation,” Medical Image Analysis, vol. 4, 2000, pp. 57-66.

[58] R.V. O'Toole, R.R. Playter, T.M. Krummel, W.C. Blank, N.H. Cornelius, W.R. Roberts, W.J. Bell, e M.

Raibert, “Measuring and developing suturing technique with a virtual reality surgical simulator.,” Journal

of the American College of Surgeons, vol. 189, 1999, pp. 114.

[59] C.Y. Ro, I.K. Toumpoulis, R.C.A. Jr, T. Jebara, C. Schulman, G.J. Todd, J.J.D. Jr, e J.J. Mcginty, “The LapSim:

a learning environment for both experts and novices,” Studies in Health Technology and Informatics, vol.

111, 2005, pp. 414-417.

[60] “SurgicalScience.”, http://www.surgical-science.com/

[61] N.W. John, M. Riding, N.I. Phillips, S. Mackay, L. Steineke, B. Fontaine, G. Reitmayr, V. Valencic, N. Zimic,

A. Emmen, E. Manolakaki, e D. Theodoros, “Web-based surgical educational tools,” Studies in Health

Technology and Informatics, vol. 81, 2001, pp. 212-7.

[62] N. Brown, S. Natsupakpong, S. Johannsen, S. Manjila, Q. Cai, V. Liberatore, A.R. Cohen, e M.C. Cavusoglu,

“Virtual environment-based training simulator for endoscopic third ventriculostomy,” Studies in Health

Technology and Informatics, vol. 119, 2006, pp. 73-5.

[63] O.V. Larsen, J. Haase, L.R. Østergaard, K.V. Hansen, e H. Nielsen, “The Virtual Brain Project--development

of a neurosurgical simulator.,” Studies in health technology and informatics, vol. 81, 2001, pp. 256.

[64] D. d'Aulignac, M.C. Cavusoglu, e C. Laugier, “Modeling the dynamics of the human thigh for a realistic

echographic simulator with force feedback,” Lecture Notes in Computer Science, 1999, pp. 1191-1198.


112

[65] S. Cover, N. Ezquerra, J. O'Brien, R. Rowe, T. Gadacz, e E. Palm, “Interactively deformable models for

surgery simulation,” Computer Graphics and Applications, IEEE, vol. 13, 1993, pp. 68-75.

[66] Y. Bhasin e A. Liu, “Bounds for damping that guarantee stability in mass-spring systems,” Studies in

Health Technology and Informatics, vol. 119, 2006, pp. 55-60.

[67] K. Miller e K. Chinzei, “Constitutive modelling of brain tissue: experiment and theory,” Journal of

Biomechanics, vol. 30, 1997, pp. 1115-1121.

[68] C. Drapaca, G. Tenti, K. Rohlf, e S. Sivaloganathan, “A Quasi-linear Viscoelastic Constitutive Equation for

the Brain: Application to Hydrocephalus,” Journal of Elasticity, vol. 85, Ott. 2006, pp. 65-83.

[69] E. Takhounts, J. Crandall, e B. Matthews, “Experimental Determination of Constitutive Equations for

Brain Tissue. 21e congrès annuel de l'American Society of Biomechanics,” 1997.

[70] G. Van Den Bergen, Collision detection in interactive 3D environments, Morgan Kaufmann, 2004.

[71] S. Gottschalk, M.C. Lin, e D. Manocha, “OBBTree: A hierarchical structure for rapid interference

detection,” Proceedings of the 23rd annual conference on Computer graphics and interactive techniques,

ACM New York, NY, USA, 1996, pp. 171-180.

[72] “X3D", http://www.web3d.org/x3d/

[73] A. De Mauro, L.T. De Paolis, E. Blasi, e G. Aloisio, “A Force Feedback Virtual Simulator for Education and

Training,” Virtual Environments, Human-Computer Interfaces and Measurement Systems, 2007. VECIMS

2007. IEEE Symposium on, 2007, pp. 135-138.

[74] “CUDA, http://www.nvidia.com/object/cuda_home.html#.”

[75] N. Hata, S. Piper, F.A. Jolesz, C.M.C. Tempany, P. Black, S. Morikawa, H. Iseki, M. Hashizume, e R. Kikinis,

“Application of Open Source Image Guided Therapy Software in MR-guided Therapies,” Lecture Notes in

Computer Science, vol. 4791, 2007, pp. 491.

[76] “P. Bergström, Iterative Closest Point Method in Mathlab.”, Matlab Central

http://www.mathworks.com/matlabcentral/fileexchange/

[77] “Camera Calibration Toolbox for Matlab.”, http://www.vision.caltech.edu/bouguetj/calib_doc/

[78] B.K.P. Horn, “Closed-form solution of absolute orientation using unit quaternions,” Journal of the Optical

Society of America A, vol. 4, 1987, pp. 629-642.

[79] S. Ourselin, R. Stefanescu, e X. Pennec, “Robust registration of multi-modal images: Towards real-time

clinical applications,” Lecture Notes in Computer Science, 2002, pp. 140-147.

MENTIS - Microscope Embedded Neurosurgical Training and ...

Documents

Transcript of MENTIS - Microscope Embedded Neurosurgical Training and ...