Weakly supervised segmentation and Firma convenzione ...

Firma convenzione

Politecnico di Milano e Veneranda Fabbrica

del Duomo di Milano

Aula Magna – Rettorato

Mercoledì 27 maggio 2015

Weakly supervised segmentation and recognition of surgical gestures

Tutor: Elena De Momi, PhD

Co-tutor: Danail Stoyanov, PhD

Co-tutor: Hirenkumar Nakawala, PhD

Student: Beatrice van Amsterdam

850736

Academic Year: 2016-2017

Nome Cognome, assoc.prof. ABC Dept.

IntroductionRobot-Assisted Minimally Invasive Surgery (RAMIS)

Robot-Assisted Minimally Invasive

Surgery (RAMIS)

✓ Video and robot kinematics recoding

✘ Lack of tactile sensation✓✘ Improved visibility (3D)

✓ Improved precision and ergonomic comfort✓ Reduced trauma, pain, recovery time, costs

✘ Extended operation time and learning curve

Minimally Invasive Surgery (MIS)

✓ Reduced trauma, pain, recovery time, costs

✘ Lack of tactile sensation✘ Kinematic restrictions

✘ Reduced visibility (2D)

Open surgery

✓Maximum visibility✓ Tactile sensation✓ No kinematic restrictions✘ Substantial trauma and pain✘ Long recovery time

Beatrice van Amsterdam


IntroductionBackground clinical problem

Surgical technical errors are the mostcommon reason (30%) for post-surgicalcomplications including re-operation andre-admission. (Stahel, 2014)

(Birkmeyer, 2013)

The issue of surgical errors has becomeparticularly relevant with the advent ofnew surgical techniques that offer greatadvantages to the patient but require ahigh level of visuomotor skill from thesurgeon, such as MIS and RAMIS. (White,

2015)



IntroductionAction recognition for Computer Assisted Interventions

❖ Extended learning curve Computational models for surgical skill assessment

❖ Critical and time-consuming tasks Intra-operative assistance and automation

Surgery involves dexterous human motion. The eventual goal for studying surgical

motion could improve safety and effectiveness of surgical patient care (Gao, 2014) .


ACTION RECOGNITION

Understand where the surgeonneeds improvement

Short segments are easier toautomate


IntroductionAction granularity


Procedure

Phase

Activity

Surgeme

Dexeme

E.g. Suturing

E.g. Closure

E.g. Lumbar Disc Herniation

E.g. Grasping the needle

E.g. Turning left

(Lalys, 2014)


Related work

Supervised Approaches

• Hidden Markov Models (Tao, 2012)

• Conditional Random Fields (Tao, 2013)

• Deep Learning (Lea, 2017)

Unsupervised Approaches

• Gaussian Mixture Models (Murali, 2016)

• Hidden Markov Models unsup. (Tanwani, 2016)

• Bottom-up clustering (Fard, 2016)

Weakly supervised Approaches

• Video retrieval (Quellec, 2014)

• CNN (Wang, 2017)

• Ordering constrained clustering (Bojanowski, 2014)


✓ Good results✘ Manual labelling (burden, subjective)✘ Fixed set of classes

✓ No manual labelling

✘ Poorer performance✓ Classes learnt from data

✓ Better performance than unsupervised methods✓ ✘ Minor manual labelling✘ Fixed set of classes


Aim of the work

New approach: employ weak supervision to optimize

unsupervised algorithm initialization


Many unsupervised algorithms rely on iterative schemes: initialization problem

Expected outcome: improved alignment between

predicted and ground truth segmentation


Materials and MethodsThe JIGSAWS dataset

Beatrice van AmsterdamBeatrice van Amsterdam

JIGSAWS (Gao, 2014) is a public dataset featuring suturing demonstrations collected fromeight surgeons with different skill level (Expert, Intermediate, Novice) using the da VinciSurgical System.

Cartesian positions (3 variables)Rotation matrix (9 variables)Linear velocities (3 variables)

Angular velocities (3 variables)Gripper angle (1 variable)

The motion of each manipulator isdescribed by a local frame attached at itsend-effector using 19 kinematic variables:

Expert: > 100 training hours Novice: < 10 training hours


Materials and MethodsPre-processing

Rotation matrix converted into

quaternion

Euclidean distance signals

computation

Low pass filter

fc = 1.5 Hz

(Despinoy, 2016)

Normalization to zero mean and unit variance



Materials and MethodsSimultaneous action segmentation and recognition

After model training, each sample is assigned to itsmost likely mixture component, i.e. its most likelysurgeme label.

Each demonstration 𝒙 𝑡 ∈ ℝ𝑝 can be treated as anoisy realization of a switched linear dynamicalsystem. Each regime 𝑨𝒌 corresponds to a surgeme.

which is equivalent to solving multiple linearregression problems, when L=1 (Moldovan, 2015).

𝒙 𝑡 + 1 = 𝑨𝒌𝒙 𝑡 + 𝒘 𝑡 ,

𝑨𝒌 ∈ {𝑨𝟏, … , 𝑨𝑵}, 𝑨𝒌 ∈ ℝ𝑝×𝑝

argmin𝑨𝒌

𝑨𝒌𝑿𝒕 − 𝑿𝒕+𝟏 ,

𝑿𝒕 = [𝒙 𝑡 ,… , 𝒙 𝑇 ] ∈ ℝ𝑝×𝑇

Action recognition can then be performed byfitting a Gaussian Mixture Model (GMM) to theaugmented state 𝒏 𝑡 ,

𝒏 𝑡 = 𝒙 𝑡 , 𝒙 𝑡 + 1 ,… , 𝒙 𝑡 + 𝐿 ,

𝒏 𝑡 ∈ ℝ𝑝𝐿



Materials and MethodsGround truth segmentation redefinition: local linearity

Beatrice van AmsterdamBeatrice van Amsterdam

𝐿 = 3 (Murali, 2016) 𝒏 𝑡 = 𝒙 𝑡 , … , 𝒙 𝑡 + 𝐿 = 𝒙 𝑡 , 𝒙 𝑡 + 1 , 𝒙 𝑡 + 2 , 𝒙 𝑡 + 3

𝒙 𝑡𝒙 𝑡 + 1𝒙 𝑡 + 2𝒙 𝑡 + 3

𝒏 𝑡


Materials and MethodsGround truth segmentation redefinition: action dictionary

L1: positioning and pushing needle through the tissue

L2: reaching for the needle with the left hand

L3: pulling suture with the left hand

L4: transferring needle from left to right

L5: extracting suture from the tissue with the left hand

L6: reaching for the needle with the right hand;

L7: transferring needle from right to left

L8: using right hand to help tighten suture

L9: dropping suture

L10: moving to end points



Materials and MethodsWeakly supervised initialization


GMM0

w


Materials and MethodsOffline segmentation


w

w


Evaluation protocolExtrinsic evaluation metrics

𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚 =𝑇𝑃

𝑇𝑃 + 𝐹𝑃 + 𝐹𝑁

𝑹𝒆𝒄𝒂𝒍𝒍 =𝑇𝑃

𝑇𝑃 + 𝐹𝑁

𝑷𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 =𝑇𝑃

𝑇𝑃 + 𝐹𝑃

𝑭 − 𝒔𝒄𝒐𝒓𝒆 =2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙

𝑵𝒐𝒓𝒎𝒂𝒍𝒊𝒛𝒆𝒅𝑴𝒖𝒕𝒖𝒂𝒍 𝑰𝒏𝒇𝒐𝒓𝒎𝒂𝒕𝒊𝒐𝒏 (𝑵𝑴𝑰) =𝐼 𝛾, 𝜏

𝐻 𝛾 𝐻 𝜏

𝑹𝒆𝒄𝒂𝒍𝒍 𝑴𝒂𝒄𝒓𝒐𝑾𝒆𝒊𝒈𝒉𝒕𝒆𝒅 (𝑹𝑴𝑾)

𝑷𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 𝑴𝒂𝒄𝒓𝒐𝑾𝒆𝒊𝒈𝒉𝒕𝒆𝒅 (𝑷𝑴𝑾)

𝑭 − 𝒔𝒄𝒐𝒓𝒆𝑴𝒂𝒄𝒓𝒐𝑾𝒆𝒊𝒈𝒉𝒕𝒆𝒅 (𝑭𝑴𝑾)



Evaluation protocolIntrinsic evaluation metrics

𝑺𝒊𝒍𝒉𝒐𝒖𝒆𝒕𝒕𝒆 𝑰𝒏𝒅𝒆𝒙 (𝑺𝑰) = 𝑚𝑒𝑎𝑛 𝒔𝑺𝑰 , 𝒔𝑺𝑰 =𝑏−𝑎

max(𝑎,𝑏), 0 < 𝒔𝑺𝑰 < 1

𝑹𝒆𝒍𝒂𝒕𝒊𝒗𝒆 𝑺𝒊𝒍𝒉𝒐𝒖𝒆𝒕𝒕𝒆 𝑰𝒏𝒅𝒆𝒙 (𝑹𝑺𝑰) =𝑆𝐼

Ground Truth SI

✕

✕

✕

𝑎

𝑏

𝒔



Experiments and ResultsExperimental protocol


❖ Ground truth redefinition

❖ Initialization technique

❖ Feature selection

❖ Increased data variability

Experts only

Full dataset


Experiments and ResultsGround truth redefinition (experts)

*p<0.05 McNemar’s test



Experiments and ResultsGround truth redefinition (experts)

*p<0.05 𝜒2 test for independent proportions


Proposed annotation

Original annotation


Experiments and ResultsInitialization technique (experts)



Experiments and ResultsFeature selection (experts)

McNemar’s test



Experiments and ResultsFeature selection (experts)


output

GT

GToutput


Experiments and ResultsExtended dataset


< (Lea, 2017) supervised (85% Accuracy)< (Fard, 2016) unsupervised (0.78 Recall, 0.74 Precision)> (Murali, 2016) unsupervised GMM-based (0.31 NMI)


Conclusion and future work

Experimental results on real surgical kinematic trajectories during a training exerciseconfirm that weakly supervised initialization signicantly outperforms standard task-agnostic initialization methods, leading to 13% improvement of NMI.

Automatic recognition of surgical gestures is an important preliminary step in thedevelopment of computational models for surgical skill assessment as well as intra-operative assistance and automation algorithms.

Future work:

• Including contextual and semantic information from the video data.

• Extending the kinematic data with the trajectories of the da Vinci arm joints.

• Introducing transition probabilities between different actions (HMM).



Acknowledgments


Weakly supervised segmentation and Firma convenzione ...

Documents

Transcript of Weakly supervised segmentation and Firma convenzione ...