Engineering Support for Laparoscopic Surgery Training ...
Transcript of Engineering Support for Laparoscopic Surgery Training ...
![Page 1: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/1.jpg)
Engineering Support for Laparoscopic Surgery Training
Intelligent Performance Assessment System for Laparoscopic Surgical Box-Trainer Device – A
Research Collaboration Project Between the ECE Department and the Department of Surgery
Janos L Grantner1, Aous H. Kurdi1, Mohammed Al-Gailani1, Koloud Al-Khamaiseh1, Osamah Majeed1, Ikhlas Abdel-Qader1,
Robert G. Sawyer2, M.D. and Saad Shebrain2, M.D., WMed Surgery Residents2
1ECE Department, Western Michigan University and 2Department of Surgery, Homer Stryker M.D. School of Medicine, Western Michigan University
1
The intelligent box-trainer project was sponsored by the Homer Stryker M.D. School of Medicine, WMU (Contract #: 29-7023660)
![Page 2: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/2.jpg)
Presentation Outline
• Brief Introduction to CEAS and WMed
• Background and Motivation
• Fundamentals of Laparoscopic Surgery (FLS) Trainer
• Intelligent Box-Trainer System
• Force and Time Measurements
• Tool Tip Tracking
• Fuzzy Logic Performance Assessment System
• Illustrations of Using the Intelligent Box-Trainer System
• Conclusions and Future Research
2
![Page 3: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/3.jpg)
CEAS Building
3
![Page 4: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/4.jpg)
CEAS Building (Cont’d)
4
![Page 5: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/5.jpg)
Upjohn Building of WMed
5
![Page 6: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/6.jpg)
Background and Motivation
• In surgery practice, the number of procedures carried out using laparoscopic and robotic surgery methods keeps increasing
• Surgery residents should pass standardized laparoscopic surgery skill tests. In addition, there is need to classify laparoscopic skill levels of practicing surgeons when they are hired by hospitals
• The performance assessment method using available low-level trainers may be subjective and inconsistent
• High-end skill-level test systems using Virtual Reality simulators are rather expensive (cost over $100K) and are lack of the “touch and feel” experience
• There is a growing need for the development of relatively inexpensive laparoscopic skill test systems which provide quantitative measurements, recorded video of the tests for review and a more objective and consistent approach for performance assessment
6
![Page 7: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/7.jpg)
Typical Laparoscopic Surgical Box-Trainer Device
7
![Page 8: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/8.jpg)
FLS Trainer
• Residents should pass standardized tests (5 different types total using the FLS Trainer) successfully to get certified for performing laparoscopic surgical procedures
• The performance of the resident is monitored by a medical educator who also takes the test execution time by a stop watch
• Neither the time, nor the video recorded electronically (no playback is available)
![Page 9: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/9.jpg)
Intelligent Box-Trainer System
• The key components of the system are as follows: FLS Trainer box along with various test tools and digital video cameras, Main Station Computer (MS) with a 32” HD monitor, Microcomputer System (CB) with force sensor and push button interfaces, MS SP4 Pro Tablet as Control Panel (CP) to the system and a Router.
• Inside the FLS Trainer Box there is two HD webcams for tool tip tracking and video recording (one more will be added for pattern cutting tests). They are interfaced to the MS via USB interfaces. There is also a third HD webcam which is interfaced to a Raspberry Pi 3 B microcomputer (RP3). The RP3 sends the video images to one of the two HDMI inputs to the monitor so the residents can view what they are doing with the test platform. The grasper tools have force sensors mounted on them.
• The MS carries out essentially all computing tasks for the system, archives the results and communicates with both the CB and the CP
• The CB is tasked with force and time measurements
• The CP provides for a GUI interface to the system and functions as a control and display panel for the supervising medical faculty
9
![Page 10: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/10.jpg)
Intelligent Laparoscopic Surgical Box-Trainer System
10
![Page 11: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/11.jpg)
Intelligent Box-Trainer System Block Diagram
11
![Page 12: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/12.jpg)
Force and Time Measurements
• The force applied by the jaws of the grasping tool shouldn’t exceed 2.2 lbsdue to potential tissue damage
• Up to 100N range capacitive force sensors are flush-mounted to the handle of the grasping tool
• A calibration function has been developed to translate the measured force to the one applied by the jaws
• After the start of the test force and elapsed time values are sent to the MS PC about every 30ms
![Page 13: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/13.jpg)
Force Measurements Cont’d
• The graph shows the linearized force values applied by the jaws versus the raw measured data
• On the horizontal axis the output values of the ADC module on the sensor interface board are given
![Page 14: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/14.jpg)
Microcontroller System
• It is based upon the Cortex M4 STM32F429I uC chip
• The force sensor interface boards are connected through the I2C protocol
• There is a built in force calibration function
• Heartbeat LED indicates that the Microcontroller System is ready to operate
• It is connected to the MS PC through the UART/USB protocol
• It is interfaced to the Test Start/Stop push button
![Page 15: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/15.jpg)
Tool Tip Tracking
• In laparoscopic surgery the tool tip movements, while executing a particular task, should be confined into a 3D space of limited size
• If the tool tips moved too much beyond the target space they might cause unintended damage to internal organs in close proximity
• Two 5MP HDMI webcams with USB interface are used
• Tracking was accomplished by color identification
• Two streams at the rate of 30 frames/sec with 1028x720 pixels per frame
![Page 16: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/16.jpg)
Tool Trip Tracking Cont’d
16
![Page 17: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/17.jpg)
Fuzzy Logic Performance Assessment System Block Diagram
17
![Page 18: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/18.jpg)
Fuzzy Logic Performance Assessment System Cont’d
• In the first layer, the fuzzified right-hand and left-hand tool tip movements as well as the fuzzified forces applied by the right-hand and left-hand grasping tools are entered into their respective fuzzy knowledge bases
• A Mamdani-type inference system along with the Center of Gravity defuzzification algorithm are implemented to process these fuzzy inputs
• The defuzzified outputs are then fuzzified again as inputs for the final assessment system in the second layer. They are joined by the fuzzified time measurement input values.
• These functionally decoupled fuzzy knowledge bases allow independent fine-tuning of the performance of the key components of the assessment system
18
![Page 19: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/19.jpg)
Fuzzy Sets for Grasping Force Measurements
19
![Page 20: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/20.jpg)
Fuzzy If Then Rules for Grasping Force Assessment
RH and LH L H VH
L E G B
H G G B
VH B B B
20
![Page 21: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/21.jpg)
Fuzzy Sets for Tool Tip Movements (X, Y and Z) Assessment
21
![Page 22: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/22.jpg)
FAM Matrices for Tool Tip Tracking Assessment
22
Z X LB LG E RG RB
E Y
LB B B G B B
LG B G E G B
E G E E E G
RG B G E G B
RB B B G B B
Z X LB LG E RG RB
G Y
LB B B B B B
LG B G G G B
E B G E G B
RG B G G G B
RB B B B B B
Z X LB LG E RG RB
B Y
LB B B B B B
LG B B B B B
E B B G B B
RG B B B B B
RB B B B B B
![Page 23: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/23.jpg)
Membership Functions for Time Taken
23
![Page 24: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/24.jpg)
Membership Functions for the Final Assessment
24
![Page 25: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/25.jpg)
Simulated Assessment of Forces Applied by Grasping Tools
25
![Page 26: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/26.jpg)
Intelligent Box-Trainer System
26
![Page 27: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/27.jpg)
Illustrations of Using the Intelligent Box-Trainer System
27
![Page 28: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/28.jpg)
Illustrations Cont’d
28
![Page 29: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/29.jpg)
Illustrations Cont’d
29
![Page 30: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/30.jpg)
Illustrations Cont’d
30
![Page 31: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/31.jpg)
Illustrations Cont’d
31
![Page 32: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/32.jpg)
Illustrations Cont’d
32
![Page 33: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/33.jpg)
Illustrations Cont’d
33
![Page 34: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/34.jpg)
Conclusions and Future Research
• Two prototypes of the Intelligent Box-Trainer System have been delivered to the Medical School
• With the proposed system, the assessment of the performance of residents taking the FLS tests is decided using measured data as well as the opinion of expert surgical faculty in a quantifiable, objective and consistent manner
• The system provides recorded visual and measured data feedback for the residents which will help them to improve their laparoscopic skills
• The FLS Tests certification documents only specify crisp Pass/Fail criteria. There is a need for residents and practicing laparoscopic surgeons to assess their performance regarding various specified skill levels. This will require a lot of tests and collaboration with expert surgeons to develop and later fine-tune the respective knowledge bases.
• Currently, we are focusing on the pattern cutting and the three suturing-related tests
34
![Page 35: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/35.jpg)
Planned Cyber -Physical System for Intelligent Robotic Surgery Laboratory
Research Questions/Objectives
• Can best robotic surgery practices as identified in (1) be used to develop automated, or partially automated robotic surgery procedures using computational intelligence methods (in particular, controllers using fuzzy automata) which are deemed very valuable by expert surgeons?
• In case of tele-robotic surgery applications, what kind of safe, automated robotic surgery procedures can be developed when the internet connection between the master station operated by a surgeon and the remote slave station with the actual patient is lost for a considerable period of time?
35
![Page 36: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/36.jpg)
Planned Cyber -Physical System for Intelligent Robotic Surgery Laboratory (Cont’d)
Research Questions/Objectives
• How can sensory data and recorded video fused with computational intelligence methods (in particular, fuzzy logic) to capture domain knowledge of expert robotic surgeons be used to develop an objective, intelligent assessment support system for skills required in robotic surgery?
• Comparatively, what are the advantages of a hardware-in-the-loop cyber-physical system (a da Vinci robot operating on 3D printed phantom objects and augmented with sensor arrays and additional HD video cameras) over current 3D virtual reality simulators with respect to skill development and skill assessment?
36
![Page 37: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/37.jpg)
Intelligent Robotic Surgery Laboratory Block Diagram
Instrumented da Vinci robot
37
4K HD Monitor
- Intelligent automated surgeon machine
- Evolutionary Algorithms
- Archives
PC Workstation
NI PXI + Custom Sensors
Da Vinci Console Console to controller
Interface
Surgeon Interface
Switch
Surgeon Console
Da Vinci Factory Controller
Da Vinci Surgical Robot
Added Sensors to Da Vinci ArmsAdded Cameras to Da Vinci Arms
Body emulator with cameras and Sensors
NI PXI + Custom Sensors
4K HD Monitor
Custom Sensor Interface
Internet Security Module
Camera Interface
Audio communication
Pre-operation PlannerIntelligent Surgeon
Intelligent Automated Surgeon Machine
![Page 38: Engineering Support for Laparoscopic Surgery Training ...](https://reader031.fdocuments.net/reader031/viewer/2022032710/623c1de2835a5f3375634707/html5/thumbnails/38.jpg)
38
Thank You!
Any Questions?