Stages of Prenatal Development Zygotic (germinal) stage Embryonic stage Fetal stage Critical period.
Video Summarization of Key Events Stage I - The Critical View
description
Transcript of Video Summarization of Key Events Stage I - The Critical View
![Page 1: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/1.jpg)
Video Summarization of
Key EventsStage I - The Critical View
Michael A. Grasso, MD, PhDUniversity of Maryland School of
MedicineUMBC Computer Science
MichaelGrasso.com
![Page 2: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/2.jpg)
AbstractLaparoscopic surgery is a minimally invasive technique with unique training requirements. Video-assisted evaluation is one method that surgical residents can use to demonstrate competence. Automated video summarization can increase the efficiency of evaluations by directing the senior surgeon to key portions of a surgical procedure. We are using image classification techniques to segment videos of laparoscopic cholecystectomies to assist with surgical training and evaluation.
![Page 3: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/3.jpg)
Overview Background
Laparoscopic Surgery Image Classification
Methods Discussion
![Page 4: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/4.jpg)
Laparoscopic Surgery Minimally Invasive Surgery. First performed in 1987. Used in many surgical procedures.
Gall bladder removal (cholecystectomy). Esophageal surgery (fundoplication). Colon surgery (colectomy). Others.
![Page 5: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/5.jpg)
Laparoscopic Approach Narrow tubes (trocars)
are inserted into the abdomen through small incisions.
www.fda.gov
![Page 6: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/6.jpg)
Laparoscopic Procedure Camera is passed
through trocar. Procedure is often
videotaped. Carbon dioxide is
infused through trocar.
Instruments are passed through the trocars to cut, manipulate, and sew.
![Page 7: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/7.jpg)
Laparoscopic Aftercare Compared with an
open procedure. Smaller scars. Reduced pain. Quicker recovery.
http://www.nlm.nih.gov/medlineplus/ency/presentations/100166_1.htm
![Page 8: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/8.jpg)
Technical Challenges Access limited to small incisions. Long instruments with only the tips
visible. Two-dimensional video. Limited tactile feedback.
British Journal of Surgery. 2004 Dec;91(12):1549-1558
![Page 9: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/9.jpg)
Laparoscopic Training Traditional apprenticeship model.
Acquire skills during actual procedures. Not sufficient for laparoscopic skills.
Other methods. Box trainer with animal or synthetic
models. Virtual reality simulator. Video-based assessment.
![Page 10: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/10.jpg)
Assessment of Skills Trainee must demonstrate
competency. Evaluation by a senior surgeon.
Direct observation of the trainee. Video-based assessment.
Question: Can we organize video in order to assist in video-based assessment?
American Journal of Surgery. 1991 Mar;161(3):399-403
![Page 11: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/11.jpg)
Objective Identity key portions of surgical
procedure to aid in video-based assessment.
Stage I is to identify the "critical view".
Video
Segments
Frames
![Page 12: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/12.jpg)
Overview Background
Laparoscopic Surgery Image Classification
Methods Discussion
Summary: Organize surgical video to make it easier for expert to review.
![Page 13: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/13.jpg)
The Critical View Helps ensure that the anatomy has
been properly identified. Occurs after dissecting anatomy. Occurs before clipping the cystic
artery and cystic duct.
![Page 14: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/14.jpg)
The Critical View
Cystic artery
Liver
Cystic duct
Fundus
Netter's Atlas of Human Anatomy
![Page 15: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/15.jpg)
The Critical View
![Page 16: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/16.jpg)
Image Classification - Human Features a person might use. Spectral features.
Tonal variations. Textural features.
Spatial distribution of tonal variations. Contextual features.
Features from surrounding areas.
![Page 17: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/17.jpg)
Image Classification - Computed Features extracted from image. Spectral features.
Distribution, size, width. Textural features.
Homogeneity, contrast, correlation. Similarity/distance metrics.
Jaccard coefficient, Jeffrey divergence.Journal of WSCG. 2003; 11(1):269-273
IEEE Transaction on Systems, Man, and Cybernetics. 1973 Nov; 3(6):610-621
![Page 18: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/18.jpg)
Color Histogram Red, green, blue, or gray.
Count number of pixels for each tone. One 28 set for an 8-bit image for each color. Does not vary with translation and rotation. Ignores shape and texture.
4x4 image. 4 gray tones. H = {5, 4, 5, 2}
0 0 1 10 0 1 10 2 2 22 2 3 3
![Page 19: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/19.jpg)
Binary Histogram Quantize values for each tone to 0
or 1. Background color given less weight. Subtle changes given more weight.
HB = {1, 0, 1, 1}
0 0 0 00 0 0 00 2 2 22 2 3 3
![Page 20: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/20.jpg)
3D Histogram Distribution within a 3D color-space.
3D color space (red, green, blue). Used in object recognition & image retrieval. n3 entries, where n = number of tones.
Example. Quantized to 3 tones
for each color.
![Page 21: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/21.jpg)
Spatial-Dependency Matrix Co-occurrence
matrix. Co-occurring values
(0o, 45o, 90o, 135o). Four 28 x 28
matrices for 8-bit image.
Co-occurring Bits
0 1 2 3Reference Bits
0 4 2 1 01 2 4 0 02 1 0 6 13 0 0 1 2
0 0 1 10 0 1 10 2 2 22 2 3 3
135o 90o 45o
0o Ref 0o
45o 90o 135o
M0 =
![Page 22: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/22.jpg)
Additional Spectral Features Location of the distribution.
Mean = Σ (bin*freq) / Σ (freq). Mode = bin of the max freq.
Size of the distribution. Standard deviation.
Width of the distribution. Max(bin) - Min(bin).
![Page 23: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/23.jpg)
Additional Textural Features Homogeneity.
Number of tone transitions. Contrast.
Amount of local variation. Correlation.
Measure of linear dependencies.
![Page 24: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/24.jpg)
Similarity/Distance Metrics Jaccard Coefficient.
Similarity of two sample sets.|A B| / |A B|
Two binary sets.M11 / (M01 + M10 + M11)
Jeffrey Divergence. Distance between two vector spaces.Σ (xi log(xi/avgi) + yi log(yi/avgi))n
i=1
![Page 25: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/25.jpg)
Other Distance Metrics City Block or Manhattan Distance. Euclidean Distance. Chi-Square. Canberra Distance.
Proceedings ACM SAC. 2008;:1225-1230
![Page 26: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/26.jpg)
Related Efforts - Hysteroscopy Use Jeffrey divergence on color
histogram to identify segments. Relevant segments based on image
redundancy. No understanding
of the content of each segment.
Proceedings 27th IEEE-EMBS. 2005;:5680-5683
Mayo Clinic
![Page 27: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/27.jpg)
Related Efforts - Echocardiogram Use cosine similarity and edge
change ratio to identify video segments.
State-based modeling. Identify states in each
video segment. Diastole (resting). Systole (contracting).
IEEE Transaction on Information Technology in Biomedicine. 2008 May;12(3):366-376
Medline Plus
![Page 28: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/28.jpg)
Overview Background
Laparoscopic Surgery Image Classification
Methods Discussion
Summary: Spectral and textural features compared with similarity metrics.
![Page 29: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/29.jpg)
Methods Our objective.
Identity key portions of surgical procedure to aid in video-based assessment.
Stage I is to identify the "critical view".Video
Segments
Frames
![Page 30: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/30.jpg)
Tools FFmpeg
http://ffmpeg.mplayerhq.hu/ Extract JPEG images.
ImageJ http://rsbweb.nih.gov/ij/ Macros and Java plugins.
![Page 31: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/31.jpg)
Work Plan Identify videos for analysis. Convert videos to JPG. Evaluate ability to identify critical view.
Color histogram. Binary histogram. 3D histogram. Spatial-dependency matrix. Jaccard coefficient, Jeffrey divergence.
![Page 32: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/32.jpg)
Algorithm
Feature ExtractionImageJ Color Histograms
Binary Histograms3D Histograms
Spatial-Dependency Matrices
Similarity Metric
Critical View?
Critical View
Random ImageImage Extraction
FFmpeg
![Page 33: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/33.jpg)
Overview Background
Laparoscopic Surgery Image Classification
Methods Discussion
Summary: Attempt to identify the critical view by comparing image features with similarity metrics.
![Page 34: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/34.jpg)
Discussion Color and binary histograms do not
correlate with the critical view. They do, however, predict when we are
in the abdomen. Currently working on 3D histograms
and spatial-dependency matrices. NIH grant application under
development.
![Page 35: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/35.jpg)
Challenges Live tissue (vs. solid objects).
Deformable. Normal variation. Disease states.
May need to consider. Temporal information. Relevant clinical data of the patient. Critical view "rectangle" (contextual).
![Page 36: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/36.jpg)
Summary We are comparing image features
with similarity metrics to identify the critical view.
This is a first step in automated video summarization, to help with video-assisted evaluation of laparoscopic surgery.
![Page 37: Video Summarization of Key Events Stage I - The Critical View](https://reader035.fdocuments.net/reader035/viewer/2022062520/56815bef550346895dc9dfea/html5/thumbnails/37.jpg)
Thank You