Vision System for Wing Beat Analysis of Bats in the Wild
description
Transcript of Vision System for Wing Beat Analysis of Bats in the Wild
![Page 1: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/1.jpg)
Vision System for Wing Beat Analysis of Bats in the Wild
1Boston University Department of Computer Science2Boston University Department of Biology
Mikhail Breslav1, Nathan W. Fuller2, and Margrit Betke1
ComputerScience
![Page 2: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/2.jpg)
Motivation• Behavior• Aerial Vehicles
![Page 3: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/3.jpg)
Past Work
• Study Kinematics (Hubel. 2012)
– Wind Tunnel– High Resolution Cameras
• 3D Tracking (Wu. 2009)
– Outdoor environment – Model Bats as Points
• Behavior and Trajectory Analysis (Theriault. 2010, Fisher. 2010)
![Page 4: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/4.jpg)
Goal• Estimate Wing Beat Frequencies– Potential to improve tracking
TWing Beat Frequency: 1/T
Hubel et al. 2012
![Page 5: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/5.jpg)
Challenging Data• Unpredictable Motions• Relatively Low Resolution• In FOV for a Short Time
![Page 6: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/6.jpg)
Segmentation and Tracking• Foreground / Background Estimation• Kalman Filter
![Page 7: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/7.jpg)
Shape-time Signals• Output of Tracker• Define “Shape”
![Page 8: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/8.jpg)
Prototype Shapes
• Assumption– There are shapes that uniquely identify 3D
poses for a given camera
• Example
• Currently chosen manually
“up” “down” “neutral”
![Page 9: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/9.jpg)
Main Idea• A prototype shape is equal to a 3D
pose• Repeating prototype shapes in a
shape-time signal
Estimate Wing Beat
![Page 10: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/10.jpg)
Shape Comparison• Shape Distance– Shape Context Descriptor (Belongie et al. 2002)• Invariant to translation, scale, and optionally rotation
– Hungarian Algorithm• Establish Correspondences
• Estimate Wing Relative to Body with feature W
![Page 11: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/11.jpg)
Shape Similarity Scores• Use Shape Distance and Ratio W to assign
similarity score– Also consider the ‘none’ hypothesis
‘None’
.24 .16 .43 .17
![Page 12: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/12.jpg)
“up” “down” “neutral”
Process Shape-Time Signal
• Find confident matches to prototype shapes
![Page 13: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/13.jpg)
“up”
Process Shape-Time Signal
“down”
“neutral”
Time Axis
![Page 14: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/14.jpg)
Fast Fourier Transform
Time Axis
“up”
“down”
“neutral”
FFT
![Page 15: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/15.jpg)
Fast Fourier Transform
Periodicity Estimate of 9.76 Hz
![Page 16: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/16.jpg)
Experimental Results• 20 Bats • Both Automatic and Manual estimates
![Page 17: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/17.jpg)
Discussion• Reasonable Estimates– Deviates from manual annotations by 1.3 Hz on
average, standard deviation 1.8 Hz– Falls within 10-15 Hz as reported in biology
literature (Foehring. 1984)
• Main Contribution– System for using shapes to estimate wing beat– First to do this for bats in the wild • Vision based system
![Page 18: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/18.jpg)
Future work
• Choosing prototype shapes – Automatically and Intelligently
• Understand the mapping between 2D shapes and 3D poses for a given model– Generalize across datasets
• Try more robust shape comparison measures
![Page 19: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/19.jpg)
Thank You for your Attention!
![Page 20: Vision System for Wing Beat Analysis of Bats in the Wild](https://reader036.fdocuments.net/reader036/viewer/2022081514/5681557f550346895dc34a31/html5/thumbnails/20.jpg)
Questions?
Holding a bat!