Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday...
Transcript of Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday...
![Page 1: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/1.jpg)
Building IMU-based Gesture Recognition!!!Jennifer Wangjewang.net
tJewang.net // Hackaday Supercon ‘18t
![Page 2: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/2.jpg)
Jennifer WangSoftware Engineer & Hardware PrototyperI <3 sensors
● Phones● Wearables● Robots● Telescopes
jewang.net / [email protected]
Hello world!
tJewang.net // Hackaday Supercon ‘18t
![Page 3: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/3.jpg)
PreambleThank you to these lovely people:
● Tim ‘mithro’ Ansell● Kat Fang● Cynthia Gan● Samy Kamkar● Sophi Kravitz● Jinna Lei● Jen Tong● Tony Valderrama● Ruth Grace Wong
Please go vote!
tJewang.net // Hackaday Supercon ‘18t
![Page 8: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/8.jpg)
Wingardium (Leviosa)Levitation
FlippendoBread and butter
tJewang.net // Hackaday Supercon ‘18t
![Page 9: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/9.jpg)
Final Producthttps://github.com/jewang/gesture-demo
tJewang.net // Hackaday Supercon ‘18t
![Page 10: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/10.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 11: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/11.jpg)
Music Control Gloves (Mimu Gloves used by Imogen Heap)
IMUs + Machine Learning = Lots of fun!
Smartwatch gestures/input/stealing ATM Pins
Android Camera Control Gestures Gait-based auth
Fitness / Sleep Tracking “actigraphy” (FitBit)
Real firebending (Allen Pan)
tJewang.net // Hackaday Supercon ‘18t
![Page 12: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/12.jpg)
Sample sensor data
Run data on deep learning model
Post-process result
Output!!!
Generate Features
✨Machine Learning ✨
What will the final product look like?
tJewang.net // Hackaday Supercon ‘18t
![Page 13: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/13.jpg)
What will the final product look like?
Sample sensor data
Run data on deep learning model
Post-process result
Output!!!
Generate Features
✨Machine Learning ✨
What is a feature?Summary. Captures what is interesting.
tJewang.net // Hackaday Supercon ‘18t
![Page 25: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/25.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 26: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/26.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 27: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/27.jpg)
Define & scope the Problem
What are your resources? Time? $?
Who/when/where will it be used?
Power, Latency, Memory Req’s?
When is your model good enough?
tJewang.net // Hackaday Supercon ‘18t
![Page 28: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/28.jpg)
Define & scope the Problem
What are your resources? Time? $? Me. 1 Month! <$200?
Who/when/where will it be used?
Power, Latency, Memory Req’s?
When is your model good enough?
tJewang.net // Hackaday Supercon ‘18t
![Page 29: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/29.jpg)
Define & scope the Problem
What are your resources? Time? $? Me. 1 Month! <$200?
Who/when/where will it be used? Me. Halloween evening! Outdoors, in my neighborhood.
Power, Latency, Memory Req’s?
When is your model good enough?
tJewang.net // Hackaday Supercon ‘18t
![Page 30: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/30.jpg)
Machine Learning on an Embedded System: Pick 2
Memory-efficient (Model Size)
Real-timeFast Detection
(Latency)
Accuracy
tJewang.net // Hackaday Supercon ‘18t
![Page 31: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/31.jpg)
Machine Learning on an Embedded System: Pick 2
Memory-efficient (Model Size)
Real-timeFast Detection
(Latency)
Accuracy Ship as fast as possible
tJewang.net // Hackaday Supercon ‘18t
![Page 32: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/32.jpg)
Define & scope the Problem
What are your resources? Time? $? Me. 1 Month! <$200?
Who/when/where will it be used? Me. Halloween evening! Outdoors, in my neighborhood.
Power, Latency, Memory Req’s?
When is your model good enough?
tJewang.net // Hackaday Supercon ‘18t
![Page 33: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/33.jpg)
Define & scope the Problem
What are your resources? Time? $? Me. 1 Month! <$200?
Who/when/where will it be used? Me. Halloween evening! Outdoors, in my neighborhood.
Power, Latency, Memory Req’s? Don’t worry about it for now. Ship as fast as possible.
When is your model good enough?
tJewang.net // Hackaday Supercon ‘18t
![Page 34: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/34.jpg)
Define & scope the Problem
What are your resources? Time? $? Me. 1 Month! <$200?
Who/when/where will it be used? Me. Halloween evening! Outdoors, in my neighborhood.
Power, Latency, Memory Req’s? Don’t worry about it for now. Ship as fast as possible.
When is your model good enough? When Halloween happensWhen kids are happy :)
tJewang.net // Hackaday Supercon ‘18t
![Page 35: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/35.jpg)
tJewang.net // Hackaday Supercon ‘18t
What sensors?
What is the basic architecture?
Define & scope the hardware
![Page 36: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/36.jpg)
Common sensors for gesture recognition● Camera?
tJewang.net // Hackaday Supercon ‘18t
![Page 37: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/37.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside
tJewang.net // Hackaday Supercon ‘18t
![Page 38: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/38.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside● IR/Wireless Beacons?
tJewang.net // Hackaday Supercon ‘18t
![Page 39: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/39.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside● IR/Wireless Beacons? → No, I can’t mount these all over my
neighborhood
tJewang.net // Hackaday Supercon ‘18t
![Page 40: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/40.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside● IR/Wireless Beacons? → No, I can’t mount these all over my
neighborhood● Sonar / Audio?
tJewang.net // Hackaday Supercon ‘18t
![Page 41: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/41.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside● IR/Wireless Beacons? → No, I can’t mount these all over my
neighborhood● Sonar / Audio? → No, the environment will be noisy because
I’ll be outside! Walking around!
tJewang.net // Hackaday Supercon ‘18t
![Page 42: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/42.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside● IR/Wireless Beacons? → No, I can’t mount these all over my
neighborhood● Sonar / Audio? → No, the environment will be noisy because
I’ll be outside! Walking around!● Magic E/M Sensing?
tJewang.net // Hackaday Supercon ‘18t
![Page 43: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/43.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside● IR/Wireless Beacons? → No, I can’t mount these all over my
neighborhood● Sonar / Audio? → No, the environment will be noisy because
I’ll be outside! Walking around!● Magic E/M Sensing? → No, I am not an academic :)
tJewang.net // Hackaday Supercon ‘18t
![Page 44: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/44.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside● IR/Wireless Beacons? → No, I can’t mount these all over my
neighborhood● Sonar / Audio? → No, the environment will be noisy because
I’ll be outside! Walking around!● Magic E/M Sensing? → No, I am not an academic :)● Inertial Measurement Unit (IMU)?
tJewang.net // Hackaday Supercon ‘18t
![Page 45: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/45.jpg)
Common sensors for gesture recognition● Camera? → No, it’s dark outside● IR/Wireless Beacons? → No, I can’t mount these all over my
neighborhood● Sonar / Audio? → No, the environment will be noisy because
I’ll be outside! Walking around!● Magic E/M Sensing? → No, I am not an academic :)● Inertial Measurement Unit (IMU)? → Yes!
tJewang.net // Hackaday Supercon ‘18t
![Page 46: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/46.jpg)
What is an inertial measurement unit (IMU)?✨ Orientation! ✨
Accelerometer + Gyroscope + Magnetometer
3 Axis
tJewang.net // Hackaday Supercon ‘18t
![Page 47: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/47.jpg)
tJewang.net // Hackaday Supercon ‘18t
What sensors?
What is the basic architecture?
Define & scope the hardware
![Page 48: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/48.jpg)
tJewang.net // Hackaday Supercon ‘18t
What sensors? IMU (Bosch BNO055)
What is the basic architecture?
Define & scope the hardware
![Page 49: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/49.jpg)
Parts!UART
USB Audio
USB Power
Raspi Zero WBNO055
IMU
Phone battery
SpeakerGlue this to a cosplay wand.tJewang.net // Hackaday Supercon ‘18t
![Page 50: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/50.jpg)
Wand
Parts
Battery
Raspi Zero WIMU
Speaker goes up your sleeve ;)tJewang.net // Hackaday Supercon ‘18t
![Page 52: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/52.jpg)
tJewang.net // Hackaday Supercon ‘18t
What sensors? IMU (Bosch BNO055)
What is the basic architecture?
Define & scope the hardware
![Page 53: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/53.jpg)
tJewang.net // Hackaday Supercon ‘18t
What sensors? IMU (Bosch BNO055)
What is the basic architecture? Raspi Zero W glued to a wand. Embedded Linux.
Define & scope the hardware
![Page 54: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/54.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 55: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/55.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 56: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/56.jpg)
What algorithm should I use?
Deep learning! Signal processingTraditional Machine Learning
Domain knowledge Not much Lots! Medium
Amount of data Lots! Not as much Medium
Easy to debug Harder Easier Medium
tJewang.net // Hackaday Supercon ‘18t
![Page 57: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/57.jpg)
What algorithm should I use?
Deep learning! Signal processingTraditional
Machine Learning
Domain knowledge Not much Lots! Medium
Amount of data Lots! Not as much Medium
Easy to debug Harder Easier Medium
tJewang.net // Hackaday Supercon ‘18t
![Page 58: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/58.jpg)
What algorithm should I use?
Deep learning! Signal processingTraditional
Machine Learning
Domain knowledge Not much Lots! Medium
Amount of data Lots! Not as much Medium
Easy to debug Harder Easier Medium
1. Ship your MVP2. Get more users3. More users = more data4. Switch to deep learning
tJewang.net // Hackaday Supercon ‘18t
![Page 59: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/59.jpg)
Software Used● Language: Python3● Numerical Libraries: Pandas, Numpy● Data Notebook: Jupyter Notebook● Data Visualization: Plot.ly● Machine learning library: scikit-learn
tJewang.net // Hackaday Supercon ‘18t
![Page 68: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/68.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 69: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/69.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 70: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/70.jpg)
Planning data collection● How much data do you need?
● Look for pre-existing data
● More data = More💰 / 🕒Design collection procedure, manage data collectors, QA...
tJewang.net // Hackaday Supercon ‘18t
![Page 71: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/71.jpg)
Doing data collectionTime to code!
Lock mechanical design!
Match & record data for all expected use cases!
Who × Posture × When × Where
tJewang.net // Hackaday Supercon ‘18t
![Page 72: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/72.jpg)
Record data for all expected use cases!Wingardium Flippendo Negative
Sitting Couch ✓ ✓ ✓
Sitting Chair ✓ ✓ ✓
Sitting Floor ✓ ✓ ✓
Standing ✓ ✓ ✓
Walking ✓ ✓ ✓
tJewang.net // Hackaday Supercon ‘18t
![Page 73: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/73.jpg)
Record data for all expected use cases!Wingardium Flippendo Negative*
Sitting Couch ✓ ✓ ✓
Sitting Chair ✓ ✓ ✓
Sitting Floor ✓ ✓ ✓
Standing ✓ ✓ ✓
Walking ✓ ✓ ✓
tJewang.net // Hackaday Supercon ‘18t
![Page 75: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/75.jpg)
Final data collected for magic wandcollect_data.py → CSV
286 1.5s traces98 Wingardium99 Flippendo89 Negative
~7 minutes of data257,000+ data points
tJewang.net // Hackaday Supercon ‘18t
![Page 76: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/76.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 77: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/77.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 78: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/78.jpg)
Train model & iterate● Explorations.ipynb● Mostly data cleaning :P
tJewang.net // Hackaday Supercon ‘18t
![Page 80: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/80.jpg)
Features● max_accel● min_accel● Range_accel● mean_accel● std_accel
● max_gyro● min_gyro● range_gyro● mean_gyro● std_gyro
tJewang.net // Hackaday Supercon ‘18t
![Page 81: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/81.jpg)
Train model & iterate● Iteration #1 → Didn’t work well. 102 traces.
tJewang.net // Hackaday Supercon ‘18t
![Page 82: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/82.jpg)
Train model & iterate● Iteration #1 → Didn’t work well. 102 traces.
○ Clean Data! Collect new data!
tJewang.net // Hackaday Supercon ‘18t
![Page 83: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/83.jpg)
Train model & iterate● Iteration #1 → Didn’t work well. 102 traces.
○ Clean Data! Collect new data!● Iteration #2 → Didn’t work well. 167 traces.
tJewang.net // Hackaday Supercon ‘18t
![Page 84: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/84.jpg)
Train model & iterate● Iteration #1 → Didn’t work well. 102 traces.
○ Clean Data! Collect new data!● Iteration #2 → Didn’t work well. 167 traces.
○ Clean Data! Collect new data!
tJewang.net // Hackaday Supercon ‘18t
![Page 85: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/85.jpg)
Train model & iterate● Iteration #1 → Didn’t work well. 102 traces.
○ Clean Data! Collect new data!● Iteration #2 → Didn’t work well. 167 traces.
○ Clean Data! Collect new data! ● Iteration #3: → Worked OK, distinguish gestures. 286 traces.
tJewang.net // Hackaday Supercon ‘18t
![Page 86: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/86.jpg)
Train model & iterate● Iteration #1 → Didn’t work well. 102 traces.
○ Clean Data! Collect new data!● Iteration #2 → Didn’t work well. 167 traces.
○ Clean Data! Collect new data!● Iteration #3: → Worked OK, distinguish gestures. 286 traces.
○ Feature Design
tJewang.net // Hackaday Supercon ‘18t
![Page 89: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/89.jpg)
WingardiumAc
cel
Gyro
FlippendoWingardium has 2 accel z-axis peaks while flippendo has 3!
tJewang.net // Hackaday Supercon ‘18t
![Page 90: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/90.jpg)
WingardiumAc
cel
Gyro
FlippendoWingardium has 2 accel z-axis peaks while flippendo has 3!
Hack a75% Accurate Peak Counter
tJewang.net // Hackaday Supercon ‘18t
![Page 91: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/91.jpg)
Features● max_accel● min_accel● Range_accel● mean_accel● std_accel
● max_gyro● min_gyro● range_gyro● mean_gyro● std_gyro
tJewang.net // Hackaday Supercon ‘18t
![Page 92: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/92.jpg)
Features● max_accel● min_accel● Range_accel● mean_accel● std_accel● accel_z_peaks
● max_gyro● min_gyro● range_gyro● mean_gyro● std_gyro
tJewang.net // Hackaday Supercon ‘18t
![Page 93: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/93.jpg)
Train model & iterate● Iteration #1 → Didn’t work well. 102 traces.
○ Clean Data! Collect new data!● Iteration #2 → Didn’t work well. 167 traces.
○ Clean Data! Collect new data!● Iteration #3: → Worked OK, distinguish gestures. 286 traces.
○ Feature Design
tJewang.net // Hackaday Supercon ‘18t
![Page 94: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/94.jpg)
Train model & iterate● Iteration #1 → Didn’t work well. 102 traces.
○ Clean Data! Collect new data!● Iteration #2 → Didn’t work well. 167 traces.
○ Clean Data! Collect new data!● Iteration #3: → Worked OK, distinguish gestures. 286 traces.
○ Feature Design● Iteration #4 → Good enough. It’s Halloween! Ship it!
tJewang.net // Hackaday Supercon ‘18t
![Page 95: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/95.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 96: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/96.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 98: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/98.jpg)
Productionize the model● gesture_detector.py● <100 lines of code● Might need a little / lot of tuning ;)
tJewang.net // Hackaday Supercon ‘18t
![Page 99: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/99.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
tJewang.net // Hackaday Supercon ‘18t
![Page 100: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/100.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
jewang.net / [email protected] tJewang.net // Hackaday Supercon ‘18t
Build more cool stuff!
![Page 101: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/101.jpg)
IMUs + Machine Learning on a large scale...I <3 sensors
● City walkability● Better health phenotypes● Depression treatments● Census data● Earthquake detection
tJewang.net // Hackaday Supercon ‘18t
![Page 102: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/102.jpg)
Music Control Gloves (Mimu Gloves used by Imogen Heap)
IMUs + Machine Learning = Lots of fun!
Smartwatch gestures/input/stealing ATM Pins
Android Camera Control Gestures Gait-based auth
Fitness / Sleep Tracking “actigraphy” (FitBit)
Real firebending (Allen Pan)
tJewang.net // Hackaday Supercon ‘18t
![Page 103: Gesture IMU-based Recognition!!! Buildingjewang.net/static/jewang - magic wand for hackaday supercon.pdf · Gesture Recognition!!! Jennifer Wang jewang.net tJewang.net // Hackaday](https://reader034.fdocuments.net/reader034/viewer/2022052518/5f0b57527e708231d4300958/html5/thumbnails/103.jpg)
1. Define & scope the problem
2. Propose machine learning model / algorithm
3. Collect data
4. Train model & iterate
5. Productionize the model
How to Gesture Recognition
Build more cool stuff!
jewang.net / [email protected] tJewang.net // Hackaday Supercon ‘18t