Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy...
Transcript of Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy...
![Page 1: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/1.jpg)
Koichi Okamoto, Keiji Yanai
The Univ. of Electro-CommunicationsTokyo, Japan
![Page 2: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/2.jpg)
• Food recording habit is helpful for diet
• However to record
foods, people
have to
-take photos
-input food names
-estimate calories
Time-consuming !
Give up recording soon !!
![Page 3: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/3.jpg)
Food recording app by image recognition ◦ It requires taking meal photos before eating
inappropriate for sharing large platter, hot pot, or BBQ
![Page 4: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/4.jpg)
A novel food recording system Is applicable even for food sharing situation
Recognize eating action during a meal
Record all eaten foods, calories and amounts
smartphone Monitoring eating action
Eating meat !
![Page 5: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/5.jpg)
Mouth detection Chopsticks detection
Eating region detection
Food item
classification
Update total calories
If they come close
![Page 6: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/6.jpg)
1. Detect a face
2. Detect a mouth from the face region
(We used face and mouse detectors in OpenCV.)
Without face detection With face detection
![Page 7: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/7.jpg)
1. Detect moving areas by background subtraction
2. Detect lines from the moving areas by Hough transform
The entire image Only moving area
![Page 8: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/8.jpg)
If mouth and chopstick regions come
close to each other “eating”
Eating region containing a food item
![Page 9: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/9.jpg)
Fusion of two Image features1. Bag-of-features with ORB
2. HSV color histogram
Classifier : Linear SVM
→fast χ2feature map (Vedaldi al et. PAMI12)
![Page 10: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/10.jpg)
![Page 11: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/11.jpg)
Target meal: ‘Yakiniku Grill’◦ Japanese-style BBQ: grill thin-sliced meat & vege.
Target food items for classifiction:5 kinds of typical items in Yakiniku Meat, Rice, Pumpkin, Bell pepper, Carrot
Sharing a grill outdoor Yakiniku alone
![Page 12: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/12.jpg)
1. Evaluation of food classification accuracy
2. User study on system usability
![Page 13: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/13.jpg)
Classification accuracy : 74.8%#Training images : 450 #Test images : 50
![Page 14: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/14.jpg)
Comparison of two systems:◦ Baseline system:
manual recording system by touching food item buttons
◦ Proposed system eating action recognition
5-step evaluation ◦ 5 (better) .... 3 (soso) .... 1 (bad)
Two questions for five subjects
Screen of the
baseline system
![Page 15: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/15.jpg)
Two questions for five subjecs:
1.How easy to take eating record ?
Baseline system: 2.0
Proposed system: 4.8
2.How easy to see calorie intake during meal ?
Baseline system: 3.0Proposed system: 3.4
Better than the manual baseline system
![Page 16: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/16.jpg)
A novel food recording systemRecognize eating action during a meal
Record all eaten foods, calories and amounts
Is appropriate for food sharing situation such as “Yakiniku”
Accuracy: 74.8%
User study: Effective,but need to improve UI
![Page 17: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/17.jpg)
• Add other types of meals:
• Improve classification accuracy:→ Fisher Vector
• Estimate the food volume
![Page 18: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/18.jpg)
Yakiniku alone with !
![Page 19: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/19.jpg)
![Page 20: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/20.jpg)
Mapo doufu◦ Tofu, Minced meat…
Shrimps in chili sauce◦ Shrimps, Onion…
sweet‐and‐sour pork◦ Pork, Carrot, Pineapple…
Eat some food items in one bite
→Pre-defined fixed calories is average calories
![Page 21: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/21.jpg)
Mouth detection accuracy is high
Chopsticks detection accuracy is not high◦ There are lines in background
◦ Detected Chopsticks are longer than the actual
False Eating region
![Page 22: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/22.jpg)
Fork and knife are straight◦ Can be same method as chopsticks detection
If eat sandwiches or onigiri◦ We have to detect hand
![Page 23: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/23.jpg)
Left : real foodRight : food sample
![Page 24: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/24.jpg)
![Page 25: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/25.jpg)
Mixture of real foods and food samples for training image set◦ food samples are not good for training
image set
Food sample Real Food
![Page 26: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/26.jpg)
Koichi Okamoto and Keiji YanaiThe University of Electro-Communications, Tokyo
![Page 27: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/27.jpg)
• Recording food habit is helpful for dietary
• However,
-take photos
-input food names
-estimate calorie
-etc…
Time-consuming !
Quit recording soon !!
![Page 28: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/28.jpg)
Food recording app by image recognition ◦ It requires taking meal photos before eating
inappropriate for sharing large platter, hot pot, and BBQ
![Page 29: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/29.jpg)
Recognize eating action during meal
Automatic recording all the eaten foodfood item category, calorie, amounts…
smartphone Monitoring eating action
Eating meat !Eating meat !
Very new type of mobile food recording system
![Page 30: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/30.jpg)
Capture frame
Mouth detection Chopstick detection
segmentation
Classification of food item
Record eating action
![Page 31: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/31.jpg)
Android App : (In the current implemention, the target is a “Yakiniku” meal.)
![Page 32: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/32.jpg)
Two kinds of evaluations
1. Classification accuracy (5-fold CV)
2. Simple user study (5 subjects)
→ 74.4 %
→Comparison with the baseline systemwhich has no recognition in 5-steps
baseline GrillCam
usability 2.0 4.8
Much better than the no-recognition baseline
![Page 33: Koichi Okamoto, Keiji Yanaiimg.cs.uec.ac.jp/pub/conf14/140714okak_8_ppt.pdfClassification accuracy (5-fold CV) 2. Simple user study (5 subjects) →74.4 % →Comparison with the baseline](https://reader035.fdocuments.net/reader035/viewer/2022063021/5fe49710a9e090637f342e72/html5/thumbnails/33.jpg)
You can experience a new type of meal with !