Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey
30
Forever Young A Tribute to the Grandmaster & A Recount of Personal Journey Jiebo Luo University of Rochester
-
Upload
goergen-institute-for-data-science -
Category
Education
-
view
128 -
download
0
Transcript of Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey
Forever YoungATributetotheGrandmaster&ARecountofPersonalJourney
Jiebo LuoUniversity of Rochester
2
Why Do We Need to Do Research?
3
Why Do We Need to Do Research?
4
Why Do We Need to Do Research?
5
Learned from the GrandmasterCuriosity
Open mindedNew problemsNew techniques
Passion Love, focus - sustained
ScholarshipUnaffected by noise, hypeUncompromised integrity
Never too old to learn
6
PhD ThesisImage Processing
Image and Video Codingwavelet transform
Scene-adaptive Coding primitive/budding scene understanding
7
Kodak Years • Intelligent Image Processing and Semantic Image Understanding
– Automatic Redeye Removal
– Automatic Image Orientation Detection
– Image Classification
– Image Annotation
– Geotagging
– Context + Content
From Classification to Description Recognizing Realistic Actions from Videos "in the Wild"UCF-11 to UCF-101(CVPR 2009)
Similarity btw Videos Cross-Domain Learning
Visual Event Recognition in Videos by Learning from Web Data (CVPR2010 Best Student Paper)
Heterogeneous Feature Machine For Visual Recognition(ICCV 2009)
Image Captioning with Semantic Attention
• Motivations– Real-world Usability
• Help visually impaired people, learning-impaired– Improving Image Understanding
• Classification, Objection detection– Image Retrieval
1. a young girl inhales with the intent of blowing out a candle2. girl blowing out the candle on an ice cream
1. A shot from behind home plate of children playing baseball
2. A group of children playing baseball in the rain
3. Group of baseball players playing on a wet field
Key Elements
• Additional textual information– Leverage noisy titles, tags or captions (Web)
Key Elements
• Additional textual information– Leverage noisy titles, tags or captions (Web)– Leverage visually similar nearest neighbor images
Key Elements
• Additional textual information– Leverage noisy titles, tags or captions (Web)– Leverage visually similar nearest neighbor images– Incorporate success of low-level tasks
• Visual attribute detection
Attention Model on Attributes
• Instead of using the same set of attributes at every step
• At each step, select the attributes (attention)
m mtmt kKwatt ),(
)softmax VK(wTtt
))],,(;([),( 11 tttttt hKwattxfhxfh
Overall Framework
• Training with a bilinear/bilateral attention model
ht
pt
xt
v
{Ai}
Yt~
RNN
Image
CNN
AttrDet 1
AttrDet 2
AttrDet 3
AttrDet N
t = 0
Word
Performance
• MS-COCO Image Captioning Challenge
TGIF: A New Video Dataset and Benchmark
Examples
a skate boarder is doing trick on his skate board.
a gloved hand opens to reveal a golden ring.
a sport car is swinging on the race playground
the vehicle is moving fast into the tunnel
Machine Generated Sentence Examples
19
Two Most Important Social Signals (IMO)• User
• Sentiment
• Cross‐modality Consistent Regression
Joint Visual-Textual Sentiment Analysis
21
Infer User Demographics & Interests
• Social interactions and social activities• Public health surveillance• Web sentiment analysis and trend prediction• Cyber terrorism, extremism, and activism• Fads and infectious ideas• Marketing intelligence analytics • Traffic and human mobility patterns• Human and environment• Social unrest, protest and riot
Understanding the Pulse of Society
23
•Correctly classified flattering expressions
•Correctly classified unflattering expressions
•Correctly classified neutral expressions
• Acts like a prism to reveal the spectrum of opinions
• Competitive Vector Autoregressive Model
2012: Calling the Swing States
2016: Fine-Grained Campaign Analysis
2016: Shifting Tide? Too Close to Call
25
August 26, 2016
September 15, 2016
Social Multimedia
Visual Data
Textual Data
Computer Vision
NLP
User Demographics
User Activities
Behavior Patterns
Using Social Multimedia to Study Social Problems
Time Pattens of Underage Alcohol Use
Temporal Patterns of Underage Drinking
Drug Use Pattern from Instagram
29
When We Were Young
29
30
Forever Young
30
Forever young,I want to be forever young.Do you really want to live forever?Forever, and ever
Forever young,I want to be forever young.Do you really want to research forever?Forever, and ever
Image Processing, Computer Vision, Multimedia, Social Media, Big Data, …
(A younger version of an old song)
……
Let’s Celebrate the Forever Young Huang Academic Tree!
