Transcript of Situation Recognition from Multimodal Data Tutorial (ICME2016)
125
SITUATION RECOGNITION FROM MULTIMODAL DATA
Vivek K Singh1 Siripen Pongpaichet2 and Ramesh Jain2
1Rutgers University 2University of California Irvine
125
Todayrsquos slides
2
httpwwwspringercomusbook9783319305356Or email us for a softcopy
httpbitly29JL30M
125
Course Outline1) Concept recognition from Multimedia data (20 mins Ramesh Jain)bull Trends bull Why situation recognition is different from object event scene recognition etc2) Situation recognition across multiple research domains (20 mins Vivek Singh)bull Situation Algebra Situation Calculus Robotics hellip3) Situation recognition (45 mins Vivek Singh) bull Situation modeling bull Situation operators 4) Designing situation based applications (30 mins Siripen Pongpaichet)bull Motivation and essential requirementsbull Application scenarios Thailand flood hurricane Sandy city sensing asthma relief5) Future trends and open problems (20 mins Ramesh Jain)bull Future trends bull Open problems for Multimedia research 6) Question and Answers (15 mins)
3
125
CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)
4
125
Introduction
bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event
scene recognition etc)
5
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Todayrsquos slides
2
httpwwwspringercomusbook9783319305356Or email us for a softcopy
httpbitly29JL30M
125
Course Outline1) Concept recognition from Multimedia data (20 mins Ramesh Jain)bull Trends bull Why situation recognition is different from object event scene recognition etc2) Situation recognition across multiple research domains (20 mins Vivek Singh)bull Situation Algebra Situation Calculus Robotics hellip3) Situation recognition (45 mins Vivek Singh) bull Situation modeling bull Situation operators 4) Designing situation based applications (30 mins Siripen Pongpaichet)bull Motivation and essential requirementsbull Application scenarios Thailand flood hurricane Sandy city sensing asthma relief5) Future trends and open problems (20 mins Ramesh Jain)bull Future trends bull Open problems for Multimedia research 6) Question and Answers (15 mins)
3
125
CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)
4
125
Introduction
bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event
scene recognition etc)
5
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Course Outline1) Concept recognition from Multimedia data (20 mins Ramesh Jain)bull Trends bull Why situation recognition is different from object event scene recognition etc2) Situation recognition across multiple research domains (20 mins Vivek Singh)bull Situation Algebra Situation Calculus Robotics hellip3) Situation recognition (45 mins Vivek Singh) bull Situation modeling bull Situation operators 4) Designing situation based applications (30 mins Siripen Pongpaichet)bull Motivation and essential requirementsbull Application scenarios Thailand flood hurricane Sandy city sensing asthma relief5) Future trends and open problems (20 mins Ramesh Jain)bull Future trends bull Open problems for Multimedia research 6) Question and Answers (15 mins)
3
125
CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)
4
125
Introduction
bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event
scene recognition etc)
5
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
>
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
>
125
Fused Information
>
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
Situation recognition across multiple research domains
Related Work Data to Situations
Defining Situations Situation Calculus
Situation Calculus Quick overview
Problems with this approach
Situations
Situations commonalities
Situation Definition
Overall Framework Motivating example
Applications
Situation recognition FRamework
Slide 34
A) Situation Modeling
Building Blocks Operands
Building Blocks Operators
Situation Modeling
Slide 39
B) Situation evaluation Workflow
Data Representation
Data Model
Situation Recognition Algebra
Media processing engine
Implementation and results
Testing Data Representation + Algebra
Sample Queries
Slide 48
Flickr Social Emages
Seasonal characteristics analysis
Year average Peak of green
Other approaches
FraPPE a vocabulary to represent heterogeneous spatio-temporal
Proposed Approach
FraPPE visually
FraPPE visually (2)
FraPPE visually (3)
FraPPE visually (4)
FraPPE visually (5)
FraPPE visually (6)
City Sensing listens to the pulse of Milano Design Week on Apri
Tweeting Cameras
Physical amp Social Sensors Fusion For Situation Awareness
Real-world Events
Probabilistic Spatio-Temporal Data
Physical Sensors (Concept ldquoCrowdrdquo)
Social Sensors (MillionMarchNYC BlackLivesMatter)
Fused Information
CMage
Fusing Sensor Cmage with Social Cmage
DESIGNING SITUATION BASED APPLICATIONS
Outline
EventShop Requirement
EventShop Architecture
EventShop UI
Demo
Demo (2)
Building applications using EventShop
Asthma Management Application
Asthma Management Application
Asthma Risk Estimation
Experiment Results
Asthma Risk Estimator Model and Result
A Graph Based Multimodal Geopatial Interpolation Framework
Detecting Situations from Micro-Reports
PHOTOS as Kodak Moments
Disruption PHOTOS as Information
Reports of Events from Journalists
Reports of Events from Citizens
Micro-Blogs
Multimedia Micro-Reports (MMRs) are now and future
What are the challenges
Capturing and Reporting Events with Krumbs SDK
Capturing and Reporting Events with Krumbs SDK (2)
Real-time MMR Dashboard
Converting Multimedia Data into MMR
Converting Multimedia Data into MMR (2)
Number of photos in London per day
Evolving Photo Concepts in London
Detecting Olympic Games
Evolving Photo Concepts in Beijing
Detecting London Olympic Games
Detecting Emergency Situations
Photos from City Flood Cluster
Smart City Project in DC
Integrating MMR with other data sources for Situation Recogniti
Trash Fill Level Situation in DC
Prediction based on Events History
Conclusion
Future trends and open problems
Future Trends
This century is different from the last
Slide 113
Slide 114
Slide 115
We are immersed in Big Data
Slide 117
Data as a Platform
Contact Information
Useful links
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects