Multimedia rescue 161018
-
Upload
ramesh-jain -
Category
Technology
-
view
183 -
download
2
Transcript of Multimedia rescue 161018
PowerPoint Presentation
Mengfan Tang,Siripen Pongpaichet, andRamesh Jain
Department of Computer ScienceUniversity of California, Irvine
ACMMM 2016, BNI session Tue 18 October 2016 @2PMMultimedia FOR Emergency Situations
Society exists only as a mental concept; in the real world there are only individuals. -- Oscar Wilde
Multimedia: Machines/Devices3
1993
2016
Technology: From Features to Machine Learning
1993
2016
Going From Advertisement and Entertainment
Going From Advertisement and Entertainment to Social Good
Shit Happens!7
Hurricanes - Typhoons
Forest Fires
Earthquakes - TsunamisDisaster Happens!Infectious diseases
Disease Epidemics -- Zika
Riots
7
Life Cycle of Disaster Management8
Focus of discussion today.
Community Computing: Merging 3 Big TrendsCommunity-based AppsReal-time CoordinationPower to Create and Share Ideas
Waze
Uber, Airbnb
Twitter, Instagram
Micro-events from M reportersSituation at Macro Level
Hurricane Matthew
ResponseReport
5-1-1 road traffic3-1-1 Red CrossNOAA SuperRes Radar US appDark Sky App
10
Fundamental Problem during a Disaster 11
Providing Resources and Information Effectively, Efficiently, and Promptly In evolving Emergency Situations.
Disaster Response12Physical World
And
Information SystemsEnvironment, Resources,InformationPersonal Situation, Needs,InformationMatchingAction Signals
Environment and ResourcesDisaster Related Evolving Situations TrafficPrecipitation MeteorologyPower outageFlood levelFoodDiseases
Data SourcesWebcams, Traffic systems (Google)Weather.com NOAAGovernment: Division of Emergency Management Social Media
Personal Situation and NeedsPersonal NeedsShelterFoodWaterFamily and FriendsInformation
Data SourcesVolunteer The-Emergency-Food-Assistance-ProgramDrinking-water, flooded-wellsCitizen reportsGovernment announcements
Shelter locationsFood assistance locations
Boil Water Notice
Timely Alerts and ResponseEvacuation ZoneEvacuation Route
Communication Methods(pull)NewsRadio
Alert Methods(push)Targeted messagesMobile apps
Disaster Report and Response App
Personal Situation RecognitionSituation Recognition and PredictionData Stream Ingestion and AggregationBuilding Emergency Situation ModelsDatabaseReal-Time Situation RecognitionOffline ProcessREPORTRESPONSE
MatchingResourcesDeterminationIdentify People NeedsEmergency Situations
Actionable Decisions
Current Explorations: Clear Potential of Emerging MultimediaEventShopStarted about 8 years agoSelect and combine diverse heterogeneous geo-spatial data streams to detect situationsDemonstrated for several applicationsInspired by PhotoshopMultimedia Micro-Reports (MMR)Started about 2 years ago.Create multimedia platform for citizen sensingDemonstrated using YFCC100M photo releaseInspired by Krumbs
Prototype: Thai Flood Emergency Response
EventShop: Thai Floods
Flood level - ShelterFlood LevelShelter
Classify (Flood level - Shelter)
Taking personalized actions
FILTERLOC=CAAGGFUNC=AVGFILTERLOC=CAGROUPTHRESHOLD
Asthma HospitalizationGround Truth Asthma Risk Area without InterpolationAsthma Risk Area with interpolationAGGFUNC=AVGGROUPTHRESHOLDPM2.5 ConcentrationFrom StationsInterpolatedPM2.5 using SSGPPollenOzoneAQIEventShop: Asthma Risk Estimator Model and Result
FILTERLOC=CAAGGFUNC=AVGFILTERLOC=CAGROUPTHRESHOLD
Asthma HospitalizationGround Truth Asthma Risk Area without InterpolationAsthma Risk Area with interpolationAGGFUNC=AVGGROUPTHRESHOLDPM2.5 ConcentrationFrom StationsInterpolatedPM2.5 using SSGPPollenOzoneAQIEventShop: Asthma Risk Estimator Model and Result
FILTERLOC=CAAGGFUNC=AVGFILTERLOC=CAGROUPTHRESHOLD
Asthma HospitalizationGround Truth Asthma Risk Area without InterpolationAsthma Risk Area with interpolationAGGFUNC=AVGGROUPTHRESHOLDPM2.5 ConcentrationFrom StationsInterpolatedPM2.5 using SSGPPollenOzoneAQIEventShop: Asthma Risk Estimator Model and Result
FILTERLOC=CAAGGFUNC=AVGFILTERLOC=CAGROUPTHRESHOLD
Asthma HospitalizationGround Truth Asthma Risk Area without InterpolationAsthma Risk Area with interpolationAGGFUNC=AVGGROUPTHRESHOLDPM2.5 ConcentrationFrom StationsInterpolatedPM2.5 using SSGPPollenOzoneAQIEventShop: Asthma Risk Estimator Model and Result
FILTERLOC=CAAGGFUNC=AVGFILTERLOC=CAGROUPTHRESHOLD
Asthma HospitalizationGround Truth Asthma Risk Area without InterpolationAsthma Risk Area with interpolationAGGFUNC=AVGGROUPTHRESHOLDPM2.5 ConcentrationFrom StationsInterpolatedPM2.5 using SSGPPollenOzoneAQIEventShop: Asthma Risk Estimator Model and Result
FILTERLOC=CAAGGFILTERLOC=CA
Asthma Risk Area without InterpolationSparse PM 2.5 SensorsPollenOzone AQI
Asthma Risk Area with interpolationAGG Interpolated PM2.5
Physical model outputs
Satellite Images TrafficAssimilating Data Sources = Better Situation Recognition
correlation with asthma hospitalizations increased from 0.17 to 0.45
Is a Smartphone camera a camera?Smartphone collects all metadata related to the Event.Exposure TimeAperture DiameterFlashMetering ModeISO RatingsFocal LengthTimeLocationFace and emotionsActivity parametersContextual reasoningMyriad sensors through wearables
Camera captures photos.Smartphone camera captures events.
Media Data is about W5H{"micro_reports":[{ "where":{"geo_location":{ "latitude":32.90233332316081, "longitude":-117.2441166718801}, "when":{"start_time":"Jun 14, 2009 11:25:19 AM","end_time":"Jun 14, 2009 11:25:19 AM","time_zone":"America/Los_Angeles"}, "what":[{"concept_name":"people","confidence":0.9836078882217407,"visual_concept_provider":"CLARIFAI"}, {"concept_name":"food","confidence":0.8526291847229004,"visual_concept_provider":"CLARIFAI"}], "tag":#niceday #summer", "source":{"default_src":"https://.jpg"}}, "sub_event":[], "why":[]},]}
Photo, videoWhatWhereWhenWhoWhySound
How
Using Photo Concepts for Creating Worldwide Situation Map!
We can detect situations using bag of concepts. 5+ Billion phone photos = Situation Map of the WorldNew frontier for Machine learning.
Figure 5 shows the evolving concepts in Beijing in the year of 2008. As we know, Beijing Olympic Games was held in Beijing in the summer of 2008. This event is clearly reflected in Figure 5 where the number of these ten concepts basketball,court game,gymnastics,people,sport,stadium,swim and tennis reaches peak at the same time of Olympic Games.
24
Resource
ShelterMultimedia Micro-Reports
SituationNeeds
Rescue Project
Rescue Project
Rescue Project
Rescue Project
Rescue Project
Research ChallengesHumanReportingData ProcessingReal-time Recognition
Predictive AnalyticsDecision SupportWhat-If ScenarioSituation RecognitionCommunica-tion TechData IdentificationData Quality
Support fast data flowHandle heterogeneous types of data streamsEfficiently aggregate data at different granularitiesProvide storage system to archive both data input and system output.Create situation model and provide actionable informationContain predictive componentGeneric computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
28
Big 429
MultimediaParticipatoryReportingMultimedia Participatory Sensing.Real-timeSituation RecognitionSituation Recognition: Situation Models from past data Real time recognitionPredictive AnalyticsPredictive analyticsMultimediaSemanticsMultimedia Semantics
Support fast data flowHandle heterogeneous types of data streamsEfficiently aggregate data at different granularitiesProvide storage system to archive both data input and system output.Create situation model and provide actionable informationContain predictive componentGeneric computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
29
Data StreamsMultimedia Semantics: From Different Data Streams to Abstracted Event StreamsReal World Event Streams30
sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, human motions, etcNot only heterogeneous, and unstructured, but time is a very important dimension.
They all generate EventsWhat about frameworks that consume events for knowledge discovery?
In particular, the accommodation of time into mining techniques provides a window into the temporal arrangement of events and, thus, introduce the ability to suggest cause and effect that are overlooked when the temporal component is ignored or treated as a simple numerical attribute. Moreover, temporal data mining has the ability to mine the behavioral aspects of a system as opposed to mining rules that describe their states at a point in time. Hence, temporal data mining helps understanding why rather than merely understanding what.
30
Multimedia Micro-ReportsTwitter is a good concept for opinions.
However, it is too noisy and too subjective for solving real problems.
Designing Multimedia Micro-ReportsObjective SpontaneousCompellingUniversal (Language Independent)Requirements of reportsOpportunityPhotos and VideosTextContextual dataPersonal dataSubjective and ObjectiveUniversal
Recognition ChallengeYesterday: Feature Based Recognition
Current: Machine Learning for Objects
Tomorrow: Situation Recognition
Real Time Situation RecognitionUse appropriate data sources.
Build Situation ModelsNew Frontier for Machine Learning.
Great Opportunity: Real Time Situation Recognition algorithms classify world wide situations.
Predictive Analytics: Spatial InterpolationObservations from sensors are sparse.We need spatial interpolation. Accurate interpolation usually requires combining multiple data sources.
Predictive Analytics: Cyber Space to Cybernetic SystemsPrediction of evolving situation at every point gives us better chances to control situations.Can we extend Wiener filtering and Kalman filtering into prediction methods for streaming data?
Kalman Filetering
4
1
3
1
1
1Our Partners using EventShop.
Our DreamBuild a vibrant Multimedia Rescue communityResearch: Ready Research ProblemsImplementationExperienceDeploy and LearnMultimedia is natural for humans. Lets make multimedia technology also natural in important situations.
Join us, please!