Event Mining in Social Multimedia

Workshop on Event-based Media Integration and ProcessingBarcelona, 21-22 October 2013

Event Mining in Social MultimediaSupervised Learning and Clustering Approaches

Symeon PapadopoulosInformation Technologies Institute (ITI)Centre for Research & Technologies Hellas (CERTH)

ACM Multimedia > EBMIP 2013 Symeon Papadopoulos#2

overview

• motivation• problem definition• approaches

– unsupervised clustering + cluster classification– supervised clustering

• evaluation– implicit + user-based – mediaeval > social event detection

• summary & discussion

ACM Multimedia > EBMIP 2013 Symeon Papadopoulos

motivation


Pope Francis

Pope Benedict

2007: iPhone release

2008: Android release

2010: iPad release

http://petapixel.com/2013/03/14/a-starry-sea-of-cameras-at-the-unveiling-of-pope-francis/


entertainment

personal

news

wedding / birthday / drinks

concert / play / sports

demonstration / riot / speech


event multimedia hold value

• archiving/story-telling (personal use)

• news & media (journalists, editors)

• promotional material (organizers, artists)

• marketing (sponsors, advertisers)


event multimedia lifecycle

announcement

happening

promotional material shared online

PRE DURING POST

attendants capture the event (photos/videos)

attendants share & comment on event content

indexing & replay

annotation (tagging)

search > replay / reuse

COMMODITIZATION OF MEDIA CAPTURING & SHARING > EXPLOSIVE GROWTH OF EVENT MEDIA

EVENT MEDIA INDEXING & REPLAY TECHNOLOGIES BARELY COPE!


event media indexing wish list

• automatic: ideally parameter-free or with intuitive parameters

• fast: casual users are impatient, professional users need quick results

• scalable: possible to apply in very large collections

• serendipitous: discover non-obvious (long tail) event

multimedia#8


problem definition


multimedia event detection

event detection involves the automatic organization of a multimedia collection C into groups of items, each (group) of which corresponds to a distinct event.

COLLECTION

EVENT DETECTION

EVENT SET

E1

E2

EN

#10


event detection variants

do all input images depict events?YES NO

NO

are

we

inte

rest

ed in

all

even

ts?

partitioningfilter media +

clustering

clustering + filter events

filter media +clustering + filter events

discovery mode

detection mode

#11


variant 1

• all input media items depict events• all possible output events are of interest

• scenario: personal/professional collection consisting solely of events > need for automatic organization

• approach: produce a partitioning (non-overlapping clusters that cover the full set of media items) of the input collection into events

#12


variant 2

• input media items may depict anything • all possible output events are of interest

• scenario: media collected from the Web > discovery of interesting event media content

• approach: (a) filter non-event media items > use approach of variant 1, (b) cluster media items (hoping that resulting clusters will be purely event or non-event) and filter non-event clusters

#13


variant 3

• all input media items depict events• not all possible output events are of interest

• scenario: personal/professional collection of event content > retrieval of target events

• approach: cluster media items into events and filter based on desired event attributes (e.g. location, type, etc.)

#14


variant 4

• input media items may depict anything • not all possible output events are of interest

• scenario: media collected from the Web > retrieval of target events

• approach: (a) approach of variant 1a + filter events by desired attributes, (b) approach similar to 1b, but not only filter non-event clusters, but also non-interesting event clusters

#15


prevalent problems

• clustering– group media items into events

• cluster classification– does a particular cluster represent an event? if so, what

type of event does it represent?

• media item classification– does a media item depict an event? what type?


how to tackle them?

we are going to explore two paradigms:

• unsupervised clustering + cluster classification > variant 2 + variant 4

by Quack et al., CIVR2008[extended by Papadopoulos et al., Multimedia 2011]

• supervised clustering > variant 1 + variant 3by Reuter et al., ICMR2012[extended by Petkos et al., ICMR2012/MMM2014]

#17


approachesunsupervised clustering + cluster classification


approach abstraction

#19

media collection

feature extraction* clustering cluster

classification**

cluster naming

post-processingevent index

*also involves similarity or top-K computation**also involves cluster feature extraction


unsupervised clustering + cluster classification

Quack et al., CIVR2008• tile-based photo collection (each tile 200x200m)• build dissimilarity matrices separately per modality

– visual: SURF + feature-feature matching + RANSAC– text: stop-word* removal + modified tf-idf weighting

• hierarchical agglomerative clustering – single-/complete-/average-link (controls granularity)

• cluster classification– two features + ID3 tree for classes “object” & “event”

• cluster naming– frequent itemset mining (top 15) + Wikipedia query (via Google)– Wikipedia link scoring + verification (at least one match between any

of the Wikipedia article images and cluster images)

* extended with Flickr-specific + location-specific stop words

#20


cluster classification#u

sers

/ #

phot

os

duration

[1 day, 2 users / 10 photos]

[2 years, 50 users / 120 photos]

#21

LANDMARK

EVENT


limitations

• applicable only to geotagged images– assumes quite accurate positioning ~100m

• dissimilarity matrix computation is expensive!– hard to scale to sets much larger than 10,000

• homography mapping expensive (due to feature-feature matching)

• cluster classification sensitive to clustering results (if a landmark cluster is split into two smaller ones, it may be incorrectly classified as event)

#22


extension

Papadopoulos et al. Multimedia 2011• city-based image collection (does not require considerable

geotagging accuracy)• construction of hybrid image similarity graph

– visual: SIFT + BoW + top-20 + median similarity filtering– text: two options

• cheap: cooccurrence frequency (exclude frequent tags) + filtering• costly: tag occurrence vectors > LSI > low-dimensional vectors > top-K

• graph clustering: SCAN (Xu et al., KDD2007)• cluster classification

– two features + two tag-based features + SVM/kNN• cluster naming

– frequent tag sequence mining (from titles)

#23


graph clustering :: SCAN

outlier

hub

(μ,ε)- corestructural similarity

• resilient to spurious links (e.g. visual links that connect unrelated images)

• very fast (scales linearly to the number of edges)• leaves less-/ and over-connected items out of the clustering

#24


tag-based cluster features

• manually label clusters as “landmarks” or “events”• aggregate tags of contained images and derive corresponding

tag profiles*

• for a new cluster compute number of contained tags in each of the two profiles > two additional features

LANDMARK

EVENT

* could be city-specific or global

#25


caveats

• graph construction may affect results– k-nn versus ε-nn, parameter selection– modality combination (in our case very simplistic)

• graph clustering– does not take into account weights– sometimes it leaves out of the clusters far too many items

• cluster classification– sensitive to cluster granularity (e.g. fragmented clusters are very

challenging since first two features are misleading)

• cluster naming– unreliable for small clusters, depends a lot on contained items (quality

of metadata, text language)

#26


approachessupervised clustering


supervised event detection

• rationale: use a large number of “known” event assignments to “learn” how to classify new content into events

two main paradigms• item-to-cluster: learn whether a new item belongs to

a given event cluster or not• item-to-item: learn whether two items belong to the

same event cluster or not

#28


approach abstraction

#29

media collection

feature extraction

blocking*similarity

computation

same event classification

clusteringevent index

same event model

* optional: used for improved efficiency** applicable only to item-to-cluster methods

**

**


supervised clustering

Reuter et al., ICMR2012• blocking

– six database queries to retrieve 330 nearest events in terms of: capture time (200), upload time (50), geo-location (20), tag/title/description similarity (20/20/20)

• new image-candidate event pair described by nine features– temporal similarity (upload+capture), proximity (Haversine formula),

tag/title/description similarity using cosine and BM25• same event classification and clustering

– SVM used to rank candidate events (from blocking) based on probability that new image belongs to them + second classifier (SVM) to decide whether new image should start a new event (separate features, incl. first SVM prediction scores + time difference)

#30


limitations

• simplistic treatment of missing metadata– set similarity equal to 0 when metadata (e.g. geo-location)

is missing > could be misleading in case the two items would actually be similar if such information was available

• for some features, representing an event by a proxy (using centroids for aggregation) might not be rich enough, e.g. in cases of geo-location– this is a general characteristic of item-to-cluster methods

• does not make use of visual content– makes approach faster at the expense of missing some

associations that might only surface in the form of visual similarity (e.g. when metadata are of poor quality)

#31


extensionPetkos et al., ICMR2012/MMM2014• blocking

– similar to Reuter et al. 2012 (except that it retrieves most similar images, not events) but also includes visual similarity (VLAD + Product Quantization) [MMM2014]. Up to 350* similar images are retrieved.

• image-image pair described by 11 similarity values:– uploader (0/1), image (GIST and SURF+VLAD), text (same as in Reuter

et al., 2012), quantized time difference, geodesic distance (in km)– two separate classifiers are trained, one when both images have

location information, and one when either of the two does not• clustering

– a same-event graph is constructed based on the predictions of the classifiers

– graph clustering is carried out in two flavours: batch (by use of SCAN) and online by use of QCA (Nguyen et al., 2011) [MMM2014]

* in practice much lower (~100-200) due to overlap between candidates from different similarities

#32


online clustering of same-event graph

QCA maintains community structure incrementally following graph change operations: node & edge addition (removal operations not applicable in same event graph): based on the concept of community attraction forces

A

B

C

D

X new nodenew edge

Cu

Cw

Cz

force from Cu to Cz

force from Cz to Cu

• Depending on a test (computed based on local graph structure), community structure could remain the same, X assigned to Cu or A to Cz.

• If A is assigned to Cu, all its neighbours will be checked for potential reassignment.

#33


caveats

• the method requires maintaining the same-event graph in-memory– starts becoming hard to apply in collections bigger than

some hundreds of thousands of images– in general, item-to-item event detection methods are less

scalable compared to item-to-cluster > potential solution by use of graph databases

• in batch mode, the use of SCAN leads to images being excluded from clusters– variants of the algorithm to make it partitional if necessary

(by assigning hubs & outliers to adjacent clusters)

#34


evaluation


how to evaluate?

• different approach depending on problem variant

• for variants 2 and 4, it is hard to create ground truth (since we are interested in all possible events)– implicit measures of cluster goodness– user-based

• for variants 1 and 3, it is possible to collect or create comprehensive ground truth– mediaeval

#36


case study: landmark & event discovery in Barcelona


dataset

Geo-query to flickr API with centre in Barcelona (2010)

• 207,750 photos by 7,768 users

• tag pre-processing:– filter very short and very long tags

– tags consisting of alphanumeric characters (e.g. camera models)

– tags from a blacklist (e.g. “geotagged”)

• 33,959 tags > 173,825 photos with at least one of them

• remove tags used in more than 350 photos (e.g. “Barcelona”, “Catalunya”) > 120,742 photos with at least one of them


implicit evaluation of clustering quality

• perform the clustering without making use of location information, and then measure how coherent the resulting clusters are > measure of quality (i.e. tight clusters > more likely to not contain irrelevant images)

• we call the measure GCC, Geospatial Cluster Coherence

SCAN graph clustering

k-means data clustering

stdmean


user-based evaluation

• random selection of 33 visual and 40 tag-based clusters (from SCAN) and corresponding k-means clusters (based on member sets overlap)

• each cluster was presented to two independent evaluators and they were asked to mark (in a Web UI) the images that were not perceived as relevant > P, R* (and F) + κ-statistic

• we call this SCQ, Subjective Cluster Quality

* this is a pseudo-recall, computed by pooling “correct” images from all methods together

+ in a second study, we compared visual, tag & hybrid (all from SCAN) > hybrid were found to have an F-score 28.5% higher than visual and 19.8% than tag-based


evaluate event/non-event classification

• manual annotation of 2,056 clusters > 969 landmark, 636 events, 451 unassigned (not used)

blue: Quack et al.red: proposed extension

10 random 50-50 splits(grey: std across 10 splits)

16-23% improvementF-measure ~ 87%


popular event categories

music, concert, gigs, DJ 43.1%

conference, presentation 6.5%

local traditional, parades 4.6%

racing, motorbikes, f1 3.3%

Browse results: http://clusttour.com/index.php?content=place&id=2


social event detection


a bit of background...

• mediaeval– well-known benchmarking activity since 2010 (started as

VideoCLEF in 2008)– consists of several tasks dedicated to specific challenges

• social event detection (SED)– first run in 2011 (7 participants)– this year was the third edition of the task with a bit

different challenge definitions and increased participation!(11 participants)


task definition & dataset

• 2011 collection: 73,645 flickr photos from five cities, May 2009 find events related to two target categories

> soccer matches in Barcelona and Rome > concerts in venues Paradiso and Parc del Forum

• 2012 collection: 167,332 flickr photos from five cities, 2009-2011 find events related to three target categories

> technical events (e.g. exhibitions, fairs) in Germany > soccer events in Hamburg and Madrid > Indignados movement in Madrid

• 2013 collection 1: 437,370 flickr photos + 1,327 YouTube videos collection 2: 57,165 Instagram photos cluster collection 1 into events (attach YouTube videos to them) categorize collection 2 images into eight event types or non-

event

variant 1

variant 4

variant 4


sed2012: evaluation setup

• approach by Petkos et al., MMM2014– method designed for event detection as in variant 4 > used

only 7,779 photos belonging to events in order to assess clustering quality (=Normalized Mutual Information, NMI)

• ground truth: photos clustered around 149 events (18 technical, 79 soccer, 52 Indignados)

• assess the following aspects:– accuracy of same-event classification– compare clustering quality between item-to-cluster and

the two versions of item-to-item (batch & incremental)– measure contributions of different features– study generalization abilities of same event model


sed2012: SE accuracy & clustering quality

• same event classification accuracy 98.58% (SVM)– 10K pos/neg training, 10K pos/neg testing (random)

• clustering quality (NMI): 30/119 training/testing events [10 random splits]– incremental same or better than batch– item-to-item better than item-to-cluster (significant at 0.95 confidence)

• when non-event photos enter the dataset, NMI degrades quickly

BATCH INCREMENTAL ITEM-TO-CLUSTER

AVG 0.924 0.934 0.898

STD 0.019 0.021 0.027

NON-EVENT BATCH INCREMENTAL ITEM-TO-CLUSTER

5% 0.4824 0.5164 0.3954

10% 0.3421 0.3683 0.2899

* In the second table, results were obtained using sed2011 for training and sed2012 for testing.

*


sed2012: contribution of features

• same experiments using limited sets of features

• repeating the same experiments without the use of blocking led to significantly worse results– e.g. 0.030 for visual, 0.7148 for textual

• time is an extremely important feature

FEATUERS BATCH INCREMENTAL

VISUAL 0.8020 ∓ 0.0193 0.8179 ∓ 0.0151

TEXTUAL 0.7925 ∓ 0.0255 0.7792 ∓ 0.0310

VISUAL+TIME 0.9244 ∓ 0.0195 0.9360 ∓ 0.0183

TEXTUAL+TIME 0.9016 ∓ 0.0173 0.9049 ∓ 0.0209


sed2012: generalizing same event model

• train using one event type > test on a different one• in most cases negative impact• in few cases, performance is very high!

BATCH

soccer technical Indignados

soccer - 0.8658 0.8494

technical 0.7967 - 0.8977

Indignados 0.9645 0.8456 -

INCREMENTAL

soccer technical Indignados

soccer - 0.8892 0.8667

technical 0.7661 - 0.7735

Indignados 0.9845 0.8482 -


sed2013 (just a couple of days ago!)

• challenge 1 (full clustering into events)– modified version of method by Petkos et al. MMM2014

post-processing step to assign hubs & outliers (by SCAN) to detected events (different variations used in different runs)

– median performance (compared to other teams)ex. results: NMI = 0.9131, F = 0.7031, divergence = 0.6367

• challenge 2 (classification into event types)– method based on combining VLAD/PCA + tags/pLSA and

Approximate Laplacian Eigenmaps (Mantziou et al., 2013)– median performance (compared to other teams)

ex. Results: F1 = 0.3344, F1 div. = 0.2261, F1 (E/NE) = 0.7163, F1 div. (E/NE) =

0.2157


evaluation: main caveat

• creation strategy of benchmark dataset can dramatically affect how hard (or easy) the problem is– if events are very sparsely distributed over time, then a

simple time-based clustering could be sufficient– if events correspond to users one-to-one, then a simple

user-based look-up could yield very high accuracy– using the same source for training/testing makes it easy

• need to explore new challenging settings– multiple sources of multimedia– huge amounts of non-event content– very dense coverage of feature space by test events


summary & discussion


the many faces of event detection

• event detection in multimedia can be formulated in different ways– we examined four variants– essentially a combination of clustering & classification

• depending on the setting, unsupervised clustering or supervised learning are valid options for tackling the problem

• presented two frameworks (+extensions) for different variants of the problem

• discussed different evaluation strategies & datasets


related research problems

• event crawling– where to look for content that is likely related to events?– what kind of queries to formulate?

• event search & recommendation– assume a very large index of events– what to retrieve?

• event summarization– have found & indexed many photos for an event– how/what to present?


holy grail for event detection

• query with event name• obtain a summary of relevant media from different

sources (twitter, facebook, google+, flickr, ...)• drill down into sub-events• event analytics/statistics

• recreate considerable part of event experience from indexed media content + data


Special Issue

• Social Multimedia and Storytelling: using social media for capturing, mining and recreating experiences, events and places– place- and event-centric social multimedia discovery and collection;– social event detection;– real-world place and event mining and analytics;– place and event summarization through social content;– ...

• editors: – Pablo Cesar, Ayman Shamma, Aisling Kelliher, Ramesh Jain, me

• expected submission date: July 1st 2014• call for papers not yet online (coming soon)


the CERTH-ITI event detection team

Manos Schinas ([email protected])

Giorgos Petkos ([email protected])

Yiannis Kompatsiaris ([email protected])


Thank You!

[email protected]

Acknowledgements

Contact

https://github.com/socialsensor/social-event-detection

http://www.slideshare.net/sympapadopoulos/

@sympapadopoulos


references (i)

• Quack, T., Leibe, B., & Van Gool, L. (2008). World-scale mining of objects and events from community photo collections. In Proceedings of the 2008 international conference on Content-based image and video retrieval (pp. 47-56). ACM.

• Papadopoulos, S., Zigkolis, C., Kompatsiaris, Y., & Vakali, A. (2011). Cluster-based landmark and event detection on tagged photo collections. IEEE Multimedia 18(1), (pp. 52-63)

• Reuter, T., & Cimiano, P. (2012, June). Event-based classification of social media streams. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (p. 22). ACM.

• Petkos, G., Papadopoulos, S., & Kompatsiaris, Y. (2012). Social event detection using multimodal clustering and integrating supervisory signals. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (p. 23). ACM.

#59


references (ii)

• Petkos, G., Papadopoulos, S., Schinas, M., Kompatsiaris, Y. (2014). Graph-based Multimodal Clustering for Social Event Detection in Large Collections of Images. In Proceedings of the 20th international conference on Multimedia Modeling, to appear.

• Xu, X., Yuruk, N., Feng, Z., & Schweiger, T. A. (2007). SCAN: a structural clustering algorithm for networks. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 824-833). ACM.

• Nguyen, N. P., Dinh, T. N., Xuan, Y., & Thai, M. T. (2011). Adaptive algorithms for detecting community structure in dynamic social networks. In 2011 Proceedings of IEEE INFOCOM, (pp. 2282-2290). IEEE.

• Mantziou, E., Papadopoulos, S., & Kompatsiaris, Y. (2013). Large-scale semi-supervised learning by Approximate Laplacian Eigenmaps, VLAD and pyramids. In 14th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), 2013 (pp. 1-4). IEEE.


Justin Bieber

http://www.dailymail.co.uk/tvshowbiz/article-1309620/Justin-Bieber-makes-early-morning-airport-dash-sending-girls-crazy-Maryland-gig.html

exercise: count the cameras…


photo acknowledgements (i)http://www.flickr.com/photos/tomvu/4137577681/

http://www.flickr.com/photos/diamondgeyser/371841339/

http://www.flickr.com/photos/mattbritt00/7125302883/

http://www.flickr.com/photos/earobe6/2333185653/

http://www.flickr.com/photos/phirue/4316064876/

http://www.flickr.com/photos/mypanda/2184195068/


photo acknowledgements (ii)http://www.flickr.com/photos/cairnlee_cres/216396373/

http://www.flickr.com/photos/duncan/4510489508/

http://www.flickr.com/photos/tripu/2521042947/

Event Mining in Social Multimedia

Technology

Transcript of Event Mining in Social Multimedia