It’s not in their tweets: Modeling topical expertise of Twitter users
Crowdsourced Evaluation for Reranked Twitter Search · 2017. 8. 4. · Recent statistics indicate...
Transcript of Crowdsourced Evaluation for Reranked Twitter Search · 2017. 8. 4. · Recent statistics indicate...
University of Washington
Abstract
Crowdsourced Evaluation for Reranked Twitter Search
Garima Tiwari
Chair of the Supervisory Committee:Professor Dr Ankur Teredesai
Computing and Software Systems
Influence and information diffusion in social networks is often explained by the ’pref-
erential attachment’ growth model which possesses power law degree distributions. In re-
cent years, disseminating information in online social networks through short messages has
gained widespread popularity as evidenced by rapid growth of networks such as Twitter1,
and Facebook2. With the exponential growth of these online social networks social influ-
ence can for the first time be measured over a large population. Computing this influence
of actors in such networks has tremendous utility for various applications such as real-time
search, advertising, marketing, recommendations, expert location, and node classification
to name just a few. Inspite of well established power law models that explain the growth
of such networks, the problem of identifying influential actors is significantly challenging
because both structural as well behavioral properties of the nodes (actors) play a crucial
role in determining influence. Several algorithms have been recently proposed that utilize
the global network structure, as well as exhaustive behavioral attributes of actors. Firstly,
access to global network structures and detailed behavioral attributes is non-trivial to ob-
tain due to limits imposed by the popular platforms. Moreover, there is a lack of robust
evaluation strategy to determine which of the algorithms is better than others in computing
1http://www.twitter.com
2http://www.facebook.com
such influence.
In this paper we first propose a series of novel, efficiently computable influence measures that
only utilize local network properties thereby alleviating the need to have access to the global
network structures. Next we propose measures that combine the behavioral attributes with
local structural properties. We then describe the first ever publicly available ground-truth
data-collection tool we designed to obtain a crowdsourced relative influence/rank data for a
set of twitterers on a variety of topics. We also describe how we compute the baseline inter-
enduser discord between the influence ranks we obtained. Needless to state, we make this
dataset publicly available for future research efforts. Lastly, we describe an exhaustive se-
ries of experiments we conducted to compare and contrast the influence ranks generated by
allthe existing influence computation algorithms. We conclusively demonstrate that there
is no one single algorithm that currently correctly outperforms other algorithms and accu-
rately reflects the enduser influence ranks; but several algorithms do come close to ideal.
Surprisingly, our results indicate that local measures of influence augmented by behavioral
attributes of twitterers are nearly as accurate as any existing exhaustive global measures.
Since influence in microblogs is likely to vary over time, we also provide empirical data
on the variation of influence and discuss the temporal robustness for our proposed local
influence measures.
TABLE OF CONTENTS
Page
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
Chapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Chapter 2: Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Chapter 3: Influence measurement Strategies . . . . . . . . . . . . . . . . . . . . . 9
Chapter 4: Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Chapter 5: Facebook Application and Evaluation . . . . . . . . . . . . . . . . . . 19
Chapter 6: Experimental Results and Analysis . . . . . . . . . . . . . . . . . . . . 24
Chapter 7: Time Varying Influence . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Chapter 8: Coclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . 38
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
i
LIST OF FIGURES
Figure Number Page
5.1 Screen Shot of Facebook App Home Page . . . . . . . . . . . . . . . . . . . . 20
5.2 Screen Shot of Facebook App Rerank Tweets Page . . . . . . . . . . . . . . . 21
5.3 Screen Shot of Facebook App Result Page . . . . . . . . . . . . . . . . . . . 23
7.1 how the Tweet rank score of authors of 5 tweet about query ’kinect’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.2 how the Tweet rank score of authors of 5 tweet about query ’boxee box’changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.3 how the Tweet rank score of authors of 5 tweet about query ’google tv’changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.4 how the Tweet rank score of authors of 5 tweet about query ’nissan leaf’changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.5 how the Tweet rank score of authors of 5 tweet about query ’ipad’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
7.6 how the Flur rank score of authors of 5 tweet about query ’windows Phone7 launch’ changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . 34
7.7 how the Flur rank score of authors of 5 tweet about query ’SXSW’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
7.8 how the Flur rank score of authors of 5 tweet about query ’halloween’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
7.9 how the Tweet rank score of authors of 5 tweet about query ’mid term elec-tion’ changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . 35
7.10 how the Flur rank score of authors of 5 tweet about query ’thanksgiving’changes over a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . 35
7.11 how the Flur rank score of authors of 5 tweet about query ’Microsoft’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7.12 how the Tweet rank score of authors of 5 tweet about query ’asus’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
ii
7.13 how the Tweet rank score of authors of 5 tweet about query ’zynga’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.14 how the Flur rank score of authors of 5 tweet about query ’Verizon’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.15 how the Flur rank score of authors of 5 tweet about query ’Costco’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.16 how the Flur rank score of authors of 5 tweet about query ’revenue’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.17 how the tweet rank score of authors of 5 tweet about query ’iran’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
7.18 how the tweet rank score of authors of 5 tweet about query ’facebook’ changesover a span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
7.19 how the Flur rank score of authors of 5 tweet about query ’flu’ changes overa span of 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
iii
ACKNOWLEDGMENTS
I wish to express my sincere appreciation to University of Washington, where I have had
the opportunity to do my graduate studies and successfully complete this thesis work. I
would like to thank Dr Ankur Teredesai who motivated me to take up this research, and
whose support made it all possible. I would also like to thank Dr Jie Sheng who helped me
through my thesis work.
iv
DEDICATION
to my dear husband, Prashant and my loving daughter, Shravya
v
1
Chapter 1
INTRODUCTION
Sharing short messages through online social networks is an important component of the
Real Time Web that is becoming increasingly popular. Online social network applications
that provide this service to their users, such as Twitter1, and Facebook2 are used by people
all over the world on a daily basis. Recent statistics indicate that Twitter currently has
around 200 million users with more than 1 Billion tweets (messages of up to 140 characters)
being generated per week. Though there is little consensus among social scientists over the
reason why people use these services, it is generally accepted that the ability to express
opinions quickly and freely, and the ability to effectively reach a large audience is the main
draw. From an information consumption perspective, obtaining current trends and the
latest news in real time from a multitude of sources with diverse viewpoints is the main
attraction for the readers of such microblogs [9]. Twitter search, for instance, only provides
keyword matching based search results (i.e. it checks if the tweet contains the search query
or not), and presents them ranked in reverse chronological order. There is no guarantee that
the most interesting tweet appears on top, especially given that thousands of new tweets are
being written every hour. Filtering the Real Time Web to find the most interesting tweet
within a short timeframe is an important challenge. One way to address this challenge is
to first find influential twitterers and then re-rank the tweets based on the influence score.
Computing influence in online social network is of great utility for other domains as well.
For example, influence scores can be used in applications like real time search, advertising,
marketing,recommendation systems etc. If suppose, author a1 has higher influence score
over author a2, then in the real time search result we can show the articles by a1 above
1http://www.twitter.com
2http://www.facebook.com
2
the articles by a2, since author 1 is likely more authoritative, and so information shared by
her is likely more useful. Similarly, if any specific author a1 has higher influence score for
a given topic, say ’iPad’, it implies, he is interested in that topic and so it is beneficial to
show him the advertisement related to apple products rather than any other less relevant
advertisement. Moreover, this model for behavioral targeting can be extended to her social
network who are more likely to listen to ’iPad’ related ads due to their proximity to an
’iPad’ influencer.
Social influence can be computed using many factors including the strength of ties be-
tween actors in the network, the geodesic distance between users, temporal effects, spatial
proximity, and network specific characteristics of the individual actors within the network.
As one can already observe, designing an algorithm for social influence in a network of twit-
terers can involve studying and integrating many such factors into influence measures. Some
of these factors are purely structural while others are behavioral at a node level. Moreover,
structural factors can be global; requiring the entire connected component of the graph, or
local; requiring structural information only few (one or two) hops away from the node under
consideration). Similarly, behavioral factors can be node specific (number of posts, topics
of interest, etc) or collaborative between actors (retweets, mentions, etc).
While the utility of social networks such as Facebook and Twitter depends significantly
on how effective they can be at disseminating information in a trustworthy manner, not
all nodes in such social networks exhibit equal amount of influence. In fact it is now
widely believed that such networks exhibit power law degree distributions [2]. These degree
distributions of many real world social networks which obey a power law are of the form
f(d) ∝ dα (1.1)
with the exponent α > 0, and f(d) being the fraction of nodes with degree d. Such power-law
relations as well as many more have been reported in [1], [22], [7], [10], [13].
Intuitively, power-law-like distributions for degrees state that there exist many low degree
nodes, whereas only a few high degree nodes in real graphs. How do some nodes that start
of as low-degree nodes within a social network gain different properties and exhibit different
3
affinities with other actors and eventually become high degree nodes? Moreover, why are
structural properties not sufficient in identifying influential nodes within modern day online
social networks? Why do many low degree nodes prefer not to listen to actors with high
degrees but rather listen to very some high-degree and some low-degree actors making
influence a combination of structural and behavioral attributes ? Who and why are some
actors more influential than others? Identifying authority figures in online social networks
has drawn considerable attention in recent years.
Recently, several algorithms have been recently proposed that utilize the global network
structure, as well as exhaustive behavioral attributes of actors. Firstly, access to global
network structures and detailed behavioral attributes is non-trivial to obtain due to limits
imposed by the popular platforms. Moreover, there is a lack of robust evaluation strategy
to determine which of the algorithms is better than others in computing such influence.
Weng et al. [19] proposed an extension of page rank algorithm to measure the influence of
twitterers.Pal et al. [15] extracted some detailed metrics and computed them for each po-
tential authoritative twitterers. In another interesting paper Yamaguchi at al. [20] analyzed
User-Tweet graph to compute users’ comparative rankings. Fernandez et al. [8] used more
than 35 variables on twitter and Facebook to compute overall online influence of a user. The
measures used are complex and often require storing and traversing the entire twitter graph
iteratively. In this paper we first propose a series of novel, efficiently computable influence
measures that only utilize local (1 or 2 hop) network properties thereby alleviating the need
to have access to the global network structure and next we propose measures that combine
the behavioral attributes with these local structural properties.
These approaches use their own empirical evaluation strategies and compare results with
degree centrality type measures. Weng et al. [19] did the comparison with related algo-
rithms like indegree, page rank and topic specific page rank. Pal et al. [15] analyzed topic
specific influential twitterers and claim that their algorithm ranks the director of a toy
story 3 higher while rejecting celebrities who tweeted about the movie which proves the
correctness of their algorithm. Hence, we believe there is a substantial need to develop a
4
comprehensive evaluation strategy that takes human judgement into account when deter-
mining effectiveness of influence algorithms. Our proposed tool is one of the first attempts
to solve the evaluation problem. This publicly available tool (hosted on Facebook) collects
human judgements as ground-truth data for influence rank by allowing end-users to treat
influence ranking as a reordering game. Once we get the crowdsourced judgement on which
twitterer is relatively more important than others for a given topic/keyword/category we
compute the baseline inter-enduser discord between the relative influence ranks. We then
generate an average ranking for the twitterers and compare and contrast the accuracy of
various algorithms including the ones we designed. We plan to make this human judgement
rank dataset publicly available for future research efforts.
Since influence of authors in microblogs is likely to vary over time since the behavioral
attributes of the authors and the global structure of the social network varies with time. As
far as we know our work in this domain is the first effort to explore this. We explored the
domain of varying influence of twitterers over a span of 6 months. We provide empirical
data on the variation of influence and discuss the temporal robustness for our proposed
local influence measures.
This paper is organized as follows: We first describe our new efficiently computable
influence measures in chapter 3 that only utilize local network properties thereby alleviating
the need to have access to the global network structures. Next we propose measures that
combine the behavioral attributes with local structural properties. We then describe the
influence rank data-collection facebook app we designed in chapter 5. We describe how
to compute the baseline inter-enduser discord between the influence ranks obtained from
the app. In chapter ?? we describe a series of experiments we conducted to compare
and contrast the influence ranks generated by the various existing influence computation
algorithms. We then discuss our results which indicate that local measures of influence
augmented by behavioral attributes of twitterers are nearly as accurate as any existing
exhaustive global measures. Since influence in microblogs is likely to vary over time, we
also provide empirical data on the variation of influence values with time. Chapter ??
5
concludes the discussion and provides some avenues for future research.
6
Chapter 2
RELATED WORK
It is now well established that analyzing network properties is tremendously useful for
various applications such as identifying authorities, search algorithms [5], [6], [11], for dis-
covering the network value of customers for viral marketing [17], and to improve recommen-
dation systems [4],[18].
Finding influential nodes within a social network is a well studied problem and is often
formulated closely with the related problems of expert identification [?], identification of
hubs and authorities [3], and various variations of the Pagerank [14] algorithm. The actual
problem of finding influentials is significantly different since authority or influence on mi-
croblog networks such as Twitter is determined by a variety of parameters which include
a mix of structural properties of the underlying social network as well as the behavioral
attributes of the nodes themselves. Moreover, our efforts are specifically oriented to find
authority within the Twitter network such that the ranking of their tweets can be done most
effectively, though the measures we design can be generalized very easily to fit other asso-
ciated problems as well. Nagmoti et. al. proposed a set of preliminary measures to obtain
an effective re-ranking of twitter search results [12]. Their influence measures, though local,
suffered from significant evaluation drawbacks which we address in this effort. Weng et. al.
reformulated the problem of finding influence in microblogs to the problem of identifying
topic sensitive influential twitterers [19]. They try to exploit the phenomenon of homophily
in twitter. Homophily implies that a twitterer follows a friend because she is interested
in some topics the friend is publishing and the friend follows back because she finds they
share similar topical interest. Their Twitterrank algorithm is basically an extension of the
iterative Pagerank algorithm and measures the influence of users in twitter given a specific
topic. It measures the influence by taking into account both topical similarity between
7
users and and the link structure but is constrained by both the availability of the global
network connectivity data and dependent on the accuracy of the topic-model generated by
the LDA model [?]. Topic modeling within fast moving text streams such as microblogs can
itself be a significantly challenging task [21] and is computationally expensive. Moreover
the influence computation in Weng et.al.’s approach significantly depends on the quality of
the topic model which can be difficult to ground [16].
Other ranking algorithms that utilizes only the underlying structural properties of the
twitter graph are TURank and ObjectRank [20]. Unlike PageRank, ObjectRank takes
account of edge types and node types in order to deal with multiple kinds of edges and
nodes. Pal et. al describe an approach where a list of metrics are extracted and computed
for each potential authority [15] . They divide each tweet by a twitterers into 3 categories
original tweet(OT), conversational tweet(CT) and repeated tweet (RT). OTs are tweets that
are not RT or CT. A CT is directed at another user, denoted by the use of @username token
preceding the text. RT are produced by someone else but the user copies, or forwards, them
in-order to spread it in her network. These metrics are: RT@username, Number of original
tweets, Number of links shared, Self-similarity score that computes how similar is authors
recent tweet w.r.t. to her previous tweets, Number of keyword hashtags used , Number
of conversational tweets, Number of conversational tweets where conversation is initiated
by the author, etc. Next they extract a few features across the tweets of a user on the
topic of interest such as; (a) topical Signal - estimates how much an author is involved with
the topic irrespective of the types of tweets posted by her, (b) signal strength - measures
originality of author’s tweets and it indicates how strong is the twitterers topical signal, (c)
non chat signals - to discount the fact that the author did not start the conversation but
simply replied back out of courtesy, (d) retweet impact - indicates the impact of the content
generated by author or how many times it has been retweeted by others, (e) mention impact
- used to find how much an author is mentioned with regard to a topic of interest, and, (f)
retweet impact - indicates the impact of the content generated by the author. Many such
measures have been employed recently to identify twitterer authority and influence. One
8
such popular authority as a service API is provided by KLOUT.com 1. The scores range
from 1 to 100 with higher scores representing a wider and stronger sphere of influence. Klout
uses over 35 variables on Facebook and Twitter to measure True Reach, Amplification
Probability, and Network Score. In this paper we evaluate and demonstrate how even
KLOUT scores are inadequate indicators of influence. In another interesting paper authors
have given a brief description of the 5 algorithms which can be used to rank twitter users.
These are pagerank [14], HITS, Noderanking ,tunkrank, twitterrank [19].
The lack of conclusive comparison between approaches is one main drawback we address
in our paper. Moreover, we demonstrate that local measures of influence computed based
on one or two hop networks for any given twitterer are as good at determining authority as
any of the global network measures previously suggested in literature.
1www.klout.com
9
Chapter 3
INFLUENCE MEASUREMENT STRATEGIES
Nagmoti et al [?] defines three measures for measuring influence of authors of microblogs
and then propose how to use these measures for ranking microblogs, in combination with
some properties of the microblogs themselves, such as the length of a microblog and whether
the microblog contains a URL.
3.0.1 Ranking Authors of Microblogs
Let A denote the set of all authors and Q the set of all queries. By a ranking measure for
authors, we mean a mapping g : A×Q → R+ that associates with every author-query pair
(a, q) a nonnegative real number g(a, q), called the rank of author a w.r.t. query q. This
implies that the rank of an author can vary with the query topic, in other words that an
author can be considered as more authoratative on some topics than on others.
The first strategy proposed ranks authors based on the number of tweets they have
posted so far in the microblogging system. The underlying idea is that
active publishers might be more valuable as information sources than inactive publishers.
[TweetRank] Let a be an author and q a query, then the TweetRank of a w.r.t.q is defined
as
TR(a, q) = N(a) (3.1)
with N(a) the total number of tweets posted by a so far.
As the right hand side in Formula (3.1) clearly reveals, TweetRank is a query independent
measure, i.e., the TweetRank of an author is the same over all query topics. The second
ranking strategy that proposed is query independent as well. It is purely based on the posi-
10
tion of an author in the social network of the microblogging service. This social network is
a directed graph in which an edge from user u to user v means that u is following v. In this
case, u is called a follower of v, and u will find all posts of v automatically displayed on his
account page. Furthermore, v is called a followee of u. Intuitively, an author is influential if
he has a lot of followers. Indeed, if an author is spreading very useful information, naturally
many people follow him. FollowerRank captures this idea.
[FollowerRank] Let a be an author and q a query, then the FollowerRank of a w.r.t. q is
defined as
FR(a, q) =i(a)
i(a) + o(a)(3.2)
with i(a) being the indegree of a, i.e. the number of followers of a, and o(a) being the
outdegree of a, i.e. the number of users followed by a.
Here, the number of followers of an author is divided by his total in and out degree. This
acts as a damping factor to the FollowerRank of authors who have a unusually high number
of followers just because they are socially overactive, rather than because of the quality of
their tweets. FollowerRank varies between 0 and 1.
3.0.2 Ranking Microblogs
Let T denote the set of all tweets and Q, as before, the set of all queries. Ranking measure
for tweets mean a mapping f : T ×Q → R+ that associates with every tweet-query pair (t, q)
a nonnegative real number f(t, q), called the rank of tweet t w.r.t. query q. Furthermore,
let auth denote the T → A mapping that maps every tweet t to its author auth(t). The
ranking measures for authors proposed above can be used to rank tweets as query search
results using any of the functions defined below.
[Ranking measures for tweets] The ranking measures fTR, fFR, and fRR for tweets are
defined as
11
fTR(t, q) = TR(auth(t), q)
fFR(t, q) = FR(auth(t), q)
fRR(t, q) = RR(auth(t), q)
(3.3)
for all t ∈ T and q ∈ Q.
In addition to the social network based ranking strategies proposed above, two more
factors are considered which may indicate the amount of information shared through tweets,
namely the presence of a URL (http link) in a tweet, and the length of a tweet. The amount
of information contained within a tweet can at times be proportional to the length of the
tweet. This leads to a query dependent ranking strategy for tweets termed LengthRank, a
measure which varies between 0 and 1.
[LengthRank] Let t be a tweet and q a query, then the LengthRank of t w.r.t. q is defined
as
fLR(t, q) =l(t)
maxn∈T (q,k)
l(n)(3.4)
with T (q, k) the set of top-k tweets on query topic q, and l(t) and l(n) the length of tweet
t and n respectively.
The author’s intention behind sharing a URL is mostly to direct his audience towards
some potentially interesting information available somewhere else on the web, so the pres-
ence of a URL can be an indication of informativeness. This leads to a query independent
measure called URLRank.
[URLRank] Let t be a tweet and q a query, then the URLRank of t w.r.t. q is defined as
fUR(t, q) =
c if t contains a URL
0 otherwise(3.5)
with c a positive constant.
12
As a stand-alone ranking strategy, this measure will not be very effective because many
tweets will receive rank 0 (because they do not contain a URL) and the rest will receive
the same rank c, which does not allow for a relative ordering of tweets. Like LengthRank,
URLRank can however be used in a meaningful way in combination with any of the social
network based ranking strategies, leading among others to the ranking measures proposed
in Definition 3.0.2.
[Ranking measures for tweets]
The ranking measures fULR,fFLR and fFLUR for tweets are defined as
fULR(t, q) = fUR(t, q) + fLR(t, q)
fFLR(t, q) = (fFR(t, q) + fLR(t, q))/2
fFLUR(t, q) = fFLR(t, q) + fUR(t, q)
(3.6)
for all t ∈ T and q ∈ Q.
3.0.3 Other Ranking strategies
Apart from the influence based ranking strategies proposed by Nagmoti et al [?], we de-
veloped a couple new and innovative strategies to test against the crowdsourced evaluation
data collected by our FB app described in chapter 5.
The first strategy proposed here, called Combined Influence Score, uses the composition
of following metrics:
[OnSubject metric] Let n be the number of tweets from author a on query q and m be
the number of tweets from author a, then the OnSubject metric of i is defined as
OSM(a) =n
m(3.7)
[Chatty metric] Let n be the number of original tweets of author i, then the chatty
metric of i is defined as
13
CYM(a) =n
maxj(chatty(j))(3.8)
here j ranges over all authors in the social network graph to normalize the result.
[Concise metric] the Concise metric of a is defined as
CM(a, q) =maxtweetlength− avg(a)
maxtweetlength(3.9)
here avg(a) is the average tweet length of all tweets from author a on query q.
[Link Structure metric] Let n be the number of followers of a and m be the number of
friends of author a, then the Link Structure metric of a is defined as
LSM(a, q) =n
n+m(3.10)
[Combined Influence Rank] Let OSM be OnSubject Metric for a, CYM be Chatty
Metric for a, CM be Concise Metric for a and LSM be Link Structure Metric for a, then
the Combined Influence Score of a is defined as
CIS(a, q) = OSM + CYM + CM + LSM (3.11)
Another new strategy proposed in this thesis, called Content Based Influence Score,
which gives a lot of emphasis on the content posted by the authors in order to determine
their influence. In this strategy the tweets of authors are split up into words and then
searched through for a count of how many times each word appears.The words are divided
into 4 buckets : weak, fail, actions, and good
To compute the score of words, the Individual word scoring system is as follows :
• The author is given points for each word used.
14
• A user loses some points for each set of words between a pair of commas that is fewer
than 4 words.
• A user loses some points for each word that ends with ... and that does not start with
htt.
• All of the points for tweets are divided by the number of words to normalize them.
Each word is given a multiplier, where it is -1 for bads, -2.1 for fails, 1.2 for goods, and
1.5 for actions. For each word, if the multiplier is negative, and the frequency is less
than or equal to 0.015 out of all words, 20000*frequency*(0.015 - frequency) . If the
multiplier is negative and the frequency is greater than 0.015, the formula is instead
20000*0.015*(frequency - 0.015) . If the multiplier is positive and the frequency is
between 0 and 0.03, the formula is instead 20000*frequency*(0.03 - frequency). The
word score is then increased by 1 to make it more likely to be positive. If it is still
negative, it is replaced by 0.0000000000000001.
[FinalWordScore] Let FWS be final word score for a word w , then theFinalWordScore
of a word w defined as
FWS(w) = (wordScore)1/5 (3.12)
where wordScore is computed by above mentioned Individual word scoring system.
[Content based Influence Score] Let CBIS be Content based Influence Score for an
author , then it is defined as
CBIS(a, q) = (∑i
(∑w∈W
(FWS(w))) (3.13)
where W is the set of words in a tweet, i ranges from 0 to n n is the total number of tweets
of author a.
Also we studied recently popular authority computation service provided by KLOUT.com 1.
In our thesis work we evaluate how klout’s influence score based ordering of authors is sim-
ilar to our crowdsourced evaluation data. Klout score is the measurement of overall online
1www.klout.com
15
influence of a user. The scores range from 1 to 100 with higher scores representing a wider
and stronger sphere of influence. Klout uses over 35 variables on Facebook and Twitter
to measure True Reach(It is the size of user’s engaged audience.), Amplification Probability
(It is the likelihood that user’s content will be acted upon. How often do user’s messages
generate retweets or spark a conversation?) and Network Score( It is the influence level
of user’s engaged audience. Engagement is measured based on actions such as retweets,
@messages, follows, lists, comments, and likes.).
16
Chapter 4
DATASET
Not all nodes in a social networks exhibit equal amount of influence. Several algorithms
have been recently proposed that utilize the global network structure, as well as exhaustive
behavioural attributes of authors. There is a lack of robust evaluation strategy to determine
which of the algorithms is better than others in computing such influence. Therefore we
decided to build a publicly available ground-truth data-collection tool to obtain a crowd-
sourced relative influence/rank data for a set of twitterers on a variety of topics.
We maintain a large dataset with tweets retrieved for the 20 query words under consid-
eration, 4.2, from twitter api along with the publicly available social network information
about their authors. This information is used to compute the rankings for tweets and their
authors based on our influence measuring ranking strategies as described in Chapter ??.
Out of all the tweets available in the dataset for a query word, we select a set of random 5
tweets. In order to collect the crowdsourced evaluation data, Each set of 5 tweets is shown
to 3 unique end users of our facebook app (described in chapter 5) corresponding to the
query selected. Thus we maintain a set of 5 tweets for each query word along with their
authors and other structural and behavioural attributes. This dataset of 96 authors (Since
4 of the authors were repeated in this dataset) their 96 tweets and their graph information
that is publicly available through twitter API is used for the evaluation purpose of influence
measuring ranking strategies. Also we maintain the ordering of tweets submitted by the 3
end users for each query.Using this we compute the average ordering of tweets by the end
user. To explain this lets take an example. Suppose there are 3 end users A, B, C who
use facebook app and provide their ordering for a set of tweets T1,T2,T3,T4,T5 here T1
represent tweet1 and T2 represent tweet2 and so on. The table ??hows the ordering of
tweets by 3 end users A, B, C.
17
position A’s ordering B’s ordering C’s ordering
1 T3 T3 T3
2 T2 T2 T2
3 T1 T1 T4
4 T5 T4 T1
5 T4 T5 T5
Table 4.1: Ordering of 5 tweets by 3 authors
Influence of authors in microblogs is likely to vary over time since the behavioural at-
tributes of the authors and the global structure of the social network varies with time. In
order to be able to observe the change in the authors’ scores based on various influence
measuring strategies, we also maintain the timestamped information of these 96 authors
under consideration. Thus we maintain twitter graph information from December 2010 to
May 2011 at irregular intervals for the 96 authors under consideration.
In December 2010 when we collected the data of 96 authors,all of them were active and
their data was publicly available but during the span of 6 months of data collection we
observed that some of these 96 authors got suspended from twitter api and some of them
got their account protected and so their social network details were no longer available to
us. Unfortunately for one of the query words, ie. toy story 3, 4 out of the 5 authors under
consideration got suspended/protected. Since in this scenario, for the above mentioned
query, the ordering of authors/tweets was not possible to be maintained at various times
due to unavailability of authors’ social network information to generate influence score/rank
based on our influence measuring ranking strategies therefore we dropped this query for
evaluation purposes.
18
id query word
1 boxee box
2 i pad
3 kinect
4 nissan leaf
5 google tv
6 halloween
7 Thanskgiving
8 SXSW
9 windows phone 7 launch
10 mid term elections
11 verizon
12 costco
13 microsoft
14 asus
15 zynga
16 flu
17 facebook
18 revenue
19 toy story 3
20 iran
Table 4.2: Query Words
19
Chapter 5
FACEBOOK APPLICATION AND EVALUATION
Currently, there is no standard ground truth dataset available to evaluate ranking strate-
gies for Twitter. All the reranking strategies mentioned in chapter 3 seem promising but in
the absence of any ground truth dataset to compare the reranked tweets it was difficult to
concretely say anything about the correctness of these algorithms. Previously Nagmoti et al
[?] had evaluated the performance of these strategies using preference judgement. Later it
was realised that preference judgement is not considered widely accepted way of evaluation.
Therefore we decided to measure the accuracy of the proposed re-ranking strategies using
human assessors for evaluation and collect ground-truth data for aiding future research in
this domain. We therefore developed a Facebook application called ’Twitter Ranking’1
to collect crowd sourced evaluation data. For the facebook app (shown in 5.1) , first we
selected 4 categories of query words as in table 4.2. Each of these categories consist of 5
trending query words. When an end-user selects a query words a set of 5 tweets related to
that query word are shown to the end-user in an arbitrary order 5.2 . This set of 5 tweets
is shown to 3 unique end users of facebook app. Then the user is supposed to drag and
drop these 5 tweets in an order that seems best to him. In other words he/she can place the
most informative tweet on top and least informative tweet at the bottom. After the user
submits his/her ordering the application shows him how much he/she was similar to our
ranking strategy and who else on facebook ranked the tweets in a similar ordering 5.3. This
is done to make the Facebook application more interesting and engaging for our end-users.
This also makes it a contest to see who matches up to whom and leads to people sharing
the application with their friends.
1http// : www.apps.facebook.com/twitter ranking
20
Figure 5.1: Screen Shot of Facebook App Home Page
The authors information and the tweets’ content are used to compute the rankings for
tweets based on our ranking strategies. We make sure that each set of 5 tweets are shown
to three unique end users to get their ordering of tweets. The discord value between the
rankings of any 5 tweets by users and by algorithm is computed using Kendalls tau distance.
Kendall tau distance is a metric that counts the number of pairwise disagreements be-
tween two lists or orderings. The larger the distance the more dissimilar the two lists are.
K tau distance can be defined as total number of discordant pairs.
K-tau distance between 2 lists τ1 and τ2 is given by:
21
Figure 5.2: Screen Shot of Facebook App Rerank Tweets Page
K(τ1, τ2) = |(i, j) : i < j, (τ1(i) < τ1(j) ∧ τ2(i) > τ2(j)) ∨
(τ1(i) > τ1(j) ∧ τ2(i) < τ2(j))|
K(τ1, τ2) =
0 if the two lists are identical
n(n− 1)/2 if the two lists are reverse(5.1)
we have normalized the K-tau by dividing by (n(n-1)/2)*10 (here n = 5) so that 0 indi-
cates identical and 10 indicates maximum disagreement between the two orderings.
22
For each query word a set of 5 tweets is presented to 3 unique facebook app users and
their ordering is collected and saved using our app. Then using this saved data, the average
ordering by the 3 users who ranked a set of same 5 tweets is estimated for each query word.
Next the kendalls tau distance metrics is used to find the discordance between this average
ordering of 5 tweets by the 3 end users of FB app and the ordering of same 5 tweets by our
various reranking strategies. We also compute the inter user discord values to find out the
discordance between the 3 end users who order the same set of 5 tweets for a query word.
We have discussed this inter-user discord and the discord between humans and reranking
strategies in chapter 6.
23
Figure 5.3: Screen Shot of Facebook App Result Page
24
Chapter 6
EXPERIMENTAL RESULTS AND ANALYSIS
For the 4 categories of queries, we computed the discordance between the average or-
dering of 5 tweets by the 3 end users of FB app and the ordering of same 5 tweets by our
various reranking strategies using Kendalls tau distance metric explained in 5. Result are
shown in tables 6.1 to 6.4. We also computed the inter human discord between the 3 FB
app end users who ordered the 5 tweets for each query word to estimate how much the
humans were in agreement when they ordered the tweets using Facebook app. (shown in
the 1st columns of tables 6.1 to 6.4).
The last row in each of the tables 6.1 to 6.4 has average values of the the inter human
discord and discord with various influence measuring strategies explained in 3. Discussing
about the
Analysing the table 6.1, we see for some query words (ex. boxee box) LengthRank,
which is an author independent ranking measure, gives minimum discord with the human
assessors whereas for other query words (ex. google tv) Length rank gives very high discord
value. Also for some query words (ex. ipad ) FLUR rank seems to be closest to human
assessors while for other query words (ex. kinect) Tweet Rank Seems to be doing better
25
query human discord discord discord discord discord discord discord discord
word discord with with with with with with with with
FR LR TR FLUR ULR KR SNA1 SNA2
kinect 1.33 5 3 3 4 3 2 3 5
boxee box 2.66 8 2 7 4 4 6 6 8
google tv 2.66 9 8 9 10 8 9 9 7
nissan leaf 3.33 3 2 6 3 2 6 8 6
i pad 4.66 1 2 3 0 4 1 2 1
Average 2.928 5.2 3.4 5.6 4.2 4.2 4.8 5.6 5.4
Table 6.1: Evaluation of reranking strategies against human assessors on facebook for Prod-uct Category (The inter-user discord of the 3 users who ordered the same set of 5 tweetsfor a query word in column 2).The discord value between average of 3 FB users ordering and the ordering by variousreranking strategies lies between 0 and 10. Here FR: FollowerRank, LR: LengthRank,TR: TweetRank, FLUR:FollowerLengthURLRank, ULR: URLLengthRank, KR: KloutScor-eRank, SNA1: SocialNetworkRank1, SNA2:SocialNetworkRank2
26
query human discord discord discord discord discord discord discord discord
word discord with with with with with with with with
FR LR TR FLUR ULR KR SNA1 SNA2
windows 2.66 4 4 8 4 4 7 4 5
Phone 7
launch
SXSW 3.33 8 1 3 2 2 4 2 7
halloween 3.33 6 3 5 2 2 4 4 8
mid term 6 5 5 4 4 3 4 4 5
election
thanksgiving 6.66 5 4 5 4 3 3 7 6
Average 4.396 5.6 3.4 5.0 3.2 2.8 4.4 4.2 6.2
Table 6.2: Evaluation of reranking strategies against human assessors on facebook for EventCategory (The inter-user discord of the 3 users who ordered the same set of 5 tweets for aquery word in column 2).The discord value between average of 3 FB users ordering and the ordering by variousreranking strategies lies between 0 and 10. Here FR: FollowerRank, LR: LengthRank,TR: TweetRank, FLUR:FollowerLengthURLRank, ULR: URLLengthRank, KR: KloutScor-eRank, SNA1: SocialNetworkRank1, SNA2:SocialNetworkRank2
27
query human discord discord discord discord discord discord discord discord
word discord with with with with with with with with
FR LR TR FLUR ULR KR SNA1 SNA2
microsoft 3.33 7 9 7 5 4 5 9 6
asus 4 4 3 1 3 3 2 2 3
zynga 4 7 3 5 6 3 4 7 8
verizon 4 5 5 5 4 4 5 6 5
costco 6 6 7 6 6 7 7 9 8
Average 4.266 5.8 5.4 4.8 4.8 4.2 4.6 6.6 6.0
Table 6.3: Evaluation of reranking strategies against human assessors on facebook for Com-pany/Organization Category (The inter-user discord of the 3 users who ordered the sameset of 5 tweets for a query word in column 2).The discord value between average of 3 FB users ordering and the ordering by variousreranking strategies lies between 0 and 10. Here FR: FollowerRank, LR: LengthRank,TR: TweetRank, FLUR:FollowerLengthURLRank, ULR: URLLengthRank, KR: KloutScor-eRank, SNA1: SocialNetworkRank1, SNA2:SocialNetworkRank2
28
query human discord discord discord discord discord discord discord discord
word discord with with with with with with with with
FR LR TR FLUR ULR KR SNA1 SNA2
revenue 0.66 5 7 3 3 4 2 6 4
iran 3.33 5 6 4 5 4 4 3 5
facebook 4.66 3 2 4 3 3 5 3 5
flu 4.66 6 4 5 1 1 0 7 5
Average 3.3275 4.75 4.75 4 3 3 2.75 4.75 4.75
Table 6.4: Evaluation of reranking strategies against human assessors on facebook for Mis-cellaneous Category (The inter-user discord of the 3 users who ordered the same set of 5tweets for a query word in column 2).The discord value between average of 3 FB users ordering and the ordering by variousreranking strategies lies between 0 and 10. Here FR: FollowerRank, LR: LengthRank,TR: TweetRank, FLUR:FollowerLengthURLRank, ULR: URLLengthRank, KR: KloutScor-eRank, SNA1: SocialNetworkRank1, SNA2:SocialNetworkRank2
query average human discord
revenue 0.66
kinect 1.33
windows phone 7 launch 2.66
boxee box 2.66
google tv 2.66
Table 6.5: The query words with minimum average human discord : The average humandiscord is average of the discord values of the 3 users who ordered the same set of 5 tweetsfor a query word on FB app)
29
than FLUR rank.
Also The fact that URLLength rank,which is an author independent influence measure,
performs better for some query words (ex. nissan leaf) than other influence measuring
strategies, indicates that that despite the very constrained size restriction on tweets, the
differences in length and presence of a URL still hold useful clues on the relative informa-
tiveness of tweets.
Another interesting indication by the table 6.1 is that when we used the klout score for
ranking the authors,for some queries (ex.ipad),the discord is as low as 1 but the similar
low discord value is also achieved by a pretty simple influence measuring strategy FR. For
the query words where discord values are high when using Klout (ex. Nissan leaf), indi-
cates that the human assessors are not always in agreement with the ordering of authors’
influence by Klout, which uses over 35 variables to compute the influence score. Thus the
analysis of table 6.1 shows that there is no one single algorithm that currently correctly out-
performs other algorithms and accurately reflects the end-user influence ranks; but several
influence measuring algorithms do come close to ideal. Surprisingly, our results indicate
that local measures of influence augmented by behavioral attributes of twitterers are nearly
as accurate as any existing exhaustive global measures.
30
Chapter 7
TIME VARYING INFLUENCE
Influence of authors in microblogs is likely to vary over time since the behaviroul at-
tibutes of the authors and the global structure of the social network varies with time. As
far as we know our work in this domain is the first academic effort to explore this. We
explored the domain of varying influence of twitterers over a span of 6 months. We provide
empirical data on the variation of influence and discuss the temporal robustness for our
proposed local influence measures. To analyse this, as explained in the chapter 4, we col-
lected enough tweets related to each of the 20 query words from twitter api along with their
authors’ information at 3 timestamps. We used our Facebook app to collect the ordering
for a set of 5 tweets for each query by 3 unique human assessors. We selected 5 tweets and
corresponding 5 authors for each query word and thus observed the 96 (since 4 authors were
repeated in the dataset) authors over a span of 6 months ie. from dec 2010 to may 2011.
The authors various statistics changed with time and so their ranks based on our reranking
strategies. This change in authors’ ranks over time is important to decide if time needs to
be taken under consideration while designing the rearnking strategies.
Based on the timestamped data we collected, we have drawn graphs to show how the au-
thors rank scores for various queries change over time.For some query words the some of
the authors under consideration got suspended by Twitter or got their account protected so
their information is not available to public. Due to this only those number of authors who
exist from Dec 2010 to May 2011 are shown in the graph. For the sake of brevity we are only
showing the best performing algorithm’s change over time for each query. In order to find
out the best reranking strategy for the 20 query words, the reranking strategies described
31
in Chapter 3 have been used. The best strategy is the one which has minimum discord1
(see table 6.1) with the average end user ordering.
To explain the time varying influence graphs, lets take figure 7.2, here the comparative
influence of the related 5 authors stays the same over the span of 6 months. In other words,
author 1 who had highest influence score (Highest tweet rank value) in December 2010 stays
on the top throughout since he has the highest influence score till May 2011.Similarly the
author 2 has higher influence score than author 3 throughout the span of 6 months. thus
we can see, during this span the twitterers comparative ranks stays the same and therefore
time does not change the influence of authors in this case.
In another graph,7.3, the author 3 who had the highest score in December 2010 goes
down to 2nd highest score in May 2011.Also author 4 who had the 2nd highest score in
december 2010 goes down to the 3rd highest score in May 2011. Interestingly the author 1
who was at the 3rd spot based on his tweet rank score in December 2010, goes up to the 1st
spot in May 2011. Thus in this case the comparative ranks of influential twitterers changes
over time.
We are presenting the graphs for the 4 categories of queries ie. Products, Events, Com-
pany/organizations and Miscellaneous. The products category has 4 query words namely,
kinect, boxee box, google tv , nissan leaf, i pad. Based on the dataset the minimum discord
for kinect out of the reranking strategies described in 3 was tweet rank and therefore the
time varying tweet rank scores of related 5 authors for this query are shown in 7.1.
32
For the best 5 query words for which the average human discord was minimum as
described in chapter 6(see table 6),in 2 out of the 5 cases the comparative ordering of
authors changed with time.
After observing all the timestamped social network properties of authors and correspond-
ing graph we found that more than 75 percent times the comparative ranking of authors
stays the same and does not change with time. As we have shown in graph 7.2 where the
authors’ hold their comparative rank and it does not change with time. In other words the
author 1 had highest TR score and Author 5 had least tweet rank score in December and
this pattern stays the same till May 2011 and thus the influence score ordering of authors
stays the same across a span of 6 months. Less than 25 percent of times the ordering of
authors based on our reranking strategies change as shown in graph 7.14 where the FLUR
rank score of author 2 was higher than author 1 in December 2010 but in May 2011 The
FLUR rank score of author 1 becomes higher than that of author 2. Since less than 25
percent times there is change in authoritative ordering of authors, we can say there is not
an intense need to add temporal dimension in the reranking strategies.
1if the best strategy is constant performing algorithm over time the second best one is used in graph
33
Figure 7.1: how the Tweet rank score of authors of5 tweet about query ’kinect’ changes over a span of 6months
Figure 7.2: how the Tweet rank score of authors of 5tweet about query ’boxee box’ changes over a span of6 months
Figure 7.3: how the Tweet rank score of authors of 5tweet about query ’google tv’ changes over a span of 6months
Figure 7.4: how the Tweet rank score of authors of 5tweet about query ’nissan leaf’ changes over a span of6 months
34
Figure 7.5: how the Tweet rank score of authors of5 tweet about query ’ipad’ changes over a span of 6months
Figure 7.6: how the Flur rank score of authors of 5tweet about query ’windows Phone 7 launch’ changesover a span of 6 months
Figure 7.7: how the Flur rank score of authors of 5tweet about query ’SXSW’ changes over a span of 6months
Figure 7.8: how the Flur rank score of authors of 5tweet about query ’halloween’ changes over a span of 6months
35
Figure 7.9: how the Tweet rank score of authors of 5tweet about query ’mid term election’ changes over aspan of 6 months
Figure 7.10: how the Flur rank score of authors of 5tweet about query ’thanksgiving’ changes over a spanof 6 months
Figure 7.11: how the Flur rank score of authors of 5tweet about query ’Microsoft’ changes over a span of 6months
Figure 7.12: how the Tweet rank score of authors of5 tweet about query ’asus’ changes over a span of 6months
36
Figure 7.13: how the Tweet rank score of authors of5 tweet about query ’zynga’ changes over a span of 6months
Figure 7.14: how the Flur rank score of authors of 5tweet about query ’Verizon’ changes over a span of 6months
Figure 7.15: how the Flur rank score of authors of 5tweet about query ’Costco’ changes over a span of 6months
Figure 7.16: how the Flur rank score of authors of 5tweet about query ’revenue’ changes over a span of 6months
37
Figure 7.17: how the tweet rank score of authors of5 tweet about query ’iran’ changes over a span of 6months
Figure 7.18: how the tweet rank score of authors of 5tweet about query ’facebook’ changes over a span of 6months
Figure 7.19: how the Flur rank score of authors of 5tweet about query ’flu’ changes over a span of 6 months
38
Chapter 8
COCLUSION AND FUTURE WORK
39
BIBLIOGRAPHY
[1] Aggarwal, C. C. Social Network Data Analytics. Kluwer Academic, 2011.
[2] Barabasi, A., and Albert, R. Emergence of Scaling in Random Networks. InProceedings of the National Academy of Sciences (1999).
[3] Bell, R., Koren, Y., and Volinsky, C. Hubs, authorities, and communities. InACM Computing Surveys (CSUR)Volume 31 Issue 4es, (1999).
[4] Bell, R., Koren, Y., and Volinsky, C. Modeling relationships at multiple scales toimprove accuracy of large recommender systems. In 13th ACM SIGKDD internationalconference on Knowledge discovery and data mining, pages 95104, New York, NY,USA, (2007).
[5] Borodin, A., Roberts, G., Rosenthal, J., and Tsaparas, P. Link analysis rank-ing: algorithms, theory, and experiments. In ACM Trans. Inter. Tech., 5(1):231297(2005).
[6] Chakrabarti, S., Dom, B., Kumar, S., Raghavan, P., Rajagopalan, S.,Tomkins, A., Gibson, D., and Kleinberg, J. Mining the webs link structure.In Computer, 32(8):6067 (1999).
[7] Faloutsos, M., Faloutsos, P., and Faloutsos, C. On powerlaw relationships ofthe internet topology. In SIGCOMM, pages 251262 (1999).
[8] Fernandez, J. klout-for-business-a-sometimes-useful-metric-but-an-incomplete-view-of-customers. http://www.web-strategist.com/blog/2011/02/21/klout-for-business-a-sometimes-useful-metric-but-an-incomplete-view-of-customers/comment-154250645.
[9] Java, A., Song, X., Finin, T., and Tseng, B. Why We Twitter: UnderstandingMicroblogging Usage and Communities. In Proceedings of WebKDD and SNAKDD(2007).
[10] Kleinberg, J., Kumar, R., Raghavan, P., Rajagopalan, S., and Tomkins,A. The Web as a graph: Measurements,models and methods. In Computer Science,1627:117 (1999).
40
[11] Kumar, R., Raghavan, P., Rajagopalan, S., and Tomkins, A. Core algorithmsin the clever system. In ACM Trans. Inter. Tech.,6(2):131152, (2006).
[12] Nagmoti, R., Teredesai, A., and De Cock, M. Ranking Approaches for Mi-croblog Search. In IEEE/WIC/ACM International Conference on Web Intelligenceand Intelligent Agent Technology (2010).
[13] Newman, M. Power laws, pareto distributions and zipfs law. In Contemporary Physics(2005).
[14] Page, L., Brin, S., Motwani, R., and Winograd, T. The PageRank CitationRanking: Bringing Order to the Web. In in Proceedings of the Seventh InternationalWeb Conference(WWW 98) (1998).
[15] Pal, A., and Counts, S. Identifying Topical Authorities in Microblogs. In proceedingsof WSDM 2011 (2011).
[16] Ramage, D., Dumais, S., and Liebling, D. Characterizing micorblogs with topicmodels. In In Proceedings of Fourth International AAAI Conference on Weblogs andSocial Media, pages 130137, (2010).
[17] Richardson, M., and Domingos, P. Mining knowledge-sharing sites for viral mar-keting. In KDD (2002).
[18] Victor, P., Cornelis, C., Cock, M., and Teredesai, A. Trust- and Distrust-Based Recommendations for Controversial Reviews. In Intelligent Systems, IEEE,Volume: 26, Issue: 1 (2011), pp. 48 – 55.
[19] Weng, J., Peng, E., Jiang, J., and He, Q. Twitter Rank: Finding Topic SensitiveInfluencial Twitterers. In Proceedings of the third ACM international conference onWeb search and data mining (2010).
[20] Yamaguchi, Y., Takahashi, T., Amagasa, T., and Kitagawa, H. TUrank:Twitter User Ranking based on User-Tweet Graph Analysis. In Proceedings of Wise2010:11th International Conference (2010).
[21] Yao, L., Mimno, D., and McCallum, A. Efficient Methods for Topic Model Infer-ence on Streaming Document Collections. In Knowledge Discovery and Data Mining(KDD) (2009).
[22] Zhan, Y.and Chakrabarti, D., and Faloutsos, C. R-MAT: A recursive modelfor graph mining. In SIAM Int. Conf. on Data Mining (2004).