photo credit donsolo, CC BY-‐NC-‐SA 2.0
From Site to Inter-‐site User Engagement Jane;e Lehmann
Barcelona, February 26, 2015
Advisors: Ricardo Baeza-‐Yates Co-‐Advisor: Mounia Lalmas
• User engagement is a quality of the user experience that emphasizes the posiLve aspects of interacLon with a website – in parLcular the fact of being capLvated by the website.
• In-‐the-‐moment engagement Users stay on a website over a long Lme.
• Long-‐term engagement Users come back frequently and over a long-‐term.
IntroducLon 2
User Engagement DefiniLon
Successful websites are not just used, they are engaged with.
User Engagement Measuring
3 IntroducLon
Before we can design engaging websites, it is crucial that we are able to measure engagement.
“If you can measure it, you can improve it.” Sir William Thomson
Analysis/Planning
Design Changes Measuring
Main Research Goals
4 IntroducLon
Primary goal Can we define new engagement metrics that Measuring enhance our understanding of engagement?
Secondary goal Can we idenLfy ways to influence engagement?Analysis/Planning
Analysis/Planning
Design Changes Measuring
IntroducLon 5
Analysis/Planning
Design Changes Measuring Online mulLtasking
Inter-‐site engagement
Site engagement
Effect of providing off-‐site content Effect of hyperlinks
IntroducLon 6
Analysis/Planning
Design Changes Measuring
Site engagement
Measuring Engagement InteracLon data
7 Site engagement
Data Browsing events provided by Yahoo toolbar (client-‐side).
Engagement Analysing the data using online behaviour metrics.
Online session:
Visit on Yahoo News
Site engagement 8
Measuring Engagement Online behaviour metrics
K. Rodden, H. Hutchinson, X. Fu. Measuring the user experience on a large scale: User-‐centered metrics for web applicaHons. CHI, 2010. E. Peterson, J. Carrabis. Measuring the immeasurable: Visitor engagement. Web AnalyHcs DemysHfied, 2008. B. Haven, S. ViWal. Measuring engagement. Forrester Research, 2008. B. Weischedel and E. Huizingh. Website opHmizaHon with web metrics: A case study. Conference on Electronic commerce, 2006.
Site engagement 9
Measuring Engagement Online behaviour metrics
Popularity
#Users Number of users.
#Visits Number of visits.
#Clicks Number of clicks.
AcCvity (within a visit) In-‐the-‐moment engagement
PageViews Avg. number of page views per visit.
DwellTime Avg. Lme on site per visit.
Loyalty (across visits) Long-‐term engagement
ReturnRate Number of Lmes a user visited the site.
AcLveDays Number of days a user visited the site.
Site engagement 10
Measuring Engagement Differences in engagement
ComScore, Alexa, GoogleAnalyHcs,…
Shopping Users do not come frequently, but stay long
Games Not many users, but they stay long
News Users come frequently and stay long
Measuring Engagement Problem
11 Site engagement
Isolated view: The metrics focus on engagement with a single site.
RelaLonships to other sites are not considered.
IntroducLon 12
Analysis/Planning
Design Changes Measuring Online mulLtasking
Site engagement
Online mulLtasking 13
MoCvaCon In-‐the-‐moment engagement
ComScore, Alexa, GoogleAnalyHcs,…
What web analyCcs think we do…
1 visit with 4 page views.
Online mulLtasking 14
MoCvaCon In-‐the-‐moment engagement
ComScore, Alexa, GoogleAnalyHcs,…
… and what we really do:
3 visit with on average 1.3 page views.
Online mulLtasking 15
MoCvaCon Online mulLtasking.
Problem
• Engagement metrics do not capture such behaviour.
• Measuring acLvity on a site can lead to incorrect conclusions.
Online mulCtasking Users visit several sites and switch between them
during an online session, to perform related or totally unrelated tasks.
Research QuesCon
16 Online mulLtasking
How can we measure engagement by accounLng for user mulLtasking behaviour?
Analysis/Planning
Design Changes Measuring
Extent of mulCtasking • 10.2 disLnct sites, 2 visits per site.
Absence Cme • 50% of sites are revisited aker < 1min.
InterrupHon of a task
• There are revisits aker long breaks. Performing a new task
Online mulLtasking 17
Online MulCtasking CharacterisLcs
0.00
0.25
0.50
0.75
1.00
10ï2
10ï1
100
101
102
Cum
ula
tive
pro
bab
ility
Absence time [min]
news (finance)
news (tech)
social media
2.09
1.76
2.28
2.09
#Visits Absencetime [min]
3.85
3.95
4.47
6.86
Absence time: Time between two visits
AcCvity paPerns
• Four types: Decreasing, increasing, constant, complex. • Successive visits can belong together (i.e. to the same task). • Complex cases refer to no specific pa;ern or repeated pa;ern.
Online mulLtasking 18
Online MulCtasking CharacterisLcs
1 2 3 4ith visit on site
1 2 3 4ith visit on site
1 2 3 4ith visit on site
1 2 3 4ith visit on site
Pro
por
tion
of to
tal
dw
ell tim
e on
site
0.23
0.28
0.33 p-value = 0.09m = -0.01
p-value = 0.07m = -0.02
p-value = 0.79m = 0.00
news (finance) sitesmail sites social media sites news (tech) sites
decreasing attention increasing attention constant attention complex attention
Online mulLtasking 19
Measuring Engagement Online mulLtasking metrics
Extent of mulCtasking
SessSites Total number of sites accessed (#tasks).
SessVisits Number of visits to site (site switching).
Absence Cme
CumAct Aggregates the dwell Lmes of the visits with accounLng for the Lme between the visits.
AcCvity paPern
A;Shik A;Range Describe the four cases of a;enLon shiks.
20
CASE STUDY: MulCtasking PaPerns • ObjecCve: Analyse mulLtasking acLvity on sites;
idenLfy mulLtasking pa;erns (clustering).
• Metrics: Site DwellTime, MulLtasking metrics.
• Data: July 2012, 2.5M users, 760 sites (shopping, news, search, etc.).
21
Case Study: MulCtasking PaPerns Results
No mulCtasking MulCtasking
Quick Focused Rapid ConCnuous Recurring Checking weather
Reading mails
Following link to off-‐site content
Purchasing an item
Performing search
Site DwellTime -‐-‐ ++ ++ ++ -‐-‐
Extent of mulCtasking -‐-‐ -‐-‐ ++ ++ ++
Absence Cme -‐-‐ ++ ++
ImplicaCons Provide
interesHng off-‐site content
Shopping takes more than
one visit
Support user by finishing tasks quickly
Online mulLtasking
-- low value ++ high value
22
Case Study: MulCtasking PaPerns Results
No mulCtasking MulCtasking
Quick Focused Rapid ConCnuous Recurring Checking weather
Reading mails
Following link to off-‐site content
Purchasing an item
Performing search
Site DwellTime -‐-‐ ++ ++ ++ -‐-‐
Extent of mulCtasking -‐-‐ -‐-‐ ++ ++ ++
Absence Cme -‐-‐ ++ ++
AcCvity paPern
Online mulLtasking
De In CmCn
60%
0%De In CmCn
60%
0%De In CmCn
60%
0%
Activity pattern: De – Decreasing In – Increasing Cn – Constant Cm - Complex -- low value ++ high value
23
CASE STUDY: Wikipedia (on-‐site mulCtasking) • ObjecCve: Analyse reading acLvity on Wikipedia
arLcles; idenLfy reading pa;erns (clustering).
• Metrics: ArLcle DwellTime, #ArLcles in session, #Views to focal arLcle.
• Data: Sep 2011 – Sep 2012, 500K users, 10K biography arLcles.
24
Case Study: Wikipedia Approach
Online mulLtasking
Users’ reading behaviour on an Wikipedia arCcle
ArLcle DwellTime How much Lme do users spend on an arLcle? #ArLcles in session Do users view also other arLcles during an
online session? #Views on focal arLcle How oken do users view the arLcle?
25
Case Study: Wikipedia Results
No mulCtasking MulCtasking
Focus ExploraCon Passing Focus is on focal arHcle
Exploring topic around the focal arHcle
Exploring topic and pass through the focal arHcle
ArCcle DwellTime ++ -‐-‐
#ArCcles in session -‐-‐ ++ ++
#Views to focal arCcle ++ -‐-‐
ImplicaCons Content quality is important
Links to addiHonal content are important
ArHcles might need to be extended
Online mulLtasking
On-‐site mulCtasking
• MulLtasking between news arLcles of a provider. • MulLtasking between different tasks on a social media
site (e.g. sharing, chapng, updaLng profile). • … Inter-‐site mulCtasking
• MulLtasking when purchasing items online (comparing offers, product reviews, search, etc.)
• …
Online mulLtasking 26
Further Use Cases
Take Aways
• AccounLng for mulLtasking leads to a be;er understanding on how users engage with sites.
• Leaving a site does not necessarily
entail less engagement, as users oken return to the site later on.
Publications J. Lehmann, M. Lalmas, G. Dupret, and R. Baeza-Yates. Online multitasking and user engagement. CIKM 2013.
J. Lehmann, C. Müller-Birn, D. Laniado, M. Lalmas, and A. Kaltenbrunner. Reader preferences and behavior on Wikipedia. HT 2014, Ted Nelson Newcomer Paper Award.
J. Lehmann, C. Müller-Birn, D. Laniado, M. Lalmas, and A. Kaltenbrunner. What and how users read: Transforming reading behavior into valuable feedback for the Wikipedia community. Wikimania 2014.
Online mulLtasking 27
IntroducLon 28
Analysis/Planning
Design Changes Measuring Online mulLtasking
Inter-‐site engagement
Site engagement
Inter-‐site engagement 29
MoCvaCon Large online service providers
ComScore, Alexa, GoogleAnalyHcs,…
Engagement Popularity: #Users, #Visits, … AcLvity: DwellTime, PageViews, … Loyalty: ReturnRate, AcLveDays, …
Inter-‐site engagement 30
MoCvaCon Large online service providers
frontpage
tv sports
shopping
autos
search
daLng
jobs
news
shine
groups
maps local
health answer
weather
games
omg
homes travel
flickr
finance
Large online service providers (AOL, Google, Yahoo, etc.) have not only one site, but many sites.
tumblr
Inter-‐site engagement 31
MoCvaCon Large online service providers
frontpage
tv sports
shopping
autos
search
daLng
jobs
news
shine
groups
maps local
health answer
weather
games
omg
homes travel
flickr
finance
Providers want that users engage with many of their sites.
tumblr
Inter-‐site engagement 32
MoCvaCon Online mulLtasking
Problem
• Engagement metrics do not measure engagement across sites. • How to adapt them is not obvious.
Inter-‐site engagement Users visit sites that belong to the
same network of sites.
Research QuesCon
33 Inter-‐site engagement
How can we measure engagement by also considering the
relaLonships between sites?
Analysis/Planning
Design Changes Measuring
Inter-‐site engagement 34
Traffic Networks Modelling
We model sites (nodes) and user traffic (edges) between them as a network. Provider network G=(N, E, λ)
N: Sites E: User traffic λ(e): Traffic volume (#Clicks)
4 clicks
2 clicks
50 clicks 10 clicks
Inter-‐site engagement 35
Measuring Engagement Inter-‐site engagement metrics: Network-‐level
Traffic distribuCon
Flow Extent to which users navigate between sites.
Density1 Diversity of inter-‐site engagement.
Reciprocity2 Homogeneity of traffic between sites.
External traffic
EntryDisparity Variability of in-‐going traffic to the network.
ExitDisparity Variability of out-‐going traffic from the network.
[1] S. Wasserman. Social network analysis: Methods and applicaHons, 1994. [2] T. SquarHni, F. Picciolo, F. RuzzenenH, and D. Garlaschelli. Reciprocity of weighted networks. Nature: ScienHfic reports, 2013.
Inter-‐site engagement 36
Measuring Engagement Inter-‐site engagement metrics: Node-‐level
Traffic distribuCon
PageRank1 Probability that a user will visit the site.
Downstream Probability that a user will conLnue browsing to other sites.
External traffic
EntryProb Probability that a user enters the network in this site.
ExitProb Probability that a user leaves the network in this site.
[1] L. Page, S. Brin, R. Motwani, T. Winograd. The pagerank citaHon ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
37
CASE STUDY: Yahoo Provider Networks • ObjecCve: Compare networks; characterise the sites
in a network.
• Metrics: Network DwellTime, Site DwellTime, Inter-‐site engagement metrics.
• Data: February 2014, 3.2M clicks/network, 4 country-‐based networks, 31 sites per network.
38
Case Study: Yahoo Comparing provider networks
Network 1 Network 2 Network 3 Network 4 High
engaging Users engage quickly
with many sites Users engage to a subset of sites
Low engaging
Network DwellTime ++ -‐-‐ ++ -‐-‐
Traffic DistribuCon ++ ++ Flow ++
Density -‐-‐ -‐-‐
Entry Disparity ++ -‐-‐ ++
ImplicaCons The network is performing
well.
This should be looked into.
MoHvate users to visit other sites.
This should be looked into.
Inter-‐site engagement
-- low value ++ high value
39
Case Study: Yahoo Sites within a provider network
Traffic Hub Supporter Focused Engagement
Shared Engagement
Search, front pages Support, services Leisure, support News, leisure
Site DwellTime -‐-‐ -‐-‐ ++ ++
Traffic DistribuCon ++ -‐-‐ -‐-‐ ++
Entry Probability ++ -‐-‐ ++ -‐-‐
ImplicaCons The sites
forward traffic to other sites.
Users visit sites for specific needs and support.
MoHvate users to visit other sites.
The sites are performing
well.
Inter-‐site engagement
-- low value ++ high value
Comparing networks
• Device, Lme, upstream traffic, user. • SimulaLons (effect of adding/removing sites). • … Network types
• Network of pages (e.g. compare language-‐based Wikipedia networks)
• Network of sites from different providers (e.g. shopping sites, news providers)
• …
Inter-‐site engagement 40
Further Use Cases
Take Aways
• Inter-‐site engagement allows for a more comprehensive look at user engagement by also considering the relaLonships between sites.
• Deeply engaged users do not only
engage with one site, but with many sites in a network.
Publications J. Lehmann, M. Lalmas, and R. Baeza-Yates. Measuring Inter-Site Engagement. Handbook of Statistics, Elsevier, 2015. To appear.
J. Lehmann, M. Lalmas, R. Baeza-Yates, and E. Yom-Tov. Networked User Engagement. ACM Workshop on User engagement optimization at CIKM, 2013.
J. Lehmann, M. Lalmas, and R. Baeza-Yates. Temporal Variations in Networked User Engagement. TNETS Satellite at ECCS, 2013.
Some of the metrics were employed to characterise online news reading across news sites: J. Lehmann, C. Castillo, M. Lalmas, and R. Baeza-Yates. Story-Focused Reading in Online News. Submitted for publication.
Inter-‐site engagement 41
IntroducLon 42
Analysis/Planning
Design Changes Measuring Online mulLtasking
Inter-‐site engagement
Site engagement
Effect of providing off-‐site content
43
CASE STUDY: Online News • Hypothesis: It may be beneficial (long-‐term) to
enLce users to leave a site by offering interesLng off-‐site content.
• Data: October 2013, 57K users, 50 news sites, 26K news arLcles.
Types of reading sessions
No click Did not follow a related link.
Off-‐site click Followed a related link to content on another site.
Effect on engagement
Short-‐term Dwell Lme per reading session.
Long-‐term Probability that user starts next reading session within the next 12h.
44
Case Study: Online News
Related off-‐site content
Approach
Effect of providing off-‐site content
Providing links to related off-‐site content has a no short-‐term effect, but a posiCve long-‐term effect.
45
Case Study: Online News Results
Effect of providing off-‐site content
News provider
Dw
ell tim
e per
ses
sion
News provider
p(a
bse
nce
12
h)
No Click Off-site click
IntroducLon 46
Analysis/Planning
Design Changes Measuring Online mulLtasking
Inter-‐site engagement
Site engagement
Effect of providing off-‐site content Effect of hyperlinks
47
CASE STUDY: Yahoo Provider Network • Hypothesis: We can use hyperlinks to influence
inter-‐site engagement in a provider network.
• Data: February 2014, 235M clicks, Yahoo US network, 73 sites.
Hyperlink vs. traffic network On-‐site Links/Traffic to pages
within the same site. Inter-‐site Links/Traffic to pages to
other sites in the network.
External Links/Traffic to
somewhere else on the Web.
48
Case Study: Yahoo Approach
frontpage
sports
search
news
shine
groups
answer
weather
omg
homes
flickr
Effect of hyperlinks
Hyperlinks can be used to influence site and inter-‐site engagement in a provider network.
However, both types of engagement influence each other.
49
Case Study: Yahoo Results
Effect of hyperlinks
TrafficOn-site Inter-site External
HyperlinksOn-siteInter-siteExternal
0.54-0.40
-
-0.450.50
-
-0.38-
0.39
IntroducLon 50
Analysis/Planning
Design Changes Measuring Online mulLtasking
Inter-‐site engagement
Site engagement
Effect of providing off-‐site content Effect of hyperlinks
Two new perspecHves for measuring engagement which consider the relaLonships between sites. Online mulCtasking Accounts for user mulLtasking behaviour. Inter-‐site engagement Accounts for the traffic between sites.
ContribuLons and future work 51
Main ContribuCons Measuring engagement
Analysis/Planning
Design Changes Measuring
AccounLng for the new perspecLves when influencing engagement. Online news Providing related off-‐site content influences long-‐term engagement. Provider network Hyperlinks affect site and inter-‐site engagement, but both influence each other.
ContribuLons and future work 52
Main ContribuCons Analysis/Planning
Analysis/Planning
Design Changes Measuring
Wikipedia Providing informaLon about readers’ engagement to the editor community.
Yahoo Using inter-‐site engagement metrics to make informed decisions about design changes (hyperlinks).
Spiegel Online Measuring and improving engagement by providing interesLng off-‐site content.
ContribuLons and future work 53
What next? Ongoing and future work
Analysis/Planning
Design Changes Measuring
photo credit donsolo, CC BY-‐NC-‐SA 2.0
Thank you!
Jane;e Lehmann
Barcelona, February 26, 2015 [email protected]
Acknowledgements Ricardo Baeza-‐Yates Mounia Lalmas Claudia Müller-‐Birn Carlos CasLllo David Laniado Andreas Kaltenbrunner Elad Yom-‐Tov Georges Dupret Guy Shaked Fabrizio Silvestri Gabriele Tolomei Ethan Zuckerman John Agapiou Andy Haines Diego Sáez-‐Trumper Hemant Purohit Noora Al Emadi Mohammed El-‐Haddad Nasir Khan
• Mounia Lalmas and Janette Lehmann. “Models of User Engagement”. In H. L. O’Brien and M. Lalmas (Eds.), Why Engagement Matters: Cross-disciplinary Perspectives and Innovations on User Engagement with Digital Media. Springer, 2015, in progress.
• Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. “Models of user engagement.” International Conference on User Modeling, Adaptation, and Personalization (UMAP 2012), pp. 164-175, Montreal, Canada, July, 2012.
• Janette Lehmann, Mounia Lalmas, Georges Dupret, and Ricardo Baeza-Yates. “Online multitasking and user engagement.” ACM International Conference on Information and Knowledge Management (CIKM 2013), pp. 519-528, San Francisco, United States, October, 2013.
• Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Measuring Inter-Site Engagement.”. In V. Govindaraju, V. V. Raghavan, and C. R. Rao (Eds.), Handbook of Statistics, Elsevier, 2015.
• Janette Lehmann, Mounia Lalmas, Ricardo Baeza-Yates, and Elad Yom-Tov. “Networked User Engagement.”, ACM Workshop on User engagement optimization at CIKM, pp. 7-10, San Francisco, United States, October, 2013.
• Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Temporal Variations in Networked User Engagement.”, TNETS Satellite at European Conference on Complex Systems (ECCS), Barcelona, Spain, September, 2013.
• Mounia Lalmas, Janette Lehmann, Guy Shaked, Fabrizio Silvestri, and Gabriele Tolomei. “Measuring Post-click User Experience with Mobile Native Advertising on Streams.”, submitted for publication.
• Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “Reader preferences and behavior on Wikipedia.”, ACM International Conference on Hypertext and Social Media (HT 2014), pp. 88-97, Santiago, Chile, September, 2014, Ted Nelson Newcomer Paper Award.
• Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “What and how users read: Transforming reading behavior into valuable feedback for the Wikipedia community.”, Presentation at Wikimania, London, UK, August, 2014.
• Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ricardo Baeza-Yates. “Story-Focused Reading in Online News.”, submitted for publication.
• Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Transient News Crowds in Social Media.” International AAAI Conference on Weblogs and Social Media (ICWSM 2013), Boston, USA, July, 2013.
• Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Finding News Curators in Twitter.” ACM International Conference on World Wide Web Companion (WWW 2013 Companion), 863-870, Rio de Janeiro, Brazil, May, 2013.
55
PublicaCons
User engagement
• Mounia Lalmas, Heather L O’Brien, and Elad Yom-Tov. Measuring user engagement. Synthesis Lectures on Sample Series #1. Morgan and cLaypool publishers, 2014.
• Heather L O’Brien and Elaine G Toms. What is user engagement? a conceptual framework for defining user engagement with technology. American Society for Information Science and Technology (ASIS&T), 59(6):938–955, 2008.
• Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. Towards a science of user engagement (position paper). In Proc. Workshop on User Modelling for Web Applications, WSDM, 2011.
• Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010.
Online behaviour metrics
• Brian Haven and Suresh Vittal. Measuring engagement. Forrester Research, 2008.
• Eric T Peterson and Joseph Carrabis. Measuring the immeasurable: Visitor engagement. Web Analytics Demystified, 2008.
• Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010.
• Georges Dupret and Mounia Lalmas. Absence time and user engagement: evaluating ranking functions. In Proc. Conference on Web Search and Data Mining, WSDM, pages 173–182. ACM, 2013.
• Randolph E Bucklin and Catarina Sismeiro. A model of web site browsing behavior estimated on clickstream data. Journal of Marketing Research, 40(3):249–267, 2003.
• Birgit Weischedel and Eelko KRE Huizingh. Website optimization with web metrics: a case study. In Proc. Conference on Electronic commerce: The new e-commerce: innovations for conquering current barriers, obstacles and limitations to
conducting successful business on the internet, pages 463–470. ACM, 2006.
• Peifeng Yin, Ping Luo, Wang-Chien Lee, and Min Wang. Silence is also evidence: interpreting dwell time for recommendation from psychological perspective. In Proc. Conference on Knowledge Discovery and Data Mining, SIGKDD, pages 989–997. ACM, 2013.
56
Selected References
Online multitasking
• Qing Wang and Huiyou Chang. Multitasking bar: prototype and evaluation of introducing the task concept into a browser. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 103–112. ACM, 2010.
• Hartmut Obendorf, Harald Weinreich, Eelco Herder, and Matthias Mayer. Web page revisitation revisited: implications of a long-term click-stream study of browser usage. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 597–606. ACM, 2007.
• Jeff Huang and Ryen W White. Parallel browsing behavior on the web. In Proc. Conference on Hypertext and Hypermedia, HT, pages 13–18. ACM, 2010.
• Patrick Dubroy and Ravin Balakrishnan. A study of tabbed browsing among mozilla firefox users. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 673–682. ACM, 2010.
Inter-site engagement
• Mark EJ Newman. The structure and function of complex networks. SIAM review, 45(2):167–256, 2003. 76, 77, 165
• Anna Chmiel, Kamila Kowalska, and Janusz A Hołyst. Scaling of human behavior during portal browsing.
• Mark R Meiss, Filippo Menczer, Santo Fortunato, Alessandro Flammini, and Alessandro Vespignani. Ranking web sites with real user traffic. In Proc. Conference on Web Search and Data Mining, WSDM, pages 65–76. ACM, 2008.
• Young-Hoon Park and Peter S Fader. Modeling browsing behavior at multiple websites. Marketing Science, 23(3):280–303, 2004.
• Qiqi Jiang, Chuan-Hoo Tan, and Kwok-Kee Wei. Cross-website navigation behavior and purchase commitment: A pluralistic field research. In Proc. Pacific Asia Conference on Information Systems, PACIS, 2012.
• Kevin Koidl, Owen Conlan, and Vincent Wade. Cross-site personalization: assisting users in addressing information needs that span independently hosted websites. In Proc. Conference on Hypertext and Hypermedia, HT, pages 66–76. ACM, 2014.
• The PEW Research Center. Understanding the participatory news consumer. http://www.pewinternet.org/~/media/Files/Reports/ 2010/PIP_Understanding_the_Participatory_News_Consumer. pdf, 2010.
• Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012.
57
Selected References
Link economy
• Joseph Turow and Lokman Tsui. The hyperlinked society. The University of Michigan Press, 2008.
• Juliette De Maeyer. Hyperlinks and journalism: where do they connect? In Proc. Future of Journalism Conference, 2011.
• Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink
formation in content networks. Management Science, 59(10):2360–2379, 2013.
• Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012.
• Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink
formation in content networks. Management Science, 59(10):2360–2379, 2013.
• Hakan Ceylan, Ioannis Arapakis, Pinar Donmez, and Mounia Lalmas. Automatically embedding newsworthy links to articles. In
Proc. Conference on Information and Knowledge Management, CIKM, pages 1502–1506. ACM, 2012.
Recommendation
• Richard McCreadie, Craig Macdonald, and Iadh Ounis. News vertical search: when and what to display to users. In Proc.
Conference on Research and Development in Information Retrieval, SIGIR, pages 253–262. ACM, 2013.
• Samuel Ieong, Mohammad Mahdian, and Sergei Vassilvitskii. Advertising in a stream. In Proc. Conference on World Wide
Web, WWW, pages 29–38. ACM, 2014.
• Eric Sodomka, Sébastien Lahaie, and Dustin Hillard. A predictive model for advertiser value-per-click in sponsored search. In
Proc. Conference on Information and Knowledge Management, CIKM, pages 1179–1190. ACM, 2013.
• Narongsak Thongpapanl and Abdul Rehman Ashraf. Enhancing online performance through website content and
personalization. Journal of Computer Information Systems, 52(1):3, 2011.
• Jian Wang and Yi Zhang. Utilizing marginal net utility for recommendation in e-commerce. In Proc. Conference on Research
and Development in Information Retrieval, SIGIR, pages 1003–1012. ACM, 2011.
• Joshua Porter. Designing for the social web. Peachpit Press, 2010.
58
Selected References
ATTACHMENT: IntroducCon
• “In a world full of choices where the fleeCng aPenCon of the user becomes a prime resource, it is essenLal that [...] providers do not just design [websites] but that they design engaging experiences.” [A}ield].
• In addiLon to uLlitarian factors, such as usability and usefulness, we must consider other factors of interacLng with websites, such as fun, fulfillment, play, and user engagement.
Successful websites are not just used, they are engaged with.
• In order to design engaging websites, it is crucial to understand what user engagement is and how to measure it.
IntroducLon 60
MoCvaCon Why is it important to engage users?
Methodology InteracLon data, online sessions and site visits.
61 IntroducLon
t0
t1
t2
t3
t4
t5
t6
t7
sessionend
sessionstart
time
Online session
Browsing activity on Wikipedia
https://ie-mg42.mail.yahoo.com
http://en.wikipedia.org/wiki/Freddie˙Mercury
http://www.bbc.com/news/uk-29149115
http://www.bbc.com/news/uk-england-nottinghamshire-29643802
http://en.wikipedia.org/wiki/Star Wars
http://en.wikipedia.org/wiki/Yoda
http://en.wikipedia.org/wiki/Albert˙Einstein
https://www.facebook.com/janette.lehmann.5
t0
t1
t2
t3
t4
t5
t6
t7
bc0
bc0
bc0
bc0
bc0
bc0
bc0
bc0
BCookie Timestamp URL
-
-
-
http://www.bbc.com/news/uk-29149115
-
http://en.wikipedia.org/wiki/Star Wars
http://en.wikipedia.org/wiki/Yoda
-
ReferrerURL
Interaction data
Page view on Wikipedia Page view on other site
IntroducLon 62
Thesis structure
Metrics that account for sitepopularity, activity and loyalty
AdvertisingChapter 7
Site engagement
How users experience ads on desktop and
mobile devices?
Does ad qualityaffect the engagement
with the publisher?
How can we identifyhigh quality ads?
Site engagementChapter 4
MultitaskingChapter 5
Inter-site engagementChapter 6
Metrics that account for traffic between sites
Metrics that account for user multitasking behaviour
(II
I+IV
) A
pplica
tion
s (
II)
Fund.
WikipediaChapter 8
Site engagementand multitasking
How users readarticles
in Wikipedia?
Does the activityof editors align with theengagement of readers?
How can readers bevaluable for editors?
YahooChapter 9
Inter-site engagement
How users engage with a providernetwork of sites?
Does the hyperlink structure affect site and inter-site engagement?
Online newsChapter 10+11
Inter-site engagement
How users readstories across
news providers?
Do hyperlinks torelated content influenceprovider engagement?
How can we automaticallydetect related content?
Characterising user engagement Comparing site characteristics and user engagement Applications to impact user engagement
ATTACHMENT: Site engagement
0-1 1-0.5 0.5Kendall’s tau with p-value < 0.05
('-' insignificant correlations)
Site engagement 64
EvaluaCon CorrelaLons between engagement metrics.
High correlaCons within metric groups.
Low correlaCons between metric groups.
[PO
P]
#U
se
rs
[PO
P]
#V
isits
[PO
P]
#C
licks
[AC
T]
Pa
ge
Vie
wsV
[AC
T]
Dw
ellT
ime
V
[LO
Y]
Active
Da
ys
[LO
Y]
Re
turn
Ra
te
#Users [POP] 0.82 0.75 - - 0.43 0.34
#Visits [POP] 0.82 0.85 - - 0.60 0.52
#Clicks [POP] 0.75 0.85 0.16 0.18 0.59 0.51
PageViewsV [ACT] - - 0.16 0.33 - -
DwellTimeV [ACT] - - 0.18 0.33 - -
ActiveDays [LOY] 0.43 0.60 0.59 - - 0.79
ReturnRate [LOY] 0.34 0.52 0.51 - - 0.79
0.69
Site engagement 65
PaPerns of Site Engagement Engagement depends on the site at hand.
Games Not many users, but they stay long
Search Users come frequently, but do not stay long
Social media Users come frequently and stay long
Shopping Users do not come frequently, but stay long
News Users come frequently and stay long
Service Users do not come frequently, but stay long
ATTACHMENT: MulCtasking
Online mulLtasking 67
MoCvaCon
Users switch between sites, to perform related or totally unrelated tasks. Switching between tasks (sites) “…within-‐session page revisits represent the most common form of revisitaLon, covering 73,54% of all revisits.” [Herder] Performing tasks (sites) in parallel using browser tabs “Most of our parLcipants switched tabs more oken than they used the back bu;on.” [Dubroy]
[Herder] E. Herder. CharacterizaHons of user web revisit behavior. WWW Workshop ABIS, 2005. [Dubroy] P. Dubroy, R. Balakrishnan. A study of tabbed browsing among mozilla firefox users. SIGCHI, 2010.
Online mulLtasking 68
Data Dataset and site categories.
Cat. Subcat. %Sites Description
news
22.1%
news 5.79%news (soc.) 5.13% societynews (sport) 2.63%news (enter.) 2.24% music, movies, tv, etc.news �¿QDQFH� 1.97%news (life) 1.58% health, housing, etc.news (tech) 1.58% technologynews (weather) 1.18%
service
15.5%
service 7.63% translators, banks, etc.mail 3.95%maps 3.03%organisation 0.92% bookmarks, calendar, etc.
search
15.3% search 12.63%
search (special) 1.58% search for lyrics, jobs, etc.directory 1.05%
sharing
9.6%
blogging 3.55%knowledge 3.55% collaborative creation and collection of contentsharing 2.50% sharing of videos, ¿OHV� etc.
navi
9.3%
front page 6.58%front page (p.) 1.84% personalised front pagessitemap 0.92%
leisure
8.7%
adult 2.76%games 1.97%social media 1.97%dating 1.05%entertainment 0.92% sites with music, tv, etc.
support
8.7%
support 1.58% sites that provide products and support for themdownload 7.11% downloading software
shopping
7.9%
shopping 4.34%auctions 2.11%comparison 1.45% sites to compare prices of products
settings
2.9%
login 1.71%site settings 1.18% prR¿Oe setting, site personalisation
InteracCon data • July 2012 • 2.5M users • 785M page views
NavigaCon model • We defined a new navigaLon
model (see paper for details)
Site categories • 760 sites from 70 countries/
regions • 11 categories • 33 subcategories
Online mulLtasking 69
MulCtasking Metrics CumAct accounts for the acLvity between site visits.
CumulaCve acCvity The metric is defined as follows:
InterpretaCon High CumAct à High engagement If users return aker short Lme, they return to conLnue with same task. If users return aker longer Lme, they return to perform a new task – a sign of loyalty.
CumActk = log10 (v1 + ivik •vi
i=2
n
∑ )
Browsing acLvity during the ith visit Browsing acLvity between the (i-‐1)th and ith visit Rescaling factor for ivi
k = 3
viivi
1 4 3 10 3
CumAct= log10 (3+1
3 •4+103 •3)= 3.48
Site visit
Online mulLtasking 70
MulCtasking Metrics AWRange and AWShik describe changes between the visits.
APenCon shie and range The metrics is defined as follows:
InterpretaCon AWShik models the shik of a;enLon, and AWRange models the fluctuaLons in the browsing acLvity.
AttShiftn =invn −min Invn
| max Invn |− | min Invn |
AttRangen =σ (Vn )µ(Vn )
Variance in the visit acLvity Average of the visit acLvity Number of visits in session ModificaLon of the “Inversion number”
n = 4
σµi
Inv
0 >0
-‐1 constant decreasing
0
constant complex
+1
constant increasing
AWenHon range
AWenHo
n shik
0-1 1-0.5 0.5Spearman’s rho with p-value < 0.05
('-' insignificant correlations)
Online mulLtasking 71
EvaluaCon CorrelaLons between mulLtasking and acLvity metrics.
[MT]
Ses
sVis
its
[MT]
Ses
sSite
s
[MT]
Cum
Act
[MT]
AttS
hift
[MT]
AttR
ange
[AC
T] D
wel
lTim
eS
SessSites [MT] 0.42
CumAct [MT] 0.41 -
AttShift [MT] 0.09 - -
AttRange [MT] - - -0.38 0.27
DwellTimeS [ACT] 0.20 0.24 0.12 0.32 0.08
DwellTimeV [ACT] -0.40 - - 0.14 - 0.50
No or only weak correlaCons between
the metrics.
All metrics convey different aspects about users’ online behaviour.
Online mulLtasking 72
MulCtasking PaPerns Cluster centers, site categories and acLvity pa;erns.
Cat
egor
ies
Multitas
kin
g
DwellTimeV CumAct SessVisitsDwellTimeS
sitemapsite settings
news (wheather)download
+75%+73%+69%+67%
PD
139 sites
Quick task Continuousmultitasking
SessSitesBars from left to right:
111 sites
auctionsshopping
adultdating
+79%+71%+71%+62%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
PD - Probability difference
Act
ivity
Activity pattern: De - Decreasing In - Increasing Cn - Constant Cm - Complex
De In CmCn De In CmCn
60%
0%
147 sites
Recurring task
searchfront page (p.)
front pageorganisation
+77%+62%+57%+24%
PD
-1.0
1.0
0.0
De In CmCn
0.6
0.0
137 sites
Focused task
news (tech)news (life)
supportmail
+66%+66%+65%+64%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
De In CmCn
60%
0%
60%
0%
142 sites
Rapid multitasking
news (enter.)knowledge
comparisonservice
+64%+63%+62%+59%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
De In CmCn
60%
0%
Single-task-oriented browsing Multitask-oriented browsing
ATTACHMENT: Inter-‐site engagement
Inter-‐site engagement 74
Data Dataset, network and site categories.
InteracCon data • August 2013 to July 2014 • 53M sessions
Provider network G=(N, E, λ) N: 155 Yahoo sites
from five countries E: User traffic λ(e): Traffic volume (#Clicks)
Site categories • 155 sites from 5 countries • 5 categories
Cat. %Sites Description
35%19%13%23%10%
newsserviceleisureproviderfront page
mail, calendar, etc.social media, games, etc.account settings, help, etc.front pages, site maps
servicefront page news providerleisure
Inter-‐site engagement 75
Inter-‐site Engagement Metrics Flow accounts for the extent users navigate between sites.
Traffic Flow The metric is defined as follows:
InterpretaCon High Flow à High inter-‐site engagement Users navigate oken between the sites of the network.
Flow =wi, ji, j∑vii∑
#Clicks between node i and j #Visits on node i
wi, j
vi
Flow = 30/60 = 0.5
105
2020
20
10 5
11
2020
20
1 1
Flow = 4/60 = 0.07
Inter-‐site engagement 76
Inter-‐site Engagement Metrics Density describes the connecLvity of the network.
Density We use the density measure of [Wasserman]:
InterpretaCon High Density à High inter-‐site engagement Users navigate between many different sites (inter-‐site engagement is highly diverse).
[Wasserman] S. Wasserman. Social network analysis: Methods and applicaHons, 1994.
Density = #Edges#Possible_Edges
Density = 4/6 = 0.7
Flow = 2/6 = 0.3
Inter-‐site engagement 77
Inter-‐site Engagement Metrics Reciprocity measures the homogeneity of traffic between two sites.
Reciprocity We use the reciprocity measure of [SquarLni]:
InterpretaCon High Reciprocity à High inter-‐site engagement Users navigate between two sites in both direcLons (inter-‐site engagement is highly homogenious).
[SquarHni] T. SquarHni, F. Picciolo, F. RuzzenenH, and D. Garlaschelli. Reciprocity of weighted networks. Nature: ScienHfic reports, 2013.
#Clicks between node i and j wi, j
RP =min[wi, j,wj,i ]i< j∑
wi, ji≠ j∑
110 5
20
1
Reciprocity = 15/50 = 0.3
Reciprocity = 2/37 = 0.05
1010 5
20
5
Inter-‐site engagement 78
Inter-‐site Engagement Metrics Entry/ExitDisp measures how the traffic to/from the network is distributed over the sites.
Entry disparity and exit disparity We use the group degree measure of [Freeman] and adapt it as follows:
InterpretaCon High Entry/ExitDisp à Low inter-‐site engagement The network is more vulnerable to outages, because only few sites are used to enter (leave) the network.
EntryDisp =(ginmax − g
ini )i∑
| N |• ginii∑
[Freeman] L. C Freeman. Centrality in social networks conceptual clarificaHon. Social networks, 1979.
Number of visits that started at node ni (user entered the network) Maximum value of gin Number of nodes | N |
giin
ginmax
EntryDisp = 20/3�40 = 0.17
20 10
10
40 5
5
EntryDisp = 70/3�50 = 0.47
Inter-‐site engagement 79
EvaluaCon: Network-‐level CorrelaLons between inter-‐site and network engagement metrics.
[IS
] D
en
sity
[IS
] R
ecip
rocity
[IS
] E
ntr
yD
isp
arity
[IS
] E
xitD
isp
arity
[PO
P]
#S
essio
ns
[AC
T]
Dw
ellT
ime
S
[AC
T]
#S
ite
s
Flow [IS] - 0.15 0.23 0.30 - 0.35 0.65
Density [IS] 0.48 -0.61 -0.60 0.92 -0.45 -0.25
Reciprocity [IS] -0.38 -0.32 0.42 - 0.25
EntryDisparity [IS] 0.84 -0.54 0.33 -
ExitDisparity [IS] -0.55 0.38 0.20
0-1 1-0.5 0.5Spearman’s rho with p-value < 0.01
('-' insignificant correlations)
Density and #Sessions The more users are
visiCng the network, the more diverse is the inter-‐
site engagement.
Entry-‐ and ExitDisparity Volume of in-‐ and out-‐going traffic of the nodes depend on each other.
Flow and #Sites The more sites are visited
during a session, the higher the flow of traffic.
Inter-‐site engagement 80
EvaluaCon: Node-‐level CorrelaLons between inter-‐site and site engagement metrics.
[IS
] D
ow
nstr
eam
[IS
] E
ntr
yP
rob
[IS
] E
xitP
rob
[PO
P] #S
essio
ns
[A
CT
] D
wellT
imeS
[M
T] #V
isits
[MT
] C
um
Act
PageRank [IS] 0.30 -0.08 -0.10 0.85 0.06 0.08 0.31
Downstream [IS] -0.27 -0.22 0.17 0.04 0.02 -0.02
EntryProb [IS] 0.79 0.12 -0.19 0.13 0.35
ExitProb [IS] 0.08 -0.18 0.18 0.32
0-1 1-0.5 0.5Spearman’s rho with p-value < 0.01
('-' insignificant correlations)
PageRank and #Sessions Popular sites in the
provider network, are also visited frequently when browsing through
the network.
Entry-‐ and ExitProb Nodes that are used to enter the network are also frequently used to
exit the network.
Inter-‐site engagement 81
Comparing Provider Networks
Country2
Country1
Country3
Country4
Country5
Flow Reciprocity EntryDisparityDensity DwellTimeBars from left to right:
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
Inter-‐site engagement 82
PaPerns of Inter-‐site Engagement C
ateg
orie
sE
nga
gem
ent
PageRank EntryProb DwellTimeDownstream PD - Probability difference
46 sites
Focused eng.
front pageservice
providerleisure
news
+80%+63%-100%-100%-100%
PD
23 sites
Traffic hub
46 sites
Supporter
40 sites
Shared eng.
CumActBars from left to right:
-1.0
1.0
0.0
providerservice
newsfront page
leisure
+31%+19%
-2%-10%
-100%
PDleisure
providerservice
newsfront page
+67%+48%-21%-94%
-100%
PDnews
leisureprovider
front pageservice
+63%-61%
-100%-100%-100%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
-1.0
1.0
0.0
ATTACHMENT: NaCve AdverCsing
NaLve AdverLsing 84
Effect on User Engagement
0%
200%
400%
600%
short ad clicks long ad clicks
ad c
lick
diffe
renc
e
��
��
���
���
short ad clicks long ad clicks
clic
ks p
er d
ay d
iffer
ence
PosiLve experience has a strong effect on users clicking on ads again, and a small effect on user engagement with the stream.
NaLve AdverLsing 85
Mobile vs. Desktop
Ad post-‐click experience between mobile and desktop differs. For dwell Lme we obtain rho = 0.50; this value is even smaller for bounce rate with rho = 0.23.
0.00
0.05
0.10
0.15
�� �� �� �� �� �� ���dwell time difference
p(dw
ell t
ime
diffe
renc
e)
higher on mobilehigher on desktop
0.00
0.05
0.10
0.15
��� �� ��� ��� ���� ����bounce rate difference
p(bo
unce
rate
diff
eren
ce) higher on mobilehigher on desktop
NaLve AdverLsing 86
Mobile OpCmised Landing Pages
Dwell Cme: The distribuLon is very similar for both groups. Bounce rate: Decreases by 6.9% (median decreases by 30.4%) for Opt landing pages but increases by 13.4% (median decreases by 11.5%) for Npt landing pages.
not mobile optimized mobile optimized
0.0
0.1
0.2
0.3
�� �� �� ����� �� �� ���dwell time difference
p(dw
ell t
ime
diffe
renc
e)
higher on mobilehigher on desktop
mobile opt.
not mobile opt.
0.0
0.1
0.2
��� �� ��� ��� ���� ����bounce rate difference
p(bo
unce
rate
diff
eren
ce) higher on mobilehigher on desktop
mobile opt.
not mobile opt.
ATTACHMENT: Wikipedia
Wikipedia 88
Wikipedia Research Literature review by Okoli et al.: The people’s encyclopedia under the gaze of the sages: A systemaLc review of scholarly research on wikipedia.
Wikipedia 89
Reading Preferences
Popularitylow high
Art
icle
Len
gth
shor
tlo
ng borderline casesII I
III IV
Jeanne Tsai
Douglas Adams
LuisPalomino
Anne Stears
PeterEhrlich
AlecMango
Stephen D.Lovejoy
1st Dalai Lama
Dexter Jackson (safety)
Katie GreenBrittanyBorman
Anthony Anenih
Ronnie Bird
Jan Anderson(scientist)
FitchRobertson
Sean Bennett
For 4.2% (group IV) of the articles editing activity is low, but reading activity is high.!
Wikipedia 90
Reading PaPerns A
rtic
le
topic
Rea
din
gbeh
avio
r
ArticleViewsa
SessionArticlesa
Popularitya
ReadingTimea
CA - Percentage in topic
4,826 articles11,579 behavior vectors
sportspersonmusician
media pers.
28%26%23%
CA
Exploration
artist/writerhistorical fig.
polit./businessp.
43%41%37%
CA
5,278 articles10,605 behavior vectors
Focus
3,876 articles14,267 behavior vectors
historical fig.criminal/victim
musican
42%38%38%
CA
Trending
5,684 articles13,470 behavior vectors
media pers.sportsperson
musician
27%27%19%
CA
Passing
28K [16K,51K]
11 [5,23]
7.7%
38K [21K,69K]
20 [9,41]
16.9%
26K [15K,45K]
10 [5,21]
10.5%
16K [10K,27K]
8 [3,18]
5.1%
ArtLen
#Edits
%HQA
#Edits - Number of edits
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
%HQA - Percentage of high quality articlesArtLen - Article length
Wikipedia 91
Reading PaPerns over Time
Stability • 30% of the arLcles are popular in 1 month • 10% are popular over the whole 13-‐months • Almost all arLcles have one reading pa;ern
half of their life Lme
TransiCons • TransiLons are temporary – arLcles move
temporarily to another cluster • High reciprocity – similar number of
transiLons in both direcLons • “Focus” cluster is isolated -‐ ArLcles in that
cluster are the most stable ones • Strong connecLon between the “Passing”,
“ExploraLon”, and “Trending” clusters – many arLcles adopt all three pa;erns
ATTACHMENT: Yahoo
93
Upstream Traffic
TeleportaCon Social media / News Search / Ext-‐Yahoo Users engage (quickly)
to many sites. Users conHnue with same acHvity inside the provider network.
Users visit site they are interested in, perform a quick task, and leave.
Network DwellTime -‐-‐ ++ -‐-‐
Traffic DistribuCon ++ -‐-‐ -‐-‐
Entry Disparity -‐-‐
Yahoo
Users engage differently depending on where they are coming from.
94
Network Effect PaPern
Yahoo
Sites change their popularity in the same way. Ac>vity (dwell >me) on a site depends more on the site itself,
but there are some nega>ve dependencies.
Pat
tern
exam
ple
s
41 patterns
Simple star-like
6 patterns
Complex star-like
1 pattern
Cluster-like
3.00 [3.00,4.00]
0.67 [0.00,0.89]
0 [0,0]
8.00 [7.00,18.00]
0.76 [0.56,0.84]
0 [0,0]
52
0.91
0.51
N
Recip
Trans
N - Number of nodes Recip - Reciprocity Trans - Transitivityservicefront page news providerleisure
(4) (5) (6)(1) (2) (3)
95
Hyperlink Performance
Yahoo
0%
25%
50%
75%
100%
Onïs
ite
links
front page providerservice news leisure
Inte
rïsi
te lin
ks
front page providerservice news leisure
0%
20%
40%
60%
80%
Exte
rnal
lin
ks
front page providerservice news leisure
20%
40%
60%
(a) PageRank and downstream.
TrafficPageRank Downstream
HyperlinksPageRank 0.54 -Downstream - -
(b) On-site, inter-site, and external.
TrafficOn-site Inter-site External
HyperlinksOn-site 0.54 -0.45 -0.38Inter-site -0.40 0.50 -External - - 0.39
ATTACHMENT: Online News
Online news 97
Focused versus Non-‐focused Sessions Internal
Non-focused sessionsFocused sessions●
(b) Duration
(d) p(focused session)
(a) %Sessions
(f) Flow
25
15
5
60%
20%
0.6
0.2
0.2
0.1
2 3 4 5 6 7 7 2 3 4 5 6 7 7 2 3 4 5 6 7 7#Articles #Articles #Articles
●
●
●
●●
●
●
(c) #Providers2.5
2.0
1.5
●
●
●●
●●
●
●●
●●
●●●● ●●
●●
●●(e) EntryDisparity
0.5
0.3
0.1 ●
●
●
●
●●
●
When users focus on a news story, they spend more >me reading the ar>cles and the inter-‐site engagement between providers is higher.
Online news 98
Hyperlink Performance
Number of Inline Links • <10 links may be wasLng an opportunity • 10-‐29 links does not result in more clicks • >29 links may harm the user experience PosiCon of Inline Links • 30% at the end, 16% at the beginning, 46%
are distributed within the arLcle text. • Performance of links located at the
beginning of the text is very low (-‐28%) • Best performance is achieved with links at
the end of the arLcle text (+35%)
Link popularity● Link performance
Position in article textLin
k po
pula
rity
[0.0,0.1[ [0.3,0.4[ [0.6,0.7[ [0.9,1.0]
10%
20%
30%
-0.2
0.0
0.2
Lin
k p
erforman
ce
●●
●●
●●●●
●● ●● ●● ●●●●
●●
●●
●●●● ●● ●● ●● ●● ●● ●●
●●
Number of inline links in article
Clic
ks p
er li
nk
0.0
0.2
0.4
0.6
[0,2] [9,11] [18,20] [27,29] [36,38]
Number of inline links in article
Num
ber of
clic
ks
[0,2] [9,11] [18,20] [27,29] [36,38]
2.5
5.0
7.5
Online news 99
Effect on User Engagement
Internal Focused Short-‐term: Only 3 (out of 50) providers have their corresponding average dwell Lme lower for the story-‐focused provider sessions. The average increase in dwell Lme from non-‐story-‐focused to story-‐focused provider sessions is 50%. Long-‐term: For 78% of the providers, we find that there are more users that return earlier aker they have a story-‐focused provider session.
Internal
News provider
Dw
ell tim
e per
ses
sion
Non-focused Focused Ext-focused
News providerp(a
bse
nce
12
h)
Non-focused Focused Ext-focused
Online news 100
Effect on User Engagement
External Focused Short-‐term: We do not observe an effect on the dwell Lme (neither posiLve nor negaLve). The average increase is only 5.5%, and based on the K-‐S test we cannot confirm that the distribuLons are different (p-‐value=0.36). Long-‐term: For 70% of these news sites, the probability that users return within the following 12 hours increases (the average increase is 76%).
External
News provider
Dw
ell tim
e per
ses
sion
Non-focused Focused Ext-focused
News providerp(a
bse
nce
12
h)
Non-focused Focused Ext-focused
Online news 101
Discovering Story-‐related Content in TwiPer
Top Related