From Site to Inter-site User Engagement

photo credit donsolo, CC BY-‐NC-‐SA 2.0

From Site to Inter-‐site User Engagement Jane;e Lehmann

Barcelona, February 26, 2015

Advisors: Ricardo Baeza-‐Yates Co-‐Advisor: Mounia Lalmas

•  User engagement is a quality of the user experience that emphasizes the posiLve aspects of interacLon with a website – in parLcular the fact of being capLvated by the website.

•  In-‐the-‐moment engagement Users stay on a website over a long Lme.

•  Long-‐term engagement Users come back frequently and over a long-‐term.

IntroducLon 2

User Engagement DefiniLon

Successful websites are not just used, they are engaged with.

User Engagement Measuring

3 IntroducLon

Before we can design engaging websites, it is crucial that we are able to measure engagement.

“If you can measure it, you can improve it.” Sir William Thomson

Analysis/Planning

Design Changes Measuring

Main Research Goals

4 IntroducLon

Primary goal Can we define new engagement metrics that Measuring enhance our understanding of engagement?

Secondary goal Can we idenLfy ways to influence engagement?Analysis/Planning

Analysis/Planning


IntroducLon 5

Analysis/Planning

Design Changes Measuring Online mulLtasking

Inter-‐site engagement

Site engagement

Effect of providing off-‐site content Effect of hyperlinks

IntroducLon 6

Analysis/Planning


Site engagement

Measuring Engagement InteracLon data

7 Site engagement

Data Browsing events provided by Yahoo toolbar (client-‐side).

Engagement Analysing the data using online behaviour metrics.

Online session:

Visit on Yahoo News

Site engagement 8

Measuring Engagement Online behaviour metrics

K. Rodden, H. Hutchinson, X. Fu. Measuring the user experience on a large scale: User-‐centered metrics for web applicaHons. CHI, 2010. E. Peterson, J. Carrabis. Measuring the immeasurable: Visitor engagement. Web AnalyHcs DemysHfied, 2008. B. Haven, S. ViWal. Measuring engagement. Forrester Research, 2008. B. Weischedel and E. Huizingh. Website opHmizaHon with web metrics: A case study. Conference on Electronic commerce, 2006.

Site engagement 9

Measuring Engagement Online behaviour metrics

Popularity

#Users Number of users.

#Visits Number of visits.

#Clicks Number of clicks.

AcCvity (within a visit) In-‐the-‐moment engagement

PageViews Avg. number of page views per visit.

DwellTime Avg. Lme on site per visit.

Loyalty (across visits) Long-‐term engagement

ReturnRate Number of Lmes a user visited the site.

AcLveDays Number of days a user visited the site.

Site engagement 10

Measuring Engagement Differences in engagement

ComScore, Alexa, GoogleAnalyHcs,…

Shopping Users do not come frequently, but stay long

Games Not many users, but they stay long

News Users come frequently and stay long

Measuring Engagement Problem

11 Site engagement

Isolated view: The metrics focus on engagement with a single site.

RelaLonships to other sites are not considered.

IntroducLon 12

Analysis/Planning


Site engagement

Online mulLtasking 13

MoCvaCon In-‐the-‐moment engagement


What web analyCcs think we do…

1 visit with 4 page views.


MoCvaCon In-‐the-‐moment engagement


… and what we really do:

3 visit with on average 1.3 page views.


MoCvaCon Online mulLtasking.

Problem

•  Engagement metrics do not capture such behaviour.

•  Measuring acLvity on a site can lead to incorrect conclusions.

Online mulCtasking Users visit several sites and switch between them

during an online session, to perform related or totally unrelated tasks.

Research QuesCon

16 Online mulLtasking

How can we measure engagement by accounLng for user mulLtasking behaviour?

Analysis/Planning


Extent of mulCtasking •  10.2 disLnct sites, 2 visits per site.

Absence Cme •  50% of sites are revisited aker < 1min.

InterrupHon of a task

•  There are revisits aker long breaks. Performing a new task


Online MulCtasking CharacterisLcs

0.00

0.25

0.50

0.75

1.00

10ï2

10ï1

100

101

102

Cum

ula

tive

pro

bab

ility

Absence time [min]

news (finance)

news (tech)

social media

mail

2.09

1.76

2.28

2.09

#Visits Absencetime [min]

3.85

3.95

4.47

6.86

Absence time: Time between two visits

AcCvity paPerns

•  Four types: Decreasing, increasing, constant, complex. •  Successive visits can belong together (i.e. to the same task). •  Complex cases refer to no specific pa;ern or repeated pa;ern.


Online MulCtasking CharacterisLcs

1 2 3 4ith visit on site




Pro

por

tion

of to

tal

dw

ell tim

e on

site

0.23

0.28

0.33 p-value = 0.09m = -0.01

p-value = 0.07m = -0.02

p-value = 0.79m = 0.00

news (finance) sitesmail sites social media sites news (tech) sites

decreasing attention increasing attention constant attention complex attention


Measuring Engagement Online mulLtasking metrics

Extent of mulCtasking

SessSites Total number of sites accessed (#tasks).

SessVisits Number of visits to site (site switching).

Absence Cme

CumAct Aggregates the dwell Lmes of the visits with accounLng for the Lme between the visits.

AcCvity paPern

A;Shik A;Range Describe the four cases of a;enLon shiks.

20

CASE STUDY: MulCtasking PaPerns •  ObjecCve: Analyse mulLtasking acLvity on sites;

idenLfy mulLtasking pa;erns (clustering).

•  Metrics: Site DwellTime, MulLtasking metrics.

•  Data: July 2012, 2.5M users, 760 sites (shopping, news, search, etc.).

21

Case Study: MulCtasking PaPerns Results

No mulCtasking MulCtasking

Quick Focused Rapid ConCnuous Recurring Checking weather

Reading mails

Following link to off-‐site content

Purchasing an item

Performing search

Site DwellTime -‐-‐ ++ ++ ++ -‐-‐

Extent of mulCtasking -‐-‐ -‐-‐ ++ ++ ++

Absence Cme -‐-‐ ++ ++

ImplicaCons Provide

interesHng off-‐site content

Shopping takes more than

one visit

Support user by finishing tasks quickly

Online mulLtasking

-- low value ++ high value

22

Case Study: MulCtasking PaPerns Results


Quick Focused Rapid ConCnuous Recurring Checking weather

Reading mails

Following link to off-‐site content

Purchasing an item

Performing search

Site DwellTime -‐-‐ ++ ++ ++ -‐-‐

Extent of mulCtasking -‐-‐ -‐-‐ ++ ++ ++

Absence Cme -‐-‐ ++ ++

AcCvity paPern

Online mulLtasking

De In CmCn

60%

0%De In CmCn

60%

0%De In CmCn

60%

0%

Activity pattern: De – Decreasing In – Increasing Cn – Constant Cm - Complex -- low value ++ high value

23

CASE STUDY: Wikipedia (on-‐site mulCtasking) •  ObjecCve: Analyse reading acLvity on Wikipedia

arLcles; idenLfy reading pa;erns (clustering).

•  Metrics: ArLcle DwellTime, #ArLcles in session, #Views to focal arLcle.

•  Data: Sep 2011 – Sep 2012, 500K users, 10K biography arLcles.

24

Case Study: Wikipedia Approach

Online mulLtasking

Users’ reading behaviour on an Wikipedia arCcle

ArLcle DwellTime How much Lme do users spend on an arLcle? #ArLcles in session Do users view also other arLcles during an

online session? #Views on focal arLcle How oken do users view the arLcle?

25

Case Study: Wikipedia Results


Focus ExploraCon Passing Focus is on focal arHcle

Exploring topic around the focal arHcle

Exploring topic and pass through the focal arHcle

ArCcle DwellTime ++ -‐-‐

#ArCcles in session -‐-‐ ++ ++

#Views to focal arCcle ++ -‐-‐

ImplicaCons Content quality is important

Links to addiHonal content are important

ArHcles might need to be extended

Online mulLtasking

On-‐site mulCtasking

•  MulLtasking between news arLcles of a provider. •  MulLtasking between different tasks on a social media

site (e.g. sharing, chapng, updaLng profile). •  … Inter-‐site mulCtasking

•  MulLtasking when purchasing items online (comparing offers, product reviews, search, etc.)

•  …


Further Use Cases

Take Aways

•  AccounLng for mulLtasking leads to a be;er understanding on how users engage with sites.

•  Leaving a site does not necessarily

entail less engagement, as users oken return to the site later on.

Publications J. Lehmann, M. Lalmas, G. Dupret, and R. Baeza-Yates. Online multitasking and user engagement. CIKM 2013.

J. Lehmann, C. Müller-Birn, D. Laniado, M. Lalmas, and A. Kaltenbrunner. Reader preferences and behavior on Wikipedia. HT 2014, Ted Nelson Newcomer Paper Award.

J. Lehmann, C. Müller-Birn, D. Laniado, M. Lalmas, and A. Kaltenbrunner. What and how users read: Transforming reading behavior into valuable feedback for the Wikipedia community. Wikimania 2014.


IntroducLon 28

Analysis/Planning



Site engagement

Inter-‐site engagement 29

MoCvaCon Large online service providers


Engagement Popularity: #Users, #Visits, … AcLvity: DwellTime, PageViews, … Loyalty: ReturnRate, AcLveDays, …



frontpage

tv sports

shopping

autos

search

daLng

jobs

news

shine

groups

maps local

health answer

weather

games

mail

omg

homes travel

flickr

finance

Large online service providers (AOL, Google, Yahoo, etc.) have not only one site, but many sites.

tumblr



frontpage

tv sports

shopping

autos

search

daLng

jobs

news

shine

groups

maps local

health answer

weather

games

mail

omg

homes travel

flickr

finance

Providers want that users engage with many of their sites.

tumblr


MoCvaCon Online mulLtasking

Problem

•  Engagement metrics do not measure engagement across sites. •  How to adapt them is not obvious.

Inter-‐site engagement Users visit sites that belong to the

same network of sites.

Research QuesCon

33 Inter-‐site engagement

How can we measure engagement by also considering the

relaLonships between sites?

Analysis/Planning



Traffic Networks Modelling

We model sites (nodes) and user traffic (edges) between them as a network. Provider network G=(N, E, λ)

N: Sites E: User traffic λ(e): Traffic volume (#Clicks)

4 clicks

2 clicks

50 clicks 10 clicks


Measuring Engagement Inter-‐site engagement metrics: Network-‐level

Traffic distribuCon

Flow Extent to which users navigate between sites.

Density1 Diversity of inter-‐site engagement.

Reciprocity2 Homogeneity of traffic between sites.

External traffic

EntryDisparity Variability of in-‐going traffic to the network.

ExitDisparity Variability of out-‐going traffic from the network.

[1] S. Wasserman. Social network analysis: Methods and applicaHons, 1994. [2] T. SquarHni, F. Picciolo, F. RuzzenenH, and D. Garlaschelli. Reciprocity of weighted networks. Nature: ScienHfic reports, 2013.


Measuring Engagement Inter-‐site engagement metrics: Node-‐level

Traffic distribuCon

PageRank1 Probability that a user will visit the site.

Downstream Probability that a user will conLnue browsing to other sites.

External traffic

EntryProb Probability that a user enters the network in this site.

ExitProb Probability that a user leaves the network in this site.

[1] L. Page, S. Brin, R. Motwani, T. Winograd. The pagerank citaHon ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.

37

CASE STUDY: Yahoo Provider Networks •  ObjecCve: Compare networks; characterise the sites

in a network.

•  Metrics: Network DwellTime, Site DwellTime, Inter-‐site engagement metrics.

•  Data: February 2014, 3.2M clicks/network, 4 country-‐based networks, 31 sites per network.

38

Case Study: Yahoo Comparing provider networks

Network 1 Network 2 Network 3 Network 4 High

engaging Users engage quickly

with many sites Users engage to a subset of sites

Low engaging

Network DwellTime ++ -‐-‐ ++ -‐-‐

Traffic DistribuCon ++ ++ Flow ++

Density -‐-‐ -‐-‐

Entry Disparity ++ -‐-‐ ++

ImplicaCons The network is performing

well.

This should be looked into.

MoHvate users to visit other sites.

This should be looked into.



39

Case Study: Yahoo Sites within a provider network

Traffic Hub Supporter Focused Engagement

Shared Engagement

Search, front pages Support, services Leisure, support News, leisure

Site DwellTime -‐-‐ -‐-‐ ++ ++

Traffic DistribuCon ++ -‐-‐ -‐-‐ ++

Entry Probability ++ -‐-‐ ++ -‐-‐

ImplicaCons The sites

forward traffic to other sites.

Users visit sites for specific needs and support.

MoHvate users to visit other sites.

The sites are performing

well.



Comparing networks

•  Device, Lme, upstream traffic, user. •  SimulaLons (effect of adding/removing sites). •  … Network types

•  Network of pages (e.g. compare language-‐based Wikipedia networks)

•  Network of sites from different providers (e.g. shopping sites, news providers)

•  …


Further Use Cases

Take Aways

•  Inter-‐site engagement allows for a more comprehensive look at user engagement by also considering the relaLonships between sites.

•  Deeply engaged users do not only

engage with one site, but with many sites in a network.

Publications J. Lehmann, M. Lalmas, and R. Baeza-Yates. Measuring Inter-Site Engagement. Handbook of Statistics, Elsevier, 2015. To appear.

J. Lehmann, M. Lalmas, R. Baeza-Yates, and E. Yom-Tov. Networked User Engagement. ACM Workshop on User engagement optimization at CIKM, 2013.

J. Lehmann, M. Lalmas, and R. Baeza-Yates. Temporal Variations in Networked User Engagement. TNETS Satellite at ECCS, 2013.

Some of the metrics were employed to characterise online news reading across news sites: J. Lehmann, C. Castillo, M. Lalmas, and R. Baeza-Yates. Story-Focused Reading in Online News. Submitted for publication.


IntroducLon 42

Analysis/Planning



Site engagement

Effect of providing off-‐site content

43

CASE STUDY: Online News •  Hypothesis: It may be beneficial (long-‐term) to

enLce users to leave a site by offering interesLng off-‐site content.

•  Data: October 2013, 57K users, 50 news sites, 26K news arLcles.

Types of reading sessions

No click Did not follow a related link.

Off-‐site click Followed a related link to content on another site.

Effect on engagement

Short-‐term Dwell Lme per reading session.

Long-‐term Probability that user starts next reading session within the next 12h.

44

Case Study: Online News

Related off-‐site content

Approach


Providing links to related off-‐site content has a no short-‐term effect, but a posiCve long-‐term effect.

45

Case Study: Online News Results


News provider

Dw

ell tim

e per

ses

sion

News provider

p(a

bse

nce

12

h)

No Click Off-site click

IntroducLon 46

Analysis/Planning



Site engagement


47

CASE STUDY: Yahoo Provider Network •  Hypothesis: We can use hyperlinks to influence

inter-‐site engagement in a provider network.

•  Data: February 2014, 235M clicks, Yahoo US network, 73 sites.

Hyperlink vs. traffic network On-‐site Links/Traffic to pages

within the same site. Inter-‐site Links/Traffic to pages to

other sites in the network.

External Links/Traffic to

somewhere else on the Web.

48

Case Study: Yahoo Approach

frontpage

sports

search

news

shine

groups

answer

weather

mail

omg

homes

flickr

Effect of hyperlinks

Hyperlinks can be used to influence site and inter-‐site engagement in a provider network.

However, both types of engagement influence each other.

49

Case Study: Yahoo Results

Effect of hyperlinks

TrafficOn-site Inter-site External

HyperlinksOn-siteInter-siteExternal

0.54-0.40

-

-0.450.50

-

-0.38-

0.39

IntroducLon 50

Analysis/Planning



Site engagement


Two new perspecHves for measuring engagement which consider the relaLonships between sites. Online mulCtasking Accounts for user mulLtasking behaviour. Inter-‐site engagement Accounts for the traffic between sites.

ContribuLons and future work 51

Main ContribuCons Measuring engagement

Analysis/Planning


AccounLng for the new perspecLves when influencing engagement. Online news Providing related off-‐site content influences long-‐term engagement. Provider network Hyperlinks affect site and inter-‐site engagement, but both influence each other.


Main ContribuCons Analysis/Planning

Analysis/Planning


Wikipedia Providing informaLon about readers’ engagement to the editor community.

Yahoo Using inter-‐site engagement metrics to make informed decisions about design changes (hyperlinks).

Spiegel Online Measuring and improving engagement by providing interesLng off-‐site content.


What next? Ongoing and future work

Analysis/Planning


photo credit donsolo, CC BY-‐NC-‐SA 2.0

Thank you!

Jane;e Lehmann

Barcelona, February 26, 2015 [email protected]

Acknowledgements Ricardo Baeza-‐Yates Mounia Lalmas Claudia Müller-‐Birn Carlos CasLllo David Laniado Andreas Kaltenbrunner Elad Yom-‐Tov Georges Dupret Guy Shaked Fabrizio Silvestri Gabriele Tolomei Ethan Zuckerman John Agapiou Andy Haines Diego Sáez-‐Trumper Hemant Purohit Noora Al Emadi Mohammed El-‐Haddad Nasir Khan

•  Mounia Lalmas and Janette Lehmann. “Models of User Engagement”. In H. L. O’Brien and M. Lalmas (Eds.), Why Engagement Matters: Cross-disciplinary Perspectives and Innovations on User Engagement with Digital Media. Springer, 2015, in progress.

•  Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. “Models of user engagement.” International Conference on User Modeling, Adaptation, and Personalization (UMAP 2012), pp. 164-175, Montreal, Canada, July, 2012.

•  Janette Lehmann, Mounia Lalmas, Georges Dupret, and Ricardo Baeza-Yates. “Online multitasking and user engagement.” ACM International Conference on Information and Knowledge Management (CIKM 2013), pp. 519-528, San Francisco, United States, October, 2013.

•  Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Measuring Inter-Site Engagement.”. In V. Govindaraju, V. V. Raghavan, and C. R. Rao (Eds.), Handbook of Statistics, Elsevier, 2015.

•  Janette Lehmann, Mounia Lalmas, Ricardo Baeza-Yates, and Elad Yom-Tov. “Networked User Engagement.”, ACM Workshop on User engagement optimization at CIKM, pp. 7-10, San Francisco, United States, October, 2013.

•  Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Temporal Variations in Networked User Engagement.”, TNETS Satellite at European Conference on Complex Systems (ECCS), Barcelona, Spain, September, 2013.

•  Mounia Lalmas, Janette Lehmann, Guy Shaked, Fabrizio Silvestri, and Gabriele Tolomei. “Measuring Post-click User Experience with Mobile Native Advertising on Streams.”, submitted for publication.

•  Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “Reader preferences and behavior on Wikipedia.”, ACM International Conference on Hypertext and Social Media (HT 2014), pp. 88-97, Santiago, Chile, September, 2014, Ted Nelson Newcomer Paper Award.

•  Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “What and how users read: Transforming reading behavior into valuable feedback for the Wikipedia community.”, Presentation at Wikimania, London, UK, August, 2014.

•  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ricardo Baeza-Yates. “Story-Focused Reading in Online News.”, submitted for publication.

•  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Transient News Crowds in Social Media.” International AAAI Conference on Weblogs and Social Media (ICWSM 2013), Boston, USA, July, 2013.

•  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Finding News Curators in Twitter.” ACM International Conference on World Wide Web Companion (WWW 2013 Companion), 863-870, Rio de Janeiro, Brazil, May, 2013.

55

PublicaCons

User engagement

•  Mounia Lalmas, Heather L O’Brien, and Elad Yom-Tov. Measuring user engagement. Synthesis Lectures on Sample Series #1. Morgan and cLaypool publishers, 2014.

•  Heather L O’Brien and Elaine G Toms. What is user engagement? a conceptual framework for defining user engagement with technology. American Society for Information Science and Technology (ASIS&T), 59(6):938–955, 2008.

•  Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. Towards a science of user engagement (position paper). In Proc. Workshop on User Modelling for Web Applications, WSDM, 2011.

•  Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010.

Online behaviour metrics

•  Brian Haven and Suresh Vittal. Measuring engagement. Forrester Research, 2008.

•  Eric T Peterson and Joseph Carrabis. Measuring the immeasurable: Visitor engagement. Web Analytics Demystified, 2008.

•  Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010.

•  Georges Dupret and Mounia Lalmas. Absence time and user engagement: evaluating ranking functions. In Proc. Conference on Web Search and Data Mining, WSDM, pages 173–182. ACM, 2013.

•  Randolph E Bucklin and Catarina Sismeiro. A model of web site browsing behavior estimated on clickstream data. Journal of Marketing Research, 40(3):249–267, 2003.

•  Birgit Weischedel and Eelko KRE Huizingh. Website optimization with web metrics: a case study. In Proc. Conference on Electronic commerce: The new e-commerce: innovations for conquering current barriers, obstacles and limitations to

conducting successful business on the internet, pages 463–470. ACM, 2006.

•  Peifeng Yin, Ping Luo, Wang-Chien Lee, and Min Wang. Silence is also evidence: interpreting dwell time for recommendation from psychological perspective. In Proc. Conference on Knowledge Discovery and Data Mining, SIGKDD, pages 989–997. ACM, 2013.

56

Selected References

Online multitasking

•  Qing Wang and Huiyou Chang. Multitasking bar: prototype and evaluation of introducing the task concept into a browser. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 103–112. ACM, 2010.

•  Hartmut Obendorf, Harald Weinreich, Eelco Herder, and Matthias Mayer. Web page revisitation revisited: implications of a long-term click-stream study of browser usage. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 597–606. ACM, 2007.

•  Jeff Huang and Ryen W White. Parallel browsing behavior on the web. In Proc. Conference on Hypertext and Hypermedia, HT, pages 13–18. ACM, 2010.

•  Patrick Dubroy and Ravin Balakrishnan. A study of tabbed browsing among mozilla firefox users. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 673–682. ACM, 2010.

Inter-site engagement

•  Mark EJ Newman. The structure and function of complex networks. SIAM review, 45(2):167–256, 2003. 76, 77, 165

•  Anna Chmiel, Kamila Kowalska, and Janusz A Hołyst. Scaling of human behavior during portal browsing.

•  Mark R Meiss, Filippo Menczer, Santo Fortunato, Alessandro Flammini, and Alessandro Vespignani. Ranking web sites with real user traffic. In Proc. Conference on Web Search and Data Mining, WSDM, pages 65–76. ACM, 2008.

•  Young-Hoon Park and Peter S Fader. Modeling browsing behavior at multiple websites. Marketing Science, 23(3):280–303, 2004.

•  Qiqi Jiang, Chuan-Hoo Tan, and Kwok-Kee Wei. Cross-website navigation behavior and purchase commitment: A pluralistic field research. In Proc. Pacific Asia Conference on Information Systems, PACIS, 2012.

•  Kevin Koidl, Owen Conlan, and Vincent Wade. Cross-site personalization: assisting users in addressing information needs that span independently hosted websites. In Proc. Conference on Hypertext and Hypermedia, HT, pages 66–76. ACM, 2014.

•  The PEW Research Center. Understanding the participatory news consumer. http://www.pewinternet.org/~/media/Files/Reports/ 2010/PIP_Understanding_the_Participatory_News_Consumer. pdf, 2010.

•  Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012.

57

Selected References

Link economy

•  Joseph Turow and Lokman Tsui. The hyperlinked society. The University of Michigan Press, 2008.

•  Juliette De Maeyer. Hyperlinks and journalism: where do they connect? In Proc. Future of Journalism Conference, 2011.

•  Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink

formation in content networks. Management Science, 59(10):2360–2379, 2013.

•  Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012.

•  Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink

formation in content networks. Management Science, 59(10):2360–2379, 2013.

•  Hakan Ceylan, Ioannis Arapakis, Pinar Donmez, and Mounia Lalmas. Automatically embedding newsworthy links to articles. In

Proc. Conference on Information and Knowledge Management, CIKM, pages 1502–1506. ACM, 2012.

Recommendation

•  Richard McCreadie, Craig Macdonald, and Iadh Ounis. News vertical search: when and what to display to users. In Proc.

Conference on Research and Development in Information Retrieval, SIGIR, pages 253–262. ACM, 2013.

•  Samuel Ieong, Mohammad Mahdian, and Sergei Vassilvitskii. Advertising in a stream. In Proc. Conference on World Wide

Web, WWW, pages 29–38. ACM, 2014.

•  Eric Sodomka, Sébastien Lahaie, and Dustin Hillard. A predictive model for advertiser value-per-click in sponsored search. In

Proc. Conference on Information and Knowledge Management, CIKM, pages 1179–1190. ACM, 2013.

•  Narongsak Thongpapanl and Abdul Rehman Ashraf. Enhancing online performance through website content and

personalization. Journal of Computer Information Systems, 52(1):3, 2011.

•  Jian Wang and Yi Zhang. Utilizing marginal net utility for recommendation in e-commerce. In Proc. Conference on Research

and Development in Information Retrieval, SIGIR, pages 1003–1012. ACM, 2011.

•  Joshua Porter. Designing for the social web. Peachpit Press, 2010.

58

Selected References

ATTACHMENT: IntroducCon

•  “In a world full of choices where the fleeCng aPenCon of the user becomes a prime resource, it is essenLal that [...] providers do not just design [websites] but that they design engaging experiences.” [A}ield].

•  In addiLon to uLlitarian factors, such as usability and usefulness, we must consider other factors of interacLng with websites, such as fun, fulfillment, play, and user engagement.

Successful websites are not just used, they are engaged with.

•  In order to design engaging websites, it is crucial to understand what user engagement is and how to measure it.

IntroducLon 60

MoCvaCon Why is it important to engage users?

Methodology InteracLon data, online sessions and site visits.

61 IntroducLon

t0

t1

t2

t3

t4

t5

t6

t7

sessionend

sessionstart

time

Online session

Browsing activity on Wikipedia

https://ie-mg42.mail.yahoo.com

http://en.wikipedia.org/wiki/Freddie˙Mercury

http://www.bbc.com/news/uk-29149115

http://www.bbc.com/news/uk-england-nottinghamshire-29643802

http://en.wikipedia.org/wiki/Star Wars

http://en.wikipedia.org/wiki/Yoda

http://en.wikipedia.org/wiki/Albert˙Einstein

https://www.facebook.com/janette.lehmann.5

t0

t1

t2

t3

t4

t5

t6

t7

bc0

bc0

bc0

bc0

bc0

bc0

bc0

bc0

BCookie Timestamp URL

-

-

-

http://www.bbc.com/news/uk-29149115

-

http://en.wikipedia.org/wiki/Star Wars

http://en.wikipedia.org/wiki/Yoda

-

ReferrerURL

Interaction data

Page view on Wikipedia Page view on other site

IntroducLon 62

Thesis structure

Metrics that account for sitepopularity, activity and loyalty

AdvertisingChapter 7

Site engagement

How users experience ads on desktop and

mobile devices?

Does ad qualityaffect the engagement

with the publisher?

How can we identifyhigh quality ads?

Site engagementChapter 4

MultitaskingChapter 5

Inter-site engagementChapter 6

Metrics that account for traffic between sites

Metrics that account for user multitasking behaviour

(II

I+IV

) A

pplica

tion

s (

II)

Fund.

WikipediaChapter 8

Site engagementand multitasking

How users readarticles

in Wikipedia?

Does the activityof editors align with theengagement of readers?

How can readers bevaluable for editors?

YahooChapter 9


How users engage with a providernetwork of sites?

Does the hyperlink structure affect site and inter-site engagement?

Online newsChapter 10+11


How users readstories across

news providers?

Do hyperlinks torelated content influenceprovider engagement?

How can we automaticallydetect related content?

Characterising user engagement Comparing site characteristics and user engagement Applications to impact user engagement

ATTACHMENT: Site engagement

0-1 1-0.5 0.5Kendall’s tau with p-value < 0.05

('-' insignificant correlations)

Site engagement 64

EvaluaCon CorrelaLons between engagement metrics.

High correlaCons within metric groups.

Low correlaCons between metric groups.

[PO

P]

#U

se

rs

[PO

P]

#V

isits

[PO

P]

#C

licks

[AC

T]

Pa

ge

Vie

wsV

[AC

T]

Dw

ellT

ime

V

[LO

Y]

Active

Da

ys

[LO

Y]

Re

turn

Ra

te

#Users [POP] 0.82 0.75 - - 0.43 0.34

#Visits [POP] 0.82 0.85 - - 0.60 0.52

#Clicks [POP] 0.75 0.85 0.16 0.18 0.59 0.51

PageViewsV [ACT] - - 0.16 0.33 - -

DwellTimeV [ACT] - - 0.18 0.33 - -

ActiveDays [LOY] 0.43 0.60 0.59 - - 0.79

ReturnRate [LOY] 0.34 0.52 0.51 - - 0.79

0.69

Site engagement 65

PaPerns of Site Engagement Engagement depends on the site at hand.

Games Not many users, but they stay long

Search Users come frequently, but do not stay long

Social media Users come frequently and stay long

Shopping Users do not come frequently, but stay long

News Users come frequently and stay long

Service Users do not come frequently, but stay long

ATTACHMENT: MulCtasking


MoCvaCon

Users switch between sites, to perform related or totally unrelated tasks. Switching between tasks (sites) “…within-‐session page revisits represent the most common form of revisitaLon, covering 73,54% of all revisits.” [Herder] Performing tasks (sites) in parallel using browser tabs “Most of our parLcipants switched tabs more oken than they used the back bu;on.” [Dubroy]

[Herder] E. Herder. CharacterizaHons of user web revisit behavior. WWW Workshop ABIS, 2005. [Dubroy] P. Dubroy, R. Balakrishnan. A study of tabbed browsing among mozilla firefox users. SIGCHI, 2010.


Data Dataset and site categories.

Cat. Subcat. %Sites Description

news

22.1%

news 5.79%news (soc.) 5.13% societynews (sport) 2.63%news (enter.) 2.24% music, movies, tv, etc.news �¿QDQFH� 1.97%news (life) 1.58% health, housing, etc.news (tech) 1.58% technologynews (weather) 1.18%

service

15.5%

service 7.63% translators, banks, etc.mail 3.95%maps 3.03%organisation 0.92% bookmarks, calendar, etc.

search

15.3% search 12.63%

search (special) 1.58% search for lyrics, jobs, etc.directory 1.05%

sharing

9.6%

blogging 3.55%knowledge 3.55% collaborative creation and collection of contentsharing 2.50% sharing of videos, ¿OHV� etc.

navi

9.3%

front page 6.58%front page (p.) 1.84% personalised front pagessitemap 0.92%

leisure

8.7%

adult 2.76%games 1.97%social media 1.97%dating 1.05%entertainment 0.92% sites with music, tv, etc.

support

8.7%

support 1.58% sites that provide products and support for themdownload 7.11% downloading software

shopping

7.9%

shopping 4.34%auctions 2.11%comparison 1.45% sites to compare prices of products

settings

2.9%

login 1.71%site settings 1.18% prR¿Oe setting, site personalisation

InteracCon data •  July 2012 •  2.5M users •  785M page views

NavigaCon model •  We defined a new navigaLon

model (see paper for details)

Site categories •  760 sites from 70 countries/

regions •  11 categories •  33 subcategories


MulCtasking Metrics CumAct accounts for the acLvity between site visits.

CumulaCve acCvity The metric is defined as follows:

InterpretaCon High CumAct à High engagement If users return aker short Lme, they return to conLnue with same task. If users return aker longer Lme, they return to perform a new task – a sign of loyalty.

CumActk = log10 (v1 + ivik •vi

i=2

n

∑ )

Browsing acLvity during the ith visit Browsing acLvity between the (i-‐1)th and ith visit Rescaling factor for ivi

k = 3

viivi

1 4 3 10 3

CumAct= log10 (3+1

3 •4+103 •3)= 3.48

Site visit


MulCtasking Metrics AWRange and AWShik describe changes between the visits.

APenCon shie and range The metrics is defined as follows:

InterpretaCon AWShik models the shik of a;enLon, and AWRange models the fluctuaLons in the browsing acLvity.

AttShiftn =invn −min Invn

| max Invn |− | min Invn |

AttRangen =σ (Vn )µ(Vn )

Variance in the visit acLvity Average of the visit acLvity Number of visits in session ModificaLon of the “Inversion number”

n = 4

σµi

Inv

0 >0

-‐1 constant decreasing

0

constant complex

+1

constant increasing

AWenHon range

AWenHo

n shik

0-1 1-0.5 0.5Spearman’s rho with p-value < 0.05



EvaluaCon CorrelaLons between mulLtasking and acLvity metrics.

[MT]

Ses

sVis

its

[MT]

Ses

sSite

s

[MT]

Cum

Act

[MT]

AttS

hift

[MT]

AttR

ange

[AC

T] D

wel

lTim

eS

SessSites [MT] 0.42

CumAct [MT] 0.41 -

AttShift [MT] 0.09 - -

AttRange [MT] - - -0.38 0.27

DwellTimeS [ACT] 0.20 0.24 0.12 0.32 0.08

DwellTimeV [ACT] -0.40 - - 0.14 - 0.50

No or only weak correlaCons between

the metrics.

All metrics convey different aspects about users’ online behaviour.


MulCtasking PaPerns Cluster centers, site categories and acLvity pa;erns.

Cat

egor

ies

Multitas

kin

g

DwellTimeV CumAct SessVisitsDwellTimeS

sitemapsite settings

news (wheather)download

+75%+73%+69%+67%

PD

139 sites

Quick task Continuousmultitasking

SessSitesBars from left to right:

111 sites

auctionsshopping

adultdating

+79%+71%+71%+62%

PD

-1.0

1.0

0.0

-1.0

1.0

0.0

PD - Probability difference

Act

ivity

Activity pattern: De - Decreasing In - Increasing Cn - Constant Cm - Complex

De In CmCn De In CmCn

60%

0%

147 sites

Recurring task

searchfront page (p.)

front pageorganisation

+77%+62%+57%+24%

PD

-1.0

1.0

0.0

De In CmCn

0.6

0.0

137 sites

Focused task

news (tech)news (life)

supportmail

+66%+66%+65%+64%

PD

-1.0

1.0

0.0

-1.0

1.0

0.0

De In CmCn

60%

0%

60%

0%

142 sites

Rapid multitasking

news (enter.)knowledge

comparisonservice

+64%+63%+62%+59%

PD

-1.0

1.0

0.0

-1.0

1.0

0.0

De In CmCn

60%

0%

Single-task-oriented browsing Multitask-oriented browsing

ATTACHMENT: Inter-‐site engagement


Data Dataset, network and site categories.

InteracCon data •  August 2013 to July 2014 •  53M sessions

Provider network G=(N, E, λ) N: 155 Yahoo sites

from five countries E: User traffic λ(e): Traffic volume (#Clicks)

Site categories •  155 sites from 5 countries •  5 categories

Cat. %Sites Description

35%19%13%23%10%

newsserviceleisureproviderfront page

mail, calendar, etc.social media, games, etc.account settings, help, etc.front pages, site maps

servicefront page news providerleisure


Inter-‐site Engagement Metrics Flow accounts for the extent users navigate between sites.

Traffic Flow The metric is defined as follows:

InterpretaCon High Flow à High inter-‐site engagement Users navigate oken between the sites of the network.

Flow =wi, ji, j∑vii∑

#Clicks between node i and j #Visits on node i

wi, j

vi

Flow = 30/60 = 0.5

105

2020

20

10 5

11

2020

20

1 1

Flow = 4/60 = 0.07


Inter-‐site Engagement Metrics Density describes the connecLvity of the network.

Density We use the density measure of [Wasserman]:

InterpretaCon High Density à High inter-‐site engagement Users navigate between many different sites (inter-‐site engagement is highly diverse).

[Wasserman] S. Wasserman. Social network analysis: Methods and applicaHons, 1994.

Density = #Edges#Possible_Edges

Density = 4/6 = 0.7

Flow = 2/6 = 0.3


Inter-‐site Engagement Metrics Reciprocity measures the homogeneity of traffic between two sites.

Reciprocity We use the reciprocity measure of [SquarLni]:

InterpretaCon High Reciprocity à High inter-‐site engagement Users navigate between two sites in both direcLons (inter-‐site engagement is highly homogenious).

[SquarHni] T. SquarHni, F. Picciolo, F. RuzzenenH, and D. Garlaschelli. Reciprocity of weighted networks. Nature: ScienHfic reports, 2013.

#Clicks between node i and j wi, j

RP =min[wi, j,wj,i ]i< j∑

wi, ji≠ j∑

110 5

20

1

Reciprocity = 15/50 = 0.3

Reciprocity = 2/37 = 0.05

1010 5

20

5


Inter-‐site Engagement Metrics Entry/ExitDisp measures how the traffic to/from the network is distributed over the sites.

Entry disparity and exit disparity We use the group degree measure of [Freeman] and adapt it as follows:

InterpretaCon High Entry/ExitDisp à Low inter-‐site engagement The network is more vulnerable to outages, because only few sites are used to enter (leave) the network.

EntryDisp =(ginmax − g

ini )i∑

| N |• ginii∑

[Freeman] L. C Freeman. Centrality in social networks conceptual clarificaHon. Social networks, 1979.

Number of visits that started at node ni (user entered the network) Maximum value of gin Number of nodes | N |

giin

ginmax

EntryDisp = 20/3�40 = 0.17

20 10

10

40 5

5

EntryDisp = 70/3�50 = 0.47


EvaluaCon: Network-‐level CorrelaLons between inter-‐site and network engagement metrics.

[IS

] D

en

sity

[IS

] R

ecip

rocity

[IS

] E

ntr

yD

isp

arity

[IS

] E

xitD

isp

arity

[PO

P]

#S

essio

ns

[AC

T]

Dw

ellT

ime

S

[AC

T]

#S

ite

s

Flow [IS] - 0.15 0.23 0.30 - 0.35 0.65

Density [IS] 0.48 -0.61 -0.60 0.92 -0.45 -0.25

Reciprocity [IS] -0.38 -0.32 0.42 - 0.25

EntryDisparity [IS] 0.84 -0.54 0.33 -

ExitDisparity [IS] -0.55 0.38 0.20



Density and #Sessions The more users are

visiCng the network, the more diverse is the inter-‐

site engagement.

Entry-‐ and ExitDisparity Volume of in-‐ and out-‐going traffic of the nodes depend on each other.

Flow and #Sites The more sites are visited

during a session, the higher the flow of traffic.


EvaluaCon: Node-‐level CorrelaLons between inter-‐site and site engagement metrics.

[IS

] D

ow

nstr

eam

[IS

] E

ntr

yP

rob

[IS

] E

xitP

rob

[PO

P] #S

essio

ns

[A

CT

] D

wellT

imeS

[M

T] #V

isits

[MT

] C

um

Act

PageRank [IS] 0.30 -0.08 -0.10 0.85 0.06 0.08 0.31

Downstream [IS] -0.27 -0.22 0.17 0.04 0.02 -0.02

EntryProb [IS] 0.79 0.12 -0.19 0.13 0.35

ExitProb [IS] 0.08 -0.18 0.18 0.32



PageRank and #Sessions Popular sites in the

provider network, are also visited frequently when browsing through

the network.

Entry-‐ and ExitProb Nodes that are used to enter the network are also frequently used to

exit the network.


Comparing Provider Networks

Country2

Country1

Country3

Country4

Country5

Flow Reciprocity EntryDisparityDensity DwellTimeBars from left to right:

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0

-1.0

0.0

1.0


PaPerns of Inter-‐site Engagement C

ateg

orie

sE

nga

gem

ent

PageRank EntryProb DwellTimeDownstream PD - Probability difference

46 sites

Focused eng.

front pageservice

providerleisure

news

+80%+63%-100%-100%-100%

PD

23 sites

Traffic hub

46 sites

Supporter

40 sites

Shared eng.

CumActBars from left to right:

-1.0

1.0

0.0

providerservice

newsfront page

leisure

+31%+19%

-2%-10%

-100%

PDleisure

providerservice

newsfront page

+67%+48%-21%-94%

-100%

PDnews

leisureprovider

front pageservice

+63%-61%

-100%-100%-100%

PD

-1.0

1.0

0.0

-1.0

1.0

0.0

-1.0

1.0

0.0

ATTACHMENT: NaCve AdverCsing

NaLve AdverLsing 84

Effect on User Engagement

0%

200%

400%

600%

short ad clicks long ad clicks

ad c

lick

diffe

renc

e

ï��

��

��

��

short ad clicks long ad clicks

clic

ks p

er d

ay d

iffer

ence

PosiLve experience has a strong effect on users clicking on ads again, and a small effect on user engagement with the stream.

NaLve AdverLsing 85

Mobile vs. Desktop

Ad post-‐click experience between mobile and desktop differs. For dwell Lme we obtain rho = 0.50; this value is even smaller for bounce rate with rho = 0.23.

0.00

0.05

0.10

0.15

ï�� ï�� ï�� ï�� ï�� dwell time difference

p(dw

ell t

ime

diffe

renc

e)

higher on mobilehigher on desktop

0.00

0.05

0.10

0.15

ï�� ï�� bounce rate difference

p(bo

unce

rate

diff

eren

ce) higher on mobilehigher on desktop

NaLve AdverLsing 86

Mobile OpCmised Landing Pages

Dwell Cme: The distribuLon is very similar for both groups. Bounce rate: Decreases by 6.9% (median decreases by 30.4%) for Opt landing pages but increases by 13.4% (median decreases by 11.5%) for Npt landing pages.

not mobile optimized mobile optimized

0.0

0.1

0.2

0.3

ï�� ï�� ï�� ï�� ï�� ï�� dwell time difference

p(dw

ell t

ime

diffe

renc

e)

higher on mobilehigher on desktop

mobile opt.

not mobile opt.

0.0

0.1

0.2

ï�� ï�� bounce rate difference

p(bo

unce

rate

diff

eren

ce) higher on mobilehigher on desktop

mobile opt.

not mobile opt.

ATTACHMENT: Wikipedia

Wikipedia 88

Wikipedia Research Literature review by Okoli et al.: The people’s encyclopedia under the gaze of the sages: A systemaLc review of scholarly research on wikipedia.

Wikipedia 89

Reading Preferences

Popularitylow high

Art

icle

Len

gth

shor

tlo

ng borderline casesII I

III IV

Jeanne Tsai

Douglas Adams

LuisPalomino

Anne Stears

PeterEhrlich

AlecMango

Stephen D.Lovejoy

1st Dalai Lama

Dexter Jackson (safety)

Katie GreenBrittanyBorman

Anthony Anenih

Ronnie Bird

Jan Anderson(scientist)

FitchRobertson

Sean Bennett

For 4.2% (group IV) of the articles editing activity is low, but reading activity is high.!

Wikipedia 90

Reading PaPerns A

rtic

le

topic

Rea

din

gbeh

avio

r

ArticleViewsa

SessionArticlesa

Popularitya

ReadingTimea

CA - Percentage in topic

4,826 articles11,579 behavior vectors

sportspersonmusician

media pers.

28%26%23%

CA

Exploration

artist/writerhistorical fig.

polit./businessp.

43%41%37%

CA


Focus


historical fig.criminal/victim

musican

42%38%38%

CA

Trending


media pers.sportsperson

musician

27%27%19%

CA

Passing

28K [16K,51K]

11 [5,23]

7.7%

38K [21K,69K]

20 [9,41]

16.9%

26K [15K,45K]

10 [5,21]

10.5%

16K [10K,27K]

8 [3,18]

5.1%

ArtLen

#Edits

%HQA

#Edits - Number of edits

-1.0

0.5

-0.5

0.0

1.0

-1.0

0.5

-0.5

0.0

1.0

-1.0

0.5

-0.5

0.0

1.0

-1.0

0.5

-0.5

0.0

1.0

%HQA - Percentage of high quality articlesArtLen - Article length

Wikipedia 91

Reading PaPerns over Time

Stability •  30% of the arLcles are popular in 1 month •  10% are popular over the whole 13-‐months •  Almost all arLcles have one reading pa;ern

half of their life Lme

TransiCons •  TransiLons are temporary – arLcles move

temporarily to another cluster •  High reciprocity – similar number of

transiLons in both direcLons •  “Focus” cluster is isolated -‐ ArLcles in that

cluster are the most stable ones •  Strong connecLon between the “Passing”,

“ExploraLon”, and “Trending” clusters – many arLcles adopt all three pa;erns

ATTACHMENT: Yahoo

93

Upstream Traffic

TeleportaCon Social media / News Search / Ext-‐Yahoo Users engage (quickly)

to many sites. Users conHnue with same acHvity inside the provider network.

Users visit site they are interested in, perform a quick task, and leave.

Network DwellTime -‐-‐ ++ -‐-‐

Traffic DistribuCon ++ -‐-‐ -‐-‐

Entry Disparity -‐-‐

Yahoo

Users engage differently depending on where they are coming from.

94

Network Effect PaPern

Yahoo

Sites change their popularity in the same way. Ac>vity (dwell >me) on a site depends more on the site itself,

but there are some nega>ve dependencies.

Pat

tern

exam

ple

s

41 patterns

Simple star-like

6 patterns

Complex star-like

1 pattern

Cluster-like

3.00 [3.00,4.00]

0.67 [0.00,0.89]

0 [0,0]

8.00 [7.00,18.00]

0.76 [0.56,0.84]

0 [0,0]

52

0.91

0.51

N

Recip

Trans

N - Number of nodes Recip - Reciprocity Trans - Transitivityservicefront page news providerleisure

(4) (5) (6)(1) (2) (3)

95

Hyperlink Performance

Yahoo

0%

25%

50%

75%

100%

Onïs

ite

links

front page providerservice news leisure

Inte

rïsi

te lin

ks


0%

20%

40%

60%

80%

Exte

rnal

lin

ks


20%

40%

60%

(a) PageRank and downstream.

TrafficPageRank Downstream

HyperlinksPageRank 0.54 -Downstream - -

(b) On-site, inter-site, and external.

TrafficOn-site Inter-site External

HyperlinksOn-site 0.54 -0.45 -0.38Inter-site -0.40 0.50 -External - - 0.39

ATTACHMENT: Online News

Online news 97

Focused versus Non-‐focused Sessions Internal

Non-focused sessionsFocused sessions●

(b) Duration

(d) p(focused session)

(a) %Sessions

(f) Flow

25

15

5

60%

20%

0.6

0.2

0.2

0.1

2 3 4 5 6 7 7 2 3 4 5 6 7 7 2 3 4 5 6 7 7#Articles #Articles #Articles

●

●

●

●●

●

●

(c) #Providers2.5

2.0

1.5

●

●

●●

●●

●

●●

●●

●●●● ●●

●●

●●(e) EntryDisparity

0.5

0.3

0.1 ●

●

●

●

●●

●

When users focus on a news story, they spend more >me reading the ar>cles and the inter-‐site engagement between providers is higher.

Online news 98

Hyperlink Performance

Number of Inline Links •  <10 links may be wasLng an opportunity •  10-‐29 links does not result in more clicks •  >29 links may harm the user experience PosiCon of Inline Links •  30% at the end, 16% at the beginning, 46%

are distributed within the arLcle text. •  Performance of links located at the

beginning of the text is very low (-‐28%) •  Best performance is achieved with links at

the end of the arLcle text (+35%)

Link popularity● Link performance

Position in article textLin

k po

pula

rity

[0.0,0.1[ [0.3,0.4[ [0.6,0.7[ [0.9,1.0]

10%

20%

30%

-0.2

0.0

0.2

Lin

k p

erforman

ce

●●

●●

●●●●

●● ●● ●● ●●●●

●●

●●

●●●● ●● ●● ●● ●● ●● ●●

●●

Number of inline links in article

Clic

ks p

er li

nk

0.0

0.2

0.4

0.6

[0,2] [9,11] [18,20] [27,29] [36,38]

Number of inline links in article

Num

ber of

clic

ks

[0,2] [9,11] [18,20] [27,29] [36,38]

2.5

5.0

7.5

Online news 99


Internal Focused Short-‐term: Only 3 (out of 50) providers have their corresponding average dwell Lme lower for the story-‐focused provider sessions. The average increase in dwell Lme from non-‐story-‐focused to story-‐focused provider sessions is 50%. Long-‐term: For 78% of the providers, we find that there are more users that return earlier aker they have a story-‐focused provider session.

Internal

News provider

Dw

ell tim

e per

ses

sion

Non-focused Focused Ext-focused

News providerp(a

bse

nce

12

h)


Online news 100


External Focused Short-‐term: We do not observe an effect on the dwell Lme (neither posiLve nor negaLve). The average increase is only 5.5%, and based on the K-‐S test we cannot confirm that the distribuLons are different (p-‐value=0.36). Long-‐term: For 70% of these news sites, the probability that users return within the following 12 hours increases (the average increase is 76%).

External

News provider

Dw

ell tim

e per

ses

sion


News providerp(a

bse

nce

12

h)


Online news 101

Discovering Story-‐related Content in TwiPer

From Site to Inter-site User Engagement

Data & Analytics

Transcript of From Site to Inter-site User Engagement