Challenges doing Personalization at Web-scalegotocon.com/dl/jaoo-melbourne-2011/slides/... · where...

CHALLENGES DOING

PERSONALIZATION

AT WEB-SCALE

Peter Bailey (pbailey)

Bing Contextual Relevance, Microsoft

YOW! Australia :: Dec 2011

Meet Bruce

He’s an Australian

expatriate …

YOW! Australia 2011 © Microsoft Bing : Peter Bailey 2

Meet Bruce

From Sydney … … who still likes to read

his hometown

newspaper


Meet Bruce

He lives in Sarasota,

Florida, and has a long

term history of health

related queries. Must be

all that sun …

… so it’s lucky the

Sarasota Memorial

Hospital has a full array

of skin cancer services


Meet Bruce

He traveled to Seattle

for work this week,

where he is depressed

by the bleak weather.

He’s searched on

Seasonal Affective

Disorder and similar

maladies.


Meet Bruce

Bruce’s most recent

searches however

include {BBH} and {PPH}

… which happen to be

stock codes for some

HOLDRS ETFs from

Merril-Lynch


Meet Bruce

Bruce’s next query is:

{smh}

What do we do?!


WHAT IS THIS TALK ABOUT? It’s all about the users …


User history in context: Peter Bailey

Search work:

• ANU (twice)

• NUIX

• Synop

• CSIRO

• Microsoft

Other work:

• Object Technology

International (IBM)

Products:

• Parallel programming

software systems (for

Fujitsu AP1000)

• VisualAge for Smalltalk

• Panoptic (Funnelback)

• Sytadel CMS

• IE Suggested Sites

• Bing

• 4/5TH of a CORBA ORB

YOW! Australia 2011 : What is this talk about? © Microsoft Bing : Peter Bailey 9

Focus areas

Kinds of personalization

Measurement

Experimentation

User feedback systems

Performance

Another thing

YOW! Australia 2011 : What is this talk about? © Microsoft Bing : Peter Bailey 10

© Andrew Sieber

© Todd Lappin

http://creativecommons.org/licenses/by-nc/2.0/

http://www.flickr.com/photos/smartjunco/

http://www.flickr.com/photos/smartjunco/

http://creativecommons.org/licenses/by-nc/2.0/

http://www.flickr.com/photos/telstar/

http://www.flickr.com/photos/telstar/

KINDS OF

PERSONALIZATION


Several core types of online systems

Pull information

e.g. Search engines

Push information

e.g. Advertisements

Interaction

e.g. Games

Transactional

e.g. Travel bookings

YOW! Australia 2011 : Kinds of personalization © Microsoft Bing : Peter Bailey 12

Browse information e.g. Web sites

Kinds of

Personalization

Level Description Prerequisites

0 No personalization

1

Contextualization - all people who

belong to some easily identified

cohort receive the same experience;

but this experience is different across

cohorts (e.g. your location)

The system can identify

you belong to a particular

cohort

2

You see different things because you

have a different "view" onto some

underlying data than someone else

(e.g. your bank account)

You have an "account" of

sorts with the system - it

knows who "you" are wrt

to other users - that is, an

online identity exists

3

You can do different things or receive

different information because of how

the system maps certain access rights

associated with your identity (e.g. your

security groups)

Some form of "role-

based" access controls,

typically hard-coded

through business rules

4

You experience different results and

interactions with the system based on

pre-computed user models about you

(e.g. your credit history)

User models are

computed in batch mode,

and are used to adjust

the interaction algorithms

of the system

5

You experience different results and

interactions with the system based on

your real-time interaction with it (e.g.

your searches and browsing patterns)

Dynamic behavior

analysis and user model

updates are used to

adjust the interaction

algorithms of the system

0. None

1. Context-only

2. Different data

3. ACLs

4. Static user models

5. Dynamic user models

YOW! Australia 2011 : Kinds of personalization © Microsoft Bing : Peter Bailey 13

MEASUREMENT


Measure driven development (MDD)

Measurement first

• Know what you’re trying

to improve for the user

• Know how you’re going

to tell if you succeed

Antonym

“shipping and hoping”

Challenges:

• Changing mindsets

• Finding or designing

user-sensitive measures

• Finding repeatable

measures

YOW! Australia 2011 : Measurement © Microsoft Bing : Peter Bailey 15

User-centric behavior

Measurement of user

behavior is fundamental

Measures posit some

model of user task

achievement

• Typical user behavior is

not unambiguous

Challenges:

• HTTP is a stateless

protocol

• User-centric systems

require additional

infrastructure

• Different basis to

logging activity

• More data to log


User data may be sensitive

Personally identifiable

information (PII)

Most individual PII and

user-associated

behavioral data is not

computationally

interesting other than in

relation to aggregated

data

Challenges:

• Storage

• Access

• Retention

• Data mining

• PII to non-PII boundaries


EXPERIMENTATION


Scale of experimentation data

In-person user study

Web-scale user behavior

YOW! Australia 2011 : Experimentation © Microsoft Bing : Peter Bailey 19

Crowd-sourced investigation

Culture of experimentation

Measurement matters! • And an organizational

culture that respects it

Good user-based experimentation processes

• These are real people

Failure is acceptable • Indeed, it’s expected!

• But fail fast otherwise you’ll lose your users

Challenges • Infrastructure complexity

• E.g. 2 versions of your user models

• Testing pre-production systems involving users

• Measurement, evaluation, and statistical expertise

• Fast failure monitoring and recovery systems


Experimental design with users

Typical experimental

design seeks to

eliminate variance due

to users

• What happens when

that’s the whole point?

Determine that your

user sample will reflect

behavior at large

Challenges

• User sampling

• Sufficient user scale

• Interaction effects unless

experiment is completely

naturalistic


Efficiency

Online • Online expt is really

expensive! • Real users

• Not directly repeatable

• Production quality code

• Failure responsibilities

• Worse user experience if experiment not successful

• Wasted finite user “bandwidth” if experiment wrongly designed or has bugs

• But ultimately you have to test online because it’s all about the users …

Offline • Find simple surrogates for

online user behavior

• Use a preliminary version of any new system with you and many others

• Use representative scenarios, not just hand-picked ones

• Build offline experimentation platforms that are repeatable and scalable, with a low barrier to entry for non-developers


USER FEEDBACK SYSTEMS


Explicit

Explicit feedback from

users should provide

discernible value to users

Explicit feedback should

be:

• Non-invasive

• Painless (one-click)

• Obvious

• Progressive for the

enthusiastic

Challenges:

• Low volume (high value)

• Spam-able

• Explicability of

personalization choices

• Potential for “creepiness”

YOW! Australia 2011 : User feedback © Microsoft Bing : Peter Bailey 24

Implicit

Implicit feedback can be an invaluable source of information from your users

Probably the most vital data source for personalization • The more users, the

more implicit feedback, the better

Challenges: • High volume

• High noise

• Interpretability of actions

• Interpretability of changes in actions

• Additional code, storage, and tooling

YOW! Australia 2011 : User feedback © Microsoft Bing : Peter Bailey 25

PERFORMANCE


Amdahl’s Law redux

1

1 − 𝑃 + 𝑃/𝑆 𝑞𝑝𝑠 =

𝐶

𝑇

where C is the scale-unit CPUs to deliver 1 query in time T, without personalization

𝑢𝑞𝑝𝑠 =𝐶

𝑇 − 𝑈 + 𝑈. 𝑛

where U is the personalization time within T (akin to 1-P in Amdahl’s Law), and n is the number of users making requests at any time

With simplifying assumptions, sustaining your original qps requires scaling C by

𝑞𝑝𝑠

𝑢𝑞𝑝𝑠≅ 1 +

𝑛.𝑈

𝑇

YOW! Australia 2011 : Performance © Microsoft Bing : Peter Bailey 27

Amdahl’s law redux

1 +𝒏.𝑼

𝑻

Challenges: • It’s all about what you can’t

speedup

• Personalization breaks most of your preconceptions about using caching for performance optimization

• Be ultra-focused on optimizing personalization code if you have a large number of users • Milliseconds matter!


(At) page load time

Fast sites get more use

Personalization makes page load slower

Well engineered design can mask additional latencies

Challenges: • Get user id and data as

quickly as possible

• Separate personalized vs non-personalized content for progressive loading

• Avoid caching of other people’s personalized information at any point in the server-to-client delivery chain


ANOTHER THING


Thoughts on the “filter bubble”

My thoughts are not about whether search engines have an

ethical responsibility to deliver diversity of opinion

• Though that’s a great debate to have

We should acknowledge that people with an ability to

interact with global search engines have (speed of) access

to information unparalleled in the history of humanity

• Though there are many who still do not have access

But people have no more time than they ever did

YOW! Australia 2011 : Another thing © Microsoft Bing : Peter Bailey 31


It’s all about trust

• Do your users grow to believe you are trying to do the best things

possible for them?

• How do you demonstrate this?

• How do you let them tell you that you got it wrong?

• And how do you learn continuously?

• And do it, and know that you’re doing it, better the next time?



In the end, all non-trivial information system interactions

will be personalized to some degree

• I encourage you to start thinking about what that may mean for

your development practices sooner rather than later!


© Daniel Agostini

http://creativecommons.org/licenses/by-nc-nd/2.0/deed.en

http://www.flickr.com/photos/dhammza/

http://www.flickr.com/photos/dhammza/

QUESTIONS? And thank you for listening!


Challenges doing Personalization at Web-scalegotocon.com/dl/jaoo-melbourne-2011/slides/... · where...

Documents

Transcript of Challenges doing Personalization at Web-scalegotocon.com/dl/jaoo-melbourne-2011/slides/... · where...