Post on 06-May-2015
description
Recommendations and Discoveryat StumbleUpon
Sumanth Kolar,
Director, Engineering
@_5K
StumbleUpon’s Mission
Help users find content they did not expect to find
Be the best way to discover new and interesting things from across
the Web.
How StumbleUpon works
1. Register 2. Tell us your interests 3. Start Stumbling and rating web pages
We use your interests and behavior to recommend new content for you!
There is a ongoing shift from search to discovery
Discovery is very different from search
Discovery at StumbleUpon Search
Serendipitous Intent driven
One at a time List of articles
Never repeats Always repeats
Constantly adapting Fixed results
Tailored for you Impersonal
StumbleUpon
StumbleUpon Overview
Discovery Crawled
Ingestion Pipeline
Sampling Pass?
Rec Engine
Users Automated
URL Index
Yes
1
2
3
What are the key challenges to good recommendations?
Pillars of good recommendations
Understand who the user is and what he is interested in.
Separate good content from the bad.
Learn from your recommendations.
Explore various techniques for matching users to content.
Pillars of good recommendations
Understand who the user is and what he is interested in.
Separate good content from the bad.
Learn from your recommendations.
Explore various techniques for matching users to content.
User self reports topics of interest
Part of the sign up flow…
User’s Interest Graph
Food/Cooking User
Cars
VintageCars
Italian Recipes
Continually Enhance a User’s Interest Graph
Analyze user’s StumbleUpon history to expand on interest preferences:
• Add/remove topics• Follow/block particular domains
Continually Enhance a User’s Interest Graph
Leverage social network data:
• Find friends & people to follow
• Find content trending in your social circles
• Find additional interests
Continually Enhance a User’s Interest Graph
Mine internal StumbleUpon rating and sharing data to suggest other stumblers, topics.
Enhanced Interest Graph
Food/Cooking
User
Cars
VintageCars
Italian Recipes
nasa.gov
1x.com
News
Friends
Trending
Pillars of good recommendations
Understand who the user is and what he is interested in.
Separate good content from the bad.
Learn from your recommendations.
Explore various techniques for matching users to content.
On average hundreds of URLs are ingested into the
StumbleUpon pipeline every minute.
• Sampling key goals:
1. Determine which URLs to sample and which to skip completely
2. Examine sampling results to identify good URLs
• URL features used when sampling:
• Known domain performance(ratings, timespent)• Content related features (#images, #ads, url length etc)• User features of the discoverer (spammer vs trusted user)
Sampling
Vote
Yes
Yes
NoYes
Recommend
Classifier based on User Feedback (Timespent, Ratings)
Yes
Random Forest
Webpage
Recommendations at StumbleUpon: Sampling
Rating Timespent
Good 35sec
Good 22sec
Bad 15sec
Good 45sec
Good 14sec
Good 28sec
• Users who thumb-up good content and thumb-down bad content
• For example– Joe DiMaggio – Baseball– Julia Child- Food/Cooking– Da Vinci- Art and Architecture
• Ratings from Experts are more trustworthy and earn more weight.
Leveraging In-Network Experts
Expe
rtRecommendations at StumbleUpon: Experts
Non
Exp
ert
Pillars of good recommendations
Understand who the user is and what he is interested in.
Separate good content from the bad.
Learn from your recommendations.
Explore various techniques for matching users to content.
Challenge: User expectations are different
“I LOVE cars!”-Anonymous Stumbler
“Me too!”-Another Stumbler
• Find users who like content similar to the content you do
• Signals can be ratings, time spent, interests, etc.
• Use the content they’ve liked
Like-Minded Users
NeuroscienceAstronomySpace ExplorationComedy Movies
Astronomy Space ExplorationPhysics Classic Movies
Vintage CarsAction moviesAstronomyRobotics
Science
Space
Movies
Cars
PLSI based like-minded
Total Pairwise Similarity Calculations
= 50K users * 5 million users * 1K features
= 250 Trillion Probabilistic Latent Semantic Index (PLSI)
based similarity over 500 trillion calculations PLSI based similarity framework computes in
less than an hour
Like-Minded Users: Challenges Scaling
Food/Cooking
User
Cars
VintageCars
Italian Recipes
nasa.gov
1x.com
News
Experts Friends
Trending
Grow User’s Interest Graph: Implicit + Explicit
LikemindedUsers
Different methods perform differently for different users at different times
User 1 User 2 User 3 User 4 User 50%
25%
50%
75%
100%
TrendingFollowBias domainsExpertsNewsLike-minded
Recommendation context
Pillars of good recommendations
Understand who the user is and what he is interested in.
Separate good content from the bad.
Learn from your recommendations.
Explore various techniques for matching users to content.
Two Main Signals from Recommendation
Rating Time Spent
Both present numerous challenges . . .
# Ra
tings
Time
Users rate more during their initial experience
Why is this happening?
Ratings: volume decay
Images
T5 sec
?
Text
T4 sec
• Ratings are sparse• < 10% of recommendations have explicit ratings.
• Using time spent decide whether the stumble was skipped• Timespent on videos is longer than images. • Solution: Estimate p(Like | Timespent)
• Model based on user, content patterns
Video
T3 sec
?
Images
T2 sec
Time Spent
Video
T1 sec
Installed plugin
Stumble Bar
Mobile / Tablets
Challenges: Time spent on different devices
5th percentile time spent per stumble
Med
ian
time
spen
t per
stum
ble
Pillars of good recommendations
Understand who the user is and what he is interested in.
Separate good content from the bad.
Learn from your recommendations.
Explore various techniques for matching users to content.
How do we know we are doing a good job?
Extensive A/B Testing
AB Tests on metrics such as session length, retention, rating behavior etc
Measurable Improvements In Rec Quality
12/1/0
8 0:00
2/1/0
9 0:00
4/1/0
9 0:00
6/1/0
9 0:00
8/1/0
9 0:00
10/1/0
9 0:00
12/1/0
9 0:00
2/1/1
0 0:00
4/1/1
0 0:00
6/1/1
0 0:00
8/1/1
0 0:00
10/1/1
0 0:00
12/1/1
0 0:00
2/1/1
1 0:00
4/1/1
1 0:00
6/1/1
1 0:00
8/1/1
1 0:00
10/1/1
1 0:00
12/1/1
1 0:00
2/1/1
2 0:00
4/1/1
2 0:00
6/1/1
2 0:00
8/1/1
2 0:000
2
4
6
8
10
12
14
16
R² = 0.737311794306772
Normalized Likes vs Dislikes
Recent Months
+111% improvement!
• Dupe detection• Anti-spam• News• Topic classification• Metrics, quality analysis• Trending• Search• User biases, mood• Many more…
Many other interesting problems…
We are HIRING !!!