Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender...
Transcript of Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender...
![Page 1: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/1.jpg)
Recommender System on Big Data
![Page 2: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/2.jpg)
www.4paradigm.com
Netflix, Inc. is an American provider of flat rate DVD-by-mail in the United States, where mailed DVDs are sent via Permit Reply Mail.
The Netflix Prize, begun on October 2, 2006, was an open competition for the best collaborative filtering algorithm to predict user ratings for films. The grand prize is US$1,000,000.
The Netflix Prize
![Page 3: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/3.jpg)
www.4paradigm.com
Emerged by US$1,000,000 …
480,189 !users
17,770 movies
100,480,507 ratings
480,189 !users
17,770 movies
K
K ×
new technology
."
."
."
."
.
traditional algorithms
![Page 4: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/4.jpg)
www.4paradigm.com
On the other hand …
1/3 products are sold via Amazon Recommender System.
Google earns 42 billion US dollars via Ads Recommender System in 2012.
![Page 5: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/5.jpg)
www.4paradigm.com
In China, …
Great success in Baidu & Taobao Ads Recommender Systems, which earn > 10 billion RMB every year.
![Page 6: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/6.jpg)
www.4paradigm.com
Backend Technology
model ads
service
query
prediction
log training
store
training!data
upload
![Page 7: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/7.jpg)
www.4paradigm.com
Backend Technology
![Page 8: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/8.jpg)
www.4paradigm.com
Query
Query-Ad Query-Ad-User
Query-Ad
Query-Ad-User Query-Ad-User
Query-Ad-User
Optimization
{fea1, fea2, fea3, fea4, fea5, {fea6,fea7,fea8}}"
{fea1, fea2, fea3, fea4, fea5, fea6}"
{fea1, fea2, fea3, fea4, fea5, fea7}"
{fea1, fea2, fea3, fea4, fea5, fea8}"
transformation �
![Page 9: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/9.jpg)
www.4paradigm.com
Model Training on MPI
![Page 10: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/10.jpg)
www.4paradigm.com
Key Feature for Successful Recommender Systems
service
query
prediction
log training
store
training!data
upload 480,189 !
users
17,770 movies
100,480,507 ratings
![Page 11: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/11.jpg)
www.4paradigm.com
Recommender System on Big Data
- serving users to retrieve big data!- learning from big data to improve service
react!(accept/reject)
recommend!
(service)
log database log
modeling
![Page 12: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/12.jpg)
www.4paradigm.com
Recommendation Technology: 3 Generations 1. Collaborative Filtering
2. Sparse Linear Prediction Model
3. Deep Learning (developing)
……
![Page 13: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/13.jpg)
www.4paradigm.com
1st Generation – Collaborative Filtering
1. Nearest Neighbor
2. Topic Modeling
3. Matrix Factorization
![Page 14: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/14.jpg)
www.4paradigm.com
1.5G – Ensemble Learning Basic Idea: training multiple models, and voting
like dislike like
like
![Page 15: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/15.jpg)
www.4paradigm.com
Problem – weak for new users ???
new user
![Page 16: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/16.jpg)
www.4paradigm.com
2nd Generation – Sparse Linear Prediction Model
model ads
> 100 billion features
user demographic
relevance short-term behaviors
![Page 17: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/17.jpg)
www.4paradigm.com
2.5G – Feature based Collaborative Filtering
(Tianqi Chen, et al. ICML 2012)
![Page 18: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/18.jpg)
www.4paradigm.com
Problem – cost on feature engineering > 100 billion features
user demographic
relevance short-term behaviors
too many people to manage these features
1. Need a lot of domain experts to design features.!
2. Managing the expert team is not easy.!
3. Hard to repeat the successful experience to other application.
![Page 19: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/19.jpg)
www.4paradigm.com
中间计算层
框架统一接口
S_direct S_combine S_plsa ...
特征抽取
切词 意图 飘红 ...
公共计算
PLSA
Query Cmatch-rank AdClusterID ...
特征配置
Managing Feature Engineering Team
![Page 20: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/20.jpg)
www.4paradigm.com
3rd Generation – Deep Learning
![Page 21: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/21.jpg)
www.4paradigm.com
Deep Learning for Recommendation
like/dislike
![Page 22: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/22.jpg)
www.4paradigm.com
Deep Recommender Systems
l Advantage:"p Less domain experts for feature engineering"
n Deep (3G) Systems: 5~10 Top Scientists/Engineers + Supporting"n Flat (2G) Systems: 5~10 Top Scientists/Engineers + 50 Domain Experts +
Supporting"
p Easy to duplicate to other applications"
l Challenge"p System & Algorithm design"
n Need top scientists/engineers"
![Page 23: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/23.jpg)
www.4paradigm.com
Summary
l Recommender systems have already achieved great success in several companies."
l As deep learning technology develops, recommender systems will easily come into more applications."
l In the future, there will be several flat (2G) recommender systems with high cost, and a lot of deep (3G) recommender systems with low cost."
![Page 24: Recommender System on Big Data - Department of ...kaichen/courses/spring2015/comp6611/...Recommender System on Big Data - serving users to retrieve big data! - learning from big data](https://reader031.fdocuments.net/reader031/viewer/2022030818/5b2d2a127f8b9adc6e8ba1e7/html5/thumbnails/24.jpg)
www.4paradigm.com
24" Thanks!