LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides ›...
Transcript of LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides ›...
![Page 1: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/1.jpg)
Learning and Reasoning on Graph forRecommendation
© Copyright NExT++. All Right Reserved.
Xiangnan HeUniversity of Science and
Technology of China
Xiang WangNational University of
Singapore
Tutorial @CIKM 2019
Tat-Seng ChuaNational University of
Singapore
Slides in https://next-nus.github.io/
1
![Page 2: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/2.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation (~40mins)
• Part II: Random Walk for Recommendation (~20mins)
• Part III: Network Embedding for Recommendation(~20 mins)
• Part III: Graph Neural Networks for Recommendation(~100 mins)
2
Slides in https://next-nus.github.io/
![Page 3: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/3.jpg)
Graph Neural Networks for Collaborative Filtering and Knowledge-aware Recommendation
© Copyright NExT++. All Right Reserved.
Dr. HE Xiangnan
University of Science and Technology ofChina (USTC)
44
Age of Information Explosion
Serious Issue of Information OverloadingØ Weibo: >500M posts/dayØ Flickr:>300M images/dayØ Kuaishou: >20M microvideos/day
… …
![Page 4: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/4.jpg)
Ubiquitous Personalized Recommendation
Recommendation has been widely applied in online services:• E-commerce, Content Sharing, Social Networking, Forum …
Ad & ProductRecommendation
Search results of Taobao5
![Page 5: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/5.jpg)
Ubiquitous Personalized Recommendation
Recommendation has been widely applied in online services:• E-commerce, Content Sharing, Social Networking, Forum …
Image & VideoRecommendation
Search results of Pinterest 6
![Page 6: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/6.jpg)
Ubiquitous Personalized Recommendation
Recommendation has been widely applied in online services:• E-commerce, Content Sharing, Social Networking, Forum …
FriendRecommendation
Screenshot of Facebook 7
![Page 7: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/7.jpg)
Ubiquitous Personalized Recommendation
Recommendation has been widely applied in online services:• E-commerce, Content Sharing, Social Networking, Forum …
POI & PostRecommendation
Screenshot of TripAdvisor8
![Page 8: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/8.jpg)
Values of Recommender System (RecSys)
RecSys has become a major monetization tool for customer-oriented online services:• E-commerce, Content Sharing, Social Networking, Forum …
Ad systems are technically supported by recommendationsolutions:• The key is Click-Through Rate (CTR) prediction.
Some Statistics:• YouTube Homepage: 60%+ clicks [Davidson et al. 2010]
• Netflix: 80%+ movie watches [Gomze-Uribe et al 2016]
• Amazon: 30%+ page views [Smith and Linden, 2017]9
![Page 9: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/9.jpg)
The Era of Connected World
The world is more closely connected than you might think!
𝒖𝟐 𝒖𝟑
𝒊𝟏Shape of You
𝒊𝟐I See Fire
𝒊𝟑Skin
𝒊𝟒Castle on the Hill
𝒆𝟏Ed Sheeran
𝒆𝟐÷ 𝒆𝟑
Pop
𝒆𝟒Folk
User-User Connections• Social Relations• Same Profiles …𝒖𝟏 𝒖𝟒
Item-Item Connections• Same Attributes• External Knowledge …
User-Item Interactions• Implicit Feedback• Explicit Feedback …
11
![Page 10: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/10.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation• Problem Formulation
• Unified View for Recommendation Paradigm
• Limitations of Previous Works
• Part II: Random Walk for Recommendation
• Part III: Network Embedding for Recommendation
• Part III: Graph Neural Networks for Recommendation
12
Slides in https://next-nus.github.io/
![Page 11: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/11.jpg)
Overview of Recommendation Engine
User Interest is implicitly reflected in:- Interaction history- Demographics … - Contexts
User, Contexts
Items of Interest Item Corpus
Items can be: Products, News, Movies, Videos, Friends …
Key challenge: user-item semantic gap• user and item are two different types of entities
and are represented by different features.
Ranking CandidateGeneration
MillionsHundreds
Recommendation Engine
Tens
13
![Page 12: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/12.jpg)
Problem Formulation
?
User Profile:- User ID- Rating history- Age, Gender- Clicks- Income level
…….
Item Profile:- Item ID- Description- Image- Category- Price
…….
• Input: historical user-item interactions or additional sideinformation (e.g., user profile, item profile)
• Output: given a target Item (e.g., movie, song, product),how likely a user would interact with it (e.g., click, view,or purchase)
There may be no overlap between user features and item features.14
![Page 13: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/13.jpg)
Research on Prevalent RecSys
2009
2008~2013
1994~2004
2016~2018
Memory-basedo User Similarity [Thomas]o Item Similarity [Greg Linden]o Cosine Similarity[Stuart]o Pearson Correlation [Paul]
Model-based (User CF)o MF [Koren]o BPR-MF [Rendle]
Model-based (Item CF)o FISM [Kabbur]o SLIM [Ning]o SVD++ [Yehuda]
Deep Learning-basedo NeuMF [He]o ONCF [He]o DeepMF [Xue]o ACF [Chen]o NAIS [He]o DeepICF [Xue]
2010~2016 Factorization Machineso FM [Rendle]o FFM [Juan]
2016~2019
Deep Learning-basedo NFM [He]o DeepCross [Shan]o YouTube Recommender
[Covington]o Wide&Deep [Cheng]o DeepFM [Guo]o xDeepFM [Lian]o FNN [Zhang]o PNN [Qu]o CrossNet [Wang]o TEM [Wang]o …
Collaborative Filtering Models Generic Feature-based Models
15
![Page 14: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/14.jpg)
Research on Collaborative Filtering Models
2009
2008~2013
1994~2004
2016~2018
Memory-basedo User Similarity [Thomas]o Item Similarity [Greg Linden]o Cosine Similarity[Stuart]o Pearson Correlation [Paul]
User-basedo MF [Koren]o BPR-MF [Rendle]
Item-basedo FISM [Kabbur]o SLIM [Ning]o SVD++ [Yehuda]
Deep Learning-basedo NeuMF [He]o ONCF [He]o DeepMF [Xue]o ACF [Chen]o NAIS [He]o DeepICF [Xue]
Collaborative Filtering Models
Input Data:• User-Item Interaction Data
• Explicit Feedback (e.g., rating)• Implicit Feedback (e.g., clicks)
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
(𝒖𝟑 𝒊𝟐 )
(𝒖𝟑 𝒊𝟑 )
…… ……
𝒚𝒖𝟏𝒊𝟏
𝒚𝒖𝟏𝒊𝟏
𝒚𝒖𝟏𝒊𝟏
𝒚𝒖𝟏𝒊𝟏
𝒚𝒖𝟏𝒊𝟏
16
![Page 15: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/15.jpg)
Collaborative Filtering (CF)
• CF is the most well-known technique for recommendation.• “CF makes predictions (filtering) about a user’s interest by
collecting preferences information from many users(collaborating)” ---Wikipedia
• Collaborative Signals à Behavior Similarity of Users• Similar users would have similar preference on items.
1. Memory-based CF: Predict by memorizing similarusers’ (or items’) ratings
2. Model-based CF: Predict by inferring from anunderlying model.
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
(𝒖𝟑 𝒊𝟐 )
(𝒖𝟑 𝒊𝟑 )
(𝒖𝟑 𝒊𝟒 )…… ……
𝟓
𝟑
𝟒
𝟏
𝟐
𝟒
5 ? ? ? …
3 4 ? ? …
? 1 2 4 …
... … … … …
𝟏
𝟐
𝟑
𝟏 𝟐 𝟑 𝟒
user
item
Interaction Matrix17
![Page 16: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/16.jpg)
Memory-based CF
Problem: predict user 𝑢’s rating on item 𝑖.• User-based CF leverages the ratings of 𝑢’s
similar users on the target item 𝑖.
• Item-based CF leverages the ratings of 𝑢 on other similar items of 𝑖.
• Many similarity measures can be used, e.g., Jaccard, Cosine, Pearson Correlation. Recent advance learns the similarity from data.
5 ? ? ? …
3 4 ? ? …
? 1 2 4 …
... … … … …
𝟏
𝟐
𝟑
𝟏 𝟐 𝟑 𝟒
user
item
Interaction Matrix
Similar users of 𝑢Rating of a similar user on 𝑖
Similar items of 𝑖Rating of 𝑢 on a similar item
18
![Page 17: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/17.jpg)
Model-based CF
Matrix Factorization (MF) is the most popular and effective model-based CF method.• It represents a user and an item as a vector of latent factors.• The score is estimated as the inner product of user latent vector
and item latent vector.
• Optimizing a loss to minimize the prediction error on trainingdata can get the latent vectors.
5 ? ? ? …
3 4 ? ? …
? 1 2 4 …
... … … … …
𝟏
𝟐
𝟑
𝟏 𝟐 𝟑 𝟒
user
item
Interaction Matrix
.𝑦01 =< 𝑢, 𝚤>User latent vector 𝑢
Item latent vector 𝚤
19
![Page 18: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/18.jpg)
Item-based CF
Instead of only using an ID to encode a user, we can make theencoding more meaningful by using the user’s rated items.• This can be interpreted as an item-based CF model.
• E.g., FISM [Kabbur], SLIM [Ning]
User multi-hot encodingon rated items
Item one-hot encoding
* * *
+
Use all items as neighbors Factorize item similarity in the latent space
20
![Page 19: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/19.jpg)
Fusing User-based & Item-based CF
• MF (user-based CF) represents a user as her ID.• Directly projecting the ID into latent space
• FISM (item-based CF) represents a user as her interacted items.• Projecting interacted items into latent space
• SVD++ fuses the two types of models in the latent space:
o This is the best single model for rating prediction in theNetflix challenge [Koren, KDD ’08].
User representation in latent space
21
![Page 20: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/20.jpg)
Two Widely-Used Loss
Pointwise loss à e.g., log loss• Cast the recommendation task as a classification problem• Rating Prediction, CTR Prediction …
Pairwise loss à e.g., Bayesian Personalized Ranking (BPR) loss• Cast the recommendation task as a ranking problem• Top-N Recommendation, Preference Ranking …
Force the prediction scores tobe close to the target scores
Relative order betweenobserved & unobserved
interactions 22
![Page 21: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/21.jpg)
Deep Learning Meets CF (1)• Methods of representation learning
• Enhance representation ability/expressiveness of models
Model Input Data RepresentationLearning
InteractionLearning
DeepMF[Xue, IJCAI’17]
User: Historical itemsItem: User group
Multi-LayerPerceptron
Inner product
AutoRec[Sedhain, WWW’15]
User: Historical itemsItem: ID
Multi-LayerPerceptron
Inner Product
CDAE[Wu, WSDM’16]
User: Historical items + IDItem: ID
Multi-LayerPerceptron
Inner Product23
![Page 22: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/22.jpg)
Deep Matrix Factorization (Xue, IJCAI’17)
Representation Learning à Multi-layer perceptron• Deep Neural Networks are adopted to learn representations of
users & items
User: Row vector ofinteraction matrix
Item: Column vectorof interaction matrix
User RL Item RL
24
![Page 23: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/23.jpg)
Deep Learning Meets CF (2)• Methods of interaction function learning
• Capture complex patterns of user-item relationships
Model Input Data RepresentationLearning
InteractionLearning
NCF[He, WWW’17]
User: IDItem: ID
ID embedding Multi-LayerPerceptron
NNCF[Bai, CIKM’17]
User: User neighborsItem: Item neighbors
Embeddings Multi-LayerPerceptron
ONCF[He, IJCAI’28]
User: IDItem: ID
ID embedding ConvolutionalNeural Network26
![Page 24: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/24.jpg)
Neural Matrix Factorization (He, WWW’17)
Interaction Modeling à MF + MLP over users and items• MF uses inner product to capture the low-rank relation• MLP is more flexible in using DNN to learn the matching function.
27
![Page 25: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/25.jpg)
NNCF: Neighbor-based NCF (Bai, CIKM’17)
Interaction Modeling à MF + MLP over user and item neighbors• Feeding user and item neighbors into the NCF framework
28
![Page 26: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/26.jpg)
Research on Feature-based Models
2010~2016 Factorization Machineso FM [Rendle]o FFM [Juan]
2016~2019
Deep Learning-basedo NFM [He]o DeepCross [Shan]o YouTube Recommender
[Covington]o Wide&Deep [Cheng]o DeepFM [Guo]o xDeepFM [Lian]o FNN [Zhang]o PNN [Qu]o CrossNet [Wang]o TEM [Wang]o …
Generic Feature-based Models
Input Data:• User-Item Interaction Data• Other Information
• User Data• Item Data• Context Data
E.g., user gender,age, occupationpersonality … E.g., item category,
description, image …
rating data
context data
userdata
item data
E.g., location, time,weather, mood …
29
![Page 27: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/27.jpg)
Feature-based Models
E.g., user gender,age, occupationpersonality … E.g., item category,
description, image …
rating data
context data
userdata
item data
E.g., location, time,weather, mood …
Raw Features:• Categorical Features:
o One-hot encoding on ID features
• Continuous Features:o E.g., time, frequency. Need
feature normalization
Transformed Features:• Categorical Features:
o Cross Features are important(e.g., AND(A=True, B=True))
• Continuous Features:o E.g., outputs of other models
like visual embeddings. 30
![Page 28: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/28.jpg)
Factorization Machine (FM)
• FM is inspired from previous factorization models• It represents each feature as a latent vector (embedding), and
models the second-order feature interactions:
• FM allows easy feature engineering for recommendation, andcan mimic many existing models (that are designed for a specifictask) by inputting different features.• E.g., MF, SVD++, timeSVD [Koren, KDD’09], PIFT [Rendle, WSDM’10] etc.
Second-order:Pair-wise interactions between features
First-order:Linear Regression
31
![Page 29: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/29.jpg)
Wide&Deep (Cheng et al, RecSys’16)
• The wide part is linear regression for memorizing seen featureinteractions, which requires careful engineering on cross features.• E.g., AND(gender=female, language=en) is 1 if both single features are 1
• The deep part is DNN for generalizing to unseen feature interactions.• Cross feature effects are captured in an implicit way.
32
![Page 30: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/30.jpg)
Neural Factorization Machine (He et al, SIGIR’17)
• Inspired by FM, NFM models pairwise interactions betweenfeature embeddings with multiplication.
Capture bilinear interaction
Capture higher-order feature interactions
33
![Page 31: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/31.jpg)
A General ParadigmTransform each observation, a user-item pair (𝑢, 𝑖) or with sideinformation 𝑢, 𝑖, 𝑐 , into a separate data instance• Initiate representations for each feature à Representation Learning
• Design whatever features as you want• Perform predictions based on interactions à Interaction Modeling
• Design whatever networks as you like
InteractionModeling
9𝑦01RepresentationLearning
Data instance(𝑢, 𝑖) or (𝑢, 𝑖, 𝑐)
User Item Other Features
…
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
(𝒖𝟑 𝒊𝟐 )
(𝒖𝟑 𝒊𝟑 )
…… ……
34
![Page 32: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/32.jpg)
Information Isolated Island Issue (1)
Treating each observation as an independent instance• Forgoing relationships among instances
InteractionModeling
9𝑦01RepresentationLearning
Data instance(𝑢, 𝑖) or (𝑢, 𝑖, 𝑐)
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
…… ……
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
User-Item Interactions• Behavior similarity among users• Audience similarity among items
(𝒖𝟏 𝒊𝟏
(𝒖𝟐 𝒊𝟏
(𝒖𝟐 𝒊𝟐
𝒖𝟑
𝒖𝟑 )
𝒖𝟑 )
User-Item Interactions + Social Ties• Shared friends as bridge among users
à mouth marketing
𝒖𝟒 )
35
![Page 33: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/33.jpg)
Information Isolated Island Issue (2)
Treating each observation as an independent instance• Limited Representation Ability
• Instance representation is dependent on its own features merely• SVD++, NAIS: CF with neighbors as input are more expressive
InteractionModeling
9𝑦01RepresentationLearning
Data instance(𝑢, 𝑖) or (𝑢, 𝑖, 𝑐)
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
…… ……
• Suffer from sparsity issue• inactive users, unpopular
items, infrequent features àinsufficient information tolearn optimal representation 36
![Page 34: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/34.jpg)
Information Isolated Island Issue (3)
Treating each observation as an independent instance• Suboptimal Model Capacity
• Suboptimal representations lead to unsatisfactory interactionmodel, especially for unseen (user-item or feature) interactions
InteractionModeling
9𝑦01RepresentationLearning
Data instance(𝑢, 𝑖) or (𝑢, 𝑖, 𝑐)
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
…… ……
The methods have to rely oncomplex interaction functionto make up for the deficiencyof suboptimal embeddings
37
![Page 35: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/35.jpg)
Information Isolated Island Issue (4)
Treating each observation as an independent instance• Components work as a black-box
• hardly exhibit the reasons behind a recommendation• Make the decision-making process opaque to understand
InteractionModeling
9𝑦01RepresentationLearning
Data instance(𝑢, 𝑖) or (𝑢, 𝑖, 𝑐)
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
…… ……
(𝒖𝟏 𝒊𝟏 )
Why 𝒊𝟏 is recommended to 𝒖𝟏? Whichone is more important?• Collaborative Signals?• Mouth Marketing?• Item Knowledge?
38
![Page 36: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/36.jpg)
How to Solve Such Issue?
InteractionModeling
9𝑦01RepresentationLearning
Data instance(𝑢, 𝑖) or (𝑢, 𝑖, 𝑐)
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
(𝒖𝟑 𝒊𝟐 )
(𝒖𝟑 𝒊𝟑 )
…… ……
Explore & Exploit Relationsamong Instances
Apply Techniques of GraphLearning & Reasoning
39
![Page 37: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/37.jpg)
References• Greg Linden, Brent Smith, and Jeremy York. Amazon.com recommendations: Item-to-item collaborative filtering. IEEE
Internet Computing• Thomas Hofmann. Latent semantic models for collaborative filtering. ACM Trans. Inf. Syst., 22(1):89–115, 2004.• Stuart James Fitz-Gerald and Bob Wiggins. Introduction to modern information retrieval, 3rd ed., C.G. chowdhury.
facit publishing, london (2010). Int J. Information Management, 30(6):573–574, 2010• Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. Grouplens: An open architecture
for collaborative filtering of netnews. In CSCW, pages 175–186, 1994.• Y. Koren, R. Bell, and C. Volinsky, “Matrix factorization techniques for recommender systems,” Computer, no. 8, pp.
3037, 2009.• S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme, “Bpr: Bayesian personalized ranking from implicit
feedback,” in Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence, ser. UAI ’09, 2009,pp. 452–461.
• S. Kabbur, X. Ning, and G. Karypis, “FISM: factored item similarity models for top-n recommender systems,” in The19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA,August 11-14, 2013, 2013, pp. 659–667.
• Xia Ning, George Karypis: SLIM: Sparse Linear Methods for Top-N Recommender Systems. ICDM 2011: 497-506• Yehuda Koren: Factorization meets the neighborhood: a multifaceted collaborative filtering model. KDD 2008: 426-
434• S. Rendle, “Factorization machines,” in IEEE 10th International Conference on Data Mining, 2010, pp. 995–1000.• Yu-Chin Juan, Yong Zhuang, Wei-Sheng Chin, Chih-Jen Lin: Field-aware Factorization Machines for CTR Prediction.
RecSys 2016: 43-50• X., X. Du, X. Wang, F. Tian, J. Tang, and T. Chua, “Outer product-based neural collaborative filtering,” in Proceedings of
the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018, pp. 2227–2233.• H.-J. Xue, X.-Y. Dai, J. Zhang, S. Huang, and J. Chen, “Deep matrix factorization models for recommender systems,” in
Proceedings of the 26th International Joint Conference on Artificial Intelligence, ser. IJCAI ’17, 2017, pp. 3203–3209.• X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua, “Neural collaborative filtering,” in Proceedings of the 26th
International Conference on World Wide Web, ser. WWW ’17, 2017, pp. 173–182.40
![Page 38: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/38.jpg)
References• J. Chen, H. Zhang, X. He, L. Nie, W. Liu, and T.-S. Chua, “Attentive collaborative filtering: Multimedia recommendation
with item- and component-level attention,” SIGIR ’17, 2017• X. He, Z. He, J. Song, Z. Liu, Y.-G. Jiang, and T.-S. Chua, “Nais: Neural attentive item similarity model for
recommendation,” TKDE, 2018.• F. Xue, X. He, X. Wang, J. Xu, K. Liu, and R. Hong, “Deep itembased collaborative filtering for top-n recommendation,”
TOIS, vol. 37, no. 3, p. 33, 2019.• H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispir et al.,
“Wide & deep learning for recommender systems,” DLRS ’16, 2016, pp. 7–10.• H. Guo, R. Tang, Y. Ye, Z. Li, and X. He, “Deepfm: A factorization-machine based neural network for CTR prediction,”
IJCAI, 2017, pp. 1725–1731.• J. Lian, X. Zhou, F. Zhang, Z. Chen, X. Xie, and G. Sun, “xdeepfm: Combining explicit and implicit feature interactions
for recommender systems,” KDD, 2018, pp. 1754–1763.• Y. Shan, T. R. Hoens, J. Jiao, H. Wang, D. Yu, and J. Mao, “Deep crossing: Web-scale modeling without manually crafted
combinatorial features,” KDD pp. 255–262.• P. Covington, J. Adams, and E. Sargin, “Deep neural networks for youtube recommendations,” RecSys ’16, 2016, pp.
191–198.• X. Wang, X. He, F. Feng, L. Nie, and T.-S. Chua, “Tem: Treeenhanced embedding model for explainable
recommendation,” WWW, 2018, pp. 1543–1552.• R. Wang, B. Fu, G. Fu, and M. Wang, “Deep & cross network for ad click predictions,” in Proceedings of the ADKDD’17,
ser. ADKDD’17, 2017, pp. 12:1–12:7.• Weinan Zhang, Tianming Du, Jun Wang: Deep Learning over Multi-field Categorical Data - - A Case Study on User
Response Prediction. ECIR 2016: 45-57• Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, Jun Wang: Product-Based Neural Networks for User
Response Prediction. ICDM 2016: 1149-1154
41
![Page 39: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/39.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation
• Part II: Random Walk for Recommendation• Random Walk
• Absorption, ItemRank, TriRank, Pixie, RecWalk
• Part III: Network Embedding for Recommendation
• Part III: Graph Neural Networks for Recommendation
42
Slides in https://next-nus.github.io/
![Page 40: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/40.jpg)
Random Walk
Graph Data 𝐺:• 𝑉 is the vertex set• 𝐸 = 𝑉× 𝑉 is the edge set
Random Walk à exhibit high-order proximity among nodes1. Given an initial vertex (node) 𝑣?, select randomly an adjacent node
𝑣@;2. Move to this neighbor 𝑣@ and treat 𝑣@ as the starting node;3. Repeat Steps 1& 2.
B
A
D
C
E
F
A à B à D à E à F
𝑝1B = P(𝑣DE@ = 𝑗|𝑣D = 𝑖) = H1|𝑁1|
, if 𝑖, 𝑗 ∈ 𝐸
0, otherwise
43
![Page 41: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/41.jpg)
Motivation —— Preference Propagation
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
(𝒖𝟑 𝒊𝟐 )
(𝒖𝟑 𝒊𝟑 )
(𝒖𝟑 𝒊𝟒 )…… ……
𝟓
𝟑
𝟒
𝟏
𝟐
𝟒
𝒖𝟐
𝒖𝟑
𝒖𝟏
𝒖𝟒
𝒊𝟏
𝒊𝟐
𝒊𝟑
𝒊𝟒
Interaction GraphInteraction Data
Target User 𝒖𝟐
High-order Proximityà Label Propagation à Preference Distribution• Label (preference) propagation from the target user’s historical
item nodes assigns unseen items with expected labels.
Item ranking 𝑖V > 𝑖X ≈ 𝑖Z
44
![Page 42: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/42.jpg)
Absorption: Random Walk Through View Graph
Absorption from [Baluja et al, WWW’2008]:• Interactions à item-item co-viewed graph or user-item graph
• Edges à two video items are often co-viewed
Bakuja et al, Video Suggestion and Discovery for YouTube: Taking Random Walks Through the View Graph.WWW’2008:
The probability of reaching 𝒖from 𝒗 in one random walk step
The probability of pickinga neighbor 𝒘 of 𝒖
Take a starting node𝒗 for a random walk& output a labeldistribution
45
![Page 43: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/43.jpg)
ItemRank: A Random-Walk Scoring Algorithm
ItemRank from [Gori et al, IJCAI’2007]:• Interactions à item-item correlation graph
• Edges à the shared user groups
• Inspired by Classic PageRank [Kamvar et al., 2003a]:
• ItemRank
Gori et al, ItemRank: A Random-Walk Based Scoring Algorithm for Recommender Engines. IJCAI’2017
Normalized connectivitymatrix for graph
Importance scorefor every node
User preference recordedin training set à bias
Preference scorefor an item node& user profile
Restart
46
![Page 44: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/44.jpg)
TriRank: Ranking over Tripartite Graph
TriRank from [He et al, CIKM’2015]:• User-Item Interactions + Item Aspects à Tripartite Graph
• User 𝒖 previously rated item 𝒑 with mentioning aspect 𝒂
• Ranking score for all nodes of a node• User-User à user similarity;• User-Aspect à interests on aspects• User-Item à preference on items
• Smoothness Constrain
• Fitting Constrain
He et al, TriRank: Review-aware Explainable Recommendation by Modeling Aspects. CIKM’2015
Ranking scores should adhere tothe observations (i.e., initial values).
Local consistency à rankingscores of nearby nodesshould not vary too much
47
![Page 45: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/45.jpg)
Pixie Random Walk
Pixie from [Eksombatchai et al, WWW’2018]:• Undirected Pin-Board Graph
• An edge between a pin 𝒑 and a board 𝒃 if a user saved 𝒑 to 𝒃
• Input: a user-specific input query pin 𝑞• Output: relevant pin 𝑝
Basic Random Walk:• Simulate many random walks on 𝐺, starting from 𝑞• record visit count for each candidate pin 𝑝• The more often 𝑝 is visited à More related it is to 𝑞
Pixie Random Walk:• Bias the random walk towards user-specific pins (𝑞, 𝑈) à personalized results
for even the same query set• Perform queries based on multiple pins (𝑞 ∈ 𝑄, 𝑈) each with a different
importance à consider the history of usersEksombatchai et al, Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time.WWW’2018
48
![Page 46: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/46.jpg)
RecWalk: Nearly Uncoupled Random Walks
RecWalk from [Nikolakopoulos et al, WSDM’2019]:• Interactions à User-Item Bipartite Graph
Transition probability matrix to govern the random sampler
1. One based on bipartite graph
2. The other designed to capture item relations
Recommendation Strategies• the probability the random walker lands on nodes after steps.
Nikolakopoulos et al, RecWalk: Nearly Uncoupled Random Walks for Top-N Recommendation. WSDM’2019
Interaction matrixAdjacency matrix
49
![Page 47: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/47.jpg)
Summary: Random Walk for RecommendationGraph Data Random Walk
Absorption Item-item co-viewed graph Basic
ItemRank Item-item correlation graph User-specific transition probability
TriRank User-item-aspect tripartitegraph
Smoothness & fitting constrains
Pixie Pin-board graph User-specific multi-pin transitionprobability
RecWalk User-item bipartite graph Basic + item relation-guidedtransition probability
Limitations• Efficiency Issue:
• for every user, generate ranking scores on all items each step à hard toapply large-scale graphs
• Lack model parameters to optimize recommendation objective• heuristic-based, rather than learning-based paradigm 50
![Page 48: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/48.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation
• Part II: Random Walk for Recommendation
• Part III: Network Embedding for Recommendation• Network Embedding
• HPE, HOP-Rec, CES
• Part III: Graph Neural Networks for Recommendation
51
Slides in https://next-nus.github.io/
![Page 49: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/49.jpg)
Network Embedding
Also well known as graph representation learning, node embedding,graph embedding.
Input: graph Data 𝐺• 𝑉 is the vertex set• 𝐸 = 𝑉× 𝑉 is the edge set
Output: 𝑍 ∈ 𝑅|e|×f latent feature representation matrix
Snips from: Jure Leskovec et al. Representation Learning on Networks, Tutorial@WWW201852
![Page 50: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/50.jpg)
Intuition
Goal:• Find embedding of nodes to 𝑑–dimensions• Similarity in the embedding space approximates similarity in the
original network
Need to define:• Encoder• Similarity function
Slides from: Jure Leskovec et al. Representation Learning on Networks, Tutorial@WWW201853
![Page 51: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/51.jpg)
Two Main Components
• Encoder• To embed each node to a low-dimension vector representation
• Similarity Function• To specify how relationships in the embedding space map to
relationships in the original network.
Slides from: Jure Leskovec et al. Representation Learning on Networks, Tutorial@WWW2018
ENC(𝑣) = 𝑧l Node in input graph
similarity(𝑣, 𝑢) = 𝑧0q𝑧lSimilarity in the originalnetwork• Are connected?• Share neighbors?• Have similar structural
roles?
Similarity in theembedding space
54
![Page 52: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/52.jpg)
One- & Multi-hop Similarity
Similarity Function• the edge weight between 𝑣 and 𝑢 in the original network.
• One-hop Similarity à Adjacency Similarity• e.g., [Ahmed et al. WWW’2013]
• Multi-hop Similarity à overlap between node multi-hop neighbors• e.g., [Cao et al. CIKM’2015], [Ou et al. KDD’2016]
Slides from: Jure Leskovec et al. Representation Learning on Networks, Tutorial@WWW2018
ℒ = s(0,l)∈t×t
||𝑧0q𝑧l − 𝐴0l||V(Weighted) adjacencymatrix for input graph• Only consider the
existence of directconnectionsAll node pairs
ℒ = s(0,l)∈t×t
||𝑧0q𝑧l − 𝑆0l||V Neighborhood overlapbetween nodes
55
![Page 53: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/53.jpg)
Random Walk-based Similarity
Similarity Function• Probability that 𝑢 and 𝑣 co-occur on a random walk over the graph• E.g., DeepWalk, node2vec …
Slides from: Jure Leskovec et al. Representation Learning on Networks, Tutorial@WWW2018Slide from: Tang Jie et al. Representation Learning on Networks, Tutorial@WWW2019
Random walks
𝑤1
𝑤1yV𝑤1y@
𝑤1E@𝑤1EV
v1 v2v3 v5v3v1 v3v2 v5v1 v5v3 v3v1 v1v2 v3 v4
v4
v3
Skip gram
56
![Page 54: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/54.jpg)
Heterogeneous Preference Embedding (HPE)
HPE from [Yang et al, RecSys’2016]• Interactions + Side information à heterogeneous graph• Random walk similarity
• treat indirect user-item interactions as user context
Yang et al, Query-based Music Recommendations via Preference Embedding. RecSys’2016
Indirect connected nodes as thecontextual information of egonode
Learn the preference embeddingsvia contextual nodes
Random walk steps
57
![Page 55: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/55.jpg)
HOP-Rec: High-Order Proximity for Recommendation
HOP-Rec from [Yang et al, RecSys’2018]• Interactions à user-item bipartite graph• Random walk similarity
• In a path, nodes with different orders à different confidence• Involve indirect user-item interactions into user preference
Yang et al, HOP-Rec: High-Order Proximity for Implicit Recommendation. RecSys’201858
![Page 56: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/56.jpg)
HOP-Rec: High-Order Proximity for Recommendation
HOP-Rec from [Yang et al, RecSys’2018]
Yang et al, HOP-Rec: High-Order Proximity for Implicit Recommendation. RecSys’2018
A random walk with a decay factor forconfidence weighting 𝑪(𝒌)• For a given walk sequence, item with
order 𝑘 that the user potentialprefers à treated as positiveinstances
Matrix Factorization with BPR loss• Random walk enriches the positive
observations
59
![Page 57: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/57.jpg)
Collaborative Similarity Embedding (CES)
CES from [Chen et al, WWW’2019]• Interactions à user-item bipartite graph
Chen et al, Collaborative Similarity Embedding for Recommender Systems. WWW’2019.
Maximize the likelihood ofobserved pairs
• Conduct k-step random walk• Get neighborhoods of a user (or item)
pair• Neighborhood proximity à user (or item)
similarity60
![Page 58: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/58.jpg)
Summary: Network Embedding forRecommendation
Graph Data Connectivity/Proximity
HPE Heterogeneous graph Indirect connections
HOP-Rec User-item bipartite graph • Direct connections• Indirect connections
CES User-item bipartite graph • Direct similarity• Neighborhood proximity similarity
Limitations• Not end-to-end Learning:
• Random walk is conducted first to get multi-hop neighbors
• Not fully explore high-order connectivity• Multi-hop neighbors are used to enrich the training data, rather than
directly contributing to the representation learning61
![Page 59: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/59.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation
• Part II: Random Walk for Recommendation
• Part III: Network Embedding for Recommendation
• Part III: Graph Neural Networks for Recommendation• Graph Neural Networks
62
Slides in https://next-nus.github.io/
![Page 60: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/60.jpg)
Recent Graph Neural Network (GNN) Research
2017 GCN
2017 GraphSage
2018 GAT
2018 SGC
63
![Page 61: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/61.jpg)
Graph Data 𝐺• 𝑉 is the vertex set• 𝐸 = 𝑉× 𝑉 is the edge set
• Undirected: social relations, user-item interactions …• Directed: triplets in knowledge graph
• 𝐴 ∈ 𝑅|e|×|e| is the adjacency matrix
• 𝐴1,B = }𝑎1,B > 0 𝑖, 𝑗 ∈ 𝐸0 𝑖, 𝑗 ∉ 𝐸
• 𝑋 ∈ 𝑅f×|e| is a matrix of node features• Categorical attributes, text, image data• Node degrees, clustering coefficient, …• Indicator vectors (i.e., one-hot encoding of each node)
• 𝑋′ ∈ 𝑅f�×|t| is a matrix of edge features• Relations 64
![Page 62: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/62.jpg)
Graph Convolution Network (GCN)• At the core of GCN
1. Model a local structural information (neighborhood) of a nodeas the receptive field
2. Apply the graph convolution operation• Spectral domain:
• Laplacian eigen-decomposition [Bruna et al. ICLR’2014]
• Chebyshev polynomials [Defferrard et al, NeurIPS’2016]
• However, computationally expensive
• Spatial domain:• Node (Neighborhood) aggregation [William et al, NeurIPS’2017]
3. Update its representation.• 𝑍 ∈ 𝑅f×|e| latent feature representation matrix
65
![Page 63: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/63.jpg)
Neighborhood Aggregation (1)
Key Idea:• Generate node embeddings based on local neighbors• Neighborhood defines a computation graph.
B
A
D
C
E
A
B
C
E
Computation Graph ofnode A
F
?One-hop Neighborhood of node A:
𝑵� = {𝐵, 𝐶, 𝐸}
66
![Page 64: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/64.jpg)
Neighborhood Aggregation (2)
Message Passing/ Information Propagation:• A node aggregates information from their neighbors via neural
networks
B
A
D
C
EOne-hop Neighborhood of node A:
𝑵� = {𝐵, 𝐶, 𝐸}
Computation Graph ofnode A
F
A
B
C
E
One-hop Neighborhood Aggregation:𝒉�� = 𝑓(𝒉�, 𝒉�, 𝒉t)
67
![Page 65: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/65.jpg)
Neighborhood Aggregation (3)
Moreover:• Every neighboring node has its won computation graph!
B
A
D
C
E
Computation Graph ofeach node
F
B
C
E
A
D
A
A
D
F
Two-hop Neighborhood of node A:𝑵� = {𝐷, 𝐹}
Unseen in one-hopneighborhood
68
![Page 66: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/66.jpg)
Neighborhood Aggregation (4)
Stacking more neighborhood aggregation layers• Nodes have embeddings at each layer• Model can be arbitrary depth• At Layer 0, embedding of node v ∈ 𝑉 is its input feature, i.e., 𝑥l.
A
B
C
E
A
D
A
A
D
Layer 2 Layer 0Layer 1
𝑥�
𝑥�
𝑥�
𝑥�
𝑥�
F 𝑥�
Slide from: Jure Leskovec et al. Representation Learning on Networks, Tutorial@WWW2018
B
A
D
C
E
F
69
![Page 67: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/67.jpg)
Graph Convolution
Neighborhood aggregation can be viewed as a center-surround filter inconvolutional neural network (CNN).
Slide from: Jure Leskovec et al. Representation Learning on Networks, Tutorial@WWW2018
Mathematically related to spectral graph convolutions (Bronstein et al., 2017)
Now• How to aggregate information across layers!• i.e., how to design the neural networks!
70
![Page 68: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/68.jpg)
Component 1: Information Construction
Generally speaking, the first main component:1. Information Construction
Construct the information being propagated from one neighboringnode to the target node.
𝑚l→0(Dy@) = 𝑓@
(Dy@) (ℎlDy@ , ℎ0
Dy@ , 𝑝l0(Dy@))
The information beingpropagated from node𝑣 to the target 𝑢
Nodes’previous layerembeddings
Decay factor ornormalization
term
Neuralnetwork
A
B
C
E71
![Page 69: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/69.jpg)
Component 2: Neighborhood Aggregation
Generally speaking, the second main component:2. Neighborhood Aggregation
Aggregate the information from the whole neighborhood.
ℎ𝒩�(Dy@) = 𝑓V
(Dy@) ( ℎlDy@ , for 𝑣 ∈ 𝒩0)
The informationcombining neighbors’
information
The informationobtained from the first
component
AggregationFunction
A
B
C
E72
![Page 70: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/70.jpg)
Component 3: Representation Update
Generally speaking, the third main component:3. Representation Update
Integrate the neighborhood information with its ownrepresentation.
ℎ0(D) = 𝑓X
(Dy@) (ℎ𝒩�Dy@ , ℎ0
(Dy@))
The updatedrepresentation of 𝑢 at
the 𝑘-th layer
The representation ofthe target node 𝑢 atthe (𝑘 − 1)-th layer
IntegrationFunction
A
B
C
E73
![Page 71: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/71.jpg)
Graph Convolutional Network (GCN)
GCN from [Kipf et al., ICLR’2017]:
Kipf et al., ICLR’2017: Semisupervised Classification with Graph Convolutional Network.
ℎ0(D) = 𝜎 ( 𝑊D s
l∈𝒩� ∪0
ℎlDy@
𝒩0 𝒩l)
per-neighbor normalization:𝑝0l(Dy@) in Comp. 1
• Normalization varies acrossneighbors
• Down-weights high degreeneighbors
Weighted sum:Aggregation in
Comp. 2
The same neural networkfor self and neighborembeddings in Comp.3• More parameter sharing
Nonlinearactivation function
74
![Page 72: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/72.jpg)
GCN in Matrix Form
Can be rewritten in the matrix form• Which is efficiently implemented using sparse batch operations
• Time complexity is 𝑂 𝐸
Kipf et al., ICLR’2017: Semisupervised Classification with Graph Convolutional Network.
𝐻(D) = 𝜎 (𝐷y@V 𝐴 + 𝐼 𝐷y
@V𝐻(Dy@)𝑊D)
Laplacian Matrix for Graph 𝐺
Representation Matrixat the 𝑘-th layer
𝑍 = 𝐻(�)
Final representationmatrix
75
![Page 73: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/73.jpg)
GraphSage
GraphSage from [Hamilton et al., NeurIPS’2017]:• The most distinction is the generalized aggregation
Hamilton et al., NeurIPS’2017. Inductive Representation Learning on Large Graphs.
ℎ0(D) = 𝜎 ( 𝑊@
D AGG ℎlDy@ , ∀𝑣 ∈ 𝒩0 ,𝑊V
D ℎ0(Dy@) )
ℎ𝒩�(Dy@) = 𝑓V
(Dy@) ( ℎlDy@ , for 𝑣 ∈ 𝒩0)
Generalized Aggregation Function in Comp. 2à Any differentiable function that maps a set of vectors to a single vector.
Integration Function in Comp.3 à concatenateneighbors & self embeddings, instead of sum
76
![Page 74: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/74.jpg)
Generalized Aggregation in GraphSage• Mean
• Pool• Transform neighbor vectors and apply symmetric vector
function.
• LSTM• Apply LSTM to random permutation of neighbors.
Hamilton et al., NeurIPS’2017. Inductive Representation Learning on Large Graphs.Slide from: Jure Leskovec et al. Representation Learning on Networks, Tutorial@WWW2018
AGG = s∀l ∈𝒩�
ℎlDy@
|𝒩l|
AGG = 𝛾({𝑄ℎlDy@ , ∀𝑣 ∈ 𝒩0})
AGG = LSTM([ℎlDy@ , ∀𝑣 ∈ 𝒩0])
Element-wise mean/max
77
![Page 75: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/75.jpg)
Graph Attention Network (GAT)
GAT from [Velickovic et al., ICLR’2018]:• The most distinction is the attentive neighborhood aggregation
ℎ𝒩�(Dy@) = 𝑓V
(Dy@) ( ℎlDy@ , for 𝑣 ∈ 𝒩0)
Attentive Aggregation Function in Comp. 1&2à Different neighbors have varying contributions when propagating information.à Instead of fixed heuristic-based decay factor like GCN, GraphSage
Velickovic et al. Graph Attention Networks. ICLR 2018
A
B
C
E
0.5
0.3
0.2
ℎ0(D) = 𝜎( s
l∈𝒩�
𝛼0,l 𝑊(D)ℎl(Dy@))
Learned attentive weights
78
![Page 76: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/76.jpg)
Attention Weights in GAT• Attention Network
• Multi-head Attention
Velickovic et al. Graph Attention Networks. ICLR 2018
ℎ0(D) = 𝜎(
1𝐿s®@
¯
sl∈𝒩�
𝛼0,l() 𝑊(,D)ℎl
(Dy@))
With 𝐿 Heads
79
![Page 77: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/77.jpg)
Simple Graph Convolution (SGC)
SGC from [Wu et al., ICML’2019]:• Unnecessary complexity & redundant computation of GCN
Wu et al. Simplifying Graph Convolutional Networks. ICML 2019
Remove the nonlinearities between GCN layers
Use a single linear transformation
80
![Page 78: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/78.jpg)
SGC in Matrix Form
• SGC improves the efficiency of GCN largely without sacrificing accuracy, & even outperforms GCN on some tasks.
Wu et al. Simplifying Graph Convolutional Networks. ICML 2019
𝐻(D) = 𝜎 (𝐷y@V 𝐴 + 𝐼 𝐷y
@V𝐻(Dy@)𝑊D)
Only linear featurepropagation is remained
𝑍 = 𝐻(�)GCN:
𝐻(D) = 𝐷y@V 𝐴 + 𝐼 𝐷y
@V𝐻(Dy@) 𝑍 = 𝐻(�)SGC:
81
![Page 79: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/79.jpg)
Training• After 𝐾 graph convolution layers (e.g., GCN, GraphSage, GAT, SGC),
we get output embeddings for each node.
• Upon these embeddings, we can define a loss function for a specifictask:
• Run stochastic gradient descent to train the aggregation parameters
𝑍 = 𝐻(�)
Node classificationℒ(𝑍0)
Graph classificationℒ(𝑍±)
Link predictionℒ(𝑍0, 𝑍l)
B
A
D
C
E
F
B
A
D
C
E
F
B
A
D
C
E
F
82
![Page 80: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/80.jpg)
e.g., Node Classification Task
ℒ = ∑l∈e 𝑦l log 𝜎 𝑧lq𝜃 + (1 − 𝑦l)log 1 − 𝜎 𝑧lq𝜃
B
A
D
C
E
F
Estimate the label of the target node:• positive or negative?• Belonging to one of 𝐶 classes
Ground-truth labelTrainable weights in the
classifier
GNN embeddings can be plug-and-play & serve other semi-supervised & unsupervised & supervised tasks.
83
![Page 81: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/81.jpg)
Recent Research on GNN• More Details in Previous Tutorials:
• Jure Leskovec et al. Representation Learning on Networks,Tutorial@WWW2018
• Hamiltion & Jie Tang. Graph Representation Learning, Tutorial@AAAI2019• Jie Tang et al. Representation Learning on Networks, Tutorial@WWW2019
• More Details in Survey Papers:• Zhou et al., Graph Neural Networks: A Review of Methods and Applications• Zhang et al., Deep Learning on Graphs: A Survey• Wu et al., A Comprehensive Survey on Graph Neural Networks
• More Paper Collections in Github:• https://github.com/thunlp/GNNPapers#survey-papers• https://github.com/naganandy/graph-based-deep-learning-literature
84
![Page 82: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/82.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation
• Part II: Random Walk for Recommendation
• Part III: Network Embedding for Recommendation
• Part III: Graph Neural Networks for Recommendation• Collaborative Filtering: GC-MC, SpectralCF, NGCF
85
Slides in https://next-nus.github.io/
![Page 83: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/83.jpg)
Recap Collaborative Filtering (CF)
• Collaborative Signals à Behavior Similarity of Users• Similar users would have similar preference on items.
• User-Item Interaction Data à User-Item Bipartite Graph• Edges indicate the user behaviors.
(𝒖𝟏 𝒊𝟏 )
(𝒖𝟐 𝒊𝟏 )
(𝒖𝟐 𝒊𝟐 )
(𝒖𝟑 𝒊𝟐 )
(𝒖𝟑 𝒊𝟑 )
(𝒖𝟑 𝒊𝟒 )…… ……
𝟓
𝟑
𝟒
𝟏
𝟐
𝟒
5 ? ? ? …
3 4 ? ? …
? 1 2 4 …
... … … … …
𝟏
𝟐
𝟑
𝟏 𝟐 𝟑 𝟒
user
item
Interaction Matrix
𝒖𝟐
𝒖𝟑
𝒖𝟏
𝒖𝟒
𝒊𝟏
𝒊𝟐
𝒊𝟑
𝒊𝟒
Interaction Graph86
![Page 84: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/84.jpg)
Graph Convolutional Matrix Completion (GC-MC)
GC-MC from [Rianne et al., KDD’2018]• View matrix completion as link prediction on interaction graph
• Rating Prediction à predict links in bipartite user-item graph
Rianne et al. Graph Convolutional Matrix Completion. KDD 2018
Generate high-quality embeddings of users anditems on the graph in an end-to-end fashion• Previous solutions separate the graph feature
model and link prediction model87
![Page 85: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/85.jpg)
Graph Convolutional Encoder in GC-MC
• Comp.1: Information Construction:
• Comp.2: Neighborhood Aggregation:
• Comp.3: Representation Update:
Rianne et al. Graph Convolutional Matrix Completion. KDD 2018
Different 𝑊µ are assigned to differentrating level 𝑟.
Accumulation operation over neighborsat all rating levels
88
![Page 86: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/86.jpg)
Rating Prediction in GC-MC• Rating Prediction
• Model Training
• Others:• One graph convolution layer achieved the best performance.• Structural information can be combined with interaction graph
• Social networks, knowledge graphs, …
Rianne et al. Graph Convolutional Matrix Completion. KDD 2018
Trainable weights fordifferent rating levels
Negative log likelihood
89
![Page 87: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/87.jpg)
Spectral Collaborative Filtering
Zheng et al. Spectral Collaborative Filtering. RecSys 2018
SpectralCF from [Zheng et al., RecSys’2018]• User-Item Interaction Graph
• GC-MC: use existing connectivity• SpectralCF: discover hidden connectivity in the spectral domain
The connectivity between 𝑢@ and 𝑖V, 𝑖X, 𝑖Z• Uncovered in the frequency domain• Discovered in the spectral domain
90
![Page 88: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/88.jpg)
Spectral Convolution Filtering in SpectralCF
Zheng et al. Spectral Collaborative Filtering. RecSys 2018
However, eigen-decomposition of graph adjacency matrix is required• A rather high complexity• Difficult to support large-scale graphs
Spectral Convolution Filtering
Polynomial Approximation
91
![Page 89: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/89.jpg)
Neural Graph Collaborative Filtering (NGCF)
NGCF from [Wang et al., SIGIR’2019]• Revisit CF via high-order connectivity
• The paths that reach 𝑢@ from any node with the path length 𝑙larger than 1 à unseen connectivity argued in SpectralCF!
• A natural way to encode collaborative signal in the interactiongraph structure
Wang et al. Neural Graph Collaborative Filtering. SIGIR 2019
Why 𝑢@ may like 𝑖Z?• 𝑢@ ← 𝑖V ← 𝑢V ← 𝑖Z• 𝑢@ ← 𝑖X ← 𝑢X ← 𝑖Z
92
![Page 90: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/90.jpg)
First-order Connectivity Modeling
Wang et al. Neural Graph Collaborative Filtering. SIGIR 2019
Inspired by GNNs1. Propagate embeddings recursively on the user-item graph2. Construct information flows in the embedding space
• Comp.1: Information Construction:
• Comp.2 & 3: Neighbor Aggregation & Representation Update:
message passed from 𝑖 to 𝑢
• message dependent on the affinity,distinct from GCN, GraphSage, etc.
• Pass more information to similar nodes
all neighbors of 𝑢self-connections
discount factor
93
![Page 91: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/91.jpg)
High-order Connectivity Modeling
Wang et al. Neural Graph Collaborative Filtering. SIGIR 2019
• Stack more embedding propagation layers to explore the high-orderconnectivity
• The collaborative signal like u1 ← i2 ← u2 ← i4 can be captured inthe embedding propagation process.
• Collaborative signal can be injected into the representationlearning process.
94
![Page 92: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/92.jpg)
Overall Framework
Wang et al. Neural Graph Collaborative Filtering. SIGIR 2019
The representations at different layers• emphasize the messages
passed over different connections
• have different contributions in reflecting user preference
95
![Page 93: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/93.jpg)
Experiment Results —— Sparsity Issue
Wang et al. Neural Graph Collaborative Filtering. SIGIR 2019
user groups with different group sparsity levels
• NGCF consistently outperforms all other baselines on most user groups.
• Exploiting high-order connectivity facilitates the representation learning for inactive users.
• It might be promising to solve the sparsity issue in recommender systems96
![Page 94: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/94.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation
• Part II: Random Walk for Recommendation
• Part III: Network Embedding for Recommendation
• Part III: Graph Neural Networks for Recommendation• Social Recommendation: GraphRec, DiffNet, DANSER
102
Slides in https://next-nus.github.io/
![Page 95: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/95.jpg)
tie
Influence
•User Profiling•Behavior Modeling
•Social Tie•Social Influence
•Triad & Group Formation• Information Diffusion
Social Network Analysis
Social Recommendation
Social relation is of importance to help users filter information• Tow graphs à user-item interaction graph + user-user social graph.
Snips from: Tang Jie et al. Representation Learning on Networks, Tutorial@WWW2019103
![Page 96: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/96.jpg)
Graph Neural Networks for SocialRecommendation (GraphRec)
Fan et al. Graph Neural Networks for Social Recommendation. WWW 2019
GraphRec from [Fan et al, WWW’2019]• User-Item Graph
• Interactions between users and items• Users’ opinions on items (i.e., explicit feedback, ratings)
• User-User Graph• Social relations have heterogeneous strengths
• Strong & weak ties are mixed together• Users are likely to share more similar tastes with strong ties than weak ties.
104
![Page 97: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/97.jpg)
User Modeling in GraphRecThese two graphs provide user information from different angles• Item Aggregation
• Item space: leverage user-item interactions to get user representations
• Social Aggregation• Social space: use social relationships to get user representations
Opinion-aware representation of an interaction
105
![Page 98: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/98.jpg)
Item Modeling in GraphRec• User Aggregation
• Consider both interactions & opinions to get item representations
• Rating Prediction• Feed the concatenation of user & item representation into a neural network
(MLP) to get predictions.
Opinion-aware representation of an interaction
Attention network to differentiate the importance weight
106
![Page 99: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/99.jpg)
Neural Influence Diffusion Model (DiffNet)DiffNet from [Wu et al, SIGIR’2019]• Social Influence in Social Recommendation
• A user’s preference is influenced by her trusted users, with these trustedusers are influenced by their own social connections à high-orderconnectivity
• Social influence recursively propagates & diffuses in social network!
Wu et al. A Neural Influence Diffusion Model for Social Recommendation. SIGIR 2019
One-hop trusted users Two-hop trusted users
107
![Page 100: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/100.jpg)
Diffusion Influence Aggregation in DiffNet
• Comp.1: Information Construction
• Comp.2: Diffusion Influence Aggregation
• Comp.3: Representation Update
Wu et al. A Neural Influence Diffusion Model for Social Recommendation. SIGIR 2019
User features
User ID Embeddings
Ego social network
108
![Page 101: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/101.jpg)
DANSERDANSER from [Wu et al, WWW’2019]
Wu et al. Dual Graph Attention Networks for Deep Latent Representation of Multifaceted Social Effects in Recommender Systems. WWW 2019
Social homophily• User static preference• Unchanged & independent of
external contexts
Social influence• User dynamic preference• Change dynamically with
specific contexts
109
![Page 102: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/102.jpg)
DANSERDANSER from [Wu et al, WWW’2019]
Wu et al. Dual Graph Attention Networks for Deep Latent Representation of Multifaceted Social Effects in Recommender Systems. WWW 2019
Item-to-item homophily• Item static attribute
Item-to-item influence• Item dynamic attribute• Depends on a specific context
110
![Page 103: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/103.jpg)
Dual GAT in User & Item Domains
Wu et al. Dual Graph Attention Networks for Deep Latent Representation of Multifaceted Social Effects in Recommender Systems. WWW 2019
111
![Page 104: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/104.jpg)
Summary: GNN for Social Recommendation
Graph Data User Modeling Item ModelingGraphRec • Interaction Graph
• Social NetworkFirst-order Connectivity• historical items• social relations
First-order Connectivity• user feedback
DiffNet • Social Network High-order Connectivity• social influence
First-order Connectivity• user feedback
DANSER • Interaction Graph• Social Network
First-order Connectivity• social homophily
(static)• social influence
(dynamic)
First-order connectivity• item homophily
(static)• item influence
(dynamic)
Social recommendation needs more guides from social network analysis:• Behavior modeling, social influence, group formation, information
diffusion à from micro to macro! 112
![Page 105: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/105.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation
• Part II: Random Walk for Recommendation
• Part III: Network Embedding for Recommendation
• Part III: Graph Neural Networks for Recommendation• Sequential Recommendation: SR-GNN, DGRec
113
Slides in https://next-nus.github.io/
![Page 106: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/106.jpg)
Sequential Recommendation
Sequential (Session-based) recommendation• Given historical interactions, to predict the successive items that a
user is likely to interact with à sequential needs of users
• User interests are dynamic in sessions.• Sequential pattern is of crucial importance.
Listen To Your Heart Dreaming Alone The Oak Tree Follow Your Heart
114
![Page 107: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/107.jpg)
Session-based Recommendation with GNNs (SR-GNN)
SR-GNN from [Wu et al, AAAI’2019]• Sequential pattern of a transition à one session sequence• Complex patterns of item transitions à all session sequences
Wu et al. Session-based Recommendation with Graph Neural Networks. AAAI 2019
Reorganize all session sequences into graphstructured data à session graph• A directed graph 𝐺¾ over items• Each edge (𝑣¾,1y@, 𝑣¾,1) means a user
clicks item 𝑣¾,1 after 𝑣¾,1y@ in session 𝑠
Ø Present the global preference in session𝑠
treating a session sequences as a sessionsubgraphØ Present the current interest &
sequential needs of the user in session 𝑠115
![Page 108: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/108.jpg)
Session Graphs with GNNs
Wu et al. Session-based Recommendation with Graph Neural Networks. AAAI 2019
𝐴¾ = [𝐴¾(À0Á), 𝐴¾
(1Â)]RNN Aggregator over
subgraphs
𝑦1 = MLP(Att(𝑣Â, 𝑣1))
Prediction conducted by aMLP + Attention network
116
![Page 109: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/109.jpg)
Dynamic Graph Attention Network for Session-basedSocial Recommendation (DGRec)
DGRec from [Song et al, WSDM’2019]• Session + Social Recommendation à social networks
• User interests change across sessions, due to:• Short-term preferences of user friends• Long-term preferences of user friends
Song et al. Session-based Social Recommendation via Dynamic Graph Attention Networks. WSDM 2019117
![Page 110: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/110.jpg)
Dynamic Social Recommendation
• Dynamic individual interest in current session à RNN
• Friends’ interest• Short-term preferences à a friend’s latest online session
• Long-term preferences à a friend’s average interests, which is no item-sensitive
• Unified representation
Song et al. Session-based Social Recommendation via Dynamic Graph Attention Networks. WSDM 2019
User sessionrepresentation
LSTM adopted on thesession sequence
Session sequence ofthe user
User IDembedding
118
![Page 111: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/111.jpg)
Dynamic Graph Attention Network
• Dynamic feature graph• User + Friends graph• Dynamic features à updated whenever a user consumed a new item.
• Attentive social aggregation
Song et al. Session-based Social Recommendation via Dynamic Graph Attention Networks. WSDM 2019
Level of influence or importanceof a friend to the target user
Final representation combining usersession interests & social influence
119
![Page 112: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/112.jpg)
Summary: GNN for Sequential Recommendation
Graph Data User Modeling Item Modeling
SR-GNN Directed SessionGraph
• Global preference in allsession sequences
• Local preference in asession sequence
• Without ID information
Graph representations
DGRec Social Network • Short-term preference inthe latest session(sequential pattern)
• Long-term preference inall sessions (IDInformation)
• Social influence (graph)
ID embeddings
Sequential recommendation needs new & reasonable angles to organizesequence data in the form of graph.
120
![Page 113: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/113.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation
• Part II: Random Walk for Recommendation
• Part III: Network Embedding for Recommendation
• Part III: Graph Neural Networks for Recommendation• Knowledge Graph-based Recommendation: KGCN, KGNN-LS,
KGAT
121
Slides in https://next-nus.github.io/
![Page 114: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/114.jpg)
Knowledge Graph-based Recommendation
Knowledge Graph (KG):• Background knowledge on
items• Rich semantics & Relations• Structural information
Benefit for Recommendation• Narrow down search space• Explore user interests
reasonably• Offer explanations
𝒖𝟏 𝒖𝟐
𝒊𝟏Shape of You
𝒊𝟐I See Fire
𝒊𝟑Skin
𝒊𝟒Castle on the Hill
𝒓 𝟏
Interact 𝒓𝟏Interact
𝒓𝟏Interact
𝒆𝟏Ed Sheeran
𝒆𝟐÷ 𝒆𝟑
Pop
𝒆𝟒Folk
𝒓𝟐IsSongOf 𝒓𝟑
SungBy𝒓𝟒
Genre
𝒓𝟓Genre
122
![Page 115: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/115.jpg)
Prior Works: Supervised Learning-based
Feature Engineering• 𝑢@- 𝑖Z interaction as an data instance• transfer item knowledge into a feature
vector is �⃗�=<𝑢@, 𝑖Z, 𝑒@, 𝑒V, 𝑒Z>
𝒖𝟏 𝒖𝟐
𝒊𝟏Shape of You
𝒊𝟐I See Fire
𝒊𝟑Skin
𝒊𝟒Castle on the Hill
𝒓 𝟏
Interact 𝒓𝟏Interact
𝒓𝟏
Interact
𝒆𝟏Ed Sheeran
𝒆𝟐÷ 𝒆𝟑
Pop
𝒆𝟒Folk
𝒓𝟐IsSongOf 𝒓𝟑
SungBy𝒓𝟒
Genre
𝒓𝟓Genre
To estimate 𝒖𝟏′s preference on 𝒊𝟐
Prediction Modeling• A supervised learning model• e.g., FM, NFM, Wide&Deep ……
Limitations• Semantic relations are ignored• Graph structure is ignored• CF signals are captured in an implicit fashion• High-order connectivity/relation are ignored 123
![Page 116: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/116.jpg)
Prior Works: Regularized-based
Representation Learning• 𝑖Z-related KG triplets regularize the
learning of its representation• Translational Principle
• Head + Relation ≈ Tail
𝒖𝟏 𝒖𝟐
𝒊𝟏Shape of You
𝒊𝟐I See Fire
𝒊𝟑Skin
𝒊𝟒Castle on the Hill
𝒓 𝟏
Interact 𝒓𝟏Interact
𝒓𝟏
Interact
𝒆𝟏Ed Sheeran
𝒆𝟐÷ 𝒆𝟑
Pop
𝒆𝟒Folk
𝒓𝟐IsSongOf 𝒓𝟑
SungBy𝒓𝟒
Genre
𝒓𝟓Genre
To estimate 𝒖𝟏′s preference on 𝒊𝟐
Interaction Modeling• Reconstruct direct user-item
interactions• e.g., NCF, MF, …
Limitations• High-order connectivity between user and item
nodes are modeled in an implicit fashion• It fails to synthesize high-order relations. 124
![Page 117: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/117.jpg)
Prior Works: Path-based
Representation Learning• Paths connecting 𝑢@ and 𝑖Z to represent
their connectivity• 𝑢@ → 𝑖@ → 𝑒@ → 𝑖V.
𝒖𝟏 𝒖𝟐
𝒊𝟏Shape of You
𝒊𝟐I See Fire
𝒊𝟑Skin
𝒊𝟒Castle on the Hill
𝒓 𝟏
Interact 𝒓𝟏Interact
𝒓𝟏
Interact
𝒆𝟏Ed Sheeran
𝒆𝟐÷ 𝒆𝟑
Pop
𝒆𝟒Folk
𝒓𝟐IsSongOf 𝒓𝟑
SungBy𝒓𝟒
Genre
𝒓𝟓Genre
To estimate 𝒖𝟏′s preference on 𝒊𝟐
Interaction Modeling• Information fusion of multiple paths• A supervised learning model
Limitations• Require domain knowledge to define meta-paths• Require labor-intensive feature engineering to
extract qualified paths• Have rather high complexity 125
![Page 118: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/118.jpg)
Knowledge Graph Convolution Network (KGCN)
KGCN from [Wang et al, WWW’2019]• Item graph à KG entities are used to enrich item representation
Wang et al. Knowledge Graph Convolutional Networks for Recommender Systems. WWW 2019
Comp.1 & 2 Comp.3 Prediction
KG entities connectedwith the target item
Attention scoreof user-relation
User ID embeddings àusers are excluded from
the propagation.127
![Page 119: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/119.jpg)
Knowledge Graph Neural Networks with LabelSmoothness Regularization (KGNN-LS)
KGNN-LS is an extension of KGCN from [Wang et al, KDD’2019]• Label smoothness à adjacent items in KG are likely to have similar
user relevance labels/scores.
Wang et al. Knowledge-aware Graph Neural Networks with Label Smoothness Regularization forRecommender Systems. KDD 2019
128
![Page 120: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/120.jpg)
Knowledge Graph Attention Network (KGAT)
KGAT from [Wang et al, KDD’2019]
Wang et al. KGAT: Knowledge Attention Network for Recommendation. KDD 2019
Knowledge Graph• Item-Item External Connections
𝑖@ →µÅ 𝑒@
User-Item Bipartite Graph• User-Item Direct Interactions
𝑢@→µÆ 𝑖@
Collaborative Knowledge Graph• High-order connectivity between users
and items𝑢@→
µÆ 𝑖@ →µÅ 𝑒@
yµÅ 𝑖V è 𝑢@→µÆ 𝑖V
• Reasoning ability & Explainability
𝒖𝟏 𝒖𝟐
𝒊𝟏Shape of You
𝒊𝟐I See Fire
𝒊𝟑Skin
𝒊𝟒Castle on the Hill
𝒓 𝟏
Interact 𝒓𝟏Interact
𝒓𝟏
Interact
𝒆𝟏Ed Sheeran
𝒆𝟐÷ 𝒆𝟑
Pop
𝒆𝟒Folk
𝒓𝟐IsSongOf 𝒓𝟑
SungBy𝒓𝟒
Genre
𝒓𝟓Genre
129
![Page 121: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/121.jpg)
Attentive Embedding Propagation in KGAT𝒖𝟏 𝒖𝟐
𝒊𝟏Shape of You
𝒊𝟐I See Fire
𝒊𝟑Skin
𝒊𝟒Castle on the Hill
𝒓 𝟏
Interact 𝒓𝟏Interact
𝒓𝟏Interact
𝒆𝟏Ed Sheeran
𝒆𝟐÷ 𝒆𝟑
Pop
𝒆𝟒Folk
𝒓𝟐IsSongOf 𝒓𝟑
SungBy𝒓𝟒
Genre
𝒓𝟓Genre
Attentive Embedding Propagation,inspired by GNNs• Propagate embeddings recursively
on the graph
• Reveal the importance of a high-order connectivity via relation-aware attentions
• Construct information flows in theembedding space
130
![Page 122: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/122.jpg)
Attentive Embedding Propagation in KGAT
• Comp.1: Information Propagation
• Comp.2: Knowledge-aware Attention Aggregation
• Comp.3: Representation Update
𝒊𝟑
𝒖𝟐
𝒆𝟏
𝒆𝟑
𝒓𝟏𝒓𝟐
𝒓𝟑The messages accounting forfirst-order connectivity
The set of triples, where thetarget node is the head entity
Tail representation
decay factor on each propagation
the attention score is dependent on the distance of 𝑒Á and 𝑒Ç in 𝑟’s space
Similar to NGCF131
![Page 123: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/123.jpg)
Model Training
Similar to NGCF, the representations at different layers• emphasize the messages passed
over different connections• have different contributions in
reflecting user preference
132
![Page 124: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/124.jpg)
Experiment Results —— Sparsity Issue
user groups with different group sparsity levels
• KGAT outperforms the other models in most cases, especially on the two sparsest user groups in Amazon-Book and Yelp2018.
• It again verifies the significance of high-order connectivity modeling:• contains the lower-order connectivity• enriches the representations of inactive users via recursive embedding
propagation 133
![Page 125: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/125.jpg)
Case Study for Explainable Recommendation
• KGAT captures the behavior-based and attribute-based high-order connectivity, which play a key role to infer user preferences.
• The explanation can be “The Last Colony is recommended since you have watched Old Man’s War written by the same author John Scalzi.” 134
![Page 126: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/126.jpg)
© Copyright NExT++. All Right Reserved.
OUTLINE
• Introduction
• Part I: Preliminary of Recommendation
• Part II: Random Walk for Recommendation
• Part III: Network Embedding for Recommendation
• Part III: Graph Neural Networks for Recommendation
• Summary & Future Directions
135
Slides in https://next-nus.github.io/
![Page 127: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/127.jpg)
Summary
The data is more closely connected that we might think!
User-Item Interactions User/Item Profiles Knowledge Graph
𝒖𝟏
𝒖𝟐
𝒖𝟑
𝒊𝟏
𝒊𝟐
𝒊𝟑
𝒊𝟒
𝒊𝟓
𝒖𝟏
𝒖𝟑
Age 18-24
Gender: Male
Geo: Beijing
Age 25-30
𝒊𝟑Logan
𝒊𝟏The Greatest Showman
𝒊𝟐X-Men
𝒆𝟐Hugh Jackman
𝒓𝟒DirectedBy
𝒓𝟑IsCastOf
𝒆𝟏James Mangold
𝒊𝟒The Prestige
𝒓𝟐ProducedBy
𝒆𝟏James Mangold
Limited Representation Ability
Suboptimal Model Capacity
Limited Reasoning Ability
Information Propagation along with the connections
High-order connectivity complementary to user-iteminteractions
High-order connectivity interpreting user intents136
![Page 128: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/128.jpg)
• Dynamic Graphs• Graph for recommendation evolves over time
• Online User-Item Interactions, Trending of (fashion) items,CTR prediction …
• Challenges:• How to efficiently & incrementally update representations?• How to incorporate edge timing?• How to forget old/irrelevant information? …
• Adversarial Learning• Attack & Defense
• Node Features + Edge Features + Graph Structure• Applications:
• Malicious detection, Fraud detection …137
Future Direction (1)
![Page 129: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/129.jpg)
• Casual Inference• Get intents behind user behaviors
• What contexts à what behaviors are reasonable• Towards explainable recommendation
• Neural Symbolic Reasoning• Mimic Human reasoning• Study & Understand user behaviors
138
Future Direction (2)
![Page 130: LearningandReasoningonGraphfor Recommendationstaff.ustc.edu.cn › ~hexn › slides › cikm19-tutorial-graph-rec.pdf · Recommendation Search results of Taobao5. UbiquitousPersonalizedRecommendation](https://reader035.fdocuments.net/reader035/viewer/2022062414/5f0bdc4c7e708231d432929e/html5/thumbnails/130.jpg)
© Copyright NExT++. All Right Reserved.
THANK YOU!
139