Predicting the Performance of Recommender Systems: An...
Transcript of Predicting the Performance of Recommender Systems: An...
![Page 1: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/1.jpg)
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy IRGIR Group @ UAM
Predicting the Performance of
Recommender Systems:
An Information Theoretic Approach
Alejandro Bellogín, Pablo Castells, Iván CantadorEscuela Politécnica Superior
Universidad Autónoma de Madrid
@abellogin
![Page 2: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/2.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
RS suggests “interesting” items to users
• Most common: explicit ratings
• Goal: predict rating rjk
i1 … ik … im
u1
…
uj ?
…
un
Recommender Systems
items
users
![Page 3: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/3.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Performance Prediction in IR
Estimation of the system’s performance in response to a specific
query
Predictors: query scope, query clarity, query drift, …
We focus on query clarity:
Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting query performance. SIGIR 2002.
He, B., Ounis, I.: Inferring query performance using pre-retrieval predictors. SPIRE 2004.
Mitra, M.,Singhal, A., Buckley, C.: Improving automatic query expansion. SIGIR 1998.
|clarity | log
w Vc
p w qq p w q
p w
| | ; | |
| | |
| | 1
q
q
w q
d R
ml c
p d q p q d p d p q d p w d
p w q p w d p d q
p w d p w d p w
![Page 4: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/4.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Predictive Models of Recommendation Performance
User clarity:
• Distance between the user’s and the system’s probability model
• X may be: users, items, ratings, or a combination (vocabulary space)
|clarity | log
x X c
p x uu p x u
p u
background model
user model
![Page 5: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/5.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Predictive Models of Recommendation Performance
Three user clarity formulations:
|clarity | log
x X c
p x uu p x u
p u
background model
user model
Name Vocabulary User modelBackground
model
Rating-based Ratings
Item-based Items
Item-and-rating-based Items rated by the user
|p r u
|p i u
| ,p r i u
cp r
cp i
|mlp r i
![Page 6: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/6.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Predictive Models of Recommendation Performance
Seven user clarity models implemented:
Name Formulation User model Background model
RatUser Rating-based
RatItem Rating-based
ItemSimple Item-based
ItemUser Item-based
IRUser Item-and-rating-based
IRItem Item-and-rating-based
IRUserItem Item-and-rating-based
cp r
cp r
cp i
cp i
|mlp r i
|mlp r i
|mlp r i
| ,Up r i u
| ,Ip r i u
| ,UIp r i u
|Rp i u
|URp i u
| , ; |U URp r i u p i u
| , ; |I URp r i u p i u
![Page 7: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/7.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Predictive Models of Recommendation Performance
Seven user clarity models implemented:
Name Formulation User model Background model
RatUser Rating-based
RatItem Rating-based
ItemSimple Item-based
ItemUser Item-based
IRUser Item-and-rating-based
IRItem Item-and-rating-based
IRUserItem Item-and-rating-based
cp r
cp r
cp i
cp i
|mlp r i
|mlp r i
|mlp r i
| ,Up r i u
| ,Ip r i u
| ,UIp r i u
|Rp i u
|URp i u
| , ; |U URp r i u p i u
| , ; |I URp r i u p i u
| , | ,Up r i u p u r i
| , , |UIp r i u p u i r
| , | ,Ip r i u p i r u
Wang, J., de Vries, A.P., Reinders, M.J.T.: Unified relevance models for rating prediction in collabotative filtering. ACM TOIS 2008.
![Page 8: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/8.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Predictive Models of Recommendation Performance
Seven user clarity models implemented:
Name Formulation User model Background model
RatUser Rating-based
RatItem Rating-based
ItemSimple Item-based
ItemUser Item-based
IRUser Item-and-rating-based
IRItem Item-and-rating-based
IRUserItem Item-and-rating-based
cp r
cp r
cp i
cp i
|mlp r i
|mlp r i
|mlp r i
| | |R ml ml
r
p i u p i r p r u
R
| ,Up r i u
| ,Ip r i u
| ,UIp r i u
|Rp i u
|URp i u
| , ; |U URp r i u p i u
| , ; |I URp r i u p i u
| | , |UR ml
r
p i u p i u r p r u
R
![Page 9: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/9.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Predictive Models of Recommendation Performance
Seven user clarity models implemented:
Name Formulation User model Background model
RatUser Rating-based
RatItem Rating-based
ItemSimple Item-based
ItemUser Item-based
IRUser Item-and-rating-based
IRItem Item-and-rating-based
IRUserItem Item-and-rating-based
cp r
cp r
cp i
cp i
|mlp r i
|mlp r i
|mlp r i
,
| | , |r u i r
p r u p r i u p i u
| ,Up r i u
| ,Ip r i u
| ,UIp r i u
|Rp i u
|URp i u
| , ; |U URp r i u p i u
| , ; |I URp r i u p i u
![Page 10: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/10.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Examples
Comparison of models for two users:
User Number of ratings ItemUser clarity RatItem clarity IRUserItem clarity
u1 51 216.01 28.60 6.85
u2 52 243.32 43.63 13.56
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Item-rating pairs
p(r|u,i)log(p(r|u,i)/p(r|i))u1
u2
Term contributions for each user, ordered by their corresponding
contribution to the user language model. IRUserItem clarity model.
![Page 11: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/11.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Examples
Comparison of models for two users:
User Number of ratings ItemUser clarity RatItem clarity IRUserItem clarity
u1 51 216.01 28.60 6.85
u2 52 243.32 43.63 13.56
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Item-rating pairs
p(r|u,i)log(p(r|u,i)/p(r|i))u1
u2
Term contributions for each user, ordered by their corresponding
contribution to the user language model. IRUserItem clarity model.
(u2, “Mc Hale’s Navy”, 3)
(4, “Mc Hale’s Navy”)
(com, “Mc Hale’s Navy”, 1)
![Page 12: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/12.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Examples
Comparison of models for two users:
User Number of ratings ItemUser clarity RatItem clarity IRUserItem clarity
u1 51 216.01 28.60 6.85
u2 52 243.32 43.63 13.56
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Item-rating pairs
p(r|u,i)log(p(r|u,i)/p(r|i))u1
u2
Term contributions for each user, ordered by their corresponding
contribution to the user language model. IRUserItem clarity model.
(u1, “Donnie Brasco”, 5)
(2, “Donnie Brasco”)
(com, “Donnie Brasco”, 4)
![Page 13: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/13.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Examples
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
4 3 5 2 1 4 3 5 2 1Ratings
RatItem p_c(x)
p(x|u1)
p(x|u2)
User language model sorted by collection probability
![Page 14: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/14.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Examples
-1.E-04
7.E-19
1.E-04
2.E-04
3.E-04
4.E-04
5.E-04
6.E-04
7.E-04
8.E-04
9.E-04
Items
ItemUser p_c(x)
p(x|u1)
p(x|u2)
User language model sorted by collection probability
![Page 15: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/15.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Examples
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Item-rating pairs
IRUserItem p_c(x)
p(x|u1)
p(x|u2)
User language model sorted by collection probability
![Page 16: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/16.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Experiments
Dataset: Movielens 100K, 5-fold
Recommenders:
• four collaborative filtering (CF)
• one content-based (CBF)
Predictors: seven user clarity variations
Analyse correlation between predictors and recommender
performance
• Against more than one recommender
• Pearson / Spearman
• nDCG / MAP (@N)
![Page 17: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/17.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Results
Pearson’s correlation wrt. nDCG@50 for different recommenders
Predictor CBF IB TF-L1 TF-L2 UB
ItemSimple 0.257 0.146 0.521 0.564 0.491
ItemUser 0.252 0.188 0.534 0.531 0.483
RatUser 0.234 0.182 0.507 0.516 0.469
RatItem 0.191 0.184 0.442 0.426 0.395
IRUser 0.171 -0.092 0.253 0.399 0.257
IRItem 0.218 0.152 0.453 0.416 0.372
IRUserItem 0.265 0.105 0.523 0.545 0.444
![Page 18: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/18.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Results
Pearson’s correlation wrt nDCG@50 for different recommenders
Predictor CBF IB TF-L1 TF-L2 UB
ItemSimple 0.257 0.146 0.521 0.564 0.491
ItemUser 0.252 0.188 0.534 0.531 0.483
RatUser 0.234 0.182 0.507 0.516 0.469
RatItem 0.191 0.184 0.442 0.426 0.395
IRUser 0.171 -0.092 0.253 0.399 0.257
IRItem 0.218 0.152 0.453 0.416 0.372
IRUserItem 0.265 0.105 0.523 0.545 0.444
![Page 19: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/19.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Results
Pearson’s correlation wrt nDCG@50 for different recommenders
Predictor CBF IB TF-L1 TF-L2 UB
ItemSimple 0.257 0.146 0.521 0.564 0.491
ItemUser 0.252 0.188 0.534 0.531 0.483
RatUser 0.234 0.182 0.507 0.516 0.469
RatItem 0.191 0.184 0.442 0.426 0.395
IRUser 0.171 -0.092 0.253 0.399 0.257
IRItem 0.218 0.152 0.453 0.416 0.372
IRUserItem 0.265 0.105 0.523 0.545 0.444
Performance 0.061 0.004 0.093 0.239 0.044
![Page 20: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/20.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Results
Performance prediction depends on
• Actual recommender performance
• Input sources used by the recommender
In Information Retrieval, typically:
• Only one system (or the mean/median of several) is reported
• Language modelling retrieval systems are used
![Page 21: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/21.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Conclusions and Future Work
Adaptations of query clarity predictors in Recommender Systems
Strong correlation values
Revision of the grey sheep concept?
Applications
• Dynamic neighbour weighting
• Dynamic adjustment of recommender ensembles
Additional performance predictors
• With explicit recommender dependence
![Page 22: Predicting the Performance of Recommender Systems: An ...ir.ii.uam.es/~alejandro/2011/ictir-slides.pdf · 3rd International Conference on the Theory of Information Retrieval (ICTIR](https://reader033.fdocuments.net/reader033/viewer/2022060520/604f06af893a6674d3298e71/html5/thumbnails/22.jpg)
IRGIR Group @ UAM
3rd International Conference on the Theory of Information Retrieval (ICTIR ’11)
September 12-14, Bertinoro, Italy
Predicting the Performance of
Recommender Systems:
An Information Theoretic Approach
Alejandro Bellogín, Pablo Castells, Iván CantadorEscuela Politécnica Superior
Universidad Autónoma de Madrid
@abellogin
Thank you!