Users and Noise: The Magic Barrier of Recommender Systems
-
Upload
alan-said -
Category
Technology
-
view
1.511 -
download
4
description
Transcript of Users and Noise: The Magic Barrier of Recommender Systems
![Page 1: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/1.jpg)
Competence Center Information Retrieval & Machine Learning
@alansaid, @saschanarr, @matip
Users and Noise: The Magic Barrier of Recommender Systems
Alan Said, Brijnesh J. Jain, Sascha Narr, Till Plumbaum
![Page 2: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/2.jpg)
Outline
►The Magic Barrier
►Empirical Risk Minimization
►Deriving the Magic Barrier
►User Study
►Conclusion
20 July 2012 The Magic Barrier 2
![Page 3: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/3.jpg)
The Magic Barrier
20 July 2012 The Magic Barrier 3
![Page 4: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/4.jpg)
The Magic Barrier
►No magic involved....
►Coined by Herlocker et al. in 2004
“...an algorithm cannot be more accurate than the variance in a user’s ratings for the same item.”
The maximum level of prediction that a recommender algorithm can attain.
►What does this mean?
20 July 2012 The Magic Barrier 4
![Page 5: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/5.jpg)
The Magic Barrier
20 July 2012 The Magic Barrier 5
![Page 6: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/6.jpg)
The Magic Barrier
►Even a “perfect” recommender should not reach RMSE = 0 or Precision @ N = 1
►Why?
People are inconsistent and noisy in their ratings
“perfect” accuracy is not perfect
►So?
Knowing the highest possible level of accuracy, we can stop optimizing our algorithms at “perfect” (before overfitting)
20 July 2012 The Magic Barrier 6
![Page 7: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/7.jpg)
The Magic Barrier
So – how do we find the magic barrier?
We employ the Empirical Risk Minimization principle and a statistical model for user inconsistencies
20 July 2012 The Magic Barrier 7
![Page 8: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/8.jpg)
The Magic Barrier – User Inconsistencies
Assumption:
If a user were to re-rate all previously rated items, keeping in mind the inconsistency, the ratings would differ, i.e.
𝑟𝑢𝑖 = 𝜇𝑢𝑖 + 𝜀𝑢𝑖
where 𝜇𝑢𝑖 is the expected rating, and
𝜀𝑢𝑖 the rating error (has zero mean)
20 July 2012 The Magic Barrier 8
![Page 9: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/9.jpg)
Empirical Risk Minimization
►… is a principle in statistical learning theory which defines a family of learning algorithms and is used to give theoretical bounds on the performance of learning algorithms.[Wikipedia]
20 July 2012 The Magic Barrier 9
![Page 10: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/10.jpg)
Empirical Risk Minimization
►We formulate our risk function as
𝑅 𝑓 = 𝑝 𝑢, 𝑖, 𝑟𝑢,𝑖,𝑟 𝑓 𝑢, 𝑖 − 𝑟2
►Keeping the assumption in mind, we formulate the risk for a true, unknown, rating function as the sum of the noise variance, i.e.
𝑅 𝑓∗ = 𝑝 𝑢, 𝑖𝑢,𝑖 𝕍 𝜀𝑢𝑖
where 𝕍 𝜀𝑢𝑖 is the noise variance
20 July 2012 The Magic Barrier 10
The probability of user u rating item i with score r
The prediction error
![Page 11: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/11.jpg)
Deriving the Magic Barrier
►We want to express the risk function in terms of a magic barrier for RMSE – we take the root of the risk function
ℬ𝒰×ℐ = 𝑝 𝑢, 𝑖 𝕍 𝜀𝑢𝑖𝑢,𝑖
RMSE=0 iff 𝜀𝑢𝑖 = 0 over all ratings users and items
► In terms of RMSE we can express this as
𝐸𝑅𝑀𝑆𝐸 𝑓 = ℬ𝒰×ℐ + 𝐸𝑓 > ℬ𝒰×ℐ
where 𝐸𝑓 is the error
20 July 2012 The Magic Barrier 11
![Page 12: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/12.jpg)
Estimating the Magic Barrier
1. For each user-item pair in our population
a) Sample ratings on a regular basis, i.e. re-ratings
b) Estimate the expected value of ratings
𝜇 𝑢𝑖 =1
𝑚 𝑟𝑡𝑢𝑖
𝑚
𝑡=1
c. Estimate the rating variance
𝜀 𝑢𝑖2 =
1
𝑚 𝜇 𝑢𝑖 − 𝑟𝑡𝑢𝑖
2𝑚
𝑡=1
2. Estimate the magic barrier by taking the average
ℬ =1
𝒳 𝜀 𝑢𝑖
2
𝑢𝑖 ∈𝒳
20 July 2012 The Magic Barrier 12
![Page 13: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/13.jpg)
A real-world user study
20 July 2012 The Magic Barrier 13
![Page 14: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/14.jpg)
A User Study
►We teamed up with moviepilot.de
Germany’s largest online movie recommendation community
Ratings scale 1-10 stars (Netflix: 1-5 stars)
►Created a re-rating UI
Users were asked to re-rate at least 20 movies
1 new rating (so-called opinions) per movie
Collected data:
306 users
6,299 new opinions
2,329 movies
20 July 2012 The Magic Barrier 14
![Page 15: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/15.jpg)
A User Study
20 July 2012 The Magic Barrier 15
User study moviepilot
![Page 16: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/16.jpg)
A User Study
20 July 2012 The Magic Barrier 16
Predictions vs Ratings
Overall Magic Barrier
Ratings above user’s average
Opinions above user’s average
Ratings below user’s average
Opinions below user’s average
~4 ratings steps
~1 rating steps
Room for improvement
![Page 17: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/17.jpg)
Conclusion
►We created a mathematical characterization of the magic barrier
►We performed a user study on a commercial movie recommendation website and estimated its magic barrier
►We concluded the commercial recommender engine still has room for improvement
►No magic
20 July 2012 The Magic Barrier 17
![Page 18: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/18.jpg)
More?
► Estimating the Magic Barrier of Recommender Systems: A User Study
SIGIR 2012
► Magic Barrier explained
http://irml.dailab.de
► Movie rating and explanation user study
http://j.mp/ratingexplain
► Recommender Systems Wiki
www.recsyswiki.com
► Recommender Systems Challenge
www.recsyschallenge.com
20 July 2012 The Magic Barrier 18
![Page 19: Users and Noise: The Magic Barrier of Recommender Systems](https://reader034.fdocuments.net/reader034/viewer/2022042607/554bb627b4c90594278b48fb/html5/thumbnails/19.jpg)
Questions?
►Thank You for Listening!
20 July 2012 The Magic Barrier 19