Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong...
-
Upload
timothy-beach -
Category
Documents
-
view
220 -
download
2
Transcript of Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong...
Interactively Optimizing Information Retrieval Systems as
a Dueling Bandits Problem
ICML 2009
Yisong Yue Thorsten Joachims
Cornell University
Learning To Rank
• Supervised Learning Problem– Extension of classification/regression– Relatively well understood– High applicability in Information Retrieval
• Requires explicitly labeled data– Expensive to obtain– Expert judged labels == search user utility?– Doesn’t generalize to other search domains.
Our Contribution
• Learn from implicit feedback (users’ clicks)– Reduce labeling cost– More representative of end user information needs
• Learn using pairwise comparisons– Humans are more adept at making pairwise judgments– Via Interleaving [Radlinski et al., 2008]
• On-line framework (Dueling Bandits Problem)– We leverage users when exploring new retrieval functions– Exploration vs exploitation tradeoff (regret)
Team-Game Interleaving
1. Kernel Machines http://svm.first.gmd.de/
2. Support Vector Machinehttp://jbolivar.freeservers.com/
3. An Introduction to Support Vector Machineshttp://www.support-vector.net/
4. Archives of SUPPORT-VECTOR-MACHINES ...http://www.jiscmail.ac.uk/lists/SUPPORT...
5. SVM-Light Support Vector Machine http://ais.gmd.de/~thorsten/svm light/
1. Kernel Machines http://svm.first.gmd.de/
2. SVM-Light Support Vector Machine http://ais.gmd.de/~thorsten/svm light/
3. Support Vector Machine and Kernel ... Referenceshttp://svm.research.bell-labs.com/SVMrefs.html
4. Lucent Technologies: SVM demo applet http://svm.research.bell-labs.com/SVT/SVMsvt.html
5. Royal Holloway Support Vector Machine http://svm.dcs.rhbnc.ac.uk
1. Kernel Machines T2http://svm.first.gmd.de/
2. Support Vector Machine T1http://jbolivar.freeservers.com/
3. SVM-Light Support Vector Machine T2http://ais.gmd.de/~thorsten/svm light/
4. An Introduction to Support Vector Machines T1http://www.support-vector.net/
5. Support Vector Machine and Kernel ... References T2http://svm.research.bell-labs.com/SVMrefs.html
6. Archives of SUPPORT-VECTOR-MACHINES ... T1http://www.jiscmail.ac.uk/lists/SUPPORT...
7. Lucent Technologies: SVM demo applet T2http://svm.research.bell-labs.com/SVT/SVMsvt.html
f1(u,q) r1 f2(u,q) r2
Interleaving(r1,r2)
(u=thorsten, q=“svm”)
Interpretation: (r2 Â r1) ↔ clicks(T2) > clicks(T1)
Invariant: For all k, in expectation same number of team members in top k from each team.
NEXTPICK
[Radlinski, Kurup, Joachims; CIKM 2008]
Dueling Bandits Problem
• Continuous space bandits F – E.g., parameter space of retrieval functions (i.e., weight vectors)
• Each time step compares two bandits– E.g., interleaving test on two retrieval functions– Comparison is noisy & independent
Dueling Bandits Problem
• Continuous space bandits F – E.g., parameter space of retrieval functions (i.e., weight vectors)
• Each time step compares two bandits– E.g., interleaving test on two retrieval functions– Comparison is noisy & independent
• Choose pair (ft, ft’) to minimize regret:
• (% users who prefer best bandit over chosen ones)
T
tttT ffPffP
1
1)'*()*(
T
tttT ffPffP
1
1)'*()*(
•Example 1•P(f* > f) = 0.9•P(f* > f’) = 0.8•Incurred Regret = 0.7
•Example 2 •P(f* > f) = 0.7•P(f* > f’) = 0.6•Incurred Regret = 0.3
•Example 3•P(f* > f) = 0.51•P(f* > f) = 0.55•Incurred Regret = 0.06
Modeling Assumptions
• Each bandit f 2F has intrinsic value v(f)– Never observed directly– Assume v(f) is strictly concave ( unique f* )
• Comparisons based on v(f)– P(f > f’) = σ( v(f) – v(f’) )– P is L-Lipschitz
– For example: )exp(1
1)(
xx
Probability Functions
Dueling Bandit Gradient Descent
• Maintain ft
– Compare with ft’ (close to ft -- defined by step size)
– Update if ft’ wins comparison
• Expectation of update close to gradient of P(ft > f’)– Builds on Bandit Gradient Descent [Flaxman et al., 2005]
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
Analysis (Sketch)
• Dueling Bandit Gradient Descent– Sequence of partially convex functions ct(f) = P(ft > f)
– Random binary updates (expectation close to gradient)
• Bandit Gradient Descent [Flaxman et al., SODA 2005]
– Sequence of convex functions – Use randomized update
(expectation close to gradient)
– Can be extended to our setting
(Assumes more information)
Analysis (Sketch)
• Convex functions satisfy
– Both additive and multiplicative error– Depends on exploration step size δ – Main analytical contribution: bounding multiplicative error
*)()(*)()( xxxcxcxc
Regret Bound
• Regret grows as O(T3/4):
• Average regret shrinks as O(T-1/4)– In the limit, we do as well as knowing f* in
hindsight
T
tttT ffPffP
1
1)'*()*(
RdLTT 102E 4/3
δ = O(1/T-1/4 )γ = O(1/T-1/2 )
Practical Considerations
• Need to set step size parameters– Depends on P(f > f’)
• Cannot be set optimally– We don’t know the specifics of P(f > f’)– Algorithm should be robust to parameter settings
• Set parameters approximately in experiments
00.10.20.30.40.50.60.70.80.9
1
10 570
1130
1690
2250
2810
3370
3930
4490
5050
5610
6170
6730
7290
7850
8410
8970
9530
Aver
age
Regr
et
Regret Comparison DBGD vs BGD
DBGD
BGD 1
BGD 2
• 50 dimensional parameter space• Value function v(x) = -xTx• Logistic transfer function• Random point has regret almost 1
More experiments in paper.
Web Search Simulation
• Leverage web search dataset– 1000 Training Queries, 367 Dimensions
• Simulate “users” issuing queries– Value function based on NDCG@10 (ranking measure)– Use logistic to make probabilistic comparisons
• Use linear ranking function.
• Not intended to compete with supervised learning– Feasibility check for online learning w/ users– Supervised labels difficult to acquire “in the wild”
• Chose parameters with best final performance• Curves basically identical for validation and test sets (no over-fitting)• Sampling multiple queries makes no difference
0.480.5
0.520.540.560.58
0.60.62
063
0000
1260
000
1890
000
2520
000
3150
000
3780
000
4410
000
5040
000
5670
000
6300
000
6930
000
7560
000
8190
000
8820
000
9450
000
Trai
ning
NDC
G @
10Web Simulation Results
Sample 1
Sample 10
Sample 100
Ranking SVM
What Next?
• Better simulation environments– More realistic user modeling assumptions
• DBGD simple and extensible – Incorporate pairwise document preferences– Deal with ranking discontinuities
• Test on real search systems– Varying scales of user communities– Sheds on insight / guides future development
Extra Slides
Active vs Passive Learning
• Passive Data Collection (offline)– Biased by current retrieval function
• Point-wise Evaluation– Design retrieval function offline– Evaluate online
• Active Learning (online)– Automatically propose new rankings to evaluate– Our approach
Relative vs Absolute Metrics
• Our framework based on relative metrics– E.g., comparing pairs of results or rankings– Relatively recent development
• Absolute Metrics– E.g., absolute click-through rate– More common in literature – Suffers from presentation bias– Less robust to the many different sources of noise
What Results do Users View/Click?
[Joachims et al., TOIS 2007]
Analysis (Sketch)
• Convex functions satisfy
– We have both multiplicative and additive error– Depends on exploration step size δ – Main technical contribution: bounding multiplicative error
*)()(*)()( xxxcxcxc
T
tttt ffPffP
1
*)()(E
Existing results yields sub-linear bounds on:
Analysis (Sketch)
• We know how to bound
• Regret:
• We can show using Lipschitz and symmetry of σ:
T
tttT ffPffP
1
1)'*()*(
LTffPffPT
ttttT
1
*)()(E2E
T
tttt ffPffP
1
*)()(E
More Simulation Experiments
• Logistic transfer function σ(x) = 1/(1+exp(-x))• 4 choices of value functions
• δ, γ set approximately
TR
NDCG• Normalized Discounted Cumulative Gain• Multiple Levels of Relevance
• DCG:– contribution of ith rank position:
– Ex: has DCG score of
• NDCG is normalized DCG – best possible ranking as score NDCG = 1
)1log(
12
i
iy
45.5)6log(
1
)5log(
0
)4log(
1
)3log(
3
)2log(
1
Considerations
• NDCG is discontinuous w.r.t. function parameters– Try larger values of δ, γ– Try sampling multiple queries per update
• Homogenous user values– NDCG@10– Not an optimization concern– Modeling limitation
• Not intended to compete with supervised learning– Sanity check of feasibility for online learning w/ users