Ranking methods in machine learning: A tutorial introduction
Transcript of Ranking methods in machine learning: A tutorial introduction
![Page 1: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/1.jpg)
Ranking Methods in Machine Learning
A Tutorial Introduction
Shivani Agarwal
Computer Science & Artificial Intelligence LaboratoryMassachusetts Institute of Technology
![Page 2: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/2.jpg)
Example 1: Recommendation Systems
![Page 3: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/3.jpg)
Example 2: Information Retrieval
![Page 4: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/4.jpg)
information
Example 2: Information Retrieval
![Page 5: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/5.jpg)
information
Example 2: Information Retrieval
![Page 6: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/6.jpg)
Problem: Millions of structures in a chemical library.How do we identify the most promising ones?
Example 3: Drug Discovery
![Page 7: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/7.jpg)
Human genetics is now at a critical juncture. The molecular methods used successfully to identify the genes underlying rare mendelian syndromes are failing to find the numerous genes causing more common, familial, non-mendelian diseases . . .
Nature 405:847–856, 2000
Example 4: Bioinformatics
![Page 8: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/8.jpg)
Nature 405:847–856, 2000
With the human genome sequence nearing completion, new opportunities are being presented for unravelling the complex genetic basis of nonmendelian disorders based on large-scale genomewide studies . . .
Example 4: Bioinformatics
![Page 9: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/9.jpg)
Types of Ranking Problems
Instance Ranking
Label Ranking
Subset Ranking
Rank Aggregation
?
![Page 10: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/10.jpg)
Instance Ranking
> , 10
> , 20
…
doc1 doc2
doc3 doc5
![Page 11: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/11.jpg)
Label Ranking
doc1
…
sports > politicshealth > money…
doc2science > sportsmoney > politics…
![Page 12: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/12.jpg)
Subset Ranking
doc1
…
doc2query 1 > , doc3 doc5> …
doc2query 2 > , …doc4 doc11 doc3>
![Page 13: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/13.jpg)
Rank Aggregation
…
query 1 …
query 2
resultsof search engine 1
resultsof search engine 2
desiredranking
resultsof search engine 1
…resultsof search engine 2
desiredranking
![Page 14: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/14.jpg)
Types of Ranking Problems
Instance Ranking
Label Ranking
Subset Ranking
Rank Aggregation
? This tutorial
![Page 15: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/15.jpg)
Tutorial Road Map
Part I: Theory & Algorithms
Part II: Applications
Further Reading & Resources
Bipartite Rankingk-partite RankingRanking with Real-Valued LabelsGeneral Instance Ranking
RankSVMRankBoostRankNet
Applications to BioinformaticsApplications to Drug DiscoverySubset Ranking and Applications to Information Retrieval
![Page 16: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/16.jpg)
Part ITheory & Algorithms
[for Instance Ranking]
![Page 17: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/17.jpg)
Bipartite Ranking
Relevant (+) …doc1+ doc2+ doc3+
Irrelevant (-) …doc1- doc2- doc3- doc4-
![Page 18: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/18.jpg)
Bipartite Ranking
![Page 19: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/19.jpg)
Bipartite Ranking
![Page 20: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/20.jpg)
Is Bipartite Ranking Different from Binary Classification?
![Page 21: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/21.jpg)
Is Bipartite Ranking Different from Binary Classification?
![Page 22: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/22.jpg)
Is Bipartite Ranking Different from Binary Classification?
![Page 23: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/23.jpg)
Is Bipartite Ranking Different from Binary Classification?
![Page 24: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/24.jpg)
Is Bipartite Ranking Different from Binary Classification?
![Page 25: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/25.jpg)
Is Bipartite Ranking Different from Binary Classification?
![Page 26: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/26.jpg)
Bipartite Ranking:Basic Algorithmic Framework
![Page 27: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/27.jpg)
Bipartite RankSVM Algorithm
[Herbrich et al, 2000; Joachims, 2002; Rakotomamonjy, 2004]
![Page 28: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/28.jpg)
Bipartite RankSVM Algorithm
![Page 29: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/29.jpg)
Bipartite RankBoost Algorithm
[Freund et al, 2003]
![Page 30: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/30.jpg)
Bipartite RankBoost Algorithm
![Page 31: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/31.jpg)
Bipartite RankNet Algorithm
[Burges et al, 2005]
![Page 32: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/32.jpg)
k-partite Ranking
Rating k …doc1k doc2k doc3k
Rating 1 …doc11 doc21 doc31 doc41
Rating 2 …doc12 doc22 doc32 doc42
…
![Page 33: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/33.jpg)
k-partite Ranking
![Page 34: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/34.jpg)
k-partite Ranking:Basic Algorithmic Framework
![Page 35: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/35.jpg)
Ranking with Real-Valued Labels
…
doc1
doc2
doc3
y1
y2
y3
![Page 36: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/36.jpg)
Ranking with Real-Valued Labels
![Page 37: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/37.jpg)
Ranking with Real-Valued Labels:Basic Algorithmic Framework
![Page 38: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/38.jpg)
General Instance Ranking
> , r1
> , r2
…
doc1 doc1’
doc2 doc2’
![Page 39: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/39.jpg)
General Instance Ranking
ri
![Page 40: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/40.jpg)
General Instance Ranking
![Page 41: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/41.jpg)
General Instance Ranking:Basic Algorithmic Framework
![Page 42: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/42.jpg)
General RankSVM Algorithm
[Herbrich et al, 2000; Joachims, 2002]
![Page 43: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/43.jpg)
General RankBoost Algorithm
[Freund et al, 2003]
![Page 44: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/44.jpg)
General RankNet Algorithm
[Burges et al, 2005]
![Page 45: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/45.jpg)
Tutorial Road Map
Part I: Theory & Algorithms
Part II: Applications
Further Reading & Resources
Bipartite Rankingk-partite RankingRanking with Real-Valued LabelsGeneral Instance Ranking
RankSVMRankBoostRankNet
Applications to BioinformaticsApplications to Drug DiscoverySubset Ranking and Applications to Information Retrieval
![Page 46: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/46.jpg)
Part IIApplications
[and Subset Ranking]
![Page 47: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/47.jpg)
Human genetics is now at a critical juncture. The molecular methods used successfully to identify the genes underlying rare mendelian syndromes are failing to find the numerous genes causing more common, familial, non-mendelian diseases . . .
Nature 405:847–856, 2000
Application to Bioinformatics
![Page 48: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/48.jpg)
Nature 405:847–856, 2000
With the human genome sequence nearing completion, new opportunities are being presented for unravelling the complex genetic basis of nonmendelian disorders based on large-scale genomewide studies . . .
Application to Bioinformatics
![Page 49: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/49.jpg)
Identifying Genes Relevant to a DiseaseUsing Microarrray Gene Expression Data
![Page 50: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/50.jpg)
Identifying Genes Relevant to a DiseaseUsing Microarrray Gene Expression Data
![Page 51: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/51.jpg)
Identifying Genes Relevant to a DiseaseUsing Microarrray Gene Expression Data
![Page 52: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/52.jpg)
Identifying Genes Relevant to a DiseaseUsing Microarrray Gene Expression Data
![Page 53: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/53.jpg)
Identifying Genes Relevant to a DiseaseUsing Microarrray Gene Expression Data
![Page 54: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/54.jpg)
Identifying Genes Relevant to a DiseaseUsing Microarrray Gene Expression Data
![Page 55: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/55.jpg)
Formulation as a BipartiteRanking Problem
Not relevant
Relevant …
…
![Page 56: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/56.jpg)
Microarray Gene Expression Data Sets[Golub et al, 1999; Alon et al, 1999]
![Page 57: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/57.jpg)
Selection of Training Genes
![Page 58: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/58.jpg)
Top-Ranking Genes for Leukemia Returned by RankBoost
[Agarwal & Sengupta, 2009]
![Page 59: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/59.jpg)
Biological Validation
[Agarwal et al, 2010]
![Page 60: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/60.jpg)
Problem: Millions of structures in a chemical library.How do we identify the most promising ones?
Application to Drug Discovery
![Page 61: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/61.jpg)
Formulation as a Ranking Problemwith Real-Valued Labels
![Page 62: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/62.jpg)
Cheminformatics Data Sets[Sutherland et al, 2004]
![Page 63: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/63.jpg)
DHFR Results Using RankSVM
[Agarwal et al, 2010]
![Page 64: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/64.jpg)
Application to Information Retrieval (IR)
![Page 65: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/65.jpg)
Learning to Rank in IR
![Page 66: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/66.jpg)
Learning to Rank in IR
![Page 67: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/67.jpg)
Learning to Rank in IR
![Page 68: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/68.jpg)
Learning to Rank in IR
![Page 69: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/69.jpg)
Learning to Rank in IR
![Page 70: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/70.jpg)
Learning to Rank in IR
![Page 71: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/71.jpg)
Learning to Rank in IR
![Page 72: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/72.jpg)
Learning to Rank in IR
![Page 73: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/73.jpg)
Learning to Rank in IR
![Page 74: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/74.jpg)
Learning to Rank in IR
![Page 75: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/75.jpg)
General Subset Ranking
doc1
…
doc2query 1 > , doc3 doc5> …
doc2query 2 > , …doc4 doc11 doc3>
![Page 76: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/76.jpg)
General Subset Ranking
![Page 77: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/77.jpg)
Subset Ranking with Real-ValuedRelevance Labels
doc1
…
doc2query 1 y1 , doc3 …
doc1query 2 …doc2 doc3
y2 , y3
y1 , y2 , y3
![Page 78: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/78.jpg)
Subset Ranking with Real-ValuedRelevance Labels
![Page 79: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/79.jpg)
RankSVM Applied to IR/Subset Ranking
[Joachims, 2002]
Standard RankSVM
![Page 80: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/80.jpg)
RankSVM Applied to IR/Subset Ranking
RankSVM with Query Normalization & Relevance Weighting
[Agarwal & Collins, 2010; also Cao et al, 2006]
![Page 81: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/81.jpg)
Ranking Performance Measures in IR
Mean Average Precision (MAP)
![Page 82: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/82.jpg)
Ranking Performance Measures in IR
Normalized Discounted Cumulative Gain (NDCG)
![Page 83: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/83.jpg)
Ranking Algorithms for OptimizingMAP/NDCG
![Page 84: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/84.jpg)
LETOR 3.0/OHSUMED Data Set[Liu et al, 2007]
![Page 85: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/85.jpg)
OHSUMED Results – NDCG
![Page 86: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/86.jpg)
OHSUMED Results – MAP
![Page 87: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/87.jpg)
Further Reading & Resources[Incomplete!]
![Page 88: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/88.jpg)
Early Papers on Ranking
W. W. Cohen, R. E. Schapire, and Y. Singer, Learning to order things, Journal of Artificial Intelligence Research, 10:243–270, 1999.
R. Herbrich, T. Graepel, and K. Obermayer, Large margin rank boundaries for ordinal regression. Advances in Large Margin Classifiers, 2000.
T. Joachims, Optimizing search engines using clickthrough data, KDD 2002.
Y. Freund, R. Iyer, R. E. Schapire, and Y. Singer, An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933–969, 2003.
C.J.C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, G. Hullender, Learning to rank using gradient descent, ICML 2005.
![Page 89: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/89.jpg)
Generalization Bounds for Ranking
S. Agarwal, T. Graepel, R. Herbrich, S. Har-Peled, D. Roth, Generalization bounds for the area under the ROC curve, Journal of Machine Learning Research, 6:393—425, 2005.
S. Agarwal, P. Niyogi, Generalization bounds for ranking algorithms via algorithmic stability, Journal of Machine Learning Research, 10:441—474,2009.
C. Rudin, R. Schapire, Margin-based ranking and an equivalence between AdaBoost and RankBoost, Journal of Machine Learning Research, 10: 2193—2232, 2009
![Page 90: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/90.jpg)
Bioinformatics/Drug DiscoveryApplications
S. Agarwal and S. Sengupta, Ranking genes by relevance to a disease, CSB 2009.
S. Agarwal, D. Dugar, and S. Sengupta, Ranking chemical structures for drug discovery: A new machine learning approach. Journal of Chemical Information and Modeling, DOI 10.1021/ci9003865, 2010.
![Page 91: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/91.jpg)
Other Applications
Natural Language Processing
M. Collins and T. Koo, Discriminative reranking for natural language parsing, Computational Linguistics, 31:25—69, 2005.
Collaborative Filtering
M. Weimer, A. Karatzoglou, Q. V. Le, and A. Smola, CofiRank - Maximum margin matrix factorization for collaborative ranking, NIPS 2007.
Manhole Event Prediction
C. Rudin, R. Passonneau, A. Radeva, H. Dutta, S. Ierome, and D. Isaac , A process for predicting manhole events in Manhattan, Machine Learning, DOI 10.1007/s10994-009-5166, 2010.
![Page 92: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/92.jpg)
IR Ranking Algorithms
Y. Cao, J. Xu, T.-Y. Liu, H. Li, Y. Hunag, and H.W. Hon, Adapting ranking SVM to document retrieval, SIGIR 2006.
C.J.C. Burges, R. Ragno, and Q.V. Le, Learning to rank with non-smooth cost functions. NIPS 2006.
J. Xu and H. Li, AdaRank: A boosting algorithm for information retrieval. SIGIR 2007.
Y. Yue, T. Finley, F. Radlinski, and T. Joachims, A support vector method for optimizing average precision. SIGIR 2007.
M. Taylor, J. Guiver, S. Robertson, T. Minka, Softrank: optimizing non-smooth rank metrics. WSDM 2008.
![Page 93: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/93.jpg)
IR Ranking Algorithms
S. Chakrabarti, R. Khanna, U. Sawant, and C. Bhattacharyya, Structured learning for nonsmooth ranking losses. KDD 2008.
D. Cossock and T. Zhang, Statistical analysis of Bayes optimal subset ranking, IEEE Transactions on Information Theory, 54:5140–5154, 2008.
T. Qin, X.D. Zhang, M.F. Tsai, D.S. Wang, T.Y. Liu, and H. Li. Query-level loss functions for information retrieval. Information Processing and Management, 44:838–855, 2008.
O. Chapelle and M. Wu, Gradient descent optimization of smoothed information retrieval metrics. Information Retrieval (To appear), 2010.
S. Agarwal and M. Collins, Maximum margin ranking algorithms for information retrieval, ECIR 2010.
![Page 94: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/94.jpg)
NIPS Workshop 2005Learning to Rank
NIPS Workshop 2009Advances in Ranking
SIGIR Workshops 2007-2009Learning to Rank for Information Retrieval
American Institute of MathematicsWorkshop in Summer 2010 The Mathematics of Ranking
![Page 95: Ranking methods in machine learning: A tutorial introduction](https://reader033.fdocuments.net/reader033/viewer/2022050916/58a2d2b01a28ab4a0f8c3c8a/html5/thumbnails/95.jpg)
Tie-Yan Liu, Learning to Rank for Information Retrieval, Foundations & Trends in Information Retrieval, 2009.
Shivani Agarwal, A Tutorial Introduction to Ranking Methods in Machine Learning, In preparation.
Shivani Agarwal (Ed.), Advances in Ranking Methods in Machine Learning, Springer-Verlag, In preparation.
Tutorial Articles & Books