Probabilistic Information Retrieval Approach for Ranking of Database Query Results
Probabilistic Information Retrieval
-
Upload
harsh-thakkar -
Category
Education
-
view
228 -
download
0
Transcript of Probabilistic Information Retrieval
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
A Probabilistic model of Information Retrieval
Harsh Thakkar
DA-IICT, Gandhinagar
2015-04-27
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 1 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Overview
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 2 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Outline
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 3 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Why Probability in IR?
Queries are representations of user’s information need
Relevance is binary
Retrieval is inherently uncertain, since the needs of users arevague in nature i.e. change with time
Probability deals with uncertainity
Provides a good estimate of which documents to choose,hence more reliable
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 4 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 5 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Outline
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 6 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Relevance feedback
In relevance feedback, the user marks documents asrelevant/nonrelevant
Given some known relevant and nonrelevant documents, wecompute weights for non-query terms that indicate how likelythey will occur in relevant documents
Develop a probabilistic approach for relevance feedback andalso a general probabilistic model for IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 7 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Relevance feedback
In relevance feedback, the user marks documents asrelevant/nonrelevant
Given some known relevant and nonrelevant documents, wecompute weights for non-query terms that indicate how likelythey will occur in relevant documents
Develop a probabilistic approach for relevance feedback andalso a general probabilistic model for IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 7 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Relevance feedback
In relevance feedback, the user marks documents asrelevant/nonrelevant
Given some known relevant and nonrelevant documents, wecompute weights for non-query terms that indicate how likelythey will occur in relevant documents
Develop a probabilistic approach for relevance feedback andalso a general probabilistic model for IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 7 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Relevance feedback
In relevance feedback, the user marks documents asrelevant/nonrelevant
Given some known relevant and nonrelevant documents, wecompute weights for non-query terms that indicate how likelythey will occur in relevant documents
Develop a probabilistic approach for relevance feedback andalso a general probabilistic model for IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 7 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic Approach to Retrieval
Given a user information need (represented as a query) and acollection of documents (transformed into documentrepresentations), a system must determine how well thedocuments satisfy the query
An IR system has an uncertain understanding of the userquery, and makes an uncertain guess of whether a documentsatisfies the query
Probability theory provides a principled foundation for suchreasoning under uncertainty
Probabilistic models exploit this foundation to estimate howlikely it is that a document is relevant to a query
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 8 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic Approach to Retrieval
Given a user information need (represented as a query) and acollection of documents (transformed into documentrepresentations), a system must determine how well thedocuments satisfy the query
An IR system has an uncertain understanding of the userquery, and makes an uncertain guess of whether a documentsatisfies the query
Probability theory provides a principled foundation for suchreasoning under uncertainty
Probabilistic models exploit this foundation to estimate howlikely it is that a document is relevant to a query
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 8 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic Approach to Retrieval
Given a user information need (represented as a query) and acollection of documents (transformed into documentrepresentations), a system must determine how well thedocuments satisfy the query
An IR system has an uncertain understanding of the userquery, and makes an uncertain guess of whether a documentsatisfies the query
Probability theory provides a principled foundation for suchreasoning under uncertainty
Probabilistic models exploit this foundation to estimate howlikely it is that a document is relevant to a query
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 8 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic Approach to Retrieval
Given a user information need (represented as a query) and acollection of documents (transformed into documentrepresentations), a system must determine how well thedocuments satisfy the query
An IR system has an uncertain understanding of the userquery, and makes an uncertain guess of whether a documentsatisfies the query
Probability theory provides a principled foundation for suchreasoning under uncertainty
Probabilistic models exploit this foundation to estimate howlikely it is that a document is relevant to a query
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 8 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic Approach to Retrieval
Given a user information need (represented as a query) and acollection of documents (transformed into documentrepresentations), a system must determine how well thedocuments satisfy the query
An IR system has an uncertain understanding of the userquery, and makes an uncertain guess of whether a documentsatisfies the query
Probability theory provides a principled foundation for suchreasoning under uncertainty
Probabilistic models exploit this foundation to estimate howlikely it is that a document is relevant to a query
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 8 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic IR Models at a Glance
Classical probabilistic retrieval model
Probability ranking principle
Binary Independence Model, BestMatch25 (Okapi)
Bayesian networks for text retrieval
Language model approach to IR
Probabilistic methods are one of the oldest but also one of thecurrently hottest topics in IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 9 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic IR Models at a Glance
Classical probabilistic retrieval model
Probability ranking principle
Binary Independence Model, BestMatch25 (Okapi)
Bayesian networks for text retrieval
Language model approach to IR
Probabilistic methods are one of the oldest but also one of thecurrently hottest topics in IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 9 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic IR Models at a Glance
Classical probabilistic retrieval modelProbability ranking principle
Binary Independence Model, BestMatch25 (Okapi)
Bayesian networks for text retrieval
Language model approach to IR
Probabilistic methods are one of the oldest but also one of thecurrently hottest topics in IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 9 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic IR Models at a Glance
Classical probabilistic retrieval modelProbability ranking principle
Binary Independence Model, BestMatch25 (Okapi)
Bayesian networks for text retrieval
Language model approach to IR
Probabilistic methods are one of the oldest but also one of thecurrently hottest topics in IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 9 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic IR Models at a Glance
Classical probabilistic retrieval modelProbability ranking principle
Binary Independence Model, BestMatch25 (Okapi)
Bayesian networks for text retrieval
Language model approach to IR
Probabilistic methods are one of the oldest but also one of thecurrently hottest topics in IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 9 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic IR Models at a Glance
Classical probabilistic retrieval modelProbability ranking principle
Binary Independence Model, BestMatch25 (Okapi)
Bayesian networks for text retrieval
Language model approach to IR
Probabilistic methods are one of the oldest but also one of thecurrently hottest topics in IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 9 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic IR Models at a Glance
Classical probabilistic retrieval modelProbability ranking principle
Binary Independence Model, BestMatch25 (Okapi)
Bayesian networks for text retrieval
Language model approach to IR
Probabilistic methods are one of the oldest but also one of thecurrently hottest topics in IR
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 9 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Outline
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 10 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets
The paper provides a common platform to a variety ofperformance scattered over many other papers from1992-1999
All collections from TREC 1-7 are used
Also older results from datasets such as NPL, UKCIS,Cranfield and a newly created TREC (described below) arereproduced.
Cranfield collection: has short initial manual indexdescriptions based on the whole documentNLP: has short initial automatic descriptions from abstractsUKCIS: has only the title of the documentsTREC: has automatic initial descriptions in natural text formmostly from full documents but in some cases from abstracts.TREC is the largest of all the collectionsTREC had a mixture of Long, Medium and Very short requests(L,M,V)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 11 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets
The paper provides a common platform to a variety ofperformance scattered over many other papers from1992-1999
All collections from TREC 1-7 are used
Also older results from datasets such as NPL, UKCIS,Cranfield and a newly created TREC (described below) arereproduced.
Cranfield collection: has short initial manual indexdescriptions based on the whole documentNLP: has short initial automatic descriptions from abstractsUKCIS: has only the title of the documentsTREC: has automatic initial descriptions in natural text formmostly from full documents but in some cases from abstracts.TREC is the largest of all the collectionsTREC had a mixture of Long, Medium and Very short requests(L,M,V)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 11 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets
The paper provides a common platform to a variety ofperformance scattered over many other papers from1992-1999
All collections from TREC 1-7 are used
Also older results from datasets such as NPL, UKCIS,Cranfield and a newly created TREC (described below) arereproduced.
Cranfield collection: has short initial manual indexdescriptions based on the whole documentNLP: has short initial automatic descriptions from abstractsUKCIS: has only the title of the documentsTREC: has automatic initial descriptions in natural text formmostly from full documents but in some cases from abstracts.TREC is the largest of all the collectionsTREC had a mixture of Long, Medium and Very short requests(L,M,V)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 11 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets
The paper provides a common platform to a variety ofperformance scattered over many other papers from1992-1999
All collections from TREC 1-7 are used
Also older results from datasets such as NPL, UKCIS,Cranfield and a newly created TREC (described below) arereproduced.
Cranfield collection: has short initial manual indexdescriptions based on the whole document
NLP: has short initial automatic descriptions from abstractsUKCIS: has only the title of the documentsTREC: has automatic initial descriptions in natural text formmostly from full documents but in some cases from abstracts.TREC is the largest of all the collectionsTREC had a mixture of Long, Medium and Very short requests(L,M,V)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 11 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets
The paper provides a common platform to a variety ofperformance scattered over many other papers from1992-1999
All collections from TREC 1-7 are used
Also older results from datasets such as NPL, UKCIS,Cranfield and a newly created TREC (described below) arereproduced.
Cranfield collection: has short initial manual indexdescriptions based on the whole documentNLP: has short initial automatic descriptions from abstracts
UKCIS: has only the title of the documentsTREC: has automatic initial descriptions in natural text formmostly from full documents but in some cases from abstracts.TREC is the largest of all the collectionsTREC had a mixture of Long, Medium and Very short requests(L,M,V)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 11 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets
The paper provides a common platform to a variety ofperformance scattered over many other papers from1992-1999
All collections from TREC 1-7 are used
Also older results from datasets such as NPL, UKCIS,Cranfield and a newly created TREC (described below) arereproduced.
Cranfield collection: has short initial manual indexdescriptions based on the whole documentNLP: has short initial automatic descriptions from abstractsUKCIS: has only the title of the documents
TREC: has automatic initial descriptions in natural text formmostly from full documents but in some cases from abstracts.TREC is the largest of all the collectionsTREC had a mixture of Long, Medium and Very short requests(L,M,V)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 11 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets
The paper provides a common platform to a variety ofperformance scattered over many other papers from1992-1999
All collections from TREC 1-7 are used
Also older results from datasets such as NPL, UKCIS,Cranfield and a newly created TREC (described below) arereproduced.
Cranfield collection: has short initial manual indexdescriptions based on the whole documentNLP: has short initial automatic descriptions from abstractsUKCIS: has only the title of the documentsTREC: has automatic initial descriptions in natural text formmostly from full documents but in some cases from abstracts.TREC is the largest of all the collections
TREC had a mixture of Long, Medium and Very short requests(L,M,V)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 11 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets
The paper provides a common platform to a variety ofperformance scattered over many other papers from1992-1999
All collections from TREC 1-7 are used
Also older results from datasets such as NPL, UKCIS,Cranfield and a newly created TREC (described below) arereproduced.
Cranfield collection: has short initial manual indexdescriptions based on the whole documentNLP: has short initial automatic descriptions from abstractsUKCIS: has only the title of the documentsTREC: has automatic initial descriptions in natural text formmostly from full documents but in some cases from abstracts.TREC is the largest of all the collectionsTREC had a mixture of Long, Medium and Very short requests(L,M,V)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 11 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets 2
Figure : Dataset descriptions from (Jones et al., 2000).PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 12 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Datasets 3
Figure : Dataset statistics (Jones et al., 2000).
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 13 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Outline
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 14 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
For events A and B
Joint probability P(A ∩ B) of both events occurringConditional probability P(A|B) of event A occurring given thatevent B has occurred
Chain rule gives fundamental relationship between joint andconditional probabilities:
P(AB) = P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)
Similarly for the complement of an event P(A):
P(AB) = P(B|A)P(A)
Partition rule: if B can be divided into an exhaustive set ofdisjoint subcases, then P(B) is the sum of the probabilities ofthe subcases. A special case of this rule gives:
P(B) = P(AB) + P(AB)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 15 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
For events A and B
Joint probability P(A ∩ B) of both events occurringConditional probability P(A|B) of event A occurring given thatevent B has occurred
Chain rule gives fundamental relationship between joint andconditional probabilities:
P(AB) = P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)
Similarly for the complement of an event P(A):
P(AB) = P(B|A)P(A)
Partition rule: if B can be divided into an exhaustive set ofdisjoint subcases, then P(B) is the sum of the probabilities ofthe subcases. A special case of this rule gives:
P(B) = P(AB) + P(AB)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 15 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
For events A and BJoint probability P(A ∩ B) of both events occurring
Conditional probability P(A|B) of event A occurring given thatevent B has occurred
Chain rule gives fundamental relationship between joint andconditional probabilities:
P(AB) = P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)
Similarly for the complement of an event P(A):
P(AB) = P(B|A)P(A)
Partition rule: if B can be divided into an exhaustive set ofdisjoint subcases, then P(B) is the sum of the probabilities ofthe subcases. A special case of this rule gives:
P(B) = P(AB) + P(AB)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 15 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
For events A and BJoint probability P(A ∩ B) of both events occurringConditional probability P(A|B) of event A occurring given thatevent B has occurred
Chain rule gives fundamental relationship between joint andconditional probabilities:
P(AB) = P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)
Similarly for the complement of an event P(A):
P(AB) = P(B|A)P(A)
Partition rule: if B can be divided into an exhaustive set ofdisjoint subcases, then P(B) is the sum of the probabilities ofthe subcases. A special case of this rule gives:
P(B) = P(AB) + P(AB)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 15 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
For events A and BJoint probability P(A ∩ B) of both events occurringConditional probability P(A|B) of event A occurring given thatevent B has occurred
Chain rule gives fundamental relationship between joint andconditional probabilities:
P(AB) = P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)
Similarly for the complement of an event P(A):
P(AB) = P(B|A)P(A)
Partition rule: if B can be divided into an exhaustive set ofdisjoint subcases, then P(B) is the sum of the probabilities ofthe subcases. A special case of this rule gives:
P(B) = P(AB) + P(AB)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 15 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
For events A and BJoint probability P(A ∩ B) of both events occurringConditional probability P(A|B) of event A occurring given thatevent B has occurred
Chain rule gives fundamental relationship between joint andconditional probabilities:
P(AB) = P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)
Similarly for the complement of an event P(A):
P(AB) = P(B|A)P(A)
Partition rule: if B can be divided into an exhaustive set ofdisjoint subcases, then P(B) is the sum of the probabilities ofthe subcases. A special case of this rule gives:
P(B) = P(AB) + P(AB)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 15 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
For events A and BJoint probability P(A ∩ B) of both events occurringConditional probability P(A|B) of event A occurring given thatevent B has occurred
Chain rule gives fundamental relationship between joint andconditional probabilities:
P(AB) = P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)
Similarly for the complement of an event P(A):
P(AB) = P(B|A)P(A)
Partition rule: if B can be divided into an exhaustive set ofdisjoint subcases, then P(B) is the sum of the probabilities ofthe subcases. A special case of this rule gives:
P(B) = P(AB) + P(AB)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 15 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
Bayes’ Rule for inverting conditional probabilities:
P(A|B) =P(B|A)P(A)
P(B)=
[P(B|A)∑
X∈{A,A} P(B|X )P(X )
]P(A)
Can be thought of as a way of updating probabilities:
Start off with prior probability P(A) (initial estimate of howlikely event A is in the absence of any other information)
Derive a posterior probability P(A|B) after having seen theevidence B, based on the likelihood of B occurring in the twocases that A does or does not hold
Odds of an event provide a kind of multiplier for how probabilitieschange:
Odds: O(A) =P(A)
P(A)=
P(A)
1− P(A)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 16 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
Bayes’ Rule for inverting conditional probabilities:
P(A|B) =P(B|A)P(A)
P(B)=
[P(B|A)∑
X∈{A,A} P(B|X )P(X )
]P(A)
Can be thought of as a way of updating probabilities:
Start off with prior probability P(A) (initial estimate of howlikely event A is in the absence of any other information)
Derive a posterior probability P(A|B) after having seen theevidence B, based on the likelihood of B occurring in the twocases that A does or does not hold
Odds of an event provide a kind of multiplier for how probabilitieschange:
Odds: O(A) =P(A)
P(A)=
P(A)
1− P(A)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 16 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
Bayes’ Rule for inverting conditional probabilities:
P(A|B) =P(B|A)P(A)
P(B)=
[P(B|A)∑
X∈{A,A} P(B|X )P(X )
]P(A)
Can be thought of as a way of updating probabilities:
Start off with prior probability P(A) (initial estimate of howlikely event A is in the absence of any other information)
Derive a posterior probability P(A|B) after having seen theevidence B, based on the likelihood of B occurring in the twocases that A does or does not hold
Odds of an event provide a kind of multiplier for how probabilitieschange:
Odds: O(A) =P(A)
P(A)=
P(A)
1− P(A)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 16 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
Bayes’ Rule for inverting conditional probabilities:
P(A|B) =P(B|A)P(A)
P(B)=
[P(B|A)∑
X∈{A,A} P(B|X )P(X )
]P(A)
Can be thought of as a way of updating probabilities:
Start off with prior probability P(A) (initial estimate of howlikely event A is in the absence of any other information)
Derive a posterior probability P(A|B) after having seen theevidence B, based on the likelihood of B occurring in the twocases that A does or does not hold
Odds of an event provide a kind of multiplier for how probabilitieschange:
Odds: O(A) =P(A)
P(A)=
P(A)
1− P(A)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 16 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
Bayes’ Rule for inverting conditional probabilities:
P(A|B) =P(B|A)P(A)
P(B)=
[P(B|A)∑
X∈{A,A} P(B|X )P(X )
]P(A)
Can be thought of as a way of updating probabilities:
Start off with prior probability P(A) (initial estimate of howlikely event A is in the absence of any other information)
Derive a posterior probability P(A|B) after having seen theevidence B, based on the likelihood of B occurring in the twocases that A does or does not hold
Odds of an event provide a kind of multiplier for how probabilitieschange:
Odds: O(A) =P(A)
P(A)=
P(A)
1− P(A)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 16 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Probability Theory
Bayes’ Rule for inverting conditional probabilities:
P(A|B) =P(B|A)P(A)
P(B)=
[P(B|A)∑
X∈{A,A} P(B|X )P(X )
]P(A)
Can be thought of as a way of updating probabilities:
Start off with prior probability P(A) (initial estimate of howlikely event A is in the absence of any other information)
Derive a posterior probability P(A|B) after having seen theevidence B, based on the likelihood of B occurring in the twocases that A does or does not hold
Odds of an event provide a kind of multiplier for how probabilitieschange:
Odds: O(A) =P(A)
P(A)=
P(A)
1− P(A)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 16 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model
The probability of a document (D) being relevant (L) to auser is modeled as:
P(L|D) =P(D|L)P(L)
P(D)
We use the Log-Odds to quantify the change since it can bederived from probabiliy by an oder-preserving transformation
Thus, the equation becomes:
logP(L|D)
P(L|D)= log
P(D|L)P(L)
P(D|L)P(L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 17 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model
The probability of a document (D) being relevant (L) to auser is modeled as:
P(L|D) =P(D|L)P(L)
P(D)
We use the Log-Odds to quantify the change since it can bederived from probabiliy by an oder-preserving transformation
Thus, the equation becomes:
logP(L|D)
P(L|D)= log
P(D|L)P(L)
P(D|L)P(L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 17 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model
The probability of a document (D) being relevant (L) to auser is modeled as:
P(L|D) =P(D|L)P(L)
P(D)
We use the Log-Odds to quantify the change since it can bederived from probabiliy by an oder-preserving transformation
Thus, the equation becomes:
logP(L|D)
P(L|D)= log
P(D|L)P(L)
P(D|L)P(L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 17 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model
The probability of a document (D) being relevant (L) to auser is modeled as:
P(L|D) =P(D|L)P(L)
P(D)
We use the Log-Odds to quantify the change since it can bederived from probabiliy by an oder-preserving transformation
Thus, the equation becomes:
logP(L|D)
P(L|D)= log
P(D|L)P(L)
P(D|L)P(L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 17 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model
The probability of a document (D) being relevant (L) to auser is modeled as:
P(L|D) =P(D|L)P(L)
P(D)
We use the Log-Odds to quantify the change since it can bederived from probabiliy by an oder-preserving transformation
Thus, the equation becomes:
logP(L|D)
P(L|D)= log
P(D|L)P(L)
P(D|L)P(L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 17 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model
The probability of a document (D) being relevant (L) to auser is modeled as:
P(L|D) =P(D|L)P(L)
P(D)
We use the Log-Odds to quantify the change since it can bederived from probabiliy by an oder-preserving transformation
Thus, the equation becomes:
logP(L|D)
P(L|D)= log
P(D|L)P(L)
P(D|L)P(L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 17 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model
The probability of a document (D) being relevant (L) to auser is modeled as:
P(L|D) =P(D|L)P(L)
P(D)
We use the Log-Odds to quantify the change since it can bederived from probabiliy by an oder-preserving transformation
Thus, the equation becomes:
logP(L|D)
P(L|D)= log
P(D|L)P(L)
P(D|L)P(L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 17 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Matching Score
=logP(D|L)
P(D|L)+ log
P(L)
P(L)(1)
Here we define the concept of matching score: MS(D)
Since the function is primitive, thus named MS-PRIM(D),which is as follows:
MS − PRIM(D) = logP(L|D)
P(L|D)− log
P(L)
P(L)(2)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 18 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Matching Score
=logP(D|L)
P(D|L)+ log
P(L)
P(L)(1)
Here we define the concept of matching score: MS(D)
Since the function is primitive, thus named MS-PRIM(D),which is as follows:
MS − PRIM(D) = logP(L|D)
P(L|D)− log
P(L)
P(L)(2)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 18 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes
The basic model of probabilistic IR has been built on the”Independence Assumption”, stated as:
”Given relevance, the attributes are statistically independent”
Thus, from the probability of statistically independent events,we have:
P(D|L) = ΠP(Ai = ai |L)
P(D|L) = ΠP(Ai = ai |L)
Here, Ai is the i th attribute with value equal to ai for thatspecific document
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 19 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes
The basic model of probabilistic IR has been built on the”Independence Assumption”, stated as:
”Given relevance, the attributes are statistically independent”
Thus, from the probability of statistically independent events,we have:
P(D|L) = ΠP(Ai = ai |L)
P(D|L) = ΠP(Ai = ai |L)
Here, Ai is the i th attribute with value equal to ai for thatspecific document
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 19 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes
The basic model of probabilistic IR has been built on the”Independence Assumption”, stated as:
”Given relevance, the attributes are statistically independent”
Thus, from the probability of statistically independent events,we have:
P(D|L) = ΠP(Ai = ai |L)
P(D|L) = ΠP(Ai = ai |L)
Here, Ai is the i th attribute with value equal to ai for thatspecific document
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 19 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes
The basic model of probabilistic IR has been built on the”Independence Assumption”, stated as:
”Given relevance, the attributes are statistically independent”
Thus, from the probability of statistically independent events,we have:
P(D|L) = ΠP(Ai = ai |L)
P(D|L) = ΠP(Ai = ai |L)
Here, Ai is the i th attribute with value equal to ai for thatspecific document
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 19 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (2)
The MS-PRIM(D) derived ealier can now be written as:
MS − PRIM(D) = ΣlogP(Ai = ai |L)
P(Ai = ai |L)(3)
Thus, we could calculate a score for each document, based onthe sum of the products of all independent attributes relatingto rhe document (D).This function takes into account both the relevant andnon-relevant attributesFor simplicity we take into account the relevant attributes andtake others as ”zero”We define a new measure called MS-BASIC(D) which is as:
MS − BASIC (D) = MS − PRIM(D)− ΣlogP(Ai = 0|L)
P(Ai = 0|L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 20 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (2)
The MS-PRIM(D) derived ealier can now be written as:
MS − PRIM(D) = ΣlogP(Ai = ai |L)
P(Ai = ai |L)(3)
Thus, we could calculate a score for each document, based onthe sum of the products of all independent attributes relatingto rhe document (D).
This function takes into account both the relevant andnon-relevant attributesFor simplicity we take into account the relevant attributes andtake others as ”zero”We define a new measure called MS-BASIC(D) which is as:
MS − BASIC (D) = MS − PRIM(D)− ΣlogP(Ai = 0|L)
P(Ai = 0|L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 20 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (2)
The MS-PRIM(D) derived ealier can now be written as:
MS − PRIM(D) = ΣlogP(Ai = ai |L)
P(Ai = ai |L)(3)
Thus, we could calculate a score for each document, based onthe sum of the products of all independent attributes relatingto rhe document (D).This function takes into account both the relevant andnon-relevant attributes
For simplicity we take into account the relevant attributes andtake others as ”zero”We define a new measure called MS-BASIC(D) which is as:
MS − BASIC (D) = MS − PRIM(D)− ΣlogP(Ai = 0|L)
P(Ai = 0|L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 20 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (2)
The MS-PRIM(D) derived ealier can now be written as:
MS − PRIM(D) = ΣlogP(Ai = ai |L)
P(Ai = ai |L)(3)
Thus, we could calculate a score for each document, based onthe sum of the products of all independent attributes relatingto rhe document (D).This function takes into account both the relevant andnon-relevant attributesFor simplicity we take into account the relevant attributes andtake others as ”zero”
We define a new measure called MS-BASIC(D) which is as:
MS − BASIC (D) = MS − PRIM(D)− ΣlogP(Ai = 0|L)
P(Ai = 0|L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 20 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (2)
The MS-PRIM(D) derived ealier can now be written as:
MS − PRIM(D) = ΣlogP(Ai = ai |L)
P(Ai = ai |L)(3)
Thus, we could calculate a score for each document, based onthe sum of the products of all independent attributes relatingto rhe document (D).This function takes into account both the relevant andnon-relevant attributesFor simplicity we take into account the relevant attributes andtake others as ”zero”We define a new measure called MS-BASIC(D) which is as:
MS − BASIC (D) = MS − PRIM(D)− ΣlogP(Ai = 0|L)
P(Ai = 0|L)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 20 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (3)
Plugging in the value of MS-PRIM(D) from equation 4, wehave:
MS−BASIC (D) = ΣlogP(Ai = ai |L)
P(Ai = ai |L)− log
P(Ai = 0|L)
P(Ai = 0|L)(4)
which can be further simplified to:
=ΣlogP(Ai = ai |L)P(Ai = 0|L)
P(Ai = ai |L)P(Ai = 0|L)(5)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 21 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (3)
Plugging in the value of MS-PRIM(D) from equation 4, wehave:
MS−BASIC (D) = ΣlogP(Ai = ai |L)
P(Ai = ai |L)− log
P(Ai = 0|L)
P(Ai = 0|L)(4)
which can be further simplified to:
=ΣlogP(Ai = ai |L)P(Ai = 0|L)
P(Ai = ai |L)P(Ai = 0|L)(5)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 21 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (3)
We define this a our Weight function, W (Ai = ai ), Thus,
MS − BASIC (D) = ΣW (Ai = ai )
This function (W), provides a weight for each value of eachattribute and the matching score for document is somply thesum of the weights.
W (Ai = 0) is always zero, i.e. for a randomly chosen term,which is irrelevant to the query, we can reasonably assume theweight to be zero.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 22 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (3)
We define this a our Weight function, W (Ai = ai ), Thus,
MS − BASIC (D) = ΣW (Ai = ai )
This function (W), provides a weight for each value of eachattribute and the matching score for document is somply thesum of the weights.
W (Ai = 0) is always zero, i.e. for a randomly chosen term,which is irrelevant to the query, we can reasonably assume theweight to be zero.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 22 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model - Independent attributes (3)
We define this a our Weight function, W (Ai = ai ), Thus,
MS − BASIC (D) = ΣW (Ai = ai )
This function (W), provides a weight for each value of eachattribute and the matching score for document is somply thesum of the weights.
W (Ai = 0) is always zero, i.e. for a randomly chosen term,which is irrelevant to the query, we can reasonably assume theweight to be zero.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 22 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model-Term presence and absence
We can further simplify the above model by using the casewhere attribute Ai is simply the presence or absence of a termti
We denote P(tipresent|L) by pi and P(tipresent|L) by pi
Substituting in previous eqation, the new weight formulabecomes:
wi = Σlogpi (1− pi )
pi (1− pi )(6)
Hence, the matching score for the document is just the sumof the weights of the terms present.
This formula is later used in the BIM termed as RSV function.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 23 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model-Term presence and absence
We can further simplify the above model by using the casewhere attribute Ai is simply the presence or absence of a termti
We denote P(tipresent|L) by pi and P(tipresent|L) by pi
Substituting in previous eqation, the new weight formulabecomes:
wi = Σlogpi (1− pi )
pi (1− pi )(6)
Hence, the matching score for the document is just the sumof the weights of the terms present.
This formula is later used in the BIM termed as RSV function.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 23 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model-Term presence and absence
We can further simplify the above model by using the casewhere attribute Ai is simply the presence or absence of a termti
We denote P(tipresent|L) by pi and P(tipresent|L) by pi
Substituting in previous eqation, the new weight formulabecomes:
wi = Σlogpi (1− pi )
pi (1− pi )(6)
Hence, the matching score for the document is just the sumof the weights of the terms present.
This formula is later used in the BIM termed as RSV function.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 23 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model-Term presence and absence
We can further simplify the above model by using the casewhere attribute Ai is simply the presence or absence of a termti
We denote P(tipresent|L) by pi and P(tipresent|L) by pi
Substituting in previous eqation, the new weight formulabecomes:
wi = Σlogpi (1− pi )
pi (1− pi )(6)
Hence, the matching score for the document is just the sumof the weights of the terms present.
This formula is later used in the BIM termed as RSV function.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 23 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Basic Model-Term presence and absence
We can further simplify the above model by using the casewhere attribute Ai is simply the presence or absence of a termti
We denote P(tipresent|L) by pi and P(tipresent|L) by pi
Substituting in previous eqation, the new weight formulabecomes:
wi = Σlogpi (1− pi )
pi (1− pi )(6)
Hence, the matching score for the document is just the sumof the weights of the terms present.
This formula is later used in the BIM termed as RSV function.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 23 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Outline
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 24 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
The Document Ranking Problem
Ranked retrieval setup: given a collection of documents, theuser issues a query, and an ordered list of documents isreturned
Assume binary notion of relevance: Rd ,q is a randomdichotomous variable, such that
Rd,q = 1 if document d is relevant w.r.t query qRd,q = 0 otherwise
Probabilistic ranking orders documents decreasingly by theirestimated probability of relevance w.r.t. query: P(R = 1|d , q)
Assume that the relevance of each document is independentof the relevance of other documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 25 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
The Document Ranking Problem
Ranked retrieval setup: given a collection of documents, theuser issues a query, and an ordered list of documents isreturned
Assume binary notion of relevance: Rd ,q is a randomdichotomous variable, such that
Rd,q = 1 if document d is relevant w.r.t query qRd,q = 0 otherwise
Probabilistic ranking orders documents decreasingly by theirestimated probability of relevance w.r.t. query: P(R = 1|d , q)
Assume that the relevance of each document is independentof the relevance of other documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 25 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
The Document Ranking Problem
Ranked retrieval setup: given a collection of documents, theuser issues a query, and an ordered list of documents isreturned
Assume binary notion of relevance: Rd ,q is a randomdichotomous variable, such that
Rd,q = 1 if document d is relevant w.r.t query qRd,q = 0 otherwise
Probabilistic ranking orders documents decreasingly by theirestimated probability of relevance w.r.t. query: P(R = 1|d , q)
Assume that the relevance of each document is independentof the relevance of other documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 25 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
The Document Ranking Problem
Ranked retrieval setup: given a collection of documents, theuser issues a query, and an ordered list of documents isreturned
Assume binary notion of relevance: Rd ,q is a randomdichotomous variable, such that
Rd,q = 1 if document d is relevant w.r.t query q
Rd,q = 0 otherwise
Probabilistic ranking orders documents decreasingly by theirestimated probability of relevance w.r.t. query: P(R = 1|d , q)
Assume that the relevance of each document is independentof the relevance of other documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 25 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
The Document Ranking Problem
Ranked retrieval setup: given a collection of documents, theuser issues a query, and an ordered list of documents isreturned
Assume binary notion of relevance: Rd ,q is a randomdichotomous variable, such that
Rd,q = 1 if document d is relevant w.r.t query qRd,q = 0 otherwise
Probabilistic ranking orders documents decreasingly by theirestimated probability of relevance w.r.t. query: P(R = 1|d , q)
Assume that the relevance of each document is independentof the relevance of other documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 25 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
The Document Ranking Problem
Ranked retrieval setup: given a collection of documents, theuser issues a query, and an ordered list of documents isreturned
Assume binary notion of relevance: Rd ,q is a randomdichotomous variable, such that
Rd,q = 1 if document d is relevant w.r.t query qRd,q = 0 otherwise
Probabilistic ranking orders documents decreasingly by theirestimated probability of relevance w.r.t. query: P(R = 1|d , q)
Assume that the relevance of each document is independentof the relevance of other documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 25 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
The Document Ranking Problem
Ranked retrieval setup: given a collection of documents, theuser issues a query, and an ordered list of documents isreturned
Assume binary notion of relevance: Rd ,q is a randomdichotomous variable, such that
Rd,q = 1 if document d is relevant w.r.t query qRd,q = 0 otherwise
Probabilistic ranking orders documents decreasingly by theirestimated probability of relevance w.r.t. query: P(R = 1|d , q)
Assume that the relevance of each document is independentof the relevance of other documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 25 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability Ranking Principle (PRP)
PRP in brief
If the retrieved documents (w.r.t a query) are rankeddecreasingly on their probability of relevance, then theeffectiveness of the system will be the best that is obtainable
PRP in full
If [the IR] system’s response to each [query] is a ranking of thedocuments [...] in order of decreasing probability of relevanceto the [query], where the probabilities are estimated asaccurately as possible on the basis of whatever data have beenmade available to the system for this purpose, the overalleffectiveness of the system to its user will be the best that isobtainable on the basis of those data
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 26 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability Ranking Principle (PRP)
PRP in brief
If the retrieved documents (w.r.t a query) are rankeddecreasingly on their probability of relevance, then theeffectiveness of the system will be the best that is obtainable
PRP in full
If [the IR] system’s response to each [query] is a ranking of thedocuments [...] in order of decreasing probability of relevanceto the [query], where the probabilities are estimated asaccurately as possible on the basis of whatever data have beenmade available to the system for this purpose, the overalleffectiveness of the system to its user will be the best that isobtainable on the basis of those data
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 26 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability Ranking Principle (PRP)
PRP in brief
If the retrieved documents (w.r.t a query) are rankeddecreasingly on their probability of relevance, then theeffectiveness of the system will be the best that is obtainable
PRP in full
If [the IR] system’s response to each [query] is a ranking of thedocuments [...] in order of decreasing probability of relevanceto the [query], where the probabilities are estimated asaccurately as possible on the basis of whatever data have beenmade available to the system for this purpose, the overalleffectiveness of the system to its user will be the best that isobtainable on the basis of those data
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 26 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability Ranking Principle (PRP)
PRP in brief
If the retrieved documents (w.r.t a query) are rankeddecreasingly on their probability of relevance, then theeffectiveness of the system will be the best that is obtainable
PRP in full
If [the IR] system’s response to each [query] is a ranking of thedocuments [...] in order of decreasing probability of relevanceto the [query], where the probabilities are estimated asaccurately as possible on the basis of whatever data have beenmade available to the system for this purpose, the overalleffectiveness of the system to its user will be the best that isobtainable on the basis of those data
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 26 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability Ranking Principle (PRP)
PRP in brief
If the retrieved documents (w.r.t a query) are rankeddecreasingly on their probability of relevance, then theeffectiveness of the system will be the best that is obtainable
PRP in full
If [the IR] system’s response to each [query] is a ranking of thedocuments [...] in order of decreasing probability of relevanceto the [query], where the probabilities are estimated asaccurately as possible on the basis of whatever data have beenmade available to the system for this purpose, the overalleffectiveness of the system to its user will be the best that isobtainable on the basis of those data
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 26 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model (BIM)
Traditionally used with the PRP
Assumptions:
‘Binary’ (equivalent to Boolean): documents and queriesrepresented as binary term incidence vectors
E.g., document d represented by vector ~x = (x1, . . . , xM),where xt = 1 if term t occurs in d and xt = 0 otherwiseDifferent documents may have the same vector representation
‘Independence’: no association between terms (not true, butpractically works - ‘naive’ assumption of Naive Bayes models)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 27 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model (BIM)
Traditionally used with the PRP
Assumptions:
‘Binary’ (equivalent to Boolean): documents and queriesrepresented as binary term incidence vectors
E.g., document d represented by vector ~x = (x1, . . . , xM),where xt = 1 if term t occurs in d and xt = 0 otherwiseDifferent documents may have the same vector representation
‘Independence’: no association between terms (not true, butpractically works - ‘naive’ assumption of Naive Bayes models)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 27 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model (BIM)
Traditionally used with the PRP
Assumptions:
‘Binary’ (equivalent to Boolean): documents and queriesrepresented as binary term incidence vectors
E.g., document d represented by vector ~x = (x1, . . . , xM),where xt = 1 if term t occurs in d and xt = 0 otherwiseDifferent documents may have the same vector representation
‘Independence’: no association between terms (not true, butpractically works - ‘naive’ assumption of Naive Bayes models)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 27 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model (BIM)
Traditionally used with the PRP
Assumptions:
‘Binary’ (equivalent to Boolean): documents and queriesrepresented as binary term incidence vectors
E.g., document d represented by vector ~x = (x1, . . . , xM),where xt = 1 if term t occurs in d and xt = 0 otherwise
Different documents may have the same vector representation
‘Independence’: no association between terms (not true, butpractically works - ‘naive’ assumption of Naive Bayes models)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 27 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model (BIM)
Traditionally used with the PRP
Assumptions:
‘Binary’ (equivalent to Boolean): documents and queriesrepresented as binary term incidence vectors
E.g., document d represented by vector ~x = (x1, . . . , xM),where xt = 1 if term t occurs in d and xt = 0 otherwiseDifferent documents may have the same vector representation
‘Independence’: no association between terms (not true, butpractically works - ‘naive’ assumption of Naive Bayes models)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 27 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model (BIM)
Traditionally used with the PRP
Assumptions:
‘Binary’ (equivalent to Boolean): documents and queriesrepresented as binary term incidence vectors
E.g., document d represented by vector ~x = (x1, . . . , xM),where xt = 1 if term t occurs in d and xt = 0 otherwiseDifferent documents may have the same vector representation
‘Independence’: no association between terms (not true, butpractically works - ‘naive’ assumption of Naive Bayes models)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 27 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary incidence matrix
Anthony Julius The Hamlet Othello Macbeth . . .and Caesar Tempest
CleopatraAnthony 1 1 0 0 0 1Brutus 1 1 0 1 0 0Caesar 1 1 0 1 1 1Calpurnia 0 1 0 0 0 0Cleopatra 1 0 0 0 0 0mercy 1 0 1 1 1 1worser 1 0 1 1 1 0. . .
Each document is represented as a binary vector ∈ {0, 1}|V |.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 28 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
To make a probabilistic retrieval strategy precise, need to estimatehow terms in documents contribute to relevance
Find measurable statistics (term frequency, documentfrequency, document length) that affect judgments aboutdocument relevance
Combine these statistics to estimate the probability P(R|d , q)of document relevance
Next: how exactly we can do this
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 29 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
To make a probabilistic retrieval strategy precise, need to estimatehow terms in documents contribute to relevance
Find measurable statistics (term frequency, documentfrequency, document length) that affect judgments aboutdocument relevance
Combine these statistics to estimate the probability P(R|d , q)of document relevance
Next: how exactly we can do this
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 29 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
To make a probabilistic retrieval strategy precise, need to estimatehow terms in documents contribute to relevance
Find measurable statistics (term frequency, documentfrequency, document length) that affect judgments aboutdocument relevance
Combine these statistics to estimate the probability P(R|d , q)of document relevance
Next: how exactly we can do this
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 29 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
To make a probabilistic retrieval strategy precise, need to estimatehow terms in documents contribute to relevance
Find measurable statistics (term frequency, documentfrequency, document length) that affect judgments aboutdocument relevance
Combine these statistics to estimate the probability P(R|d , q)of document relevance
Next: how exactly we can do this
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 29 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
P(R|d , q) is modeled using term incidence vectors as P(R|~x , ~q)
P(R = 1|~x , ~q) =P(~x |R = 1, ~q)P(R = 1|~q)
P(~x |~q)
P(R = 0|~x , ~q) =P(~x |R = 0, ~q)P(R = 0|~q)
P(~x |~q)
P(~x |R = 1, ~q) and P(~x |R = 0, ~q): probability that if arelevant or nonrelevant document is retrieved, then thatdocument’s representation is ~x
Use statistics about the document collection to estimate theseprobabilities
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 30 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
P(R|d , q) is modeled using term incidence vectors as P(R|~x , ~q)
P(R = 1|~x , ~q) =P(~x |R = 1, ~q)P(R = 1|~q)
P(~x |~q)
P(R = 0|~x , ~q) =P(~x |R = 0, ~q)P(R = 0|~q)
P(~x |~q)
P(~x |R = 1, ~q) and P(~x |R = 0, ~q): probability that if arelevant or nonrelevant document is retrieved, then thatdocument’s representation is ~x
Use statistics about the document collection to estimate theseprobabilities
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 30 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
P(R|d , q) is modeled using term incidence vectors as P(R|~x , ~q)
P(R = 1|~x , ~q) =P(~x |R = 1, ~q)P(R = 1|~q)
P(~x |~q)
P(R = 0|~x , ~q) =P(~x |R = 0, ~q)P(R = 0|~q)
P(~x |~q)
P(~x |R = 1, ~q) and P(~x |R = 0, ~q): probability that if arelevant or nonrelevant document is retrieved, then thatdocument’s representation is ~x
Use statistics about the document collection to estimate theseprobabilities
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 30 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
P(R|d , q) is modeled using term incidence vectors as P(R|~x , ~q)
P(R = 1|~x , ~q) =P(~x |R = 1, ~q)P(R = 1|~q)
P(~x |~q)
P(R = 0|~x , ~q) =P(~x |R = 0, ~q)P(R = 0|~q)
P(~x |~q)
P(R = 1|~q) and P(R = 0|~q): prior probability of retrieving arelevant or nonrelevant document for a query ~q
Estimate P(R = 1|~q) and P(R = 0|~q) from percentage ofrelevant documents in the collection
Since a document is either relevant or nonrelevant to a query,we must have that:
P(R = 1|~x , ~q) + P(R = 0|~x , ~q) = 1
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 31 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
P(R|d , q) is modeled using term incidence vectors as P(R|~x , ~q)
P(R = 1|~x , ~q) =P(~x |R = 1, ~q)P(R = 1|~q)
P(~x |~q)
P(R = 0|~x , ~q) =P(~x |R = 0, ~q)P(R = 0|~q)
P(~x |~q)
P(R = 1|~q) and P(R = 0|~q): prior probability of retrieving arelevant or nonrelevant document for a query ~q
Estimate P(R = 1|~q) and P(R = 0|~q) from percentage ofrelevant documents in the collection
Since a document is either relevant or nonrelevant to a query,we must have that:
P(R = 1|~x , ~q) + P(R = 0|~x , ~q) = 1
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 31 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
P(R|d , q) is modeled using term incidence vectors as P(R|~x , ~q)
P(R = 1|~x , ~q) =P(~x |R = 1, ~q)P(R = 1|~q)
P(~x |~q)
P(R = 0|~x , ~q) =P(~x |R = 0, ~q)P(R = 0|~q)
P(~x |~q)
P(R = 1|~q) and P(R = 0|~q): prior probability of retrieving arelevant or nonrelevant document for a query ~q
Estimate P(R = 1|~q) and P(R = 0|~q) from percentage ofrelevant documents in the collection
Since a document is either relevant or nonrelevant to a query,we must have that:
P(R = 1|~x , ~q) + P(R = 0|~x , ~q) = 1
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 31 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
P(R|d , q) is modeled using term incidence vectors as P(R|~x , ~q)
P(R = 1|~x , ~q) =P(~x |R = 1, ~q)P(R = 1|~q)
P(~x |~q)
P(R = 0|~x , ~q) =P(~x |R = 0, ~q)P(R = 0|~q)
P(~x |~q)
P(R = 1|~q) and P(R = 0|~q): prior probability of retrieving arelevant or nonrelevant document for a query ~q
Estimate P(R = 1|~q) and P(R = 0|~q) from percentage ofrelevant documents in the collection
Since a document is either relevant or nonrelevant to a query,we must have that:
P(R = 1|~x , ~q) + P(R = 0|~x , ~q) = 1
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 31 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Binary Independence Model
P(R|d , q) is modeled using term incidence vectors as P(R|~x , ~q)
P(R = 1|~x , ~q) =P(~x |R = 1, ~q)P(R = 1|~q)
P(~x |~q)
P(R = 0|~x , ~q) =P(~x |R = 0, ~q)P(R = 0|~q)
P(~x |~q)
P(R = 1|~q) and P(R = 0|~q): prior probability of retrieving arelevant or nonrelevant document for a query ~q
Estimate P(R = 1|~q) and P(R = 0|~q) from percentage ofrelevant documents in the collection
Since a document is either relevant or nonrelevant to a query,we must have that:
P(R = 1|~x , ~q) + P(R = 0|~x , ~q) = 1
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 31 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (1)
Given a query q, ranking documents by P(R = 1|d , q) ismodeled under BIM as ranking them by P(R = 1|~x , ~q)
Easier: rank documents by their odds of relevance (gives sameranking)
O(R|~x , ~q) =P(R = 1|~x , ~q)
P(R = 0|~x , ~q)=
P(R=1|~q)P(~x |R=1,~q)P(~x |~q)
P(R=0|~q)P(~x |R=0,~q)P(~x |~q)
=P(R = 1|~q)
P(R = 0|~q)· P(~x |R = 1, ~q)
P(~x |R = 0, ~q)
P(R=1|~q)P(R=0|~q) is a constant for a given query - can be ignored
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 32 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (1)
Given a query q, ranking documents by P(R = 1|d , q) ismodeled under BIM as ranking them by P(R = 1|~x , ~q)
Easier: rank documents by their odds of relevance (gives sameranking)
O(R|~x , ~q) =P(R = 1|~x , ~q)
P(R = 0|~x , ~q)=
P(R=1|~q)P(~x |R=1,~q)P(~x |~q)
P(R=0|~q)P(~x |R=0,~q)P(~x |~q)
=P(R = 1|~q)
P(R = 0|~q)· P(~x |R = 1, ~q)
P(~x |R = 0, ~q)
P(R=1|~q)P(R=0|~q) is a constant for a given query - can be ignored
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 32 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (2)
It is at this point that we make the Naive Bayes conditionalindependence assumption that the presence or absence of a word ina document is independent of the presence or absence of any otherword (given the query):
P(~x |R = 1, ~q)
P(~x |R = 0, ~q)=
M∏t=1
P(xt |R = 1, ~q)
P(xt |R = 0, ~q)
So:
O(R|~x , ~q) = O(R|~q) ·M∏t=1
P(xt |R = 1, ~q)
P(xt |R = 0, ~q)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 33 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (2)
It is at this point that we make the Naive Bayes conditionalindependence assumption that the presence or absence of a word ina document is independent of the presence or absence of any otherword (given the query):
P(~x |R = 1, ~q)
P(~x |R = 0, ~q)=
M∏t=1
P(xt |R = 1, ~q)
P(xt |R = 0, ~q)
So:
O(R|~x , ~q) = O(R|~q) ·M∏t=1
P(xt |R = 1, ~q)
P(xt |R = 0, ~q)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 33 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (3)
Since each xt is either 0 or 1, we can separate the terms:
O(R|~x , ~q) = O(R|~q)·∏
t:xt=1
P(xt = 1|R = 1, ~q)
P(xt = 1|R = 0, ~q)·∏
t:xt=0
P(xt = 0|R = 1, ~q)
P(xt = 0|R = 0, ~q)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 34 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (3)
Since each xt is either 0 or 1, we can separate the terms:
O(R|~x , ~q) = O(R|~q)·∏
t:xt=1
P(xt = 1|R = 1, ~q)
P(xt = 1|R = 0, ~q)·∏
t:xt=0
P(xt = 0|R = 1, ~q)
P(xt = 0|R = 0, ~q)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 34 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (3)
Since each xt is either 0 or 1, we can separate the terms:
O(R|~x , ~q) = O(R|~q)·∏
t:xt=1
P(xt = 1|R = 1, ~q)
P(xt = 1|R = 0, ~q)·∏
t:xt=0
P(xt = 0|R = 1, ~q)
P(xt = 0|R = 0, ~q)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 34 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (4)
Let pt = P(xt = 1|R = 1, ~q) be the probability of a termappearing in relevant document
Let pt = P(xt = 1|R = 0, ~q) be the probability of a termappearing in a nonrelevant document
Can be displayed as contingency table:
document relevant (R = 1) nonrelevant (R = 0)
Term present xt = 1 pt ptTerm absent xt = 0 1− pt 1− pt
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 35 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (4)
Let pt = P(xt = 1|R = 1, ~q) be the probability of a termappearing in relevant document
Let pt = P(xt = 1|R = 0, ~q) be the probability of a termappearing in a nonrelevant document
Can be displayed as contingency table:
document relevant (R = 1) nonrelevant (R = 0)
Term present xt = 1 pt ptTerm absent xt = 0 1− pt 1− pt
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 35 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (4)
Let pt = P(xt = 1|R = 1, ~q) be the probability of a termappearing in relevant document
Let pt = P(xt = 1|R = 0, ~q) be the probability of a termappearing in a nonrelevant document
Can be displayed as contingency table:
document relevant (R = 1) nonrelevant (R = 0)
Term present xt = 1 pt ptTerm absent xt = 0 1− pt 1− pt
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 35 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (4)
Let pt = P(xt = 1|R = 1, ~q) be the probability of a termappearing in relevant document
Let pt = P(xt = 1|R = 0, ~q) be the probability of a termappearing in a nonrelevant document
Can be displayed as contingency table:
document relevant (R = 1) nonrelevant (R = 0)
Term present xt = 1 pt ptTerm absent xt = 0 1− pt 1− pt
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 35 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (4)
Let pt = P(xt = 1|R = 1, ~q) be the probability of a termappearing in relevant document
Let pt = P(xt = 1|R = 0, ~q) be the probability of a termappearing in a nonrelevant document
Can be displayed as contingency table:
document relevant (R = 1) nonrelevant (R = 0)
Term present xt = 1 pt ptTerm absent xt = 0 1− pt 1− pt
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 35 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms (4)
Let pt = P(xt = 1|R = 1, ~q) be the probability of a termappearing in relevant document
Let pt = P(xt = 1|R = 0, ~q) be the probability of a termappearing in a nonrelevant document
Can be displayed as contingency table:
document relevant (R = 1) nonrelevant (R = 0)
Term present xt = 1 pt ptTerm absent xt = 0 1− pt 1− pt
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 35 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Additional simplifying assumption: terms not occurring in thequery are equally likely to occur in relevant and nonrelevantdocuments
If qt = 0, then pt = pt
Now we need only to consider terms in the products that appear inthe query:
O(R|~x , ~q) = O(R|~q) ·∏
t:xt=qt=1
ptpt·
∏t:xt=0,qt=1
1− pt1− pt
The left product is over query terms found in the documentand the right product is over query terms not found in thedocument
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 36 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Additional simplifying assumption: terms not occurring in thequery are equally likely to occur in relevant and nonrelevantdocuments
If qt = 0, then pt = pt
Now we need only to consider terms in the products that appear inthe query:
O(R|~x , ~q) = O(R|~q) ·∏
t:xt=qt=1
ptpt·
∏t:xt=0,qt=1
1− pt1− pt
The left product is over query terms found in the documentand the right product is over query terms not found in thedocument
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 36 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Additional simplifying assumption: terms not occurring in thequery are equally likely to occur in relevant and nonrelevantdocuments
If qt = 0, then pt = pt
Now we need only to consider terms in the products that appear inthe query:
O(R|~x , ~q) = O(R|~q) ·∏
t:xt=qt=1
ptpt·
∏t:xt=0,qt=1
1− pt1− pt
The left product is over query terms found in the documentand the right product is over query terms not found in thedocument
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 36 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Including the query terms found in the document into the rightproduct, but simultaneously dividing by them in the left product,gives:
O(R|~x , ~q) = O(R|~q) ·∏
t:xt=qt=1
pt(1− pt)
pt(1− pt)·∏
t:qt=1
1− pt1− pt
The left product is still over query terms found in thedocument, but the right product is now over all query terms,hence constant for a particular query and can be ignored.
→ The only quantity that needs to be estimated to rankdocuments w.r.t a query is the left product
Hence the Retrieval Status Value (RSV) in this model:
RSVd = log∏
t:xt=qt=1
pt(1− pt)
pt(1− pt)=
∑t:xt=qt=1
logpt(1− pt)
pt(1− pt)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 37 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Including the query terms found in the document into the rightproduct, but simultaneously dividing by them in the left product,gives:
O(R|~x , ~q) = O(R|~q) ·∏
t:xt=qt=1
pt(1− pt)
pt(1− pt)·∏
t:qt=1
1− pt1− pt
The left product is still over query terms found in thedocument, but the right product is now over all query terms,hence constant for a particular query and can be ignored.
→ The only quantity that needs to be estimated to rankdocuments w.r.t a query is the left product
Hence the Retrieval Status Value (RSV) in this model:
RSVd = log∏
t:xt=qt=1
pt(1− pt)
pt(1− pt)=
∑t:xt=qt=1
logpt(1− pt)
pt(1− pt)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 37 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Including the query terms found in the document into the rightproduct, but simultaneously dividing by them in the left product,gives:
O(R|~x , ~q) = O(R|~q) ·∏
t:xt=qt=1
pt(1− pt)
pt(1− pt)·∏
t:qt=1
1− pt1− pt
The left product is still over query terms found in thedocument, but the right product is now over all query terms,hence constant for a particular query and can be ignored.
→ The only quantity that needs to be estimated to rankdocuments w.r.t a query is the left product
Hence the Retrieval Status Value (RSV) in this model:
RSVd = log∏
t:xt=qt=1
pt(1− pt)
pt(1− pt)=
∑t:xt=qt=1
logpt(1− pt)
pt(1− pt)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 37 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Including the query terms found in the document into the rightproduct, but simultaneously dividing by them in the left product,gives:
O(R|~x , ~q) = O(R|~q) ·∏
t:xt=qt=1
pt(1− pt)
pt(1− pt)·∏
t:qt=1
1− pt1− pt
The left product is still over query terms found in thedocument, but the right product is now over all query terms,hence constant for a particular query and can be ignored.
→ The only quantity that needs to be estimated to rankdocuments w.r.t a query is the left product
Hence the Retrieval Status Value (RSV) in this model:
RSVd = log∏
t:xt=qt=1
pt(1− pt)
pt(1− pt)=
∑t:xt=qt=1
logpt(1− pt)
pt(1− pt)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 37 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Equivalent: rank documents using the log odds ratios for the termsin the query ct :
ct = logpt(1− pt)
pt(1− pt)= log
pt(1− pt)
− logpt
1− pt
The odds ratio is the ratio of two odds: (i) the odds of theterm appearing if the document is relevant (pt/(1− pt)), and(ii) the odds of the term appearing if the document isnonrelevant (pt/(1− pt))
ct = 0: term has equal odds of appearing in relevant andnonrelevant docs
ct positive: higher odds to appear in relevant documents
ct negative: higher odds to appear in nonrelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 38 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Equivalent: rank documents using the log odds ratios for the termsin the query ct :
ct = logpt(1− pt)
pt(1− pt)= log
pt(1− pt)
− logpt
1− pt
The odds ratio is the ratio of two odds: (i) the odds of theterm appearing if the document is relevant (pt/(1− pt)), and(ii) the odds of the term appearing if the document isnonrelevant (pt/(1− pt))
ct = 0: term has equal odds of appearing in relevant andnonrelevant docs
ct positive: higher odds to appear in relevant documents
ct negative: higher odds to appear in nonrelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 38 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Equivalent: rank documents using the log odds ratios for the termsin the query ct :
ct = logpt(1− pt)
pt(1− pt)= log
pt(1− pt)
− logpt
1− pt
The odds ratio is the ratio of two odds: (i) the odds of theterm appearing if the document is relevant (pt/(1− pt)), and(ii) the odds of the term appearing if the document isnonrelevant (pt/(1− pt))
ct = 0: term has equal odds of appearing in relevant andnonrelevant docs
ct positive: higher odds to appear in relevant documents
ct negative: higher odds to appear in nonrelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 38 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Equivalent: rank documents using the log odds ratios for the termsin the query ct :
ct = logpt(1− pt)
pt(1− pt)= log
pt(1− pt)
− logpt
1− pt
The odds ratio is the ratio of two odds: (i) the odds of theterm appearing if the document is relevant (pt/(1− pt)), and(ii) the odds of the term appearing if the document isnonrelevant (pt/(1− pt))
ct = 0: term has equal odds of appearing in relevant andnonrelevant docs
ct positive: higher odds to appear in relevant documents
ct negative: higher odds to appear in nonrelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 38 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Deriving a Ranking Function for Query Terms
Equivalent: rank documents using the log odds ratios for the termsin the query ct :
ct = logpt(1− pt)
pt(1− pt)= log
pt(1− pt)
− logpt
1− pt
The odds ratio is the ratio of two odds: (i) the odds of theterm appearing if the document is relevant (pt/(1− pt)), and(ii) the odds of the term appearing if the document isnonrelevant (pt/(1− pt))
ct = 0: term has equal odds of appearing in relevant andnonrelevant docs
ct positive: higher odds to appear in relevant documents
ct negative: higher odds to appear in nonrelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 38 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Term weight ct in BIM
ct = log pt(1−pt) − log pt
1−pt functions as a term weight.
Retrieval status value for document d : RSVd =∑
xt=qt=1 ct .
So BIM and vector space model are identical on anoperational level . . .
. . . except that the term weights are different.
In particular: we can use the same data structures (invertedindex etc) for the two models.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 39 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Term weight ct in BIM
ct = log pt(1−pt) − log pt
1−pt functions as a term weight.
Retrieval status value for document d : RSVd =∑
xt=qt=1 ct .
So BIM and vector space model are identical on anoperational level . . .
. . . except that the term weights are different.
In particular: we can use the same data structures (invertedindex etc) for the two models.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 39 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Term weight ct in BIM
ct = log pt(1−pt) − log pt
1−pt functions as a term weight.
Retrieval status value for document d : RSVd =∑
xt=qt=1 ct .
So BIM and vector space model are identical on anoperational level . . .
. . . except that the term weights are different.
In particular: we can use the same data structures (invertedindex etc) for the two models.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 39 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Term weight ct in BIM
ct = log pt(1−pt) − log pt
1−pt functions as a term weight.
Retrieval status value for document d : RSVd =∑
xt=qt=1 ct .
So BIM and vector space model are identical on anoperational level . . .
. . . except that the term weights are different.
In particular: we can use the same data structures (invertedindex etc) for the two models.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 39 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Term weight ct in BIM
ct = log pt(1−pt) − log pt
1−pt functions as a term weight.
Retrieval status value for document d : RSVd =∑
xt=qt=1 ct .
So BIM and vector space model are identical on anoperational level . . .
. . . except that the term weights are different.
In particular: we can use the same data structures (invertedindex etc) for the two models.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 39 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Term weight ct in BIM
ct = log pt(1−pt) − log pt
1−pt functions as a term weight.
Retrieval status value for document d : RSVd =∑
xt=qt=1 ct .
So BIM and vector space model are identical on anoperational level . . .
. . . except that the term weights are different.
In particular: we can use the same data structures (invertedindex etc) for the two models.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 39 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
How to compute probability estimates
For each term t in a query, estimate ct in the whole collectionusing a contingency table of counts of documents in thecollection, where n is the number of documents that contain term t:
documents relevant nonrelevant Total
Term present xt = 1 r n − r nTerm absent xt = 0 R − r (N − n)− (R − r) N − n
Total R N − R N
pt = r/R
pt = (n − r)/(N − R)
ct = K (N, n,R, r) = logr/(R − r)
(n − r)/((N − n)− (R − r))
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 40 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
How to compute probability estimates
For each term t in a query, estimate ct in the whole collectionusing a contingency table of counts of documents in thecollection, where n is the number of documents that contain term t:
documents relevant nonrelevant Total
Term present xt = 1 r n − r nTerm absent xt = 0 R − r (N − n)− (R − r) N − n
Total R N − R N
pt = r/R
pt = (n − r)/(N − R)
ct = K (N, n,R, r) = logr/(R − r)
(n − r)/((N − n)− (R − r))
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 40 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Avoiding zeros
If any of the counts is a zero, then the term weight is notwell-defined.
Maximum likelihood estimates do not work for rare events.
To avoid zeros: add 0.5 to each count (expected likelihoodestimation = ELE)
For example, use R − r + 0.5 in formula for R − r . Thus,fomula becomes:
RSVd = Σ logr + 0.5/(R − r + 0.5)
(n − r + 0.5)/((N − n + 0.5)− (R − r + 0.5))(7)
This was previously known as relevance weight (RW) ormatching score relevance weight (MS-RW) (Roberston andJones, 1976)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 41 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Avoiding zeros
If any of the counts is a zero, then the term weight is notwell-defined.
Maximum likelihood estimates do not work for rare events.
To avoid zeros: add 0.5 to each count (expected likelihoodestimation = ELE)
For example, use R − r + 0.5 in formula for R − r . Thus,fomula becomes:
RSVd = Σ logr + 0.5/(R − r + 0.5)
(n − r + 0.5)/((N − n + 0.5)− (R − r + 0.5))(7)
This was previously known as relevance weight (RW) ormatching score relevance weight (MS-RW) (Roberston andJones, 1976)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 41 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Avoiding zeros
If any of the counts is a zero, then the term weight is notwell-defined.
Maximum likelihood estimates do not work for rare events.
To avoid zeros: add 0.5 to each count (expected likelihoodestimation = ELE)
For example, use R − r + 0.5 in formula for R − r . Thus,fomula becomes:
RSVd = Σ logr + 0.5/(R − r + 0.5)
(n − r + 0.5)/((N − n + 0.5)− (R − r + 0.5))(7)
This was previously known as relevance weight (RW) ormatching score relevance weight (MS-RW) (Roberston andJones, 1976)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 41 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Avoiding zeros
If any of the counts is a zero, then the term weight is notwell-defined.
Maximum likelihood estimates do not work for rare events.
To avoid zeros: add 0.5 to each count (expected likelihoodestimation = ELE)
For example, use R − r + 0.5 in formula for R − r . Thus,fomula becomes:
RSVd = Σ logr + 0.5/(R − r + 0.5)
(n − r + 0.5)/((N − n + 0.5)− (R − r + 0.5))(7)
This was previously known as relevance weight (RW) ormatching score relevance weight (MS-RW) (Roberston andJones, 1976)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 41 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Avoiding zeros
If any of the counts is a zero, then the term weight is notwell-defined.
Maximum likelihood estimates do not work for rare events.
To avoid zeros: add 0.5 to each count (expected likelihoodestimation = ELE)
For example, use R − r + 0.5 in formula for R − r . Thus,fomula becomes:
RSVd = Σ logr + 0.5/(R − r + 0.5)
(n − r + 0.5)/((N − n + 0.5)− (R − r + 0.5))(7)
This was previously known as relevance weight (RW) ormatching score relevance weight (MS-RW) (Roberston andJones, 1976)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 41 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Avoiding zeros
If any of the counts is a zero, then the term weight is notwell-defined.
Maximum likelihood estimates do not work for rare events.
To avoid zeros: add 0.5 to each count (expected likelihoodestimation = ELE)
For example, use R − r + 0.5 in formula for R − r . Thus,fomula becomes:
RSVd = Σ logr + 0.5/(R − r + 0.5)
(n − r + 0.5)/((N − n + 0.5)− (R − r + 0.5))(7)
This was previously known as relevance weight (RW) ormatching score relevance weight (MS-RW) (Roberston andJones, 1976)
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 41 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Simplifying assumption
Assuming that relevant documents are a very smallpercentage of the collection, approximate statistics fornonrelevant documents by statistics from the whole collection
Hence, (pt) (the probability of term occurrence in nonrelevantdocuments for a query) is n/N and
log[(1− (pt))/(pt)] = log[(N − n)/n] ≈ logN/n
The above approximation cannot easily be extended torelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 42 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Simplifying assumption
Assuming that relevant documents are a very smallpercentage of the collection, approximate statistics fornonrelevant documents by statistics from the whole collection
Hence, (pt) (the probability of term occurrence in nonrelevantdocuments for a query) is n/N and
log[(1− (pt))/(pt)] = log[(N − n)/n] ≈ logN/n
The above approximation cannot easily be extended torelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 42 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Simplifying assumption
Assuming that relevant documents are a very smallpercentage of the collection, approximate statistics fornonrelevant documents by statistics from the whole collection
Hence, (pt) (the probability of term occurrence in nonrelevantdocuments for a query) is n/N and
log[(1− (pt))/(pt)] = log[(N − n)/n] ≈ logN/n
The above approximation cannot easily be extended torelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 42 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Simplifying assumption
Assuming that relevant documents are a very smallpercentage of the collection, approximate statistics fornonrelevant documents by statistics from the whole collection
Hence, (pt) (the probability of term occurrence in nonrelevantdocuments for a query) is n/N and
log[(1− (pt))/(pt)] = log[(N − n)/n] ≈ logN/n
The above approximation cannot easily be extended torelevant documents
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 42 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in relevance feedback
Statistics of relevant documents (pt) in relevance feedbackcan be estimated using maximum likelihood estimation or ELE(add 0.5).
Use the frequency of term occurrence in known relevantdocuments.
This is the basis of probabilistic approaches to relevancefeedback weighting in a feedback loop
What we just saw was a probabilistic relevance feedbackexercise since we were assuming the availability of relevancejudgments.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 43 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in relevance feedback
Statistics of relevant documents (pt) in relevance feedbackcan be estimated using maximum likelihood estimation or ELE(add 0.5).
Use the frequency of term occurrence in known relevantdocuments.
This is the basis of probabilistic approaches to relevancefeedback weighting in a feedback loop
What we just saw was a probabilistic relevance feedbackexercise since we were assuming the availability of relevancejudgments.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 43 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in relevance feedback
Statistics of relevant documents (pt) in relevance feedbackcan be estimated using maximum likelihood estimation or ELE(add 0.5).
Use the frequency of term occurrence in known relevantdocuments.
This is the basis of probabilistic approaches to relevancefeedback weighting in a feedback loop
What we just saw was a probabilistic relevance feedbackexercise since we were assuming the availability of relevancejudgments.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 43 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in relevance feedback
Statistics of relevant documents (pt) in relevance feedbackcan be estimated using maximum likelihood estimation or ELE(add 0.5).
Use the frequency of term occurrence in known relevantdocuments.
This is the basis of probabilistic approaches to relevancefeedback weighting in a feedback loop
What we just saw was a probabilistic relevance feedbackexercise since we were assuming the availability of relevancejudgments.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 43 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in relevance feedback
Statistics of relevant documents (pt) in relevance feedbackcan be estimated using maximum likelihood estimation or ELE(add 0.5).
Use the frequency of term occurrence in known relevantdocuments.
This is the basis of probabilistic approaches to relevancefeedback weighting in a feedback loop
What we just saw was a probabilistic relevance feedbackexercise since we were assuming the availability of relevancejudgments.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 43 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in adhoc retrieval
Ad-hoc retrieval: no user-supplied relevance judgmentsavailable
In this case: assume that pt is constant over all terms xt inthe query and that pt = 0.5
Each term is equally likely to occur in a relevant document,and so the pt and (1− pt) factors cancel out in the expressionfor RSV
Weak estimate, but doesn’t disagree violently withexpectation that query terms appear in many but not allrelevant documents
Combining this method with the earlier approximation for pt ,the document ranking is determined simply by which queryterms occur in documents scaled by their idf weighting
For short documents (titles or abstracts) in one-pass retrievalsituations, this estimate can be quite satisfactory
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 44 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in adhoc retrieval
Ad-hoc retrieval: no user-supplied relevance judgmentsavailable
In this case: assume that pt is constant over all terms xt inthe query and that pt = 0.5
Each term is equally likely to occur in a relevant document,and so the pt and (1− pt) factors cancel out in the expressionfor RSV
Weak estimate, but doesn’t disagree violently withexpectation that query terms appear in many but not allrelevant documents
Combining this method with the earlier approximation for pt ,the document ranking is determined simply by which queryterms occur in documents scaled by their idf weighting
For short documents (titles or abstracts) in one-pass retrievalsituations, this estimate can be quite satisfactory
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 44 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in adhoc retrieval
Ad-hoc retrieval: no user-supplied relevance judgmentsavailable
In this case: assume that pt is constant over all terms xt inthe query and that pt = 0.5
Each term is equally likely to occur in a relevant document,and so the pt and (1− pt) factors cancel out in the expressionfor RSV
Weak estimate, but doesn’t disagree violently withexpectation that query terms appear in many but not allrelevant documents
Combining this method with the earlier approximation for pt ,the document ranking is determined simply by which queryterms occur in documents scaled by their idf weighting
For short documents (titles or abstracts) in one-pass retrievalsituations, this estimate can be quite satisfactory
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 44 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in adhoc retrieval
Ad-hoc retrieval: no user-supplied relevance judgmentsavailable
In this case: assume that pt is constant over all terms xt inthe query and that pt = 0.5
Each term is equally likely to occur in a relevant document,and so the pt and (1− pt) factors cancel out in the expressionfor RSV
Weak estimate, but doesn’t disagree violently withexpectation that query terms appear in many but not allrelevant documents
Combining this method with the earlier approximation for pt ,the document ranking is determined simply by which queryterms occur in documents scaled by their idf weighting
For short documents (titles or abstracts) in one-pass retrievalsituations, this estimate can be quite satisfactory
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 44 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in adhoc retrieval
Ad-hoc retrieval: no user-supplied relevance judgmentsavailable
In this case: assume that pt is constant over all terms xt inthe query and that pt = 0.5
Each term is equally likely to occur in a relevant document,and so the pt and (1− pt) factors cancel out in the expressionfor RSV
Weak estimate, but doesn’t disagree violently withexpectation that query terms appear in many but not allrelevant documents
Combining this method with the earlier approximation for pt ,the document ranking is determined simply by which queryterms occur in documents scaled by their idf weighting
For short documents (titles or abstracts) in one-pass retrievalsituations, this estimate can be quite satisfactory
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 44 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in adhoc retrieval
Ad-hoc retrieval: no user-supplied relevance judgmentsavailable
In this case: assume that pt is constant over all terms xt inthe query and that pt = 0.5
Each term is equally likely to occur in a relevant document,and so the pt and (1− pt) factors cancel out in the expressionfor RSV
Weak estimate, but doesn’t disagree violently withexpectation that query terms appear in many but not allrelevant documents
Combining this method with the earlier approximation for pt ,the document ranking is determined simply by which queryterms occur in documents scaled by their idf weighting
For short documents (titles or abstracts) in one-pass retrievalsituations, this estimate can be quite satisfactory
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 44 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probability estimates in adhoc retrieval
Ad-hoc retrieval: no user-supplied relevance judgmentsavailable
In this case: assume that pt is constant over all terms xt inthe query and that pt = 0.5
Each term is equally likely to occur in a relevant document,and so the pt and (1− pt) factors cancel out in the expressionfor RSV
Weak estimate, but doesn’t disagree violently withexpectation that query terms appear in many but not allrelevant documents
Combining this method with the earlier approximation for pt ,the document ranking is determined simply by which queryterms occur in documents scaled by their idf weighting
For short documents (titles or abstracts) in one-pass retrievalsituations, this estimate can be quite satisfactory
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 44 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Outline
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 45 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
History and summary of assumptions
Among the oldest formal models in IR
Maron & Kuhns, 1960: Since an IR system cannot predict withcertainty which document is relevant, we should deal withprobabilities
Assumptions for getting reasonable approximations of theneeded probabilities (in the BIM):
Boolean representation of documents/queries/relevanceTerm independenceOut-of-query terms do not affect retrievalDocument relevance values are independent
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 46 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
History and summary of assumptions
Among the oldest formal models in IR
Maron & Kuhns, 1960: Since an IR system cannot predict withcertainty which document is relevant, we should deal withprobabilities
Assumptions for getting reasonable approximations of theneeded probabilities (in the BIM):
Boolean representation of documents/queries/relevanceTerm independenceOut-of-query terms do not affect retrievalDocument relevance values are independent
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 46 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
History and summary of assumptions
Among the oldest formal models in IR
Maron & Kuhns, 1960: Since an IR system cannot predict withcertainty which document is relevant, we should deal withprobabilities
Assumptions for getting reasonable approximations of theneeded probabilities (in the BIM):
Boolean representation of documents/queries/relevanceTerm independenceOut-of-query terms do not affect retrievalDocument relevance values are independent
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 46 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
History and summary of assumptions
Among the oldest formal models in IR
Maron & Kuhns, 1960: Since an IR system cannot predict withcertainty which document is relevant, we should deal withprobabilities
Assumptions for getting reasonable approximations of theneeded probabilities (in the BIM):
Boolean representation of documents/queries/relevanceTerm independenceOut-of-query terms do not affect retrievalDocument relevance values are independent
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 46 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
History and summary of assumptions
Among the oldest formal models in IR
Maron & Kuhns, 1960: Since an IR system cannot predict withcertainty which document is relevant, we should deal withprobabilities
Assumptions for getting reasonable approximations of theneeded probabilities (in the BIM):
Boolean representation of documents/queries/relevance
Term independenceOut-of-query terms do not affect retrievalDocument relevance values are independent
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 46 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
History and summary of assumptions
Among the oldest formal models in IR
Maron & Kuhns, 1960: Since an IR system cannot predict withcertainty which document is relevant, we should deal withprobabilities
Assumptions for getting reasonable approximations of theneeded probabilities (in the BIM):
Boolean representation of documents/queries/relevanceTerm independence
Out-of-query terms do not affect retrievalDocument relevance values are independent
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 46 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
History and summary of assumptions
Among the oldest formal models in IR
Maron & Kuhns, 1960: Since an IR system cannot predict withcertainty which document is relevant, we should deal withprobabilities
Assumptions for getting reasonable approximations of theneeded probabilities (in the BIM):
Boolean representation of documents/queries/relevanceTerm independenceOut-of-query terms do not affect retrieval
Document relevance values are independent
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 46 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
History and summary of assumptions
Among the oldest formal models in IR
Maron & Kuhns, 1960: Since an IR system cannot predict withcertainty which document is relevant, we should deal withprobabilities
Assumptions for getting reasonable approximations of theneeded probabilities (in the BIM):
Boolean representation of documents/queries/relevanceTerm independenceOut-of-query terms do not affect retrievalDocument relevance values are independent
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 46 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
How different are vector space and BIM?
They are not that different.
In either case you build an information retrieval scheme in theexact same way.
For probabilistic IR, at the end, you score queries not bycosine similarity and tf-idf in a vector space, but by a slightlydifferent formula motivated by probability theory.
Next: how to add term frequency and length normalization tothe probabilistic model.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 47 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
How different are vector space and BIM?
They are not that different.
In either case you build an information retrieval scheme in theexact same way.
For probabilistic IR, at the end, you score queries not bycosine similarity and tf-idf in a vector space, but by a slightlydifferent formula motivated by probability theory.
Next: how to add term frequency and length normalization tothe probabilistic model.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 47 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
How different are vector space and BIM?
They are not that different.
In either case you build an information retrieval scheme in theexact same way.
For probabilistic IR, at the end, you score queries not bycosine similarity and tf-idf in a vector space, but by a slightlydifferent formula motivated by probability theory.
Next: how to add term frequency and length normalization tothe probabilistic model.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 47 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
How different are vector space and BIM?
They are not that different.
In either case you build an information retrieval scheme in theexact same way.
For probabilistic IR, at the end, you score queries not bycosine similarity and tf-idf in a vector space, but by a slightlydifferent formula motivated by probability theory.
Next: how to add term frequency and length normalization tothe probabilistic model.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 47 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
How different are vector space and BIM?
They are not that different.
In either case you build an information retrieval scheme in theexact same way.
For probabilistic IR, at the end, you score queries not bycosine similarity and tf-idf in a vector space, but by a slightlydifferent formula motivated by probability theory.
Next: how to add term frequency and length normalization tothe probabilistic model.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 47 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Overview
Okapi BM25 is a probabilistic model that incorporates termfrequency (i.e., it’s nonbinary) and length normalization.
BIM was originally designed for short catalog records of fairlyconsistent length, and it works reasonably in these contexts
For modern full-text search collections, a model should payattention to term frequency and document length
BestMatch25 (a.k.a BM25 or Okapi) is sensitive to thesequantities
BM25 is one of the most widely used and robust retrievalmodels
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 48 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Overview
Okapi BM25 is a probabilistic model that incorporates termfrequency (i.e., it’s nonbinary) and length normalization.
BIM was originally designed for short catalog records of fairlyconsistent length, and it works reasonably in these contexts
For modern full-text search collections, a model should payattention to term frequency and document length
BestMatch25 (a.k.a BM25 or Okapi) is sensitive to thesequantities
BM25 is one of the most widely used and robust retrievalmodels
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 48 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Overview
Okapi BM25 is a probabilistic model that incorporates termfrequency (i.e., it’s nonbinary) and length normalization.
BIM was originally designed for short catalog records of fairlyconsistent length, and it works reasonably in these contexts
For modern full-text search collections, a model should payattention to term frequency and document length
BestMatch25 (a.k.a BM25 or Okapi) is sensitive to thesequantities
BM25 is one of the most widely used and robust retrievalmodels
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 48 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Overview
Okapi BM25 is a probabilistic model that incorporates termfrequency (i.e., it’s nonbinary) and length normalization.
BIM was originally designed for short catalog records of fairlyconsistent length, and it works reasonably in these contexts
For modern full-text search collections, a model should payattention to term frequency and document length
BestMatch25 (a.k.a BM25 or Okapi) is sensitive to thesequantities
BM25 is one of the most widely used and robust retrievalmodels
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 48 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Overview
Okapi BM25 is a probabilistic model that incorporates termfrequency (i.e., it’s nonbinary) and length normalization.
BIM was originally designed for short catalog records of fairlyconsistent length, and it works reasonably in these contexts
For modern full-text search collections, a model should payattention to term frequency and document length
BestMatch25 (a.k.a BM25 or Okapi) is sensitive to thesequantities
BM25 is one of the most widely used and robust retrievalmodels
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 48 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Overview
Okapi BM25 is a probabilistic model that incorporates termfrequency (i.e., it’s nonbinary) and length normalization.
BIM was originally designed for short catalog records of fairlyconsistent length, and it works reasonably in these contexts
For modern full-text search collections, a model should payattention to term frequency and document length
BestMatch25 (a.k.a BM25 or Okapi) is sensitive to thesequantities
BM25 is one of the most widely used and robust retrievalmodels
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 48 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Starting point
The simplest score for document d is just idf weighting of thequery terms present in the document:
RSVd =∑t∈q
logN
n
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 49 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Starting point
The simplest score for document d is just idf weighting of thequery terms present in the document:
RSVd =∑t∈q
logN
n
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 49 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Starting point
The simplest score for document d is just idf weighting of thequery terms present in the document:
RSVd =∑t∈q
logN
n
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 49 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25: Starting point
The simplest score for document d is just idf weighting of thequery terms present in the document:
RSVd =∑t∈q
logN
n
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 49 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 Basic Weighting
Improve idf term [log N/n] by factoring in term frequency anddocument length.
RSVd =∑t∈q
log
[N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
tftd : term frequency in document d
Ld (Lave): length of document d (average document length inthe whole collection)
k1: tuning parameter controlling the document termfrequency scaling
b: tuning parameter controlling the scaling by documentlength
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 50 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 Basic Weighting
Improve idf term [log N/n] by factoring in term frequency anddocument length.
RSVd =∑t∈q
log
[N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
tftd : term frequency in document d
Ld (Lave): length of document d (average document length inthe whole collection)
k1: tuning parameter controlling the document termfrequency scaling
b: tuning parameter controlling the scaling by documentlength
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 50 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 Basic Weighting
Improve idf term [log N/n] by factoring in term frequency anddocument length.
RSVd =∑t∈q
log
[N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
tftd : term frequency in document d
Ld (Lave): length of document d (average document length inthe whole collection)
k1: tuning parameter controlling the document termfrequency scaling
b: tuning parameter controlling the scaling by documentlength
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 50 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 Basic Weighting
Improve idf term [log N/n] by factoring in term frequency anddocument length.
RSVd =∑t∈q
log
[N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
tftd : term frequency in document d
Ld (Lave): length of document d (average document length inthe whole collection)
k1: tuning parameter controlling the document termfrequency scaling
b: tuning parameter controlling the scaling by documentlength
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 50 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 Basic Weighting
Improve idf term [log N/n] by factoring in term frequency anddocument length.
RSVd =∑t∈q
log
[N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
tftd : term frequency in document d
Ld (Lave): length of document d (average document length inthe whole collection)
k1: tuning parameter controlling the document termfrequency scaling
b: tuning parameter controlling the scaling by documentlength
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 50 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 Basic Weighting
Improve idf term [log N/n] by factoring in term frequency anddocument length.
RSVd =∑t∈q
log
[N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
tftd : term frequency in document d
Ld (Lave): length of document d (average document length inthe whole collection)
k1: tuning parameter controlling the document termfrequency scaling
b: tuning parameter controlling the scaling by documentlength
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 50 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 weighting for Long queries
For long queries, use similar weighting for query terms
RSVd =∑t∈q
[log
N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
·(k3 + 1)tftqk3 + tftq
tftq: term frequency in the query q
k3: tuning parameter controlling term frequency scaling of thequery
No length normalization of queries (because retrieval is beingdone with respect to a single fixed query)
The above tuning parameters should ideally be set to optimizeperformance on a development test collection. In the absenceof such optimization, experiments have shown reasonablevalues are to set k1 and k3 to a value between 1.2 and 2 andb = 0.75
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 51 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 weighting for Long queries
For long queries, use similar weighting for query terms
RSVd =∑t∈q
[log
N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
·(k3 + 1)tftqk3 + tftq
tftq: term frequency in the query q
k3: tuning parameter controlling term frequency scaling of thequery
No length normalization of queries (because retrieval is beingdone with respect to a single fixed query)
The above tuning parameters should ideally be set to optimizeperformance on a development test collection. In the absenceof such optimization, experiments have shown reasonablevalues are to set k1 and k3 to a value between 1.2 and 2 andb = 0.75
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 51 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 weighting for Long queries
For long queries, use similar weighting for query terms
RSVd =∑t∈q
[log
N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
·(k3 + 1)tftqk3 + tftq
tftq: term frequency in the query q
k3: tuning parameter controlling term frequency scaling of thequery
No length normalization of queries (because retrieval is beingdone with respect to a single fixed query)
The above tuning parameters should ideally be set to optimizeperformance on a development test collection. In the absenceof such optimization, experiments have shown reasonablevalues are to set k1 and k3 to a value between 1.2 and 2 andb = 0.75
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 51 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 weighting for Long queries
For long queries, use similar weighting for query terms
RSVd =∑t∈q
[log
N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
·(k3 + 1)tftqk3 + tftq
tftq: term frequency in the query q
k3: tuning parameter controlling term frequency scaling of thequery
No length normalization of queries (because retrieval is beingdone with respect to a single fixed query)
The above tuning parameters should ideally be set to optimizeperformance on a development test collection. In the absenceof such optimization, experiments have shown reasonablevalues are to set k1 and k3 to a value between 1.2 and 2 andb = 0.75
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 51 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 weighting for Long queries
For long queries, use similar weighting for query terms
RSVd =∑t∈q
[log
N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
·(k3 + 1)tftqk3 + tftq
tftq: term frequency in the query q
k3: tuning parameter controlling term frequency scaling of thequery
No length normalization of queries (because retrieval is beingdone with respect to a single fixed query)
The above tuning parameters should ideally be set to optimizeperformance on a development test collection. In the absenceof such optimization, experiments have shown reasonablevalues are to set k1 and k3 to a value between 1.2 and 2 andb = 0.75
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 51 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi BM25 weighting for Long queries
For long queries, use similar weighting for query terms
RSVd =∑t∈q
[log
N
n
]· (k1 + 1)tftdk1((1− b) + b × (Ld/Lave)) + tftd
·(k3 + 1)tftqk3 + tftq
tftq: term frequency in the query q
k3: tuning parameter controlling term frequency scaling of thequery
No length normalization of queries (because retrieval is beingdone with respect to a single fixed query)
The above tuning parameters should ideally be set to optimizeperformance on a development test collection. In the absenceof such optimization, experiments have shown reasonablevalues are to set k1 and k3 to a value between 1.2 and 2 andb = 0.75
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 51 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Okapi at TREC 7*
All the TREC-7 searches used varieties of Okapi BM25, suchas:∑
t∈q
[log
N
n
]· (k1 + 1)tftd
K + tftd· (k3 + 1)tftq
k3 + tftq+ k2|Q|
Lave − LdLave + Ld
Where K = k1((1− b) + b × (Ld/Lave))
k2 is a parameter depending on the nature of the query andpossibly on the database
k2 was always zero in all searches of TREC-7, simplying theequation to: ∑
t∈qw1 · (k1 + 1)tftd
K + tftd· (k3 + 1)tftq
k3 + tftq
Where w1 =[log N
n
], aka Roberstons-Jones weight
(Roberston et.al.,
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 52 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Outline
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 53 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Performance measures used
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 54 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Outline
1 Inception
2 Probabilistic Approach to IR
3 Data
4 Basic Probability Theory
5 Probability Ranking Principle
6 Extensions to BIM: Okapi
7 Performance measure
8 Comparision of Models
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 55 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic model vs. other models
Boolean model
Probabilistic models support ranking and thus are better thanthe simple Boolean model.
Vector space model
The vector space model is also a formally defined model thatsupports ranking.Why would we want to look for an alternative to the vectorspace model?
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 56 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic model vs. other models
Boolean model
Probabilistic models support ranking and thus are better thanthe simple Boolean model.
Vector space model
The vector space model is also a formally defined model thatsupports ranking.Why would we want to look for an alternative to the vectorspace model?
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 56 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic model vs. other models
Boolean model
Probabilistic models support ranking and thus are better thanthe simple Boolean model.
Vector space model
The vector space model is also a formally defined model thatsupports ranking.Why would we want to look for an alternative to the vectorspace model?
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 56 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic model vs. other models
Boolean model
Probabilistic models support ranking and thus are better thanthe simple Boolean model.
Vector space model
The vector space model is also a formally defined model thatsupports ranking.Why would we want to look for an alternative to the vectorspace model?
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 56 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic model vs. other models
Boolean model
Probabilistic models support ranking and thus are better thanthe simple Boolean model.
Vector space model
The vector space model is also a formally defined model thatsupports ranking.
Why would we want to look for an alternative to the vectorspace model?
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 56 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic model vs. other models
Boolean model
Probabilistic models support ranking and thus are better thanthe simple Boolean model.
Vector space model
The vector space model is also a formally defined model thatsupports ranking.Why would we want to look for an alternative to the vectorspace model?
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 56 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic vs. vector space model
Vector space model: rank documents according to similarityto query.
The notion of similarity does not translate directly into anassessment of “is the document a good document to give tothe user or not?”
The most similar document can be highly relevant orcompletely nonrelevant.
Probability theory is arguably a cleaner formalization of whatwe really want an IR system to do: give relevant documentsto the user.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 57 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic vs. vector space model
Vector space model: rank documents according to similarityto query.
The notion of similarity does not translate directly into anassessment of “is the document a good document to give tothe user or not?”
The most similar document can be highly relevant orcompletely nonrelevant.
Probability theory is arguably a cleaner formalization of whatwe really want an IR system to do: give relevant documentsto the user.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 57 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic vs. vector space model
Vector space model: rank documents according to similarityto query.
The notion of similarity does not translate directly into anassessment of “is the document a good document to give tothe user or not?”
The most similar document can be highly relevant orcompletely nonrelevant.
Probability theory is arguably a cleaner formalization of whatwe really want an IR system to do: give relevant documentsto the user.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 57 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic vs. vector space model
Vector space model: rank documents according to similarityto query.
The notion of similarity does not translate directly into anassessment of “is the document a good document to give tothe user or not?”
The most similar document can be highly relevant orcompletely nonrelevant.
Probability theory is arguably a cleaner formalization of whatwe really want an IR system to do: give relevant documentsto the user.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 57 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Probabilistic vs. vector space model
Vector space model: rank documents according to similarityto query.
The notion of similarity does not translate directly into anassessment of “is the document a good document to give tothe user or not?”
The most similar document can be highly relevant orcompletely nonrelevant.
Probability theory is arguably a cleaner formalization of whatwe really want an IR system to do: give relevant documentsto the user.
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 57 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Which ranking model should I use?
I want something basic and simple → use vector space withtf-idf weighting.
I want to use a state-of-the-art ranking model with excellentperformance → use language models or BM25 with tunedparameters
In between: BM25 or language models with no or just onetuned parameter
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 58 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Which ranking model should I use?
I want something basic and simple → use vector space withtf-idf weighting.
I want to use a state-of-the-art ranking model with excellentperformance → use language models or BM25 with tunedparameters
In between: BM25 or language models with no or just onetuned parameter
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 58 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Which ranking model should I use?
I want something basic and simple → use vector space withtf-idf weighting.
I want to use a state-of-the-art ranking model with excellentperformance → use language models or BM25 with tunedparameters
In between: BM25 or language models with no or just onetuned parameter
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 58 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Which ranking model should I use?
I want something basic and simple → use vector space withtf-idf weighting.
I want to use a state-of-the-art ranking model with excellentperformance → use language models or BM25 with tunedparameters
In between: BM25 or language models with no or just onetuned parameter
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 58 / 59
Inception Probabilistic Approach to IR Data Basic Probability Theory Probability Ranking Principle Extensions to BIM: Okapi Performance measure Comparision of Models
Resources
Chapter 11 of IIR
Resources at http://cislmu.org
Presentation on ”Probabilistic Information Retrieval”by Dr.Suman Mitra at:http://www.irsi.res.in/winter-school/slides/
ProbabilisticModel.ppt
PhD Comprehensive presentation Part 1: Probabilistic Information Retrieval 59 / 59