Designing Test Collections for Comparing Many Systems

Designing Test Collections for Comparing Many Systems

Tetsuya SakaiWaseda University, Japan

@tetsuyasakai

November 4, 2014@CIKM 2014, Shanghai

Acknowledgement

This research is a part of Waseda University’s project “Taxonomising and Evaluating Web Search Engine User Behaviours,” supported by Microsoft Research.

THANK YOU!

Takeaways (1)

• Using one‐way ANOVA‐based power analysis, researchers can determine the topic set size n by specifying:

α: Type I error probability β: Type II error probabilityminD: minimum detectable range (performance diff between best and worst systems) for ensuring a statistical power of 1‐βm: number of systems to be compared

: estimated variance of each system• Different measures have different s, so researchers should decide on the evaluation measure at the test collection design phase

Takeaways (2)

• Our method can provide different test collection designs (n, pd) that satisfy the same statistical requirement.

n: topic set size pd: pool depth• The assessment cost of a pd=100 test collection can be reduced to 18% or less while keeping it statistically equally reliable.

• Our method can be used to compare evaluation measures in terms of practical significance = judgment cost.

• Our tools and data are available athttp://www.f.waseda.jp/tetsuya/tools.htmlhttp://www.f.waseda.jp/tetsuya/data.html

TALK OUTLINE

1. How Information Retrieval (IR) test collections are constructed2. Statistical reform3. How test collections SHOULD be constructed4. Experimental results5. Conclusions and future work

Test collections = standard data sets for evaluation

Test collection A Test collection BEvaluationmeasurevalues

Evaluationmeasurevalues

An Information Retrieval (IR) test collection

Topic Relevance assessments(relevant/nonrelevant documents)

Document collection



: :

Topic set “Qrels = query relevance sets”

CIKM 2014 home page cikm2014.fudan.edu.cn/: highly relevant

cikmconference.org/: partially relevantwww.cikm2013.org: nonrelevant

How IR people build test collections (1)

Okay, let’s build a test collection…

Organiser


…with maybe n=50topics (search requests)…

Well n>25 sounds good for statistical significance testing, but why 50? Why not 100? Why not 30?

TopicTopicTopicTopicTopic 1



50 topicsOkay folks, give me your runs (search results)!

run run run

Participants



50 topicsPool depth pd=100 looks

affordable…

run run run

Top pd=100 documentsfrom each run

Pool for

Topic 1Document collection too large to doexhaustive relevance assessments sojudge pooled documents only



50 topics


Pool for

Topic 1Relevance assessments

Highly relevant

Partially relevant

Nonrelevant

An Information Retrieval (IR) test collection


Document collection



: :

Topic set “Qrels = query relevance sets”

CIKM 2014 home page cikm2014.fudan.edu.cn/: highly relevant

cikmconference.org/: partially relevantwww.cikm2013.org: nonrelevant

n=50topics…why?

Pool depth pd=100(not exhaustive)

TALK OUTLINE


NHST = null hypothesis significance testing (1)

EXAMPLE: paired t‐test for comparing systems X and Y with n topics

Assumptions:

Null hypothesis:

Test statistic:

Population means are the same


EXAMPLE: paired t‐test for comparing systems X and Y with n topics

Null hypothesis:

Test statistic:

Under H0, t0 obeys a t distribution with n‐1 degrees of freedom.


EXAMPLE: paired t‐test for comparing systems X and Y with n topicsNull hypothesis:Under H0, t0 obeys a t distribution with n‐1 degrees of freedom.

Given a significance criterion α(=0.05), reject H0 if |t0| >= t(n‐1; α).

0

0.1

0.2

0.3

0.4

‐t(n‐1; α)

n=50

t(n‐1; α)

“H0 is probably not true because the chance of observing t0 under H0

is very small”


EXAMPLE: paired t‐test for comparing systems X and Y with n topicsNull hypothesis:Given a significance criterion α(=0.05), reject H0 if |t0| >= t(n‐1; α).

0

0.1

0.2

0.3

0.4

‐t(n‐1; α)

n=50

t(n‐1; α)

0

0.1

0.2

0.3

0.4

‐t(n‐1; α)

n=50

t(n‐1; α)

Conclusion:X ≠ Y!

t0 t0Conclusion:

H0 not rejected, so don’t know

NHST is not good enough [Cumming12]

• Dichotomous thinking ( “different or not different?” )A more important question is “what is the magnitude of the difference?” Another is “How accurate is my estimate?”• p‐values a little more informative than “significant at α=0.05” but…

0

0.1

0.2

0.3

0.4

‐t(n‐1; α)

n=50

t(n‐1; α)

t0

Probability of observing t0 or something more extreme under H0

The p‐value is not good enough either [Ellis10,Nagata03]

Reject H0 if |t0| >= t(n‐1; α) where

But a large |t0| could mean two things:(1) Sample effect size (ES) is large;(2) Topic set size n is large.

If you increase the sample size n, you can always achieve statistical significance!

Difference between X and Y measured in standard deviation

units

Statistical reform – effect sizes [Cumming12,Okubo12]

• ES: “how much difference is there?”• ES for paired t test measures difference in standard deviation unitsPopulation ES =

Sample ES as an estimate of the above =

In several research disciplines such as psychology and medicine, it is required to report ESs! In this study, we determine the topic set size n by ensuring high power 1‐β whenever ES is large.

Statistical reform – confidence intervals

• CIs are much more informative than NHST(point estimate + uncertainty/accuracy)• Estimation thinking, not dichotomous thinking[Cumming12]

In several research disciplines such as psychology and medicine, it is required to report CIs! See [Sakai14FIT] (Designing Test Collections that Provide Tight Confidence Intervals)

[Sakai14SIGIRforum]

TALK OUTLINE


Overview of our approaches

• Using the paired t‐test to determine n (for m=2 systems)INPUT: α, β, minDt (minimum detectable difference for ensuring power=1‐β), (estimated variance of performance difference)• Using one‐way ANOVA to determine n (for m>=2 systems)INPUT: α, β, m, minD (minimum detectable range for ensuring power=1‐β), (estimated variance of system performance)• Methods for estimating and INPUT: test collections with runs and an evaluation measure(topic‐by‐run matrices)

See Appendix

The ANOVA approach (1)

Assume

i=1,…,m (systems)j=1,…,n (topics)

Homoscedasticity(equal variance)

Let

where

Hypotheses

: ai ≠ 0 for some i

System effect

No system effect

The ANOVA approach (2)i=1,…,m (systems)j=1,…,n (topics)

Total variation

can be decomposed intoST = SA + SE where

Between‐systemvariation

Within‐systemvariation

Sample grand mean

Sample system mean


Test statistic F0 = VA/VE where VA = SA/φA, VE = SE/φE,φA = m‐1, φE = m(n‐1)

Under H0,F0～ F distribution with (φA, φE) degrees of freedom.One‐way ANOVA rejects H0if F0 >= F(φA; φE; α).

F(φA; φE; α)F0

α

1‐α

How large is the between‐system variance compared to the within‐system variance?


The probability of rejecting H0

Under H0, this is exactly α (rejecting H0 that is true)

F(φA; φE; α)F0

α

1‐α



Under H1, this is exactly the power 1‐β (rejecting H0 that is false)

Under H1, F0～ noncentral F distribution with (φA, φE) degrees of freedomand a noncentrality parameter λ = nΔ, where

Measures total system effects in variance units


The power (probability of rejecting H0 that is false)

1‐βF0～ noncentralF distribution

For a random variable F’ that obeys a noncenral F distribution,Pr{ F’ <= w } can be approximated using a normal distribution[Nagata03] (Eqs. 14 and 15 in my paper).

Given α, n, Δ, m, the power 1‐β can be computed.But what we want is: Given α, β, Δ, m, compute n!

Under H0, we know Δ=0. But under H1, we only know that Δ ≠ 0.Δ needs to be specified to guarantee power=1‐β.Let’s guarantee power=1‐β (i.e. correctly reject H0 with 100(1‐β)% confidence)whenever Δ >= minΔ (minimum detectable delta).

How shall we set minΔ?


Let

where minD is the minimum detectable range that you specify.Whenever the difference between the best system and the worst system is minD or more, we guarantee power=1‐β.

Given α, n, Δ, m, the power 1‐βcan be computed.Using α, n, minD, , m, the worst‐case power can be computed (See Eq.17 in my paper).


Estimate variance from past data

μ

best

worst

μi

minDai = μi ‐ μ


Given (α, β, minD, , m),

Here, λ can be obtained if we use the following approximation [Nagata03]:

λ = nΔ: Noncentrality parameter

1‐β ≒

Let φE = m(n‐1) ≒∞ ～ noncentral chi‐square distribution with φA degrees of

freedom and the same noncentrality parameter λ


Given (α, β, minD, , m), λ = nΔ: Noncentrality parameter

φA = m‐1

For noncentral chi‐square distributions, use the following λ values [Nagata03]:


Given (α, β, minD, , m),

Obtain n using λ from the table, and check if (α, n, minΔ, m) satisfies the power requirement:

λ = nΔ: Noncentrality parameter

1‐β

Normal approximation available (Eqs. 15, 16 in my

paper)

If not, n++ and try the above again.

Demo: determine n from (α, β, minDt, , m)

http://www.f.waseda.jp/tetsuya/CIKM2014/samplesizeANOVA.xlsx

Obtaining

A time‐honoured method using one‐way ANOVA statistics [Okubo12]:

Estimate of the population between‐system variance

Estimate of the population within‐system variance

Estimate of the populationvariance

Given multiple ANOVA statistics (test collections + runs),the estimated variances can be pooled to enhance reliability:

Overview of our approaches

• Using the paired t‐test to determine n (for m=2 systems)INPUT: α, β, minDt (minimum detectable difference for ensuring power=1‐β), (estimated variance of performance difference)• Using one‐way ANOVA to determine n (for m>=2 systems)INPUT: α, β, m, minD (minimum detectable range for ensuring power=1‐β), (estimated variance of system performance)• Methods for estimating and INPUT: test collections with runs and an evaluation measure(topic‐by‐run matrices)

See Appendix

TALK OUTLINE


Data for estimating

Data #topics runs pd #docsTREC03new 50 78 125 528,155 news articlesTREC04new 49 78 100 dittoTREC11w 50 37 25 One billion web pagesTREC12w 50 28 20/30 dittoTREC11wD 50 25 25 dittoTREC12wD 50 20 20/30 ditto

Adhoc news IR

Adhoc web IR

Diversified web IR

We have a topic‐by‐run matrix for each data set and evaluation measure

Evaluation measures

News(l=10,1000)Web (l=10)

Web (l=10)

l: measurement depth

t‐test vs ANOVAWhen m=2,• minD for ANOVA (range) reducesto minDt for t‐test (difference).• Results are similar,with ANOVA giving slightly larger estimates of n.Henceforth we discussANOVA as it can also considerm>3 and we prefer to“err on the side of oversampling”[Ellis10] .

0

1000

2000

3000

4000

5000

6000

0 20 40 60 80 100 120 140 160 180 200AP Q nDCG nERR

(a2) adhoc/news (l=10)(α, β, minD)=(0.05, 0.20, 0.05)

m

n

For comparing m=100 systems,Q/nDCG/AP/nERR require

2198/2382/2863/4063 topics

0

500

1000

1500

2000

2500

3000

3500

4000

0 20 40 60 80 100 120 140 160 180 200AP Q nDCG nERR

(b) adhoc/web (l=10)(α, β, minD)=(0.05, 0.20, 0.05)

m

n

For comparing m=100 systems,Q/nDCG/AP/nERR require

1240/1291/2801/2921 topics

0

500

1000

1500

2000

2500

3000

3500

4000

0 20 40 60 80 100 120 140 160 180 200α‐nDCG nERR‐IA D‐nDCG D#‐nDCG

(c) diversity/web (l=10)(α, β, minD)=(0.05, 0.20, 0.05)

m

n

For comparing m=100 systems,D/D#/α/nERR‐IA require

1201/1749/2662/2869 topics

What if we reduce the pool depth pd?


n=50 topics


Pool for

Topic 1Relevance assessments

Highly relevant

Partially relevant

Nonrelevant

For adhoc/news l=1000 (pd=100) only

when pd is reduced

As pd gets smaller,• Average #judged/topic decreases (naturally)• Variance increases (fewer data points hurts stability)Re‐estimate n for (α, β, minD, new )

0

200

400

600

800

1000

1200

1400

1600

0 100 200 300 400 500 600 700 800AP Q nDCG nERR

pd=50

n

(α, β, minD)=(0.05, 0.20, 0.05)m=10

pd=100pd=70

pd=30pd=10

#Average judged/topic

Total cost for AP:96 docs/topic *

879 topics = 84,384 docs

Total cost for AP:731 docs/topic *

652 topics = 476,612 docs

TREC ad hocpool depth

Alternative design with costreduced to 18%

TALK OUTLINE


Takeaways (1)

• Using one‐way ANOVA‐based power analysis, researchers can determine the topic set size n by specifying:

α: Type I error probability β: Type II error probabilityminD: minimum detectable range (performance diff between best and worst systems) for ensuring a statistical power of 1‐βm: number of systems to be compared

: estimated variance of each system• Different measures have different s, so researchers should decide on the evaluation measure at the test collection design phase

Takeaways (2)

• Our method can provide different test collection designs (n, pd) that satisfy the same statistical requirement.

n: topic set size pd: pool depth• The assessment cost of a pd=100 test collection can be reduced to 18% or less while keeping it statistically equally reliable.

• Our method can be used to compare evaluation measures in terms of practical significance = judgment cost.

• Our tools and data are available athttp://www.f.waseda.jp/tetsuya/tools.htmlhttp://www.f.waseda.jp/tetsuya/data.html

Future work

• Investigating the relationship between our power‐based approach and a CI (confidence interval)‐based approach: DONE [Sakai14EVIA]

• Estimating n for various tasks (not just IR) – our methods are applicable to any paired‐data evaluation tasks

• Given a set of statistically equally reliable designs (n,pd), choose the best one based on reusability and assessment cost

Can we evaluate new systems fairly?

References[Cumming12] Cumming, G.: Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta‐Analysis. Routledge, 2012.[Ellis10] Ellis, P.D.: The Essential Guide to Effect Sizes, Cambridge, 2010.[Nagata03] Nagata, Y.: How to Design the Sample Size. Asakura Shoten, 2003.[Okubo12] Okubo, M. and Okada, K. Psychological Statistics to Tell Your Story: Effect Size, CondenceInterval (in Japanese). Keiso Shobo, 2012.[Sakai14PROMISE] Sakai, T.: Metrics, Statistics, Tests. PROMISE Winter School 2013: Bridging between Information Retrieval and Databases (LNCS 8173), pp.116‐163, Springer, 2014. [Sakai14SIGIRforum] Sakai, T.: Statistical Reform in Information Retrieval?, SIGIR Forum, 48(1), 2014. [Sakai14FIT] Sakai, T.: Designing Test Collections that Provide Tight Confidence Intervals, FIT 2014, RD‐003, 2014.[Webber08] Webber, W., Moffat, A. and Zobel, J.: Statistical power in Retrieval Experimentation. ACM CIKM 2008, pp.571–580, 2008.

Appendices

• The t‐test approach• Delta over [Webber08]

The t‐test approach (1)

vs.

Hypotheses

Assume

⇒

where

Systems X and Y are equally effective


Test statistic

Under H0, t0～ t distribution with φ=n‐1 degrees of freedom.The paired t test rejects H0 if |t0| >= t(φ; α).

α/2 α/2

t(φ; α)‐t(φ; α)

1‐αt0

Two‐sided critical t value



α/2 α/2

t(φ; α)‐t(φ; α)

1‐αt0

Under H0, this is exactly α (rejecting H0 that is true)

The t‐test approach (4)The probability of rejecting H0

Under H1, this is exactly the power 1‐β (rejecting H0 that is false)

Under H1, t0～ noncentral t distribution with φ=n‐1 degrees of freedom anda noncentrality parameter λt =where

Effect size

The t‐test approach (5)The power (probability of rejecting H0 that is false)

1‐β =

t0～ noncentralt distribution

For a random variable t’ that obeys a noncenral t distribution,Pr{ t’ <= w } can be approximated using a normal distribution[Nagata03] (Eqs. 4 and 5 in my paper).

Given α, n, effect size Δt, the power 1‐β can be computed.But what we want is: Given α, β, Δt, compute n!

The t‐test approach (6)Given α, n, effect size Δt, the power 1‐β can be computed.But what we want is: Given α, β, Δt, compute n!

Under H0, we know Δt = 0. But under H1, we need to specify Δt to discuss power.So let’s correctly reject H0 with 100(1‐β)% confidencewhenever |Δt| >= minΔt (minimum detectable effect).

Don’t miss a real difference if it’s minΔt or larger!


Given (α, β, minΔt), the required n can be approximated [Nagata03]:

zp: one‐sided critical z value

1‐β <=

Check if the above n actually satisfies the power requirement:

t0～ noncentralt distribution

If not , n++ and try the above again.


Effect size

In practice, instead of setting minΔt (in terms of effect size),set minDt (minimum detectable difference):

|μX – μY | > = minDtthen convert it to

Need a variance estimatefrom past data!

Demo: determine n from (α, β, minDt, )

http://www.f.waseda.jp/tetsuya/CIKM2014/samplesizeTTEST.xlsx

Delta over [Webber08]

• They addressed the problem of building a test collection incrementally (add a topic, judge, re‐estimate variance…).

• They considered the t‐test only.

• They used heuristics to estimate the variance.

• They considered AP and adhoc IR only.

We ask the direct question: “How many topics do we need to create?”

We use both the t‐test and ANOVA to handle m(>=2) systems

We use estimates from ANOVA

We consider a variety of graded‐relevance measures and three different IR tasks.

Designing Test Collections for Comparing Many Systems

Technology

Transcript of Designing Test Collections for Comparing Many Systems