Comparing Benchmark Task and Insight Evaluation Methods

Purvi Saraiya, Chris North, Karen DucaVirginia Tech

BELIV 2010

1 2 3 4 5 6 (Timepoints)

Non-Smokers

Smokers

1 2 3 4 5 6 (Timepoints)

Single-Time Viz

Multi-Time Viz

Multi-Graph Viz

Comparison of Results

Fast / accurate

Slow / inaccurate Not tested

More insight

Confirm

Timepoint analysis on Single-Time Viz

Refute

???

Extend

Hypothesis tasks on Single-Time Viz

Outlier node tasks on Multi-Time Viz

Less insight

Refute

Topology tasks on Multi-Graph Viz

Confirm

Timepoint analysis on Multi-Time Viz

Benchmark task method result

Insight method result

Other interesting stuffImportance of interaction to insight

Subjectivity Threat

Task method = choice of tasks ecological validity

Insight method = coding of insights repeatability

Avg Time

8.5 9.2 8.8

20.5

16.9

10.8

0

5

10

15

20

25

1 Tpt.* M Tpt.* M Graphs

Tim

e (

min

)

Task Insight

Question

Which is more important to know:

Tasks a visualization supports

or

Tasks a visualization promotes

?

Studies using VAST Challenge

Controlled Ethnography !?

VAST Challenges – overcoming the problems with ethnography (esp. in intell)

(contextual) interviews

observational lab study

Gain rich insight into analysts’ processes,Quantify findings with repeatability, Introduce interventions

generate scenario

confirm findings