Post on 25-May-2015
A graphical grammar + graphical inference =
a grammar of graphical inference?
Hadley Wickham, Rice University
1. Motivation
2. Resampling methods (graphical inference)
3. Graphical display (grammar of graphics)
4. Future work
−6
−4
−2
0
2
4
6
8
5
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
−6 −4 −2 0 2 4 6
species● Concinna● Heikert.● Heptapot.
−6−4−2
02468
−6−4−2
02468
−6−4−2
02468
1
● ●●
●
● ●●●●●●●●
●●
●● ●
●●●● ●
●●
●
●
●●
●●
●
●●●●
●
●
●●
●
●●
●
●●
●
●●●
●
●
●
●
●●
●●●●●●
●●●●●●●●●●
●
●
5
●●●
● ●
●●
● ●●
●
●
●
●
●
●●
●
●● ●
●
●●●
●●
●
●
●●●
●
●
●
●●
●●
●
●
●●
●●
●
●●● ●
●
●
●
●●
●●●●
●
●●
●
●●
● ●
●
●
●
●●
●
●
9
● ●●
●● ●●●
●●
●
●●●
●
●
●
●●
●
●
●●●●
●
● ●● ●
●●
●
●
●
●
●● ●
●
●●
●
●● ●●
●●
●
●
●
●●
●●●●
●
●●● ●●●●●
● ●●
●
●●●
−6−4−2 0 2 4 6
2
●
●
●●
●
●●●●●
●
●
●
●●
●
●
●
●●
●●●
●
●
●
●
●
●●
●●●●●●●●●●
●●
●●●
●
●
●
●
●●●●
●●●●
●
●
●●
●
●● ●●
●●
●●
●● ●●
6
●●●●
●●●●
● ●●
●
●●
●●
●●●
●
●
●●
●
●
●● ●●●●
●●
●●
●● ●●
●●●
●
●●
●
●●
●
●
●●●●
●
●●
●
●
●
●●
●● ●●
●
●●
●●
●●
●
10
●●●
●●
●
●●●
●
● ●
●
●●●●
●●●
●
●●
●
●
●
●
●
●
●●●●●
●
●●●● ●●●
●●
●
●
●● ●●●
●●
●
●
●●●
● ●
●
●
●●
●●
●● ●
●
● ●●
●
−6−4−2 0 2 4 6
3
●
●●
● ●●●●
●●●
●●
●
●●
●
●
●●
●
●●●●●
●●●
●
●●
●●●●
●●●
● ●●●●
●● ●●●
●● ●●
●●●
●●●●●●
●●●
●
●
●●
●●
●●
●
7
●●
●
●
●●● ●● ●
●
●
●●●●●●
●●●
●●●
●● ●●●●●
●
●●●
●●
●
●●
●
●
●● ●●
●●●●●●
●●
●●●
●
●
●
●●●
●●●
●●
●●
●●
●●
11
●●
●
●●
●
●●●●
●● ●
●
●●
●
●
●
●●
●
●●●●●●
●●
●
●
●●
●●●
●
●●●●●
●●●
●
●
●
●●
●
●
●
●● ●●
●●
●
● ●
●●
●
●
●
●
●●
●●
●
−6−4−2 0 2 4 6
4
● ●●●
●●●
●●●
●●
●
●
●●
●●
●
●●●●
●●●●
●
●●
●
●
●●
●●
●●●
●●●●
●●
●●
●
●●●
●
●●
●
●●●
●
●
●●●
●
●●●●
●●
● ●
● ●
8
●● ●●
●●
● ●●●
● ●●
● ●●●
●●●●
●
●
●●●●●
●●
●●
●●●
●
●
●●●
●●
●●
●
●
● ●●●●
●●●
●
●●
● ●
●
●
●
●
●
●
●●
●●●
● ●
●
●
12
●●
●●
●
●
●
●
●●
●
●● ●●●
●●
●●●
●
●●
●
●●●
●●
●●●
● ●●●●●
●●
● ●
●●
●
●●
● ●●
●●
●
●
●●
●
●●
●
●
●●●●
●
●
●
●●
●
●●
−6−4−2 0 2 4 6
species● Concinna● Heikert.● Heptapot.
−4
−2
0
2
4
−4
−2
0
2
4
−4
−2
0
2
4
1
●
●
●● ●
●
●
●●
●
●
●
●
●
●●●
●●
●
● ●
●●●
●
●●
●
●
●
●
●
●
●
●●
●● ●●
●●
●●
●
●
●
●●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
● ●●
●
●
●
●
5
●
●
●
●
●
●●●●
●●
●
●●
●●●
●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
● ●
● ●●
●
● ●
●●
9
●●
●●
●
●
●
●●●●
●●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●● ●
●
● ●
●
●● ●
●
●
●●
●●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
● ●
●
●
−4 −2 0 2 4
2
●●
●
●●●
●
●
● ●
●
●
●
●●
●●
●●●
●
● ●●
●●●
●
●
●
●●
● ●●●
●
● ●●
●●●
●●●
●
● ●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
6
●●
●
●
●●
●
●
● ●
●●
●
●
●●
●●
●
● ● ●
●
●
●●
●
● ●
●
●
●
●
●
●
●●●
●●
●●
●
●
●
● ●●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●●●
●
●
●
●
●●
10
● ●
●●
●●
●
●
●
●
●
●
●
●
●●●
●●● ●
●
●
●
●
●
●
●
●●●
●●
●
● ●●●●
●●
●
●
●
●●
●
●● ●
●
●
●● ●
●
●●
●●●
●
●
●
●
●
●●
●
●
●●●
●
−4 −2 0 2 4
3
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●●●●
●
●●
●
● ●●
●
●●
●
●●
●
●● ●
●●●●
●
●
●
●
●
●●
●
●
●●●
●
7
●●
●
●
●●●
●
●
●
●
●
●●
●
●●●●
● ●
● ●●
●
●●●
●
●●
●●
●
●
●
●●● ●
●
●●
●
●
●
●
●
●●●
● ●
●
●
●●
●
●
●
●●
●●
●●
●●
●●
● ●●
●
11
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●●
●●●
●
●●●
●● ●
●
●
●
●●
● ●●
●
● ● ●●
●
●●
●●
●
●
●●
●
●●●●
●
●
● ●●
−4 −2 0 2 4
4
●
●
●●
●●
●
●●
●
●●●
●
●●
●
●
●●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●●
●●
●● ● ● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
8
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
● ●●●
●
●
●●
●
●●
●
●●
●
●●
●
●
●●●
●●
●
●
●●
●
●●
●
●
●
● ●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●●
●
12
●
●
●
●
●
●●
● ●
●●
●●
●
●● ●
●
●
●
●●●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
●●
●●
●●● ●
●●●
●● ●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
−4 −2 0 2 4
species● Concinna● Heikert.● Heptapot.
ProblemWant to make it very easy to make these types of display - should be able to create as easy as doing as simple numerical test.
With the right tools, it turns out to be really easy!
Two basic problems: how to do resampling under the null, and how to define the graphic.
Resampling
Many resampling methods have particularly elegant expression in R code. For example, if null hypothesis is that of independence:
df <- transform(df, resp = sample(resp))
However, better to generalise the pattern.
Generalisationpermute_var <- function(var) { function(df) { df[[var]] <- sample(df[[var]]) df }}
f <- permute_var("mpg")f(mtcars)permute_var("mpg")(mtcars)
Advantages
Separate description from implementation.
Call describes action.
Can later replace implementation if better approach discovered.
Unimportant details concealed.
n resamples
Each method gives us a single resample, but we need n.
Trivial to repeat n times with rdply() from the plyr package.
(rdply() is a generalisation of replicate() that returns a data frame with a column that labels each replicate.)
Overallresamp <- function(true, method, n = 19, pos = sample(n + 1, 1)) { samples <- rdply(n, method(true)) if (missing(pos)) { message("True data in position ", pos) } add_true(samples, true, pos)}
add_true <- function(samples, true, n) { samples$.n <- with(samples, ifelse(.n >= n, .n + 1, .n)) true$.n <- n all <- rbind(samples, true) all[order(all$.n), ]}
Will see application shortly
Other nulls
Need functions for other common null hypotheses.
Experimental null_model(), which computes rotation residuals given a specified linear model. (Demo a little later)
Display
How is the plot of the simulated data with true data different from a single plot? Just need to repeat the same display n times and label appropriately.
This is easy if we have a description of the plot, independent of the data.
Grammar of GraphicsThis is one principle of the grammar of graphics: should describe the graphic we want, not how to create it.
Implemented with the ggplot2 package in R.
Easy to modify a graphic after it has been created. For graphical inference, we need to change the data and facetting.
Example
displ
cty
10
15
20
25
30
35
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●●
●●●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●●
●●
●●
●
●
● ● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●●
●
●
●●
●●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
● ●
●● ●●
●
●
●
● ●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●●
●
●●
●●
●●
●
●
●
●
●
● ● ●
● ●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●●
●
●
●
2 3 4 5 6 7
factor(year)● 1999● 2008
displ
1/ct
y * 1
00
4
6
8
10
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
● ●●● ●● ●
●
●
●
●
●●
●●
●
●●
●●
●
●●
●
●
●●●●
●●
●
●
● ● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●●
●
●
●●
●●
●●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●●
●
● ●
●
● ●●
●
●
●
● ●
●
●
●●
●
●
●
●
● ●● ●
●
●
●●
●
●●●
● ●●
●
●●
●
●
● ●
●
●
●
●
●
● ● ●
● ●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●●
●
●
●
2 3 4 5 6 7
factor(year)● 1999● 2008
displ
1/ct
y * 1
00
468
10
468
10
468
10
468
10
1
●●●
●●
●●
●
●●
●●
●●
●
●
● ●●●
●
●●●●●● ●
●
●●
●
●
●
●●●
●
●
●● ●●
●●
●
●●
●
●●●
●●● ●●●●●
●●
●
●
● ●●
●●
●●
●
●● ●
● ●
●●●●●●●●●
● ●●
●●● ●
●●
●●●●
●
●●●
●●
●●● ●
●
●●●●
●● ●
●●●
●
●●
●●●●●
●●
●●
●●
● ●● ●●●
●●●
●
● ●
●
●●
●
●
●●●● ●●●
●● ●
●●
●●●●
●●●●●●
●●●
●●
●●●
●
●●●
● ●
●●
●
●●
●
●
● ●●●
●●●●●●
●●
●
●●●
●●●
●●
●
●
● ●
●●●
●
●
●● ●
●
●●
●●
●●
6
●●●●
●●●
●
●●
●
●
●
●
●●
●
●●●●●
●
●●●● ●
●
● ●
●●●
●
●
●●●
●●
●●●
●●●●
●●●●
●●●●●●
●● ●●
●●
●●
●●● ●
●●
●
●● ●
●
●
●●●●●●
●●●●●
●●
●
●● ●
●●●
●●
●●●
●
●●● ●●
●●
●●
●
●● ●●●●
●●●●
●
●
●●●
●
●
●
●●
●●●●
●●
●● ●
●●
●
●
●
●
●
●●
● ●● ●●
●● ●●
●
●●
●●
●●●●●●●●
●●●●
●
●
●●
●
●● ●
●●
●
●●
●●
● ●●●
●●●●
●●
●
●●●●
●●
●●
●
●●●
●● ●●
●●
●
●
●●●● ●●●
●●●
11
●●
●●●
●●
●●●●●
●
●
●●
●
●
●●●●●●
●
●●
●●
● ●●
●●●●
●
●
●●●●● ●●●
●
●●●●
●●●●
●●●●●● ●●●
●●●●●
●
●●
●
●● ●
●●
●●
●
●●
●●●●
●●●●●●
● ●●●
●
●●●●●
●●●
●●
●●
●
● ●
●
● ●●●●●
●●
●●●●
●● ●
●
●●
●●●●●●●●●● ●●
●●
●
●
●
●● ●● ●●
●●●● ●
●●●●
●
●●●●
●●●
●
●●
●●●
●●●
●
●● ●● ●
●●
●
●●●
●●●●
●●●
●●●●
●●●
●
●
● ●●●
●●●●
●
●
●●● ●
●●
●
● ●●●●
●● ●
16
●●●●●
●
●
●●
●●
●
●
●●
●
●
●
●●● ●●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●●●
●●●●
●●
●●●●● ●●
●●●
●●● ●
● ●
●
●●●
●●
●
●●●●
●●●●●
●●
●●●●● ●
●●●
●●●
●
●●
● ●●
●●
●
●●● ●●
●
●● ●
●●
●●●
●
●●●
●
●
●
●
●● ●
●
●●
●●
●●
●●● ●
●
●●
●
●●●●●
●
●●
●
● ●●
●● ●●
●
●
●●●●
●
●●●●●●
●●
●●●
●●●●
●●
●
●●
●●
●●
●●
●
● ●
●
●●●●●
●●●●●
●●
●●
●●●●
●
●●
●
●●
●●
●
●
●●
●
●●● ● ●
2 3 4 5 6 7
2
●●●
●●●
●●
●●
●
●
●●
●
●●●●●●●●
●●
●
● ●
●
●
●
●
●● ●●
●●●●
●●
●●
●
●●●●
●●●●
●●
●●
● ● ●●●
● ●●
●●
●
●●
●
●
●●●●●
●●●●●●
●●●
●
●●●
●●●
●
●●●●
●
●
● ●
●●
●●
●
●●
● ●●●●●
● ●
●●●
●
●●●●
●●
●
●● ●
●
●
●
●●
●●●●
●
●●
●●●
●●
●
●
●
●●●●
●●
●
●●
●●●
●
●
●●
●●●●
●●●●
●●● ●●●●●
●●●
●
●●
●●●
●
●● ●●● ●
●●
●●●●●●
●●
●●●●
●●●
●
●●
● ●●
●●
●
●●● ●
●●
● ●
●
7
●
●
●
●●●
●
●
●●
●
●
●●
●● ●●
●
●●●●
●● ●
●
●
●
●
●
●
●●
●
●
●
● ●
●●● ●●
●
●●●
●●●●
●●
● ●●●
●●●
●●
●●
●●●
●●●
●●●
● ●
●●●●●●
●●●●
●
●●●●●
●
●●●
●
●●
●●
●●
●
●● ●●
●●
●
● ●●●● ●●●●
●●●●
●
●●●
●
●●●
●●
●●
●
●●●●●
●● ●
●
●
●
●
●●●
● ●●
●
●●
●●●
●
●
●
●●●●●●●
●●●●●●●
●●
●
●
●●●
●
●● ●●
● ●●
●
●
● ●●●
●●●●●
●●
●●●
●●● ●
●●●●
●●
●
●●●
●●
●
●●
●
●●●
●●
12
●
●
●
●●
●
●
●●● ●
●
●
●
●
● ●● ●● ●
●● ●● ●● ●
●
●●
●
● ●●
●●●
●
●●●●
●●●●●●
●●●●
●●
●
●●
●●●
●● ●
●●
●●
● ●
●●
●
●●
●●●●●●●●●
●●●
●●●●●
●●
●●
●
●
●● ●●
●●
●
●● ●●
●
● ●●●●
● ●●●●
●●●●
●
●●●●●
●
●
●
●
●
● ●●●●
●
●●●
●● ●
●
●●
●
● ●●●
●
●
●
●● ●●
●●●
●●
●●●
●
●
●●●
●●
●●●●●●● ●●●
●
●
●
●●●
●
●●●
●
●●●●
●●●●
●● ●● ●
●●●
●●●●
●
●●●
●●
●
●●
●●
●
● ●●
●
17
●●
●●●
●
●●●●
●
●
● ● ●●
●●
●
●●●●
●●●
●● ●●
●●
●
●
●●
●
●●
●●
●●
● ●●●
●
●●●●●
●●●●
●●●
●●●
●
●
●
● ●
● ●
● ●●●
●
●●
●●●●●●●●
● ●
●● ●
●●●●
●●●●
●●
●●●
●
●
●●● ●●
●
●
● ●●
●●●
●●
●
●●●
●
●
●●
● ●
●
● ● ●● ●●
●
●●
●
●●
● ●●●●
●
● ●● ●
●●●●● ●
●●
●●
●●●●
●●●●● ●●●●
●●●
●
●●●
●●●
●●
●●●
● ●
●●
●
●●●●●
●●●●
●●
●
●
●●●●
●●●●
●
●
●●● ●
●
●●
●
●
● ●●●●
●
2 3 4 5 6 7
3
●●
●
●●●●●
●
●●●
● ●
●
●
●
●●●●●
●
●●●● ●●
●
●
●
●
●
●●
●●
●● ●●●
●●●● ●●●
●●●
●
●●●●
●●●
●●
●
●
●●
●●●
●●
●● ●
●
●●●●●
●●●●●
●●●
●
●●
●●●
●●
●
●●●
●●
●●●●
●
●
●● ●
●
●●
●
●●●●
●●●
●
●
●●
●● ● ●
●
●
●●●●●●●
●●
●
●
●
●
●●●
●
●
●●●●●
●
●●
●●●
●●●●●
●●● ●●●
●●●
●●
●
●●
●●●
●● ●●
●
●
●●
●●
●
●●
●●●●●●●
●●
●●●● ●
●●
●●●●
●
●
●●
●● ●
●
●● ●●●●●
● ●
8
●●●●●
●●
●●
●
●
●●●
●●
●
●
●
●●
●●●● ●●●●
●● ●
●
●
●
●
●
● ●
● ●●
●●
●●●
●●
●●●●●●●
●●●●●●●
●●
●
● ●●
●● ●
●
●●
●
●●
●●●
●●●●●
●●
●●●
●●●● ●
●●●●
●
●
● ●●
●●
●
●● ●
●
●●
●●
●
●
●●●●●●●●
●●●
●
●●●
●
●●
●●
●
●
●
● ●●
●●●●
●
●● ●
●●●● ●●●
●●
●
●
●●●●
●
●●● ●●
●●●●
●
●
●
●●●● ●
●
●●
●●
●●●●
●
● ●
●●●●●●●
●
●●
●●
●
●●●●
●
●●
● ●
●●
●
●●
●
●
●● ●●●● ● ●
13
●●●●●●
●●●
●
●
●●●
● ●●
● ●● ●●●
●●●
●●
● ●
●●●●
●
●
●●
●●●●
●
●
●
●●
●●●●
●●●●● ●●●●●
●●●
●●●
● ●●
●● ●
●● ●● ●
●●
● ●●
●●●●●●
● ●
●●●●●●
●
●●●
●●
●●
●
● ●●
●
●●
●●●
● ●●●●●
●●●●●
● ●●
●●
●
●
●●●●
●
●●●●
●●●
●●
● ●
●
●
●
●
● ●●
●●●
●●●
●●●
●●●●
●●●●
●●●●●
●●
●
●●
●
●
●●
●●
●
●
●●●
●●
●●●●●
● ●●●
●●●● ●
●●●●●●●
●
●
●●
●●
●
●●●
●
●● ●●●
●
18
●●●
●●
●
●●●
●
●●●
●
●●
● ●●●● ●
● ●●
●
●
●
●
●
●
●●●
●
●
●●
●●● ●●●
●●●
●●●
●●●●
●●●●
●●●●●
●●●
●
●●
●●
●
●
●
●●
●
●●●
●●●
●●
●●● ●
●●
●●●
●●●
●
●●
● ●●
●●●
●●
●
●●
● ●●●●
●
●●●●●
●●●
●
●●●●
●●
●
●
●●●●●
●
●●●●●●
●
●
●
●
●●
●
●●●
●●
●
●
●
●
●●●●
●●●●
●●●
●●
●●
●
●
●●
●●
●● ●
●●
●
●●●●
●
●●
●●●●●
●●●●●●
●●●
●
●●●●●●●
●
●●
●
●
●●● ●
●●
● ●●● ●
2 3 4 5 6 7
4
●●●●
●●●●
●
●
●
●
●●
● ●●
●
●●●
●●
●●
●
●
●
● ●
●
●
●●● ●
●●●● ●●●
●
●
●
●●●
●●
●●●●●
●●
●
●
●● ● ●
●
●●
●●
●●●
●
●
●●
●
●●●●●●●
● ●
●●●●●● ●
●●●●
●●●
●
●●
●
●●●
●
● ●
●●
●
●●
●●●
●
●●●
●
●
●●●
●
●
●
●
●●
●●
●
●
●● ●●●●
●●●
●
●
●
● ●●●●
●
●
●●
●●●
●●●●
●●
●●●●●●
●●●● ●●●
●●
●●●●
●●●●
● ●●
●●
●●
● ●●●●●●●●
●●
●
●
●
●●
●●
●
●●●
●
●
●
●●● ●
●
●● ●●
●●
●
9
●●●●
●
●●●●
●●●●
● ●
●
●●●●
●
●●●●● ●
●
●
●
●●●
●
●
●
●● ●●
●●
●●●
● ●
●●
●●●
●●●●●●
●●●●
●●
●●●
●●●
●●●
●●
● ●
●
●
●●
●●●●●●
●●
●
●●●●
●●●
●●
●
●
●● ●●
●●
●●
●
●
●●
● ●● ●●
●
●●
●
●
●●●
●
●●
●
●
●
●
●●
● ●●●●
●●●
●
●
●
●
●
●
●
●
●●
●
●● ●●●
●
●●
●●
●●
●●● ●●●
●
●
●●
●● ● ●●
●●
●●●
●●● ●
●●
● ●●
●●●●●●●●
● ●●
●● ●
●●●
●
●●●
●●●
●●
●
●●● ●
●
● ●●●●●
●
14
●●
●●●●
●
●●●●
●
●●
●●
●
●
●
● ●●●●●
●
●●
●
●
●
●
● ●
●●●
● ●
●●●●
●
●●●●●●
●●●● ●●●
●●
●●●
●
●●
●
●
●● ●
●●
●●●●
●●●●●●●●
●●
●
●●
●●
●●●●
●●
●●
●●●
●
●
●●● ●●
●
●
●● ●●●
●●●
●
●
●●
●
●●
●● ●
●
●
●●● ●
●
●●●
●● ●●
●
●
● ●
●
●
●
●
●●
●
●●
●
● ●●
●● ●●●
●●
●
●●●●
●●
●●●●●●●
●●
●
●
●●
●● ●●
● ●
●●
●●
●
●
●●●●●●●
● ●●
●●●● ●
●●●
●●
●
●
●
●●●
●●
●
●
●●●
●●
19
●●
●●●
●●●●
●
●●
●
●
● ●● ●●●
●●
●●
●●
●
● ●●● ●● ●
●●●
●●●
●●
●
●●
●●
●●●●●
●●●●●●
●●
● ● ●●
●●
●● ●●
●●
●●●●●
●●●●
●●
●●
●●●●
●●●●
●●●
●
●●
●●
●●
●
●
●●
●●
●●
●●
●●
●●
●●●
●●●●●
●
●
●
●
●
●● ●
●
●
●
●●●●
●
●●●● ●●
●
●
●
● ●●
●
●●●
●
●●
● ●● ●
●●
●●●
●●●●●●
●●●
●●●●
●
●
●●
●●● ●
● ●
●●
●●●
●
●
●●
●●●●●●●●
●
●
●●●
●●
●
●●
●
●●●
●●●● ●●●●●●
●
2 3 4 5 6 7
5
●●●
●
●
●●●●
●
●
●
●●
●● ●
●
●
●
●●●●
●●●
●●
●
●
●●
● ●
●
●●●
●
● ●●
●●
●●●● ●
●
●
●●●●●●
●●●●● ●●●●
●●●
●
●●
●
●
●●
●●●
●●
●
●●●
●●●●
●●
●●● ●
●●●
●
●
●
●●
●●
●
●
●● ●
●●
●●●
●
●●●●●
●
●●
●
●
●●
●
● ●●●●●
●
●●● ●
● ●
●
●
●●
●
●
● ●
●●● ●●
●
●●●
●●●●
●●
●●
●
●●●
●
●
●
●●
● ●●
●● ●
●
●
●●
●
●●
●●●●
●●●●●●
●●●
●
●●●
●●●
●
●●●●
● ●● ●●
●●●● ●
●●
● ●●●● ● ●
10
●
●●●●●●
●●
●
●●
●
●●
●●
●●
●
●●●
●● ●●
●●
●
●
●
●●
●
●
●●
●●●
● ●
●●●●●
●
●●●●
●●●●●●
●●●●
●
●
●
●●
●●
●●● ●
●●
●
●●●●
●●●●●●●
●● ●
●●●●●●
●
●● ●
●
●●
●
●
●● ●
●●
●
● ●●●●●●●●●
●●
●●●
●
●
●
●● ●●
●●● ●●●●
●●●
●
● ●●
●
●●●
●●●●
●●●●●
●●
●
●●
●
●●●●
●●●●
●●
●●
●●
●●●
● ●●●
●●
●
●
●
●
●●●●●●●●
●●●
●●●●● ●
●●●●
●●●
●
●●
●
●●●● ●
●
● ●●
●● ●
15
●●
●●●●●
●●●
●
●●
●
●
●●
●●●● ●●●
●●
●●
●
●●
●●
● ●
●
● ●●●
●● ●●● ●
●●●
●●●●
●●●●
●●●●●●●
●●●
●
●●● ●
●
● ●●
●
●
●●
●●●●●●●●
●●
●
●●●●●
●
●
●
●●●
●●●●●● ●
●●
● ●●●●●●●
●
●
●●●
●
●
●●
●●
●●
●
●
●●
●
●●
●● ●
●
●
●
●●
●
●●●
●
●
●
●●
●●●●
●
●
●
●●●●
●● ●●●
●●●
●
●●
●● ●
●● ●●
●●●
●● ●
●●
●●●
●●●●●●
● ●●
●●● ●
●●●●
●●●●
●
●●
●●
●
●
●
● ●
●●
●●
●● ●
20
●●●
●
●●●●●
●●
●●
● ●
●●
● ●●●
●
●●●● ●●
●
●● ●
●
●●
●●●●
●
● ●●●
●● ●
●●●
●●●●●
●●●●●●●
●●●●
●
●
●
●●●
●●●
●●●
●
●
●
● ●●
●●●●●●●
●●
●●
●●●●
●●
● ● ●●
●●●
●
●●● ●
●
●● ●●●●●●
●
●●
●
●●●
●●
●
●●
●● ●●●
●●
●●
●●
●
●
●●
●
●
●
●
●●
●●●●
●
●●
●●●●
●●●
●●●
●●●●
●
●●●
●●
●
● ●●● ●
●●
●●●●●
●●●●
●●
●●
●●●●●
●●●●●●
●
●
●●
●
●
●
●●
●
●●
●● ●●
●●
●●
2 3 4 5 6 7
factor(year)● 1999● 2008
displ
1/ct
y * 1
00
468
10
468
10
468
10
468
10
1
●●●
●●
●●
●
●●
●●
●●
●
●
● ●●●
●
●●●●●● ●
●
●●
●
●
●
●●●
●
●
●● ●●
●●
●
●●
●
●●●
●●● ●●●●●
●●
●
●
● ●●
●●
●●
●
●● ●
● ●
●●●●●●●●●
● ●●
●●● ●
●●
●●●●
●
●●●
●●
●●● ●
●
●●●●
●● ●
●●●
●
●●
●●●●●
●●
●●
●●
● ●● ●●●
●●●
●
● ●
●
●●
●
●
●●●● ●●●
●● ●
●●
●●●●
●●●●●●
●●●
●●
●●●
●
●●●
● ●
●●
●
●●
●
●
● ●●●
●●●●●●
●●
●
●●●
●●●
●●
●
●
● ●
●●●
●
●
●● ●
●
●●
●●
●●
6
●●●●
●●●
●
●●
●
●
●
●
●●
●
●●●●●
●
●●●● ●
●
● ●
●●●
●
●
●●●
●●
●●●
●●●●
●●●●
●●●●●●
●● ●●
●●
●●
●●● ●
●●
●
●● ●
●
●
●●●●●●
●●●●●
●●
●
●● ●
●●●
●●
●●●
●
●●● ●●
●●
●●
●
●● ●●●●
●●●●
●
●
●●●
●
●
●
●●
●●●●
●●
●● ●
●●
●
●
●
●
●
●●
● ●● ●●
●● ●●
●
●●
●●
●●●●●●●●
●●●●
●
●
●●
●
●● ●
●●
●
●●
●●
● ●●●
●●●●
●●
●
●●●●
●●
●●
●
●●●
●● ●●
●●
●
●
●●●● ●●●
●●●
11
●●
●●●
●●
●●●●●
●
●
●●
●
●
●●●●●●
●
●●
●●
● ●●
●●●●
●
●
●●●●● ●●●
●
●●●●
●●●●
●●●●●● ●●●
●●●●●
●
●●
●
●● ●
●●
●●
●
●●
●●●●
●●●●●●
● ●●●
●
●●●●●
●●●
●●
●●
●
● ●
●
● ●●●●●
●●
●●●●
●● ●
●
●●
●●●●●●●●●● ●●
●●
●
●
●
●● ●● ●●
●●●● ●
●●●●
●
●●●●
●●●
●
●●
●●●
●●●
●
●● ●● ●
●●
●
●●●
●●●●
●●●
●●●●
●●●
●
●
● ●●●
●●●●
●
●
●●● ●
●●
●
● ●●●●
●● ●
16
●●●●●
●
●
●●
●●
●
●
●●
●
●
●
●●● ●●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●●●
●●●●
●●
●●●●● ●●
●●●
●●● ●
● ●
●
●●●
●●
●
●●●●
●●●●●
●●
●●●●● ●
●●●
●●●
●
●●
● ●●
●●
●
●●● ●●
●
●● ●
●●
●●●
●
●●●
●
●
●
●
●● ●
●
●●
●●
●●
●●● ●
●
●●
●
●●●●●
●
●●
●
● ●●
●● ●●
●
●
●●●●
●
●●●●●●
●●
●●●
●●●●
●●
●
●●
●●
●●
●●
●
● ●
●
●●●●●
●●●●●
●●
●●
●●●●
●
●●
●
●●
●●
●
●
●●
●
●●● ● ●
2 3 4 5 6 7
2
●●●
●●●
●●
●●
●
●
●●
●
●●●●●●●●
●●
●
● ●
●
●
●
●
●● ●●
●●●●
●●
●●
●
●●●●
●●●●
●●
●●
● ● ●●●
● ●●
●●
●
●●
●
●
●●●●●
●●●●●●
●●●
●
●●●
●●●
●
●●●●
●
●
● ●
●●
●●
●
●●
● ●●●●●
● ●
●●●
●
●●●●
●●
●
●● ●
●
●
●
●●
●●●●
●
●●
●●●
●●
●
●
●
●●●●
●●
●
●●
●●●
●
●
●●
●●●●
●●●●
●●● ●●●●●
●●●
●
●●
●●●
●
●● ●●● ●
●●
●●●●●●
●●
●●●●
●●●
●
●●
● ●●
●●
●
●●● ●
●●
● ●
●
7
●
●
●
●●●
●
●
●●
●
●
●●
●● ●●
●
●●●●
●● ●
●
●
●
●
●
●
●●
●
●
●
● ●
●●● ●●
●
●●●
●●●●
●●
● ●●●
●●●
●●
●●
●●●
●●●
●●●
● ●
●●●●●●
●●●●
●
●●●●●
●
●●●
●
●●
●●
●●
●
●● ●●
●●
●
● ●●●● ●●●●
●●●●
●
●●●
●
●●●
●●
●●
●
●●●●●
●● ●
●
●
●
●
●●●
● ●●
●
●●
●●●
●
●
●
●●●●●●●
●●●●●●●
●●
●
●
●●●
●
●● ●●
● ●●
●
●
● ●●●
●●●●●
●●
●●●
●●● ●
●●●●
●●
●
●●●
●●
●
●●
●
●●●
●●
12
●
●
●
●●
●
●
●●● ●
●
●
●
●
● ●● ●● ●
●● ●● ●● ●
●
●●
●
● ●●
●●●
●
●●●●
●●●●●●
●●●●
●●
●
●●
●●●
●● ●
●●
●●
● ●
●●
●
●●
●●●●●●●●●
●●●
●●●●●
●●
●●
●
●
●● ●●
●●
●
●● ●●
●
● ●●●●
● ●●●●
●●●●
●
●●●●●
●
●
●
●
●
● ●●●●
●
●●●
●● ●
●
●●
●
● ●●●
●
●
●
●● ●●
●●●
●●
●●●
●
●
●●●
●●
●●●●●●● ●●●
●
●
●
●●●
●
●●●
●
●●●●
●●●●
●● ●● ●
●●●
●●●●
●
●●●
●●
●
●●
●●
●
● ●●
●
17
●●
●●●
●
●●●●
●
●
● ● ●●
●●
●
●●●●
●●●
●● ●●
●●
●
●
●●
●
●●
●●
●●
● ●●●
●
●●●●●
●●●●
●●●
●●●
●
●
●
● ●
● ●
● ●●●
●
●●
●●●●●●●●
● ●
●● ●
●●●●
●●●●
●●
●●●
●
●
●●● ●●
●
●
● ●●
●●●
●●
●
●●●
●
●
●●
● ●
●
● ● ●● ●●
●
●●
●
●●
● ●●●●
●
● ●● ●
●●●●● ●
●●
●●
●●●●
●●●●● ●●●●
●●●
●
●●●
●●●
●●
●●●
● ●
●●
●
●●●●●
●●●●
●●
●
●
●●●●
●●●●
●
●
●●● ●
●
●●
●
●
● ●●●●
●
2 3 4 5 6 7
3
●●
●
●●●●●
●
●●●
● ●
●
●
●
●●●●●
●
●●●● ●●
●
●
●
●
●
●●
●●
●● ●●●
●●●● ●●●
●●●
●
●●●●
●●●
●●
●
●
●●
●●●
●●
●● ●
●
●●●●●
●●●●●
●●●
●
●●
●●●
●●
●
●●●
●●
●●●●
●
●
●● ●
●
●●
●
●●●●
●●●
●
●
●●
●● ● ●
●
●
●●●●●●●
●●
●
●
●
●
●●●
●
●
●●●●●
●
●●
●●●
●●●●●
●●● ●●●
●●●
●●
●
●●
●●●
●● ●●
●
●
●●
●●
●
●●
●●●●●●●
●●
●●●● ●
●●
●●●●
●
●
●●
●● ●
●
●● ●●●●●
● ●
8
●●●●●
●●
●●
●
●
●●●
●●
●
●
●
●●
●●●● ●●●●
●● ●
●
●
●
●
●
● ●
● ●●
●●
●●●
●●
●●●●●●●
●●●●●●●
●●
●
● ●●
●● ●
●
●●
●
●●
●●●
●●●●●
●●
●●●
●●●● ●
●●●●
●
●
● ●●
●●
●
●● ●
●
●●
●●
●
●
●●●●●●●●
●●●
●
●●●
●
●●
●●
●
●
●
● ●●
●●●●
●
●● ●
●●●● ●●●
●●
●
●
●●●●
●
●●● ●●
●●●●
●
●
●
●●●● ●
●
●●
●●
●●●●
●
● ●
●●●●●●●
●
●●
●●
●
●●●●
●
●●
● ●
●●
●
●●
●
●
●● ●●●● ● ●
13
●●●●●●
●●●
●
●
●●●
● ●●
● ●● ●●●
●●●
●●
● ●
●●●●
●
●
●●
●●●●
●
●
●
●●
●●●●
●●●●● ●●●●●
●●●
●●●
● ●●
●● ●
●● ●● ●
●●
● ●●
●●●●●●
● ●
●●●●●●
●
●●●
●●
●●
●
● ●●
●
●●
●●●
● ●●●●●
●●●●●
● ●●
●●
●
●
●●●●
●
●●●●
●●●
●●
● ●
●
●
●
●
● ●●
●●●
●●●
●●●
●●●●
●●●●
●●●●●
●●
●
●●
●
●
●●
●●
●
●
●●●
●●
●●●●●
● ●●●
●●●● ●
●●●●●●●
●
●
●●
●●
●
●●●
●
●● ●●●
●
18
●●●
●●
●
●●●
●
●●●
●
●●
● ●●●● ●
● ●●
●
●
●
●
●
●
●●●
●
●
●●
●●● ●●●
●●●
●●●
●●●●
●●●●
●●●●●
●●●
●
●●
●●
●
●
●
●●
●
●●●
●●●
●●
●●● ●
●●
●●●
●●●
●
●●
● ●●
●●●
●●
●
●●
● ●●●●
●
●●●●●
●●●
●
●●●●
●●
●
●
●●●●●
●
●●●●●●
●
●
●
●
●●
●
●●●
●●
●
●
●
●
●●●●
●●●●
●●●
●●
●●
●
●
●●
●●
●● ●
●●
●
●●●●
●
●●
●●●●●
●●●●●●
●●●
●
●●●●●●●
●
●●
●
●
●●● ●
●●
● ●●● ●
2 3 4 5 6 7
4
●●●●
●●●●
●
●
●
●
●●
● ●●
●
●●●
●●
●●
●
●
●
● ●
●
●
●●● ●
●●●● ●●●
●
●
●
●●●
●●
●●●●●
●●
●
●
●● ● ●
●
●●
●●
●●●
●
●
●●
●
●●●●●●●
● ●
●●●●●● ●
●●●●
●●●
●
●●
●
●●●
●
● ●
●●
●
●●
●●●
●
●●●
●
●
●●●
●
●
●
●
●●
●●
●
●
●● ●●●●
●●●
●
●
●
● ●●●●
●
●
●●
●●●
●●●●
●●
●●●●●●
●●●● ●●●
●●
●●●●
●●●●
● ●●
●●
●●
● ●●●●●●●●
●●
●
●
●
●●
●●
●
●●●
●
●
●
●●● ●
●
●● ●●
●●
●
9
●●●●
●
●●●●
●●●●
● ●
●
●●●●
●
●●●●● ●
●
●
●
●●●
●
●
●
●● ●●
●●
●●●
● ●
●●
●●●
●●●●●●
●●●●
●●
●●●
●●●
●●●
●●
● ●
●
●
●●
●●●●●●
●●
●
●●●●
●●●
●●
●
●
●● ●●
●●
●●
●
●
●●
● ●● ●●
●
●●
●
●
●●●
●
●●
●
●
●
●
●●
● ●●●●
●●●
●
●
●
●
●
●
●
●
●●
●
●● ●●●
●
●●
●●
●●
●●● ●●●
●
●
●●
●● ● ●●
●●
●●●
●●● ●
●●
● ●●
●●●●●●●●
● ●●
●● ●
●●●
●
●●●
●●●
●●
●
●●● ●
●
● ●●●●●
●
14
●●
●●●●
●
●●●●
●
●●
●●
●
●
●
● ●●●●●
●
●●
●
●
●
●
● ●
●●●
● ●
●●●●
●
●●●●●●
●●●● ●●●
●●
●●●
●
●●
●
●
●● ●
●●
●●●●
●●●●●●●●
●●
●
●●
●●
●●●●
●●
●●
●●●
●
●
●●● ●●
●
●
●● ●●●
●●●
●
●
●●
●
●●
●● ●
●
●
●●● ●
●
●●●
●● ●●
●
●
● ●
●
●
●
●
●●
●
●●
●
● ●●
●● ●●●
●●
●
●●●●
●●
●●●●●●●
●●
●
●
●●
●● ●●
● ●
●●
●●
●
●
●●●●●●●
● ●●
●●●● ●
●●●
●●
●
●
●
●●●
●●
●
●
●●●
●●
19
●●
●●●
●●●●
●
●●
●
●
● ●● ●●●
●●
●●
●●
●
● ●●● ●● ●
●●●
●●●
●●
●
●●
●●
●●●●●
●●●●●●
●●
● ● ●●
●●
●● ●●
●●
●●●●●
●●●●
●●
●●
●●●●
●●●●
●●●
●
●●
●●
●●
●
●
●●
●●
●●
●●
●●
●●
●●●
●●●●●
●
●
●
●
●
●● ●
●
●
●
●●●●
●
●●●● ●●
●
●
●
● ●●
●
●●●
●
●●
● ●● ●
●●
●●●
●●●●●●
●●●
●●●●
●
●
●●
●●● ●
● ●
●●
●●●
●
●
●●
●●●●●●●●
●
●
●●●
●●
●
●●
●
●●●
●●●● ●●●●●●
●
2 3 4 5 6 7
5
●●●
●
●
●●●●
●
●
●
●●
●● ●
●
●
●
●●●●
●●●
●●
●
●
●●
● ●
●
●●●
●
● ●●
●●
●●●● ●
●
●
●●●●●●
●●●●● ●●●●
●●●
●
●●
●
●
●●
●●●
●●
●
●●●
●●●●
●●
●●● ●
●●●
●
●
●
●●
●●
●
●
●● ●
●●
●●●
●
●●●●●
●
●●
●
●
●●
●
● ●●●●●
●
●●● ●
● ●
●
●
●●
●
●
● ●
●●● ●●
●
●●●
●●●●
●●
●●
●
●●●
●
●
●
●●
● ●●
●● ●
●
●
●●
●
●●
●●●●
●●●●●●
●●●
●
●●●
●●●
●
●●●●
● ●● ●●
●●●● ●
●●
● ●●●● ● ●
10
●
●●●●●●
●●
●
●●
●
●●
●●
●●
●
●●●
●● ●●
●●
●
●
●
●●
●
●
●●
●●●
● ●
●●●●●
●
●●●●
●●●●●●
●●●●
●
●
●
●●
●●
●●● ●
●●
●
●●●●
●●●●●●●
●● ●
●●●●●●
●
●● ●
●
●●
●
●
●● ●
●●
●
● ●●●●●●●●●
●●
●●●
●
●
●
●● ●●
●●● ●●●●
●●●
●
● ●●
●
●●●
●●●●
●●●●●
●●
●
●●
●
●●●●
●●●●
●●
●●
●●
●●●
● ●●●
●●
●
●
●
●
●●●●●●●●
●●●
●●●●● ●
●●●●
●●●
●
●●
●
●●●● ●
●
● ●●
●● ●
15
●●
●●●●●
●●●
●
●●
●
●
●●
●●●● ●●●
●●
●●
●
●●
●●
● ●
●
● ●●●
●● ●●● ●
●●●
●●●●
●●●●
●●●●●●●
●●●
●
●●● ●
●
● ●●
●
●
●●
●●●●●●●●
●●
●
●●●●●
●
●
●
●●●
●●●●●● ●
●●
● ●●●●●●●
●
●
●●●
●
●
●●
●●
●●
●
●
●●
●
●●
●● ●
●
●
●
●●
●
●●●
●
●
●
●●
●●●●
●
●
●
●●●●
●● ●●●
●●●
●
●●
●● ●
●● ●●
●●●
●● ●
●●
●●●
●●●●●●
● ●●
●●● ●
●●●●
●●●●
●
●●
●●
●
●
●
● ●
●●
●●
●● ●
20
●●●
●
●●●●●
●●
●●
● ●
●●
● ●●●
●
●●●● ●●
●
●● ●
●
●●
●●●●
●
● ●●●
●● ●
●●●
●●●●●
●●●●●●●
●●●●
●
●
●
●●●
●●●
●●●
●
●
●
● ●●
●●●●●●●
●●
●●
●●●●
●●
● ● ●●
●●●
●
●●● ●
●
●● ●●●●●●
●
●●
●
●●●
●●
●
●●
●● ●●●
●●
●●
●●
●
●
●●
●
●
●
●
●●
●●●●
●
●●
●●●●
●●●
●●●
●●●●
●
●●●
●●
●
● ●●● ●
●●
●●●●●
●●●●
●●
●●
●●●●●
●●●●●●
●
●
●●
●
●
●
●●
●
●●
●● ●●
●●
●●
2 3 4 5 6 7
factor(year)● 1999● 2008
qplot(displ, cty, data = mpg, colour = factor(year))qplot(displ, 1 / cty * 100, data = mpg, colour = factor(year))
mpg_perm <- resamp(mpg,permute_var("year"))last_plot() %+% mpg_perm + facet_wrap(~ .n)
Is a linear model with displacement as single
predictor adequate?
displ
gp10
0m
2468
10
2468
10
2468
10
2468
10
1
●
● ●●●
●●
●
●
●●●
●●
●
●● ●●●
●●●
●
●●
●
● ●
●●
●● ●
●●
●
●
●
●●
●
●
●●
●●
●●●●
●●●●●
●●
● ●● ●●●
●
●
●●
●●
●●● ●
●
●●
●●●●
●
●
●
●
●●●●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●●●
●●●●●
●●
●
●●●
●●● ●
●●
●●●● ●●
●●
●●
●
●●●
●●●
●
●
●
●●
●●●●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●
● ●●
●●
●●●
●●
●●●●●
●●
●
●
●
●
●
●●
●
●
● ● ●
●●
●
●
● ●
●
●
●
●
●●
●●●
●
●
●●●
●
●
●
6
●
● ●
●●●
●
●
●
●●●●
●
● ●●
●●●
●●
●
●
●●●
●●●
●●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●
●
●
●● ●
●●●
●●●
●●
●●
●
● ●●
●
●●●
●
●●●
●●●●
●●●●
●●
●●
●●●● ●
●
●
●
●
●●●
●
●
●●
●
●
●● ●
●●
●
●●
●●
●
●●
●
●
●
● ●
●
● ●●●
● ●●● ●
●
●
●
●
●
●
●● ●●●
●●●●●
●
●
●
●
●
●●●●
●
●
●
●●
●●●
●● ●
●●●●●
●●●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
● ●●
●●
●
●
●
●●●●●
●●●
●●
●●
●●
● ●●●
●
●●
●
●●●●
●●●
●
●●
●
11
●● ●
●
●●
●
●●
●
●
●●
●
●●
●●●●
●
●
●●
●●
●
●
●
●●
●
●●
●●●
●
●
●●●
●
●
● ●
●
●
●●●●
●
●●●●
●
●
●●
●●
●
●
●
●●
●●●●●
●●
●●●●
●●●●
●
●
●●
●
●
●
● ●●●
●
●
●
●●
●
●
●●●
●●●
●
●
●
●●
●
●
●
●●●●
●●●
●
●
●
● ●●
●
●
●● ●
●●
● ●●
●●●
●
●
●● ●
●
●●●
●●
●●●●●●
●●
●●●
●
●
● ●●
●
●●●●
●
● ●●
●
●●
●●
●
●● ●
●●
●
●
● ●
●●
●●●
●●
●
●
●●●●●
●
●
●●
●
●●
●●●
●
●●
●
●
●
●
●
●●●
●●●●●●
●
16
●●
●
●
●
●
●●
●●●●● ●
● ●
●
●●●
●
●●
●
●●●
●●
●●
●●
●
●●●
●
●
●
●● ●
●
● ●●
●
●
●
●
●
●●●●
●●
●
●● ●
●●
●●
●
●
●●●●●
●●●
●●●●●
●●●●
●●
●
●
●
● ●●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●●●
●●
●●●
●
●●● ●
●
●
●
●
●
● ●●
● ●●
●
●●
●
●
●
●●●●●
●
●
●●
●
●●●
●
●●●● ●
●
●●●
●
●
●●●
●
●●●
●●●
●●●
●●●
●
●
●●
●●
●●
●
●●
●●
●
●
●●
●● ●
●●
●●
●●●●
●
●
●●
●●●
●●
●
●
● ●
●
●
●●●
●
●
●●
●●
●
2 3 4 5 6 7
2
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●
●●●
●●●
●
●
●
● ●
●
● ●
●●
●
●
●
●●
●●●
●
●
●●●
●
●●●
●●
●●●
●
●
●
●●
●
●●
● ●●
●
●
●
●●
● ●
●
●●●
●●●●●
●●
●●
●
●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●
●
●●●●
●
●
●●
●
●
●
●
●
●
● ●●●●
●
●● ●●
●
●
●●
●
●●
●●●
●●
●●●●●
●●
●●●
●● ●●
●●● ●
●●
●
●
●
●●●●
●● ●●●
●●
●●
●
●
●
●
●●●
●
●●
●
●●●●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
● ●●
●●●● ●
●
●
●
●
● ●
7
●●
●●●●
●●
●
●
●
●●
●
● ●
●
●
●●
●●●
● ●
●
●
●●●●●●
●●●●
●
●
●
●●
●
●
●●●
●
●
●
●● ●
●
●
●●●
●
●● ●
●●● ●
●●
●
●●
●
●
●●●
●●●
●●●●
●●
●●
●●
●
●
●
●
●●
●
●●
● ●
●
●●
●●●
●
●●●
●●●
●
●●
●●● ●●
●
●●●
●●
●
●
●●●
●
●
●
●●●
●●●
●●
●
●
●●●
●
●●
●
●
●●●●
●
●●
●
●●●
●●●
●
●
●●
●●
●
● ●●
●
●●
●
● ●●●
●●●● ●
●
●
●●●●●
●
●
● ●
●●
●
●●●●● ● ●
●●●
●●
●●
●
●
●
●
●●●
●●
●
●●
●
●●
●
12
●
●
●
●
●●
●
●●
●●
●
●●
●●●●
●
●●●
●●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●●
●
●●●●●
● ●
●
●
● ●●●●
●●●●●●
●
●●
●
●●
●●●●●
●●●
●●
●●● ●●
●
●
●
●
●●
●●
●●
●
●●●
●
●
●●
●
●
●●
●●●
●
●
●
●
●●●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●●●
●●●
●
●●
●● ●
●●
●●●
●●●●
●●●
●●●●●●
● ●●●
●
●●
●●●
●
●
●●● ●
●● ●
●
●
●●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●●
●●
●
●
17
●●
●●
●
●●
●●
●
●
●
●
●
●
●● ●●
●
●
●
●
●
●
●●
● ●
●
● ●● ●●
●●
●
●●
●
●
●
●
●
●●
●●●
●●
●
●●●
●●
●
●
●●
●
●● ●
●
● ●●●●●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●
●●
●●●
●
●
●●
●
●
●
●
●●●●
●
●
●●
●
●
●●
●●●
●
●●
●
●
●
●●●●●●●●
●●
●●●●●
●
●
●●●●●● ●
●
●
●●
●
●
●
●●●
●●●
●●
●
●●
●●
●●●
●
●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●
●●●●
●●●
●
●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●●
●
●
●
2 3 4 5 6 7
3
●
● ●●●
●
●●●
●
●
●●●
●
●● ●●●
●●●
●●●
●●
●●●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●●●●●
●
●
●
●●
● ●
●● ●
●
● ●●
●
●
●●
●
●
●●
●●●●
●
●●●
●
●
●●
●
●●
●
●
●
●●●
●
●●
●●
●●
●●●
●
●
●
●
●●
●●● ●●
●
●
●
●●●
●
●
●●●●
●
●●
●
●●●
●●
●●
●
●●●●
●
●●
●●●
●●●
●●●
● ●
●● ●
● ●●●●● ●
●●●
●
●
● ●
●
● ●●
●●
●●
● ●
●
●●
●●
●
●●
●●● ●
●
●
●
●
●●●●●●●
●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●●●●
●●●
●
8
●
●
●
●●●
●●●
●
●
●●●
●
●
●
●
●
●●●
●
●
●●●
● ●
●●
●
●
●●
●●
●●
●
●●
●
●●
●
●
●
●
●●● ●●●
●
●●
● ●●●
●
●●
●
●●
●●
●●●
●●
●●●
●
●●●
●●
●●●
●
●
●●
●●●●
●
●
●
●
●●●●
●●●●
●●●●●●●●
●●
●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●
●
●
● ●
●●
● ●●
●
●
●● ●●
●●●●●
●
●● ●
●
●
●●
●
●●
●●●
●●●●
●●
●●●●
●
●●
●
●●
●
●
● ●●● ●
●●●
●●●
●
●●●
●●●●
●
●
●●
●
●●●
●
●
●
●
●●
●●
●● ●
●
●●●
●
●
13
●
●●
●
●
●
●●
●
●●●●
●
●
●
●●●
● ●●
●●
●
●
●●
●
●● ●●
●
●
●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●●●
●●
●
●●
●
●
●
●
●●●●●●●
●
●●●
●
●
●
●
●●●
●
●
●●
●
●
●●●●●●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●●●●
●
●●●
●
●
●
●
●●
●
●●●●●●
●●●
●
●●
●
●
●●
●
●
●
●
●
●●●
● ●●
●
●●
●
●
●
●●●
●
●●● ●
●
●
●●●●●
●
●
●
●
●●●●
●
●
●
●●
●●
●
●
●
●
● ●
●●●
●
●
●
●
●
●●
●●
●
● ●
●
●●
●●
●
●
●●
●
●●●
●●
●
● ●
●●●
●●
●
18
●
●
●
●
●●
●
●●
●●●● ●
●
●●
●
●●
●●●
●●●●●
●●
●
●
●
●●
●●
●●●
●
●
●
●●●
●
●●
●●●
●●●●
●●
●
●●
● ●
●
●
●
●●●●●●
●●●
●
●
●●●
●
●
● ●●
●●●
●●
●
●●●●●
●
●●
●
●
●●
●●●
●
●
●●
●●●●●
●
●
●● ●●●
●
●●
●
●●
● ●●● ●●●
●
●●
●
●●
●
●●●
●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●● ●
●
●●
●
●
●●●
●●
●
●
●●
●●●●
●●●
●
●●●
●●●
●
●
●●●
●
●●
●
●●
●
●●●●
●
●
●●
●
●●
●
●
●●
●
●●●●
●●
●●
●
●
●
●● ●
2 3 4 5 6 7
4
●●
●
●
●●●
●●
●
●
●●
●
●
●●
●
●● ●
●●●●●●●
●
●
● ●●●
●●●
●
●●
●
●
●
●●
●
●
●●●●●
●●●
●
●●
● ●
●●
●●
● ●
●●
●●●●
● ●●●
●●●●
●
●
●●
●●●
●●
●●
●●
●●●
●
●● ●●
●●
●
●●
●
●●
●●●
●●●
●●●●●●
●
●●●
● ●
●● ●
●
●●●
●
●
●
●●●
●
●
●●
●●●●●●
●●
●●
●●
●●
●
●
●
●
●
●●●
●
●●
●
●●●●
●●
●●
●●
●
●
● ●●●
●
●
● ●
●●
●●●
●●● ●●
●
●●●●●
●●
●
●
●
●●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●●
●●●
9
●●
●●●
● ●●●
●
●●
●
●
● ●● ●●●●●●●
●●
●
● ●●● ●● ●
●●●
●●●
●●
●
●●
●●
●●●●●
●●●●●●
●●
● ● ●●
●●
●● ●●
●●
●●●●●
●●●●
●● ●●
●●●●
●●●●
●●●
●
●● ●●
●●
●
●
●●
●●
●●
●●
●●
●●
●●●
●●●●● ●
●
●
●
●
●● ●●
●
●●●●●
●
●●●● ●●
●
●
●
● ●●
●
●●●
●
●●
● ●● ●
●●●●
●
●●●●●●
●●●●●●●
●
●
●●
●●● ●
● ●
●●●●●
●
●
●●
●●●●●●●●
●●
●●●
●●●
●●
●
●●●
●●●● ●●●●●●
●
14
●●●●
●●
●●●
●●●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●
●●
●●
●●
●
●●
●●●●●
●
●
●
●
●
●
●
●● ●
●
●
●●
●● ●●●
●
● ●●●●●
●●●●●
●●●
●
●●
●
●●
●●●
●
●
●● ●
●
●
●
●●●
●
●●
●●●●
●
●
●●
●●
●
●
●
●●● ●
●●●
●
●● ●
●
●
●
●●●
●●●
●
●
●●
●●●
●
● ●
●
●●
●
●●
●
●●●●
●
●
● ●
●●
●●●●
●
●
●
●
●●●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●●●●
●
●
●
●
● ●
●
●
●●
●
●
●
●●
●●●
●
●
●●●●●
●
19
●●●
●
●
● ●●●
●●●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●
●●●
●
● ●●●●
●
●
●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
● ●
● ●
●●●
●
●
●
●●●●●
●
●
●●●●●
●●
●●●
●●
●
●
●
● ●
●●●
●
●●● ●
●
●●
●
●
●● ●
●●
●
●●
●
●●●
●●
●
●●
●
●
●● ●
●●
●
●●
●
●
●
● ●
●
●
●
●
●
●●●
●●●
●●
●
●
●
●
●
●●●
●
●● ●
●●
●
●
●●
●
●
●
●●●●
●●●
●●●
● ●
●
●
●●
●
●
●
●●
●●
●●●
●
●●
●
●
●
●●●●●●
●
●
●●
●
●●
●
●●
●
●●●
●
●
●● ●
●●●
●
●
●
2 3 4 5 6 7
5
●
●
●●
●●
●●●
●●
●● ●
●●
●
●●●
●●
●
●
●●● ●
●●●
●●
●●●●
●
●
●
●● ●
●● ●
●
●
●●
●
● ●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●●
●
●●●
●●●
●●
●
●
●
●
●●●
●●
●
●● ●●
●
●
●●
●
●●
●
●●
●
●
●
●
●●●
●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
●
●●●
●●●●
●
●
●
●
●
●
●●
●●●●●
●●
●
●●
●
●
●
●
● ●
●
●●
●
●
● ●
●
●
●●●●
●
●
●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●●
●●
●
●
●●
●●
●●
●
10
●
●●●
●
●
●
●●
●
●
●●
●
●●● ●●●
●●
●
● ●●
●
●●
●● ●
●
●●●●
●●●●
●
●●●
●
●
●●●●●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●●
●●●
●●
●●●●
●●
●
●
●●
●●●●
●●
●●●●●
●
●●●●
●●
●●●●
●●●●
●
●
●
●●●●● ●
●●
●
●
●
●●●
●
●
●●
●
●●
●
●●
●●
●
●●●●
●●●●●● ●●
●●●●● ●●
●
●
● ●
●
●
●●
●
●
●●
●●●
●
●
●●●●●
●●
● ●●
●●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●●
● ●●●
●
●
●●
●●
●
●●●●●
●●
●
15
●
●
●●
●●
●
●
●
●●●● ●
●●
● ●
●●
●
●
●
●
●
●
●
● ●●●
●
●
●●
●
●
●●●
●●
●
●●
●
●
●
●●●●
●
●●
●
●●
● ●
● ●
●●
●
●
●
●
●
●●●●
●●
●
●
●●●
●
●●
●
●
●●
●●●● ●
●●●
●
●
●
●●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●●
●●● ●
●●
●
●
●●●
●
●
●●
●●
●●●
●●
●●
●
●
●
●
●
●●
●
●●
●●●
●●
●
●
●●
●
●
●
●●●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●●●
● ●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●●●
●
●●●
●
●●
20
●●
●
●
●●
●
●●
●●●●
●
●
●● ●
●●
●
●●●
●●
●●
●
●●●
●
●
●●●
●
●●●●
●●●
●●
●●
●
●
● ●●●●
●●
●
●
●
●
●●
● ●
●
●
●
●●●●
●●●
●●
●
●
●
●
● ●
●●● ●
●
●
●●●
●●●
●●●
●
●
●●
●
●
●
● ●●
●
●
●●
●● ●
●●
●●
●●●●●
●
●
●●
●
●
● ●
●
●
●
●●
●
●●●●
●
●
●●●●●●
●●
●
●
●
●●
●●
●
●●●●● ●
●
●
●●
●●
●
●
●
●●
●●●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
● ●
2 3 4 5 6 7
factor(year)● 1999● 2008
displ
gp10
0m
2468
10
2468
10
2468
10
2468
10
1
●
● ●●●
●●
●
●
●●●
●●
●
●● ●●●
●●●
●
●●
●
● ●
●●
●● ●
●●
●
●
●
●●
●
●
●●
●●
●●●●
●●●●●
●●
● ●● ●●●
●
●
●●
●●
●●● ●
●
●●
●●●●
●
●
●
●
●●●●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●●●
●●●●●
●●
●
●●●
●●● ●
●●
●●●● ●●
●●
●●
●
●●●
●●●
●
●
●
●●
●●●●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●
●
● ●●
●●
●●●
●●
●●●●●
●●
●
●
●
●
●
●●
●
●
● ● ●
●●
●
●
● ●
●
●
●
●
●●
●●●
●
●
●●●
●
●
●
6
●
● ●
●●●
●
●
●
●●●●
●
● ●●
●●●
●●
●
●
●●●
●●●
●●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●
●
●
●● ●
●●●
●●●
●●
●●
●
● ●●
●
●●●
●
●●●
●●●●
●●●●
●●
●●
●●●● ●
●
●
●
●
●●●
●
●
●●
●
●
●● ●
●●
●
●●
●●
●
●●
●
●
●
● ●
●
● ●●●
● ●●● ●
●
●
●
●
●
●
●● ●●●
●●●●●
●
●
●
●
●
●●●●
●
●
●
●●
●●●
●● ●
●●●●●
●●●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
● ●●
●●
●
●
●
●●●●●
●●●
●●
●●
●●
● ●●●
●
●●
●
●●●●
●●●
●
●●
●
11
●● ●
●
●●
●
●●
●
●
●●
●
●●
●●●●
●
●
●●
●●
●
●
●
●●
●
●●
●●●
●
●
●●●
●
●
● ●
●
●
●●●●
●
●●●●
●
●
●●
●●
●
●
●
●●
●●●●●
●●
●●●●
●●●●
●
●
●●
●
●
●
● ●●●
●
●
●
●●
●
●
●●●
●●●
●
●
●
●●
●
●
●
●●●●
●●●
●
●
●
● ●●
●
●
●● ●
●●
● ●●
●●●
●
●
●● ●
●
●●●
●●
●●●●●●
●●
●●●
●
●
● ●●
●
●●●●
●
● ●●
●
●●
●●
●
●● ●
●●
●
●
● ●
●●
●●●
●●
●
●
●●●●●
●
●
●●
●
●●
●●●
●
●●
●
●
●
●
●
●●●
●●●●●●
●
16
●●
●
●
●
●
●●
●●●●● ●
● ●
●
●●●
●
●●
●
●●●
●●
●●
●●
●
●●●
●
●
●
●● ●
●
● ●●
●
●
●
●
●
●●●●
●●
●
●● ●
●●
●●
●
●
●●●●●
●●●
●●●●●
●●●●
●●
●
●
●
● ●●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●●●
●●
●●●
●
●●● ●
●
●
●
●
●
● ●●
● ●●
●
●●
●
●
●
●●●●●
●
●
●●
●
●●●
●
●●●● ●
●
●●●
●
●
●●●
●
●●●
●●●
●●●
●●●
●
●
●●
●●
●●
●
●●
●●
●
●
●●
●● ●
●●
●●
●●●●
●
●
●●
●●●
●●
●
●
● ●
●
●
●●●
●
●
●●
●●
●
2 3 4 5 6 7
2
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●
●●●
●●●
●
●
●
● ●
●
● ●
●●
●
●
●
●●
●●●
●
●
●●●
●
●●●
●●
●●●
●
●
●
●●
●
●●
● ●●
●
●
●
●●
● ●
●
●●●
●●●●●
●●
●●
●
●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●●
●
●●●●
●
●
●●
●
●
●
●
●
●
● ●●●●
●
●● ●●
●
●
●●
●
●●
●●●
●●
●●●●●
●●
●●●
●● ●●
●●● ●
●●
●
●
●
●●●●
●● ●●●
●●
●●
●
●
●
●
●●●
●
●●
●
●●●●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
● ●●
●●●● ●
●
●
●
●
● ●
7
●●
●●●●
●●
●
●
●
●●
●
● ●
●
●
●●
●●●
● ●
●
●
●●●●●●
●●●●
●
●
●
●●
●
●
●●●
●
●
●
●● ●
●
●
●●●
●
●● ●
●●● ●
●●
●
●●
●
●
●●●
●●●
●●●●
●●
●●
●●
●
●
●
●
●●
●
●●
● ●
●
●●
●●●
●
●●●
●●●
●
●●
●●● ●●
●
●●●
●●
●
●
●●●
●
●
●
●●●
●●●
●●
●
●
●●●
●
●●
●
●
●●●●
●
●●
●
●●●
●●●
●
●
●●
●●
●
● ●●
●
●●
●
● ●●●
●●●● ●
●
●
●●●●●
●
●
● ●
●●
●
●●●●● ● ●
●●●
●●
●●
●
●
●
●
●●●
●●
●
●●
●
●●
●
12
●
●
●
●
●●
●
●●
●●
●
●●
●●●●
●
●●●
●●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●●
●
●●●●●
● ●
●
●
● ●●●●
●●●●●●
●
●●
●
●●
●●●●●
●●●
●●
●●● ●●
●
●
●
●
●●
●●
●●
●
●●●
●
●
●●
●
●
●●
●●●
●
●
●
●
●●●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●●●
●●●
●
●●
●● ●
●●
●●●
●●●●
●●●
●●●●●●
● ●●●
●
●●
●●●
●
●
●●● ●
●● ●
●
●
●●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●●
●●
●
●
17
●●
●●
●
●●
●●
●
●
●
●
●
●
●● ●●
●
●
●
●
●
●
●●
● ●
●
● ●● ●●
●●
●
●●
●
●
●
●
●
●●
●●●
●●
●
●●●
●●
●
●
●●
●
●● ●
●
● ●●●●●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●●●
●●
●●●
●
●
●●
●
●
●
●
●●●●
●
●
●●
●
●
●●
●●●
●
●●
●
●
●
●●●●●●●●
●●
●●●●●
●
●
●●●●●● ●
●
●
●●
●
●
●
●●●
●●●
●●
●
●●
●●
●●●
●
●
●●
●
●●
●
●●
●
●●
●●●
●●
●
●
●●●●
●●●
●
●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●●●
●
●
●
2 3 4 5 6 7
3
●
● ●●●
●
●●●
●
●
●●●
●
●● ●●●
●●●
●●●
●●
●●●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●
●●●●●
●
●
●
●●
● ●
●● ●
●
● ●●
●
●
●●
●
●
●●
●●●●
●
●●●
●
●
●●
●
●●
●
●
●
●●●
●
●●
●●
●●
●●●
●
●
●
●
●●
●●● ●●
●
●
●
●●●
●
●
●●●●
●
●●
●
●●●
●●
●●
●
●●●●
●
●●
●●●
●●●
●●●
● ●
●● ●
● ●●●●● ●
●●●
●
●
● ●
●
● ●●
●●
●●
● ●
●
●●
●●
●
●●
●●● ●
●
●
●
●
●●●●●●●
●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●●●●
●●●
●
8
●
●
●
●●●
●●●
●
●
●●●
●
●
●
●
●
●●●
●
●
●●●
● ●
●●
●
●
●●
●●
●●
●
●●
●
●●
●
●
●
●
●●● ●●●
●
●●
● ●●●
●
●●
●
●●
●●
●●●
●●
●●●
●
●●●
●●
●●●
●
●
●●
●●●●
●
●
●
●
●●●●
●●●●
●●●●●●●●
●●
●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●
●
●
● ●
●●
● ●●
●
●
●● ●●
●●●●●
●
●● ●
●
●
●●
●
●●
●●●
●●●●
●●
●●●●
●
●●
●
●●
●
●
● ●●● ●
●●●
●●●
●
●●●
●●●●
●
●
●●
●
●●●
●
●
●
●
●●
●●
●● ●
●
●●●
●
●
13
●
●●
●
●
●
●●
●
●●●●
●
●
●
●●●
● ●●
●●
●
●
●●
●
●● ●●
●
●
●
●
●●
●
●●
●
●●
●●
●
●●●
●
●●●●
●●
●
●●
●
●
●
●
●●●●●●●
●
●●●
●
●
●
●
●●●
●
●
●●
●
●
●●●●●●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●●●●
●
●●●
●
●
●
●
●●
●
●●●●●●
●●●
●
●●
●
●
●●
●
●
●
●
●
●●●
● ●●
●
●●
●
●
●
●●●
●
●●● ●
●
●
●●●●●
●
●
●
●
●●●●
●
●
●
●●
●●
●
●
●
●
● ●
●●●
●
●
●
●
●
●●
●●
●
● ●
●
●●
●●
●
●
●●
●
●●●
●●
●
● ●
●●●
●●
●
18
●
●
●
●
●●
●
●●
●●●● ●
●
●●
●
●●
●●●
●●●●●
●●
●
●
●
●●
●●
●●●
●
●
●
●●●
●
●●
●●●
●●●●
●●
●
●●
● ●
●
●
●
●●●●●●
●●●
●
●
●●●
●
●
● ●●
●●●
●●
●
●●●●●
●
●●
●
●
●●
●●●
●
●
●●
●●●●●
●
●
●● ●●●
●
●●
●
●●
● ●●● ●●●
●
●●
●
●●
●
●●●
●●●
●
●
● ●
●
●
●
●
●
●
●
●●
●● ●
●
●●
●
●
●●●
●●
●
●
●●
●●●●
●●●
●
●●●
●●●
●
●
●●●
●
●●
●
●●
●
●●●●
●
●
●●
●
●●
●
●
●●
●
●●●●
●●
●●
●
●
●
●● ●
2 3 4 5 6 7
4
●●
●
●
●●●
●●
●
●
●●
●
●
●●
●
●● ●
●●●●●●●
●
●
● ●●●
●●●
●
●●
●
●
●
●●
●
●
●●●●●
●●●
●
●●
● ●
●●
●●
● ●
●●
●●●●
● ●●●
●●●●
●
●
●●
●●●
●●
●●
●●
●●●
●
●● ●●
●●
●
●●
●
●●
●●●
●●●
●●●●●●
●
●●●
● ●
●● ●
●
●●●
●
●
●
●●●
●
●
●●
●●●●●●
●●
●●
●●
●●
●
●
●
●
●
●●●
●
●●
●
●●●●
●●
●●
●●
●
●
● ●●●
●
●
● ●
●●
●●●
●●● ●●
●
●●●●●
●●
●
●
●
●●●
●
●
●●●
●
●
●●
●
●
●●
●
●
●●
●●●
9
●●
●●●
● ●●●
●
●●
●
●
● ●● ●●●●●●●
●●
●
● ●●● ●● ●
●●●
●●●
●●
●
●●
●●
●●●●●
●●●●●●
●●
● ● ●●
●●
●● ●●
●●
●●●●●
●●●●
●● ●●
●●●●
●●●●
●●●
●
●● ●●
●●
●
●
●●
●●
●●
●●
●●
●●
●●●
●●●●● ●
●
●
●
●
●● ●●
●
●●●●●
●
●●●● ●●
●
●
●
● ●●
●
●●●
●
●●
● ●● ●
●●●●
●
●●●●●●
●●●●●●●
●
●
●●
●●● ●
● ●
●●●●●
●
●
●●
●●●●●●●●
●●
●●●
●●●
●●
●
●●●
●●●● ●●●●●●
●
14
●●●●
●●
●●●
●●●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●●
●●
●●
●●
●
●●
●●●●●
●
●
●
●
●
●
●
●● ●
●
●
●●
●● ●●●
●
● ●●●●●
●●●●●
●●●
●
●●
●
●●
●●●
●
●
●● ●
●
●
●
●●●
●
●●
●●●●
●
●
●●
●●
●
●
●
●●● ●
●●●
●
●● ●
●
●
●
●●●
●●●
●
●
●●
●●●
●
● ●
●
●●
●
●●
●
●●●●
●
●
● ●
●●
●●●●
●
●
●
●
●●●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●●●●
●
●
●
●
● ●
●
●
●●
●
●
●
●●
●●●
●
●
●●●●●
●
19
●●●
●
●
● ●●●
●●●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●
●●●
●
● ●●●●
●
●
●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
● ●
● ●
●●●
●
●
●
●●●●●
●
●
●●●●●
●●
●●●
●●
●
●
●
● ●
●●●
●
●●● ●
●
●●
●
●
●● ●
●●
●
●●
●
●●●
●●
●
●●
●
●
●● ●
●●
●
●●
●
●
●
● ●
●
●
●
●
●
●●●
●●●
●●
●
●
●
●
●
●●●
●
●● ●
●●
●
●
●●
●
●
●
●●●●
●●●
●●●
● ●
●
●
●●
●
●
●
●●
●●
●●●
●
●●
●
●
●
●●●●●●
●
●
●●
●
●●
●
●●
●
●●●
●
●
●● ●
●●●
●
●
●
2 3 4 5 6 7
5
●
●
●●
●●
●●●
●●
●● ●
●●
●
●●●
●●
●
●
●●● ●
●●●
●●
●●●●
●
●
●
●● ●
●● ●
●
●
●●
●
● ●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●●
●
●●●
●●●
●●
●
●
●
●
●●●
●●
●
●● ●●
●
●
●●
●
●●
●
●●
●
●
●
●
●●●
●
●●
●
●●●
● ●●●
●
●●
●
●
●
●
●
●
●●●
●●●●
●
●
●
●
●
●
●●
●●●●●
●●
●
●●
●
●
●
●
● ●
●
●●
●
●
● ●
●
●
●●●●
●
●
●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●●
●●
●
●
●●
●●
●●
●
10
●
●●●
●
●
●
●●
●
●
●●
●
●●● ●●●
●●
●
● ●●
●
●●
●● ●
●
●●●●
●●●●
●
●●●
●
●
●●●●●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●●
●●●
●●
●●●●
●●
●
●
●●
●●●●
●●
●●●●●
●
●●●●
●●
●●●●
●●●●
●
●
●
●●●●● ●
●●
●
●
●
●●●
●
●
●●
●
●●
●
●●
●●
●
●●●●
●●●●●● ●●
●●●●● ●●
●
●
● ●
●
●
●●
●
●
●●
●●●
●
●
●●●●●
●●
● ●●
●●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●●
● ●●●
●
●
●●
●●
●
●●●●●
●●
●
15
●
●
●●
●●
●
●
●
●●●● ●
●●
● ●
●●
●
●
●
●
●
●
●
● ●●●
●
●
●●
●
●
●●●
●●
●
●●
●
●
●
●●●●
●
●●
●
●●
● ●
● ●
●●
●
●
●
●
●
●●●●
●●
●
●
●●●
●
●●
●
●
●●
●●●● ●
●●●
●
●
●
●●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●●
●●● ●
●●
●
●
●●●
●
●
●●
●●
●●●
●●
●●
●
●
●
●
●
●●
●
●●
●●●
●●
●
●
●●
●
●
●
●●●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●●●
● ●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●●●
●
●●●
●
●●
20
●●
●
●
●●
●
●●
●●●●
●
●
●● ●
●●
●
●●●
●●
●●
●
●●●
●
●
●●●
●
●●●●
●●●
●●
●●
●
●
● ●●●●
●●
●
●
●
●
●●
● ●
●
●
●
●●●●
●●●
●●
●
●
●
●
● ●
●●● ●
●
●
●●●
●●●
●●●
●
●
●●
●
●
●
● ●●
●
●
●●
●● ●
●●
●●
●●●●●
●
●
●●
●
●
● ●
●
●
●
●●
●
●●●●
●
●
●●●●●●
●●
●
●
●
●●
●●
●
●●●●● ●
●
●
●●
●●
●
●
●
●●
●●●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
● ●
2 3 4 5 6 7
factor(year)● 1999● 2008
mpg$gp100m <- 100 / mpg$ctyqplot(displ,gp100m, data = mpg, colour = factor(year))
mpg_rotate <- resamp(mpg, null_model(gp100m ~ displ))last_plot() %+% mpg_rotate + facet_wrap(~ .n)
Maybe there are fewer bigger cars?
displ
count
5
10
15
20
25
30
35
2 3 4 5 6 7
factor(year)19992008
displ
coun
t
10203040
10203040
10203040
10203040
1
6
11
16
2 3 4 5 6 7
2
7
12
17
2 3 4 5 6 7
3
8
13
18
2 3 4 5 6 7
4
9
14
19
2 3 4 5 6 7
5
10
15
20
2 3 4 5 6 7
factor(year)19992008
qplot(displ, data = mpg, colour = factor(year) geom = "freqpoly", binwidth = 1)last_plot() %+% mpg_perm + facet_wrap(~ .n)
Future work
Methods for more null hypotheses.
Is it possible to guess plausible null hypotheses from the plot specification?