Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015...
Transcript of Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015...
![Page 1: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/1.jpg)
“My job ain’t a job, it’s a damn good time; City to city I’m runnin’ my rhymes”
Mark Allman International Computer Science Institute
Passive and Active Measurement Conference March 2015
Towards Better Internet Empiricalism
![Page 2: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/2.jpg)
Allman
PAM 2002
2
![Page 3: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/3.jpg)
Allman
PAM 2002
3
![Page 4: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/4.jpg)
Allman
Data Sharing Goals
• Broader datasets in everyone’s hands …… leads to more conclusive results
• Broadens participation beyond those who are data affluent
• Offers transparency to reproduce results
• Fosters longitudinal studies by building an aggregate global dataset
4
![Page 5: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/5.jpg)
Allman
PAM 2014
5
![Page 6: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/6.jpg)
Allman
PAM 2014
6
![Page 7: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/7.jpg)
Allman
State of Data Sharing
7
P (S
hare
Dat
a)
Effort + Risk
2002
2015
2015 ??
![Page 8: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/8.jpg)
Allman
Now What?
8
“Some of the old songs I sing often, because they help me to reflect on where I've been and that's important
for me to do - so I don't lose track of where I am going.” —Johnny Cash
![Page 9: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/9.jpg)
Allman
PAM 2015
9
![Page 10: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/10.jpg)
Allman
PAM 2015
• Revisit the broad notion of bringing better science to bear to understand the Internet
10
![Page 11: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/11.jpg)
Allman
Embrace Insight (not numbers)
11
![Page 12: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/12.jpg)
Allman
“I Know A Little”
12
“The bigger the city, well the brighter the lights; The bigger the dog, well the harder the bite”
—Skynyrd
![Page 13: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/13.jpg)
Allman
A Few Things I Know …
• Most TCP connections are short …… but most bytes belong to long connections
• Most paths have little packet reordering …… but some paths reorder many packets
• Most routes are fairly stable …… at least at the timescales of transactions
• Scanning is incessant
• Some countries censor their users
13
![Page 14: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/14.jpg)
Allman
A Few More Things I Know …
• Open DNS resolvers are quite prevalent
• ECN is not used much
• DNS pre-fetching is prevalent
• CDNs carry much of the Internet’s popular content
• Spoofing is possible from a non-trivial fraction of edge networks
• …
14
![Page 15: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/15.jpg)
Allman
Embracing Insight
• These knowns are all established via empirical observation
• And, none of these knowns contain numbers!
15
![Page 16: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/16.jpg)
Allman
Embrace Calibration
16
![Page 17: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/17.jpg)
Allman
“I Know A Little [More]”
17
“There are three kinds of lies: lies,
damned lies, and statistics.”
—???
Networking Addendum:… and traceroute.
![Page 18: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/18.jpg)
Allman
Measurements vs. Reality
• Measurements are an approximation of reality
18
![Page 19: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/19.jpg)
Allman
Measurements vs. Reality
• We well understand that network behavior naturally varies
• e.g., RTT varies over time
!
• But, we also need error bars for our collection and processing!
19
![Page 20: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/20.jpg)
Allman
Measurements vs. Reality
• Before we can get to the insight we need to understand the contours of the approximation
• I.e., we need to calibrate our data collection process
20
![Page 21: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/21.jpg)
Allman
Measurements vs. Reality
21
Median HTTP Flow Sizes - FTTH Network (bytes)0
600
1200
1800
2400
All "Connections" Established Connections
![Page 22: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/22.jpg)
Allman
Calibration
• Sound calibration of data lends credibility to results
• I.e., by developing the difference between network behavior and measurement artifacts
• I.e., by demonstrating carefulness
22
![Page 23: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/23.jpg)
Allman
Calibration Process
• Always validate basic properties in the data
• “realness” of events, timing accuracy, etc.
!
• Calibration is a process, not an initial phase
• Plan to re-visit calibration as necessary
23
![Page 24: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/24.jpg)
Allman
Calibration Process
• Not always appreciated by reviewers …
• Mysterious to me …
!
• Stop beating each other up for being careful!
24
![Page 25: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/25.jpg)
Allman
Embrace Hoarding
25
![Page 26: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/26.jpg)
Allman
How We Work …
• We compartmentalize …
• … a paper
• … a project
• … an internship
• … a semester
• … a student
• … a talk
26
![Page 27: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/27.jpg)
Allman
Circle of Research
27
DATA
![Page 28: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/28.jpg)
Allman
How We Work …
• Compartmentalization is good for organizing ourselves …
• … but suboptimal for data collection
28
![Page 29: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/29.jpg)
Allman
Hoarder’s Circle of Research
29
DATA
![Page 30: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/30.jpg)
Allman
Why Hoard?
• When data is on hand …
• … you can quickly execute on ideas
• … you can develop longitudinal understanding
• … you can act on unexpected events
!
• Anecdotally: colleagues who hoard get more done than colleagues who use purpose-driven data collection
30
![Page 31: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/31.jpg)
Allman
Hoarding Examples
• E.g., CAIDA’s skitter (ark) traceroute data
• E.g., ISI’s Internet census data
• E.g., Paxson’s myriad use of LBL connection summaries
• E.g., Bailey’s dark net data
• E.g., UCSD’s spam feeds
31
![Page 32: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/32.jpg)
Allman
Hoarding• Do not blindly expend effort collecting every
possible thing you can think to collect
!
• If data is cheap and easy to collect, save it.
!
• If you have to expend energy to setup a collection, keep it going
• … even if there is no clear reason
• … even though it will take a bit of (ongoing) effort
32
![Page 33: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/33.jpg)
Allman
Hoarding
• Do not collect data without ever looking at it
• … you’ll some day find you have NO data!
• … simple automated analyses suffice
!
• Don’t delegate hoarding to students
!
• Realize you’ll still need to augment with purpose-driven data collection
• Realize hoarding is sometimes too expensive
33
![Page 34: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/34.jpg)
Allman
Embrace Heterogeneity
34
![Page 35: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/35.jpg)
Allman
Heterogeneous By Metric
35
Czyz, et. al. SIGCOMM 2014
IPv6 Advertisement
IPv6 Traffic
![Page 36: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/36.jpg)
Allman
Heterogeneous By Time
36
Richter, et. al. CCR, Apr. 2015
![Page 37: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/37.jpg)
Allman
Heterogeneous By Path
37Craven, et. al. SIGCOMM 2014
![Page 38: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/38.jpg)
Allman
We All Know A Little
• We all accept heterogeneity …
38
![Page 39: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/39.jpg)
Allman
How Much Is Enough?
39
Internet Measurement Workshop, 2001
![Page 40: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/40.jpg)
Allman
We All Know A Little
• We all accept heterogeneity …
• … right up until the moment we walk into the PC meeting!
40
![Page 41: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/41.jpg)
Allman
But, We Forget A Little
• Among my least favorite review comments …
• “data comes only from one university …”
• “authors watch only one mobile carrier …”
• “the data only encompasses one day …”
• “the user population is ‘small’ …”
!
• “… so, the study is not representative.”
41
![Page 42: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/42.jpg)
Allman
On Representativeness
• These review comments are simultaneously …
• … correct
• … vacuous
• … hypocritical
!
• We need to stop beating each other up for conducting sound empirical work!
42
![Page 43: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/43.jpg)
Allman
Embrace Reappraisal
43
![Page 44: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/44.jpg)
Allman
Validating Results
44
A large percentage of the Internet measurement studies
currently published are not verified by the community due to the inability of researchers to access others' data and
measurement/analysis tools.
Allman, et.al. PAM 2002
![Page 45: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/45.jpg)
Allman
So You Wanna Reappraise?
45
IMC 2005
![Page 46: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/46.jpg)
Allman
So You Wanna Reappraise?
46
Share your data
so the community can validate
your results.
Give me your data so I can use it for
my stuff!
![Page 47: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/47.jpg)
Allman
Why I Know What I Know
• I believe in heavy tails …
• … not because Leland told me
• … or because Willinger told me
• … or because Feldmann told me
• … or because Crovella told me
• … or because Paxson told me
• … but because they all told me!
47
![Page 48: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/48.jpg)
Allman
Why I Know What I Know
• I believe in heavy tails …
• … but not because a bunch of researchers re-produced numbers from some dataset
• … but, rather, because a bunch of researchers arrived at the same insights across datasets!
48
![Page 49: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/49.jpg)
Allman
Reappraisal
• Insights are stronger when they come from multiple …
• researchers
• datasets
• vantage points
• methodologies
!
• Yet, in general we do little reappraisal
49
![Page 50: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/50.jpg)
Allman
We Hate Reappraisal• We are novelty junkies …
• … reappraisal is boring
• … so, reappraisal is unwelcome
• … even when it is!
!
• Reappraisal is not viewed as on par with new contributions
• But, it does strengthen our understanding
• We should stop beating up sound empirical work!
50
![Page 51: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/51.jpg)
Allman
Embrace Risk
51
![Page 52: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/52.jpg)
Allman
Ground Truth
• Ground truth in Internet data is elusive
!
• This makes understanding the efficacy of inferences difficult (or impossible)
52
![Page 53: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/53.jpg)
Allman
Risky Business
53
IMC 2014
![Page 54: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/54.jpg)
Allman
Accepting Risk
• We should be willing to accept some risk
• … or we will never make progress
54
![Page 55: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/55.jpg)
Allman
Risk Mitigation
• We should expect some validation …
• lab-based experiments
• using alternate methodologies to cross-check results
• small amounts of ground truth
55
![Page 56: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/56.jpg)
Allman
Do Not Embrace Sloppiness
• Risk from sloppy work is always unacceptable
56
![Page 57: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/57.jpg)
Allman
Risk Should Not Be …
57
… a crutch for the lazy
…a beating for the diligent
![Page 58: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/58.jpg)
Allman
Embrace The Whole Bag Of Tricks
58
![Page 59: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/59.jpg)
Allman
Bag Of Tricks
59
Insight Calibration
Reappraisal RiskHeterogeneity
Hoarding
• Individual notions have merit
• Also, feed on each other
![Page 60: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/60.jpg)
Allman
Comment #1
60
Insight Calibration Hoarding
• These provide a foundation for solid individual pieces of work
![Page 61: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/61.jpg)
Allman
Comment #2
61
Reappraisal RiskHeterogeneity
• These let us start thinking beyond individual pieces of solid work
• Rather, we lean on a community-based body of work
![Page 62: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/62.jpg)
Allman
Composition #1
62
Calibration
Risk
• Building confidence that we know the difference between network properties and measurement artifacts reduces risk
![Page 63: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/63.jpg)
Allman
Composition #2
63
Calibration Hoarding
• Hoarding with a given measurement apparatus leads to easier calibration
![Page 64: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/64.jpg)
Allman
Composition #3
64
Reappraisal Risk
• Fostering reappraisal reduces the overall risk of any particular study being wrong
![Page 65: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/65.jpg)
Allman
Composition #4
65
Insight
Reappraisal
• Focusing on insight makes reappraisal easier
![Page 66: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/66.jpg)
Allman
Composition #5
66
Reappraisal
Hoarding
• Pack ratting data leads to more opportunities to reappraise
• data is on hand
• Reduces the cost to, for instance, make reappraisal a first student project
![Page 67: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/67.jpg)
Allman
Composition #6
67
ReappraisalHeterogeneity
• Reappraisal allows for a variety of perspectives from a variety of researchers
• Ultimately we have stronger and “more representative” insights
![Page 68: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/68.jpg)
Allman
Keeping Our Eye on the Ball
68
“Some of the old songs I sing often, because they help me to reflect on where I've been and that's important
for me to do - so I don't lose track of where I am going.” —Johnny Cash
![Page 69: Towards Better Internet Empiricalism - ICIR · —Johnny Cash. Allman PAM 2015 9. Allman PAM 2015 ... Hoarder’s Circle of Research 29 A. Allman Why Hoard?](https://reader035.fdocuments.net/reader035/viewer/2022081614/5fc221028ef92960872a7e42/html5/thumbnails/69.jpg)
Allman
Thanks!
• Jelena Mirkovic & the PAM organizers
!
• Mike Bailey, Paul Barford, Rob Beverly, Ethan Blanton, kc claffy, Wes Eddy, Sally Floyd, John Heidemann, Christian Kreibich, Boris Nechaev, Shawn Ostermann, Craig Partridge, Vern Paxson, Matt Roughan, Walter Willinger, …
69