Big Data in NATO and Your Role
-
Upload
jay-gendron -
Category
Data & Analytics
-
view
203 -
download
4
description
Transcript of Big Data in NATO and Your Role
![Page 1: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/1.jpg)
BIG DATA IN NATO: WHAT IT MEANS TO YOU
Jay Gendron
jaygendron
October 29, 2014
![Page 2: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/2.jpg)
U
![Page 3: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/3.jpg)
Our Journey
• BIG DATA– What is BIG?– What is DATA?
• Two Workhorses of Big Data– Enterprise Data Warehousing– Predictive Analytics
• Weaponry Available– Open Source Tools– Open Source Data
• Use Cases• Future Trends
![Page 4: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/4.jpg)
BIG DATA: HOW BIG IS BIG?
![Page 5: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/5.jpg)
Need something small…
20lb copy paper = 0.004” = 0.1 mm
Let 1 byte = 1 paper thickness
Image: http://pencilgrinder.wordpress.com/
![Page 6: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/6.jpg)
01000010 01101001 01100111 00100000 01000100 01100001 01110100 01100001 00100000 01101001 01101110 00100000 01001110 01000001 01010100 01001111 00111010 00100000 01010111 01101000 01100001 01110100 00100000 01101001 01110100 00100000 01001101 01100101 01100001 01101110 01110011 00100000 01110100 01101111 00100000 01011001 01101111 01110101 00101110
This is 39 bytes…or in our thinking…
a pile of paper 39 pages thick
So 1 MB is a million sheets thick…
![Page 7: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/7.jpg)
1 MB
Imagine the front squareCompletely covered with 10,000
sheetsStacked 100 sheets high
That’s under one-half inch thick
…1 GB?
![Page 8: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/8.jpg)
33 feet tall10 meters
1 GB
![Page 9: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/9.jpg)
266 feet tall81 meters8GB
So what does 1 TB look like?
![Page 10: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/10.jpg)
6.3 miles (10 km) high. Plane! Duck!
1 TB
![Page 11: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/11.jpg)
BIG data
There is no technical definition
3V’s =VolumeVelocityVariety
“Big Data is at the heart of modern science and business…the necessity of grappling with Big Data, and the desirability of unlocking the information hidden within it, is now a key theme in all the sciences – arguably the key scientific theme of our times.”
-Francis X. Diebold , University of PennsylvaniaA Personal Perspective on theOrigin(s) and Development of “Big Data”:The Phenomenon, the Term, and the Discipline
(2012)
Laney, D. (2001)
![Page 12: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/12.jpg)
Volume
Large Synoptic SurveyTelescope (LSST)
40TB/day
100+PB in 10-year lifetime
Illumina HiSeq 2000DNA Sequencer
~1TB/day; 30 TB/month
Images: https://d396qusza40orc.cloudfront.net/datasci/lecture_slides/week1/005_escience.pdf
![Page 13: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/13.jpg)
250 miles Exosphere
186 miles Thermosphere
25 miles Mesosphere
6 miles Troposhere
40 TB: 10,000 x 4,000,000,000 sheets high
![Page 14: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/14.jpg)
How Big is the Internet?
Size of the Internet as of 31st Dec 2013
14.3 Trillion - Webpages, live on the Internet.48 Billion - Webpages indexed by Google Inc.14 Billion - Webpages indexed by Microsoft's Bing.672 Exabytes - 672,000,000,000 Gigabytes (GB) of accessible data.
Source: http://www.factshunt.com/2014/01/total-number-of-websites-size-of.html
1 EB = 1,000,000,000,000,000,000 = 1 x 1018
![Page 15: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/15.jpg)
![Page 16: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/16.jpg)
![Page 17: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/17.jpg)
Velocity
43.6 EB = Total Internet Traffic in 2013
![Page 18: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/18.jpg)
Volume + Velocity + Variety
Image: http://www.mediabistro.com/alltwitter/files/2011/06/internet-60-seconds-infographic.jpg
![Page 19: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/19.jpg)
DATA IS…
![Page 20: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/20.jpg)
…that which aids Decision Making
Data is…
![Page 21: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/21.jpg)
The “WebMaster”
Image: http://www.fivem.be/
![Page 22: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/22.jpg)
The “DataMaster”
• Hiring a room of PhD’s won’t solve Big Data
• They have a role…as does IT
• Ultimately Big Data will also be a team effort like the web buildout
• …and You have a role on that team
Image: http://www.fivem.be/
![Page 23: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/23.jpg)
BIG DATA: THE WORKHORSES
![Page 24: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/24.jpg)
Difficult Data is more apt
Images: Elephant - http://www.marcolotz.com/?p=77 Word Cloud - http://www.fotolia.com/id/36647313?by=serie
Enterprise DataWarehousing
PredictiveAnalytics
![Page 25: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/25.jpg)
Enterprise Data Warehousing
• What began with MapReduce in 2004• Evolved in open source like Hadoop
• Permanent contributions of evolution:– Fault tolerance – running on many machines
and accounting for failures– Schema-on-Read – more flexibility in working
with data in different forms– User Defined Functions – giving developers
more freedom in where to place queries
Source: https://class.coursera.org/datasci-002/lecture/15
![Page 26: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/26.jpg)
Predictive Analytics
• Business Intelligence• Statistics• Visualization• Programming• Machine Learning
![Page 27: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/27.jpg)
Image: http://www.ted.com/talks/nate_silver_on_race_and_politics?language=en
Desired End State
![Page 28: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/28.jpg)
A team approach
Predict
BI
Stats
Analyst
Viz
S/W + scale
+ algorithm
+ statistics
+ programming + data products
![Page 29: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/29.jpg)
Impact of the Phenomenon
Images: Galileo - http://www.crystalinks.com/galileo.html; Formulae - https://msschwarzeducationstation.wordpress.com/page/2/; Computers - http://www.utsystem.edu/blog/2011/09/26/ut-austin-awarded-50-million-build-faster-more-powerful-supercomputer; Book Cover - http://radar.oreilly.com/2011/09/building-data-science-teams.html
Theoretical
ComputationalEmpirical +
Empirical
![Page 30: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/30.jpg)
Human-Computer Symbiosis
Sankar, S. (2012, June). The rise of human-computer cooperation [TEDGlobal 2012]. Podcast retrieved from https://www.ted.com/talks/shyam_sankar_the_rise_of_human_computer_cooperation?language=en
![Page 31: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/31.jpg)
People and Culture Count
People are “tinkerers” & “hobbyists”
We ALL have 1 thing in commonWE ARE ALL DIFFERENT!
ExperiencePerspective
![Page 32: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/32.jpg)
THE WEAPONRY: OPEN SOURCES
![Page 33: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/33.jpg)
Source: Sankar (2012). https://www.ted.com/talks/shyam_sankar_the_rise_of_human_computer_cooperation?language=en
![Page 34: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/34.jpg)
Repurposing Data
![Page 35: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/35.jpg)
Open Source Tools
![Page 36: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/36.jpg)
Visual Text Analytics
Image: http://hermeneuti.ca/voyeur/tools
![Page 37: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/37.jpg)
Open Datasets
![Page 38: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/38.jpg)
Again…team approach
SCIENCE
•Stats•S/W
DOMAI
NS
•SME•BI•Analyst
INFO TECH
•DBA•VizT TT TTT
![Page 39: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/39.jpg)
Skill Building
• Free Courses
• Meetups
• Hackathons
• Podcasts
Image: http://www.amazon.com/Data-Science-Business-data-analytic-thinking/dp/1449361323
![Page 40: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/40.jpg)
…to make the point
Fawcett, T. & Provost, F. (2013, August 9). Data Science for Business. Retrieved from https://www.safaribooksonline.com/library/view/data-science-for/9781449374273/.
…on page 13
![Page 41: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/41.jpg)
DATA USE CASES
![Page 42: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/42.jpg)
Remember…You have a role
Informal poll at Univ of WA:
How much time do you spend “handling data” as opposed to “doing science”?
Most given response: 90%
![Page 43: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/43.jpg)
Image: http://www.ibmbigdatahub.com/infographic/infographic-big-data-exploration
![Page 44: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/44.jpg)
Image: http://www.ibmbigdatahub.com/infographic/infographic-enhanced-360-view-customer
![Page 45: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/45.jpg)
Image: http://www.ibmbigdatahub.com/infographic/security-intelligence-extension
![Page 46: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/46.jpg)
Fraud Detection
Image: http://www.ibmbigdatahub.com/infographic/countering-fraud-big-data-world
![Page 47: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/47.jpg)
Human-Computer Symbiosis
Sankar, S. (2012, June). The rise of human-computer cooperation [TEDGlobal 2012]. Podcast retrieved from https://www.ted.com/talks/shyam_sankar_the_rise_of_human_computer_cooperation?language=en
![Page 48: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/48.jpg)
Image: http://www.ibmbigdatahub.com/infographic/operations-analytics
![Page 49: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/49.jpg)
Data Warehouse Augmentation
Existing + NEW
Operational Efficiencies
Images: Arrow - http://canadawebservices.com/7-powerful-ways-increase-website-traffic/; Explore - http://www.keadventure.com/page/explore_more!.html
More Data
![Page 50: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/50.jpg)
Leveraging Use Cases
Business and Commerce
• Big Data Exploration
• 360 Degree View
• Security & Intelligence• Operational Analysis
• Data Warehouse Augmentation
NATO Enterprise• FMN (“experiment”
and data mining)• Enterprise (supporting
commands)• Cyber Defence (threat
tactics)• C2 (requirements text
analysis)• Ent. Architecture &
Technology (req’ts)
![Page 51: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/51.jpg)
FUTURE TRENDS
![Page 52: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/52.jpg)
According to Gartner
Image: http://www.forbes.com/sites/gilpress/2014/08/18/its-official-the-internet-of-things-takes-over-big-data-as-the-most-hyped-technology/
![Page 53: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/53.jpg)
According to IBM
• More Analytics – Less Gut• Data security and privacy• Leaders with data knowledge• Data-centric applications• Integrating internal and external• Investments in platforms
Cho, I. (2013, February 3). 6 trends in big data and analytics [IBM Big Data Hub]. Podcast retrieved from http://www.ibmbigdatahub.com/podcast/6-trends-big-data-and-analytics
An example
![Page 54: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/54.jpg)
Visual Analytics
Koblin, A. (2011, March). Visualizing ourselves ... with crowd-sourced data [TED2011]. Podcast retrieved from http://www.ted.com/talks/aaron_koblin?language=en
![Page 55: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/55.jpg)
Art and science meet
Miebach, N. (2011, July). Art made of storms [TEDGlobal 2011]. Podcast retrieved from http://www.ted.com/talks/nathalie_miebach?language=en
![Page 56: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/56.jpg)
Summary
• Big Data – has implications• Big Data = Data + Analytics• Open Source – tools and data• Use Case – leverage others’ results• Future– More analytics and applications– Need for data fluency among managers– Need processes to encourage exploring
![Page 57: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/57.jpg)
You know WHAT
![Page 58: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/58.jpg)
…the SO WHAT
![Page 59: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/59.jpg)
NOW WHAT?
![Page 60: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/60.jpg)
Empower
Make apps
Find data
![Page 61: Big Data in NATO and Your Role](https://reader038.fdocuments.net/reader038/viewer/2022103018/558d31ccd8b42ad5268b4738/html5/thumbnails/61.jpg)
ReferencesCho, I. (2013, February 3). 6 trends in big data and analytics [IBM Big Data Hub]. Podcast retrieved from
http://www.ibmbigdatahub.com/podcast/6-trends-big-data-and-analytics.
Diebold, F.X. (2012, November 26). A personal perspective on the origin(s) and development of “big
data”: The phenomenon, the term, and the discipline. Retrieved from
http://www.ssc.upenn.edu/~fdiebold/papers/paper112/Diebold_Big_Data.pdf.
Fawcett, T. & Provost, F. (2013, August 9). Data Science for Business. Retrieved from
https://www.safaribooksonline.com/library/view/data-science-for/9781449374273/. (on page 13)
Howe, B. (2013). Data science in science [PDF document]. Retrieved from Lecture Notes Online Web site:
https://d396qusza40orc.cloudfront.net/datasci/lecture_slides/week1/005_escience.pdf.
Koblin, A. (2011, March). Visualizing ourselves ... with crowd-sourced data [TED2011]. Podcast retrieved
from http://www.ted.com/talks/aaron_koblin?language=en.
Laney, D. (2001), 3-D data management: Controlling data volume, velocity and variety. META Group
Research Note, February 6. Retrieved from http://
blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf
.
Miebach, N. (2011, July). Art made of storms [TEDGlobal 2011]. Podcast retrieved from
http://www.ted.com/talks/nathalie_miebach?language=en.
Sall, E. (2013, February 12). Top 5 big use cases [IBM Big Data Hub]. Podcast retrieved from http://
www.ibmbigdatahub.com/podcast/top-5-big-data-use-cases.
Sankar, S. (2012, June). The rise of human-computer cooperation [TEDGlobal 2012]. Podcast retrieved
from
https://www.ted.com/talks/shyam_sankar_the_rise_of_human_computer_cooperation?language=en.